Survey signals — ask users, then train the ranker on their answers

Engagement logs tell us only what users did. A survey asks users directly. It asks "Was this post worth your time?" or "How well does this video match your interests?" and uses the answer as a label to change the ranker.

The problem comes down to one point: survey responses are sparser than engagement logs by four or more orders of magnitude.

"the model has a limited number of labeled positive and negative examples for user-item pairs, accounting for less than one ten-thousandth of the total video exposures"

verified — USM (TikTok), arXiv:2412.10674, 2024. It is the only primary source that gives survey coverage as a number.

Meta says the same thing but only qualitatively: "User perception feedback collected using surveys are extremely sparse" verified engineering.fb.com, 2026-01-14.

Data this scarce cannot train a ranker directly. So every published system has an intermediate mechanism that converts the survey into a dense signal. The sentence "we reflect surveys in ranking" explains nothing. What that intermediate mechanism is makes up the substance of this note.

Four approaches — how sparse surveys move the ranker

This note divides the ways sparse surveys move the ranker into four approaches. Each approach puts the survey in a different role. The role is a weight outside the model, a model that predicts survey responses, a light layer on top of existing predictions, or a head inside the weighted sum verified (sections ①–④ below).

The evidence for each item is quoted one by one in the sections below verified.

① is the only approach that leaves the model untouched. The other three all work around sparsity by building a "function that predicts survey responses."

The difference between ② and ③ matters in particular. ② predicts the survey anew from raw features. ③ adds one light layer on top of engagement predictions that are already trained. ③ converges with far fewer labels inferred (reason: the input dimension shrinks to about the number of heads, and the main model has already finished representation learning).

① Set the weights from surveys — Meta 2021

In its 2021 News Feed ranking post, Meta wrote that it sets how much each prediction counts according to the actions that people say in surveys are more meaningful and worth their time verified.

"the way we take each prediction into account for Juan is based on the actions that people tell us (via surveys) are more meaningful and worth their time" "People with higher correlation gain more value from that specific event, as long as we make this method incremental and control for potential confounding variables."

verified — engineering.fb.com, 2021-01-26

The survey is the answer to how w is chosen in the value model V = Σ w·Y. For understanding the mechanism, the second sentence matters more: what Meta means by "correlation" is measuring the correlation between survey responses and each engagement event to adjust the weights inferred. In other words, the survey serves not as a label but as a reference point for weight calibration. The formula, the correlation coefficient, and the update cycle are not public.

The formula and head structure are in multi-task-ranking.md, and the Meta context is in facebook.md.

② Build a model that predicts survey responses — YouTube 2021

YouTube collected surveys in which users rate videos they watched with stars, and trained a model that predicts those responses for every user verified. It is the oldest and most widely known approach.

• The survey has users rate the video they watched with stars: "rate the video you watched from one to five stars" verified
• Follow-up questions come after both low and high scores verified.
  • A score of 1–2 triggers a question about why it was low.
  • A score of 4–5 triggers this question: "was it inspirational or meaningful?"
• valued watchtime is watch time counted only for watches that received 4–5 stars verified.
  • YouTube's definition is: "the time spent watching a video that you consider valuable"
• The answer to sparsity is a model that predicts survey responses for all users verified.
  • YouTube wrote: "we trained a machine learning model to predict potential survey responses for everyone"
• Whether predictions match actual responses is monitored continuously with a holdout verified.
  • It wrote that it deliberately leaves some responses out of training: "purposely hold back some of the survey responses from the training"

verified — On YouTube's recommendation system, Cristos Goodrow, 2021-09-15

Structurally, this is the boldest choice. It does not use the survey as an auxiliary signal; it changed the optimization target itself into the survey prediction (watchtime → valued watchtime). In exchange, if the proxy model is wrong, the whole objective is wrong. This appears to be why YouTube states the holdout monitoring explicitly inferred.

The model structure and where it is fed in (which head of the ranker, and at what weight) do not appear in the blog post observed 2026-08-09. The corpus examined is one YouTube blog post body quoted in this section.

③ Add an alignment layer on top of existing predictions — Meta UTIS, 2026

Meta trained a light alignment layer on survey responses that takes as input the engagement predictions the main ranking model already produces verified. The survey is a single-question survey shown in a randomly chosen subset of watch sessions each day, and responses are binarized for training verified.

verified — Adapting the Facebook Reels RecSys AI Model Based on User Feedback, engineering.fb.com, 2026-01-14. This post (the UTIS blog post) is the primary source that most concretely discloses the mechanism of survey signals among the sources this repository found.

UTIS stands for User True Interest Survey. The survey asks on a 1–5 scale how well the video matches the user's interests verified (UTIS blog post).

"Daily, a certain proportion of users viewing sessions on the platform are randomly chosen to display a single-question survey asking, 'To what extent does this video match your interests?' on a 1-5 scale."

The core mechanism is contained in these three sentences verified (UTIS blog post):

"User perception feedback collected using surveys are extremely sparse but such feedback can be generalized in large scale recommendation systems using our novel model 'Perception Layer' architecture that uses existing event predictions as additional features."

"We trained a lightweight UTIS alignment model layer on the collected user survey responses using existing predictions of the main model as input features."

"The survey responses used to train our model were binarized for easy modelling and denoises variance in responses."

This structure is the answer to the sparsity problem. The survey model's input is not raw features but the engagement prediction vector the main ranking model already produces verified (UTIS blog post). Because the input dimension shrinks to about the number of heads, training works even with tens of thousands of labels inferred.

Also, it binarizes the 1–5 scale to suppress response variance verified (UTIS blog post). It gave up 5-level regression and chose to reduce noise.

UTIS is used in two places verified (UTIS blog post).

• In the Late Stage Ranking (LSR) stage, UTIS is not a head but an additional input term of the value formula.

  "UTIS is deployed in parallel to the LSR model, providing an additional input feature into the final value formula."

• At the Early Stage / retrieval stage, it uses two-step distillation. It builds the LSR's UTIS predictions from surveys, then uses those predictions as labels to align the retrieval model.

  "large sequences based on user-to-item retrieval models are aligned using knowledge distillation based objectives trained on UTIS predictions from LSR as labels."

  Survey responses do not go directly to retrieval.

The UTIS blog post reported three offline metrics together with the A/B results on 10M+ users verified.

⚠ Read this metric with care. "high survey ratings +5.4%" means high-rating responses rose by 5.4% (low ratings fell by 6.84%), and this model was trained to predict exactly that survey. The objective and the evaluation metric measure the same thing. The evidence that breaks out of this loop is that engagement +5.2% rose along with it. This means survey optimization did not reduce engagement inferred.

⚠ The offline baseline is not disclosed. The post does not say whether the 59.5% accuracy is compared against "the main model without the survey head" or against "majority-class prediction" observed 2026-08-09 (one UTIS blog post body).

④ Add it as a head, but correct for "who answers" — TikTok USM, 2024

TikTok USM puts the survey response prediction into the weighted sum of the final score alongside the engagement heads verified. It also predicts separately the probability that a user answers the survey, and corrects the response bias that comes from who answers verified.

verified — USM: Unbiased Survey Modeling for Limiting Negative User Experiences in Recommendation Systems, TikTok, 2024

• There are two kinds of surveys verified.
  • Satisfaction is a 3-point scale: "I like it" / "Neither like nor dislike it" / "I don't like it"
  • Inappropriate has users choose a reason: sexually suggestive, disgusting, hateful, violent, spammy, uninteresting
• The survey prediction enters the weighted sum alongside the engagement heads verified. It is the only case where the combination formula is public verified.
  • The combination formula is final_s(item) = Σ wᵢ·p(surveyᵢ) + other_s(item).
• The key contribution is response-bias correction verified. A separate survey-submit model predicts the response probability, and applies IPW (a correction that weights samples by the inverse of the response probability) to samples with 1/P(ans|ss) verified.
  • The starting point is this sentence: "users' willingness to answer and submit the survey is not uniform"
• As online results, reports, dislikes, and the survey inappropriate rate all fell verified.
  • reports −1.75%, dislikes −2.57%, survey inappropriate rate −2.06%

This section alone covers a problem that ①②③ do not address. People who answer surveys are not a random sample. If angry people answer more often, a survey model overestimates dissatisfaction. Neither the Meta UTIS post nor the YouTube blog post mentions this correction observed 2026-08-09 (corpus checked: two blog post bodies).

↑ Contents

X — a survey prompt exists, but no path to ranking is visible

X actually shows a survey prompt in the For You timeline. However, in the public code we cannot find a connecting path that delivers those responses to the ranker. We cannot say whether X uses surveys in ranking; we can say only that what is public is the collection mechanism inferred.

The public code has a mechanism that shows the survey prompt and counts impressions verified (list below). The the-algorithm items are as of the 2025-09-03 commit (c54bec0) of twitter/the-algorithm.

• For You has a candidate pipeline dedicated to the survey prompt verified — the-algorithm, ForYouRelevancePromptCandidatePipelineConfig.scala.
  • The candidate ID is the string literal "feed-survey-prompt" (:55).
• Client events define this prompt's component name and action verified — the-algorithm, ClientEventsBuilder.scala:54,60,113-114.
  • The component name is "for_you_survey_feed", and the action is "served_prompts".
• The prompt UI consists of a title, three buttons (positive / negative / neutral), a free-text follow-up input, and four display types verified — the-algorithm.
  • The evidence files are RelevancePromptFollowUpFeedbackType.scala and RelevancePromptDisplayType.scala.
  • The free-text follow-up input is RelevancePromptFollowUpTextInput.
  • The display types are Normal / Compact / Large / ThumbsUpAndDown.
• The impression-frequency cap and the insertion position are stated explicitly as parameters verified — the-algorithm, ForYouParam.scala:568-585.
  • The minimum insertion interval for_you_relevance_prompt_min_injection_interval_minutes has default 1440 minutes = 24 hours (min 0 / max 144000 minutes ≈ 100 days).
  • The insertion position for_you_relevance_prompt_position has default 15.
• The master switch is off by default, and the question and button text are not in the repo verified — the-algorithm.
  • The master switch for_you_relevance_prompt_enable is default false (:562-566).
  • All four question and button texts are default "" (:587-609).
  • This matches this repo's consistent pattern (values at runtime, only the shape in code).
• The 2026 Rust generation still requests this prompt verified — xai-org/x-algorithm, home-mixer/sources/prompts_source.rs.
  • PromptsSource sends PromptType::RELEVANCE_PROMPT in supported_prompt_types.

Three things are absent from the public code, and the conclusion of this section rests on them verified (list below):

1. Home timeline responses are not structured into a training signal stream (UUA) verified.
   • BaseFeedbackSubmitClientEvent branches on eventNamespace.page, handles only case "search" and discards the rest with case _ => None (:16,39).
   • Only two modules on the search results page are structured. relevance_prompt_module is structured as TweetRelevantToSearch, and did_you_find_it_module as DidYouFindItSearch.
2. Home-side logging counts only "how many were shown" verified — ClientEventsBuilder.scala:113-114.
   • Because eventValue = count(relevancePrompt), what is recorded is an impression count, not the response content.
3. The ranker has no survey head verified — multi-task-ranking.md.
   • Phoenix is X's Transformer ranker, and Navi is the production heavy ranker of the 2025 Scala generation.
   • We checked 19 public Phoenix checkpoints, 22 production ones, and 15 for 2025 Navi.
   • No list has a survey-family head.

X appears to use surveys for product metrics or manual analysis rather than as ranking labels, or the consumption path lies outside the public scope inferred. The evidence is the combination of the three points above. We cannot conclude either that "X uses surveys in ranking" or that it does not. We can say only that what is public is the collection mechanism alone.

The surface and ranker context is in x.md.

↑ Contents

Netflix — explicit ratings, not surveys, plus an onboarding taste survey

Netflix's public material covers not a random-sample survey but a product control that every user can press at any time (stars → thumbs → two thumbs up) inferred. Its structure differs from the topic of this note. It is sparse but not random, and the nature of its response bias is different too.

The primary sources below were obtained in the 2026-08-09 update. They are promoted from "no primary source found" in the earlier version.

• The 2017 announcement of the switch from stars to thumbs defines the meaning of the buttons as ranking behavior verified — Goodbye Stars, Hello Thumbs, about.netflix.com.
  • Thumbs up means "recommend more like this," and thumbs down means "stop recommending."
  • Rating activity rose 200% compared with stars. It is a measurement that the coarser the scale, the more responses it yields.
  • It is a product-level choice in the same direction as UTIS's binarization and YouTube's collapsing to 4–5 points.
• The same announcement explains % Match as a display score computed "from viewing habits and behavior analysis alone". We read this as a separation: explicit ratings are an input to recommendation, while the percentage on screen is computed from viewing behavior alone inferred. The original text does not state this separation directly.
• The 2022 Two Thumbs Up makes "really liked it" a separate signal and reflects it more strongly in recommendations verified — announcement.
  • It restores a 3-level explicit scale (5-level stars → binary → 3-level).
• The explicit taste survey (onboarding title picker), where users pick titles they like at sign-up, runs from the 2015 paper to current help documents verified — TMIS 2015 §5.5, help.netflix.com.
  • It is optional, and if skipped, the user starts with a "diverse and popular" default set.
  • It is an elicitation approach (collecting preferences by asking directly) limited to cold start.
  • It is cold-start elicitation that belongs to none of this note's four approaches. ② applies a proxy model that predicts survey responses to all impressions, so it differs from this.

There is still no primary source that uses a random-sample survey (UTIS-type) as a ranking label inferred. The corpus inspected is the recommendation-related posts on netflixtechblog.com and about.netflix.com (read 2026-08-09), and it is not evidence that undisclosed material has none.

Because the reward function r=f(play, complete, thumb) in the 2024 long-term satisfaction blog post includes thumb as a term, it is confirmed that explicit ratings are used in ranking verified — netflix.md.

Similar features (Meta's Show more/less, X's not_interested) are properly classified as user controls, not surveys. They are covered in the "User controls" section of facebook.md.

↑ Contents

Inconsistency between Meta's own documents — Reels

The UTIS blog post names Facebook Reels, but the word "survey" never appears in the Facebook Reels system card. The system card is Meta's public regulatory-compliance document. The 10 prediction tasks the card lists (watch time, full-screen entry, completion, skip, like, hide, "see less" request, session start, move to next reel, etc.) are all engagement or negative actions, and there is no survey item.

observed 2026-08-09 — The corpus checked is one body of the fb-reels system card (last updated 2026-06-19), in which the string survey appears 0 times. The comparison target is the 2026-01-14 post of the UTIS blog post.

⚠ Note that citations of the card have a validity period. It is covered in the "Validity period of system card citations" section of facebook.md.

Two interpretations are possible, and there is no evidence to tell them apart inferred.

• One reading is that UTIS is not a head but an additional input term of the value formula, so it is not in the "prediction tasks" list. It is consistent with the blog's wording.
• Another reading is that the system card has not yet reflected it as of its update date.

This is a real-world case of this repository's rule. The engineering blog is ahead of the regulatory disclosure, and only combining both documents reveals the whole structure. If the regulatory document were treated as canon, UTIS would not be visible at all.

↑ Contents

What generalizes from this case

A statement about survey signals is informative only if it says where in the ranker the survey enters. All three services that disclosed a fix for sparsity built a function that predicts the survey, and TikTok alone also corrects response bias.

• The sentence "surveys are used in ranking" carries no information. The four approaches are entirely different systems.
  • The four approaches are ① setting weights, ② building a proxy model, ③ adding on top of existing predictions, and ④ adding as a head.
  • A statement that does not say which one is not worth quoting.
• The fix for sparsity is, in the end, "a function that predicts the survey." When labels are under 0.01%, direct training is impossible, so the survey is converted into a dense prediction that can be applied to all impressions, and that is fed into ranking. The three services independently reached the same form.
• Taking the main model's output as the input requires far fewer labels. This is the core of Meta UTIS, and this choice is what makes a "lightweight layer" possible.
  • Conversely, training the proxy model anew from raw features (the YouTube approach) needs more data, and the whole objective depends on that model.
• Only one service handles response bias. Only TikTok corrects submit propensity (the probability of submitting a survey) with IPW. When comparing survey-based metrics, check whether this correction exists.
• Survey optimization results are reported in survey metrics, so they are circular. The only evidence that escapes the circle is whether a different kind of metric, such as engagement or integrity, improved at the same time.
• The 1–5 scale is not used as is. Meta binarizes it, and YouTube counts only 4–5 points as valued. Both discard the ordering information of the five levels.
• A survey prompt in the code does not mean it is a ranking signal. X is exactly that case. The collection mechanism and the consumption path must be checked separately.

↑ Contents

Who uses this

Among the services these notes cover, Meta, YouTube, and TikTok disclosed even how they use surveys in ranking verified (Four approaches). X disclosed only the collection mechanism, and Netflix's primary material covers explicit ratings, not random-sample surveys inferred (X section, Netflix section).

• Facebook / Meta disclosed the most detail about running surveys verified — facebook.md.
  • It introduced surveys in 2019-05: close-friends list, "worth your time", Page interest, Group importance.
  • In 2021-01 it disclosed the basis for setting the weights w.
  • In 2021-04 it expanded to inspirational posts, topic interest, and posts with many angry reactions.
  • In 2026-01 it released UTIS.
  • The surface context and the value model are in facebook.md.
• X actually has a pipeline that serves feed-survey-prompt in For You, and this pipeline remains in the 2026 generation verified (collection) — x.md.
  • However, the path by which responses reach ranking is not in the public code inferred (no consumption).
• YouTube trains a proxy model on 1–5 point surveys and uses valued watchtime as the objective. This repository has no service note for it verified.
• TikTok puts a survey head directly into the weighted sum and corrects response bias with IPW. This repository has no service note for it verified.
• Netflix has no primary source for random-sample surveys, but its primary sources on explicit ratings are the most thorough verified — netflix.md.
  • Primary sources exist for the following three.
    • thumbs (2017, activity up 200%)
    • two thumbs(2022)
    • onboarding title picker (2015 to current)
  • It is also confirmed that thumb is a term of the reward function.

↑ Contents

Not yet verified

This note could not confirm, from the sources it read, the internal structure and weights of UTIS, Meta's survey sampling rate and whether it corrects response bias, or where YouTube's proxy model is injected.

• Meta's survey sampling rate. The only public expression is "a certain proportion of users viewing sessions". There is no Meta figure corresponding to TikTok's < 1/10,000.
• The actual structure of the UTIS alignment layer. The number of layers, how many prediction heads are used as input, and the training cycle are not in the blog post.
• The weight when UTIS enters the value formula. Meta has never disclosed actual values of w in any document (facebook.md).
• Whether Meta corrects response bias (who answers the survey). It is not mentioned at all.
• Where the YouTube proxy model enters the ranker. We could not confirm whether it is a head or a separate model, or whether valued watchtime reaches retrieval.
• The "correlation" methodology in the 2021 Meta blog post. We could not confirm how the correlation between survey responses and engagement events is measured or how often it is updated.
• Whether Instagram / Threads run surveys. The two service notes (instagram.md, threads.md) have no primary source saying surveys are used in ranking. The weight-setting methods for Instagram that instagram.md summarizes are offline replay·tuning and Bayesian optimization.

↑ Contents

Sources

• Adapting the Facebook Reels RecSys AI Model Based on User Feedback — 2026, blog post (engineering.fb.com). It covers UTIS and is the primary source that most concretely discloses the mechanism of survey signals.
• How machine learning powers Facebook's News Feed ranking algorithm — 2021, blog post. It contains the only statement that surveys set the value model weights.
• Using Surveys to Make News Feed More Personal — 2019, blog post (newsroom). Introduced four kinds of surveys.
• Incorporating More Feedback Into News Feed Ranking — 2021, blog post (newsroom). Expanded to inspirational, topic, and angry.
• Facebook Reels system card — updated 2026-06-19, regulatory-compliance document. It mentions surveys 0 times.
• On YouTube's recommendation system — 2021, blog post (Cristos Goodrow). Explains the 1–5 point survey, valued watchtime, and the proxy model.
• USM: Unbiased Survey Modeling for Limiting Negative User Experiences in Recommendation Systems — 2024, paper (TikTok). The only source with the survey coverage figure and the combination formula. We read the HTML version, not the PDF.

Netflix (primary)

• Goodbye Stars, Hello Thumbs — 2017, announcement. Covers the definition of what thumbs mean, the 200% increase in activity, and % Match.
• Two Thumbs Up, Even Better Recommendations — 2022, announcement.
• The Netflix Recommender System (TMIS) §5.5 — 2015, paper. Describes the onboarding title picker.
• How Netflix's recommendations system works — current help document. Confirmed that the picker still exists.

Secondary (reference only — demotion lifted now that primary sources are secured)

• Variety — Netflix Launches 'Two Thumbs Up' · TechCrunch — 2022, reporting. Used only for details absent from the primary sources, such as the timing of the global test in the second half of 2021.

Public source code

• twitter/the-algorithm — commit c54bec0 of 2025-09-03. ForYouRelevancePromptCandidatePipelineConfig.scala, ForYouParam.scala:562-609, ClientEventsBuilder.scala:54-114, BaseFeedbackSubmitClientEvent.scala, ItemTypeFilterPredicates.scala, RelevancePrompt*.scala.
• xai-org/x-algorithm — 2026, home-mixer/sources/prompts_source.rs.

↑ Contents