2026-09-19

1Ads ranking training splits delayed feedback into three streams — X

Delayed feedback refers to the condition where training labels arrive late in time. It arises when conversions observed over several days after an ad impression (purchase, install, lead form completion, etc.) are treated as training labels. In Gen4+6 (2026-08-21), X expanded ad conversion actions by 20 and first added three config flags that address this problem. The three flags are condition_conversion_on_click, train_view_through_heads, and split_head_training_by_source. In Gen4+23 (2026-09-17), the delay window was subdivided into 3 tiers (original, tier 2, tier 3). This commit (Gen4+24) unified these three flags into a single ads_head_masking flag and three sample-source stream IDs.

• The ads ranking training pipeline redefined delayed feedback as 3 streams along the sample-source axis.
• Phoenix (the ranking Transformer of the X For You feed) got two new channels in its wire proto: time-window counts of the user's ads engagement history, and a search lexical match.
• The safety pipeline of grox (a Grok-based nearline classifier) got a new spam-clone detection flow. It compares a reference video bundle with candidate videos using a VLM (a model that takes images and text together as input).

Many small cleanups also came with it.

A. The delayed feedback flag set is merged into one, and training samples are split into three streams

Three flags that handled delayed feedback in the ads ranking training config were removed, and one flag took their place. Turning this flag on determines, per head, which stream's samples it trains on.

• This change is in the following files verified — phoenix/xrex/models/recsys_model.py (+50/−55), new phoenix/xrex/models/ads_head_masking.py (67 lines), phoenix/xrex/data/recsys/constants.py.

• In the model config, all three delayed-feedback fields were deleted and one field, off by default, was added verified.

  • The three fields condition_conversion_on_click, train_view_through_heads, and split_head_training_by_source were deleted from RecsysAggregatedModelConfig.
  • The field that took their place is the single ads_head_masking, declared as ads_head_masking: bool = False.

• Three stream ID constants indicating the sample origin were newly added verified.

  FRESH_STREAM_ID = 0
  EARLY_RELABEL_STREAM_ID = 1
  LATE_SPLIT_STREAM_ID = 2
  

• When the flag is on, a new function assembles a per-head loss mask verified.

  • When ads_head_masking=True, the loss mask is assembled by the function ads_head_masking_factor(targets, source_id, num_actions).
  • The function returns a real-valued tensor of size (batch, seq, num_actions). Each action index is turned on by the rules below.

• The stream used for training differs by head type verified.

  • Engagement heads are num_actions minus the CT, VT, and twin indices, and are 1 only on FRESH_STREAM_ID samples.
  • CT heads (click-conditioned, CLICK_CONDITIONED_ACTION_INDICES) are 1 if there was a click in EARLY_RELABEL_STREAM_ID samples.
  • VT heads (view-through, VIEW_THROUGH_ACTION_INDICES) are 1 if there was no click in EARLY_RELABEL_STREAM_ID samples.
  • Twin heads (late-window delayed, new list ADS_LATE_WINDOW_TWIN_HEAD_INDICES) are 1 only for LATE_SPLIT_STREAM_ID samples that had a click and have no early counterpart.

• Twin heads are the 10 base tier delayed conversions introduced in Gen4+6 verified — phoenix/xrex/data/recsys/ads_late_window.py (29 lines, new).

  • The 10 names are ADS_ATTRIBUTED_KEY_CLICK_CONVERSION_DELAYED, ADS_ATTRIBUTED_CLICK_CONVERSION_DELAYED, ADS_PURCHASE_CONVERSION_DELAYED, ADS_MID_FUNNEL_CONVERSION_DELAYED, ADS_ADD_TO_CART_CONVERSION_DELAYED, ADS_UPPER_FUNNEL_CONVERSION_DELAYED, ADS_WEB_CONVERSION_DELAYED, ADS_SEARCH_CONVERSION_DELAYED, ADS_SIGN_UP_CONVERSION_DELAYED, and ADS_CHECKOUT_INITIATED_CONVERSION_DELAYED (action index 204-213, introduced in Gen4+6).

• This commit attached the twin logic to only these 10 base tier types verified. Tier 2 (214-223) and tier 3 (224-233), introduced yesterday (2026-09-17) in Gen4+23, have not yet been added to this list.

• For each twin head, a fresh-stream counterpart that suppresses it was defined together verified.

  • The ADS_LATE_WINDOW_TWIN_SUPPRESSOR dictionary, introduced at the same time, maps each twin head to the index of its suppressor, the fresh-stream counterpart.

    ADS_LATE_WINDOW_TWIN_SUPPRESSOR = {
        ADS_ATTRIBUTED_KEY_CLICK_CONVERSION_DELAYED: ADS_ATTRIBUTED_KEY_CLICK_CONVERSION,
        ADS_ATTRIBUTED_CLICK_CONVERSION_DELAYED:     ADS_ATTRIBUTED_CLICK_CONVERSION,
        ADS_PURCHASE_CONVERSION_DELAYED:             ADS_PURCHASE_CONVERSION,
        ADS_MID_FUNNEL_CONVERSION_DELAYED:           ADS_MID_FUNNEL_CONVERSION,
        ADS_ADD_TO_CART_CONVERSION_DELAYED:          ADS_ADD_TO_CART_CONVERSION,
        ADS_UPPER_FUNNEL_CONVERSION_DELAYED:         ADS_UPPER_FUNNEL_CONVERSION,
        ADS_WEB_CONVERSION_DELAYED:                  ADS_ATTRIBUTED_CLICK_CONVERSION,
        ADS_SEARCH_CONVERSION_DELAYED:               ADS_SEARCH_CONVERSION,
        ADS_SIGN_UP_CONVERSION_DELAYED:              ADS_SIGN_UP_CONVERSION,
        ADS_CHECKOUT_INITIATED_CONVERSION_DELAYED:   ADS_CHECKOUT_INITIATED_CONVERSION,
    }
    

    — phoenix/xrex/data/recsys/ads_late_window.py, 2026-09-18

• If the fresh stream's counterpart conversion is already captured for the same (user, ad) sample, the late-split twin head is excluded from training to avoid double counting verified.

  • Inside ads_head_masking_factor, it looks up this suppressor with no_early_counterpart = 1 - targets[:, :, suppressor_index] and multiplies the late-split twin head mask by it.

• The delayed head for web conversion uses the attributed-click conversion as its suppressor, not the web conversion verified. This appears to mean that a web conversion is treated as a delayed observation of an attributed-click conversion inferred.

  • The counterpart of ADS_WEB_CONVERSION_DELAYED is mapped to ADS_ATTRIBUTED_CLICK_CONVERSION, not ADS_WEB_CONVERSION.

• The metric slices were also redefined verified. A new slice looks at samples in the late-split stream that have a click and no early counterpart.

  • build_metric_masks lost the two parameters condition_conversion_on_click and split_head_training_by_source and now takes only ads_head_masking.
  • The new slice is late_split_no_early (= mask × late × click × no_early). The existing fresh, delayed_clicked, and delayed_non_clicked are kept.

Why it matters. This is a decision to handle delayed feedback labeling explicitly as three streams in the production training pipeline. The delayed feedback literature since Chapelle 2014 has mostly been discussed along the axis of "wait for the label observation window vs. keep updating the prediction". X is closer to the latter: it names three observation points, "fresh (immediate), early (short window), late (long window)", each as a sample source.

The earlier practice of turning off view-through heads by default with train_view_through_heads=False is gone. The 8 VT heads introduced in Gen4+6 are now automatically included in training in the EARLY_RELABEL slice when there is no click, if ads_head_masking=True. The deployment point can be confirmed in the commit where this config is set to "ads_head_masking": True in mparams (the model parameters passed to the training config).

B. The purchase value loss now trains only on samples from the early-relabel stream

The range of samples that the loss predicting ad purchase value by regression treats as valid has narrowed. In Gen4+22, all non-fresh samples were valid, and now only samples from the early-relabel stream are valid.

• This change is in phoenix/xrex/models/loss_recsys.py (+75/−10) verified.

• The condition of the valid mask narrowed to the early-relabel stream verified. That is, the late-split stream does not train purchase value regression and trains only the twin head classification.

  • The condition of purchase_value_valid_mask narrowed from sample_source > 0 to sample_source == EARLY_RELABEL_STREAM_ID.
  • When the purchase value loss was formally wired in Gen4+22 (2026-09-16), all "delayed (= not fresh) samples" counted as valid, but now only the early-relabel stream is treated as valid.

• Most loss stat names were also changed to new names (the sum key _purchase-value-sums is unchanged) verified.

  • They changed from purchase-value-* to purchase-value_delayed_clicked-*.

  • This aligns the names with the mask condition above. The stat name itself says that the effective slice on which this head is trained is the subset "clicked among delayed samples".

    "purchase-value_delayed_clicked-loss": loss, ... "_purchase-value-sums": sums, — phoenix/xrex/models/loss_recsys.py, 2026-09-18

• A new function publishes purchase value metrics smoothed with an exponential moving average (EMA) verified.

  • The new function purchase_value_smoothed_stats publishes EMA-smoothed loss, ratio-mae, target-ratio, baseline-mean-usd, usd-mae, and calib.
  • The default smoothing windows are (1 << 28, 1 << 30). They are exponential moving averages over sample sizes of 2^28 ≈ 268 million and 2^30 ≈ 1.07 billion samples, respectively.
  • trainer_recsys.py added the hook _purchase_value_ema_keys to keep a shape=(10,) tensor as the EMA state for these two windows. If a shape mismatch occurs when a checkpoint is reloaded, it is reinitialized to zeros.

• USD MAE and calibration metrics are also newly published verified.

  • USD MAE is purchase-value_delayed_clicked-usd-mae.
  • Calibration is purchase-value_delayed_clicked-calib, the total actual ratio versus the total predicted ratio.

• This commit has no actual wiring for how the ads revenue prediction is re-adjusted in the auction verified. The default purchase_value_loss_weight=0.0 is also kept.

Why it matters. The training scope of the ads revenue regression loss attached in Gen4+22 has narrowed. Previously, anything "inside the delayed observation window" went into training unconditionally; now only "the slice where a delayed observation occurred slightly (early)" does. The late-split slice now trains only the twin classification heads and no purchase value regression.

In effect, X's ads ranking is settling into a two-branch training stack. Short-lag observations train purchase value by regression, and long-lag observations train only whether a conversion occurred, by classification.

C. Phoenix requests got fields that carry ads engagement counts by time window

The Phoenix serving request format now has room for a table of per-action engagement counts over several time windows. The same format is used in two places: the user's ads engagement history and the candidate ad's own engagement history. This commit has no training or batch encoding that consumes this value.

• This change is in two proto files verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto (+22), phoenix/python/common/xai-proto/proto/recsys.proto (+22).

  • The new message is EngagementCounts.

  message EngagementCountRow {
    int32 action = 1;
    repeated uint32 counts = 2;
  }
  
  message EngagementCounts {
    bool present = 1;
    repeated uint32 window_secs = 2;
    repeated EngagementCountRow rows = 3;
    int64 updated_at_ms = 4;
    int64 window_end_ms = 5;
  }
  

• The count table is connected to two slots of the request verified.

  • PredictNextActionsRequest.user_ads_engagement_counts = 24: a time-window count matrix of the ads engagement history carried in the user context.
  • AdInfo.engagement_counts = 23: time-window counts of the engagement history that each candidate ad itself has.

• This matrix is a two-dimensional (action × time window) count table, carried in the request payload into the serving ranker verified.

  • window_secs is a list of time window lengths, and each element of rows lists the counts observed in those windows for a particular action index.
  • updated_at_ms and window_end_ms are the time reference points of this matrix.

• Server-side training and batch encoding that consume this field are not reflected in this commit verified. Only the proto layer was opened, and there is no related parsing in xrex/data/recsys/*.

Why it matters. This field follows up on Gen4+9 (2026-08-26), where ConvAssetIds carried the user's recent ad creative impression history in the request. This time the target is per-action time-window counts, not creative assets. A distinguishing point is that each ad candidate's own engagement history and the user's ads engagement history were opened symmetrically. As a signal that recent engagement statistics of user–item pairs are entering ads ranking as explicit features, the direction matches the standard "user × item time-window count" features of the YouTube/Meta family inferred.

D. Phoenix's post info message got a search-term match signal

A search lexical match field (whether the query words appear in the post verbatim) was newly added to the post info message in the Phoenix wire proto. This commit has no point that uses this value in training or serving.

• A new field was added to TweetInfo in the same two proto files.

  message TweetInfo {
    // ... 37 existing fields ...
    SearchLexicalMatch searchLexicalMatch = 38;
  }
  
  message SearchLexicalMatch {
    float matchedWordFraction = 1;
    bool exactPhrase = 2;
  }
  

• The field type is SearchLexicalMatch, made of two values: the matched word fraction and whether there is an exact match.

  • matchedWordFraction is the fraction of search query words that appear in the tweet.
  • exactPhrase is whether the query string appeared as an exact match.

• It appears it will be consumed in the search ranking config, but this commit has no training or serving consumption point.

  • That config is xrecsys_search.

Why it matters. A channel has opened through which a lexical match can pass from a candidate generation signal to a ranking signal. Since Gen4 of X's open-source mirror, two practices have largely been maintained on the search ranking line: the xrecsys_search config runs separately, and EarlyBird-style lexical matching is absorbed at the candidate generation stage. This time the lexical match was promoted to a wire field of the candidate itself.

Instead of Elasticsearch/Lucene-family BM25 (a classic search score based on term frequency) scores, it passes simple statistics on two axes, (fraction, exact_phrase). In X's query domain, which mixes short text, hashtags, and handles, these two axes may actually be more robust to noise.

E. grox added spam-clone detection that compares confirmed violating videos with candidate videos using a VLM

grox's safety pipeline got a new detection flow. It picks high-engagement video posts and passes them to a VLM together with storyboards (sequences of frames in time order) of confirmed violating videos for comparison.

• This flow is in the following files verified — grox/flows/ptos/plan_safety_ptos_media_injected_adult_infrared_video_spam_detection.py (44 lines, new), task_safety_ptos_media_injected_adult_infrared_video_spam_detection.py (188 lines), task_safety_ptos_media_injected_adult_infrared_video_spam_detection_filter.py (55 lines), grox/core/data_loaders/media_reference_bundle.py (146 lines, new).
• The flow is attached to grox's PTOS (policy-based violation category verdict), and one Kafka topic triggers it verified.
  • The flow name is long: safety_ptos_media_injected_adult_infrared_video_spam_detection.
  • A streaming task generator sourced from the Kafka topic safety-ptos-media-injected-adult-infrared-video-spam-detection triggers this flow.
• Targets are high-engagement video posts not already flagged NSFW verified. The eligibility filter requires all of the following conditions.
  • There is a user.
  • The post or the quoted post has a video.
  • The fav count is at least media_hydration.deluxe_fav_count_threshold (a high-engagement post).
  • It is not a post already flagged NSFW.
  • The filter name is TaskSafetyPtosMediaInjectedAdultInfraredVideoSpamDetectionFilter.
• Posts that pass go to the VLM together with a reference video bundle verified.
  • MediaReferenceBundleLoader.current("media_injected_adult_infrared_video_spam") fetches the reference video bundle from the blobstore.
  • Then the reference video storyboards and the candidate video storyboard are passed to the VLM together.
• The reference bundle holds up to 10 reference videos and is reloaded every 15 minutes verified. The bundle loader's specification is as follows.
  • It reads manifest.json from a Blobstore URI (format blobstore://<namespace>/<subpath>) and loads up to 10 reference video specs.
  • Each video has at most 10 frames, and each frame is at most 2 MiB.
  • It refreshes every 15 minutes (refresh_interval_s=900 default), and retries every 60 seconds if the initial load fails.
  • It keeps a 12-character SHA-256 digest of the manifest and frame bytes for bundle identity.
• One vision sampler config is called as the VLM with a 60-second timeout verified.
  • That config is GROK_4_1_FAST_HEDGEHOG_CRITICAL, looked up with grox_config.get_model.
• The prompt puts the reference video frames first, then the post to be judged, and instructs a comparison verified.
  • The system message is the new template media_injected_adult_infrared_video_spam_detection.j2 (body undisclosed).
  • The user message starts with Reference storyboards from N confirmed violating videos (frames in chronological order): and interleaves the reference video frames in order.
  • After that it renders the candidate post (descendants removed) with Post under review:.
  • Finally it instructs: Compare the video storyboard(s) of post {post.id} against the reference storyboards and provide the requested JSON object.
  • The assistant seed is THINKING_CONTROL_END. That is, it opens with a thinking control token.
• If the response is a violation, a violation policy is attached to the safety state, and the result goes to the safety pipeline sink verified.
  • It extracts the pattern <json>{"violation": bool, "reason": "..."}</json> from the response and parses it into MediaInjectedAdultInfraredVideoSpamVerdict.
  • If violation=True, it attaches a violation policy to SafetyPtosState in the form SafetyPolicy(category=..., type=..., ...).
  • Then TaskWriteSafetyPostAnnotationsResultSink passes the result on to the safety pipeline sink.

Why it matters. X's grox pipeline expanded in the form of "visual comparison against a set of confirmed violating video storyboards". In the literature, near-duplicate video detection (finding nearly identical videos) is standardly embedding-based, but instead of embedding matching, X chose an approach that puts storyboard frames directly into the VLM prompt and leaves the visual matching verdict to it. Because the reference bundle is refreshed from the blobstore every 15 minutes, when the operations team adds newly confirmed clone candidate videos to the bundle, they are reflected in verdicts without a separate redeployment. The infrared in the flow name appears to refer to a particular capture mode (e.g., adult video clones shot with night vision), but the exact category definition cannot be confirmed without the text being released inferred.

F. Other minor changes

• The Kafka consumers and producers of abuse-enforcement-service (a service that takes account actions based on model scores) can now set cluster/zone per topic. The changed files are abuse-enforcement-service/service-lib/src/lib.rs (+390/−56) and growthbook.rs (+35/−6).
  • Consumer and producer config can now be overridden per topic instead of using the service-wide mTLS cluster/zone.
  • The new metric abuse_enforcement_kafka_consumer_start_total{topic,cluster,result} records the consumer start result at boot time as labels.
  • A filter was also added so that consumer_error_total is scoped only to the set of watched topics.
• The order in which grox collects URLs changed, and the domain set now preserves insertion order. The changed file is grox/core/data_loaders/data_types.py.
  • In _collect_urls, user URLs moved behind card URLs.
  • Domain dedup uses dict[str, None] instead of set, so insertion order is preserved. A side effect is that the order in which domains are exposed to downstream verdicts becomes stable.
• The allowed domains grox passes to the web search tool are now truncated to at most 5. Previously the full list was passed as is. The changed file is grox/libs/grok_sampler/eapi_sampler.py.
  • The allowed_domains that eapi_sampler passes to the web_search tool is truncated to search_allowed_domains[:MAX_SEARCH_DOMAINS] (MAX_SEARCH_DOMAINS = 5).
• The log label name in the grox reply spam verdict changed. The changed file is grox/flows/reply_spam/task_spam_detection.py.
  • "lower than 1000 follower bucket" became "Gemma bucket". The label was unified as the term for the Gemma score = 0 verdict after the follower branch in Gen4+22.
• The parameter mirror of home-mixer (the service that assembles the For You feed) changed only its sync time, with no actual default change. The changed file is home-mixer/params/param.rs.
  • This file is a cron mirror that syncs the production values of weights and constants every day.
  • The sync time changed to 2026-09-16T16:23:12Z → 2026-09-17T20:46:44Z.
• The account list of the Brazil2026 election filter (a filter that hardcodes accounts acted on by the Brazilian electoral court and removes them) was updated. The changed file is home-mixer/filters/brazil_2026_election_filter.rs (+143/−4).
  • It hardcoded 12 new accounts and updated 4 "no live account found" comments. The practice of regular updates continues.
• Phoenix's Cargo config moved the thrift crate source from a git tag to the crates.io registry. The changed file is phoenix/Cargo.toml.
  • The version is the same, 0.24.0, but a checksum lock was added, which strengthens supply-chain reproducibility.
• When exporting a Phoenix serving bundle, fields to fill from the request are now separated from fields fixed to template values. The code is Phoenix xrex/train/recsys_bundle_export.py.
  • A new constant REQUEST_BATCH_KEYS (a set of about 30 batch keys) was defined.
  • At serving bundle export time, fields belonging to these keys are marked source: "request" and filled from the request payload.
  • The rest are marked source: "template", and the uniform values of example_data are fixed as fill and included in the bundle. A non-uniform template raises an error.
  • In other words, with the request schema now explicitly defined, which fields come from the client at serving time and which are fixed constants is locked as a contract.
• A default-fill syntax was added to one test in visibility-filtering (the service that decides, post by post, whether a post may be shown to this viewer). It appears to be a side change because a struct gained a field elsewhere.
  • ..Default::default() was added to the gizmoduck_hydrator test.

• ⚠ Claim changed #1. The three config options that split delayed-feedback training were deleted and unified into one flag.
  • The "three delayed-feedback training branch config options (condition_conversion_on_click, train_view_through_heads, split_head_training_by_source)" that the canon has described since Gen4+6 were all deleted in Gen4+24 and unified into the single ads_head_masking.
  • The three options in the earlier table no longer exist in the code. However, the 3-stream masking itself strengthens the combination of the earlier three options, so the conceptual core remains.
• ⚠ Claim changed #2. The effective scope of the Gen4+22 purchase value loss narrowed to the early-relabel stream.
  • The statement that the Gen4+22 purchase value loss is "valid for all delayed samples (sample_source > 0)" was overturned. It is now valid only on EARLY_RELABEL_STREAM_ID (the early-relabel stream).
  • On the late-split stream, purchase value regression is not trained and only the twin classification head is trained.
• ⚠ Partial claim change #3. The tier 2/3 question left open by Gen4+23 was not resolved in Gen4+24 either.
  • The Gen4+23 statement "the lag window definition and training loss weight of delayed conversion tier 2/3 are not stated in this commit" is still open in Gen4+24.
  • Gen4+24 added twin masking, but its target is the base tier (204-213), and tier 2 (214-223) and tier 3 (224-233) are still not connected. That is, the tier 2/3 actions introduced yesterday have not been incorporated into today's training pipeline.
• Since Gen4+9 (ConvAssetIds), a second channel has opened through which the ads request payload explicitly carries the user's ads history. This time it differs in that it is an (action × time-window) count matrix rather than creative asset IDs.
• A lexical match signal appeared for the first time as a wire field in the search ranking line. This opens the possibility that lexical matching, which had been absorbed only into candidate generation, expands to ranking.

Corrections

• The three delayed-feedback options the canon described were deleted, and the validity condition of the purchase value loss narrowed. See subsections A and B of ## 1 above.
  • The three options condition_conversion_on_click, train_view_through_heads and split_head_training_by_source were deleted.
  • The condition changed from sample_source > 0 to sample_source == EARLY_RELABEL_STREAM_ID.

What to check next

Radar

• Today we screened 30 arXiv cs.IR papers, and 0 passed. The number of parsed <entry> elements is 30.
  • Yesterday (2026-09-18 daily, Gen4+23), four papers (LIGE-GR, SARA, ANGLE and PCap) were already folded in.
  • None of the 30 papers that appeared today meets the acceptance criteria. The criteria are platform-company authors, production deployment or A/B result claims, and novelty in ranking or retrieval mechanisms.
  • Most are academic increments limited to RAG, LLM re-ranking, retrieval benchmarks, and medical, legal and science domains.
• Industry blogs are scanned only on Mondays. Today is Saturday, so they were skipped.