2026-09-16

1Slate diversity observation and ads purchase-value prediction head added — X

• A new connection opened in the ads Phoenix ranker that regression-predicts the purchase value itself as a continuous head (previously only click and conversion probability heads existed).
  • A continuous head is an output head that predicts a continuous value by regression.
• The attention kernel in the H100 two-tower training config moved to the same CUTe DSL family as GB300.
• The Kafka OTel export interval was relaxed from 30→120 seconds.
• The root follower threshold of the coordinated spam filter rose from 10k→30k.

• A new observation side effect that measures response diversity in three stages was added to home-mixer (the service that assembles the For You feed) verified — home-mixer/side_effects/response_diversity_stats_side_effect.rs (186 lines, new)

  • A side effect is a pipeline stage that runs in addition without changing the response.
  • By default it runs on only 5% of requests, and inside an experiment bucket it runs on 100%. Each stage emits 12–15 metrics as histograms.
  • The other files changed together are home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs (+2) and home-mixer/params/param.rs (+7).

• This side effect was wired into the Phoenix candidate pipeline as the 6th observation side effect verified(same commit).

  • It is named ResponseDiversityStatsSideEffect, and its place is after ScoredStatsSideEffect and before AuthorServedMetricsSideEffect.

• There are two run conditions verified(same commit).

  • Default: if no experiment bucket is set on the feature-switch param EnableResponseDiversityStatsExperimentBucket, DEFAULT_SAMPLING_RATE = 0.05 applies, that is, it runs on only 5% of requests.
  • Experiment: if an experiment bucket is set on this param, it runs on 100% inside that bucket and emits each metric with ddg (experiment id) and bucket scope tags. This means the diversity distribution can be observed separately for each experiment arm.

  if experiment_buckets.is_empty() && random::<f64>() >= DEFAULT_SAMPLING_RATE {
      return Ok(());
  }
  

  — response_diversity_stats_side_effect.rs

• Each of the three stages is measured verified(same commit).

  • final: the finally selected candidates (input.selected_candidates) sorted by score in descending order. This is the slate actually delivered to the user.
  • top10: the top 10 of final (TOP_POSITIONS = 10).
  • pre_heuristic: merges selected_candidates and non_selected_candidates and takes the top K by weighted_score in descending order (K = final slate size).
  • weighted_score is the raw score (head-weight sum) obtained by multiplying per-head probabilities by weights and adding them, the value before applying DPP, author-decay, and the OON multiplier.
  • In other words, pre_heuristic is the counterfactual of "what diversity would the slate have had if only the raw-score top-K had gone out without the heuristic layer".

• Each stage emits the metrics below, and all names carry the ResponseDiversity.* prefix verified(same commit).

• A new primitive Composition that extracts multiset statistics for any axis was added verified(same commit). This struct is reused when the hydrator in C below computes the new slate bits.
  • Its fields are size, unique, max_count, hhi, and entropy_norm, and the file is home-mixer/util/composition.rs (63 lines, new).
  • A single from_keys<K: Hash + Eq>(iter) produces statistics regardless of the axis.
• The combination of a 5% sampling probability and three stages is a tuning that keeps observation accuracy while barely affecting ranking latency.
  • There are two sorts, and the entropy and HHI computations are O(n) passes over the vector.
  • Observation is 100% only in experiment buckets, which leaves open the verification of even the tail distribution.

• A new function collects the ids of positively engaged authors from the viewer's recent interaction history carried in the request verified — home-mixer/util/viewer_history.rs (61 lines, new)

  • The function is positively_engaged_author_ids(columnar_sequence: &bytes::Bytes) -> Option<HashSet<u64>>.
  • The input is the Arrow IPC stream ScoredPostsQuery.columnar_scoring_sequence carried in the request. It holds the user's recent interaction history as columnar batches.
  • It collects and returns the authorId of rows where authorId is greater than 0 and at least one of the 18 positive actions is on in the actionNameMultiHot fixed-size list.

• The 18 positive actions are stated explicitly in the following list verified(same file).

  ServerTweetFav, ServerTweetReply, ServerTweetQuote, ServerTweetRetweet,
  ClientTweetClick, ClientQuotedTweetClick, ClientTweetOpenLink,
  ClientTweetClickProfile, ClientTweetPhotoExpand, ClientQuotedTweetPhotoExpand,
  ClientTweetVideoOpen, ClientTweetVideoQualityView, ClientQuotedTweetVideoQualityView,
  ClientTweetShare, ClientTweetClickSendViaDirectMessage, ClientTweetShareViaCopyLink,
  ClientTweetBookmark, ClientTweetFollowAuthor
  

  • That is, negative signals such as report, mute, block, and not_dwelled, and exposure-type signals such as dwell, are intentionally excluded. Two video quality views are in the list.

• This set is a new constant set that has appeared nowhere else in X's public code. It fixes in code for the first time the definition of "authors with whom a relationship was formed through active engagement".

• Which ranking head this set corresponds to, that is, whether it exactly matches the positive engagement head definition at phoenix/runners.py:233-253, is outside the scope of this commit.

• This function is used to compute one slate bit in C below (AUTHOR_NOT_ENGAGED_BY_VIEWER).

• 12 new slate-axis bits were added to the bitset that records attributes per candidate verified — home-mixer/candidate_hydrators/tweet_type_metrics_hydrator.rs (+100/−4)
  • The target is the TweetTypeMetrics bitset, and the predicate slots are 239-244 and 315-320.
  • The other files changed together are home-mixer/util/tweet_type_metrics.rs (+27) and home-mixer/models/candidate.rs (+8).
• Until now, TweetTypeMetricsHydrator set the bitset for each candidate with attributes of the tweet itself.
  • Examples are whether it is a retweet, the media type, the tweet age bucket, and six tiers of author follower scale.
• This commit added six bits (239-244) that reflect slate-wide statistics onto every candidate in the slate verified(same commit). These bits are set identically for all candidates.
  • UNIQUE_AUTHOR_RATIO_LTE_50_PCT = 239: slate unique_ratio ≤ 0.5
  • UNIQUE_AUTHOR_LTE_5/10/15 = 240/241/242
  • SINGLE_AUTHOR_GTE_25_PCT/50_PCT = 243/244
• It also added six bits (315-320) reflecting the context each candidate has within the slate verified(same commit). These bits are set differently for each candidate.
  • AUTHOR_REPEAT_IN_SLATE = 315, AUTHOR_REPEAT_GTE_3_IN_SLATE = 316: they turn on from the moment the same author_id is met for the second, and the third or later, time while iterating in descending score order. They are not on at the first encounter. This bit expresses "which appearance of the same author this tweet is in the slate".
  • AUTHOR_NOT_ENGAGED_BY_VIEWER = 317: turns on when the candidate's author is not an account the viewer follows and is not in the positive-engagement set of B either. It is the bit that marks new authors unrelated to the viewer.
  • HAS_SEMANTIC_IDS = 318: turns on when semantic_ids are assigned to the candidate.
  • SID_L1_REPEAT_IN_SLATE = 319, SID_L2_REPEAT_IN_SLATE = 320: in descending-score iteration, they turn on for the second and later candidates when the same SID L1 (or L2) prefix is already in the slate.
• A helper that safely cuts only the top codes from the SID array was added verified(same commit).
  • PostCandidate::semantic_id_prefix(levels: usize) -> Option<&[i32]> returns a slice of the top levels codes, and returns None if levels is greater than the array length.
• These bits are merged with the per-candidate base bitset and then stored compressed verified(same commit).
  • In hydrate, the slate-level bit set and the per-position bit set are merged by union into the per-candidate base bitset, compressed, and stored as the tweet_type_metrics payload.
• TweetTypeMetrics has so far been an axis used in ranking training and serving logs and offline analysis, so slate diversity is now logged as a per-candidate feature and becomes accessible to training and modeling. However, this commit has no training config change that actually uses these bits, so only the connection was opened. Whether they are actually used is something to watch in follow-up commits.

• A continuous head predicting the actual purchase value after an ads web click and a per-advertiser purchase-value baseline message were added to the Phoenix proto verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto (+9)

  • The head is named ADS_WEB_CT_PURCHASE_VALUE = 5, and the message is named PurchaseValueBaseline.
  • The other files changed together are phoenix/crates/serving/xai-recsys-proto/src/purchase_value.rs (114 lines, new) and phoenix/python/common/xai-proto/proto/recsys.proto (+9).

• The new head is the fifth value of the ContinuousActionName enum verified(same commit).

  • The existing four were DWELL_TIME = 1, CLICK_DWELL_TIME = 2, ACTIVE_SECS_5M_RESIDUAL_NORM = 3, and BRIDGE_PROBABILITY = 4.
  • ACTIVE_SECS_5M_RESIDUAL_NORM is the residual normalization of the 5-minute moving average of active seconds. It predicts the active seconds of the next 5 minutes in this session, normalized by the previous moving average.
  • BRIDGE_PROBABILITY is the bridge probability between segments.

• The new ADS_WEB_CT_PURCHASE_VALUE = 5 is the regression target for the purchase value that occurs after a web click-through conversion verified(same commit).

  • CT is an abbreviation of Click-Through, and here it means an actual purchase made on the landing page after an ads click.

• A new field 23 was added to the PredictNextActionsRequest request payload verified(same commit).

  repeated PurchaseValueBaseline purchase_value_baselines = 23;
  
  message PurchaseValueBaseline {
    int64 impression_id = 1;
    int64 advertiser_account_id = 2;
    double mean_value_usd_28d = 3;
  }
  

  • Each baseline is a tuple of (impression_id, advertiser_account_id, advertiser's average purchase value over the last 28 days in USD).

• The Rust helper purchase_value.rs provides baseline validation and lookup functions verified(same commit).

  • is_valid_baseline checks that the id is positive and the USD value is a positive finite number. It also provides find_baseline(request, impression_id, advertiser_account_id).

• Five unit tests pin down wire tag stability and backward compatibility verified(same commit).

  • Wire tag: field 23 = 0xBA 0x01, and it is independent of returnLogitsList field 22 = 0xB0 0x01 0x01.
  • Backward compatibility: an old request decodes normally with an empty baseline list.

If the advertiser's average purchase baseline over the last 28 days is carried in the request payload, the residual of how far above or below that baseline this impression falls can be trained and served. However, this commit does not state whether the amount is the direct training target or a baseline offset. As the probability that this combination of user and ad drives an unusually large (small) purchase relative to the advertiser's average purchase value enters the ads ranking score, X ads ranking is ready to move toward a more advanced value-optimizing auction structure inferred. This is in the same direction as what Meta and Google call VBB (Value-Based Bidding) and tCPA/tROAS, respectively inferred.

That the baseline is carried in each request means a separate offline pipeline must update the 28-day average of advertiser accounts daily and feed it in parallel into ad serving requests. Given the concern about serving traffic overhead, baselines for all advertiser accounts are likely not carried; it reads as attaching only the baselines of those advertisers whose ads will be seriously scored for this viewer at request time, selected by pre-pruning inferred.

• The attention implementation in the H100 training settings changed to a kernel of the same CUTe DSL family as GB300 verified — phoenix/xrex/configs/xrecsys_two_tower.py (+1/−1)

  • The changed value is pallas_ranker_varlen_attn → cutedsl_ranker_varlen_attn.

    _H100_OVERRIDES = {
        "bs_per_device": 480,
        "ep": 128,
    -    "attn_impl": "pallas_ranker_varlen_attn",
    +    "attn_impl": "cutedsl_ranker_varlen_attn",
    }
    
    _GB300_OVERRIDES = {"bs_per_device": 960, "ep": 64, "attn_impl": "cutedsl_ranker_varlen_attn"}
    

• cutedsl_ranker_varlen_attn is a variable-length attention kernel based on NVIDIA CUTLASS/CUTe DSL. This kernel had been in _GB300_OVERRIDES since the Gen4 release (47c1bcd, 2026-08-13) (xrecsys_two_tower.py:340). In Gen4+17 (49815da, 2026-09-08), the SM80 and SM90 forward and backward kernels were all released as open source (phoenix/xrex/cutedsl/ranker_fa4/).

• The SM80 kernel is for A100 (compute capability 8.0), the SM90 kernel is for H100 (9.0), and GB300 (10.3) takes the arch 100 path verified — NVIDIA compute capability table, cutedsl_arch in ranker_attention_fa4.py.

  • So with this switch, H100 training uses the SM90 kernel. The two GPU generations use the same attn_impl name, but the kernel that actually runs differs by generation.

• We could not confirm whether this commit lifts the constraint that the seqpack (packing that concatenates multiple user sequences into one batch row) training config forces pallas_ranker_varlen_attn on H100 only. This constraint was observed in Gen4+11 (bc8e5f0).

  • If the NotImplementedError check stated explicitly in the seqpack code path still remains, the combination of seqpack and FA4 (flash attention v4 based on CUTe DSL) is still blocked. This is something to watch in follow-up commits.

• The Kafka OTel (OpenTelemetry) export interval in Phoenix training grew from 30s → 120s, easing the metric export load by 4× verified — phoenix/xrex/data/streaming/kafkaloader.py (+2/−2)
  • The default export_interval_ms of _init_otel_metrics and PhoenixKafkaDataset.metrics_export_interval_ms both changed from 30_000 → 120_000.
• This change reads as a pair with item H of Gen4+20 (6bb4594, 2026-09-12), the Kafka mTLS connection, but the direction is opposite. mTLS adds authentication overhead, and the relaxed export reduces metric load.
• Exporting OTel metrics 4× less often is a trade-off that gives up observation tail resolution of the training job and reduces the collector burden.
• This relaxation is evidence that metric host cost had actually become a burden inferred.

• The follower threshold of the coordinated spam filter in Grox (the Grok-based nearline annotation pipeline) rose from 10k → 30k verified — grox/flows/reply_spam/task_filter.py (+1/−1)

  class TaskCoordinatedSpamFilter(TaskFilterWithPost):
  -    FOLLOWER_COUNT_THRESHOLD_FOR_SPAM_DETECTION = 10000
  +    FOLLOWER_COUNT_THRESHOLD_FOR_SPAM_DETECTION = 30000
  

• The Grox reply-spam pipeline has two separate follower thresholds.

  • TaskSpamFilter·TaskReplyRankingFilter: since the Gen4 release it rose seven times: 15k → 30k (Gen4+1) → 40k (Gen4+2) → 60k (Gen4+3) → 80k (Gen4+6) → 100k (Gen4+7) → 120k (Gen4+8) → 150k (Gen4+17) verified — per-commit values of task_filter.py.
  • Spam classification includes only conversations where both the root and the reply-target account have followers at or below the threshold (lines 77-93). So raising it means expanding the classification scope (inspecting conversations of larger accounts as well).
  • TaskCoordinatedSpamFilter: it was 1k at the Gen4 release, and after 5k (Gen4+15) and 10k (Gen4+17) it rose for the third time, now to 30k verified — per-commit values of the same file. Coordinated spam is a type in which multiple accounts pair up and repeatedly post the same spam pattern.
  • This filter skips the conversation when the root author's followers are below the threshold (lines 157-161, reason low_blast_radius). So raising it removes conversations under small accounts from judgment.

• Both thresholds are rising, and the judgment scope of both filters moves toward conversations of large accounts verified — same file. General spam classification widens to conversations of larger accounts, and coordinated spam classification drops conversations under small accounts.

• It reads as an adjustment to concentrate the cost of coordinated spam judgment on high-exposure conversations inferred. The basis is that the skip reason is named low_blast_radius.

1. The observation scope widened to slate diversity. Until now, scoring observation in X's public code went only as far as per-candidate scores and per-head probabilities, and how much the slate-layer heuristics (DPP, author-decay, OON multiplier) change the candidate set was not measured.
   • The parallel observation of the three stages pre_heuristic/top10/final in this commit is the first connection that can measure the actual impact of the heuristic layer.
   • In particular, since pre_heuristic is a counterfactual ("without the heuristic"), it makes it possible to measure the trade-off between diversity gain and ranking loss empirically.
   • The reorganization of the slate layer has continued through the slate context pipeline of Gen4+4, the VMRanker DPP simplification of Gen4+8, and the seed anchor pinning of Gen4+20. This observation reads as the observation-side pair of that line.
2. Ads ranking widened from probability prediction to value prediction.
   • The 20 conversion actions of the 2026-08-21 commit (Gen4+6) were still heads predicting probability (whether a click or conversion occurs).
   • The new continuous head takes the amount itself as its regression target. It also prepared a connection that carries the per-advertiser 28-day baseline in the request, enabling residual training.
   • A value-optimizing auction means advertiser ROI optimization is handled directly in the serving layer, so there will likely be effects on both the ads ranking score computation and the auction logic after it.

Actual consumption (promoting the new bits and head in the training config, a production experiment ramp) is something to watch in follow-up commits.

↑ Contents

Canon update summary

Of the canon, only services/x.md was edited, and the pipeline diagram was left as is.

• Added a row to the Gen4 release table in services/x.md and created a new ### Gen4+21 detail subsection.
• The diagram was not regenerated. The pipeline structure (stages, sources, figures, generation) did not change, and this change is an observation side effect, proto fields, and an attention kernel switch, so assets/x-pipeline.svg is left as is.

↑ Contents

Corrections

No claims were overturned.

↑ Contents

Radar

In this batch the arXiv cs.IR API kept responding with 429 (rate limit), so triage could not be done. It was abandoned after 6 retries, and the number of passed items folded in is also 0.

↑ Contents

What to check next

↑ Contents