2026-10-21

Other changes reviewed today:

• In X's For You, the video carousel no longer picks the same media as a video already shown in the main feed. The carousel is still off by default.
• X added, off by default, a metric that measures how many topics the feed shows that were not in the viewer's recent history.
• Alibaba quantified that reassigning SIDs in generative retrieval sharply degrades search quality, and reported an A/B gain from an update that keeps existing SIDs fixed.

1Meta ads represent items with semantic tokens instead of IDs — Meta

• GenRecs Tokenization represents an item with a set of tokens chosen from a stable semantic vocabulary instead of giving each item a dedicated ID verified — blog post.
  • It applies to ads, products, pages, and pixels (conversion tracking entities on advertiser sites), and is used in both retrieval and ranking verified.
• Used together with GEM (Meta's central ads generative model) and the Adaptive Ranking Model, it raised advertiser conversions by +0.35% across all surfaces verified — same post.
  • The footnote reads "Meta Internal Data, measured in February 2026". The measurement is 8 months old verified.
  • It did not disclose which surfaces are included in this figure.

"Paired with Meta's GEM and Adaptive Ranking Model, GenRecs Tokenization delivered a +0.35% lift in conversions." — GenRecs Tokenization, 2026-10-19

• A pretrained multimodal model turns an item's text and images into embeddings, and the tokenizer adds engagement-based representations to them and quantizes the result into a short sequence of discrete tokens verified — same post.
  • The pipeline has three stages: encode into an embedding space, decompose into independent subspaces, and convert each subspace into a codebook (a list of representative vectors used for quantization) entry verified.
  • It states that many ad representations use a prefix n-gram that combines a learned encoder, residual decomposition, and a learned vector codebook verified.
• It writes that tokenizer quality could be estimated in advance with intrinsic metrics, without downstream training verified — same post.
  • The intrinsic metrics are distribution balance (the degree to which items are not concentrated on a few IDs) and codebook utilization (the share of codes to which at least one item is assigned).
  • It states that across 8 tested tokenizer versions, these metrics correlated strongly with ad CTR prediction accuracy verified. It did not disclose the correlation figures.
• It also mentions two-tower (a retrieval structure that embeds the user and the item separately and finds items by dot product) work built with a shared LLM encoder on the same vocabulary verified. It only wrote that it reached the existing two-tower performance with part of the training data.

"As recommendation systems evolve from discriminative scoring to include generative reasoning, the data foundation benefits from evolving from ephemeral IDs to stable semantic representations." — GenRecs Tokenization, 2026-10-19

↑ Contents

2The video carousel now excludes videos that match the main feed — X

• The carousel skips videos whose media ID or MD5 hash matches a video in the top 35 of the main feed verified — home-mixer/util/video_carousel.rs:52-108.

  • It also skips a video whose media matches a video picked earlier within the carousel verified.
  • The filter passes the main feed range as &kept[..RESULT_SIZE], and the candidate pool in the same file comes after it (skip(RESULT_SIZE)) verified — filters/video_carousel_filter.rs:26-29. RESULT_SIZE is 35.
  • The media ID and MD5 hash are read from TES (Tweet Entity Service, X's internal tweet metadata lookup service). The existing aspect-ratio hydrator was renamed VideoMediaHydrator and fills in both values verified.

• This rule has no parameter verified. The carousel itself (EnableVideoCarousel) is still off by default in this commit verified — params/param.rs:1102.

  • A test pins the expected behavior. If the main feed has media 100 and hash "a", candidate 1 (media 100) and candidate 2 (hash "a") are dropped.

  let main_feed = vec![
      with_media(video(90, 1.7778, 0.9), Some(100), &[]),
      with_media(video(91, 0.5625, 0.8), None, &["a"]),
  ];
  // candidates 1(media 100), 2(md5 "a"), 3, 4, 5
  // => select_videos(...) == vec![3, 4, 5]
  

• A calculation was added that counts the first 1 and 2 levels of the SID of posts the viewer recently engaged with as topics verified — home-mixer/util/viewer_history.rs.

  • Engagement is judged by the same 18 positive actions as the existing author-history calculation verified.
  • The calculation switch HistoryTopicsEnabled is false by default, and the window is HistoryTopicsWindowDays=28 verified — params/param.rs:508-529.

• A new side effect (a post-processing stage that only leaves logs or metrics without changing the response) compares the topics of response posts with this history and emits the "new topic share" as a metric verified — side_effects/serendipity_stats_side_effect.rs.

  • The metric name prefix is Serendipity, and it emits NewTopicShareL1, NewTopicShareL2, and NewTopicShareL1Top10.
  • Without an experiment bucket, it measures a sample of only 5% of requests. The history window on the metric side is fixed at 28 days verified — scoring_sequence_query_hydrator.rs:22, 57-68.

  const METRIC_PREFIX: &str = "Serendipity";
  const DEFAULT_SAMPLING_RATE: f64 = 0.05;
  const TOP_POSITIONS: usize = 10;
  

• A switch VMRankerSendHistoryTopics, which sends the same history topics in requests to VMRanker (a DPP slate reordering service), was also added, off by default verified — same param file. The code where VMRanker uses this value is not in the public repository verified.

Shadow traffic is a path that also sends a portion of real requests to an experimental Phoenix (X's Transformer ranking model) cluster for scoring and keeps those scores only in logs.

• Each shadow cluster can now specify its own aggregation method for the user behavior sequence verified — home-mixer/util/shadow.rs:35-49.
  • The default is the single value Experiment16Fou:DENSE_WITH_LONG_DWELL_HOME_EXCURSION verified — params/param.rs:103-108.
  • The definition of this aggregation is not public. From the name, we infer a history that includes long-dwell events and excursions away from Home inferred.
• A path was added that stores shadow scores in Redis keyed by prediction request ID and reads them back verified — util/shadow_scores.rs.
  • The reference ID for shadow sampling also changed from the request ID to the prediction ID (prediction_id) verified.

• video_6h and video_6hto2day were added to the SID windows of the Phoenix retrieval index verified — phoenix-rankall/src/config/mod.rs:212-213.
  • Until now, the shortest of the video windows was 48 hours. A function that builds hour-unit window names and tests were added together.
• Phoenix ranking and retrieval clients were consolidated into one xDS (service discovery) client, and the retrieval fallback switch was removed verified — phoenix_candidate_pipeline.rs.
  • The default load-balancing policy for retrieval also changed from tonic to penalized_peak_ewma verified — home-mixer/main.rs.
• 6 fields were added to TweetInfo in the ranker proto: the not-interested count and the notification send, open, click, dismiss, and see-less counts verified — recsys.proto. This diff has no code that fills in the values verified.
• visibility-filtering (a service that decides per post whether it can be shown) gained a pass that judges the quoted-post author separately; the mode is set by an environment variable and is off by default verified — visibility-filtering/quote.rs.
• The SID decoder was changed to restore a group of duplicate SIDs only once, and the FA4 attention kernel was upgraded to CUTLASS DSL 4.7 verified — phoenix/xrex/models/recsys_sid.py. There is no policy change.
• abuse-enforcement-service gained a backfill path, and the checkpoint KMS retry budget was raised to 300 seconds verified — 74d49a4. It is unrelated to feed ranking.

↑ Contents

3Reassigning SIDs breaks generative retrieval — Alibaba

• Updating the assignment lowered HitRate by 41.78–62.05% at beam 1, and by only 8.33–10.85% at beam 1000 verified — abstract.
  • The paper itself states the interpretation that a large beam produces many candidates and masks the degradation.
• An update combining fixed SIDs for existing items, insertion of new items, and periodic warm-start produced GMV +1.16% and UCTCVR +0.29% in a 3-week A/B test verified — HTML version.
  • All 7 authors are affiliated with Taobao & Tmall Group of Alibaba.

"Experiments show approximately 9% improvement on old items and 109% on new items; a three-week online A/B test yields UCTCVR +0.29% and GMV +1.16%." — arXiv 2610.26105 (HTML), 2026-10-19

↑ Contents

Radar

• Don't Rewrite the SID Map (arXiv 2610.26105): Added Alibaba's SID update A/B to the generative retrieval thread as additional evidence.

↑ Contents

What to check next

↑ Contents

Lookup status

• X: xai-org/x-algorithm had 1 new commit (74d49a4). There is no release.
• Meta: engineering.fb.com had 3 posts dated 2026-10-19. There is 1 recommendation post, GenRecs Tokenization; the AI agent router post and the Prometheus supercluster post were skipped as infrastructure. ai.meta.com/blog and transparency.meta.com were not checked this time.
• Netflix: The latest post in the RSS is dated 2026-10-16 and was already checked in the 2026-10-19 batch. research.netflix.com/publications was not checked.
• arXiv cs.IR: 30 entries were parsed. The latest submission is 2026-10-19, and 1 passed: the Alibaba SID paper. FLMNet, OneChord, and MOSS (diversity recommendation), whose platform affiliation is not shown, were dropped because they had only public benchmark results.
• WebSearch cross-checking was not done this time.

↑ Contents