2026-10-04

1SID retrieval server released and popular posts candidate source added — X

• The first stage is the 2026-10-01 commit (Gen4+33, b79b947). A sketch of the slot that extracts SIDs appeared in the retrieval wire of Phoenix (the Grok-based Transformer used for For You candidate retrieval and ranking).
• In the second stage, the 2026-10-02 commit (Gen4+34, 76843a5), four files for a dedicated Transformer in Phoenix (training config, model, trainer, runner) were released. home-mixer (the service that assembles the For You feed) got a SidSource candidate source skeleton and a SidRetrievalClient gRPC client.
• However, that commit had no server crate to receive actual requests and return responses. home-mixer sends calls to xai-recsys-sid-retrieval.prod.fou.twitter.biz:443, but what the Rust server behind that endpoint looks like could not be known from outside.

This commit released that server crate and changed three other things.

• Server crate: 7 new src files under phoenix/crates/serving/xai-recsys-sid-retrieval/, 1,018 lines (8 files, 1,050 lines with Cargo.toml).
• Popular posts candidate source: home-mixer got a new OON (posts from accounts the user does not follow) candidate source, PopularPostsSource. It fills the gap in OON recommendation by feeding top-K popular authors into Thunder (the in-network index holding the latest posts of followed accounts) as "virtual following".
• Ads brand safety: the default of EnableAdsBrandSafetyVerdictV2 changed from false → true. A new parameter, EnableAdsAuthorBrandSafetyFallback, which uses the author-level Grok label instead when a post label is missing, was also added (off by default).
• Replay and capture: a new subsystem of 6 new files, 1,334 lines, appeared under visibility-filtering/staging/ on the VF (visibility-filtering, the service that decides whether a post may be shown to a viewer) side. Including 3 moved files, the additions total 1,636 lines.

• The SID retrieval server crate was newly released with 7 src files (8 including Cargo.toml) verified — phoenix/crates/serving/xai-recsys-sid-retrieval/.

  • Component files: src/service.rs:208, src/index.rs:223, src/loader.rs:170, src/http_api.rs:109, src/main.rs:115, src/seed_sids.rs:114, src/metrics.rs:79.

• The service interface is defined by a separate proto crate and has only one RPC verified — sid_retrieval.proto:46.

  • The proto crate is xai-recsys-sid-retrieval-proto, and the RPC is rpc Retrieve(RetrieveRequest) returns (RetrieveResponse).
  • The request fields are seed_post_ids/max_results/max_per_seed/min_prefix_depth/max_prefix_depth/caller.
  • The response carries candidates, seeds (marking where each seed came from with Index/SidService/Missing labels), snapshot_timestamp_secs, and index_posts.

• The index packs the SID into a single integer key and keeps two post lists sorted in different orders verified — src/index.rs:5-110.

  • SID_LEVELS = 6, BITS_PER_LEVEL = 8, MAX_CODE = 255. Each level has 2^8=256 codes.
  • The full SID is packed into a 48-bit u64 key, shifting left by 8 bits per level starting from the top level. In theory the key space is 256^6 ≈ 2.8 × 10^14.
  • It keeps two vectors of IndexedPost { key, post_id, author_id }. by_key is sorted by (key ASC, post_id DESC) and used for prefix-range search. by_post is sorted by (post_id ASC) and used for binary search from seed post_id to SID key.
  • prefix_range(key, depth) returns [lo, hi), keeping only the top depth levels and setting the lower (6-depth)*8 bits to 0. It calls by_key.partition_point with this range to pick the posts inside it. Each depth level costs O(log N) + O(result size).

• Search collects candidates by descending from the deepest prefix depth to the shallowest verified — src/index.rs:138-193.

  • Each post enters the result "only once, at the deepest prefix level it shares". A post pulled at depth 6 is not pulled again at depths 5–3. This is implemented by walking the depths in reverse and subtracting the region already scanned (consumed).
  • Within the same depth, posts with a larger post_id (= more recent) come first. It cuts the top K with select_nth_unstable_by(post_id DESC) and then sorts.
  • The seed post itself is filtered out by a global excluded set. When requesting with multiple seeds, seed₂ does not get mixed into the results of seed₁.

  for depth in (params.min_prefix_depth..=max_depth).rev() {
      let (lo, hi) = prefix_range(seed.key, depth);
      let (start, end) = self.key_range(lo, hi);
      let mut level: Vec<&IndexedPost> = match consumed {
          None => self.by_key[start..end].iter().collect(),
          Some((inner_start, inner_end)) => self.by_key[start..inner_start]
              .iter()
              .chain(self.by_key[inner_end..end].iter())
              .collect(),
      };
      level.retain(|p| !excluded.contains(&p.post_id));
      ...
      picked.extend(level.into_iter().map(|p| Candidate {
          post_id: p.post_id, author_id: p.author_id,
          seed_post_id, shared_prefix_depth: depth as u32,
      }));
      consumed = Some((start, end));
  }
  

  — src/index.rs:138-179

• Candidates from multiple seeds are taken alternately by round-robin verified — src/index.rs:196-222.

  • It cycles seed 1 → seed 2 → ... → seed N → seed 1 → ..., and removes duplicates with a global HashSet<post_id> emitted.
  • It stops when it reaches max_results or when nothing more can be taken.
  • As a result, position bias is equalized so that it does not skew toward a particular seed.

• The snapshot loader checks the path every 30 seconds and, if it changed, rebuilds the whole index and swaps it in atomically verified — src/loader.rs:1-170.

  • The input format is Arrow Parquet (post_sid_snapshot.parquet). It has three columns, post_id: int64, author_id: int64, and post_sid: list<int64>, and the batch size is 65,536.
  • The timestamp is taken from the file name suffix (*_<unix_ts>.parquet), and if parsing fails, the file mtime is used instead.
  • watch_snapshots(path, poll_interval=30s, index, ready_tx) calls canonicalize(path) every 30 seconds and reloads everything if the path changed. The build runs in tokio::spawn_blocking, and on success it does an atomic swap with ArcSwapOption<SidIndex>. It then raises the initial ready flag of /readyz with ready_tx.send(true).
  • In other words, there are no real-time upserts; the pattern is to push in, in batches, a code table generated on the model side.
  • Invalid SIDs are dropped silently. pack_full_key(row_codes) returns None if len != 6 or any code is outside 0..=255, and the skipped counter goes up then.

• A seed's SID is looked up first in the local trie, and if it is missing, a separate gRPC service is asked verified — src/seed_sids.rs:1-114.

  • The external service is SidLookupService (xai-recsys-sid-proto, with the endpoint set by the --sid-endpoint flag). It looks up in batches and puts the results in a moka cache (2 million entries, 6-hour TTL).
  • That is, the service receives seed SIDs from two places. One is its own parquet snapshot (= all corpus-side codes), and the other is the external service (= presumably the real-time code publishing line on the Phoenix side).

  Historical context: SidLookupService is, in effect, the lookup service that disappeared in Gen4+15 (2026-09-04, 9b0dc31), when the "SID retrieval sidecar was removed wholesale from the Phoenix serving engine", now reborn here as a separate external service. At the time, the sidecar served as a lookup that "converts each post ID in the user history into an SID" on Phoenix requests; after it disappeared, history SIDs were absorbed into a form that arrives preloaded in the request. Now the shape has changed so that the retrieval server calls that service when a seed → SID conversion is needed.

• The server emits 3 kinds of metrics verified — src/metrics.rs.

  • sid_retrieve_requests{caller, result=ok|empty|unavailable} counter
  • sid_retrieve_seeds{source=index|sid_service|missing} counter
  • sid_retrieve_candidates{caller} and sid_retrieve_latency_seconds{caller} histograms

• The server's default behavior is set by run flags. By default it returns up to 800 results and 100 per seed, and searches prefix depths 3–6 verified — src/main.rs:26-52.

  • --grpc-port=9090 · --http-port=8080 · --snapshot-poll-secs=30 · --sid-timeout-ms=200 · --sid-cache-size=2_000_000 · --max-seeds=200 · --default-max-results=800 · --default-max-per-seed=100 · --default-min-prefix-depth=3 · --default-max-prefix-depth=6

• The new source passes the top authors by follower count to Thunder as if the user follows them, and emits the posts Thunder returns as candidates verified — home-mixer/sources/popular_posts_source.rs:68-115.

  1. PopularPostsSource runs only when !has_cached_posts && !in_network_only && EnablePopularPostsSource==true.
  2. It calls PopularAuthorsCache.maybe_spawn_refresh(now_ms). It starts a background refresh while preventing duplicates with a single AtomicBool.
  3. top_author_ids(top_k) picks the author_ids of the top K (default 1,000) by follower_count in descending order.
  4. It sends Thunder GetInNetworkPostsRequest { user_id, following_user_ids=[top K authors], max_results=500, per_author_limits={3, 0}, exclude_tweet_ids=seen_ids, algorithm, debug=false, is_video_request=false }. Thunder treats those K authors as if the user follows them and returns posts by recency from its own in-network post store.
  5. The response posts go out in PostCandidate with the label served_type=ForYouPhoenixRetrieval. It is the same label as PhoenixSource.

  let authors = self.popular_authors.top_author_ids(top_k);
  ...
  let request = GetInNetworkPostsRequest {
      user_id: query.user_id,
      following_user_ids: authors,
      max_results: query.params.get(PopularPostsMaxResults),
      exclude_tweet_ids: query.seen_ids.to_vec(),
      per_author_limits: Some(PerAuthorLimits {
          max_posts_per_author: Some(query.params.get(PopularPostsMaxPerAuthor)),
          max_replies_reposts_per_author: Some(
              query.params.get(PopularPostsMaxRepliesRepostsPerAuthor),
          ),
      }),
      ...
  };
  

  — popular_posts_source.rs:87-101

• The popular authors cache is stored in its own binary format v1 verified — util/popular_authors.rs:64-104.

  • The cache is PopularAuthorsCache, and the format is version=1(1B) + updated_at_ms(8B BE) + count(4B BE) + [author_id(8B BE), follower_count(8B BE)] * count.
  • For 1,000 authors it is 13 + 16,000 = 16,013 bytes. The design stores everything under a single key in Manhattan (X's internal key-value store).
  • The refresh period is 60 seconds when the cache is empty and 60 minutes when it is not. It is a common pattern of retrying aggressively only when the cache is empty.

• Five parameters were added, and the source itself is off by default verified — home-mixer/params/param.rs.

• The default of the ads brand safety V2 verdict switch changed from false to true verified — home-mixer/params/param.rs.

  • The switch is EnableAdsBrandSafetyVerdictV2. After the BrandSafetyVerdictV2 logic was released in Gen4+2 (2026-08-17, b089ce6), the default stayed off until today, when it became on for the first time.
  • This file is a cron mirror that syncs production config values daily.
  • The value arrived at the mirror sync point (last sync 2026-10-01T16:00:46Z → 2026-10-02T16:00:41Z), and it appears to mirror a state pushed by a production cron at some point during 2026-10-02 inferred.

• A new author label fallback switch was added, off by default verified — same file.

  • The switch is EnableAdsAuthorBrandSafetyFallback. When an author-level Grok label (GrokSfa/GrokNsfaLimited/GrokNsfa) exists but a post-level label does not, it is a path that puts the author label in the post label's place as a fallback.

• The verdict grades and the author fallback labels are defined as two enums verified — home-mixer/models/brand_safety.rs:6-80.

  #[repr(i32)]
  pub enum BrandSafetyVerdict {
      Unspecified = 0, Safe = 1, LowRisk = 2, MediumRisk = 3, HighRisk = 4,
  }
  pub enum AuthorBrandSafetyFallback { Sfa, NsfaLimited, Nsfa }
  

  — brand_safety.rs:6-21

• The verdict has 4 levels: Safe / LowRisk / MediumRisk / HighRisk. It is consumed by the adjacency matrix (MultiRiskAdsBlender in Gen4+10) that pairs with AdAdjacencyControl.brand_safety_risk on the ads blender side.

  • That is, the axis that decides "post safety → whether an ad can be inserted" on ads surfaces is on by default from today.

• A VF staging folder that did not exist until yesterday was newly created with 6 new files, 1,334 lines verified. Including 3 moved files, the additions total 1,636 lines.
  • The folder is visibility-filtering/staging/.
  • Changes by file: recording.rs(+744 new), replay_corpus.rs(+299 new), capture.rs(+221 new), reference/tweetypie.rs(+204/-81 renamed), reference_compare.rs(+63/-39 renamed), reference/mod.rs(+35/-72 renamed), main.rs(+39 new), serve.rs(+29 new), mod.rs(+6 new).
• This subsystem has its own crate entry point and is meant to run VF in a replay and capture mode fully separated from live traffic verified.
  • The entry point is staging/main.rs.
  • The referential comparison (Tweetypie reference) code that was scattered elsewhere in the VF tree until yesterday was also moved here.
• Three things to examine in a deep dive in the next window:
  • What recording.rs (744 lines) records and in what format (presumably parquet/jsonl)
  • How capture.rs samples live traffic
  • What the replay contract in replay_corpus.rs looks like

• The checkpoint restore side of the row-sharded embedding CUDA kernel bundle released in yesterday's commit (Gen4+34) was completed in Python.
  • The kernel bundle is cuda/row_emb/, and the restore file is phoenix/xrex/train/row_embedding_restore.py (+488 new). In effect, yesterday's shared adapter (recsys_async_emb.py) and today's restore utility have both arrived. A deep dive is needed.
• A module for server-side latency metrics only was added to the Phoenix ranking server, separate from the inflight metrics. It widens operational observation and is not a model-side change.
  • phoenix/crates/serving/xai-recsys-engine/src/grpc_server_latency.rs (+361 new)
• There was a reorganization that cleanly detaches utilities that the SID retrieval trainer/model, split out yesterday, had shared with the common code. The common code has a net decrease (-477).
  • trainer_recsys.py(+376/-853), recsys_model.py(+786/-431)
• A client that fetches author-level Grok labels on the ads side was added. The fallback path in C above uses this client.
  • home-mixer/clients/author_brand_safety_client.rs (+281 new)
• is_arm_gated_retrieval, deleted in yesterday's commit (Gen4+34), is used again on the author cold-start side. It was revived as a different branch specific to cold start.
  • home-mixer/scorers/author_cold_start.rs(+53)
  • That is, yesterday's deletion was not "abandoning this axis itself" but "moving it out of the retrieval branch to the author cold-start side".
• The SimClusters ANN cache client was split out from the shared crate.
  • home-mixer/clients/simclusters_ann_cache_client.rs (+211 new)
• A pacing-input axis was declared in VMRanker (the stage that reorders the candidate bundle to send to the feed). The linked param VMRankerSendPacingInputs is off by default.
  • vm-ranker/scoring/value_model.rs(+96/-5), scoring/mod.rs(+85/-2)

1. The "actual server crate" was attached today to the three axes of generative SID retrieval released yesterday (home-mixer source, Phoenix model/trainer, proto). It is the third and last piece of a release spread over three days.
   • Until yesterday there were only the training recipe and the home-mixer source skeleton, and the server was empty.
   • The xai-recsys-sid-retrieval crate released today states three things explicitly, so the server's actual query complexity can be measured from released code alone.
     • It is a fully static index that reloads the whole parquet snapshot every 30 seconds. There are no real-time upserts.
     • It does range search over a 6-level trie packed into a 48-bit u64 key, based on partition_point.
     • It uses an algorithm that trims recall with reverse depth order and round-robin interleave.
   • Until yesterday this slot read as "the server will do generative decoding". Today's release reveals the design choice that generation is done beforehand on the corpus side (a batch code table) and the server only does prefix-range search.
   • That is, there is no inference-time beam search on the server. Codes the model produced come in contained in the corpus, and the server is a KNN-like server that searches by seed code and shared prefix length.
   • Since EnableSidSource=false is still the default, the checkpoint to watch for live traffic is the day it turns ON in the FeatureSwitches cron.
2. PopularPostsSource reuses Thunder's in-network index for "global popularity". It places the results under the ForYouPhoenixRetrieval label and mixes them with Phoenix two-tower retrieval (retrieval that embeds users and posts separately and matches them) responses.
   • It is interesting that it is designed not to be distinguishable from PhoenixSource by label. To a scorer/filter it looks like "Phoenix retrieval returned K more", so it mixes in at the upper ranking stages with the same distribution as PhoenixScore.
   • The mechanism is not embedding-based retrieval. It is a retro approach that feeds the "global popularity centroid of the follow graph" into Thunder as "virtual following".
   • It is a typical global popularity backfill pattern, used "so that Thunder responds with something even if the user follows no one" or "to fill the gap when other OON sources cannot return results because user engagement signals are too few" inferred.
   • Since EnablePopularPostsSource=false is still the default, the actual activation time is the checkpoint to watch in the next window.
3. Both ads changes expand "SFA (Suitable For Advertising) labeling from post level → author level".
   • It is no coincidence that the switch of EnableAdsBrandSafetyVerdictV2=true and the introduction of EnableAdsAuthorBrandSafetyFallback arrived on the same day. The direction is to use the author's past label pattern as a fallback for posts that do not yet have a post-level Grok label.
   • This is a signal that they intend to complement, with author-level aggregation, situations where Grok classifier coverage is sparse per post (especially for new posts) inferred.

Radar

No papers were added to the radar today. Of the 30 requested in the arXiv cs.IR 2026-10-01–2026-10-02 window, all 30 <entry> items were parsed, and 0 of them cleared all three acceptance criteria.

• ① Platform authors or a production A/B test
• ② Mechanism novelty
• ③ Large-scale results

Three borderline items were also all rejected.

• arXiv 2609.39312 is Learning Multiresolution Relevance for Hierarchical Generative Retrieval by Weihao Shen and 7 others. It learns multiresolution relevance in hierarchical generative retrieval. It was rejected because the platform and deployment axis of ① is empty.
  • The author affiliation could not be confirmed directly from arXiv primary metadata, and the abstract has no production/deployment claim.
• arXiv 2609.39319 is Residual Trajectory Distillation for Generative Retrieval by the same 7 authors. It is an approach that distills the intermediate trajectory of SID RQ (residual quantization). It was rejected for the same reason.
  • It is an academic line paired with 2609.39312, and neither makes an industry deployment claim.
• arXiv 2609.39828 is KUAISHOU Explorer LLM-Rec Challenge 2026: Reasoning Generative Recommendation by Kuaishou authors. The body is a challenge overview with no mechanism novelty, so it was rejected on ②, mechanism novelty.
  • It is merely a source reconfirming that OneRec/OneRec-V2 were deployed at Kuaishou (a line already on the radar).

In the 2609.39xxx number range posted in the window after the Meta ETT% recommendation training infrastructure paper (arXiv 2610.02057) that was rejected from yesterday's radar, no new platform-affiliated mechanism worth tracking was found.

What to check next