2026-08-08

This post is a work log. Evidence grades and file:line quotes are all collected in services/x.md and topics/*.md, and are not repeated here.

Sources read

I read the directories below from the source X released as Gen3 and from the 2023 machine learning repository.

• Gen3: all of the following directories of xai-org/x-algorithm (2026-01-20 aaa167b, 2026-05-15 e414c17·0bfc279)
  • home-mixer/ (the service that assembles the For You feed)
  • thunder/ (Thunder, an in-memory index that returns the latest posts of followed accounts)
  • phoenix/ (Phoenix, the Transformer ranking model and two-tower retrieval model)
  • grox/ (nearline LLM annotation pipeline)
  • candidate-pipeline/
• Gen2: home-mixer/ and tweet-mixer/ of twitter/the-algorithm @ c54bec0 (2025-09-03 refresh)
• Gen1: simclusters_v2/, simclusters-ann/, ann/ and cr-mixer/ of twitter/the-algorithm @ 72eda9a (2023)
• projects/home/recap/ and projects/twhin/ of twitter/the-algorithm-ml (2023)

↑ Contents

The 5 most important things learned today

1. Between Gen1 and Gen2 there is a 26-month gap, and 0 files were deleted in Gen2. The 2023 file set is a proper subset of the 2025 file set. Therefore a file remaining in the repository does not mean at all that the file is live. Retirement shows up only as a change to a single README line.
2. The "in-network 50:50" rule does not exist in the code. The 2026 code has no allocator at all. OON (out-of-network, posts from accounts the user does not follow) is controlled not by a quota but by a score multiplier. In effect, the allocation problem has moved into a ranking problem.
   • This multiplier is effective_oon.
3. xai-org/x-algorithm does not build. There are 0 Cargo.toml files, and the crate::params, util and clients modules are missing entirely. 60 files reference params, but its definition is nowhere. In other words, only the shape of the logic was released; the values and the wiring were left out. This is a consistent design, not an accident.
4. The 3 scorers the README points to are cut off from the module tree. What is actually wired in is vec![phoenix_scorer, ranking_scorer, vm_ranker].
   • Settled the list of 11 dead files.
   • Also confirmed that the dual layout of models/ and candidate_pipeline/ is the criterion that separates the generations.
5. No generation has a primary source for the weight values.
   • The widely cited "reply 13.5 / report −369" table came not from a blog post but from the README of the-algorithm-ml. That document states itself: "as of 2023-04-05, subject to change at any time".
   • The 2025 Scala erased all 30 ModelWeights by setting default = 0.0. The other 203 defaults in the same file carry real values, so this should be read as a deliberate deletion, not a coding style.

Smaller findings are as follows:

• A single cache flag, has_cached_posts, is the biggest switch, turning off all sources, hydrators and Phoenix at once.
• The backup filter for removing already-seen posts (PreviouslySeenPostsBackupFilter) is a permanent no-op.
• VMRanker (a remote service that re-orders candidates to reflect diversity) fails silently without leaving a log. So it is hard to notice that diversity has disappeared.
• Two Python files in grox/ cannot even be imported because of literal substitution.

↑ Contents

What was overturned in verification (important)

When I re-verified the draft with a verifier, one conclusion came out reversed, and I also learned one new rule for citing constants.

• Overturned: The conclusion that "2025 tweet-mixer's u2i candidates are appended after the cap truncation and get preferential treatment" was wrong. Looking at only one cutoff gives the opposite direction.
  • tweet-mixer is the service that handled OON candidate generation in the 2025 generation. u2i is the path that finds candidates from user to post (user→post).
  • Because DropMaxResults truncates only result and keeps remainingCandidates, u2i sits at the back of the list.
  • The final cutoff after that is 400 (smaller than the light-ranker cap of 500), so when the front is full, u2i disappears first.
• New rule: A number found by grep must not be used as a production value as is. All three cautions actually appeared.
  • The default of args.int(name, default): the SimClusters (embeddings built by splitting the follow graph into communities) maxNeighbors is written as 400 in the code. However, the production run command recorded in the same file is --maxNeighbors 100. maxEpochs also differs: code value 3 and production value 4.
  • Literals used in benchmarks and warmup: the HNSW (approximate nearest neighbor search index) efConstruction=200 and maxM=16 are experimental values in experimental/Runner.scala. ef=800 is inside warmup().
  • Constants in files cut off from the module tree: p::AUTHOR_DIVERSITY_DECAY in the 2026 dead scorer is an example.
• Gained: Recovered from the 2025 code one constant that the 2026 code does not release.
  • The 2026 offset_score is code carried over as is from 2025 RerankerUtil.weightedScoresSum. So the undisclosed constant NEGATIVE_SCORES_OFFSET was recovered as the 2025 Epsilon = 0.001.
  • This case is an exception to the conventional wisdom that code duplication between the two repositories is 0.
  • Other undisclosed constants appear to be recoverable the same way.
• Other corrections: A few other facts were also corrected.
  • FailOpenPolicy.Always applies to only 8 of 9, so only ContentExploration is fail-closed.
  • 2025 MMR (a technique that re-orders by weighing relevance and diversity together) applies no diversity at all to the first (1-diversityRatio)×n items of the list.
  • QuoteDeduplicationFilter removes the quoted original, not the quote post.
  • The tweet-mixer pipelines number 34, not 35.

↑ Contents

Canon update

The X canon and the topics notes were all newly created.

• services/x.md: Gen3 comparison table, surfaces, 11 candidate generation (retrieval) sources, 22 Phoenix prediction heads, 17 filters, ads blending, cold start, 11-item caveat list
• topics/sequence-transformer-ranking.md: Phoenix structure and masks
• topics/llm-in-recsys.md: Grok/Grox annotation pipeline
• topics/multi-task-ranking.md: per-generation head lists, weight opacity
• topics/candidate-allocation.md: ratios, caps, interleaving and the rebuttal of "50:50"
• topics/feed-diversity.md: decay, MMR, DPP (a probabilistic model that picks items so that similar ones are not chosen together), dedup
• topics/embedding-retrieval.md: SimClusters, TwHIN, two-tower, ANN index

↑ Contents

What to check next

↑ Contents