X (Twitter)

Summary

As of the 2026-10-20 commit (74d49a4, Gen4+45), X For You gathers the latest in-network posts from Thunder and OON candidates from Phoenix retrieval, SimClusters, and a popular posts source that has been on by default since 2026-10-07, while the legacy tweet-mixer and SID retrieval sources are off by default. Phoenix, a Grok-based Transformer, predicts per-action engagement probabilities for each candidate, and since 2026-09-24 the shared crate xai-value-model applies an author-diversity decay and an OON multiplier to the weighted sum of these probabilities. There are 20 filters before scoring and 5 after selection, and the sorted candidates then pass through VMRanker DPP slate re-ranking (θ=0.65, K=150, skipped for topic requests) and blending of ads and follow recommendations.

↑ Contents

Released source up to Gen4 — read this first to keep the rest straight

X's recommendation code release is divided into four generations. Gen1 is Scala/Java and Gen2 is Scala, both from the Twitter repo, while Gen3 and Gen4 are Rust/Python code from the xAI repo. From Gen4 on, the weight parameter mirror, the safety pipeline, and the training code were also released verified.

A direct check of the git log on 2026-09-10 showed that xai-org/x-algorithm had 18 commits (+1 from the previous day: 49815da) and that twitter/the-algorithm was unchanged at 31 commits verified.

These notes number the xai-org/x-algorithm commits that came after the Gen4 release (47c1bcd, 2026-08-13) sequentially, starting from Gen4+1. Per-commit details are in the change log at the end of the file.

Four facts matter most:

1. There is a 26-month gap between Gen1 and Gen2. There was not a single commit between 2023-07-07 and 2025-09-03 verified.

2. Gen2 deleted nothing. There were 0 deletions and 0 renames. The 2023 file set of 6,906 files is a proper subset of the 2025 file set of 7,667 files. In other words, cr-mixer/, navi/, ann/, simclusters-ann/, src/, product-mixer/, and others remain exactly as they were, with not a single byte changed since 2023-04. The only top-level paths the 2025 commit touched are home-mixer/, tweet-mixer/, and README.md, with 603, 384, and 1 files respectively verified.

   • The command used for the check is git show --name-only --pretty='' c54bec0 | awk -F/ '{print $1}' | sort | uniq -c.

   A file in the repo does not mean the code is in use now. Deprecation shows up only as a line removed from the README, not as a file deletion. cr-mixer is exactly such a case. The only documentation change in the 2025 commit replaced the service name in the README's cr-mixer row with tweet-mixer and left the wording as is verified — git show c54bec0 -- README.md.

3. Gen3 is a separate repo. The first 3 commits of xai-org/x-algorithm (aaa167b→e414c17→0bfc279) rewrote the code and added grox/, reaching 216 files verified.

4. Gen4 changed the nature of Gen3. Through Gen3, only the shape of the logic was released, while values, wiring, and safety rules were not public, but Gen4 released the following three things verified — README.md #latest-update--august-13th-2026.

   • Released a production cron mirror of the weights and constants as home-mixer/params/param.rs verified — same README.
   • Released the entire safety pipeline through visibility-filtering/ and 15 labeling sub-repos verified — same README.
   • Made the system runnable end to end by including training code and synthetic data in phoenix/ verified — same README.

   The detailed analysis is in daily/2026-08-15.md.

↑ Contents

Glossary

↑ Contents

Feed surfaces

This document mainly covers For You, the feed surface the algorithm recommends. The vertical video carousel inside For You was introduced, off by default, in the 2026-10-08 commit (Gen4+38) verified (Gen4+38 in the change log); in the 2026-10-15 commit (Gen4+42), a VQV prediction term from the immersive surface model was added to the carousel ranking score, but its default weight is 0.0, so default behavior is unchanged verified (Gen4+42 in the change log). Since the 2026-10-17 commit (Gen4+44), the three dedup checks inside the carousel are on by default, and the carousel is not inserted when the same response has a Who-to-Follow module verified (Gen4+44 in the change log). Since the 2026-10-20 commit (Gen4+45), the carousel excludes videos whose media already appears in the top 35 of the main feed, and the carousel itself is still off by default as of that commit verified (daily 2026-10-21); the video-only path implemented in Thunder is turned on by a separate surface that is not public (Reels equivalent) inferred.

• For You is the algorithm-recommended feed this document mainly covers, and its gRPC methods are get_for_you_feed and get_for_you_feed_urt verified 2026 — home-mixer/server.rs:273-346
• A vertical video carousel module, which places 3–5 vertical videos in the 6th slot of the For You feed, was introduced off by default in the 2026-10-08 commit (Gen4+38) verified — home-mixer/util/video_carousel.rs
  • Off by default means EnableVideoCarousel = false in home-mixer/params/param.rs.
  • It is enabled only on For You requests and appears again only after 60 minutes (VideoCarouselFatigueMinutes) have passed since the last carousel impression.
  • The slot is the 6th position in the feed (VideoCarouselPosition). A vertical video is one whose width-to-height ratio is below 1.0.
  • Since the 2026-10-15 commit (Gen4+42), the ranking score is the video_open weight plus the VQV prediction weight of the immersive surface model (GalleryPage). Because the VQV weight defaults to 0.0, default behavior is still ordering by video_open verified — home-mixer/util/video_carousel.rs:104-123.
  • Since the 2026-10-17 commit (Gen4+44), the three dedup checks inside the carousel (same author, first 3 SID levels matching, multimodal embedding cosine of 0.72 or higher) are on by default verified — params/param.rs:1095-1112.
    • From the same commit, the carousel is not inserted when the same response has a Who-to-Follow module (no parameter) verified — selectors/blender_selector.rs:50, 156-161.
    • The carousel itself (EnableVideoCarousel) is still off by default in the 2026-10-20 commit (74d49a4) verified — params/param.rs:1102.
  • Since the 2026-10-20 commit (Gen4+45), the carousel skips videos that have the same media ID or MD5 hash as the top 35 of the main feed, and videos with the same media as one picked earlier within the carousel (no parameter) verified — util/video_carousel.rs:52-108.
    • The main feed range is cut at the same boundary as the carousel candidate pool (RESULT_SIZE) — filters/video_carousel_filter.rs:26-29. Details are in daily 2026-10-21.
  • At introduction (Gen4+38, 2026-10-08), vertical videos were grouped in order of predicted video_open probability.
  • The detailed selection path is in Gen4+38 in the change log.
• Following handles only in-network posts and, as of the 2026-10-17 commit (862a123), has a gRPC service and candidate pipelines separate from For You verified — home-mixer/lib.rs
  • The services are FollowingFeedService and RankedFollowingFeedService, and the pipelines are following_candidate_pipeline.rs, ranked_following_candidate_pipeline.rs, and reverse_chron_posts_pipeline.rs. The Gen4 release (47c1bcd) already had the same structure verified — home-mixer/server.rs:522-680, home-mixer/candidate_pipeline/
  • Both Following services set in_network_only to true after building the query verified — home-mixer/server.rs:555, 658
  • Important: Even for a For You request, if the user has allow_for_you_recommendations = false, the server demotes the request to in_network_only verified — home-mixer/server.rs:96-97
  • The product_surface value passed to Phoenix also splits into HomeTimelineRankedFollowing and HomeTimelineRanking depending on this flag verified — home-mixer/scorers/phoenix_scorer.rs:86-90.
  • Gen3 (0bfc279) had no Following-specific service and was split only by the in_network_only flag of the same pipeline verified — home-mixer/server.rs:75-76
  • In the 2026-08-18 commit (Gen4+3), a VF hydrator dedicated to reverse-chron (Following) was branched off verified — home-mixer/candidate_hydrators/vf_following_candidate_hydrator.rs
    • The new hydrator is a new file of 95 lines.
    • Another evidence location is home-mixer/candidate_pipeline/reverse_chron_posts_pipeline.rs:141-144.
    • Previously, For You and Following used the same VFCandidateHydrator.
    • The new hydrator is connected only to the Following pipeline.
    • The should_drop_ancillary helper is reused from the existing file as pub(crate).
    • The new hydrator returns only the VF lookup results (visibility_reason, drop_ancillary_posts) and resets the remaining candidate fields with ..Default::default().
    • In other words, the Following pipeline's hydration output shrank to the two VF results, and the remaining fields are filled by other hydrators.
    • VF's SafetyLevel remains TimelineHome. Following and For You share the safety level.
    • Why it matters: With this commit, the hydration field sets of For You and Following were explicitly split; this is an optimization direction that avoids spending logic and resources on candidate fields Following does not use inferred
• A topic feed is a request carrying topic_ids; if there are 7 or more topic ids (> 6), it is treated as a "bulk topic request" and the candidate sources differ verified 2026 — home-mixer/models/query.rs:201-203
• Video surface: Thunder has a dedicated video ring buffer and an is_video_request path that are fully implemented, but the only public caller hardcodes is_video_request: false verified(code) — thunder/thunder_service.rs:287
  • Another evidence location is home-mixer/sources/thunder_source.rs.
  • Because the implementation is complete and not a stub, a separate surface that is not public (Reels equivalent) turns this flag on and uses it inferred.

↑ Contents

Candidate generation (retrieval)

Candidate generation (retrieval) is the stage that gathers posts from multiple sources to hand to ranking. The 2026 pipeline runs the sources in parallel and concatenates the results as is, with per-source caps instead of an allocation ratio between in-network and OON verified (Candidate allocation section). Since the 2026-10-07 commit (Gen4+37), the source that serves a popular posts list built in advance by an offline job is on by default verified (Gen4+37 in the change log). The SID generative retrieval source released in the 2026-10-02–03 commits (Gen4+34–35) is off by default verified (Gen4+34, Gen4+35).

Current (2026-08 Gen4)

As of the Gen4 release, For You has 12 candidate sources in total: 7 internal Phoenix pipeline sources and 5 external For You pipeline sources verified. The evidence is home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs, for_you_candidate_pipeline.rs, and the EnableXxxSource defaults in home-mixer/params/param.rs.

On the same basis, the retrieval sources confirmed on by default are Thunder, which serves the latest in-network posts, and Phoenix retrieval and SimClusters, which serve OON candidates verified. The SID source and the popular posts source in the table were added later, in the 2026-10-02–03 commits (Gen4+34–35) verified (details below). The table values are as of the Gen4 release, but the rows for these two sources and the cold slice cap of PhoenixSource are items added in later commits.

The details of the three sources shortened in the table are as follows:

• In the 2026-09-18 commit (Gen4+25), a cold-start slice was wired into PhoenixSource as a first-class path via the dataset_type tag verified (Cold start section).
  • If a response candidate has dataset_type == HOME_COLD_DATASET_TYPE(=13), served_type is ForYouPhoenixRetrievalCold; otherwise it is the existing ForYouPhoenixRetrieval.
  • This slice's cap PhoenixColdStartMaxResults was introduced in Gen4+25 at 0 (off by default) and has been 200 since the 2026-09-30 commit (Gen4+32) verified — params/param.rs:16. It is still 200 in the 2026-10-17 commit (862a123).
• SidSource calls the SID generative retrieval microservice released in the 2026-10-02–03 commits (Gen4+34–35) verified (Gen4+34, Gen4+35).
  • When the user's positive engagement post IDs are passed as seeds, the service builds candidates in three steps:
    1. Convert the seed into a 6-level code. If the seed post is in the local SID trie, look it up there; otherwise, look it up in the external SidLookupService.
    2. For each seed, pick candidates from the shared-prefix range in order of decreasing depth, from max_prefix_depth down to min_prefix_depth. Ranges already consumed at a deeper level are excluded.
    3. Round-robin interleave the candidate lists of all seeds, dedup, and return the result.
  • Seeds are extracted from user engagement signals by reusing the same function as simclusters_source::post_signal_ids.
  • Each Candidate carries shared_prefix_depth as its score.
  • Response candidates are labeled served_type=ForYouPhoenixRetrievalMoe. This is the same label as PhoenixMOE.
  • The call has a 300ms gRPC deadline, and the connection path has two routes: xDS P2C → DNS fallback.
  • The server polls the parquet snapshot (post_sid_snapshot.parquet, columns post_id/author_id/post_sid) every 30 seconds and reloads it in full.
  • The trie consists of two vectors: by_key(prefix range binary search) and by_post(seed lookup).
  • Server-side limits are max_results=800 / max_per_seed=100 / min_prefix_depth=3 / max_prefix_depth=6 / max_seeds=200, and the hard caps are MAX_RESULTS_CAP=5000 · MAX_PER_SEED_CAP=2000 · MAX_CALLER_LEN=64.
• PopularPostsSource was created in the 2026-10-03 commit (Gen4+35), and its structure was replaced wholesale in the 2026-10-06 commit (Gen4+36) verified (Gen4+35, Gen4+36).
  • The replaced structure does not call Thunder at request time. It reads the popular posts list (up to 500) built in advance by two offline jobs from Manhattan KV every 60 seconds, removes only the posts the user has already seen (seen_ids), and returns the rest as candidates as is.
  • The first job, popular_authors_job (hourly), picks from a BigQuery snapshot the top 0.005% by follower count among authors who posted in the last 7 days (TOP_POSTING_AUTHORS_FRACTION=0.00005).
  • The second job, popular_posts_job (every 5 minutes), gathers those authors' original posts (excluding replies and retweets) from Thunder. It scores posts from the last 24 hours using the view counts from engagement counts, then trims to 5 per author and 500 overall.
    • The scoring formula is quality = views × (1−e^{−λ·24h}) / (1−e^{−λ·max(age,0.5h)}) (half-life 8 hours).
  • Since the 2026-10-16 commit (Gen4+43), the label is its own served type ForYouPopularPosts; before that it was ForYouPhoenixRetrieval (popular_posts_source.rs:13).
  • It is turned off for cache-reuse requests and in_network_only requests (popular_posts_source.rs:27-31).
  • PopularPostsTopAuthors, PopularPostsMaxPerAuthor, and PopularPostsMaxRepliesRepostsPerAuthor, which existed in Gen4+35, were deleted and moved to job CLI arguments (--popular-posts-per-author=5, --popular-posts-budget=500, --popular-posts-half-life-hours=8).
  • Storage format v1 is a 13-byte header followed by 24 bytes per post (post_id, author_id, quality).

Thunder is an in-network index newly built in 2026. It replaces the Earlybird search index inferred. The evidence is that the roles are the same and references to Earlybird disappeared.

• Posts are held in DashMap<post_id, LightPost>, and each author has 3 ring buffers (original, reply/retweet, video) verified — thunder/posts/post_store.rs:20-53
• LightPost has no text, no media, no engagement counts, and no safety labels verified — thunder/kafka/tweet_events_listener.rs:230-248
  • The only fields it holds are post_id, author_id, created_at, in_reply_to_*, is_retweet, is_reply, source_*, has_video, and conversation_id.
• Ranking inside Thunder is deliberately a single line of recency sorting, because Phoenix re-scores anyway verified — thunder/thunder_service.rs:334-339
• The default retention period is 2 days verified — post_store.rs:521-526
• Under overload, it rejects requests immediately instead of queuing them verified — thunder_service.rs:160-171
  • If try_acquire fails, it becomes resource_exhausted.
• The reply exposure rule is a product policy stated explicitly in code verified — post_store.rs:291-315
  • If the parent is an original post, the reply passes.
  • If the parent is not an original post, the reply passes only when the parent is a direct reply to the conversation root and the reply target is someone the viewer follows.
  • If the parent is not in the store, the reply is simply dropped. So when a thread root older than 2 days expires, all replies under it become invisible.
• A user's own retweet of themselves is blocked by one line in the hot path verified — post_store.rs:287-289

The retrieval sources that survived from the 2023 generation are tweet-mixer and SimClusters. tweet-mixer is mirrored as off by default, and SimClusters was revived. ⚠ Claim changed (2026-08-13): the canon had long stated explicitly that "SimClusters, TwHIN, UTEG/GraphJet, FRS, Earlybird, Lists, Communities — none of these names appear anywhere in home-mixer/". The Gen4 release brought the simclusters/ sub-repo back as a home-mixer OON retrieval source (EnableSimclustersSource=true, on by default). TwHIN/UTEG/GraphJet/FRS/Earlybird/Lists/Communities are not in Gen4 either verified — home-mixer/params/param.rs, README.md #candidate-sources

The details of the SimClusters core algorithm (SVD pipeline, k, index format, update cycle) need a deep dive. For the simclusters/ sub-repo, only the file list was checked and the individual files were not read. The items that need a deep dive are in the list in daily/2026-08-15.md.

Candidate allocation — per-source caps, not ratios

"In-network 50% / out-of-network 50%" is not a quota.

The 2026 pipeline has no allocator, no ratio parameter, and no global cap verified — candidate-pipeline/candidate_pipeline.rs:257-272

• It runs the sources in parallel with join_all and simply concatenates the results.
• A source that errors out silently disappears.

The max_results value each source reads from a FeatureSwitch parameter was released as a production cron mirror in the Gen4 release (2026-08-13) verified — home-mixer/params/param.rs

• ThunderMaxResults = 1200
• PhoenixMaxResults = 1000
• TweetMixerMaxResults = 800. However, it is off by default with EnableTweetMixerSource = false.
• For PhoenixMOEMaxResults, the corresponding param must be checked.

In the 2026 pipeline, the means of adjusting the in-network versus OON share is a scoring multiplier, not retrieval verified — home-mixer/scorers/ranking_scorer.rs:220-239, 272-275

let final_score = match c.in_network {
    Some(false) => after_diversity * effective_oon,
    _ => after_diversity,
};

effective_oon is computed in three branches, and all of its values were released in the Gen4 release verified — home-mixer/params/param.rs:247,269,275

A generalization note is in candidate-allocation.md.

Legacy (2025 Scala, tweet-mixer)

tweet-mixer appeared all at once on 2025-09-03 with 384 files. It has no README and not a single .md file, so every fact has to be confirmed from the code alone verified.

The retrieval families newly introduced in this generation are as follows:

• 19 new DeepRetrieval-related files appeared (0 in the 2023 snapshot) verified
• 20 new files that consume SimClusters appeared, and the SimClusters core itself is unchanged from 2023 verified — git show --diff-filter=M c54bec0 | grep -ci simclusters = 0
  • The new files include SimclusterColdPostsCandidateSource.scala.
  • The core is src/scala/com/twitter/simclusters_v2 and simclusters-ann/.
  • That is, the engine was left untouched and only the points of consumption were added.
• 15 new files that consume Earlybird appeared, and the index itself (src/java/com/twitter/search) did not change verified
• Hydra already existed in the 2023 cr-mixer as a thrift client, and in 2025 it was promoted to a scorer and an embedding generation service. "Hydra is new in 2025" is wrong verified

How SimClusters, TwHIN, and DeepRetrieval each generate embeddings, build indexes, and do lookups, along with the code defects found in them, is summarized in embedding-retrieval.md.

↑ Contents

Ranking

As of the Gen4 release (2026-08-13), Phoenix, the ranking model of X For You, predicts the probability of actions such as like, reply, and report for each candidate post verified(Phoenix section). As of the same release, the score starts from the sum of these probabilities multiplied by per-action weights, and after an offset it applies author-diversity decay, the OON multiplier, and the new-author boost in that order verified(Phoenix section, new-author boost section). Since the 2026-09-24 commit (Gen4+28), the shared crate xai-value-model does this calculation, and home-mixer's ranking scorer file was deleted verified(Phoenix section). In the params of the same release, VMRanker's DPP slate (the bundle of candidates sent out to the user) reordering is also on verified(VMRanker table in the Phoenix section). Since this release, the weight values can be read in the repo as a cron mirror that copies the production defaults daily, and some mirror values changed up to the 2026-10-14 commit (Gen4+41) verified(weights section, Gen4+41 in the change log).

Phoenix (2026, current) — Gen4 release update

Phoenix is a Transformer ported from Grok-1. It puts the user history and the candidates into one sequence and scores them in a single pass. The structure, masks, and parameter distribution are summarized in sequence-transformer-ranking.md.

There are 24 prediction heads by the documentation, divided into 5 categories verified — README.md #scoring-and-ranking (2026-08-13). The table follows this README; weight terms removed or added since then are in the weights table.

The click dwell time head and the dwell time head, which deal with dwell time, had their training approach or ranking term changed after the release verified(Gen4+22 in the change log, Gen4+23, Gen4+38).

• In the 2026-09-16 commit (fad2f71, Gen4+22), the click dwell time head began to be trained as sigmoid BCE (threshold=10s) binary classification verified(Gen4+22 in the change log). The public training code before that had no loss setting for this head verified — 2d4a03c phoenix/xrex grep.
  • The head name is click_dwell_time.
  • The serving head remains a probability [0,1]. _rescale_sigmoid_heads_for_inference skips restoring the original unit through a binary exception.
• In the 2026-09-17 commit (42266f3, Gen4+23), the ranking term for click dwell was redefined verified(Gen4+23 in the change log).
  • EnableClickDwellLowFavRatePenalty, which had existed since the Gen4 release (47c1bcd), was deleted entirely (param.rs:387-416). This heuristic used 4 tuning params (baseline, alpha, floor, cap) and (fav/baseline)^alpha to lower the dwell prediction when fav was low.
  • A new parameter EnableCdwellOnImpr (off by default) took its place. When on, it computes click_dwell_term = click_dwell_time × click_score. This redefines the click dwell term as expected dwell per impression.
  • ContClickDwellTimeWeight did not change in the mirror.
• In the 2026-10-08 commit (35650fb, Gen4+38), the training loss of the dwell_time head changed from Tweedie and MAE to CREAD verified — phoenix/xrex/configs/xrecsys.py:272-275, 715-742.
  • The newly enabled CREAD loss trains dwell time as binary classification of whether it exceeds each of 30 log-spaced thresholds (2 seconds up to the cap), then converts back to an expected value.
  • In the default home training config, two CREAD losses were turned on at 0.1 each. The cap is 120 seconds for regular surfaces and 240 seconds for gallery/immersive surfaces.
  • In the same config, the Tweedie (0.1) and MAE (0.1) weights became 0.
  • The serving head name and the ranking weight (ContDwellTimeWeight) are unchanged.

There are 2 more params that contribute to the score but are not in the README categories. The total number of effective weight params is 26.

• QuotedVqvWeight is the video quality view weight of a quoted post. It is not in the README categories, but the param exists.
• PostUnexploredWeight is the new-author boost. Details are in the new-author boost section.

The canon's long-standing observation that "the head count splits three ways within the repo (public checkpoint 19 / ranking scorer 22 / VMRanker proto 21)" is a fact up to Gen3. The Gen4 README states 24 by the documentation, and the weight symbols in home-mixer/params/param.rs are also organized as 25 (prediction head) + 1 (post_unexplored heuristic). However, whether the dead 19-field PhoenixScores struct in candidate_pipeline/candidate.rs observed in Gen3 was cleaned up in Gen4 requires a deep dive. verified (documentation) / inferred (whether the code was cleaned up)

The generalized notes on head design are in multi-task-ranking.md.

As of the Gen4 release (47c1bcd), the final score was computed in the following order verified — home-mixer/scorers/ranking_scorer.rs.

1. Take the weighted sum of 26 weights × P(head). negative_sum is subtracted with its sign flipped.
   • Only for original posts from mutual follows, add BidirectionalFollowReplyWeightBoost=15.0 to the reply weight. The condition is !in_reply_to && !retweeted && is_mutual_follow_author.
2. Apply offset_score. If the score is negative, it is (score + negative_sum) / total_sum * NEGATIVE_SCORES_OFFSET; if positive, it is score + NEGATIVE_SCORES_OFFSET.
3. Multiply by author-diversity decay. The k-th appearance of the same author is multiplied by (1-0.25)*0.5^(k-1) + 0.25, so the first appearance is 1.0×.
   • The exponent is the number of times that author has appeared earlier (ranking_scorer.rs:614-615, 635).
   • The settings are AuthorDiversityDecay=0.5, AuthorDiversityFloor=0.25, and EnableAuthorDiversity=true.
4. Multiply by the OON multiplier OonWeightFactor=0.75. For topic requests, use TopicOonWeightFactor=0.5.
5. Apply the new-author boost.

The normalize_score normalization stage that existed in the Gen3 code (0bfc279) is not in ranking_scorer.rs of the Gen4 release verified — ranking_scorer.rs:4, 259.

In the 2026-09-24 commit (Gen4+28, 44d37eb), ranking_scorer.rs was deleted, and the weighted sum, offset, author-diversity decay, and OON multiplier calculations moved to the shared crate xai-value-model verified — xai-value-model/scoring.rs.

• As of the 2026-10-17 commit (862a123), the scorer chain is PhoenixScorer, ImmersiveVideoScorer, VMRanker in that order verified — phoenix_candidate_pipeline.rs:487-488.
  • The chain at the time of the Gen4 release was vec![phoenix_scorer, ranking_scorer, vm_ranker] (phoenix_candidate_pipeline.rs:395-396).
• As of the same reference, home-mixer's VMRanker scorer first computes a local weighted sum (fuse_heads) and the new-author adjustment, then overwrites them with the score returned by the vm-ranker service verified — home-mixer/scorers/vm_ranker.rs:59-131.

The heuristic that sets the vqv weight to 0 depending on the video length and the querying user's follower count has been in the repo's home-mixer/util/candidates_util.rs since the Gen4 release verified — candidates_util.rs:19-60.

• The related identifiers are vqv_weight, quoted_vqv_weight, and the MinVideoDurationMs threshold. vqv_weight uses the weight as is when the video length exceeds MinVideoDurationMs and query.user_features.follower_count is not 10,000 or more. It also uses the weight when the follower count is empty (candidates_util.rs:4, 25-38).
• In the Gen4 release, ranking_scorer.rs:230 called this function.

The VMRanker and DPP settings are now public verified — home-mixer/params/param.rs:579-618. The values in the table are those at the time of the Gen4 release, except VMRankerClusterId, which is the 2026-10-06 value.

• VMRankerClusterId started at "Experiment3" and changed several times in between. It became "Experiment6" in Gen4+35 and "Experiment1" in Gen4+36 (2026-10-06) verified.
• In the 2026-09-24 commit (Gen4+28), the DPP settings were removed from home-mixer's param.rs and moved to vm-ranker/params.rs. As of the 2026-10-17 commit (862a123), the values in that file are DppEnabled=true, DppTheta=0.65, DppMaxSelectedRank=150, and TopicDppEnabled=false verified — vm-ranker/params.rs:235-248.

The VMRanker service itself has been in vm-ranker/ since the Gen4 release verified — vm-ranker/ranker_service.rs.

• The same directory has main.rs, dpp.rs (the DPP implementation), and scoring/dpp_model.rs verified — vm-ranker/.
• The 2026-09-23 commit (Gen4+27) added scoring/value_model.rs, params.rs, and config_sync.rs verified — vm-ranker/.
• The structure in which the VMRanker response overwrites the candidate's score has been unchanged since Gen3 verified — vm_ranker.rs:116-131.
• If the gRPC call fails, since the 2026-09-24 commit (Gen4+28) it emits the local weighted-sum score without DPP and counts the failure with the VMRanker.local_score_fallback metric verified — vm_ranker.rs:107-113, 170-178.
  • Before that, a failure returned Err for every candidate (vm_ranker.rs:44-66).

The 2026-08-18 commit (Gen4+3) newly wired 7 SID (semantic ID) fields into the VMRanker request verified — home-mixer/scorers/vm_ranker.rs:194-206:

• With these fields passed per candidate source alongside the existing fields, VMRanker can now see not only the collaborative pool rank but also the semantic ID hierarchy position in its item diversity decisions.
  • The existing fields are k, pool_rank, and pool_rank_gap.
  • VMRanker keeps the legacy name but is now responsible for DPP slate reordering.
  • This is the X-side connection corresponding to the SID family observed in Kuaishou HD-Rec/PushDualGen inferred.
• Why it matters: VMRanker's diversity judgment now has three axes of information: per-candidate-source rank, the 3-level semantic ID codeword, and the neighbor gap.
  • The existence of a 3-level codebook hierarchy (coarse → fine) is itself newly confirmed architecture information.
  • This wiring is where the rank/rerank side consumes the same SID infrastructure as the SID window in Phoenix retrieval. The SID window (video_14day, video_48h, etc.) is in the Gen4+1 entry of the Phoenix training and index section.
• Undisclosed: The following are not public:
  • The SID codebook definition (vocabulary size and training approach of each level)
  • The exact formula for sid_gap
  • How these fields are used in VMRanker's response logic

New-author boost — the third adjustment factor documented in Gen4

The new-author boost is an adjustment factor that lifts posts from authors whose impressions are below a threshold toward a target position. The README describes it as follows:

"New-Author Boost: posts from authors whose impressions are below a threshold are lifted toward a target position."

The new README states three scoring adjustment factors verified — README.md #scoring-and-ranking.

1. Author Diversity Decay
2. Out-of-Network Discount
3. New-Author Boost

The params that set this boost are in the table below verified — home-mixer/params/param.rs:351-374, #coldstart-threshold.

ColdStartImpressionThreshold was lowered from 1000 to 200 in the 2026-09-30 commit (Gen4+32) verified — params/param.rs:459(Gen4+32 in the change log). It is also 200 in the 2026-10-17 commit (862a123).

• The cold-start slot selector that uses this threshold picks only candidates whose post view_count_on_home is below the threshold verified — author_cold_start.rs:293-294.
• The Gen4 release code made the same comparison using the candidate post's view_count verified — author_cold_start.rs:179.

The canon's observation in the cold start section that "new items: the 2026 code has no mechanism at all" is a fact up to Gen3. Gen4 documented a new-author (account-level) boost. As of the Gen4 release, there was no boost attached to an individual new post (item level), and the hash embedding collision problem was unchanged verified. The later post-level mechanisms are in the cold start section.

Weights — values were released with the Gen4 release (⚠ mass claim change, 2026-08-13)

Conclusion (updated 2026-08-15): The weight values were released as a production cron mirror. The mirror file is home-mixer/params/param.rs. The earlier conclusion, "weight values have no primary source in any generation," was overturned by the Gen4 release. This section is the most important one in this document.

The new conclusion is that a mirror that syncs the production primary values daily exists, and the last sync time is stated in the file header.

The file header defines the nature of this mirror verified — home-mixer/params/param.rs:1:

// mirrored from config feature-switch defaults; last sync 2026-08-12T04:09:22Z

The README states what mirroring means verified — README.md #experiments-and-configuration:

"To help people understand the production defaults, we run cron scripts that set the defaults in this repository's code to be the primary production values"

The engagement head weights are in the table below (in file order, 2026-08-12 sync) verified.

• The later values in the click, vqv, dwell, and cont_click_dwell_time cells were confirmed in home-mixer/params/param.rs and vm-ranker/params.rs verified. vqv and dwell first changed in param.rs:342, 356 of 0d3cdd8, and click and cont_click_dwell_time in param.rs of a707cc2.

The mutual-follow boost was introduced in 2026-07 and adjusted on 2026-07-24 verified — docs/BIDIRECTIONAL_BOOST_CHANGE.md.

The reply boost value was settled in three stages verified — same file.

• 2026-07-10: Started an A/B test at 5/10/15/20 verified — same file.
• 07-13: Rolled out 20 verified — same file.
• 07-24: Lowered from 20 to 15. The adjustment reflected user feedback during the World Cup season verified — same file.

Whether a post is a boost target is decided by the function below verified — home-mixer/scorers/ranking_scorer.rs:

fn bidirectional_boost_eligible(candidate: &PostCandidate) -> bool {
    candidate.in_reply_to_tweet_id.is_none()
        && candidate.retweeted_tweet_id.is_none()
        && candidate.is_mutual_follow_author == Some(true)
}

In other words, only for original posts does the reply weight go from 5.0 to 20.0. Replies and retweets are not boost targets.

The negative head weights are in the table below verified.

The file's own comment defines the meaning of the weight signs and magnitudes verified:

"These weights reflect a combination of how much an action is valued in ranking and typical propensities of these actions across the X network (e.g. negative feedback is overall rare)."

In other words, the magnitude of report −234 is not "the relative importance of one report" but the product of value × 1/propensity. Here propensity means how often that action typically occurs across the whole X network. Because reports are so rare, a large absolute value is attached.

The 2026-08-14 commit (Gen4+1) stated this interpretation rule explicitly in a code comment and the README verified — home-mixer/params/param.rs:281-306, home-mixer/scorers/ranking_scorer.rs:415-446, README.md #scoring-and-ranking:

"Each weight multiplies the predicted probability of that action (P(favorite), P(repost), …) or a continuous value e.g. watch time -- the weights do not multiply raw engagement counts. One common misinterpretation is that you can read these weight ratios as count equivalences, e.g. the incorrect statement that 'one report cancels 468 likes' -- this is incorrect because the weights apply to the predicted probabilities rather than raw counts. And the baseline probability of a Report is more than 1000x lower than a Like, so it's weighted more to allow the prediction to affect the final ranking at all." — home-mixer/params/param.rs:283-292

The same comment also rejects the misconception that "mass reporting uniformly suppresses reach," for two reasons verified — same comment block.

1. The score multiplies your predicted probability, so reports from bot accounts mainly affect recommendations for users similar to those bots (personalization) verified — same comment.
2. Only actions taken on posts served in the Home Timeline count as ranking signals. Reactions that appear from a link shared in a group chat do not affect ranking verified — same comment.

Five caveats of the weights mirror

There are five things to watch when reading this mirror.

1. The cron sync is a mirror that follows once a day, so it lags. Production changes after the sync time in the header may not be reflected yet. verified
2. Sign and magnitude must be read with propensity in mind. The file's own comment states "value = worth × 1/propensity". The ratio of absolute values (report −234 is 468× favorite 0.5) is not the relative importance as is.
3. Since the 2026-09-23 commit (Gen4+27), the weights mirror has been split into two files, and as of the 2026-10-17 commit the weight values differ between the two files.
   • Gen4+27 introduced vm-ranker/params.rs, which holds 56 parameters, so the mirror became two files together with the existing home-mixer/params/param.rs for the home-mixer process verified.
   • The next day (Gen4+28, 2026-09-24), 22 non-weight ranking parameters were deleted from the home-mixer param.rs verified — comparison of param.rs in 1b3fec20 and 44d37eb.
     • The deleted parameters include the author diversity, OON, new-user, weight perturbation, VMRanker DPP, and value model switch families; the only weight is ContActiveSecs5mResidualNormWeight.
   • Per-action weights remain in both files, and as of the 2026-10-17 commit (862a123), five values differ: favorite, video_open, cont_dwell_time, post_unexplored, and not_interested verified — home-mixer/params/param.rs, vm-ranker/params.rs.
   • The shared crate xai-value-model was released as an actual file, so both processes compute the value model with the same code verified. However, each process reads the weights from its own parameter file verified — home-mixer/scorers/value_model.rs:8-51, vm-ranker/scoring/value_model.rs:68.
   • The phoenix training hyperparameters and the botmaker rule constants are still outside the repo. crate::util, which the Gen3 canon treated as outside the repo, has been released as home-mixer/util/ since the Gen4 release verified — home-mixer/lib.rs.
4. The Phoenix training code is "trainable", not "the same as production". The README states the difference in scale: "a small model can be trained and served end-to-end".
5. Some botmaker rules are not published to prevent gaming. The README states this explicitly. For some of the labels that visibility-filtering references, the rules are not open.

Practice of documenting parameter changes — introduced in Gen4

The practice of documenting parameter changes as code diffs began in Gen4. docs/BIDIRECTIONAL_BOOST_CHANGE.md is a model for how parameter changes will be documented as code diffs going forward verified:

"Going forward, you'll see future algorithm updates published to this open source repository, and you should be able to understand what changed by checking the diffs."

The same document reproduces the 3-stage rollout of the mutual-follow boost (A/B start → broad rollout → parameter adjustment) with 3 code diffs. No other company (Meta, Netflix, or YouTube) has had an example of this documentation approach so far.

The value of tracking the X commit log daily in this repository grows considerably. What changes in which param in the next commit becomes a lagging indicator of A/B results.

Generation comparison — why the earlier canon stated "values undisclosed"

In Gen1, Gen2, and Gen3 the values were indeed not released, and Gen4 newly opened them.

• Gen1 (2023 Scala): object ModelWeights at ScoredTweetsParam.scala:151 has 17 parameters. The defaults are 1.0 for 10 and 0.0 for 7, and they are not actual production values. Range bounds (e.g., ReportParam −20000..0) are valid only as relative-scale hints.
• Gen2 (2025 Scala): Moved to HomeGlobalParams.scala:786, and the parameters grew from 17 to 30. All 30 were intentionally removed, with default = 0.0. Only relative-scale hints remain.
• Gen3 (2026-01 Rust): The params:: module itself was dropped from the repo. Symbols such as FavoriteWeight had only call sites and no definition.
• Gen4 (2026-08 Rust): ⚠ Released the real production mirror as home-mixer/params/param.rs.

The widely quoted "reply 13.5" table is still exactly the 2023-04-05 snapshot. Only favorite 0.5 and retweet 1.0 match the Gen4 mirror, and the rest was overhauled substantially over three years. verified — twitter/the-algorithm-ml projects/home/recap/README.md

• The values in that table are as follows:
  • Positive items: fav 0.5 / retweet 1.0 / reply 13.5 / good_profile_click 12.0 / video_playback50 0.005 / reply_engaged_by_author 75.0 / good_click 11.0 / good_click_v2 10.0
  • Negative items: negative_feedback_v2 −74.0 / report −369.0

Some constants are restored across generations. Gen3 offset_score is a literal translation of Gen2 Scala RerankerUtil.weightedScoresSum. The Gen2 counterpart of NEGATIVE_SCORES_OFFSET, which was hidden in crate::params in Gen3, is val Epsilon = 0.001. Whether NEGATIVE_SCORES_OFFSET was also exposed in param.rs in Gen4 needs a grep. Details are in multi-task-ranking.md.

In the Gen2 code, X can inject multiplicative Beta-distribution noise seeded per user×date into the ranking weights verified — HeavyRankerWeightsQueryFeatureHydrator.scala, seed Objects.hash(userId, startOfDay). Whether this noise injection was kept in Gen4 needs a deep dive.

Legacy heavy ranker (2023–2025)

The legacy heavy ranker is a structure in which a multi-task neural network predicts engagement probabilities and sorts by their weighted sum. Compared with the 2026 Phoenix, the objective has the same form and the input differs. The legacy ranker uses handcrafted features as input, and Phoenix uses ID embeddings.

In the 2025 generation, two rankers run side by side verified — ScoredTweetsRerankingScoringPipelineConfig.scala:35-39.

• Navi is the production heavy ranker and has 15 heads verified — home-mixer/.../model/PredictedScoreFeature.scala:294-311.
• Phoenix is a Transformer rescorer (a model that rescores the scores assigned by the preceding ranker) and has 13 heads. It is attached after Navi through ParamGatedScorer(EnablePhoenixScorerParam, …) and shares the same ModelWeights as Navi verified — PhoenixPredictedScoreFeature.scala:174-189.

The Transformer ranker did not replace the existing heavy ranker in one step. It first joined alongside as a rescorer switched on by a param, and then fully replaced the heavy ranker in 2026. verified — the evidence is that all 8 files were newly created in c54bec0 (git log --diff-filter=A) and the pipeline wiring above.

The 2025 Phoenix call specification is below verified — home-mixer/.../util/PhoenixUtils.scala.

• Candidates are cut to 1400 (MaxCandidates = 1400 :36). There is also a config value TopLogProbsNum = 50 (:35).
• Retries are 2, 500ms per attempt, 1 second in total (:119-121). The channel is picked at random again on every attempt (:108-109).
• Because the response is a logprob (log probability), the probability is restored with math.exp(logProb) (:151-158).
• Only 5 booleans travel with a candidate: isRetweet, isForYouPage (always true), isPromotedTweet (always false), isReply, isQuote :48-68.

The score combination formula and the full per-generation head lists are in multi-task-ranking.md. This formula does not multiply the negative heads but handles them with a threshold constant penalty, and finally compresses with Epsilon = 0.001.

There is a caveat in the 2025 code. There are two files named PredictedScoreFeature, with different head sets. The 15 in model/ are actually used, and the 17 in product/scored_tweets/scorer/ are legacy. Writing the head list from the latter is wrong. verified — import path in RerankerUtil.scala:4

Grok does not contribute arithmetically to the 2025 ranking score. It enters only through filters (4 kinds, including GrokGoreFilter) and topic tags. No Grok head appears anywhere in PredictedScoreFeatures. verified — repo-wide, ScoredTweetsRecommendationPipelineConfig.scala:543-546

There is no ranking boost for verified authors. All 10 files that reference AuthorIsBlueVerifiedFeature and similar are hydration, logging, caching, or marshalling. However, it does enter as an input feature of the trained model (GizmoduckFeaturesAdapter.scala:12-17). There is no rule-based bonus; the model uses it on its own. verified — repo-wide grep

↑ Contents

Re-ranking, diversity, and filters

As of the 2026-10-17 commit (862a123), X For You has 25 filters: 20 run before scoring and 5 run after candidate selection verified (filters section). As of the Gen4 release, diversity has three layers: author-diversity decay, VMRanker's DPP slate reordering, and conversation dedup verified (diversity section, Phoenix section). Since the 2026-10-14 commit (Gen4+41), VMRanker skips DPP diversity re-ranking by default for topic requests verified (Gen4+41 in the change log). As of the 2026-10-08 commit (Gen4+38), the final arrangement is not in score order: the blender inserts ads, prompts, follow recommendations, and the video carousel (off by default) in turn and pins push-to-home posts at the front verified (final arrangement section).

21 filters (2026-08, Gen4+1) — 18 before scoring + 3 after

As of the 2026-10-17 commit (862a123), 20 filters run before scoring, in the order below verified — phoenix_candidate_pipeline.rs:419-451. As of the 2026-08-14 commit (Gen4+1), there were 18. The evidence is README.md #filtering and phoenix_candidate_pipeline.rs.

• DropDuplicatesFilter
• ZeroFavCodivertFilter (new in Gen4+44, 2026-10-17)
• CoreDataHydrationFilter
• AgeFilter
• SelfTweetFilter
• OONRetweetReplyFilter (new in Gen4)
• OONNsfwSimclustersFilter (new in Gen4)
• RetweetDeduplicationFilter
• IneligibleSubscriptionFilter
• PreviouslySeenPostsFilter
• PreviouslySeenPostsBackupFilter
• PreviouslyServedPostsFilter
• ViewerMutedKeywordFilter (replaced MutedKeywordFilter in the 2026-08-19 commit (Gen4+4))
• AuthorSocialgraphFilter
• Brazil2026ElectionFilter (new in Gen4+1, 2026-08-14)
• VideoFilter
• TopicIdsFilter
• NewUserMinEngagementFilter (new in Gen4, replaced NewUserTopicIdsFilter)
• InventoryHoldoutFilter (new in Gen4)
• FavHoldoutFilter (new in Gen4+23, 2026-09-17)

As of the 2026-10-17 commit (862a123), the filters that run after selection are 5: VFFilter, AncillaryVFFilter, AncestorVideoFilter, DedupConversationFilter, and VideoCarouselFilter verified — phoenix_candidate_pipeline.rs:518-524.

• Up to the 2026-08-14 commit (Gen4+1), there were 3, VFFilter, AncillaryVFFilter, and DedupConversationFilter, the same as Gen3 verified.
• VideoCarouselFilter was added in the 2026-10-08 commit (Gen4+38) and AncestorVideoFilter in the 2026-10-13 commit (Gen4+40) verified — phoenix_candidate_pipeline.rs in 35650fb and e1a0375.

The first 4 of the bullets below are the filters newly introduced in the Gen4 release.

• OONRetweetReplyFilter filters retweets and replies from accounts the user does not follow, as well as replies without a parent. It is separate from RetweetDeduplicationFilter.
• OONNsfwSimclustersFilter was introduced together with the SimClusters revival. It filters posts from adult-content accounts that came from the SimClusters source out of the feeds of users who do not follow those accounts.
• NewUserMinEngagementFilter removes OON posts to be shown to new users that fall below the engagement threshold. It has 6 related params.
  • NewUserMinEngagementThreshold, NewUserMinEngagementFilterMetric, NewUserMinEngagementFilterUseRatio, NewUserMinEngagementFilterMaxAccountAgeSecs, NewUserMinEngagementFilterMaxResurrectionAgeSecs, EnableNewUserMinEngagementFilter
• InventoryHoldoutFilter holds out inventory at the 3-tier ratios through deterministic user×post hashing, and its params were confirmed in the 2026-09-17 commit (Gen4+23, 42266f3) verified — home-mixer/filters/inventory_holdout_filter.rs
  • The hashing goes through splitmix64-family double mixing and is split into 100-modular buckets.
  • There are 3 params, InventoryHoldoutOriginalsPercent, InventoryHoldoutRepliesPercent, and InventoryHoldoutRetweetsPercent, each clamped with .min(100).
  • The post type is determined by retweeted_tweet_id.is_some() → Retweet and in_reply_to_tweet_id.is_some() → Reply, and anything else is treated as Original. A quoted post is treated as Original.
  • The gate opens when EnableInventoryHoldout=true and at least one of the three values is greater than 0.
  • A retweet is decided against get_original_tweet_id() (the original), so it shares the holdout decision with the original.
• FavHoldoutFilter was introduced in the 2026-09-17 commit (Gen4+23, 42266f3) and holds out by fav count tier. It is off by default with param EnableFavHoldout=false verified — home-mixer/filters/fav_holdout_filter.rs
  • The tier table is the following 23-tier one:
    • 10K+ fav = 15%, 5K+ = 15%, 3K+ = 15%, 2K+ = 14%, 1.5K+ = 13%, 1K+ = 13%, 750+ = 12%, 500+ = 11%, 300+ = 10%, 200+ = 9%, 150+ = 8%, 100+ = 7%, 75+ = 7%, 50+ = 6%, 30+ = 6%, 20+ = 5%, 15+ = 5%, 10+ = 5%, 7+ = 4%, 5+ = 4%, 3+ = 3%, 2+ = 3%, 1+ = 2%, ≤1 = 0%
  • The holdout decision reuses InventoryHoldoutFilter::is_held_out(post_id, viewer_id, percent) (the same splitmix64 hash), so it is deterministic per viewer.

Gen4+1 (2026-08-14, c65aa17) added one new filter.

• Brazil2026ElectionFilter filters out posts tied to accounts that Brazil's Electoral Court (TSE) acted on in connection with the Brazil 2026 general election, unless the viewer follows the account verified — home-mixer/filters/brazil_2026_election_filter.rs
  • The account list is a FxHashSet<u64> with 665 accounts hardcoded.
  • The fields checked on a candidate are author_id, retweeted_user_id, quoted_user_id, and ancestor_users, and if any one of them is in the set, the candidate is targeted.
  • In the pipeline it sits right after AuthorSocialgraphFilter and right before VideoFilter. Another evidence location is phoenix_candidate_pipeline.rs:356-368.
  • The source materials are the Brazilian Electoral Court open dataset and X's announcement (@XBR, 2026-08-14). The README states: "in accordance with Brazilian electoral law, For You now runs Brazil2026ElectionFilter, which removes posts from accounts reported to Brazil's Electoral Court for the 2026 election, unless the viewer explicitly follows the account."
  • In the 2026-10-14 commit 17c96bf (Gen4+41), the list was cut to 25 accounts tied to candidates in the October 25 runoff (the relevant diff range went from 5,654 lines to 63 lines) verified. It cites TSE official letter numbers in comments as evidence verified — brazil_2026_election_filter.rs:17-74
  • The code leaves @username as a comment next to each account ID ("obfuscated ID + username alongside for transparency"). Examples are 40053694 // @FlavioBolsonaro, 21069302 // @marcelvanhattem, and 48298703 // @eduardopaes.
  • The follow exception is based on the user's personal following graph, so it is symmetric regardless of political spectrum. The decision formula is is_excluded_author(user_id, followed_user_ids) = LIST.contains(user_id) && !followed_user_ids.contains(user_id).
  • The list size is the code constant 665. The code has this unusual comment: "OmarAzizSenador deleted his account at the time this code was written."
  • This filter is the first case in which a jurisdiction filter is observed directly in code. Until now, responses to regulation such as DSA, GDPR, and NetzDG were seen mostly only in regulatory filings (transparency reports). If this code-release practice continues, similar filters can be confirmed in code for each country's election cycle.

The notable points among the filters are as follows:

• AuthorSocialgraphFilter checks 6 conditions verified — filters/author_socialgraph_filter.rs:8-61
  • mute, block, author blocks me, quoted author blocks me, I block the quoted author, I block the retweet's original author
• VFFilter is still fail-closed, but since Gen4+18 (2026-09-09, 75d93d9) the basis of the decision moved from matching FilteredReason variants to an explicit Action enum verified — home-mixer/filters/vf_filter.rs:16-21
  • should_drop_action classifies Drop(_), Tombstone, and NotEvaluated as drop, and Allow, Interstitial, Avoid, and Downrank as keep.
  • The upstream StratoVfClient converts non-SafetyResult reasons (mute, block, null-cast, exclusive, etc.) to Drop at the client layer, and converts NotEvaluated inside a SafetyResult to Allow verified — visibility-filtering-client/vf_client.rs:178-187. So on this path, a missing verdict is not dropped.
  • Conversely, the Rust VF client fills posts missing from the response with NotEvaluated, and VFFilter drops them verified — vf_client.rs:360-378. Fail-closed applies to missing results on this path.
  • As of the 2026-10-17 commit (862a123), the two handlings and should_drop_action are the same verified — vf_client.rs, vf_filter.rs.
• In-network and OON get different safety levels (TimelineHome vs TimelineHomeRecommendations). Ancestor posts and quoted posts always get the stricter one verified — vf_candidate_hydrator.rs:62-67, 76-90
• TopicIdsFilter is a 572-line file containing about 90 topic constants and a hand-built supertopic hierarchy. This part sits squarely against "remove all handcrafted features". verified — filters/topic_ids_filter.rs:109-515
• PreviouslySeenPostsBackupFilter is permanently a no-op. The ImpressedPostsQueryHydrator that would fill its input is only constructed, bound to _, and not registered in the pipeline. It runs on every request but filters nothing verified — phoenix_candidate_pipeline.rs:234-236

Gen4+4 (2026-08-19, aad7179) added a mute keyword filter dedicated to the Following pipeline and the quoted-post text hydrator that supports it verified — home-mixer/filters/following_viewer_muted_keyword_filter.rs (65 new lines), home-mixer/candidate_hydrators/quoted_post_text_hydrator.rs (106 new lines). This commit continues the flow in which the Following pipeline branched to VFFollowingCandidateHydrator in the 2026-08-18 commit (Gen4+3).

• FollowingViewerMutedKeywordFilter was split into a separate file from the existing MutedKeywordFilter (for the For You pipeline).
  • It tokenizes muted_keywords with TweetTokenizer and builds the matcher with MatchTweetGroup.
  • It filters a candidate if any one of three text sources matches: the candidate's tweet_text, quoted_tweet_text, or ancestor_texts.values().
  • This matching is CPU-bound work wrapped in tokio::task::block_in_place.
  • A peculiarity is the possibility that the existing For You filter looked only at tweet_text. The evidence is that the 2026-08-18 commit (Gen4+3) had no quoted-text wiring.
  • The mute targets have now widened to quoted posts and ancestors (reply ancestors). This extension narrows the route of bypassing a mute through quotes or replies.
• QuotedPostTextHydrator is the hydrator that prepares this filter's input. For the filter to match quoted text, this field must be filled first.
  • For candidates with quoted_tweet_id, it does a bulk lookup with TESClient.get_tweet_core_datas and then fills the quoted_tweet_text field.
  • It removes duplicates with a HashSet and calls gRPC in batches.
• Why it matters: Now For You and Following use different components in the filter and hydrator stack. Paired with the VF hydrator branch in the 2026-08-18 commit (Gen4+3), the Following pipeline is being rebuilt as a parallel structure rather than a subset of the For You pipeline inferred

Gen4+5 (2026-08-20, d0cef2f) added an "AI trend" feedback context channel to For You.

A second context channel opened alongside the existing topic feedback context (a context label for inferring a user's topics of interest) verified — home-mixer/candidate_hydrators/ai_trend_feedback_context_hydrator.rs (157 new lines). In the pipeline it is wired in right after TopicFeedbackContextHydrator, side by side with it verified — home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs:424-428. Another evidence location is home-mixer/util/urt/post_marshaller.rs:72-110.

• The gate is 3-part. The switch EnableAiTrendFeedbackContext=false is off by default, and the other two conditions are !query.is_topic_request() and !query.in_network_only. That is, it does not overlap with topic surfaces or Following surfaces and targets only For You OON candidates.
• Candidates are filtered by a 4-part standalone original post condition. In other words, the AI trend label is attached only to original posts, not to replies, retweets, posts with conversation ancestors, or targets that a followed account replied to.
  • The condition expression is in_reply_to_tweet_id.is_none() && retweeted_tweet_id.is_none() && ancestors.is_empty() && following_replied_user_ids.is_empty().
• At most 2 AI trend labels are attached per page, and each trend is attached to only one candidate. This limit is a suppression device to avoid flooding the same label across multiple candidates of the same trend.
  • The selection approach is the top-2 trends and one random candidate per trend.
  • The trend_ids of the candidates are looked up with Strato batch_get_tweet_ai_trend, and the top 2 trends are picked from the frequency histogram (TOP_TREND_COUNT = 2). On a tie, the smaller trend_id takes priority.
  • For each trend, one candidate labeled with that trend is picked at random, and the display name is looked up with Strato batch_get_ai_trend_name.
• At render time, topic feedback takes priority. In post_marshaller, topic wins if topic feedback exists, and AI trend enters as a fallback only when it does not.
  • Both go out in URT as type ContextType::TOPIC.
  • AI trend fills landing_url = "https://x.com/i/trending/{trend_id}" and TopicFeedbackContext.topic_id is None.
  • That is, tapping the chip goes to the trending surface rather than the topic feedback page.
• Why it matters: This channel widens the context and signal sources upstream of ranking. The user's reaction of tapping, dismissing, or ignoring this chip becomes a label source for "does this user care about this AI trend".
• Undisclosed: The following are not public:
  • The definition of AI trend (tweet clustering with Grok? human curation?)
  • The location of the batch_get_tweet_ai_trend upstream pipeline
  • Whether EnableAiTrendFeedbackContext is actually deployed

Diversity

X For You has three layers of diversity mechanisms.

1. Author-diversity decay was applied inside the ranking scorer at the Gen4 release; since the 2026-09-24 commit (Gen4+28), the vm-ranker service computes it with the shared crate xai-value-model verified (final score calculation order in the Phoenix section).
2. VMRanker DPP optimizes candidates as a set, and the service itself has been in vm-ranker/ since the Gen4 release verified (VMRanker paragraph in the Phoenix section).
   • Gen4+20 (6bb4594, 2026-09-12) added an optional seed_tweet_id argument.
   • If an embedding exists for this id, the seed is pinned to the first slot of the greedy DPP and filtered out of the final result, so it does not consume the top_k budget while only candidates similar to the seed receive a penalty.
   • In other words, the diversity axis can now extend from within the set to relative to a reference tweet.
   • Which production surfaces carry this field is not yet public.
3. Conversation dedup keeps only the top-scoring one per thread verified — filters/dedup_conversation_filter.rs:6-48
   • The id that groups a conversation is conversation_id = min(ancestors) or the tweet itself.

The feed goes out without DPP even if the VMRanker call fails. Since the 2026-09-24 commit (Gen4+28), if gRPC fails it emits the local weighted-sum score and counts the failure with the VMRanker.local_score_fallback metric verified — vm_ranker.rs:107-113.

• Before that (Gen3 to Gen4+27), it returned Err for every candidate, and the pipeline's update_all skipped the Err without logging. As a result, a score-ordered feed lacking DPP diversity went out with no indication verified — candidate-pipeline/scorer.rs (default update_all implementation), vm_ranker.rs:44-66

X made the mechanism that measures how many new topics the feed shows the viewer off by default. In the 2026-10-20 commit (Gen4+45), a path was added that counts the first and second SID levels of posts the viewer recently engaged with as topics and records, as a metric, the share of response posts whose topic is not in that history verified — side_effects/serendipity_stats_side_effect.rs.

• History topic computation is off because the switch HistoryTopicsEnabled is false by default; when it is on, the metric is measured against a 28-day history verified — params/param.rs:508-529.
• A switch that sends the same history topics in the VMRanker request was also added, off by default verified — same file. How VMRanker uses this value is not in the public code verified.
• There is only a metric and no code that uses it for re-ranking verified. Given that measurement was added first, we infer this is preparation to test history-based topic exploration as a diversity goal inferred. Details are in daily 2026-10-21.

The 2025 Scala generation worked differently from today. MMR gave a reward for distance and the SimClusters category penalty gave a penalty for the same thing, and the two sat side by side, param-gated. There were six dedup filters. A comparison of the four approaches (decay, MMR, DPP, dedup) and the full per-generation list are in feed-diversity.md.

The final placement is not score order

The blender that handles final placement intentionally disables the framework's sorting verified. The evidence is home-mixer/selectors/blender_selector.rs:25-114. In this file's BlenderSelector, score() returns the constant 0.0 verified. The blender then runs the following stages.

• Inserts prompts at positions 0..n verified — same file
• Inserts Who-to-Follow (WTF) at designated positions verified — same file
• Pins the push-to-home post always at position 0 verified — same file
• Runs the ads blender. As of the 2026-10-17 commit (862a123), AdsBlenderType picks one of partition_organic_low_risk, safe_gap, multi_risk, and time_gap, and the default has been multi_risk since the 2026-09-01 commit (Gen4+13) verified — blender_selector.rs:55-72, params/param.rs:817
• Since the 2026-10-08 commit (Gen4+38), inserts the video carousel module, off by default, at the VideoCarouselPosition (default 6th) slot. The execution order is ads blending → prompt → WTF → carousel → push-to-home pinning → frame → survey verified — home-mixer/selectors/blender_selector.rs:72-85, 137-153

↑ Contents

Cold start

Cold start is the question of how recommendations treat users, authors, and posts that have not yet accumulated history or impressions. As of the Gen4 release, the X For You code had mechanisms for new users and new authors, but none for individual new posts verified (sections New users, new-author boost, and New items). Since the 2026-09-30 commit (Gen4+32), the Phoenix retrieval cold slice, which collects new posts with few impressions and likes separately, has been on, and the cold-start slot selector picks only posts with fewer than 200 home impressions verified (section New items). The new-author boost on in-network posts of new authors has been 0.015 in the home-mixer mirror and 0.02 in the VM-Ranker mirror since the 2026-10-07 commit (Gen4+37) verified (change log Gen4+37); Thompson sampling in the cold-start slot selector and the cold-start slice of Phoenix retrieval came in off by default and were turned on together in the commit that tuned the home-mixer cold-start parameters to production values verified.

New users (account level)

New users whose history length is below a threshold are sent to a separate model fleet for both ranking and retrieval verified. The evidence is phoenix_scorer.rs:29-43 and phoenix_source.rs:25-39. The threshold is Phoenix*NewUserHistoryThreshold, and the fleet these users go to is Phoenix*NewUserInferenceClusterId.

• In the 2026-09-18 commit (Gen4+25), the Phoenix retrieval cold-start slice was wired in as first-class verified — home-mixer/sources/phoenix_source.rs.
  • This slice is distinguished by the dataset_type axis carried in the Phoenix retrieval response.
  • Candidates tagged HOME_COLD_DATASET_TYPE=13 have their served_type mapped to ForYouPhoenixRetrievalCold.
  • record_phoenix_retrieval_cold_stats emits the PhoenixRetrievalColdTweets.* stat block.
  • A separate cap param for this slice, PhoenixColdStartMaxResults, was introduced at 0 (off by default).
  • Unlike the routing above, which splits model fleets by the user's history threshold, this slice splits the candidate set returned by the same Phoenix retrieval RPC using the dataset_type tag.
  • When introduced, it was off by default, so it had no production share.
  • This cap was later turned on at 200 in the commit that tuned the home-mixer cold-start parameters to production values verified — params/param.rs:16.
• As of the Gen4 release, the new-user routing threshold is NewUserAgeThresholdSecs = 0 verified. Because it is 0 seconds, we infer the new-user path is mirrored as effectively disabled inferred.
  • This param was dropped from the home-mixer param.rs in the 2026-09-24 commit (Gen4+28), and it is also 0 in vm-ranker/params.rs as of the 2026-10-17 commit (862a123) verified.
• NewUserMinEngagementFilter, the OON engagement-threshold filter new in Gen4, removes OON posts that fall short of the engagement threshold.
• NEW_USER_OON_WEIGHT_FACTOR, NewUserTopicIdsFilter, and EnableNewUserTopicRetrieval, which existed in Gen3, are not in the Gen4 param.rs. A deep dive is needed.

New authors (account level)

New authors are identified by impression count, and their in-network posts get the new-author boost verified (new-author boost section of ranking). The boost and threshold params were documented in Gen4, and the full set of boost params is in the table in that section.

• The new-author boost has been 0.015 in the home-mixer mirror and 0.02 in the VM-Ranker mirror since the 2026-10-07 commit (Gen4+37). It applies only to in-network posts.
  • The value documented in Gen4 is PostUnexploredWeight = 0.02.
• The Gen4 README describes authors whose impressions are below a threshold as "new authors", and the threshold param at that time was ColdStartImpressionThreshold = 1000. The code applies this threshold to the home impression count of the candidate post, not to the author verified (new-author boost section).
  • This threshold was later changed to 200 in the commit that tuned the home-mixer cold-start parameters to production values verified.

Cold-start slot selector

The cold-start slot selector chooses "which candidate fills a slot" within the set of cold-start target candidates. In the 2026-08-14 commit (Gen4+1), this selection approach was split into two branches, greedy max-score and Beta-reward Thompson sampling verified — home-mixer/scorers/author_cold_start.rs:151-190. Thompson sampling was off by default at first, and it has been on by default since the commit that tuned the home-mixer cold-start parameters to production values verified.

The param definitions and the defaults as of Gen4+1 are in param.rs verified — home-mixer/params/param.rs:687-724.

EnableColdStartThompsonSampling is a switch added for A/B. Its default in the 2026-08-14 commit (Gen4+1) is false, and the default is still greedy (current behavior unchanged). That it is off by default matters. At this release point (2026-08-14), the cold start shown to real users is still max-score greedy, and for Thompson only the A/B infrastructure is wired up.

• Thompson sampling draws one sample per candidate from the posterior, picks the top K, and then picks the candidate with the highest original score among them.
  • For each cold-start candidate i, set n_i = scale × view_count_i and x_i = min(fav_count_i, n_i).
  • The posterior is Beta(α₀ + x_i, β₀ + max(n_i − x_i, 0)), and one sample is drawn from it.
  • In other words, it does Bayesian exploration (uncertainty about which candidate will be good), but the final choice is decided by tie-break on the original score.
• α₀=0.75, β₀=49.25 set a prior of "roughly 0.75 likes per 50 impressions". The values are meant to let new authors whose measured like rate is higher than this be picked generously.
• As of the 2026-08-14 commit (Gen4+1), the instrumentation hook record_tracked_ids is an individual observation hook that lets A/B manually watch specific accounts.
  • This hook counts how many times a tweet_id/author_id registered in ColdStartTrackedIds appeared from which source of the cold-start pipeline.
  • The counter is home_mixer.cold_start_tracked_ids_total{id, source}.
• In the 2026-08-18 commit (Gen4+3), the aggregation key of this instrumentation was simplified verified — home-mixer/scorers/author_cold_start.rs:92-108.
  • The changed function is count_tracked_ids.
  • Previously, tweet_id and author_id were aggregated as separate entries.
  • Afterward, a candidate whose tweet_id or author_id is in the tracked list is counted under its single tweet_id. The separate author_id entry is gone.
  • The tracked-list filter remains, so if the list is empty nothing is counted. The function in the 2026-10-17 commit (862a123) is the same verified — author_cold_start.rs:151-185.
• In the 2026-09-04 commit (Gen4+16), the caller of record_tracked_ids and the param wiring were confirmed verified — home-mixer/scorers/author_cold_start.rs. This tracked ID observation hook is an always-on observation channel independent of A/B.
  • ColdStartParams::read(query) reads all params at once and puts them in params.tracked_ids.
  • record_tracked_ids(candidates, &params.tracked_ids) is always called regardless of EnableViewerColdStart gating.
• The same 2026-09-04 commit (Gen4+16) pinned the impression symmetry of the codivert experiment with a regression test verified — same file.
  • A pair of YAML rules was added so that the two arms of the new experiment moe_codivert_viewer split symmetrically as enable_viewer_cold_start_boost=true/false.
  • The experiment params are PhoenixMoeCodivertViewerIsControl and PhoenixMoeCodivertViewerIsTreatment.
  • apply() changed its execution order to call viewer_arm(query) and author_corpus(...) before the enabled gating.
  • Each of the two arms leaves exactly 3 moe_codivert_viewer bucket impressions and 2 moe_exp bucket impressions, which target authors. The tests both_arms_log_the_same_codivert_impressions and both_viewer_arms_log_the_same_author_impressions pin this.
  • Triggered-analysis compares only users who entered the pipeline, while intent-to-treat compares all assigned users. A methodology that guards against the mismatch between the two definitions with a regression test came in as code.
• In the same 2026-09-04 commit (Gen4+16), the candidate filter of the Treatment arm was narrowed to apply only to Phoenix MoE candidates verified — same file.
  • The condition && is_phoenix_moe(c) was added to cold_start_corpus_eligible(Treatment).
  • As of this commit, the cold-start candidates of a Treatment viewer were only those that are posts by treatment corpus authors and have served_type == ForYouPhoenixRetrievalMoe.
  • The test name also changed to treatment_viewer_cold_starts_treatment_moe_only.
  • Cold start and Phoenix MoE are bundled into one codivert experiment and isolated so that the two effects are not confounded.
  • This MoE condition was removed from the Treatment branch in the 2026-09-17 commit (Gen4+23), and cold_start_corpus_eligible in the 2026-10-17 commit (862a123) has no MoE condition verified — author_cold_start.rs in 42266f3 and 862a123.
• In the same 2026-09-04 commit (Gen4+16), engagement counts hydration was separated from cold-start gating verified — home-mixer/candidate_hydrators/engagement_counts_hydrator.rs.
  • enable() drops the dependency on EnableViewerColdStart and decides whether to turn on solely by EnableEngagementCountsHydration || is_shadow_traffic.
  • It also removed the !query.has_cached_posts gating of enable(). The has_cached_posts branch on the side that decides per candidate whether to look up remained until the 2026-10-01 commit (Gen4+33) and was removed in the 2026-10-02 commit (Gen4+34) verified — engagement_counts_hydrator.rs:123, same file at 76843a5.
  • The tight coupling of the initial Gen4+1 (2026-08-14) wiring, where cold-start Thompson sampling depended on view_count_on_home and fav_count, is loosened.

New items (post level)

As of the Gen4 release, there was no separate mechanism that boosts a new post itself regardless of its author verified. As of the 2026-10-17 commit (862a123), two mechanisms work on post-level conditions.

• In the 2026-09-24 commit (Gen4+28), HOME_COLD (13) and HOME_HOT (14) datasets were added to the Phoenix retrieval training data. HOME_COLD keeps only recent posts with fewer than 500 views and fewer than 8 likes verified — retrieval_dataset.py:280-281, cold_pool_filter.py:26-46.
  • The home-mixer has separated retrieval candidates carrying this tag (13) as a cold slice since the 2026-09-18 commit (Gen4+25), and has accepted up to 200 since the 2026-09-30 commit (Gen4+32) verified (current candidate generation).
• The cold-start slot selector picks one of the original posts whose author has 50,000 followers or fewer, that was posted within the last 2 hours, and that has fewer than 200 home impressions, and raises its ranking verified — author_cold_start.rs:296-309, params/param.rs:507-531.
  • This selector has used the candidate post's impression count as a condition since the Gen4 release, and the threshold then was 1000 (new-author boost section).

The hash embedding of a new post is the value the continually trained table currently holds, and for an ID seen for the first time it is the average of other posts mixed in by hash collisions inferred.

The 2025 Scala code had SimclusterColdPostsCandidateSource.scala. Because SimClusters was revived in Gen4, whether this source is alive in Gen4 simclusters/ needs a deep dive (open questions).

↑ Contents

Safety / visibility-filtering / labeling pipeline (released in full in Gen4)

The Gen4 release (2026-08-13) published the whole path by which X creates safety labels and uses them to decide post visibility. According to the Gen4 README, this path has four stages: content understanding, labeling rules, storage, and visibility filtering (VF) verified (4-stage Labeling Path). As of the 2026-10-15 commit, VF uses five SafetyLevels verified (2026-10-15 commit in the change log).

Through Gen3, VFFilter and AncillaryVFFilter were applied fail-closed in home-mixer/. However, who attaches the labels those filters consult, and when and how, was a complete black box. The Gen4 release added 15 new visibility-filtering and labeling sub-repos verified (released source Gen4).

4-stage Labeling Path verified — README.md #labeling-path

Stages 1–3 create and store labels, and stage 4 makes decisions with those labels on the request path verified.

Content Understanding → Labeling Rules → Storage → Visibility Filtering

Stage 1 (Content Understanding) is a nearline stage that runs outside the request path. Stage 1 of the README diagram lists six: grox, media-model-proxy, clip, agatha, bdsm, and user-cred-v2 verified. The related sub-repos are in the table below.

Stage 2 (Labeling Rules) turns events and scores into labels verified.

• scarecrow/ is a rule runner that runs in response to events, and it embeds botmaker/ as the rule engine verified.
• botmaker/ is the rule DSL, compiler, and runtime verified.
• botmaker-rules/ holds the actual rules that scarecrow loads verified. According to the README, some were not released to prevent gaming verified.
• abuse-enforcement-service/ takes account actions (labeling, challenge, suspension) based on model scores verified.
• safety-label-user-agg/ aggregates labels of posts owned by an account to the account level verified.

In stage 3 (Storage), labels are written to storage and read on the request path.

Stage 4 (Visibility Filtering) runs on the request path verified — visibility-filtering/rules/registry.rs. The client library is in visibility-filtering-client/ verified.

As of the 2026-10-15 commit (92f0918), there are five SafetyLevels, and the number of rule constant entries per level is as follows verified — visibility-filtering/rules/levels.rs.

Since the 2026-09-22 commit (Gen4+26), the verdict has been a tree split into withhold and display, rather than the three-way Allow/Interstitial/Drop verified — visibility-filtering/models/verdict.rs (Gen4+26 row of the commit summary table).

• The withhold side is Drop and Tombstone.
• The display side has media and engagement slots, and a notice slot was added in the 2026-10-06 commit (Gen4+36) verified — verdict.rs (Gen4+36 in the change log).
• Before that, the 2026-09-09 commit (Gen4+18) changed the verdict axis from FilteredReason to a flat Action enum (Gen4+18 in the change log).

At the Gen4 release (2026-08-13), there were three SafetyLevel policies verified — Gen4 registry.rs.

• FilterAll drops everything verified.
• TimelineHome is the policy for following targets and consists of 28 base rules verified.
  • The 28 rules of base_home_rules() consist of 5 account-state rules, 3 viewer-relation rules, 8 tweet-label rules, 4 tweet-state rules, 3 NSFW viewer-gating rules, 1 excluded-tweet rule, and 4 NSFW interstitial rules verified.
• TimelineHomeRecommendations is the policy for recommendation targets and consists of 54 rules, the 28 base rules plus 26 OON-only rules verified.
  • The 26 OON-only rules consist of 2 DMCA/geo-restricted media rules, 4 NSFW account rules, 9 rules for tweets labeled NSFW/violence/DoNotAmplify/spam, and 11 NSFW/spam/impersonation/deactivated/adult user-label rules verified.

In the Gen4 registry.rs, evaluation ends at the first drop verdict (short-circuit) verified. The OON-only rule group can only drop and issues no interstitial or allow verdict verified. The verdicts are three: Allow, Interstitial(reason), and Drop(FilteredReason) verified. Interstitials are handed off to UI handling, and this repo has no code that draws interstitials verified.

Verifying canon caveat list #1 and #2 (partial)

Both items were partially verified, and each leaves one question that needs a deep dive.

• VFFilter fail-closed behavior: The Gen3 observation that VFFilter unconditionally drops every FilteredReason variant that has no verdict is still valid. How the visibility-filtering/ sub-repo handles a return without a verdict (error, timeout) needs a deep dive.
• Different safety levels for in-network and OON: The Gen4 Policies struct exposes the three levels FilterAll, TimelineHome and TimelineHomeRecommendations as they were. Whether the Gen3 rule of applying the stricter level to ancestor and quote posts (vf_candidate_hydrator.rs) still holds in Gen4 needs a deep dive.

Under the Hood transparency tool — introduced alongside Gen4 verified

Under the Hood is a transparency tool where users look up the visibility-impacting labels attached to their own account and posts verified. A daily batch collects the labels, and the serving layer aggregates them verified. X stated that the purpose is to let users audit "why was my post treated this way" by checking against the code in this repo verified.

• Address: x.com/i/under_the_hood
• Sub-repo: under-the-hood/

↑ Contents

Feedback signals

The X recommendation code uses positive actions such as like, reply and repost and negative actions such as not interested, block, mute and report as feedback signals, and production also counts not dwelling on a post as a negative signal verified. Phoenix's history encoding feeds an action not taken as −1 rather than 0, and masks history entries that have no action at all verified. Already-seen posts are handled in six layers, and the user-side features include demographics such as age and inferred gender verified. In the commit that introduced the vertical video carousel module as off by default, the Phoenix feature that feeds history dwell time as a bucket embedding became on by default verified (change log).

• Positive signals are actions such as like, reply, repost, quote, click and dwell verified — phoenix/runners.py:233-253
  • The full list is favorite, reply, repost, quote, click, profile_click, photo_expand, video quality view, share (including DM and link copy), dwell, dwell_time and follow_author.
• Negative signals are not interested, block author, mute author and report, and production counts even not_dwelled (did not dwell) as a negative signal verified — ranking_scorer.rs:83
  • The identifiers of the four actions are not_interested, block_author, mute_author and report.
• In the history encoding, no action is −1, not 0 verified — phoenix/recsys_model.py:404-432
  • The formula is actions_signed = 2*actions - 1.
  • That is, "did not like this post" enters as an explicit negative signal.
  • However, a history entry with no action at all is masked as a whole entry verified — same file.
• Already-seen posts are handled in six layers verified — each file in query_hydrators/ and side_effects/
  • The six layers are the seen_ids sent by the client, the impression bloom filter, the server-side served history (last N minutes/M items), the Redis candidate cache, the Phoenix request cache and the URT cursor.
• Impressions reported by the client are sent back out to Kafka verified — side_effects/publish_seen_ids_to_kafka_side_effect.rs
• User-side features include demographics such as age, inferred gender and IP location verified — home-mixer/models/query.rs:24-95
  • The fields included are user_age_in_years, user_inferred_gender, ip_location and subscription_level.
  • The topic and starter pack fields are followed_grok_topics[32], inferred_grok_topics[32] and followed_starter_packs[20].
  • user_inferred_gender has a dedicated GenderPredictionGrpcClient.
  • The hydrator-side evidence is phoenix_candidate_pipeline.rs:210-231.

↑ Contents

Ads

In the public code, X For You inserts ads between organic posts and decides ad slots by looking at the brand safety grade of the posts next to an ad verified. The default ads blender has been the multi-risk blender since the Gen4+13 commit (2026-09-01, 7ba7768) verified (Gen4+13 in the change log). This blender places no ad next to a HighRisk post, places next to a MediumRisk post only ads whose advertiser allows that context, and gives such ads the MediumRisk slots first verified (Gen4+10 in the change log). As of the 2026-10-15 commit, the V2 verdict switch for the brand safety verdict and the fallback switch that substitutes the author-level label when there is no post-level Grok safety label are both on by default verified (2026-10-15 commit in the change log).

Ads blending and brand safety verdict

The public code contains the entire ads blending. This is the unexpected part of this release.

• At least 5 organic posts are required before ads can be attached verified — home-mixer/ads/util.rs:10
• The default ad spacing is {requested: 3, min: 2}, and the minimum requested spacing is 3 verified — ads/util.rs:12-17
• Ads are inserted only up to half of the "safety gap" verified — ads/partition_organic_blender.rs:35
  • The formula is max_from_safe = safe_count / 2.
• If the last item is an ad, it is trimmed verified — ads/util.rs:164-198
• Brand safety is an adjacency verdict. The partition_organic_low_risk blender, the default until the 2026-09-01 commit (Gen4+13), attaches no ad next to MediumRisk or HighRisk posts (has_avoid() treats both tiers as avoid). The multi_risk blender, the default since Gen4+13, first tries to place a BSR-high ad between two MediumRisk posts and, if it fails the overlap check, moves it to a Safe slot verified — multi_risk_blender.rs:68-179. The verdict is made from a hand-written rule cascade plus whether Grok scored the post and whether PTOS reviewed it verified — home-mixer/models/brand_safety.rs:5-13, 15-22, 45-48
  • The 2026-08-24 commit (Gen4+7, d011592) introduced the BrandSafetyVerdict::HighRisk = 4 tier verified — home-mixer/models/brand_safety.rs
    • Five labels previously in medium and the new SOFT_NSFW were promoted to HIGH_RISK_LABELS. v1 and v2 are symmetric.
      • The five are NSFW_HIGH_PRECISION, NSFW_HIGH_RECALL, GORE_AND_VIOLENCE_HIGH_PRECISION, PDNA and EGREGIOUS_NSFW.
    • The NSFW ancestor-inheritance escalation in ads_brand_safety_vf_hydrator.rs changed from MediumRisk to HighRisk.
    • scored_posts_server.rs added the proto HM enum mapping for five V2 labels (downstream exposure begins).
      • The five are GROK_SFA_V2, GROK_NSFA_LIMITED_V2, GROK_NSFA_V2, GROK_NSFA_EXPANDED_V2 and PTOS_REVIEWED.
• If the quote post lookup fails, the verdict is pessimistically demoted to MediumRisk verified — ads/../ads_brand_safety_hydrator.rs:144-145
• Gen4+2 (2026-08-17, b089ce6) wired in the V2 verdict function side by side, and Gen4+3 (2026-08-18, 11a71f8) redesigned its semantics a day later.
• The Gen4+2 original (b089ce6, 2026-08-17) added two label lists for V2, a migration helper and a verification test verified — home-mixer/models/brand_safety.rs
  • MEDIUM_RISK_LABELS_V2 has 16 entries. Compared with V1, GROK_NSFA was renamed GROK_NSFA_V2, GROK_NSFA_EXPANDED_V2 was introduced, and the list includes NSFA_HIGH_PRECISION.
  • LOW_RISK_LABELS_V2 has 3 entries. Compared with V1, GROK_NSFA_LIMITED was renamed GROK_NSFA_LIMITED_V2, and the list includes NSFA_LIMITED_INVENTORY.
  • The migration helper strip_v1_grok_written removes from the input label set, before the V2 verdict, any label that meets both conditions.
    • The first condition is that the label is NSFA_HIGH_PRECISION or NSFA_LIMITED_INVENTORY.
    • The second condition is that the label's botmaker rule id is one of the seven in PROMPT_OWNED_RULES = [1400, 1410, 1420, 1500, 1510, 1610, 1700].
    • That is, the V2 logic ignores labels dual-written by the V1 prompt and decides from pure V2 labels only.
  • The verification test v2_mirrors_v1_across_tier_matrix asserts that the V1 and V2 verdicts must match exactly across 6 label combinations × two tweet_id points in time (before/after PTOS_CUTOFF).
• The Gen4+3 redesign (11a71f8, 2026-08-18) deleted this helper and added a gate that decides with V2 only when at least one V2-written label is present and otherwise falls back to V1 verified — home-mixer/models/brand_safety.rs
  • strip_v1_grok_written was deleted entirely (even the Cow import was removed). That is, the seven-rule-id PROMPT_OWNED_RULES logic that the canon update folded in yesterday was discarded within a day.
  • A new constant V2_WRITTEN_LABELS was added instead. It has 4 entries: GROK_SFA_V2, GROK_NSFA_V2, GROK_NSFA_LIMITED_V2 and GROK_NSFA_EXPANDED_V2.
  • The new gate checks at the start of compute_verdict_v2 whether any of V2_WRITTEN_LABELS is in the label set. If none is, it falls back to compute_verdict(v1), and if one is, it decides from the pure V2 label set.
  • NSFA_HIGH_PRECISION was removed from MEDIUM_RISK_LABELS_V2, and NSFA_LIMITED_INVENTORY from LOW_RISK_LABELS_V2. That is, once V2 gives a verdict, the V1 heuristic labels are ignored.
  • The new test v2_defers_to_v1_when_v2_has_not_ruled explicitly asserts the decisive case.
    • In particular, for the label set NSFA_HIGH_PRECISION + GROK_NSFA + GROK_SFA_V2, the V1 verdict is MediumRisk and the V2 verdict is Safe.
    • That is, once V2 sends a verdict signal (GROK_SFA_V2 alone), the V2 decision governs and V1's traditional heuristics are ignored entirely.
  • Interpretation: The dual-write migration strategy moved from "filter specific rule ids when V2 decides" to "use the V2 decision if V2 has ruled, otherwise fall back to V1." The latter is much cleaner dual-write semantics. Posts the V2 labeler has not yet reached stay covered by the V1 safety net, and posts it has reached are fully replaced by V2.
  • Undisclosed: When the V1 verdict function will finally be removed, the V2 labeler coverage, and the origin of the four V2_WRITTEN_LABELS labels (a Grok prompt change? a pipeline retraining?)
• The Gen4+7 commit (d011592, 2026-08-24) introduced the HighRisk tier in the brand safety verdict verified — home-mixer/models/brand_safety.rs
  • A fourth grade, BrandSafetyVerdict::HighRisk = 4, was added on top of the existing Unspecified/Safe/LowRisk/MediumRisk. Because the enum discriminant is assigned 4, the >= comparison in worst_verdict() extends automatically.
  • Five labels were promoted from medium to high. They are NSFW_HIGH_PRECISION, NSFW_HIGH_RECALL, GORE_AND_VIOLENCE_HIGH_PRECISION, PDNA and EGREGIOUS_NSFW. One new label, SOFT_NSFW, is also HighRisk. In both V1 (compute_verdict) and V2 (compute_verdict_v2), the HighRisk check moved to the top.
  • V2 changed symmetrically too. The same five labels were also removed from MEDIUM_RISK_LABELS_V2. V2 originally had NSFW_HIGH_PRECISION, NSFW_HIGH_RECALL, GORE_AND_VIOLENCE_HIGH_PRECISION, PDNA and EGREGIOUS_NSFW.
  • The ads util treats both tiers as avoid. has_avoid() in home-mixer/ads/util.rs was extended to a MediumRisk | HighRisk matches (previously only MediumRisk).
    • That is, in the adjacency verdict, no ad is attached next to a HighRisk post, exactly as with MediumRisk.
    • There is no further action yet, and the severity may be for flagging or logging purposes.
  • Quote adjacency was also promoted. ads_brand_safety_vf_hydrator.rs escalates the verdict to HighRisk, not MediumRisk, when the quoted ancestor post has an NSFW author.
  • Downstream exposure began. Five V2 labels were added to the safety_label_to_proto mapping in scored_posts_server.rs.
    • The five are GROK_SFA_V2, GROK_NSFA_LIMITED_V2, GROK_NSFA_V2, GROK_NSFA_EXPANDED_V2 and PTOS_REVIEWED.
    • These labels are now actually carried out in the feed response proto. Previously, V2 labels were used only in the internal verdict and were not sent downstream.
  • DDG experiment tracking was added. ads_injection_logging_side_effect.rs exposes the EnableAdsBrandSafetyVerdictV2 experiment bucket in logging in the form ddg_experiment_bucket = "experiment_id:bucket".
    • DDG passes to the ads reporting pipeline (presumably Data Dashboard or a similar system inferred), which makes it possible to observe revenue and safety metrics for the V2 experiment group.
    • Why it matters: It indirectly shows that V2 is still at the experiment stage (experiment bucketing). Gen4+2 introduced only the V2 verdict function, Gen4+3 redesigned the semantics, and Gen4+7 put the experiment measurement infrastructure in place.
  • Undisclosed: What follow-up actions the HighRisk tier triggers beyond ads avoid (strong ranker decay, complete blocking of organic impressions, etc.) and the holdout/treatment ratio of the V2 experiment

Ads conversion action expansion — Gen4+6 (2026-08-21, 28e414f)

The Phoenix ranker's ads revenue prediction heads grew by 20 in the 2026-08-21 commit (Gen4+6), and the training loss was reorganized verified. After that, the Gen4+9 commit (2026-08-26, 45b48ba) added three pixel-fire-related heads, and the 2026-09-17 commit (Gen4+23) added 10 each of tier 2 and tier 3 to the delayed conversion actions verified.

Before Gen4+6, the ads conversion family effectively had only a few members (install and click-attributed conversions). The Gen4+6 commit expanded it as follows.

20 ActionNames were introduced in the proto verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto

• 6 MACT kinds × click/view = 12 (171–182) were added.
  • MACT is Mobile App Conversion Tracking. It attributes in-app events after an ad click or impression (purchase, add to cart, level achieved, tutorial completed, sign-up, custom) to the ad.
  • Each event has two versions: click-attributed (MACT_<event>) and view-attributed (MACT_VIEW_<event>).
• 8 view-through conversions (183–190) were added.
  • The targets are purchase, add_to_cart, checkout_initiated, sign_up, site_visit, session, landing_page_view and upper_funnel.
  • Each is a view-through head corresponding to the existing click-attributed one (ADS_PURCHASE_CONVERSION = 200, etc.).

The engagement metric group changed verified — phoenix/xrex/data/recsys/constants.py

• Added 12 new MACT mappings and 8 view-through mappings to ads_conversion_engagement_to_action_types.
• The new metric group ads_slim trains only 3 positive signals, 4 negative signals and non-primary conversion clicks. It is the slim signal combination of the ads ranking line.
  • The positive signals are open link, external link long dwell and video quality view.
  • The negative signals are not interested, block, report and mute.
• CLIENT_TWEET_SEARCH_QUERY_REFORMULATED was added to NEGATIVE_FEEDBACK_HEAD_INDICES. When a user rewrites the query because they dislike the search results, it is treated as negative feedback. This is in the same direction as the xrecsys_search config.

A delayed feedback training branch was added verified — phoenix/xrex/models/recsys_model.py

There are four new config options. The defaults in the table are as of the Gen4+6 commit.

There are two new batch fields.

• sample_source (bool) comes from the is_delayed_feedback column of the kafka record. If True, this sample is a conversion delayed re-observation.
• conversion_keep_mask (bool per-candidate) is the kafka conversionKeepMask column. If False, this candidate is excluded from conversion training.
  • In sequence packing, only all(True) is allowed (when not linked).

The new metric slices are fresh (delayed=0), delayed_clicked (delayed=1 & clicked=1) and delayed_non_clicked (delayed=1 & clicked=0). Because condition_conversion_on_click is kept in delayed training as well, even a delayed sample trains conversion under the condition "was there a click."

In the 2026-09-18 commit (Gen4+24), three options were unified into a single ads_head_masking: bool = False. The three options are split_head_training_by_source, train_view_through_heads and condition_conversion_on_click. The three slice names (fresh, delayed_clicked, delayed_non_clicked) are kept, and a new slice late_split_no_early was added. Sample source is a three-valued categorical (FRESH=0, EARLY_RELABEL=1, LATE_SPLIT=2). The twin head mapping is summarized in Gen4+24 in the change log.

Why it matters: The Gen4+6 commit attached two things to ads conversion prediction at once. One is a large increase in the number of heads (20 introduced), and the other is infrastructure that formally branches delayed feedback training. This is a signal that X ranker training is explicitly aligned with the standard treatment in the ads delayed conversion prediction literature (since Chapelle 2014). Separate from the ranking signal channel, SID and SlateContext extension axes that the canon has folded in through Gen4~+5, the ads revenue prediction stack was added as a new axis.

In Gen4+9 (2026-08-26, 45b48ba), three combined pixel-fire and long-dwell heads were added and conv_asset_ids was wired in verified — phoenix/python/common/xai-proto/proto/recsys.proto (+12)

• Other evidence files are phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto and home-mixer/models/candidate.rs (+8).
• Three new ActionNames (191–193) were added.
  • 191 ADS_PIXEL_FIRE is the firing of a conversion pixel (web tracking pixel) that an advertiser planted on their own site or app. It directly predicts web conversion events.
  • 192 ADS_LONG_DWELL_AND_PIXEL_FIRE combines pixel fire and long dwell (staying long on the landing page) with AND. That is, it is the case where the pixel did not merely fire but the user actually stayed long.
  • By the Gen4+6 convention, the prefix P_ of 193 P_LONG_DWELL_AND_PIXEL_FIRE suggests a primary or predicted target family inferred.
• A ConvAssetIds message was introduced and wired in as the field PredictNextActionsRequest.conv_asset_ids = 18.
  • It consists of three arrays: impressed_time_ms (int64), author_id (int64) and asset_id (int64).
  • It passes to Phoenix serving, along with the request, the history of ad creatives (assets) the user was recently shown.
  • This history is presumably consulted in conversion prediction when an ad asset is reused across several advertisers or when a user saw the same asset several times inferred.
• Two Option<u32> fields (SlateContext.recon_count_above, recon_gap_above) were added to the home-mixer SlateContext struct.
  • These fields break down the recon reconstruction metrics of served_slate_context.
  • They are not yet reflected on the proto side. The proto has only the field 13 reconCosMilli.
  • The home-mixer local wiring is waiting for the proto migration.
• Why it matters: This follows Gen4+6's large expansion of the ads conversion prediction heads by 20. Gen4+9 adds three heads that combine pixel fire and long dwell, and creates a channel to carry asset history in the serving request. constants.py and models/recsys_model.py do not yet reflect it. Because only the proto was introduced, the training and serving wiring is a checkpoint to watch in the next commit.

In Gen4+13 (2026-09-01, 7ba7768), the DPA product SID was carried in the serving wire format verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto, phoenix/python/common/xai-proto/proto/recsys.proto

Dynamic Product Ads (DPA) is an ad format in which an advertiser uploads a product catalog and X automatically generates and shows ads tailored to each user. This commit added repeated int32 sid_codes = 8 to the DPA DpaProductInfo. That is, SID codes for ad DPA products are carried in the request payload into the ranker. This is a separate pipeline from the SID of organic posts, which is already carried as post_sids in history_seq and candidate_seq.

In the same commit, reserved 214 to 252 was added to the DiscreteActionName enum; reserving 39 slots in advance appears to secure room for further expansion of ads conversion actions inferred.

Why it matters: Ads DPA ranking has begun to be incorporated into the SID family of signals. Paired with the Phoenix ranker's Gen4+13 SID recon mode, this opens the possibility that ads DPA products and organic posts are represented under the same SID vocabulary. However, this commit alone does not tell us the following three things:

• Whether the DPA SID codebook is the same as or different from the organic post SID codebook
• Whether a server-side connection that consumes sid_codes already exists
• Whether there are A/B results

In Gen4+23 (2026-09-17, 42266f3), delayed conversion attribution was expanded to 3 tiers verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto, phoenix/python/common/xai-proto/proto/recsys.proto

The existing reserved 214 to 252 was narrowed to reserved 234 to 252, and tier 2 and tier 3 were attached at 214-233.

• 10 Tier 2 actions (214-223) were added.
  • ADS_ATTRIBUTED_KEY_CLICK_CONVERSION_DELAYED_TIER_2·ADS_ATTRIBUTED_CLICK_CONVERSION_DELAYED_TIER_2·ADS_PURCHASE_CONVERSION_DELAYED_TIER_2·ADS_MID_FUNNEL_CONVERSION_DELAYED_TIER_2·ADS_ADD_TO_CART_CONVERSION_DELAYED_TIER_2·ADS_UPPER_FUNNEL_CONVERSION_DELAYED_TIER_2·ADS_WEB_CONVERSION_DELAYED_TIER_2·ADS_SEARCH_CONVERSION_DELAYED_TIER_2·ADS_SIGN_UP_CONVERSION_DELAYED_TIER_2·ADS_CHECKOUT_INITIATED_CONVERSION_DELAYED_TIER_2
• 10 Tier 3 actions (224-233) are the _TIER_3 symmetric counterpart of the same 10.

Why it matters: The 10 *_DELAYED actions (204-213) introduced by the 2026-09-01 commit (Gen4+12) were expanded to tier 2 and 3 in 16 days verified — recsys.proto:446-455 in 6384ca7. The delayed window is now divided not into one but into three intervals by lag. The delayed feedback literature of the Chapelle (2014) and Yasui&Sakata (2020) line has taken the approach of mitigating propensity bias with per-lag masks, and this change is in the same direction inferred. Which part of the observation window each tier covers and how the loss weights are combined across tiers are not stated in this commit. Whether values like mparams["*_tier_2_loss_weight"] appear in the training config is the checkpoint to watch.

↑ Contents

Phoenix training and index (introduced in Gen4)

Since the Gen4 release (2026-08-13), Phoenix can train and serve a small model end-to-end, but the production checkpoint, data, and infrastructure are not public verified (training code). The ranker's dense-parameter optimizer changed from Adam to Muon in the 2026-08-19 commit (Gen4+4) verified (training code). The path by which home-mixer uses the slate context computed by Phoenix serving was added in the 2026-08-20 commit (Gen4+5), and the default of its gate was false verified (prediction output). SID reconstruction input and StableHLO bundle export were added in the Gen4+13 commit (2026-09-01, 7ba7768), and both were off by default verified (input features, infrastructure).

Training code, optimizer, and training config

As of the 2026-08-21 commit (Gen4+6), the released flagship ranking config and nano config use the same Muon optimizer recipe as the production home ranker verified. The training config for search ranking was added next to the For You feed config in the 2026-08-19 commit (Gen4+4) verified.

Phoenix can be run end-to-end since the Gen4 release verified — README.md #deployment-related-code.

• A Cargo workspace, pyproject.toml, QUICKSTART, and a synthetic data generator are included verified — README.
• The scope the README states is a small model: "a small model can be trained and served end-to-end". As of this README, the production checkpoint, data, and infrastructure are still not public verified — README.
• Loss and negative sampling details need a deep dive. The optimizer was released in the 2026-08-19 commit (Gen4+4).

In the 2026-08-19 commit (Gen4+4), Phoenix's dense-parameter optimizer changed from Adam to Muon verified — phoenix/xrex/optimizers/recsys/muon.py (240 lines added).

• The same commit added phoenix/xrex/optimizers/recsys/dense_optim.py (47 lines) and changed the default config in phoenix/xrex/configs/xrecsys.py.

Muon is an optimizer that approximately orthogonalizes gradient momentum with Newton-Schulz orthogonalization. The Muon in this code wraps optax.contrib._muon and became the default dense-parameter optimizer of the Phoenix ranker. The application is hybrid: only matrix-shaped parameters use Muon, and parameters whose names match the patterns below still use Adam.

muon_adam_patterns = "embed,_emb,logits,vocab,norm,table"

In other words, embedding tables, the output logits head, vocab, and norm (layernorm/rmsnorm) stay on Adam. Only the attention and FFN weight matrices of the Transformer body go to Muon. This setup matches the standard Muon hybrid pattern (Keller Jordan 2024 and the Kimi/Moonshot K2 line).

The default config of _home_direct_packed_base() changed as in the table below.

The GB300 override was updated in the same direction. The values are Muon, LR 5e-4, half_life_steps=5000, and weight_decay=0.01.

This is the first time the code confirms Muon adoption in a large-scale production recsys training pipeline that this repository has covered. A new observation axis appears: an optimizer that started in LLM training has moved to ranker and retrieval training. An optimizer axis needs to be added to topics/sequence-transformer-ranking.md.

In the 2026-08-21 commit (Gen4+6), the Muon training recipe was formally included in the export verified — phoenix/TRAINING.md.

• The same commit also changed phoenix/README.md and phoenix/reference/README.md.

The canon's previous statement was: "The dense optimizer of the Phoenix export is standard Optax AdamW; production's tuned RMS-normalized-Adam derivative is undisclosed." Even after muon.py and dense_optim.py were exported in Gen4+4, which introduced Muon, TRAINING.md still said that only AdamW was exported. In this commit, TRAINING.md changed as follows.

Two dense-parameter optimizer families ship in this export:

• Muon (xrex/optimizers/recsys/muon.py): the production home-ranker recipe — consistent-RMS scaling with decoupled weight decay on the matrix and embedding partitions. The flagship ranking configs and home_direct_packed_nano select it (optim="muon"), so the nano trains with the same dense-optimizer recipe production runs.
• Standard Optax AdamW: the slot used by the remaining shipped configs (two-tower retrieval, gen-recs, and the legacy ranking presets). ...

— phoenix/TRAINING.md

• The flagship ranking config and home_direct_packed_nano train end-to-end with the production Muon recipe verified — TRAINING.md. Running the exported nano model as is gives the same Muon training as production.
• The AdamW arm remains only in two-tower retrieval, gen-recs, and legacy ranking presets verified — TRAINING.md. These are still the export-safe equivalent of production's tuned RMS-normalized-Adam derivative.
• It also states that three options come with the rowwise AdaGrad for embeddings in flagship and nano verified — TRAINING.md.
  • The three options are accumulator half-life decay, lazy per-row decay, and decoupled weight decay.
  • The emb_optim_config parameter (LR 0.28, half-life 2500 steps) introduced in Gen4+4 has been settled in the canon.
• The train_step.py entry in reference/README.md was also generalized to "dense optimizer + rowwise-AdaGrad" verified — reference README. The reference training script configures the AdamW arm, while the flagship and nano configs use the Muon recipe.

Why it matters: With this commit, running the exported training Python code as is gives the same training as the production Muon recipe. When Muon entered the code in Gen4+4 (2026-08-19), the canon's understanding was "the code is exported, but the training recipe is still production-specific." The reproducibility documentation has narrowed, but the data and infrastructure are still not public (the README caveat stays).

In the same commit (Gen4+6), the nano config inherits the LR and optimizer settings of the GB300 hardware-specific override verified — phoenix/xrex/configs/xrecsys.py:330-345.

• _GB300_OVERRIDES itself has existed since the Gen4 release (batch, EP, attention, remat, and LR entries) verified — xrecsys.py:305-313, 368. The optimizer entries were added in the 2026-08-19 commit (Gen4+4).
• The block below shows the values as of Gen4+6.

_GB300_OVERRIDES = {
    "bs_per_device": 512,
    "ep": 64,
    "attn_impl": "cutedsl_ranker_varlen_attn",
    "remat_policy": RematType.SAVE_GB300_RECSYS,
    "unroll_layer_stack": True,
    "learning_rate": 5e-4,
    "checkpoint_every_n": 300,
    "optim_config": RecsysDenseOptimConfig(
        optim="muon",
        muon_consistent_rms=0.2,
        muon_matrix_weight_decay=0.01,
        muon_split_fused="qkv:128",
        adam_embedding_weight_decay=0.01,
        b1=0.95,
        b2=0.98,
        ...
    ),
    ...
}

• bs_per_device=512 is 8× the previous bs_per_device=64 of _home_direct_packed_nano. It reflects the HBM headroom of GB300.
• ep=64 sets expert parallelism to 64-way.
• attn_impl="cutedsl_ranker_varlen_attn" is a variable-length attention kernel based on NVIDIA CUTLASS/CUTe DSL. It is a new implementation optimized for GB300.
  • This name is not in open-source catalogs, so it may be xAI's own kernel inferred.
• remat_policy=RematType.SAVE_GB300_RECSYS is a rematerialization policy tuned to the GB300 memory hierarchy.
• unroll_layer_stack=True compiles the Transformer layer stack unrolled. This leaves room for XLA optimization.
• checkpoint_every_n=300 saves a checkpoint every 300 steps.

_home_direct_packed_nano inherits the LR, optim_config, and emb_optim_config of this override. Previously nano had its own learning_rate=2e-3, but from this commit it is unified on the GB300 basis.

Why it matters: This is the first case where X exposes a hardware generation transition in code. NVIDIA GB300 (Grace Blackwell 300) is the latest data-center GPU, which began shipping in the second half of 2025, and the successor to Blackwell B200. LLM companies are securing this generation in bulk, and xAI applied it as is to the ranker training and serving pipeline as well. The new fact for the canon is that Phoenix production training runs on GB300. That even nano is tuned to the GB300 config means the export itself assumes a GB300 reproduction scenario.

The remat policy for H100 was added in a later commit verified (Gen4+15 D in the change log).

In the 2026-08-19 commit (Gen4+4), Phoenix expanded to search ranking by introducing the xrecsys_search config verified — phoenix/xrex/configs/xrecsys.py:435-473.

xrecsys_search was added next to xrecsys, the For You feed training config. The main differences from the For You config are as follows:

• The vocab grew substantially. The search query and click corpus has greater user and item diversity, so the vocab must be larger than the much smaller vocab of the For You config.
  • The values are user_vocab_size=100_000_000, item_vocab_size=100_000_000, author_vocab_size=30_000_000, and ip_vocab_size=10_000_000.
• The architecture values are num_layers=8, emb_size=2560, emb_table_width=1024, query_heads=20, kv_heads=4 (GQA), history_seq_len=1022, and candidate_seq_len=64.
• The distribution settings are ep=512, dp=2, fsdp=1, and tp=1. With expert-parallel 512, the EP axis is much larger than in the For You base config, and the MoE routing is also much larger in the search config.
• There are search-specific flags. use_product_surface=True turns on product surface conditioning, and output_vocab_size=128 expands the actions to 128 kinds.
  • The rest are condition_search_relevance_on_prompt=True, use_dense_action_table=True, and metric_group="search".
• The training signal uses a separate group, group_id="user_action_sequence_xrecsys", and log_q_correction=True.
• The training scale is total_samples=1e11, the same as the For You base.

Meaning: Phoenix is expanding from the For You feed ranker to a search surface ranker. This config pairs with two actions renumbered in the proto from 161/162 to 105/106 (CLIENT_TWEET_RELEVANT_TO_SEARCH = 105, CLIENT_TWEET_NOT_RELEVANT_TO_SEARCH = 106) (prediction output schema). Previously the canon understood each surface (For You, Following, search) as a separate codebase. This commit confirms a new fact: unification is under way toward serving both search and feeds with the same Phoenix Transformer architecture, swapping only the per-surface config. The direction matches the "single ranker for search and recommendation" trend of Netflix UniCoRn and GenRec (topics/llm-in-recsys.md).

In the 2026-08-21 commit (Gen4+6), three options were added to the nano config of two-tower retrieval verified — phoenix/xrex/configs/xrecsys_two_tower.py:429-431.

The options went into the xrecsys_two_tower_nano entry of MODEL_CFGS verified — xrecsys_two_tower.py:411-431. This entry does not inherit the combined base, so until then all three keys used the defaults in the model construction code verified — same file, lines 530, 627, 675, 681. In the combined base, the three keys have been False, 80.0, and True since the Gen4 release, and this commit did not change the base verified — xrecsys_two_tower.py:274-275, 318.

• qk_norm=True
• attn_logit_cap=-1: turns off logit capping.
• right_anchored_rope=True: sets the RoPE positional encoding to right-anchored.

A stability trick from LLM training has moved to the retrieval two-tower. Separately, the EVERGREEN retrieval dataset file was drastically reduced from evergreen_video_1825day.parquet to video_4to14day.parquet. The name and the content now disagree, and this may be temporary inferred.

In the same commit (Gen4+6), the retrieval post-embedding computation was split into 65k-row chunks verified — phoenix/xrex/train/trainer_recsys.py _forward_chunked.

maybe_build_retrieval_post_embeddings previously computed a whole shard with a single candidate_tower_forward_jit call. From this commit, on even shards where total_samples % data_world_size == 0, it computes in units of _chunk_rows = min(_shard_rows, 65536). This stabilizes the subroutine as training scale grows with the Muon promotion and the GB300 config.

Prediction output schema and slate context

In the 2026-08-19 commit (Gen4+4), a reward output and a 5-quantile head for link-open probability were added to the Phoenix prediction output schema verified. The path by which home-mixer receives and uses the slate context computed by Phoenix serving was introduced in the 2026-08-20 commit (Gen4+5), and the default of its gate was false verified.

In the 2026-08-19 commit (Gen4+4), a reward head and a link-open 5-quantile head were added to the Phoenix proto verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto.

The Phoenix prediction output schema expanded in two directions.

1. A RewardOutputs message was added, providing scaffolding for Phoenix to emit a reward value head and a policy logits head alongside the 24 probability prediction heads verified — recsys.proto.
   • A RewardOutputs rewardOutputs = 8 field was added to NextActionDistribution. The two fields of RewardOutputs are values, a scalar array, and policyLogits, the policy logits.
   • optional double rewardRerankSlotProb = 5 was also added to ScoreInfo.
   • It is presumably used for RL-style (actor-critic value+policy) training or reward-model-guided re-ranking slate decisions, and no scorer that consumes this field is connected yet inferred. It is still at the hidden state scaffolding stage inferred.
   • The training code of the page-level RL re-ranker for immersive surfaces, released later, reads the rewardRerankSlotProbSeq column as rl_rerank_prob and uses it as the logged probability of the page RL loss verified — recsys_batch.py:284, page_rl.py:206, 216, 233 (the page-level RL re-ranker commit in the change log).
     • The claim that this column collects the rewardRerankSlotProb field of the serving response rests on the names being the same inferred.
2. A head was added that predicts link-open probability not as a single scalar but as a distribution of five quantiles (P10/P25/P50/P75/P90) verified — recsys.proto.
   • The five slots PLACE_HOLDER_167/168/169/170/253 were renamed P_OPEN_LINK_P10, P_OPEN_LINK_P25, P_OPEN_LINK_P50, P_OPEN_LINK_P75, and P_OPEN_LINK_P90, respectively.
   • Before this commit, X's prediction heads were of the single-probability or scalar kind. This head is the first time a distribution-shaped prediction entered the ranker.

The same proto change also includes these:

3. The search relevance actions were renumbered as CLIENT_TWEET_RELEVANT_TO_SEARCH = 105 and CLIENT_TWEET_NOT_RELEVANT_TO_SEARCH = 106 verified — recsys.proto. The old 161/162 remain as *_DEPRECATED. These actions pair with the xrecsys_search config.
4. SlateContext gained seven SID fields verified — recsys.proto. The fields are sidKnown, sidK1/2/3, and sidGap1/2/3. This aligns, at the proto level, with the 2026-08-18 commit (Gen4+3) that connected SID to VMRanker.

The following questions need a deep dive:

• In which training loop is RewardOutputs actually filled? In this commit recsys_model.py changed substantially, +265/−25. The key is whether actor-critic heads were added or a reward model is trained separately.
• Is the link-open 5-quantile a quantile regression or a categorical mixture?
• Has the xrecsys_search config been deployed to production, or is it still used only for offline training?

In the 2026-08-20 commit (Gen4+5), home-mixer started receiving and using the slate context computed by Phoenix serving verified — home-mixer/models/candidate.rs:26-27, 76-77, 99-118.

• The remaining evidence is home-mixer/scorers/phoenix_scorer.rs:107-125, home-mixer/scorers/ranking_scorer.rs:700-710, 806-825, 875-895, and home-mixer/params/param.rs:243-249.

The SlateContext.sid* fields added to the Phoenix proto in Gen4+4 were connected to the consumption path of the actual pipeline in this commit. Four places changed at once across three files (candidate.rs, phoenix_scorer.rs, ranking_scorer.rs).

1. PostCandidate gained a new field served_slate_context: Option<SlateContext>. It is a second storage slot next to the existing slate_context (a value home-mixer computes locally and then caches), and it holds only the slate context that came from the Phoenix response.
2. A conversion impl between proto and Rust, From<xai_recsys_proto::SlateContext>, was added. This impl renames the proto's sid_k1/2/3 and sid_gap1/2/3 to home-mixer's sid_k_l1/l2/l3 and sid_gap_l1/l2/l3. The "l" stands for level.
3. PhoenixScorer takes the slate context out of the prediction response and puts it in this field. If the Phoenix prediction response carries a SlateContext along with the 24 candidate scores, it is used as is.
   • The SlateContext at this point contains pool_rank, pool_rank_gap, fatigue, pre_diversity_score, sid_known, sid_k_l1/2/3, and sid_gap_l1/2/3.
   • The code is predictions.candidate_slate_context(&c.get_original_tweet_id()).map(Into::into).
4. RankingScorer decides which slate context to use through a three-way fallback. The new gate is UseServedSlateContext, and its default is false.

   let persisted_contexts: Option<Vec<SlateContext>> =
       match Self::served_slate_contexts(query, candidates) {
           Some(served) => Some(served),
           None if query.has_cached_posts => Self::stored_slate_contexts(candidates),
           None => Some(Self::compute_slate_contexts(candidates, &weighted_scores)),
       };
   

   The fallbacks apply in order from the top, and both ranking branches (mpn_scoring, diversity) use the same fallback.
   1. If the gate is on and Phoenix sent the slate context for every candidate, it is used as is.
   2. If the gate is off and the Redis cache hits, the stored context is used.
   3. Otherwise, it is computed locally.

Why it matters: This is the first step of a migration that moves the place where the slate context is computed from home-mixer locally to Phoenix serving. At the moment Phoenix produces a prediction, it already knows each candidate's pool rank, fatigue, pre-diversity, and three-level SID information. Therefore, sending these in the serving response instead of recomputing them in home-mixer gains two things:

• (a) It saves ranker CPU.
• (b) It can reflect the richer context that Phoenix has. The three SID levels cannot be computed locally by home-mixer.

This matches the direction of the SID connection to VMRanker in the 2026-08-18 commit (Gen4+3). By definition, SID is a Phoenix training artifact and cannot be computed locally; this commit kept UseServedSlateContext off by default, so it only opened the gate, and the actual rollout is planned as a later parameter change. What to watch next is the commit that raises this gate to true. That commit marks the point when the slate context from Phoenix serving reaches production traffic.

In the 2026-08-21 commit (Gen4+6), SlateContext was also added to NextActionDistribution as field #9 verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto.

SlateContext first appeared in Gen4+4 as the field slateContext = 4 of ScoreInfo, the context attached to re-ranking slate decisions, and in Gen4+5 it was connected as PostCandidate.served_slate_context, which home-mixer started using. In this commit, a SlateContext slateContext = 9 field was also added to NextActionDistribution, the per-action prediction distribution of each candidate. That is, a slate context is now attached to each action distribution as well, so from this commit it can be carried in two places, ScoreInfo and NextActionDistribution. The consumption path is not yet connected, and this is at the scaffolding stage inferred.

Ranker input features and SID reconstruction

Three bool input features of the Phoenix ranker have been filled with real values since the 2026-08-14 commit (Gen4+1) verified. These three became default training inputs in the 2026-08-19 commit (Gen4+4) verified. The recon mode, which turns SID codes back into dense vectors and adds them to the ranker input, was released in the Gen4+13 commit (2026-09-01, 7ba7768), but at release its default was the existing learned mode verified.

In the 2026-08-14 commit (Gen4+1), three new Phoenix input features were added verified — phoenix/crates/common/xai-recsys/src/util.rs:456-540.

• phoenix/crates/common/xai-recsys/src/model_config.rs:40 in the same commit is also evidence.

• In the code from Gen3 to early Gen4, bool_features was stamped entirely as vec![false; ...], so the input itself had no effect (a "dead field" kind in the canon caveat list). In Gen4+1, three were filled for the first time verified.
  • The evidence is the patch replacing bool_features: vec![false; ...] → bool_features: candidate_bool_features.
• The stale signal comes in as a pair: count zeroing and a bool set. It is an explicit label telling the ranker that "these counts are low not because the post is unpopular but because it is old and its impressions have stopped." The purpose is likely to mitigate Gen3's snowball problem inferred.
• The is_mutual_follow boost (reply +15) was a condition hardcoded in the scorer. From this commit, the model receives the follow direction as separate input bools and can learn a similar pattern through training.
• In the 2026-08-19 commit (Gen4+4), all four features were promoted to train-default ON verified — phoenix/xrex/configs/xrecsys.py:270-290.
  • In the _home_direct_packed_base() default config, the four flags enable_stale_post=True, enable_is_author_followed_by_viewer=True, enable_is_author_following_viewer=True, and enable_engagement_counts=True were raised to the FeaturePrepConfig defaults.
  • The three bools filled in Gen4+1 have been default inputs of the training pipeline since this commit.
  • Separately, only the grpc_recsys dataset type overrides enable_stale_post=False, so the stale-flag stamp is off on the real-time serving path. The reason is presumably that count zeroing can be risky in real time inferred.

In the Gen4+13 commit (2026-09-01, 7ba7768), an SID recon mode was added to the Phoenix ranking model. It is off by default. verified — phoenix/xrex/models/recsys_sid.py (72 lines added).

• In the same commit, phoenix/xrex/models/recsys_model.py changed by +35/−2.

This commit is the first code release of the path by which X attaches RQ-VAE-family SIDs to the ranker input. It does not replace the retrieval model with SID; it is a path that reconstructs SID codes into input embeddings for the ranker post tower.

_load_sid_decoder(path) reads the tensors below from an offline .safetensors.

• stages is a stacked residual codebook (RQ-VAE style) of shape (num_levels, codebook_size, code_dim). For each level, stages[l, code_l, :] is picked and summed. The codebook size and number of levels are determined by the artifact and are not hardcoded in the code.
• dec_w0, dec_b0, dec_w1, and dec_b1 form a 2-layer MLP decoder (relu → linear). It restores an SID sequence to a dense vector.

The recon function reconstruct_entity_sid turns SID codes into ranker input vectors in the order below verified — phoenix/xrex/models/recsys_sid.py.

• (a) codes = sids - 1 converts 1-based to 0-based, and rows with sids[..., 0] == 0 are marked "missing".
• (b) For each level, stages[l, code_l, :] is picked to compute the residual sum.
• (c) It is restored with a 2-layer MLP and relu.
• (d) It is L2-normalized (eps=1e-12).
• (e) Missing rows are masked to 0.
• (f) The decoder is kept fully frozen with stop_gradient.
• (g) Only one linear projection from recon_dim → target_dim ({prefix}_sid_recon_proj, trainable) is a parameter. The result is returned with its dtype cast to fit the ranker post tower.

The mode is chosen by a config gate verified. Two fields were added to RecsysAggregatedModelConfig:

sid_embedding_mode: Literal["learned", "recon"] = "learned"
sid_decoder_path: str = ""

• The default "learned" keeps using the existing SID embedding table (a lookup trained inside the post tower).
• Setting "recon" bypasses the learned SID table. Instead, it builds recon vectors from the SID codes of history_seq and candidate_seq and adds them to the post tower embedding (add).
  • The requirements are the legacy build_inputs path (feature_prep_enabled=False), no sid_hash_level, no sid_cross_attn, and a specified sid_decoder_path.

Why it matters

• This is the first code release of X's SID line. This commit is the first to show in code how SID codes themselves are turned back into dense features. Until now, X's public SID-related material comprised three items:
  • (a) In Gen4+3 (11a71f8, 2026-08-18), the sid_known, sid_k_l1/2/3, and sid_gap_l1/2/3 fields were connected to VMRanker.
  • (b) In Gen4+4 (aad7179, 2026-08-19), the SlateContext.sid* fields were added to the Phoenix proto.
  • (c) In Gen4+5 (d0cef2f, 2026-08-20), a path was added by which SID is carried in the slate context of the serving response.
• The decoder is fully frozen. Because it uses stop_gradient and an offline artifact, serving only needs the codebook and MLP matrices deployed, and the only thing trained is the single recon_dim → target_dim projection. Codebook tuning happens only offline, and ranker training does not disturb the codebook.
• "learned" and "recon" become the A/B axis. That the default is still learned is read as meaning that recon mode's production validity has not yet been verified, or that it is before dial-up inferred. If sid_embedding_mode="recon" appears in a production config in the next commit, it will signal that the X SID pipeline has moved from a "trained table" to "offline codebook + recon at serving time".
• This path is not a retrieval model. Restoring vectors from the codebook and attaching them to the ranker input increases the density of the candidate representation; it does not replace candidate generation with SID. Its lineage differs from the approach of Kuaishou OneRec and Tencent TGR (radar.md generative retrieval thread), which "treat SID as the autoregressive generation target". As of this commit, X still uses SID as an input signal inside the cascade. A generative retrieval source that uses SID for candidate generation was later released off by default verified (candidate generation section).

This commit also leaves some things undisclosed.

• Code of the SID training pipeline that builds the artifacts (encoder and VQ training)
• Measured codebook size and number of levels
• Whether sid_embedding_mode="recon" is on in production
• A/B metrics for the learned and recon modes

Phoenix retrieval index

The post index that Phoenix retrieval queries, and the event layer that decides which index a post goes into, were released with Gen4 verified. A post gets a visibility verdict before it enters the index verified. Since the 2026-08-17 commit (Gen4+2), candidates with no likes are also published as indexing events verified.

The Gen4 release also included the Phoenix retrieval index services verified — README.md #retrieval-index.

• phoenix-rankall/ is the post index that Phoenix retrieval queries. It is updated when events arrive verified — README.
• phoenix-rankall-strato/ is the event layer that decides which index a post belongs to. This layer consults visibility-filtering first verified — README.
• In the 2026-08-14 commit (Gen4+1), one SID window, video_14day, was added verified — phoenix-rankall/src/config/mod.rs:177-180.
  • It pairs with the existing video_48h, video_96h, and nsfw_video_48h/168h/14day.
  • A video popularity signal over a 2-week window entered the retrieval SID.
• In the 2026-10-20 commit (Gen4+45), the hourly video windows video_6h and video_6hto2day were added, so the shortest video window of the SID index went from 48 hours to 6 hours verified — phoenix-rankall/src/config/mod.rs:212-213.

The indexing pipeline is already aligned with the ranking-side visibility rules. A post already carries a visibility verdict before it enters the index. This is an architecture fact that the canon did not know at all through Gen3.

Two indexing gates were removed in the 2026-08-17 commit (Gen4+2) verified — phoenix-rankall-strato/columns/phoenix_rank_all/phoenixRankAllCandidateProcessor.strato.

• build1FavIndex publishes every candidate from this commit. In other words, the "1fav" index name became a backward compat label, and the actual index holds all incoming candidates.
  • Before the change, only candidates with favoriteCount >= 1 published a "1fav" indexing event to Kafka.
  • After the change, only candidates with favoriteCount == 0 are instrumented with a new counter, 1fav_uec_count_race.
• buildMMEmbMetadataDump also lost its gate and fetches and dumps three versions of the multimodal embedding for every candidate.
  • Before the change, the same favoriteCount >= 1 gate meant that only candidates with favorites looked up the three versions (v3, v5_1, v6_dev) and dumped them as Kafka metadata.
• This is a new caveat. The canon's understanding from Gen3–Gen4 that "phoenix-rankall updates the index with a popularity signal" is no longer accurate from this commit. The index name does not imply a filter condition.
• Why it matters: This is an infrastructure expansion that lets Phoenix retrieval attempt embedding-based matching on long-tail posts too inferred. In the next commit we look at two things: the ratio at which the 1fav_uec_count_race counter is actually observed, and which param gates (Phoenix retrieval top-k, SID window) the index coverage expansion is turned on together with.

In the 2026-08-18 commit (Gen4+3), Phoenix postCreationEventForwarder was migrated to unified posts topic V3 verified — phoenix-rankall-strato/stream_forwarders/postCreationEventForwarder.strato.

• The same commit also changed phoenix-rankall-strato/columns/phoenix_rank_all/postCreationEventProcessor.strato.

• This migration moves the shared event stream of the content understanding domain from v1 to v3. It changed in two directions:
  • (a) It raised postId to the Kafka key, so partitioning, dedup, and inverted-index lookup can be purely key-based.
  • (b) It separated the Phoenix retrieval Kafka cluster from X main, so isolation and tuning can be done separately.
• Moving the import to hydra/sharedConfig is also a signal that the Phoenix family is being brought under hydra infrastructure.
• With this change, the upstream event path for maintaining the Phoenix retrieval index (build1FavIndex, buildMMEmbMetadataDump) fully moved from X main Kafka to the isolated Phoenix Kafka. It points the same direction as the "indexing gate removal" in Gen4+2. The direction fits if the indexing pipeline's throughput grew and a separate cluster became necessary inferred.
• Some things are also not public. They are whether a v2 existed (whether it jumped straight from v1 to v3, or this is a third migration going from v2 to v3), the scale of the Phoenix Kafka cluster and the reason for isolation, and when the v1 forwarder is finally shut down.

Training and serving infrastructure

The 2026-08-20 commit (Gen4+5) extracted the core of the embedding table loader into a function not tied to pyo3 verified. The 2026-08-21 commit (Gen4+6) released an option to return memory after hotswap and the interface of xai-o2, the internal object storage verified. The path that exports a trained ranking model as a StableHLO bundle for CUDA serving entered off by default in the Gen4+13 commit (2026-09-01, 7ba7768) verified.

In the 2026-08-20 commit (Gen4+5), emb_table.rs was split into a Python wrapper and a pyo3-free core verified — phoenix/crates/serving/xai-recsys-engine/src/emb_table.rs (+189/−169, essentially a behavior-preserving refactor).

The existing load_tensor was tied to pyo3 types and errors and could not be reused in non-Python contexts. This commit extracted its core into the function below, and the existing load_tensor was redefined as a pyo3 wrapper.

• load_tensor_into(&str, &str, &[(String, String, usize, usize)], &mut [u8], usize, usize) -> Result<(), String>

Safety was also reinforced.

• The tokio runtime's unwrap() was replaced with map_err.
• A prefix.len() < 3 guard (checkpoint path too short: {path}) was added.
• The modulo check was unified on the is_multiple_of API.

This is refactor preparation for the growing need to load large embedding tables (vocab 100M/100M/30M of Gen4+4 xrecsys_search) in multiple serving and training contexts inferred.

In the 2026-08-21 commit (Gen4+6), a malloc_trim(0) option was added to inference hotswap verified — phoenix/xrex/inference/model_runner.py.

• The same commit also changed launch_inference.py.

The new CLI option --hotswap_malloc_trim defaults to True. The _malloc_trim_after_swap() method calls libc malloc_trim(0) at the end of every hotswap cycle. The observability metric is checkpoint_reload_step_seconds.labels(step="malloc_trim").

The CLI help reads:

glibc malloc_trim(0) on the coordinator thread after each applied hotswap; returns the cycle's freed pages (glibc slots otherwise ratchet ~196 MiB/cycle). No-op under jemalloc.

Why it matters: This is the first documentation that embedding table hotswap in Phoenix serving has been leaking ~196MiB of RSS per cycle under glibc. Hotswap is long-running infrastructure in production, so the observation is that the accumulated burden was large enough to need this patch.

The same commit (Gen4+6) released the interface of the xai-o2 storage crate verified — phoenix/crates/storage/xai-o2/.

The description in Cargo.toml reads as follows:

Client interface for the xai-o2 (S3-compatible) object storage backend

The crate consists of 3 files:

• base_client.rs (12 lines) defines the BaseO2Client async trait, whose only method is put(key, data) -> Result<()>.
• In o2_client_builder.rs (115 lines), O2ClientBuilder provides 15 builder methods.
  • endpoint·bucket·prefix·access_key_id·secret_access_key·allow_anonymous·timeout·io_timeout·retry_max_times(6)·retry_min_delay(1s)·retry_max_delay(32s)·write_chunk_size_mib(8)·write_chunk_concurrency(16)·max_concurrent_write_requests(64)·trace_always
• lib.rs is 7 lines.

However, in the open-source build, build() always returns the error below.

pub const BACKEND_UNAVAILABLE: &str =
    "the xai-o2 backend is not included in this build; use the default object_store backend";

The xai-recsys-engine crate declares the dependency as xai-o2 = { workspace = true }, but in actual use it must fall back to the standard object_store crate.

Why it matters: This is the first confirmation that xAI has an internal S3-compatible object storage system named xai-o2. The 15 builder params are significant because they are items actually tuned in production. It signals that production checkpoint and data storage attach here inferred.

A new StableHLO bundle export path arrived in the same commit as SID recon mode (Gen4+13). It is off by default. verified — phoenix/xrex/train/recsys_bundle_export.py (634 new lines).

• In the same commit, phoenix/xrex/train/trainer_recsys.py changed by +53/−0.

The entry point build_bundle(trainer) lowers the ranking model's forward pass with jax.export. The only target is RecsysAggregatedModelConfig, and RecsysGenRecsModelConfig is explicitly unsupported. The export produces three outputs:

• StableHLO MLIR bytecode (.mlirbc) made with jit(...).lower().as_text(dialect="stablehlo")
• .jax_export made with jax_export.export(platforms=("cuda",)).serialize()
• MANIFEST.json

The bundle structure reveals features of the serving architecture:

• Embedding lookup is outside the exported graph. The forward input merged_embeddings is a tensor that concatenates already looked-up embedding rows.
  • MANIFEST.embedding holds {rows, rows_padded, num_shards, width, dtype, merged_slices, hash_table}, so the serving side knows how to concat the embedding server results.
  • The code reveals that the serving architecture of the X ranker is a split structure: "the embedding server does the lookup, and the GPU server runs StableHLO".
• There are multiple batch buckets. The default of export_bundle_bs_per_device is "1,2,4". A separate program is lowered per bucket, and at serving time the bucket matching the request batch size is chosen.
• The target is CUDA only. MANIFEST carries device_kind and compute_capability.
• seqpack supports only pallas_ranker_varlen_attn. The block-sparse layout of FA4 (cutedsl_ranker_varlen_attn) explicitly raises NotImplementedError.
  • Because the Gen4+6 GB300 config uses attn_impl="cutedsl_ranker_varlen_attn", exporting the GB300 training config as is requires switching the attn kernel to pallas observed. This limit is the break point in the pipeline from GB300 training to GB300 serving observed.
• Custom-call targets are recorded in the manifest. Because ranker-specific pallas/CUDA kernels are baked into the StableHLO, the serving loader must allow custom calls. The allowed range is widened with jax_export.DisabledSafetyCheck.custom_call(...).
• sid_num_levels is echoed into the manifest prep_spec. The serving side uses this value to know how many SID levels to put in a batch. This field pairs with SID recon mode.

Four new fields on Trainer turn on and tune the bundle export:

export_stablehlo_bundle: bool = False
export_bundle_bs_per_device: str = "1,2,4"
export_bundle_history_seq_len: int = 0
export_bundle_candidate_seq_len: int = 0

If export_stablehlo_bundle is on but checkpoint_config.copy_port is off, __post_init__ raises ValueError. In other words, export can be turned on only when attached to the copy_port of the checkpoint deployment pipeline (hot checkpoint deployment). With every copy_port deployment, a bundle is shipped to {OUT_PATH}/.{prefix}/export/ as well.

Why it matters

• X released for the first time code that standardizes the handoff from JAX training to CUDA serving as StableHLO. Until then the canon's understanding was "Phoenix is trained with JAX/Haiku and serving is a separate stack". With this commit, the code confirms that the serving stack takes and runs compiled StableHLO.
• The explicit separation of embeddings and dense forward is settled as an architecture fact. This accurately supports the canon's understanding that X does not hold the embedding table on a single GPU and looks it up from a separate server (the embedding server and mm_embedding_client.rs). An example of table scale is the vocab 100M/100M/30M of Gen4+4 xrecsys_search.
• It is off by default. This commit alone cannot confirm whether the production serving pipeline actually consumes StableHLO bundles. If export_stablehlo_bundle=True appears in a production config in the next commit, it will signal that serving has attached to this path.

Some things are also not yet public.

• Code of the serving loader (the runtime that consumes StableHLO)
• Detailed conventions for how the embedding server builds the merged_embeddings tensor. The merged_slices schema of MANIFEST.embedding is in the code, but the consumer-side code is not public.
• Whether this bundle is on in production

The Gen4+13 commit added an Orbax writer that encrypts and stores training checkpoints with envelope encryption. The default is off. verified — phoenix/python/training/xai-checkpointing/xai_checkpointing/orbax_encrypted.py

The linked orbax_encrypted.py is a new 288-line file, and phoenix/xrex/train/misc.py (+5/0) and phoenix/xrex/train/checkpoint_write.py (+6/0) also changed with it verified — same commit.

Envelope encryption encrypts data with a data key (DEK) and then encrypts that data key again with another key.

This writer wraps Orbax PyTreeCheckpointHandler and enforces the following:

• Every file in a checkpoint must start with a magic header. If even one file lacks the header, the commit is refused verified — same commit.

  • The magic header value is b"XAIENC01".
  • After finalize, assert_all_enveloped(path) walks the checkpoint directory. If it finds a file without a header, it raises RuntimeError("encrypted save left plaintext at rest; not committing: ...").
  • _DEK and _DEK.claim (the DEK file issued by KMS and its claim) are exceptions.

• Leaves allow only jax.Array; string, dict, and Python scalar leaves are explicitly rejected verified — same commit.

  • The guards are _ArraysOnlyRegistry and _require_array_leaves.
  • This guard removes the arbitrary code execution risk when the serving side loads this checkpoint.

• The tensorstore KvStore is also replaced with an encrypted wrapper verified — same commit.

  • The option is use_encrypted_kvstore.

• Metadata is also encrypted verified — same commit.

  • _METADATA, _CHECKPOINT_METADATA, and the sharding metadata are all written with encrypt_write.

• A process that restored from an encrypted checkpoint cannot save in plaintext (no downgrade) verified — same commit.

  • The new guard is in checkpoint_write.py.
  • Attempting a plaintext save raises ValueError("restored from an ENCRYPTED checkpoint but the orbax writer cannot produce encrypted saves — refusing to downgrade to plaintext saves").

• CheckpointConfig gained new encryption setting fields verified — same commit.

  encrypt: bool = False
  encryption_key_id: str | None = None
  encryption_context: dict[str, str] = field(default_factory=dict)
  encryption_chunk_size_bytes: int = 8 * 1024 * 1024
  

  • The encryption_context field follows the AWS-KMS encryption-context pattern as is.
  • The default chunk size is 8 MiB.

Why it matters

• A writer that encrypts training checkpoints at rest entered the code verified. We infer that X uses this writer in production inferred. The basis is that it even includes devices aimed at operational accidents, such as the downgrade-prevention guard.
  • Previously this could only be guessed to be handled at the infrastructure layer (not confirmed in the canon).
  • Now envelope encryption, KMS, and magic header enforcement are in-tree in the xrex training Python code.
  • The source did not state a regulatory framework (SOC2, FedRAMP, GDPR, etc.). It reads as general "data at rest" protection.
• CheckpointConfig.encrypt defaults to False, and as of the 2026-10-17 commit (862a123) none of the public training configs turns it on verified — phoenix/xrex/train/misc.py:56. So code alone cannot confirm whether production actually uses this writer.
• The downgrade-prevention guard stands out in particular. It blocks accidentally losing the encrypted state at the code level.
  • It points the same direction as the VF reference_compare harness in Gen4+7, which panics when booted with APP_ENV=prod and so enforced staging-only use.
  • In both cases the policy is fixed in code as a fail-close assertion, not as a config value.

The following is not public:

• The actual implementation of kms_client (within the file it is passed in from outside)
• DEK rotation period
• Actual values used for the encryption context
• Which KMS is used (AWS, Azure, or in-house)

↑ Contents

Grox (LLM annotation) — unchanged from Gen3 (+ threshold raise on 2026-08-14)

Grox is a Grok-based nearline annotation pipeline. The follower threshold that splits reply spam verdicts from reply ranking scoring targets has been 250k since the 2026-09-24 commit (Gen4+28). Replies whose thread root and reply target are both at or below this value receive a spam verdict, and replies where either exceeds it receive reply ranking scoring verified. The threshold started at 15k in the Gen4 release and was raised eleven times; each raise widened the spam verdict targets and narrowed the reply ranking scoring targets verified. The separate spam detection flow was absorbed into the reply ranking flow in the 2026-08-18 commit (Gen4+3) verified. Both thresholds changed after that, and the changed values are in the change log; see the threshold history summary in the commit that raised the coordinated spam threshold from 10k to 30k and the commit that split the reply spam model by thread followers verified.

grox/ produces the slop score, safety labels, topics, reply scores, and multimodal embeddings. In Gen4, grox/ was explicitly placed as Content Understanding stage 1 of the upstream labeling pipeline (safety section). Details are in llm-in-recsys.md.

The 2026-08-14 commit doubled the follower thresholds for reply spam verdicts and reply ranking scoring verified — grox/flows/reply_spam/task_filter.py:17,185.

• The two threshold constants in grox/flows/reply_spam/task_filter.py changed from 15000 → 30000.
  • TaskSpamFilter.FOLLOWER_COUNT_THRESHOLD_FOR_SPAM_DETECTION: 15,000 → 30,000
  • TaskReplyRankingFilter.FOLLOWER_COUNT_THRESHOLD_FOR_REPLY_RANKING: 15,000 → 30,000

Meaning: replies whose thread root and reply target both have 30k followers or fewer receive a grok spam verdict, and replies where either exceeds 30k receive reply ranking scoring verified — task_filter.py:77-92, 214-225. Doubling the threshold widened the spam verdict targets and narrowed the reply ranking scoring targets. Because the original code constant names carry "spam detection" and "reply ranking", this adjustment is a resource allocation adjustment that decides which replies get grok resources.

The 2026-08-18 commit (Gen4+3, 11a71f8) raised both thresholds again and merged the separate spam detection flow into the reply ranking flow verified — grox/flows/reply_spam/.

• Both constants went up from 40,000 → 60,000 verified — same commit. This is the third raise, following 30,000 → 40,000 in the 2026-08-17 commit (Gen4+2) (task_filter.py:17, 185 in b089ce6).
  • Only replies whose root and reply target are both at or below 60k become grok spam verdict targets, and the other replies receive reply ranking scoring.
  • The boundary between spam verdicts and reply ranking scoring moved from 40k to 60k.
• The separate task generator that detected spam over the whole post stream was deleted, and spam detection was absorbed into the reply ranking flow verified — same commit.
  • The deleted generator is PostStreamTaskGenerator. The TOPIC_UNIFIED_POSTS constant (the original, not v3) was also deleted.
  • The remaining ReplyRankingTaskGenerator injects the three plans below together and consumes from the single source TOPIC_UNIFIED_POSTS_V3.
    • PlanReplyRanking (existing)
    • PlanSpamComment (newly absorbed)
    • PlanCoordinatedSpam (newly absorbed)
• Meaning: the grox spam verdict pipeline was consolidated with a wider range of verdict target accounts (60k and below) and a single flow verified — same commit. The single flow refers to the merge with reply ranking.
  • This change came in the same commit as the Phoenix unified_posts_v3 topic migration (Phoenix training and index section). So the observation follows that grox and Phoenix both moved to the v3 content understanding stream at the same time.

The Gen4+6 commit (2026-08-21, 28e414f) raised both constants from 60,000 → 80,000 verified — grox/flows/reply_spam/task_filter.py.

• This is the fourth raise in 3 days, with a step of 1.33×.
• The threshold went up four times in the 8 days after the Gen4 release (15k → 30k → 40k → 60k → 80k).
• After the first raise (2×), the step shrank to 1.33–1.5×. This is one of the following: nearing the grok budget bottleneck, diminishing marginal utility because few spam replies come from accounts above that size, or preparation to move to full on with no threshold in the next stage inferred.

This section's change log is below. The values are the threshold shared by TaskSpamFilter and TaskReplyRankingFilter, confirmed in each commit's task_filter.py and constants.py verified.

• 2026-08-13(Gen4 release): 15k
• 2026-08-14(Gen4+1): 30k
• 2026-08-17(Gen4+2): 40k
• 2026-08-18(Gen4+3): 60k
• 2026-08-21(Gen4+6): 80k
• 2026-08-24(Gen4+7): 100k
• 2026-08-25(Gen4+8): 120k
• 2026-09-08(Gen4+17): 150k
• 2026-09-16(Gen4+22): 180k. The constant was renamed to GROK_GEMMA_FOLLOWER_SPLIT.
• 2026-09-17(Gen4+23): 200k
• 2026-09-23(Gen4+27): 225k
• 2026-09-24(Gen4+28): 250k. It is still 250k in the 2026-10-17 commit (862a123).

↑ Contents

Grox → PtOS Adult Content Cross-Validation (Gen4+2 new, 2026-08-17)

The judge in the grox safety verdict is a layer that has a Grok model judge the first-pass verdict once more. The adult content judge was introduced in the 2026-08-17 commit (Gen4+2), and the judge class (cross-validator) that also covers child safety, violent media, and illegal and regulated behaviors was introduced in the 2026-08-24 commit (Gen4+7) verified (verdict flow, model history). As of the commit that introduced each (Gen4+2 and Gen4+7), the two judges only keep or demote the first-pass verdict and never make it stricter verified (verdict flow, model history). Later, cross-validation stages were also added to the high-popularity video path and the leading-frames rescan path, and the judge model for illegal and regulated behaviors was changed verified (commit that created the high-popularity video path, Gen4+15 B, Gen4+17).

The judge model column of the table is as of the commit listed in that column, and the other columns are as of the commit that introduced the judge class.

• Since the commit that split the judge model by category (Gen4+15), the illegal and regulated behaviors judge uses the model in the table instead of EAPI_GROK_4_5_X_ALGO, and it calls this model directly without a circuit breaker verified (Gen4+15 B in the change log).
• The high-popularity video path created by the commit that built the special video pipeline based on motion-reveal also has an adult content cross-validation stage verified (commit that created this path).
  • As of that commit, only this path forces a demotion to Soft instead of keeping the original verdict when the judge call fails with an exception (other paths keep the original verdict by default) verified — same subsection.
• The adult content leading-frames rescan path introduced in the Gen4+17 commit also has a cross-validation stage verified (Gen4+17 in the change log).
  • As of Gen4+17, this path cuts only the first 10 seconds of videos that are 30 seconds or longer and re-judges them verified — same subsection.
  • As of Gen4+17, the entry targets are some of the videos whose fav count is at or above the deluxe threshold and that already received a PTOS verdict but were not judged NSFW verified — same subsection.
  • In the commit that brought broadcast into the PTOS safety pipeline (Gen4+18), this path lost the deluxe fav gate, and it is entered whenever a broadcast exists even without a video of 30 seconds or longer verified (Gen4+18 in the change log).
  • The Gen4+17 subsection treats this path as a second judge that can reverse a demoted verdict again.

Verdict flow of the adult content judge

A judge layer was added to adult content verdicts to handle verdict disagreement. The grox safety pipeline first released in Gen4 runs the two verdicts in parallel and only logged the result when they differed.

• The two verdicts are TaskSafetyPtosSafemodelSexNudity (a sex/nudity verdict based on an in-house vision model) and PtOS (a policy-based violation category verdict).
• In the 2026-08-17 commit (Gen4+2), this logic was split into two files, and a new task was added in which a Grok 4.5 judge re-evaluates when the two verdicts disagree verified — grox/flows/ptos/task_safety_ptos_adult_content_cross_validation.py (162 lines, new)
  • The safemodel task was moved to grox/flows/ptos/task_safety_ptos_safemodel_sex_nudity.py (32 lines moved).
  • The judge class was newly created at grox/flows/ptos/classifier.py:597-608.

The judge task works in four stages:

1. Compare the two verdicts. The enum CompareOutcome distinguishes BOTH_POSITIVE, BOTH_NEGATIVE, SAFEMODEL_ONLY_POSITIVE, and PTOS_ONLY_POSITIVE. is_disagreement is true for SAFEMODEL_ONLY or PTOS_ONLY.
2. Check two gates.
   • It runs only in the deluxe flow, and the regular flow only logs. The condition is is_deluxe = ctx.payload.task_type == SAFETY_PTOS_DELUXE.
   • It skips posts that are already NSFW-flagged. The flag is looked up with StratoSafetyPostAnnotationsResultDirectMh.
3. Call the Grok 4.5 judge. The new class SafetyPtosAdultContentCrossValidationJudge uses ModelName.EAPI_GROK_4_5_X_ALGO.
   • The dedicated circuit breaker has a 50% failure rate, a 600-second window, 600 seconds of recovery, and 5 half-open calls.
   • The system prompt is _strip_thinking_restrictions(adult_content_policy_prompt()). The user message is built with UserRenderer.render(user) and PostRenderer.render(post, include_reply_to=True).
   • The response is parsed with re.compile(r"(.*)<json>(.*)</json>", re.DOTALL) to separate the free-text reasoning from the JSON verdict.
4. Apply the judge result. If the judge returns AdultContentSexualHard, the original verdict is kept. For any other result, it does three things.
   • It demotes the safetyPolicy.policyType of every AdultContent violation to AdultContentSexualSoft.
   • It writes reason = "Grok 4.5 Cross Validation disagreed" on the demoted violation and leaves it in the audit log.
   • It sets safemodel_sex_nudity.positive to false.

Four points in this design are worth noting:

• There is only a demotion direction (hard → soft) and no escalation direction.
  • This judge layer works only in the direction of easing over-gating.
  • The code shows a policy direction aimed at reducing false positives rather than false negatives.
• For a reply, the parent post is also put in as context, so the context "this is a reaction to that post" influences the verdict (include_reply_to=True).
• In the judge context, the "do not expose reasoning" caution constraint in the original prompt is stripped by _strip_thinking_restrictions() so the reasoning can be seen (for audit purposes).
• The verdict layers are now three: safemodel, PtOS, and the Grok 4.5 judge. This matches the direction of Gen4's "release of the whole labeling pipeline" trend.

Three questions need a deep dive:

• Whether this judge layer expands to other categories (violence, hate, spam)
• How much Grok 4.5 inference cost can be absorbed
• How the EAPI_GROK_4_5_X_ALGO model differs from the existing EAPI_GROK_4_5_INTERNAL. Presumably the difference is x-algorithm-specific tuning versus a general internal judge.

The 2026-08-18 commit (Gen4+3) improved the logging of this task.

• The outcome attribute of Metrics.counter(metric) changed from a binary "hard"|"soft" label to the raw policyType.value of the verdict verified — grox/flows/ptos/task_safety_ptos_adult_content_cross_validation.py:128-131
  • Even if the judge returns a policyType other than hard/soft (for example, a finer adult category split), it can now be observed without loss.
  • This is an adjustment to instrumentation granularity, not a change to the verdict logic itself.

Judge models and judged categories over time

With the changes through Gen4+7, X settled on a dual stack that keeps the primary and the judge on different model generations and tunings. The two call sites in the PTOS classifier that called Grok 4.5 internal (the child safety judge and the high-fav deluxe reclassification) moved to Grok 4.6 internal across three commits from 2026-08-19 to 08-21 verified (classifier in 28e414f). Three days later, in the 2026-08-24 commit (Gen4+7), the new judge class also came to use the 4.5 X_ALGO endpoint that the Gen4+2 adult content judge used, and the scope widened to 3 categories verified (classifier in d011592).

From Gen4+4 to +6, the two call sites that used 4.5 internal moved to 4.6 internal, and in Gen4+7 the child safety judge moved to the new cross-validator class and came to use 4.5 X_ALGO. The adult content judge has used 4.5 X_ALGO unchanged since Gen4+2 verified. The flow by commit:

• Gen4+2 (b089ce6, 08-17): The adult content judge SafetyPtosAdultContentCrossValidationJudge was introduced (when safemodel and PtOS disagree). Its model is EAPI_GROK_4_5_X_ALGO, and this endpoint and its dedicated circuit breaker were also introduced in this commit verified — classifier.py:85, 100, 561.
• Gen4+3 (11a71f8, 08-18): Refined the metric outcome of the judge above.
• Gen4+4 (aad7179, 08-19): Attached Grok 4.6 internal with a 10% dial (two sites: high-fav deluxe reclassification and the child safety judge).
• Gen4+5 (d0cef2f, 08-20): Raised the 4.6 internal dial from 10% to 30%.
• Gen4+6 (28e414f, 08-21): Switched fully to 4.6 internal at 100%, and 4.5 internal was removed from the code entirely.
• Gen4+7 (d011592, 08-24): The new cross-validator class also came to use EAPI_GROK_4_5_X_ALGO, which the adult content judge had used since Gen4+2, and the scope widened to 3 categories (child safety, violent media, illegal).
  • The adult content judge is still kept as a separate class (SafetyPtosAdultContentCrossValidationJudge).

If the judge were the same model as the primary, it would amplify the same mistakes, so the rationale of this design is to let the view of a separate model only demote.

The 2026-08-19 commit (Gen4+4) attached Grok 4.6 internal to 10% of traffic and added a cache to skip NSFW reclassification. verified

The verdict stack was refined in two directions.

• First, it introduced Grok 4.6 to the child safety judge and the high-fav deluxe reclassification verified — grox/flows/ptos/classifier.py (+66/−7)

  • It registered the new model name ModelName.EAPI_GROK_4_6_INTERNAL and attached an EapiSampler instance and a dedicated circuit breaker (_eapi_4_6_internal_breaker).

  • The circuit breaker parameters are the same as 4.5 internal: a 50% failure rate, a 600-second window, 600 seconds of recovery, and 5 half-open calls.

  • Both call sites pick the model probabilistically.

    _GROK_4_6_INTERNAL_DIAL = 0.1
    ...
    if random.random() < _GROK_4_6_INTERNAL_DIAL:
        async with _eapi_4_6_internal_breaker.guard():
            raw = await self.eapi_4_6_internal.sample(...)
    else:
        async with _eapi_4_5_internal_breaker.guard():
            raw = await self.eapi_4_5_internal.sample(...)
    

  • The first site is the existing function _cross_model_validate_with_4_5, which handles cross-model validation for the child safety category. The metric name changed from safety_ptos.cross_model_validate_with_grok_4_5 to safety_ptos.child_safety_cross_model_validate_with_grok_4_5.

  • The second site is the deluxe 4.3 category reclassification inside classify_policy_for_violation. When fav_count >= 1024, it uses deluxe-4.6-internal with 10% probability instead of deluxe-4.5-internal.

    • This site is not a judge; it is a path where the first-pass classification of high-fav posts is done with this model instead of the regular sampler verified — classifier.py:461-489.

  • It is an exclusive traffic split that calls 4.6 with 10% probability per request and 4.5 for the rest verified — the code above. It is not dual-write or shadow, which would send the same request to both models.

  • The adult content judge introduced in Gen4+2 is not a target of this dial.

• Second, post_is_already_flagged_nsfw, which looks up whether a post is already flagged NSFW, was split out into a new module verified — grox/flows/ptos/prior_nsfw.py (34 lines, new)
  • The new module prior_nsfw.py checks with StratoSafetyPostAnnotationsResultDirectMh (which looks up safety annotation results from X-internal Strato http/Manhattan).
  • It caches the result in state.prior_nsfw so it does not look up again within the same task context.
  • If the lookup fails, it increments the counter safety_ptos.prior_nsfw_lookup_error.count and falls back to not-flagged.
  • This appears to be a refactor of what the Gen4+2 cross-validation task used to look up into a separate module inferred.
  • The sync_task_kwargs of the cross-validation task was changed to import post_is_already_flagged_nsfw.

Why it matters: The PTOS verdict layer is being organized around a traffic split for the 4.5 → 4.6 model switch and a separate cache for the prior classification lookup. Across the three commits Gen4+2 → +3 → +4, this verdict stack is seen being refined daily.

The 2026-08-20 commit (Gen4+5) raised the Grok 4.6 internal dial from 0.1 to 0.3 (3× in one day). verified — grox/flows/ptos/classifier.py:114

-_GROK_4_6_INTERNAL_DIAL = 0.1
+_GROK_4_6_INTERNAL_DIAL = 0.3

• The _EAPI_4_6_INTERNAL_BREAKER_CONFIG circuit breaker parameters and the places it applies (child safety cross-model validation, deluxe 4.3 category reclassification) are unchanged.
• Meaning: this promotion supports that 4.6 traffic did not trip the circuit breaker and that the verdict disagreement rate was within the acceptable range inferred. Otherwise a 3× dial promotion would not be possible.
• Introduction of the Gen4+2 adult content judge → +3 logging refinement → +4 4.6 dial 10% → +5 dial 30%: the PTOS verdict stack changed every day. The 4.6 dial applies to the child safety judge and the deluxe reclassification, not to the Gen4+2 judge.

The 2026-08-21 commit (Gen4+6) removed Grok 4.5 internal completely and switched to 4.6 internal at 100%. verified — grox/flows/ptos/classifier.py (+6/−63)

The Gen4+5 watch point ("if it jumps to 50%/100% in the next commit, a full 4.6 switch is near") was resolved at 100% in one day. Seven items were deleted:

• _EAPI_4_5_INTERNAL_BREAKER_CONFIG
• _eapi_4_5_internal_breaker (breaker instance)
• The _GROK_4_6_INTERNAL_DIAL constant (0.3 a day earlier)
• self.eapi_4_5_internal (initialization removed from two classes)
• _sample_4_5_internal method
• The probabilistic branch in _cross_model_validate_with_4_5 (now always calls 4.6 internal directly)
• The probabilistic branch for the deluxe 4.3 reclassification in classify_policy_for_violation (now always deluxe-4.6-internal)

Why it matters: The 4.6 internal dial-up (10%→30%→100%) that started in the 2026-08-19 commit (Gen4+4) finished in two days. Grok 4.5 internal is gone from both the classifier code and the ModelName list in the grox config, so it cannot be rolled back verified — grox/config/config.py:82-93. The child safety judge and the deluxe reclassification are unified on Grok 4.6 internal, and the adult content judge continues to use EAPI_GROK_4_5_X_ALGO. The watch point moves from 4.5→4.6 to the timing of the 4.6→4.7 switch.

Two other places in grox changed in the 2026-08-20 commit (Gen4+5), one day earlier verified — the diff between aad7179 and d0cef2f.

• A stop_predicate callback and aclosing were added to grok_sampler verified — grox/libs/grok_sampler/llm.py (+27/−13)
  • It wraps the raw stream in an async with aclosing(...) context, which guarantees cleanup even on early termination.
  • After yielding each token, it checks stop_predicate(resp), and if True, it increments the llm.sample.early_stop.count counter and breaks.
  • When a safety verdict response arrives in <json>...</json> tags, the tokens after the tag closes are meaningless. Early termination reduces token cost and latency.
  • This is a sampling infrastructure optimization matched to the growth in 4.6 traffic inferred.
• A profile_image field was added to the grox User schema verified — grox/core/data_loaders/data_types.py
  • The mapping is in grox/core/data_loaders/post_mapper.py. If AuthorMetadata.profileImageUrl exists, it fills in Image(url=...).
  • The Grok classifier can now reference the profile image as verdict context. For example, the profile image also becomes input for judgment in ads brand safety or the adult content judge.
  • This is a step that prepares the upstream data connection inferred.

The 2026-08-24 commit (Gen4+7) created SafetyPtosPolicyCrossValidator and also used, for this judge, the Grok 4.5 X_ALGO endpoint that had existed since Gen4+2. verified — grox/flows/ptos/classifier.py (+223/−52)

Three days after Gen4+6 removed Grok 4.5 internal completely, the 4.5 X_ALGO endpoint was also used for the new verdict judge (cross-validator). The scope widened to 3 categories.

• The endpoint is ModelName.EAPI_GROK_4_5_X_ALGO, and it uses the circuit breaker _eapi_4_5_x_algo_breaker (window 600 seconds, recovery 600 seconds, half-open 5 calls) verified — same commit. Both were already created in Gen4+2 (b089ce6) for the adult content judge.
  • It is a separate model from EAPI_GROK_4_5_INTERNAL, which was deleted completely in Gen4+6.
  • The X_ALGO suffix in the name suggests an x-algorithm-specific tuned variant inferred.
• The entry point validate(category, post, policy) of the new class SafetyPtosPolicyCrossValidator routes the three categories to dedicated methods verified — same commit.
  • ChildSafety (_validate_child_safety) is rechecked with 4.5. If the policy type matches, it is kept; if not, it is only demoted to NoViolation. Parse failures and errors also fall back to NoViolation. This absorbs the Gen4+2 child safety judge into this class.
  • ViolentMedia (_validate_violent_media) is rechecked with 4.5. If it matches, it is kept; if the judge decides NoViolation, it is demoted; if type_mismatch, it is downgraded to ViolentMediaGraphicMedia (the weaker form). The demotion options were refined into three: NoViolation, GraphicMedia, and keep.
  • IllegalAndRegulatedBehaviors (_validate_illegal_and_regulated_behaviors) is rechecked with 4.5. If it matches, the policy type is kept but the reason is overwritten with the one 4.5 gave; if not, it is demoted to NoViolation.
• All three categories build a standard prompt with _build_policy_convo and apply _strip_thinking_restrictions (reasoning enabled) verified — same commit.
  • The prompts are child_safety_policy_prompt(post_creation_time), violent_media_policy_prompt(), and illegal_and_regulated_behaviors_policy_prompt().
• It keeps the demotion-only principle verified — same commit. This is the same design philosophy as the Gen4+2 adult content judge.
  • The cross-validator cannot produce a stricter verdict than the primary.
  • The judge does not newly catch harmful content the primary missed; it works only in the direction of releasing what the primary falsely flagged.
• The metrics are counters safety_ptos.<category>_cross_model_validate_with_grok_4_5 for each of the three categories verified — same commit.
  • The counters carry an outcome tag (agreed/disagreed/unparseable/no_violation/type_mismatch/error).

In the same commit, the classifier turned on reasoning only for the two categories ViolentSpeech and SuicideOrSelfHarm, and made it include the entire thread when a parent thread exists verified.

• SafetyPtosPolicyClassifier.build_convo() applies _strip_thinking_restrictions to the prompt only for these two categories (reasoning enabled) verified — same commit.
  • Here the assistant message is left empty so the model can think freely.
  • The set is USE_REASONING_CATEGORIES = {ViolentSpeech, SuicideOrSelfHarm}.
  • Other categories still suppress thinking with THINKING_CONTROL_END (fast and cheap responses).
• Only for the same two categories, if the post has ancestors, the whole parent thread is rendered as the user message with ThreadRenderer verified — same commit.
  • The set is USE_THREAD_RENDERER_CATEGORIES = {ViolentSpeech, SuicideOrSelfHarm}.
  • The reason is fundamental. A sentence like "I want to die" cannot be judged without the thread context. If the earlier remark was about losing a game, it is harmless; if it is a self-harm support thread, it is serious.
  • Likewise for violent speech, the quote and reply context decide the verdict.
  • For other categories (adult, NSFW media, illegal), the target post itself carries the signal, so thread expansion is not needed.

Four things are not public as of the Gen4+7 commit:

• Whether EAPI_GROK_4_5_X_ALGO is the 4.5 base plus x-algorithm domain fine-tuning or a completely separate model
• The A/B results of the three-category expansion
• The next expansion candidates (spam, hate speech)
• The relationship between SafetyPtosAdultContentCrossValidationJudge and the new SafetyPtosPolicyCrossValidator (whether they will merge or be kept in parallel)
  • The two classes have the same function signature (judge vs validate) but different routing approaches.

Changes to the grox reply spam path and LLM input

The grox reply spam pipeline has had a path that writes the reply ranking score back to Strato since the Gen4 release (2026-08-13), and the 2026-08-24 commit (Gen4+7) added regional cache fanout to this path verified(storage module at 47c1bcd, storage module at d011592). As of the 2026-08-28 commit (Gen4+10), grox skips the Manhattan write when the new reply ranking score is not lower than the existing score verified(write task at 24c6094). In other words, the reply ranking score sent out on this path only goes down over time and never goes up verified(write task at 45b48ba).

The 2026-08-24 commit (Gen4+7) also changed three other places in grox.

• The follower threshold of the reply spam filter and the reply ranking filter rose from 80k to 100k (grox/flows/reply_spam/task_filter.py:17).
  • The two filters are TaskSpamFilter and TaskReplyRankingFilter in grox/flows/reply_spam/task_filter.py.
  • The threshold rose five times in 11 days (15k → 30k → 40k → 60k → 80k → 100k), so the set of accounts covered by spam verdicts kept widening.
  • The increases after Gen4+3 are 1.25–1.5×. The higher the threshold, the more threads of accounts with more followers grok checks with grok, so we infer this adjustment aims to check more spam replies in threads of large accounts inferred.
• The reply ranking score storage module now also writes the score to two regional caches verified — grox/flows/reply_spam/strato_loader.py (+17/−0).
  • This file and the ReplyRankingScoreStratoLoader class have existed since the Gen4 release (32 lines) verified — same file at 47c1bcd.
  • Since then it has stored the score with StratoReplyRankingScore.put() and emitted an event with StratoReplyRankingScoreV2Kafka.insert().
  • Gen4+7 added one more stage after put(). If score ≤ 1.0, it writes to two regional caches (StratoReplyRankingScoreCacheAtla, StratoReplyRankingScoreCachePdxa) in parallel.
  • If the score exceeds 1.0, the cache fanout is skipped verified — strato_loader.py:20, 34-41. Low-confidence verdicts stay only in the canon (strato), and the serving cache is not polluted inferred.
• X Spaces (audio) context entered the LLM verdict. The Grok safety judge recognizes posts with an audio space card attached for the first time.
  • A SpaceMetadata class was added to grox/core/data_loaders/data_types.py (a single title as of this commit). A space_metadata: SpaceMetadata | None field was added to Post.
  • post_mapper.py maps it from StratoSpaceMetadata.
  • The grox/core/lm/post.py renderer inserts it into the LLM prompt as \n\nThis post has the following audio space card attached:\nAudio Space title: {title}.
  • It does not listen to the audio content itself, but the space title text becomes input to the verdict. For example, if the topic stated in the space title is adult content or violence, it is reflected in the brand safety verdict.

Gen4+9 (2026-08-26, 45b48ba) added an option to put the follower count and bio into the reply_spam LM prompt and put a monotone-only gate on the Manhattan write. verified

• The user metadata going into the LM verdict was widened verified — grox/core/lm/post.py (+18/−6), grox/core/lm/thread.py (+25/−4)

  • The previous PostRenderer.render extracted user metadata as a single line, Metadata: <name> @<handle>. It now takes two options, include_follower_count and include_bio, and renders it split as follows:

    User Handle: @<handle>
    [User Follower Count: <N>]     # include_follower_count=True
    User Name: "<name>"
    [User Bio: "<bio>"]              # include_bio=True
    

  • When ThreadRenderer._render_thread receives the include_follower_count argument, it renders the follower count of every post in the thread. It renders the bio only for post 0 (the original post, the thread root) (include_bio=include_follower_count and post_idx == 0).

  • The follower count is also included for the reply under evaluation (the evaluate target).

  • A header # Reply Author Info\n\nThe user profile and any signals below describe the author of the reply being evaluated (the final post of the thread).\n is prepended to the message. This header tells the LLM explicitly that "this information is about the author of the reply being evaluated."

    • Below the header, UserRenderer.render(post.user) separately renders the name, handle, and bio of the reply author verified — thread.py:25-27, user.py:18-24. So the bios of two people are included: the author of the original post (post 0) and the author of the reply under evaluation.

  • Implications:

    • The follower count was already used as the grox reply_spam gating threshold (100k in Gen4+7), and now the LM verdict itself also references this signal directly.
    • Bio is strong evidence for spam and bot detection, so exposing the profiles of both the original post author and the reply-under-eval author together is expected to raise confidence.
    • The bio content that serves as evidence includes spam hints such as "link in profile", affiliation with political or commercial organizations, and interests.
    • Bios of other thread participants are not included (to prevent context explosion), and only the two authors' profiles are expanded.
• The reply ranking score sent to Manhattan only goes down over time and never goes up verified — grox/flows/reply_spam/task_write.py (+22)
  • Inside _publish_to_reply_ranking_manhattan, it looks up the existing score with ReplyRankingScoreStratoLoader.fetch_reply_ranking_score(post.id).
  • If the new score is greater than the existing one, it skips the write and increments Metrics.counter("task.write_reply_ranking_manhattan.skipped.count", reason="higher_than_existing"). If it is lower, it writes as is.
  • Implications:
    • If a reply's first verdict comes out low (close to spam), that reply keeps a low score. Even if a new verdict becomes lenient, the serving score is not updated.
    • Conversely, a reply whose first verdict comes out high (lenient) is lowered if a later verdict becomes stricter.
    • This is consistent with the monotone safety direction of the reply spam verdict (once caught as spam, always treated as spam; only normal → spam reclassification is allowed).
    • It eliminates label flapping of the reply ranking score along the time axis at the source, so the ranking of the same reply does not go up and down in the UI or in A/B tests inferred.
    • Label flapping was likely actually observed in the reply ranking channel inferred.

Gen4+10 (2026-08-28, 24c6094) added a Kafka-first storage order and tie skipping, and made the follower count always injected into the coordinated spam verdict. verified

_publish_to_reply_ranking_manhattan was tightened in three ways on top of the monotone-only logic of Gen4+9 verified — grox/flows/reply_spam/task_write.py (+15/−21)

1. Ties are not written either. The comparison operator changed from > to >=, so it skips if score >= existing.score.
   • Previously, when the new score equaled the existing one, it was written anyway (idempotent write).
   • When exactly the same score is recomputed, it saves the Manhattan put, and no Kafka event is emitted either.
   • The counter name also changed from higher_than_existing to not_lower_than_existing to make the semantics clear.
2. The Kafka event moved ahead of the Manhattan write. The previous order was save_reply_ranking_score(Manhattan) → save_reply_ranking_kafka_v2(Kafka), and the new order is the reverse: Kafka publish first, Manhattan write second.
   • Reason inferred: Kafka is append-only and can be retried on failure, whereas Manhattan can hit the monotone-only logic and get a not_lower_than_existing verdict, so the retry itself may be skipped. Manhattan is also an upsert with side effects, so it is heavier than a Kafka publish. If Kafka succeeds first, the offline reprocessing pipeline can later recover a Manhattan write failure; if Manhattan succeeds first and the Kafka publish fails, downstream (serving cache warm-back, other consumers) cannot see the new score already reflected in Manhattan. In other words, Kafka-first is the durability order that guarantees consistent observability across the whole system.
3. The try/except around fetch_reply_ranking_score was removed. Previously, if the fetch failed, it logged a warning and proceeded with existing = None, so it always wrote. Now it raises the exception upward as is, so the flow task is marked as failed and retried.
   • This blocks at the source the incident where a fetch fails silently, the monotone comparison is nullified, and an unexpected write goes through.
   • Defensive exception handling was removed, prioritizing logic correctness.

In the same commit, the follower count option of Gen4+9 was adopted as always on in the coordinated spam verdict verified — grox/flows/reply_spam/classifier_coordinated_spam.py (+5/−1)

• An argument was added to _to_convo in classifier_coordinated_spam.py. It now calls PostRenderer.render(p, max_media=MAX_MEDIA_PER_POST) with include_follower_count=True.
• The coordinated spam verdict receives the follower count of each participating account in the thread in the prompt. Bio keeps the include_bio=False default, and only the follower count is included.

Why it matters: The two axes Gen4+9 introduced changed in Gen4+10 as follows. The LM prompt user metadata expansion became always on. Manhattan monotone-only gained consistency through stronger idempotency and the Kafka-first order. The durability contract of the reply_spam verdict stack became much more explicit.

↑ Contents

BDSM (abusive account detection from account behavior sequences) — enforcement note generation logic partially released in Gen4+1

BDSM is an account scorer that extracts abuse signals from an account's time-series behavior sequence. As of the c65aa17 commit (Gen4+1), the enforcement note generation logic is partially public verified(commit summary table). The logic for the conditions to create a note and for choosing the bot heads to put in the note is public, while the template string and the tuned verdict thresholds are not public verified(resolution record). The note carries a score for which bot family was triggered and how strongly, but the text explaining which action pattern caused it is not public verified(BDSM section text).

From Gen3 through early Gen4, build_enforcement_note at bdsm/runtime/score_results_sink_focal.py:78 was a stub that only did return None. The Gen4+1 commit released the gate logic and the dominant-head selection logic, leaving only the template string as a "<redacted>" sentinel verified — bdsm/runtime/score_results_sink_focal.py:78-128, bdsm/README.md, bdsm/proto/abuse_inference.proto:98-102.

• A note is created only for a sequence with 30 or more actions in which some head other than LegitimateUser has a score above 0.5 verified — the three files in the previous paragraph.
  • The action count condition is MIN_ACTIONS = 30, total_actions >= 30, and the score condition is score > 0.5.
• If the conditions pass, the head with the highest score is chosen as dominant, and the note string carries the names and scores of up to the top 3 heads verified — the three files in the previous paragraph.
  • The note string has the form "<redacted> [model: FollowBot=0.91, also: ReplySpamBot=0.60, LikeBot=0.53]".
  • Which bot family was triggered and how strongly goes out in the note as is, but the text explaining which action pattern triggered it is not public.
• Seven bot heads were officially released verified — the three files in the previous paragraph.
  • The names are FollowBot, LikeBot, EngagementAmplifier, ReplySpamBot, TweetSpamBot, RTBot, and MultiActionBot.
• The 9.99 filling the operating point (verdict threshold) in sink_policy.yaml is a sentinel, and the README states this explicitly verified — bdsm/runtime/sink_policy.yaml:5-13.
  • The operating point is a probability in the range [0, 1], so when it is filled with 9.99 it never triggers.
  • This means the gating structure and head names are real, and the tuned real numbers were not released, to prevent gaming.
• Note that the head score carried in the note is a bot probability, not a "report reviewer decision" verified — the three files in the previous paragraph.
  • abuse-enforcement-service receives this note and decides the actual action (label/challenge/suspension).
  • Related changes on the abuse-enforcement-service (AES) side are in the AES generic_actions passthrough subsection of the change log.

↑ Contents

How to read the source — caveat list

This list collects points of caution so that one does not judge "it is in the code, so it is true" when reading X's public source. As of the 2026-08-15 update, weights and constant values are filled in Gen4 xai-org/x-algorithm, and Phoenix can also be fully built, but it is not production scale verified(item 1). The score post-processing utilities and external service client modules that the Gen3 canon regarded as outside the repo are also included in the Gen4 release verified(item 1). In Gen3 code, files not connected to any module and files with syntax broken by literal substitution were also observed verified(items 2 and 3).

1. xai-org/x-algorithm did not build through Gen3, and Gen4 partly resolved this. ⚠ This item was updated on 2026-08-15.

   • crate::params, missing in Gen3, was filled in Gen4 as home-mixer/params/param.rs verified.
     • This file is 1084 lines and is a production cron mirror of the weights and constants.
     • The canon's Gen3 observation that symbols like FavoriteWeight have only call sites and no definition is resolved verified.
   • crate::util, missing in Gen3, was filled in the Gen4 release as home-mixer/util/ verified — home-mixer/lib.rs.
     • vqv_weight and quoted_vqv_weight are in util/candidates_util.rs:19, 42.
     • score_normalizer, which Gen3 code called, is not in Gen4; the normalization stage itself is gone (Phoenix section).
   • crate::clients was also filled in the Gen4 release as home-mixer/clients/ (19 files) verified — home-mixer/clients/.
   • Phoenix became fully buildable in Gen4 verified.
     • It comes with a Cargo workspace, pyproject.toml, QUICKSTART, and synthetic data.
     • However, the README states explicitly that a small model can be trained and served: "a small model can be trained and served". It is not production scale verified.
   • Whether the neighboring crate (component_library) is still unreleased in Gen4 needs a deep dive.

   Reflecting the changes above, we redefine the character of the repo. Through Gen3 it was "only the shape of the logic released, values and wiring missing", and Gen4 moved to "values released + safety rules released + Phoenix runnable end to end, but the service glue layer still undisclosed". The missing parts are not accidental omissions but still show a consistent character.

2. In Gen3 (0bfc279) code, the three scorers named in the README were cut off from the module tree. The README wrote "Weighted Scorer / Author Diversity Scorer / OON Scorer", but the actual pipeline was vec![phoenix_scorer, ranking_scorer, vm_ranker] verified — phoenix_candidate_pipeline.rs:299-300 vs README.md:251-255

   • scorers/mod.rs does not declare the three files, and there are 0 external references verified.
     • The three files are weighted_scorer.rs, author_diversity_scorer.rs, and oon_scorer.rs, and the struct names are WeightedScorer/AuthorDiversityScorer/OONScorer.
   • Yet those files contain hardcoded constants, so a reader following the README would mistake unreachable constants for production values verified.
     • An example of a hardcoded constant is p::AUTHOR_DIVERSITY_DECAY.
   • These three are not the only unconnected files. There are 11 .rs files in home-mixer/ not hooked by any mod declaration verified.
     • The 11 files are the following:
       • candidate_hydrators/engagement_counts_hydrator.rs
       • candidate_pipeline/{candidate,candidate_features,query,query_features}.rs
       • query_hydrators/{user_action_seq,user_features}_query_hydrator.rs
       • scorers/{author_diversity,oon,weighted}_scorer.rs
       • side_effects/cache_request_info_side_effect.rs
   • How to tell (Gen3 basis): dead files import crate::candidate_pipeline::{candidate, query}, while the live ranking_scorer.rs imports crate::models::{candidate, query}. Types with the same name exist in both directories, and models/ is current and candidate_pipeline/ is the older generation inferred. This was inferred only from the direction of the connections, and the commit history has no clue.
   • pub(crate) mod is also a declaration. ads/util.rs is hooked in ads/mod.rs:3 by pub(crate) mod util;, so it is live verified.
     • Counting with a regex like ^\s*(?:pub )?mod wrongly judges this file dead verified.
   • In the Gen4 release (47c1bcd), 10 of these 11 files disappeared from the repo, and engagement_counts_hydrator.rs was declared at candidate_hydrators/mod.rs:6 and connected verified. The README of the same release also no longer lists these three scorers verified — README.md.

3. In Gen3 (0bfc279) code, there were places where string and number literals were mechanically replaced with "". Syntax was not checked, so some files had broken compilation or parsing verified.

   • For example, in grox/classifiers/content/safety_ptos.py:241-244, and is left without an operand, causing a SyntaxError, so the PTOS plan itself cannot be imported verified.
   • grox/tasks/task_summarizer_for_post_embedding.py:14 throws FileNotFoundError at import time, so Grox cannot start at all verified.
   • Neither file exists in the Gen4 release (47c1bcd) verified.

4. git log --diff-filter=D <commit> shows the deletions of all ancestors, not the deletions of that commit. It would be wrong to conclude from this command's result that "the 2025 refresh deleted DiversityScorer.scala". That file was deleted on 2023-07-07 verified.

5. If the the-algorithm-ml clone is a shallow clone, every file looks like a "new file". Weight history must not be read from such a clone verified.

6. has_cached_posts is the most impactful switch, but it is a cache threshold, not a model. When 500 or more posts are stacked in the Redis warming cache, this boolean turns on, and then the retrieval sources and many hydrators turn off verified — cached_posts_query_hydrator.rs:11, 69

   • As of the 2026-10-17 commit (862a123), 25 files in home-mixer have a !query.has_cached_posts gate. These include 8 retrieval sources and 12 candidate hydrators verified — 862a123 home-mixer/ grep. In Gen3 code (0bfc279) there were 18 files.
   • On the same basis, the Phoenix scorer also has a has_cached_posts branch verified — phoenix_scorer.rs:70. EngagementCountsHydrator dropped this gate in the 2026-09-04 commit (Gen4+16) (cold start slot selector).
   • Much of "load more" during scrolling is answered from leftover candidates of the previous request without model inference verified.
   • How large that share is cannot even be inferred. The repo has no clue.

7. In Gen3 (0bfc279) code, CoreDataHydrationFilter, despite its name, does not actually filter out candidates whose core data lookup failed. This is because the hydrator does not use author_id verified(code) — core_data_candidate_hydrator.rs:108-113, filters/core_data_hydration_filter.rs:13

   • The README describes this filter as "removes posts whose core metadata hydration failed".
   • We regard it as a bug inferred. Given the code flow, another explanation is hard to find.

8. In Gen3 (0bfc279) code, all failures are silent. There is no framework-level timeout, circuit breaker, or fallback verified — candidate-pipeline/candidate_pipeline.rs:202-218, 266, 419-428

   • Sources and query hydrators that error out disappear verified.
   • If a candidate hydrator errors, the field is left unset verified.
   • Side effects run detached verified.

9. In Gen3 (0bfc279) code, logging is sampled. The two richest log streams are unrepresentative slices verified — side_effects/reranking_kafka_side_effect.rs:14,29

   • The reranking Kafka log samples 5% in production and keeps only the top 50 verified.
   • The other one is kept only in shadow traffic verified.

10. In the 2025 Scala code, some comments disagree with the code. Two comments on the low-signal user gate say "< 25 users", but the actual constant is SmallFollowGraphSize = 5. Quoting the threshold from the comments is off by 5× verified — SignalUtils.scala:17-23 vs AllowLowSignalUserGate.scala:12, DenyLowSignalUserGate.scala:12

11. There is no single value called "ranker score". The 2025 MMR and the 2026 VMRanker both overwrite the score field. When looking at a score in logs or code, check which stage's value it is. This is why the 2026 code keeps weighted_score and score as separate fields verified — DiversityRescoringFeatureHydrator.scala:70, vm_ranker.rs:47-52, ranking_scorer.rs:277-281(both Gen3 code). In the 2026-10-17 commit (862a123), VMRanker also fills the two fields separately (vm_ranker.rs:93-131).

↑ Contents

Generalizations from this service

Among the techniques observed at X, those that span several services are collected separately in topic notes in the topics folder. Each note places X's cases side by side with cases from other services such as Meta.

↑ Contents

Open questions

Of the questions that X's public code has still not answered since the Gen4 release (2026-08-13), the most recent are those left by the 2026-10-16 and 10-17 commits (Gen4+43 and +44) (Gen4+43 in the change log, Gen4+44). Before that is the deep dive task left by the 2026-10-14 commit (Gen4+41) (Gen4+41 in the change log). The feed request context values added to the ranking request have been filled only for ads index requests since the 2026-10-13 commit (Gen4+40) verified(Gen4+40 in the change log). As of the same commit, the Phoenix ranking request side still has no answer.

Unresolved and needing deep dive (after the Gen4 release)

These are questions that public code still could not answer after the Gen4 release, or that need a deep dive. The struck-through item was resolved in the 2026-09-17 commit (Gen4+23).

• What percentage of actual impressions does the has_cached_posts path account for? Is there any way to estimate it from outside? The repo still has no clue, so even inferred is not possible.
• The details of the SimClusters core algorithm are not yet confirmed. The open points are the SVD pipeline, k, the index format, and the update cycle. The simclusters/ sub-repo needs a deep dive.
  • It also asks whether a Gen4 component corresponds to SimclusterColdPostsCandidateSource.
• What are the details of the agatha/bdsm/user-cred-v2 account scorers? The open points are each scorer's actual model, update cycle, and label threshold.
• How is pnsfwmedia calibrated? The open point is how it combines the CLIP embedding with the agatha score.
• What are the details of the phoenix/ training pipeline? The open points are the loss, negative sampling, and the synthetic data format. The optimizer was released in the 2026-08-19 commit (Gen4+4) (training code).
• How do the phoenix-rankall/ and phoenix-rankall-strato/ indexing pipelines work? The open points are the visibility pre-lookup logic and the details of the event layer.
• In abuse-enforcement-service/, which score maps to which action (label/challenge/suspension)?
• What are the DSL and rule categories of scarecrow/botmaker/botmaker-rules? The ratio of public to undisclosed rules is also an open point.
• What is the label taxonomy in under-the-hood/? The open point is how the labels shown to users connect to the rule code.
• The holdout ratio params of InventoryHoldoutFilter: resolved in the 2026-09-17 commit 42266f3 (Gen4+23), which confirmed three params (the 21 filters section).
  • The three params are InventoryHoldoutOriginalsPercent, InventoryHoldoutRepliesPercent, and InventoryHoldoutRetweetsPercent.
• Where is the RECSYS_EMBED_V4 multimodal embedding used? Not in home ranking.
• What is the architecture of the 2025 Scala-generation heavy ranker (Navi)? The open point is the details of the MaskNet/DCN family. The model definition is on the the-algorithm-ml side and must be checked separately.
• What do the Phoenix retrieval two-tower structure and the ANN index look like? I have read phoenix/recsys_retrieval_model.py. The serving-side index was partly released in Gen4 (phoenix-rankall/). The whole must be verified again.
• Was the dual layout of models/ and candidate_pipeline/ in the 2026 generation cleaned up in Gen4? This still needs to be checked.
• Was the weight noise injection of Gen2 kept in Gen4? The Gen2 noise is a Beta distribution in HeavyRankerWeightsQueryFeatureHydrator.scala. Whether it was kept needs a deep dive.
• The 2026-10-08 commit (Gen4+38) left four deep-dive tasks (change log: 35650fb).
  1. The final response size of the video carousel, and whether it overlaps with the main feed
  2. Whether VF hydration/sources/exchange.rs (528 lines) and clients/user_location_client.rs (345 lines) add new location-based verdicts
  3. The new hydration scheduler in phoenix-rankall/src/hydrate/* (about 1,300 lines)
  4. Whether the MoL two-tower is turned on in the actual experiment config
• Where are FeedRequestContext and FeedCursorDirection, which the 2026-10-09 commit (Gen4+39) added to the ranking request, filled and where are they read?
  • We need to check when the code in which the Phoenix ranker reads these values as features will be released, and from which request information home-mixer fills them.
  • In Gen4+40 (e1a0375), home-mixer started building the value with RequestContext::parse(query.request_context) and filling it only in ad index requests verified.
  • The Phoenix ranking request side is still unresolved.
• The 2026-10-13 commit (Gen4+40) left four deep-dive tasks (change log: e1a0375).
  1. The verdict threshold of CLIENT_TWEET_RECAP_DWELLED in the two-tower ld head, and whether the combined preset is production retrieval
  2. The service that calls the reply ranker reply_two_tower_v2, and its connection to the grox reply score
  3. Whether the abuse-enforcement action_control and off_switch (about 6,000 lines) and the VF explain.rs and rules/levels.rs refactors include a policy change
  4. Which side is the serving value for the four weights that diverged between the home-mixer and vm-ranker mirrors (PostUnexploredWeight, favorite, video_open, cont_dwell_time)
• The 2026-10-14 commit (Gen4+41) left four deep-dive tasks (change log: 17c96bf).
  1. Whether any serving instance turns on serve_page_decoder (off by default), and which home-mixer stage consumes the decoding result
  2. The training log already has rl_rerank_prob and rl_decode_prob; what is the behavior policy that produced these values
  3. Whether immersive_page_rl is only for the immersive video surface or is also used in Home For You
  4. Whether any experiment turns on TopicDppEnabled
• The 2026-10-16 and 10-17 commits (Gen4+43 and +44) left four questions (change log: bc3948b, 862a123).
  1. Which encoder the Phoenix model uses to read TweetInfo.postEngagers
  2. The home-mixer call site that fills RankingScoreWeights
  3. Whether the head that uses the IsBrandEngagement label is the ads ranker
  4. The author bucket scale of the ZeroFav two-sided experiment
• The 2026-10-20 commit (Gen4+45) left two questions (daily 2026-10-21).
  1. For which re-ranking objective VMRanker uses the viewer-history topics carried in the request
  2. How the history aggregation DENSE_WITH_LONG_DWELL_HOME_EXCURSION, used by the shadow cluster Experiment16Fou, differs from the existing aggregations
• What does X change in its EU DSA transparency report? The question is what X newly submits or deletes after this 8-13 release. Whether the regulatory documents and the code match (sync) is a new verification angle.

Resolved by the Gen4 release (2026-08-15)

The Gen4 release resolved the questions about the actual weight values, whether VMRanker and DPP are active, and the number of Phoenix heads. The training pipeline and cold start questions were resolved in part, with some remainder.

• ✅ Actual weight values: home-mixer/params/param.rs was released as a production cron mirror.
• ✅ Whether VMRanker/DPP are on: they are on (EnableVMRanker=true, DPP θ=0.65, K=150).
• ✅ Number of Phoenix heads: confirmed as 24 per the documentation.
  • The composition is Engagement 7 + Clicks 6 + Attention 5 + Author 1 + Negative 5.
• ✅ 2026 Rust training pipeline: phoenix/ has the training code, synthetic data, and a QUICKSTART. However, the loss, negative sampling, and optimizer need a deep dive.
• ✅ Absence of new-item cold start: Gen4 released a new-author boost (PostUnexploredWeight=0.02, ColdStartImpressionThreshold=1000). However, there is still no separate mechanism for individual new posts.

Resolved by the Gen4+1 release (2026-08-16)

This release resolved the questions about the rule for reading weight ratios, the cold start slot selector, and the jurisdiction filter. The Phoenix bool features and the BDSM enforcement note were resolved only in part.

• ✅ Rule for interpreting weight ratios: the comments in param.rs and ranking_scorer.rs and the README state explicitly "do not read the ratio as a raw count equivalent".
  • Through Gen3 the canon cited documents that interpreted this as "reply 13.5 = the value of 27 favorites". It is now confirmed that this interpretation was a framing that was wrong in principle.
• ✅ Fill status of the Phoenix bool features: the observation that all of bool_features are zero-stamped was partly resolved in Gen4+1.
  • Three were filled (IS_STALE_POST14D and IS_AUTHOR_FOLLOWED/FOLLOWING).
  • Whether the remaining bool slots are used is still not public.
• ✅ Cold start slot selector logic: two branches, greedy and Beta Thompson sampling, were connected through params. However, Thompson is off by default, and actual production is still greedy.
• ✅ Existence of a jurisdiction filter: confirmed at the code level through Brazil2026ElectionFilter. Whether election and regulation responses for other countries will be attached by the same practice is a new checkpoint to watch.
• ✅ BDSM enforcement note generation logic: the gate and dominant head logic were released. The template strings and the operating point values are still not public.

Resolved by the Gen4+2 release (2026-08-19)

This release revealed how Grok safety labels are migrated, how conflicting adult content verdicts are handled, and the indexing gate of Phoenix retrieval. The judgment on the first item was overturned the next day.

• ✅ Schema migration practice for Grok safety labels: it was confirmed that the label schema is managed by version and that its migration is built into dual-write detection logic in the code. Caution: this judgment was overturned by a redesign in Gen4+3 (the next day)(Gen4+3 record).
  • compute_verdict_v2 references six _V2 labels.
  • strip_v1_grok_written filters out V1 dual-write labels that belong to PROMPT_OWNED_RULES = [1400, 1410, 1420, 1500, 1510, 1610, 1700].
  • The v2_mirrors_v1_across_tier_matrix test asserts V1/V2 equivalence across 6 label combinations and 2 tweet_id points.
• ✅ Handling when adult content verdicts conflict: when safemodel and PtOS disagree, the Grok 4.5 judge re-decides, and only hard→soft demotion is allowed.
  • safemodel is an in-house vision model, and PtOS is a policy verdict. There is no escalation.
  • The verdict layer expanded to three layers (safemodel, PtOS, the Grok 4.5 judge).
  • A dedicated circuit breaker for EAPI_GROK_4_5_X_ALGO was added, and the audit log states the text "Grok 4.5 Cross Validation disagreed" explicitly.
• ✅ Indexing gate of Phoenix retrieval: the favoriteCount >= 1 condition was removed from build1FavIndex and buildMMEmbMetadataDump. The actual index is all incoming candidates.
  • The index name "1fav" became a backward compat label.
  • The 1fav_uec_count_race counter instruments only the cases reached without a fav.
• Deep dive needed (new):
  • Whether the Grok 4.5 judge expands to other categories
  • The actual observed rate of 1fav_uec_count_race (not observable from outside)

Resolved or overturned by the Gen4+3 release (2026-08-20)

This release overturned the dual-write judgment on brand safety V2 that was considered settled in Gen4+2 the day before. It also confirmed the SID connection in VMRanker, the hydration branch between For You and Following, the move of the Phoenix event stream to v3, and the merge of the grox reply_spam flow.

• ⚠ Claim changed (in one day): the V2 brand safety dual-write logic was completely redesigned.
  • The Gen4+2 approach was strip_v1_grok_written and seven PROMPT_OWNED_RULES rule ids. The Gen4+3 approach is four V2_WRITTEN_LABELS gates and V1 fallback semantics.
  • The seven PROMPT_OWNED_RULES disappeared from the code entirely. The deep-dive item that asked about their rule id mapping was resolved automatically (the question itself no longer exists).
  • The new semantics are much cleaner: "if even one V2 label exists, V2 decides and V1 heuristics are ignored entirely; if none exists, fall back to V1".
  • This case is strong evidence for the observation that X exposes the reality of "daily commit" dual-write migrations as they are.
• ✅ SID connection in VMRanker: seven SID fields entered the VMRanker request. We confirmed that there is a re-ranking-side SID consumer that pairs with the SID window in Phoenix retrieval.
  • The fields are sid_known, sid_k_l1/2/3, and sid_gap_l1/2/3.
  • The architectural fact that SID is a 3-level hierarchical codebook was newly revealed.
• ✅ Hydration branch between For You and Following: For You and Following started to split explicitly at the pipeline stage.
  • VFFollowingCandidateHydrator was newly connected.
  • Following now keeps only the VF result and leaves the other candidate fields at default.
• ✅ The Phoenix event stream moved to v3 and the Kafka cluster was separated.
  • content_understanding_realtime_unified_posts → ..._v3
  • main-2 → phoenix-kafka-external-bootstrap
  • ValueOnly → KeyValue(key=Long)
  • grox reply_spam also moved to v3 in the same commit, and its flow was unified.
• ✅ The grox reply_spam flow was unified and its threshold was adjusted again.
  • PostStreamTaskGenerator was deleted, and ReplyRankingTaskGenerator absorbed PlanSpamComment and PlanCoordinatedSpam.
  • The follower threshold rose from 40k to 60k. This is the third increase, after 30k → 40k in the 2026-08-17 commit (Gen4+2).
• Deep dive needed (new):
  • The origin of the four V2_WRITTEN_LABELS labels (a Grok prompt change? retraining?)
  • The SID codebook definition (vocabulary size and training approach of each level)
  • The formula for sid_gap
  • Whether Phoenix Kafka unified_posts_v2 existed (or whether it was a direct jump from v1 to v3)
  • How the ColdStartTrackedIds param actually connects to the simplified count_tracked_ids instrumentation

Resolved or new in the Gen4+4 release (2026-08-21)

This release confirmed the Phoenix optimizer in code and gave the first signal that Phoenix is expanding to the search surface. It also newly revealed the reward head and distribution prediction head, a Following-side filter, the PTOS Grok 4.6 shadow, and the VF shadow deployment base.

• ✅ The identity of the Phoenix dense-param optimizer was confirmed in code in Gen4+4. This long-undisclosed item is a hybrid of Muon (Newton-Schulz orthogonalization) and Adam.
  • muon_adam_patterns="embed,_emb,logits,vocab,norm,table"
  • base LR 1e-3 → 7.1e-4, muon_split_fused="qkv:128", muon_consistent_rms=0.2
  • Embeddings use rowwise_adagrad (LR 0.28, half-life 2500 step, lazy_decay, weight_decay 2.8e-4).
  • The GB300 override is LR 5e-4 and half-life 5000.
  • This is the first case, among the large-scale production recsys training pipelines this repository has covered, where Muon adoption is confirmed in code. A new axis of observation opened: LLM optimizers moving into ranker/retrieval training.
• ✅ The first code signal appeared that Phoenix is expanding to the search surface. A search config was newly added, paired with the two proto actions CLIENT_TWEET_(NOT_)RELEVANT_TO_SEARCH.
  • The config is xrecsys_search (vocab 100M/100M/30M, EP 512, use_product_surface=True, condition_search_relevance_on_prompt=True).
  • The earlier canon understood each surface to have its own codebase. Now the same Phoenix Transformer is being unified so that swapping only the config serves both search and feed.
• ✅ The skeleton of a reward head appeared in the Phoenix proto. A connection toward RL (actor-critic value+policy) training or reward-model-guided re-ranking slate decisions has begun. Which scorer consumes it needs a deep dive.
  • The new items are RewardOutputs (values and policyLogits) and ScoreInfo.rewardRerankSlotProb.
  • The 2026-10-14 commit 17c96bf (Gen4+41) released the page decoder (actor) and twin critic training code for the immersive surface, and this training reads the rewardRerankSlotProbSeq column as the logged probability rl_rerank_prob verified (Gen4+41 in the change log).
    • In the same commit, code outside the proto does not reference RewardOutputs (rewardOutputs and policyLogits) verified — grep over all of 17c96bf.
• ✅ The prediction schema of X's ranker began to carry a distribution shape for the first time. It predicts link-open probability not as a single scalar but with five quantile heads: P10/P25/P50/P75/P90.
• ✅ The parallel structure of the Following pipeline widened. After the hydrator branch in Gen4+3, one filter and one hydrator were newly attached in Gen4+4.
  • The Gen4+3 branch is VFFollowingCandidateHydrator.
  • FollowingViewerMutedKeywordFilter (65 lines) also matches quoted text and ancestors.
  • QuotedPostTextHydrator (106 lines) fetches the quoted text from TES.
  • For You and Following use different components in their filter and hydrator stacks, so the parallel structure is reconfirmed.
• ✅ A Grok 4.6 shadow started in the PTOS verdict layer. Only 10% of traffic goes to 4.6 in the child safety cross-validation and deluxe 4.3 categories. The dual-write stage of the 4.5 → 4.6 gradual rollout has begun.
  • The ratio is set by _GROK_4_6_INTERNAL_DIAL = 0.1.
  • The new model (EAPI_GROK_4_6_INTERNAL) and a dedicated circuit breaker were connected.
  • prior_nsfw.py (34 new lines) separates the Manhattan lookup cache.
• ✅ The VF shadow deployment infrastructure was connected. A base for verifying rule changes of the fail-closed VF through shadow was established (Gen4+4 in the change log).
  • visibility-filtering/dark_traffic_setup.rs (109 new lines) is env-gated and mirrors requests to xai-vf-shadow.prod.visibility only from specific ordinal pods.
  • It uses mTLS and reloads the /config/dark-traffic/dark_traffic.yaml rule file.
• ✅ The three Phoenix bool features filled in Gen4+1 were raised to ON by default in training.
  • The four enable_stale_post, enable_is_author_followed_by_viewer, enable_is_author_following_viewer, and enable_engagement_counts are the defaults of _home_direct_packed_base().
  • Only the grpc_recsys dataset type overrides enable_stale_post=False (presumably to avoid the count zeroing risk in real-time serving inferred).
• Deep dive needed (new):
  • In which training loop RewardOutputs is filled and which scorer consumes it (the large +265/−25 overhaul of recsys_model.py must be traced)
  • Whether the link-open 5-quantile is a quantile regression or a categorical mixture
  • Whether the xrecsys_search config was deployed to production
  • Observed performance of Muon (no public metrics)
  • The actual dark_traffic YAML (undisclosed)
  • The prompt form of condition_search_relevance_on_prompt

Resolved or new in the Gen4+5 release (2026-08-22)

This release connected the path by which home-mixer receives and uses the slate context built by Phoenix serving (the gate is false by default). AI trend context began to attach to For You candidates. The Grok 4.6 shadow ratio tripled in one day.

• ✅ A path appeared that consumes the slate context built by Phoenix serving. SlateContext.sid*, introduced in the proto in Gen4+4, was connected through to the actual pipeline. The gate was connected as false by default.
  • The components are PostCandidate.served_slate_context, the From<xai_recsys_proto::SlateContext> conversion impl, the candidate_slate_context() extraction in PhoenixScorer, and the three-way fallback in RankingScorer.
  • The fallback order is the UseServedSlateContext gate first → cache → local computation.
  • The first step is confirmed in moving the slate context computation point from home-mixer local to Phoenix serving.
• ✅ A new AI trend feedback context channel appeared. An "AI trend" label is attached to For You candidates as a second context channel alongside topic feedback. It enters as a fallback only when there is no topic feedback, so the two do not overlap.
  • TOP_TREND_COUNT = 2 (at most 2 per page).
  • One candidate is picked at random per trend.
  • A triple gate applies: standalone original post, non-topic, non-in-network.
  • The landing URL is https://x.com/i/trending/{trend_id}.
• ✅ The core of the emb_table loader was split out as pyo3-free. This prepares large embedding tables to be loaded in non-Python contexts. The direction matches the vocab 100M/100M/30M flow of xrecsys_search.
  • The core is load_tensor_into(&str, &str, &[(String, String, usize, usize)], &mut [u8], usize, usize) -> Result<(), String>, and the Python wrapper is separate.
• ✅ The Grok 4.6 traffic split dial rose 3× in one day, from 10% to 30%. This supports that the 4.6 traffic stayed within the acceptable disagreement rate range without a circuit breaker trip inferred. The flow of daily refinement continues through Gen4+2→+3→+4→+5.
• ✅ A callback to end streaming responses early was added to grok_sampler. This is a sampling infrastructure optimization matched to the growth of 4.6 traffic inferred.
  • stop_predicate and aclosing were connected, and the llm.sample.early_stop.count counter was added.
• ✅ A profile_image field was newly added to the grox User. The schema was extended so the Grok classifier can reference the profile image as verdict context.
• Deep dive needed (new):
  • The upstream definition of AI trend (Grok clustering? human curation?) and the pipeline source of batch_get_tweet_ai_trend
  • When the UseServedSlateContext gate is raised to true. That point is also when the Phoenix-served slate context reaches production traffic (the checkpoint to watch in the next commit).
  • The next move of the Grok 4.6 dial (30% → 50%/100% or a decrease)
  • Where stop_predicate is actually used (whether it serves as a callback matching the safety verdict JSON tag)

Resolved by the Gen4+6 release / new (2026-08-23)

This release added 20 ads conversion prediction heads and a delayed-feedback training branch, and the export nano config now inherits the training recipe of the GB300 override. PTOS child-safety cross-validation and deluxe reclassification moved entirely to Grok 4.6 internal.

• ✅ 20 new ads conversion prediction heads were added. Previously only app installs were predicted; now six post-install engagement events are predicted for click and view each (ads conversion subsection).
  • They were added at proto ActionName 171–190. They are 6 MACT in-app events (purchase·add_to_cart·level_achieved·tutorial_complete·sign_up·custom) × 2 families click/view = 12, plus 8 view-through conversions (web conversions).
  • Previously there was only ADS_ATTRIBUTED_MACT_CLICK_INSTALL (=140).
  • CLIENT_TWEET_SEARCH_QUERY_REFORMULATED was added to NEGATIVE_FEEDBACK_HEAD_INDICES.
• ✅ The training loss now splits on whether feedback is delayed. X ranker training is now explicitly aligned with the standard handling in the ads delayed conversion prediction literature (Chapelle 2014).
  • The split_head_training_by_source config brings kafka's is_delayed_feedback column in as sample_source.
  • Engagement heads train only on fresh samples and conversion heads only on delayed samples (the loss mask splits).
  • There are three metric slices: fresh/delayed_clicked/delayed_non_clicked.
  • The tuning knobs are mact_in_app_loss_weight, train_view_through_heads, and conversion_keep_mask.
• ✅ A new ads_slim metric group was added. It is a slim signal combination for the ads ranking line.
  • It includes 3 positive signals (open link, external link long dwell, video quality view).
  • It includes 4 negative signals (not interested, block, report, mute).
  • Non-primary conversion clicks are also included.
• ✅ The Muon training recipe was formally added to the export. The export moved to a state where running the exported training Python code as is yields production Muon training.
  • TRAINING.md, README, and reference/README.md were rewritten. They state explicitly that the flagship ranking config and home_direct_packed_nano are trained end-to-end with the production Muon recipe.
  • Embedding rowwise AdaGrad has three options: accumulator half-life decay, lazy per-row decay, and decoupled weight decay.
• ✅ home_direct_packed_nano now inherits the LR and optimizer settings of the NVIDIA GB300 hardware override. The override itself has existed since the Gen4 release verified — xrecsys.py:305-313. The same file has also had _H100_OVERRIDES since the Gen4 release (line 297 of the same file).
  • _GB300_OVERRIDES defines bs_per_device=512, ep=64, attn_impl="cutedsl_ranker_varlen_attn", remat_policy=RematType.SAVE_GB300_RECSYS, unroll_layer_stack=True, LR 5e-4, and checkpoint_every_n=300.
  • This attention kernel is a varlen attention kernel based on NVIDIA CUTLASS/CUTe DSL, and may be an xAI in-house kernel.
• ✅ It was confirmed for the first time that an xai-o2 storage system exists inside xAI. The interface of the xai-o2 crate, an S3-compatible object storage client, was released.
  • It exposes a BaseO2Client trait and 15 builder methods (endpoint, bucket, access_key, chunk_size_mib, retry, timeout, etc.).
  • However, build() always returns the error "the xai-o2 backend is not included in this build" in open-source builds.
• ✅ Grok 4.5 internal was removed entirely from PTOS child-safety cross-validation and deluxe reclassification, and 4.6 internal went to 100%. The dial-up (10%→30%→100%) that began in the 2026-08-19 commit (Gen4+4) finished in two days verified — aad7179, d0cef2f, classifier.py in 28e414f.
  • The adult-content judge (Gen4+2) keeps using EAPI_GROK_4_5_X_ALGO regardless of this dial verified — classifier.py:563.
  • All 4.5-related breaker, sampler, and dial constants were deleted. Rollback is not possible.
• ✅ SlateContext was added as field #9 of NextActionDistribution. Now a slate context is attached to each action distribution as well. The consumption path is still a skeleton.
  • Earlier, Gen4+4 introduced ScoreInfo.slateContext (re-ranking slate decisions) and Gen4+5 introduced PostCandidate.served_slate_context (consumed by home-mixer).
• ✅ The grox reply_spam follower threshold rose from 60k to 80k. It is the fourth increase within 8 days of the Gen4 release (15k→30k→40k→60k→80k), and this increase is 1.33×.
• ✅ The visibility-filtering dark_traffic target changed from shadow to staging. ⚠ The Gen4+4 canon interpretation needs to be revisited (Gen4+4 in the change log).
  • Within 3 days the shadow workload was deleted and redefined as staging mirroring.
  • The target changed from xai-vf-shadow.prod.visibility to xai-vf-service.staging.visibility:grpc.
• Deep dive needed (new):
  • The source of the is_delayed_feedback upstream pipeline (the conversion re-observation event logger that feeds this column into kafka)
  • When split_head_training_by_source is promoted to True in _home_direct_packed_base() or the flagship config (a signal that ads conversion training is moving to production)
  • When the train_view_through_heads opt-in is promoted
  • Whether the flagship also explicitly inherits the GB300 override
  • Whether the actual xai-o2 implementation will be released (unlikely inferred)
  • Which way the EVERGREEN retrieval dataset is redefined (rename vs. revert)
  • When 4.7 shadow starts after Grok 4.6 settles
  • Whether the cutedsl_ranker_varlen_attn kernel is an xAI in-house kernel

Resolved by the Gen4+7 release / new (2026-08-26)

This release added a regression verification channel that measures the consistency of the visibility filtering rewrite, and brand safety verdicts grew to four levels. The PTOS judge widened to three categories.

• ✅ The regression verification channel for the VF rewrite is connected. The consistency of the VF sub-repo rewrite, the Gen4 headline, can now be audited at production traffic scale. Rollout risk dropped considerably (Gen4+7 in the change log).
  • visibility-filtering/reference_compare.rs (812 lines, new) is layered on top of the dark_traffic stack (Gen4+4/+6), making it possible to measure actual verdict comparison.
  • It is staging-only, through VF_DUAL_CALL_HARNESS_ENABLED and an enforcement mechanism that panics when APP_ENV=prod.
• ✅ BrandSafetyVerdict grew to four levels. Previously the brand safety verdict controlled ads adjacency with only three levels (Safe/Low/Medium), and lumped labels of different risk into the same tier. This limitation is resolved.
  • The new tier is HighRisk = 4.
  • The five labels that were in medium (NSFW_HIGH_PRECISION·NSFW_HIGH_RECALL·GORE_AND_VIOLENCE_HIGH_PRECISION·PDNA·EGREGIOUS_NSFW) and the new SOFT_NSFW were promoted to HighRisk. V1 and V2 moved symmetrically.
  • The ads utility has_avoid() handles both Medium and High, and the quote adjacency escalation also changed from Medium → High.
• ✅ Five V2 labels are exposed in the downstream proto. Previously V2 labels existed only inside home-mixer. Now other services that consume the feed response can also see the V2 verdict.
  • GROK_SFA_V2, GROK_NSFA_LIMITED_V2, GROK_NSFA_V2, GROK_NSFA_EXPANDED_V2, and PTOS_REVIEWED were added to the safety_label_to_proto mapping in scored_posts_server.rs.
• ✅ DDG tracking for the V2 experiment is connected. V2 is at the experiment stage, and the infrastructure to observe revenue/safety metrics is complete. The 8-day journey of Gen4+2 (introduction) → +3 (redesign) → +7 (experiment measurement) is complete.
  • ads_injection_logging_side_effect.rs logs the EnableAdsBrandSafetyVerdictV2 experiment bucket in the ddg_experiment_bucket field.
• ✅ The PTOS judge layer widened to 3 categories. Previously there was an adult-content judge class and, separately, the child-safety cross-model validation that had been inside the classifier since the Gen4 release. Now all 3, ChildSafety, ViolentMedia, and IllegalAndRegulatedBehaviors, are grouped into one judge class (Grox → PtOS section).
  • The new judge is SafetyPtosPolicyCrossValidator. The earlier adult-content judge is SafetyPtosAdultContentCrossValidationJudge (Gen4+2).
  • The new judge class uses the EAPI_GROK_4_5_X_ALGO endpoint that the adult-content judge has used since Gen4+2. It is separate from EAPI_GROK_4_5_INTERNAL, which was removed entirely in Gen4+6 verified — classifier.py:100, 561.
  • X has settled into a dual stack that keeps the primary (4.6) and the judge (4.5 X_ALGO) on different models and tuning.
• ✅ Parent-thread rendering was turned on for the PTOS ViolentSpeech and SuicideOrSelfHarm categories. Now grox puts the post and its ancestors into the LLM prompt together.
  • The underlying reason is that these two categories cannot be judged for harm from the target post's text alone.
    • For example, whether "I want to die" is about a game or about self-harm is decided only by the thread context.
  • USE_THREAD_RENDERER_CATEGORIES is new.
  • USE_REASONING_CATEGORIES was also turned on for only these two (thinking suppression lifted).
• ✅ The Phoenix checkpoint loader was reinforced for multi-node environments. It addresses restore-path OOM issues on the Muon and GB300 generation (checkpoint stack subsection).
  • It is a follow-up verification result that came after Gen4+6 formally included flagship Muon training in the export inferred.
  • xai_checkpointing/load.py gained new arguments domains (per-tensor slice), tag (checkpoint subdir), and concurrent_gb (concurrent decoding throttle).
  • Restore is serialized per node with XAI_RESTORE_NODE_SERIALIZE and fcntl.flock().
• ✅ grox reply_spam started writing reply ranking scores to the regional cache as well. The path that writes scores back to Strato has existed since the Gen4 release, and this time regional cache fanout was added verified — storage module in d011592.
  • ReplyRankingScoreStratoLoader has put scores to Strato and emitted them to Kafka v2 since the Gen4 release (47c1bcd).
  • Since Gen4+7, when score ≤ 1.0 it also fans out to the regional caches (Atla, Pdxa).
• ✅ The grox reply_spam follower threshold rose from 80k to 100k. It rose five times in 11 days (15k→30k→40k→60k→80k→100k), so the set of spam-judgment targets keeps widening.
• ✅ X Spaces (audio) context enters the grox LLM judgment. The audio itself is not listened to, but the Space title becomes material for brand safety and adult content judgments.
  • The SpaceMetadata class and the Post.space_metadata field are new.
  • The renderer puts the audio Space card into the prompt in the form \n\nThis post has the following audio space card attached:\nAudio Space title: {title}.
• Deep dive needed (new):
  • Whether the VF reference diff goes only to logs or to a separate dashboard (definition of the promotion threshold: "promote if the diff rate is X% or lower")
  • What the reference VF wrapped by StratoVfClient actually is (a remnant of the Gen1 and Gen2 Scala VF? a separate stack?)
  • Whether the HighRisk tier triggers follow-up actions beyond ads avoid (strong ranker decay, complete block of organic impressions)
  • The holdout/treatment ratio of the V2 experiment
  • How EAPI_GROK_4_5_X_ALGO is tuned (x-algorithm fine-tuning on the 4.5 base, or a completely separate model)
  • The next category for PTOS judge expansion (spam and hate speech are presumably the next targets of the sequential expansion inferred)
  • Whether SafetyPtosAdultContentCrossValidationJudge and the new SafetyPtosPolicyCrossValidator will be unified (same function signature, different routing)
  • The meaning of the grox reply_spam score threshold 1.0 (whether the cache fanout gating threshold is a probability or a score)
  • Whether a pipeline exists in which the LLM listens to the audio content itself (Space title text only vs. STT of the real audio stream)
  • The production tuning value of concurrent_gb and how much restore time increases

Resolved by the Gen4+8 release / new (2026-08-27)

This release added a general path for the abuse-enforcement service to execute actions requested by scorers. The VMRanker path was greatly simplified, and the Following pipeline now applies block relationships up to the original author of quote and retweet chains.

• ✅ A general pipeline in which AES executes actions requested by scorers is connected. Now a scorer can add and send requested_actions directly, outside the hardcoded YAML rules (Gen4+8 in the change log).
  • The new file is abuse-enforcement-service/service-lib/src/generic_actions.rs (663 lines, new), and its rule pair is the platform_row_without_requested_actions skip and the act_requested_actions act.
  • Verification is two layers of fail-closed.
    • (a) hardcoded kind allowlist (5 user kinds, 2 post kinds, MAX 16 per msg)
    • (b) GrowthBook dynamic allowlist (kinds·suspend_policies·labels)
  • Rejected requests are not discarded but are recorded as outcome.info["requested_actions_skipped"] JSON and the GENERIC_ACTION_TOTAL{outcome=skipped, reason} metric.
  • As the first client, the panda_reports_embedding_v10_rough_spam label appeared in a temporary PlatformManipulation suspend rule. This is evidence that the embedding-based spam detector is a continuous pipeline retrained up to v10.
• ✅ The VMRanker path was greatly simplified. VMRanker passes through the score returned by the rank server as is, and DPP diversity is handled by the server's own logic.
  • AUTHOR_DIVERSITY_VALUE_MODEL_ID, the 128-line MPN fallback, the AuthorColdStart post-processing, and the 88-line ranking_scorer::effective_head_weights were all deleted.
  • The only new constant is DPP_VALUE_MODEL_ID = "dpp".
  • It is an architecture refactor that pairs with Gen4+5's UseServedSlateContext (migrating slate context to Phoenix serving). home-mixer is kept thin and the gRPC servers take over the computation.
• ✅ The Following pipeline respects block relationships up to the original author of quote/retweet chains. X explicitly blocks block circumvention through quotes/retweets.
  • The new hydrator FollowingBlockedByHydrator (65 lines) makes a batch check_blocked_by lookup to SocialGraphClient with the list of quoted/retweeted authors of each candidate.
  • AuthorSocialgraphFilter filters on six branches.
    • (i) viewer→author mute
    • (ii) viewer→author block
    • (iii) author→viewer block
    • (iv) quoted author→viewer block
    • (v) viewer→quoted author block
    • (vi) viewer→retweeted author block
  • It is connected in reverse_chron_posts_pipeline.rs (before the VF filter).
• ✅ The account list of Brazil2026ElectionFilter was updated. This is a sign that a regular update practice has taken hold. The filter semantics did not change (the follow exception is kept).
  • It reflects the TSE dadosabertos 2026-08-25 edition (+3,379 lines, 4,947 lines in the whole file).
  • The README also states it explicitly: "(Account list updated August 25, 2026.)".
• ✅ A prefloor snapshot is connected in phoenix-rankall. New candidates stay in a separate pool until they pass the min_age threshold, and once they pass they are reflected in production immediately.
  • The changed files are store/base.rs (+202/−61) and config/mod.rs (+51/−12).
  • For windows with min_age, the prefloor_<window>.parquet file is loaded as well (PREFLOOR_SLACK_SECS = 6.0 * 3600.0, 6 hours of slack).
  • It is a freshness improvement in the same direction as the removal of the favoriteCount>=1 gate in Gen4+2.
• ✅ The Phoenix copy_port rate limit became accurate. Previously, sequential polling could let actual in-flight concurrency exceed the cap. It is a serving-side reinforcement that pairs with Gen4+7's restore path reinforcement.
  • In copy_port_client.rs (+155), join_all(futures) changed to join_all(futures.map(tokio::task::spawn)).
  • Now the tokio scheduler runs them in true parallel, and the rate limit issues the next batch based on the in-flight count.
  • There are two regression tests (rate_limit_caps_in_flight_despite_spawn, rate_limit_paces_between_spawned_batches).
• ✅ The README's Latest Updates was renamed Notable Updates. It is a declaration that the README will not be updated daily even though follow-up commits are added daily inferred.
• ✅ In the home-mixer cron mirror (sync 2026-08-25), two weights and the ads brand safety V2 switch changed verified — home-mixer/params/param.rs:342, 356, 908.
  • VqvWeight changed from 0.05 to 0.0, and DwellWeight changed from 0.0 to 0.05.
  • EnableAdsBrandSafetyVerdictV2 changed from false to true. This value went back to false in the 2026-09-16 commit (Gen4+22) and became true again in the 2026-10-03 commit (Gen4+35).
• Deep dive needed (new):
  • The first production GrowthBook configuration of the generic_actions allowlist (which scorers it opens to first)
  • The evolution of panda_reports_embedding_v10 from the earlier v1–v9 and when v11 arrives
  • The actual value model architecture of DPP_VALUE_MODEL_ID (whether it is the same as the DPP (θ=0.65, K=150) in the re-ranking section or different)
  • The new consumer point of AuthorColdStart (where it went after being removed from vm_ranker)
  • Whether AuthorSocialgraphFilter is also connected to the For You pipeline
  • The first active window of the prefloor snapshot
  • The role of the grox/flows/upa/ subflow
  • The new event type in the +27 lines of URT client_event.rs

Resolved by the Gen4+9 release / new (2026-08-28)

This release made both saving and restoring of Phoenix checkpoints batch pipelines, and unified the training-step validity gating between dense and embedding. VF removed the media fallback cache, and three ads conversion heads were newly added to the proto.

• ✅ The Phoenix checkpoint stack was reorganized again so that both save and restore have an RSS cap. The entire Muon+GB300 checkpoint round trip is now a batched pipeline (checkpoint stack subsection).
  • Symmetric to Gen4+7's restore-side reinforcement (concurrent_gb throttle and node-lock), this time the save side was reorganized.
  • The save side gained a new ThrottledD2HArrayHandler. It overrides Orbax ArrayHandler.serialize to do D2H and write per batch, and the budget is set by save_concurrent_gb.
  • Saving is serialized per node with the XAI_SAVE_NODE_SERIALIZE env and the /dev/shm/xai_save_node_lock flock.
  • gc.collect() and malloc_trim(0) are called between batches.
  • The new restore-side function load_checkpoint_streamed is a restore variant that pushes stage→copy→evict tensor by tensor. auto-window = min(max(max_leaf, total/16), 32GB).
  • This function passes old/new array pairs through the on_replaced callback and separates aliased references in advance with copy_aliased_arrays.
  • The Trainer.warm_start_staging_spec hook does not make an opt_state copy in pinned-host during a params-only restore (keep_emb_opt_state=True option).
  • admission.reset_estimates resets the EWMA of the serving admission controller to zero.
  • The copy_port_client logging also records GB/s.
• ✅ valid_step gating in trainer_recsys was unified. Now if either dense or embedding fails, both keep the originals. It cuts off at the source the residual effect of dense·emb grad scale divergence on Muon+GB300.
  • The param update order was rearranged in the dense sync and async_no_deferred paths.
  • Previously dense params were updated as long as they passed the dense grad-norm check, and if the emb grad-norm failed only the emb table was rolled back, which could create an asymmetric state for a few steps.
  • The unified condition is keep_step = valid_step & emb_valid_step.
  • The async path is also synchronized with AsyncEmbGradientUpdate.pending=keep_step.
  • The grad_norm_keep_threshold is None assert in _create_async_emb_executables was removed (the async path now actually consumes the threshold too).
• ✅ An option to put follower count and bio in the grox reply_spam LM prompt was added, and Manhattan writes became monotone-only. Because the reply ranking score only goes down over time and never goes up, label flapping is eliminated at the source.
  • Two options, include_follower_count and include_bio, were added to PostRenderer.render, which renders user meta in four parts: Handle / Follower Count / Name / Bio.
  • ThreadRenderer._render_thread puts the follower count in every post of the thread when include_follower_count is on, and renders the bio for post 0 (the original) only.
  • The # Reply Author Info header states the judgment target explicitly.
  • _publish_to_reply_ranking_manhattan looks up the existing score with ReplyRankingScoreStratoLoader.fetch_reply_ranking_score. If new score > existing, it skips the write (higher_than_existing counter).
• ✅ The media fallback cache was removed entirely from VF TesHydrator. It follows the same "remove the safety net after stabilization" trend as Gen4+8's big VMRanker simplification (Gen4+9 in the change log).
  • 292 lines were deleted. The 1M-entry LRU, serve-stale/shadow mode, and both env flags (VF_MEDIA_FALLBACK_CACHE_SERVE_STALE_ENABLED, ..._POPULATE_ENABLED) are all gone.
  • The pattern is that the safety net from the early VF rewrite was removed once consistency could be audited with the Gen4+7 reference_compare harness and TES stabilized.
• ✅ dark_traffic no longer mirrors canary workloads. It prevents incidents in which canary issues contaminate shadow verdicts.
  • should_enable changed to should_mirror(workload, ordinal, max_ordinal).
  • With workload=None it is fail-closed, and workloads with a -canary suffix are not mirrored.
• ✅ Three ads conversion heads were newly added to the proto. The ranker directly predicts advertiser pixel fires, and the combination of pixel and long dwell is also a separate head.
  • They are proto ActionName 191 ADS_PIXEL_FIRE, 192 ADS_LONG_DWELL_AND_PIXEL_FIRE, and 193 P_LONG_DWELL_AND_PIXEL_FIRE.
  • They are not yet reflected in constants.py and recsys_model.py (proto introduction only). The training and serving connection is a checkpoint to watch in the next commit.
• ✅ Ads asset reuse history is carried in the serving request. It attributes cases where the same creative was shown or re-shown across multiple advertisers.
  • PredictNextActionsRequest.conv_asset_ids is connected. A new ConvAssetIds message (impressed_time_ms / author_id / asset_id 3-array) was added as field 18.
• ✅ Two fields for reconstruction metrics were added to home-mixer's local SlateContext. They refine the served_slate_context reconstruction metrics. The proto migration is pending.
  • The fields are SlateContext.recon_count_above and recon_gap_above, and both are Option<u32>.
• ✅ A new use_history_segment_ids flag was added to the retrieval user tower. It is off by default.
  • The default is RecsysTwoTowerModelConfig.use_history_segment_ids=False.
  • When on, user_tower_segment_ids(B, T, ...) gives history tokens HISTORY_SEGMENT_ID instead of zeros.
  • The segment mask of pallas varlen flash-attention isolates the user history segment from other special tokens (e.g., the user_features token).
• Deep dive needed (new):
  • The production tuning value of save_concurrent_gb and how much save time increases
  • When load_checkpoint_streamed replaces the default load_checkpoint (currently both functions coexist)
  • The first active config of warm_start_staging_spec (whether it is already reflected in the flagship)
  • The actual call trigger of admission reset_estimates (hot-swap completion? traffic shift detection?)
  • Data showing that the Manhattan reply_ranking monotone-only write actually reverses observed label flaps
  • The media feature fail-close rate when TES fails after removal of the VF media fallback cache
  • The training and serving connection of ADS_PIXEL_FIRE and ADS_LONG_DWELL_AND_PIXEL_FIRE (when constants.py is updated)
  • The actual consumption point of conv_asset_ids (which head in recsys_model uses this asset history)
  • When use_history_segment_ids is promoted to production (the commit that changes it to True in the flagship)
  • The formula for recon_count_above/recon_gap_above and when the proto migration happens

Resolved by the Gen4+10 release / new (2026-08-28)

The 2026-08-28 commit (Gen4+10) introduced the ads blender, the VF safety-label cache warmer, and Phoenix checkpoint restore and serving fixes verified (blender, warmer, Phoenix). Operational values such as rollout ratio, initial settings, and production metrics remain new open questions.

• ✅ The ads blender gained a fourth option, MultiRiskAdsBlender. It decides ad slots with an adjacency matrix that combines the ad's BSR grade and the post's risk grade.
  • The existing options are safe_gap, partition_organic, and time_gap; this commit added multi_risk.
  • Ads are split into two tiers, BsrNormal/BsrHigh, and posts into three tiers, Safe/MediumRisk/HighRisk.
  • Only BSR-high ads are allowed medium-medium slots. If an advertiser opts in to impressions in medium-risk contexts, total impressions increase.
  • A HighRisk post cannot be adjacent to any ad.
  • Three metrics, MultiRisk.enforcement, MultiRisk.slot_outcome, and MultiRisk.serving_limitation, slice two things for observation.
    • Which safety filter caused a slot to be left empty
    • Which of ads_supply/spacing/safe_posts was the binding constraint
  • The same commit also added a "partition_organic_low_risk" alias to blender_selector, bringing the options to five in total.
  • It is placed behind an experiment gate, and the default of the AdsBlenderType param has not yet been promoted.
• ✅ A warmer was added that, on a miss in the safety-label twemcache, samples a portion of the misses and fills the cache asynchronously.
  • When twemcache returns Miss (the cache does not know this key) on a visibility/baseTweetSafetyLabelMap column lookup, SampledWarmer samples with a probability set by the VF_CACHE_WARM_SAMPLE_PCT env.
  • Sampled keys are published fire-and-forget to a tokio bounded mpsc (capacity 1024).
  • A background drain groups batches with a 500ms linger and at most 50 IDs and performs a Strato batch fetch. The results are written back to twemcache.
  • NotFound (definitively absent) is not a warming target, and FallThrough (error/timeout) is not one either. The principle is that warming resolves only unfilled cache.
  • When the bounded channel is saturated, it never blocks request handling and only increments the DroppedChannelFull counter.
  • Three new envs were also introduced: VF_GIZMODUCK_CLIENT_ID, VF_TWEMCACHE_CLIENT_NAME, and APP_ENV (staging/prod are separated automatically).
• ✅ The Phoenix trainer now actually uses restore_streamed.
  • The load_checkpoint_streamed scaffolding from Gen4+9 was wired into maybe_load_checkpoint in this commit.
  • The activation condition is restore_streamed=True AND format == "orbax" AND save_method != "tensorstore".
  • The warm_start (do_not_load_opt_state) path benefits in particular. On this path, the pinned-host copy of the opt state is never created.
  • The _graft_replaced callback grafts loaded tensors in place into their original positions in the state tree. Aliased state leaves are blocked by an early assertion.
• ✅ A bug where the Phoenix gRPC compression path lost response trailers was fixed.
  • When compressing with zstd, GrpcCompressionService wrapped the response body in a new axum::body::Body::from(data) and lost all gRPC trailers (grpc-status, grpc-message).
  • Clients were likely to have treated the response as ending at EOF without a status trailer.
  • Trailers are now restored with collected.trailers().cloned() and a rebuild_response helper. This is unified for both successful and failed compression.
  • Two regression tests were added.
  • The same file gained a new histogram, recsys_engine_response_compress_ms{method} (zstd encode queuing and CPU time).
• ✅ Phoenix copy_port shuffled the shard download order per pod to even out peer load.
  • Previously, when each pod downloaded sharded tensors, all pods requested in order starting from index 0, which created a spike on the first shard peer.
  • The new shuffle_sharded_schedule(n, tensor_name) hashes POD_NAME/HOSTNAME together with the tensor name and shuffles deterministically with StdRng::seed_from_u64(hash). As a result, each pod requests in a different order.
  • Completed results are restored to the original order with restore_piece_order.
  • The two consumers, emb_table.rs and copy_port_client.rs, were unified in the same way.
  • replicated ownership keeps in-order.
  • This change extends the node-serialize and concurrent_gb hardening of Gen4+7 to the peer axis.
• ✅ home-mixer gained three counters that observe polling requests. Regular polling refreshes and a user's initial requests can now be viewed separately in serving metrics.
  • A new file, home-mixer/util/strato_context.rs (75 new lines), extracts the StratoContext.is_polling proto field from stratocontext[-bin] metadata.
  • The stat_request_context function was wired into response_stats_side_effect.rs.
  • It increments three counters per request.
    • <feed>.request.polling{polling=true|false, client}
    • <feed>.request.request_context{request_context, client}
    • <feed>.request.client{client, client_version}
• ✅ grox reply_spam now stores scores to Kafka first, and >= idempotency and fetch-failure handling also changed.
  • The monotone-only rule from Gen4+9 changed from > to >=, so writes are also skipped on ties. The counter name also changed from higher_than_existing to not_lower_than_existing.
  • The storage order was reversed from Manhattan-first to Kafka-first. Kafka append-only durability takes priority, and a Manhattan write failure can be recovered offline.
  • The defensive try/except in fetch_reply_ranking_score was removed. A fetch failure now propagates as a flow task failure.
  • In the same commit, include_follower_count=True was turned on permanently in the PostRenderer.render call in classifier_coordinated_spam.py. The coordinated spam verdict now receives each thread participant's follower count in the prompt.
• ✅ abuse-enforcement gained an AIS thrift schema conformance test. CI guarantees that AES's payload schema does not drift from the downstream thrift service.
  • The new test every_action_shape_encodes_against_the_ais_schema encodes action shapes against the actual AIS thrift schema to prevent regressions.
  • The encoded targets are eight EnforcementAction shapes built with six constructors (suspend_user, add_user_label, add_post_label, bounce_arkose, bounce_captcha, spam_liveness_check). suspend_user and add_user_label each appear in two variants verified — service.rs:2104-2139.
  • The target schema is ActionIntakeService.intakeAction.
  • When a new scorer is created or a payload field is added, this test detects the problem as an early failure.
• ✅ A regression test was added to confirm that home-mixer's CJK muted keywords match only as whole tokens.
  • The new test is test_cjk_keyword_whole_token_only in viewer_muted_keyword_filter.rs.
  • If 京都 is muted, I visited 京都 last week must be filtered out, but 東京都に行くのが楽しみ must not be (京都 is a substring of 東京都).
  • The filter itself already handles CJK word boundaries. Only the test was added this time, and its purpose is to lock down the semantics.
• ✅ The account-list comments of Brazil2026ElectionFilter were reorganized again. The filter semantics are unchanged (the follow exception is kept).
  • Previously, 665 accounts were listed with a single repeated comment such as no live account was found / deleted account.
  • This time they were reorganized into 10 "not the candidate's actual account" verdicts and many no live account found entries.
  • Separate from the list-update practice of Gen4+8, this change tidies the tone of the documentation and comments.
• ✅ A fallback was wired into the SimClustersInterestedIn2020 snapshot read in simclusters. The batch job became more tolerant of missing snapshots.
  • EntityToSimClustersEmbeddingsJob.scala directly inlined the InterestedInSources.simClustersInterestedIn2020Source(dateRange, timeZone) helper.
  • The inlined code consists of three parts.
    • DAL.readMostRecentSnapshot(..., dateRange.prepend(Days(28)(timeZone)))
    • ExplicitLocation(ProcAtla)
    • fallbackPath = "viewfs://.../simclusters_v2_interested_in_20M_145K_2020/_TMP_RECOVERY/{timestamp}"
  • It uses a 28-day lookback, a pin to a specific data center, and a fallback to a temporary recovery path together.
• The deep-dive questions newly raised by this commit are as follows.
  • The experiment rollout ratio of the multi_risk blender and the share of BSR-high opt-in advertisers
  • The uplift that medium-medium placement produces in revenue/brand safety metrics
  • The initial production value of VF_CACHE_WARM_SAMPLE_PCT and the change in twemcache miss rate
  • Which service quotas the gizmoduck_client_id() staging separation splits
  • The first active config of restore_streamed (currently presumed to default to false inferred)
  • The P99 target of recsys_engine_response_compress_ms and production observation evidence of the gRPC trailer loss bug
  • The request-rate distribution of the shard 0 peer before and after the copy_port shard shuffle was introduced
  • The actual distribution of the home-mixer polling request ratio (by feed and client)
  • Changes over time in the rate at which reply_spam skips Manhattan writes (what percentage is skipped after monotone-only and >=)
  • How often the AIS conformance test actually catches regressions in the scorer development cycle

Resolved by the Gen4+11 release / new (2026-08-28)

In the 2026-08-28 commit (Gen4+11), a reranker head tag parameter was wired into the request home-mixer sends to Phoenix, and Phoenix's embedding table receive handling was moved to a blocking pool verified (change log). The VF rule-layer refactor in the same commit did not change policy verified (change log).

• ✅ The visibility-filtering rule layer was refactored. Policy did not change, and regression-preventing assets accumulated.
  • 24 rule files were slimmed down, at +1,522/−1,151.
  • RuleContext was split into a new file, context.rs, and exposes about 25 domain methods (is_stale_tweet, viewer_country_in, has_tweet_nsfw_user_flag, etc.).
  • Condition checks in individual rules were replaced with domain method calls, and the semantics were preserved.
  • A 90-case golden corpus (golden_corpus.rs, +1039) pins verdict results across 13 categories.
  • CI enforces the every_wired_rule_decides_a_corpus_case assertion. It checks that every wired rule has at least one deciding case.
  • This is the next stage in the flow that led from Gen4+7 reference_compare to the removal of the Gen4+9 media fallback cache.
• ✅ A RerankerHeadTag parameter was wired into the request home-mixer sends to Phoenix. The server-side consumption logic is not in this commit.
  • The new parameter is param!(RerankerHeadTag, i64, "rust_home_mixer_reranker_head_tag", 0).
  • A PostCandidate.reranker_head_tag: Option<u32> field was introduced. PhoenixScorer sends this value in the request and in scoring_sequence, and it round-trips in the response.
  • It was added symmetrically in two proto places: ScoreInfo.rerankerHeadTag = 6 and UserActionSequence.rerankerHeadTag = 9 (in the Rust and Python protos together).
  • This is a wiring stage before head routing is turned on inferred.
  • It reads as a switch for choosing one of several reranker heads in the model through an A/B experiment inferred.
• ✅ Phoenix emb_table's gRPC receive handling was separated into a blocking pool.
  • Previously, when downloading large embedding tables, non-temporal copy and Adler-32 computation ran directly in the gRPC callback and held up tokio runtime threads.
  • Now data is pushed into a sync_channel(2) in 4 MiB chunks and run separately with tokio::task::spawn_blocking.
  • On the RDMA path as well, consumption up to the final summation is done with spawn_blocking.
  • The new helper JoinOnDrop<T> always terminates the blocking task and prevents handle leaks. The normal path uses join().await, and on drop it uses block_in_place and block_on(handle).
  • The 4 MiB chunks and sync_channel(2) backpressure regulate the receive rate.
• ✅ copy_port_client gained a join_transfers helper, and proto_parser gained a Bytes callback type.
  • The previous four-line join_all(...spawn) → collect → map_err idiom was extracted into a single join_transfers helper (no behavior change).
  • The new Bytes callback type in proto_parser.rs (FnMut(usize, Bytes)) can pass chunks by ownership transfer.
  • emb_table still keeps the Str signature. True zero-copy is a follow-up checkpoint to watch.
• The deep-dive questions newly raised by this commit are as follows.
  • The server-side consumption logic of RerankerHeadTag (head routing in the Phoenix Python serving code)
  • Why it was attached in two places, UserActionSequence and ScoreInfo (request/response symmetry vs. a different purpose)
  • When RerankerHeadTag is first promoted in an experiment (the commit where it changes from 0 to 1+)
  • When emb_table consumes the Bytes callback (the true zero-copy path)
  • The tail latency improvement of the tokio runtime before and after the gRPC callback separation was introduced
  • Verifying that the 90 golden corpus cases match prior production verdicts (the possibility that the refactor hid a semantic change)

Resolved by the Gen4+15 release / new (2026-09-04)

The 2026-09-04 commit (Gen4+15) changed grox reply spam verdicts, PTOS judge model assignment, Phoenix serving and training, and VF hydration all at once verified (change log). The new questions include the activation ratio of the new verdict path, the point where SID is filled into the request, and the H100 training config.

• ✅ grox gained a second reply spam path that judges per thread chain. It is named PlanMultiStepReplySpam.
  • It feeds the whole thread into Gemma 2 as a prompt and extracts a "spam_post_indexes" set.
  • There are eight entry conditions: it is a reply, ancestors are present, root followers ≥ 1,000, root author ≠ reply author, direct parent author = reply author, not high page-rank/grey badge, not Grok/Gork, and depth ≥ 2.
  • The pattern these conditions try to catch is a time-axis coordinated spam where "one author steals root impressions through a chain in which they reply to their own replies."
  • The LM output has a three-step safeguard: newest reply not flagged → drop, other-author → drop, singleton → drop.
  • If it judges a post as spam, it forcibly stores score = 0.0.
  • The plan is injected into two streams, ReplyRankingTaskGenerator and RecoveryTaskGenerator.
  • A time-axis collusion axis, alongside Coordinated_spam (account collusion), is added to the canon.
• ✅ The first concrete case of the PTOS judge stack using a different model per category was confirmed.
  • Gen4+7 assigned a single model, EAPI_GROK_4_5_X_ALGO, to the SafetyPtosPolicyCrossValidator of three categories (ChildSafety, ViolentMedia, IllegalAndRegulatedBehaviors).
  • In this commit, only IllegalAndRegulatedBehaviors was migrated to EAPI_GROK_4_1_FAST_X_ALGO.
  • This call is made directly without a Breaker, and the metric name also changed to ..._with_grok_4_1.
  • The policy of X tuning judge models separately per category was stated explicitly.
• ✅ Gemma 4 was promoted to first-pass for IllegalAndRegulatedBehaviors, and the Gemma probability dial was removed.
  • USE_GEMMA_CATEGORIES = {Spam, IllegalAndRegulatedBehaviors} (previously Spam only).
  • The if random.random() < self.use_oai_gemma4_dial probability gate was removed entirely, so both categories are routed 100% to Gemma.
  • The removed dial value had been 1.0 since the Gen4 release, so Spam behavior did not change verified — classifier.py in 47c1bcd and 85ac72a. The actual change is that IllegalAndRegulatedBehaviors moved to Gemma.
  • task_safety_ptos_policy.py also added IllegalAndRegulatedBehaviors to the reclassification trigger. This completes the three-stage stack (Gemma first-pass → deluxe cross-model → Grok 4.1-fast judge).
• ✅ The SID retrieval sidecar was removed completely from Phoenix serving. History SIDs now arrive carried in the request.
  • Five things were deleted.
    • sid_client.rs (189 lines)
    • sid_retrieval_runner.py (17 lines)
    • --sid_endpoint CLI
    • The xai-recsys-sid-proto dependency
    • The PySemanticIdClient Python binding
  • The log keeps the phrase "Parsing history SIDs from the request (sid_num_levels=%d); no sid_endpoint".
  • The responsibility for SID codeword lookup moved out of online serving. The contract is that home-mixer (or the candidate hydration layer) already puts the SIDs in when it builds the request.
  • One gRPC call per request is eliminated, and the fault domain of the SID infrastructure shrinks.
• ✅ The default of Phoenix use_pinned_d2h changed from False to True.
  • D2H transfer speed is ~50 GB/s with CUDA pinned host memory, versus ~3 GB/s for the JAX default. It saves ~29ms per inference.
  • This option was introduced in Gen4+6, and its default changed in this commit.
• ✅ Phoenix training gained a remat policy for H100. It is the first code signal that training also runs on H100 besides GB300.
  • The new policy is SAVE_H100_RECSYS = 32, and it entered custom_remat_policy as a new branch.
  • It designates the same name list as GB300 (attn_outputs, dense_outputs, gate_up_proj, query_heads, key_heads, value_heads, etc.) as rematerialization save targets.
  • A hardware-specific config block such as _H100_OVERRIDES is not yet in this commit.
  • The fact that X's training hardware is not only GB300 is itself new content to fold into the canon.
• ✅ The FA4 attention kernel now natively supports variable-length history.
  • hist_valid_len was added to the build_dense_block_sparse_layout signature.
  • Layout computation moved from numpy to jax.numpy (cache removed).
  • Two masking fields, valid_block_upper and valid_block_lower, were added so the kernel handles partial-history blocks directly. A partial-history block is a block whose valid tokens inside a tile are at most the tile size.
  • Training and inference accuracy improve for new users and short sessions.
  • The varlen semantics of Gen4+6 cutedsl_ranker_varlen_attn were extended to the history axis.
• ✅ The hydration and safety-label code in visibility-filtering was simplified considerably. Over 800 lines were removed in net.
  • The reduced files include registry.rs −378, safety_label_source/codec.rs −206, hydration/mod.rs −90, hydration/fallback_cache.rs −87, and config.rs −118.
  • The flow of "removing the safety net after the rewrite stabilized" continues. The earlier steps are the removal of the media fallback cache in Gen4+9, the deletion of 128 lines of VMRanker in +8, the rules refactor in +11, and the elimination of Box<dyn Rule> in +14.
  • In the same commit, fixtures.rs +81 and golden_corpus.rs +102 made the regression safety net thicker.
• ✅ Brazil2026ElectionFilter was refreshed on schedule. The regular refresh practice of Gen4+8 continues. It is a signal that verification of politician X account mappings continues every week.
  • 166 account IDs were added as hardcoded entries, taking the set from 2,388 to 2,554 verified — brazil_2026_election_filter.rs. Examples of added accounts are +@nanacachen, +@AecioNeves, +@lgmbrasilia, +@crismonteirosp, +@rodrigolimamdh, +@samiabomfim, and +@gabrielmagno_13.
  • A few more "not the candidate's actual account" verdicts were also added.
• ✅ PR #88 (e4dcedd) from an external contributor was merged. It is evidence that Gen4's open-source stance actually accepts external PRs.
  • Two lines were added to vf_candidate_hydrator.rs that dedup in_network_ids before the VF lookup.
• The deep-dive questions newly raised by this commit are as follows.
  • The initial active traffic ratio and verdict metrics of PlanMultiStepReplySpam (experiment gating is not in this diff)
  • The tuning history of FOLLOWER_COUNT_THRESHOLD_FOR_SPAM_DETECTION = 1000
  • How multi-step verdict results are reflected in reply ranking in experiments
  • The name and source of the hydrator that fills SID in home-mixer or the candidate hydration layer (the point that puts SID into the request after sid_client was removed)
  • The SID codeword storage backend (Manhattan? Strato? phoenix-rankall itself?)
  • The first active config of SAVE_H100_RECSYS (which recipe is trained on H100)
  • When _H100_OVERRIDES appears and the batch/attn/remat tuning differences between H100 and GB300
  • What the 206-line deletion in safety_label_source/codec.rs actually removed (which wire format or label source)
  • How EAPI_GROK_4_1_FAST_X_ALGO differs from 4.5 X_ALGO (latency, cost, and accuracy trade-offs)
  • The next category expansion candidates (the next Gemma promotion targets after Spam and IllegalAndRegulatedBehaviors, and when the ChildSafety and ViolentMedia judges move to 4.1-fast)

↑ Contents

Sources

We checked the git log of xai-org/x-algorithm (2026, public source code, Apache 2.0) directly as of 2026-10-21. Up to that point there are 46 commits after Gen4, and the most recent commit is the 2026-10-20 commit (Gen4+45).

The sections and table rows below each have their own reference commit.

• visibility-filtering-client/vf_client.rs — 2026-09-09, +130/−62. Introduced the TweetVisibility.action field and the TweetVisibility::to_visibility_reason() helper. action_from_strato_reason() converts Strato non-SafetyResult reasons to Drop and promotes NotEvaluated to Allow.
• visibility-filtering-client/models.rs — 2026-09-09, +7. Added the FilteredReason::ViewerBlocksAuthor variant and proto to/from mappings.
• home-mixer/filters/vf_filter.rs — 2026-09-09, +27/−69. should_drop_action drops only Drop(_), Tombstone, and NotEvaluated, and keeps Allow, Interstitial, Avoid, and Downrank. The tests were rewritten as an 8-case partition matrix.
• home-mixer/candidate_hydrators/vf_candidate_hydrator.rs — 2026-09-09, +159/−23. Introduced the visibility_fields helper and rewrote should_drop_ancillary to be action-based.
• home-mixer/util/urt/ad_marshaller.rs — 2026-09-09, +258/−3. Introduced marshal_header_image_prompt_ad and the HeaderImagePromptAdInfo entry branch. The entry namespace is header-image-prompt-ad, and it was connected to URT MessageContent::HeaderImagePrompt. It supports a secondary CTA and has three tests.
• grox/core/data_loaders/media_loader.py — 2026-09-09, +100/−19. Put a four-gate check for videos of 30 seconds or longer in hydrate_video_preview. _extract_broadcast_video normalizes frames with BroadcastLoader.load_broadcast, and _get_broadcast_loader uses lazy init.
• grox/core/lm/convo.py — 2026-09-09, +18. Added the Video.preview_image: bytes | None field and the PREVIEW_IMAGE_LABEL = "[Video Preview Thumbnail]" constant. interleave_storyboard and _image_part insert the preview image at the front.
• grox/flows/ptos/task_safety_ptos_adult_content_leading_frames_filter.py — 2026-09-09, +18/−8. Removed the deluxe fav gate and formally allowed broadcast entry. eligible.count is tagged with media = both|broadcast|video.
• grox/flows/ptos/task_safety_ptos_safemodel_sex_nudity.py — 2026-09-09, +21/−2. Added the _FRAME_KINDS = {video_frame, broadcast_frame} constant and _iter_broadcast_videos. Also added a branch gated by enable_local_broadcast_frame_extraction.
• phoenix/README.md — 2026-09-09, +3/−3. Fixed the Multimodal post embedding row of the nano config table.
  • It changed from v5 on xrecsys_seqpack; off on home_direct_packed to off (home_direct_packed and xrecsys_seqpack).
• phoenix/xrex/utils/gpu.py — 2026-09-09, +32/−3. Introduced _parse_cpulist and schedulable_cpus(*nodes). pin_process_to_local_numa excludes isolated CPUs via os.sched_setaffinity(0, schedulable_cpus(node)).
• home-mixer/scorers/ranking_scorer.rs — 2026-09-08, +93/−29. Introduced ScoringWeights::perturbed(), perturbation_sign(), and weights_mut(). The weights of 26 engagement heads are scaled per request by multiplying by exp(σ×md5-sign), and the sums are recomputed with recompute_sums().
• home-mixer/params/param.rs — 2026-09-08, introduced two parameters. The sync time in the header is 2026-09-08T16:24:06Z.
  • WeightPerturbationSigma: f64, default 0.0
  • WeightPerturbationSalt: String, default ""
• grox/flows/mm_emb/embedder_v8_search.py — 2026-09-08, 113 lines, new. Contains MultimodalPostEmbedderV8Search and MultimodalPostEmbedderV8SearchBackfill.
• grox/flows/mm_emb/renderer_v8_search.py — 2026-09-08, 461 lines, new. Contains V8SearchEmbedPostRenderer.render_for_embedding.
  • The rendering layout is an instruction prefix, an author block (follower humanize and tier), 4000 characters of body, 1000 characters of quoted text, 2000 characters of ASR, and a 16-frame media budget.
• grox/flows/mm_emb/adaptive_min_fav.py — 2026-09-08, 63 lines, new. A PI controller observes the target QPS and adjusts the fav threshold. The window is 30 seconds, the deadband is 0.1, and the gain is 0.3.
• grox/flows/mm_emb/task_backfill_fanout.py — 2026-09-08, 85 lines, new. Contains TaskBackfillBatchFanout, Semaphore(240), BackfillConsumerDedup, and BatchedFavHydrator.
• grox/flows/mm_emb/task_backfill_gate.py — 2026-09-08, 60 lines, new. Contains TaskBackfillFavGate and TaskBackfillPreloadGate. The latter gate skips posts already embedded with V8_SEARCH_MODEL_VERSION="v8_search_1".
• grox/flows/mm_emb/plan_post_embedding_v8_search.py, plan_post_embedding_v8_search_for_reply.py, plan_post_embedding_v8_search_backfill.py, plan_post_embedding_v8_search_backfill_batch.py — 2026-09-08, 130 lines in total, new. These are four Plans.
• grox/flows/mm_emb/backfill_consumer_dedup.py, batched_fav_hydrator.py — 2026-09-08, 114 lines in total, new.
• grox/flows/mm_emb/constants.py — 2026-09-08, +9 lines. Added constants.
  • POST_EMBEDDING_V8_SEARCH_STREAM·POST_EMBEDDING_V8_SEARCH_FOR_REPLY_STREAM·POST_EMBEDDING_BACKFILL_STREAM·TOPIC_EMBEDDING_V8_SEARCH·RECSYS_V8_SEARCH_EMBED·RECSYS_V8_SEARCH_BACKFILL_EMBED·V8_SEARCH_STATE_KEY·V8_SEARCH_MODEL_VERSION
• phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto, phoenix/python/common/xai-proto/proto/recsys.proto — 2026-09-08, added the ContentFeatures message and the TweetInfo.contentFeatures=36 and quotedContentFeatures=37 fields to both proto files at the same time.
• home-mixer/models/content_features.rs — 2026-09-08, 185 lines, new. Contains build, build_quoted, and TextMetrics::of. Five regression tests pin down the Twitter text length rules (URL=23, CJK=2, ZWJ/skin-tone modifier=0).
• grox/flows/ptos/plan_safety_ptos_adult_content_leading_frames.py — 2026-09-08, 56 lines, new. It is an 8-stage DAG.
• grox/flows/ptos/task_safety_ptos_adult_content_leading_frames_filter.py — 2026-09-08, 81 lines, new. The entry filter has 6 stages (author exists, deluxe fav, video of 30s+, prior PTOS passed, non-NSFW, prior_adult or fav≥512). It sets video.crop_seconds = 10.0.
• grox/flows/ptos/constants.py — 2026-09-08, added constants.
  • SAFETY_PTOS_ADULT_CONTENT_LEADING_FRAMES·TOPIC_ADULT_CONTENT_LEADING_FRAMES·ADULT_CONTENT_LEADING_FRAMES_CROP_SECONDS=10.0·MIN_DURATION_SECONDS=30.0·UNCONDITIONAL_FAV=512
  • DELUXE_TIER_TASK_TYPES also gained a new entry.
• phoenix/xrex/cutedsl/ranker_fa4/flash_fwd.py, flash_fwd_sm90.py, flash_bwd.py, flash_bwd_sm90.py, flash_bwd_preprocess.py, ampere_helpers.py — 2026-09-08, 5,080 lines in total, new. This is code forked from the original Dao-AILab flash-attention. It released the SM80/SM90 forward and backward fully as open source.
  • The license notice is SPDX Apache-2.0, Copyright 2026 X.AI Corp.
• phoenix/xrex/cutedsl/ranker_fa4/block_sparse_utils.py, ranker_attention_fa4.py, ranker_attention_varlen_fa4.py, mask.py — 2026-09-08, major changes to 4 files (+926/−656). The changes cover the block-sparse iterator, the fwd entry point, varlen cleanup, and the AttentionMask backward extension.
• phoenix/NOTICE — 2026-09-08, +198 lines. Added copyright notices for the flash-attention subcomponents.
• phoenix/xrex/utils/aot.py — 2026-09-08, +54/−1. Contains _sanitize_cutlass_calls and cutlass_module_device_digest. The latter function parses the ELF header and then computes the SHA256 of the device blob. AOT_CACHE_VERSION changed from "20260705"→"20260904".
• phoenix/crates/serving/xai-recsys-engine/src/copy_port_client.rs — 2026-09-08, +207/−9. Contains full_prefixes_newest_first, select_bundle_checkpoint, and download_named_files. Only a checkpoint that has MANIFEST.json is treated as a completed bundle and selected.
• phoenix/xrex/train/recsys_bundle_export.py — 2026-09-08, +48. Introduced _serving_sequence_len and added a model_sequence_len field to the MANIFEST.
• home-mixer/candidate_hydrators/quote_hydrator.rs — 2026-09-08, +142/−12. It injects MediaInfoCacheClient. get_quoted_content consists of 200ms parallel hydration of core data and media info, plus QuickCache.
• home-mixer/candidate_hydrators/media_info_hydrator.rs — 2026-09-08, +70/−15. Refactored the media info lookup.
• home-mixer/bin/pipeline_components_json.rs — 2026-09-08, deleted the entire file (76 lines).
• grox/flows/reply_spam/task_multi_step_reply_spam.py — 2026-09-04, 206 lines, new. Contains TaskMultiStepReplySpamFilter (8-axis gating), TaskMultiStepReplySpamDetection, and TaskWriteMultiStepReplySpamReplyRanking.
• grox/flows/reply_spam/plan_multi_step_reply_spam.py — 2026-09-04, 29 lines, new. It is a 4-task DAG running filter → media hydration → detection → write.
• grox/flows/reply_spam/classifier_multi_step_reply_spam.py — 2026-09-04, 226 lines, new. Contains MultiStepReplySpamScorer (Gemma 2, _thread_posts MAX_THREAD_SIZE=10, 5 before and 5 after). Post-processing has 3 stages: _drop_if_newest_reply_not_flagged, _drop_other_author_posts, and _drop_singleton.
• grox/flows/reply_spam/state_multi_step_reply_spam.py — 2026-09-04, 13 lines, new. It is the MultiStepReplySpamResult pydantic model.
• grox/flows/reply_spam/prompts.py — 2026-09-04, added multi_step_reply_spam_system_prompt(). The original prompt text was excluded from the export to prevent gameability.
• grox/flows/reply_spam/generators.py — 2026-09-04, added PlanMultiStepReplySpam.KEY to PLANS_TO_INJECT of ReplyRankingTaskGenerator and ReplyRankingRecoveryTaskGenerator.
• grox/flows/ptos/classifier.py — 2026-09-04, registered the EAPI_GROK_4_1_FAST_X_ALGO sampler. Moved _validate_illegal_and_regulated_behaviors from 4.5 to 4.1-fast. Added IllegalAndRegulatedBehaviors to USE_GEMMA_CATEGORIES and removed the use_oai_gemma4_dial probability gate.
• grox/flows/ptos/task_safety_ptos_policy.py — 2026-09-04, added IllegalAndRegulatedBehaviors to the reclassification trigger of classify_policy_for_violation.
• phoenix/crates/serving/xai-recsys-engine/src/sid_client.rs — deleted on 2026-09-04 (−189 lines). The link points to the previous commit.
• phoenix/xrex/inference/model_runner.py — 2026-09-04, removed the sid_endpoint field and the sid_client assembly code in create_server. The use_pinned_d2h default changed from False → True.
• phoenix/xrex/inference/launch_inference.py — 2026-09-04, removed the --sid_endpoint CLI argument and changed the --use_pinned_d2h default to True.
• phoenix/crates/serving/xai-recsys-engine/src/python.rs — 2026-09-04, removed the sid_client: Option<Arc<SemanticIdClient>> field from two structs. Also removed the server_impl macro argument and the add_class::<PySemanticIdClient>() registration.
  • The two structs are RecsysPredictorImpl and RecsysRetrievalPredictorImpl.
• phoenix/crates/serving/xai-recsys-engine/Cargo.toml — 2026-09-04, removed the xai-recsys-sid-proto dependency.
• phoenix/xrex/models/remat.py — 2026-09-04, introduced RematType.SAVE_H100_RECSYS = 32 and added a branch to custom_remat_policy.
• phoenix/xrex/cutedsl/ranker_attention_fa4.py — 2026-09-04, added hist_valid_len to the signature of build_dense_block_sparse_layout and removed batch_size. Moved it to jax.numpy and added the valid_block_upper and valid_block_lower fields. Removed _DENSE_BS_LAYOUT_CACHE.
• visibility-filtering/rules/registry.rs — 2026-09-04, +8/−378. Deleted a large amount of code.
• visibility-filtering/safety_label_source/codec.rs — 2026-09-04, +15/−206. Unified the wire format.
• visibility-filtering/hydration/mod.rs — 2026-09-04, +11/−90. Removed the hydration entry.
• visibility-filtering/hydration/fallback_cache.rs — 2026-09-04, +25/−87. The cache layer became thinner.
• visibility-filtering/config.rs — 2026-09-04, +7/−118. Unified the config layer.
• home-mixer/filters/brazil_2026_election_filter.rs — 2026-09-04, +339/−4. This is a regular update. It added 166 account IDs (2,388 → 2,554), with a few "not the candidate's actual account" verdicts and a small number of "no live account found".
• home-mixer/candidate_hydrators/vf_candidate_hydrator.rs — 2026-09-03, PR #88 (external contribution, joshs1017dev). Added two lines that dedup in_network_ids before the VF lookup.
• home-mixer/server.rs — 2026-09-02, hardcoded query.return_backbone_scores = true at 5 gRPC feed service entry points.
  • The entry points are ScoredPosts, ForYouFeed, ForYouFeed polling, RankedFollowingFeed, and FollowingFeed.
• home-mixer/models/candidate.rs — 2026-09-02, introduced the PostCandidate.backbone_scores: Option<PhoenixScores> field.
• home-mixer/models/query.rs — 2026-09-02, introduced the ScoredPostsQuery.return_backbone_scores: bool field (default false).
• home-mixer/scorers/phoenix_scorer.rs — 2026-09-02, consumes predictions.candidate_backbone_scores().
• home-mixer/util/phoenix_request.rs — 2026-09-02, passes the return_backbone_scores field from the request context to the proto.
• phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto — 2026-09-02, contains proto fields.
  • PredictNextActionsRequest.returnBackboneScores = 20, NextActionDistribution.backboneTopLogProbs = 10, .backboneContinuousValues = 11
• home-mixer/params/param.rs — 2026-09-02, the PhoenixRetrievalAggregationType default changed from DENSE_WITH_SHORT_DWELL to DENSE_WITH_LONG_DWELL. The sync time in the header is 2026-09-02T16:03:45Z.
• phoenix/crates/serving/xai-recsys-engine/src/python.rs — 2026-09-02, added a zero-fill guard and a copy_len > 0 condition to candidate/search-query embedding.
• scarecrow/legacy/XReviewIntakeRegistry.scala — 2026-09-02, 179 lines, new. Contains the CreateXReviewReportProd/Staging function units and the mTLS client provider.
• scarecrow/legacy/XReviewReportBuilder.scala — 2026-09-02, 59 lines, new. Contains the SubmitReportRequest payload schema.
  • The fields are report_source="proactive", subject="Botmaker report", reporter_id="0", detection_bot_id, and detection_timestamp_ms.
  • The entityType whitelist is "post"/"profile".
• scarecrow/legacy/XReviewReportSubmitter.scala — 2026-09-02, 91 lines, new. Contains dual prod/staging clients, 5 kinds of counters, and a noop fallback.
• scarecrow/legacy/DownstreamServices.scala — 2026-09-02, registered XReviewIntake as a DownstreamService.
• visibility-filtering/rules/registry.rs — 2026-09-02, contains the Policy struct and three static policies. Policies was renamed RuleEngine.
  • The three policies are FILTER_ALL_POLICY, TIMELINE_HOME_POLICY, and TIMELINE_HOME_RECOMMENDATIONS_POLICY.
• visibility-filtering/rules/mod.rs — 2026-09-02, deleted the Rule trait and the evaluate_rules function entirely (−155 lines).
• visibility-filtering/rules/tweet_rules.rs — 2026-09-02, added a new test, wired_rule_names_are_unique_and_nonempty. It verifies that the names of the 15 rule slices are unique.
• phoenix/python/training/xai-checkpointing/xai_checkpointing/dek.py — 2026-09-02, +91/−2. Contains adopt_tree_dek (sidecar or in-envelope fallback) and _header_wrapped_master. The latter function uses a 4096-byte envelope header, the magic XAIENC01, and the derivation form xai-dek1:<salt>:<master>. It also expanded rank_logger observation.
• phoenix/python/training/xai-checkpointing/xai_checkpointing/load.py — 2026-09-02, the reader uses the xai_encrypted tensorstore driver directly. It removed the KMS-proxy KvStore, unwraps the DEK once, and then decrypts symmetrically in local memory.
• phoenix/python/training/xai-checkpointing/xai_checkpointing/orbax_encrypted.py — 2026-09-02, added decrypt_read and wait_until_finished.
• phoenix/xrex/train/checkpoint_write.py — 2026-09-02, domain and run became required in encryption_context, and mint_nonce = sha256(path) is inserted automatically. It also made the downgrade-prevention message more specific.
• phoenix/xrex/train/trainer.py — 2026-09-02, _uses_tensorstore_save is no longer coupled with encrypt.
• phoenix/xrex/utils/checkpointing.py — 2026-09-02, _AsyncCheckpointer waits up to orbax_encrypted.wait_until_finished(). save_checkpoint accepts the kms_client/encryption_chunk_size arguments.
• phoenix/xrex/configs/xrecsys_two_tower.py — 2026-09-02, changed qk_norm: False → True and attn_logit_cap: 80.0 → -1.
• visibility-filtering/rules/context.rs — 2026-08-28, 214 lines. Introduced about 25 domain methods on RuleContext.
• visibility-filtering/rules/golden_corpus.rs — 2026-08-28, 1039 lines, new. Contains a regression corpus of 90 cases in 13 categories and the every_wired_rule_decides_a_corpus_case assertion.
• visibility-filtering/rules/fixtures.rs — 2026-08-28, 116 lines, new (#[cfg(test)]). Contains CandidateBuilder and viewer helpers.
• visibility-filtering/rules/registry.rs — 2026-08-28, +49/−298. Introduced the wired_rule_names(level) helper and moved the individual rule tests.
• home-mixer/params/param.rs — 2026-08-28, introduced the RerankerHeadTag parameter (default 0).
• phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto — 2026-08-28, contains ScoreInfo.rerankerHeadTag = 6 and UserActionSequence.rerankerHeadTag = 9.
• phoenix/crates/serving/xai-recsys-engine/src/emb_table.rs — 2026-08-28, +79/−12. Split the gRPC receive → destination copy path out with spawn_blocking, and contains JoinOnDrop<T>.
• phoenix/crates/serving/xai-recsys-engine/src/copy_port_client.rs — 2026-08-28, +16/−15. Removed the duplicated idiom with the join_transfers helper.
• phoenix/crates/serving/xai-recsys-engine/src/proto_parser.rs — 2026-08-28, +11. Introduced the Bytes callback type.
• home-mixer/ads/multi_risk_blender.rs — 2026-08-28, 332 lines. Introduced MultiRiskAdsBlender. The structure is an adjacency matrix of ads tier × post tier.
• home-mixer/ads/tests/multi_risk_blender_tests.rs — 2026-08-28, 215 lines. Regression tests for the adjacency matrix.
• home-mixer/selectors/blender_selector.rs — 2026-08-28, connected the "multi_risk" and "partition_organic_low_risk" aliases to the AdsBlenderType param.
• visibility-filtering/safety_label_source/warmer.rs — 2026-08-28, 203 lines, new. Contains SampledWarmer and StratoWarmFetcher.
• visibility-filtering/safety_label_source/lookup.rs — 2026-08-28, +84. Contains RemoteSource::with_warmer and wired in the Miss-only warm condition.
• visibility-filtering/config.rs — 2026-08-28, introduced four env variables: VF_CACHE_WARM_SAMPLE_PCT, VF_GIZMODUCK_CLIENT_ID, VF_TWEMCACHE_CLIENT_NAME, and APP_ENV.
• phoenix/xrex/train/trainer.py — 2026-08-28, +102/−31. Wired it to actually consume restore_streamed, and contains the _graft_replaced callback.
• phoenix/python/training/xai-checkpointing/xai_checkpointing/load.py — 2026-08-28, extended the arguments of load_checkpoint_streamed to window_gb, window_cap_gb, and on_replaced.
• phoenix/crates/serving/xai-recsys-engine/src/grpc_compression.rs — 2026-08-28, +109/−13. Fixed a gRPC trailer preservation bug and added the recsys_engine_response_compress_ms histogram.
• phoenix/crates/serving/xai-recsys-engine/src/copy_port_client.rs — 2026-08-28, +113/−10. Contains shuffle_sharded_schedule and restore_piece_order (peer-load equalization).
• phoenix/crates/serving/xai-recsys-engine/src/emb_table.rs — 2026-08-28, consumes shuffle_sharded_schedule.
• home-mixer/util/strato_context.rs — 2026-08-28, 75 lines, new. Extracts the StratoContext.is_polling proto field.
• home-mixer/side_effects/response_stats_side_effect.rs — 2026-08-28, contains three kinds of counters: <feed>.request.polling|request_context|client.
• grox/flows/reply_spam/task_write.py — 2026-08-28, changed the comparison operator from > to >= and made the save order Kafka-first. Removed the try/except.
• grox/flows/reply_spam/classifier_coordinated_spam.py — 2026-08-28, always enabled PostRenderer.render(..., include_follower_count=True).
• abuse-enforcement-service/service-lib/src/service.rs — 2026-08-28, +58. Introduced an AIS thrift schema conformance test.
• home-mixer/filters/viewer_muted_keyword_filter.rs — 2026-08-28, +18. Introduced a CJK whole-token matching regression test.
• home-mixer/filters/brazil_2026_election_filter.rs — 2026-08-28, reorganized the account list comments again. The filter semantics are unchanged.
• simclusters/simclusters_v2/scalding/embedding/EntityToSimClustersEmbeddingsJob.scala — 2026-08-28, inlined the SimClustersInterestedIn2020 snapshot fallback path.
• README.md #latest-updates — latest as of 2026-08-21. The README itself was not updated in Gen4+4, +5, or +6. The August 14th entry from Gen4+1 is still the latest post.
• phoenix/xrex/utils/checkpointing.py — 2026-08-26, +215/−6. Introduced ThrottledD2HArrayHandler, _NodeBatchLock (save), and the save_concurrent_gb argument.
• phoenix/python/training/xai-checkpointing/xai_checkpointing/load.py — 2026-08-26, +311/−88. Introduced load_checkpoint_streamed. It contains auto-window and the on_replaced callback, and the helpers were refactored as well.
• phoenix/xrex/train/trainer.py — 2026-08-26, contains the warm_start_staging_spec hook and the RecsysTrainer.keep_emb_opt_state option.
• phoenix/xrex/train/trainer_recsys.py — 2026-08-26, does lock-step gating with keep_step = valid_step & emb_valid_step (+29/−19). Removed the grad_norm_keep_threshold is None assert.
• phoenix/xrex/models/recsys_two_tower_model.py — 2026-08-26, contains the use_history_segment_ids config flag and the user_tower_segment_ids utility.
• phoenix/crates/serving/xai-recsys-engine/src/admission.rs, request_metrics.rs — 2026-08-26, admission reset_estimates zeroes the EWMA and syncs the Prometheus gauge.
• phoenix/crates/serving/xai-recsys-engine/src/copy_port_client.rs — 2026-08-26, the dense and emb_table load logs now record total bytes together with GB/s.
• phoenix/python/common/xai-proto/proto/recsys.proto — 2026-08-26, contains ActionName 191 ADS_PIXEL_FIRE / 192 ADS_LONG_DWELL_AND_PIXEL_FIRE / 193 P_LONG_DWELL_AND_PIXEL_FIRE. It also has the ConvAssetIds message, PredictNextActionsRequest.conv_asset_ids = 18, and SlateContext.reconCosMilli = 13.
• home-mixer/models/candidate.rs — 2026-08-26, contains two Option<u32> fields, SlateContext.recon_count_above and recon_gap_above.
• grox/core/lm/post.py, grox/core/lm/thread.py — 2026-08-26, contain the include_follower_count and include_bio options and the # Reply Author Info header in PostRenderer/ThreadRenderer.
• grox/flows/reply_spam/task_write.py — 2026-08-26, made the Manhattan write monotone-only (skipping higher_than_existing).
• visibility-filtering/hydration/tes_hydrator.rs, config.rs, server_deps.rs, hydration/mod.rs — 2026-08-26, −323 in total. Removed the TesHydrator media fallback cache and two env flags entirely.
• visibility-filtering/dark_traffic_setup.rs — 2026-08-26, extended should_mirror with a canary opt-out and fail-closed handling of a missing workload.
• phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto — 2026-08-21, introduced 20 ads conversions in ActionName 171–190 and added the NextActionDistribution.slateContext = 9 field.
• phoenix/xrex/data/recsys/constants.py — 2026-08-21, introduced MACT_IN_APP_LOSS_ACTION_INDICES, VIEW_THROUGH_ACTION_INDICES, SOURCE_SPLIT_CONVERSION_HEAD_INDICES, and ads_slim_engagement_to_action_types.
• phoenix/xrex/models/recsys_model.py — 2026-08-21, contains split_head_training_by_source, train_view_through_heads, mact_in_app_loss_weight, conversion_keep_mask, and a branch for the delayed feedback loss mask.
• phoenix/xrex/data/recsys/recsys_batch.py — 2026-08-21, wired in sample_source (is_delayed_feedback). Also wired in conversion_keep_mask (conversionKeepMask).
• phoenix/TRAINING.md — 2026-08-21, organized Muon flagship and nano training into an official document.
• phoenix/xrex/configs/xrecsys.py — 2026-08-21, _home_direct_packed_nano inherits the LR and optimizer settings of _GB300_OVERRIDES, which have existed since the Gen4 release.
• phoenix/crates/storage/xai-o2/ — 2026-08-21, BaseO2Client trait and O2ClientBuilder (154 lines, new; implementation undisclosed).
• grox/flows/reply_spam/task_filter.py — 2026-08-21, follower threshold 60k → 80k.
• visibility-filtering/dark_traffic_setup.rs — 2026-08-21, dark_traffic target switched from shadow to staging.
• grox/flows/ptos/classifier.py — 2026-08-21, Grok 4.5 internal removed entirely; Grok 4.6 internal switched to 100%.
• phoenix/xrex/train/trainer_recsys.py — 2026-08-21, retrieval post-embedding chunked into 65k rows.
• phoenix/xrex/inference/model_runner.py — 2026-08-21, introduced _glibc_malloc_trim() and _malloc_trim_after_swap() (fixed a hotswap leak of ~196MiB/cycle).
• phoenix/xrex/configs/xrecsys_two_tower.py — 2026-08-21, added three options: qk_norm=True, attn_logit_cap=-1, and right_anchored_rope=True.
• phoenix/xrex/data/retrieval_dataset.py — 2026-08-21, narrowed the EVERGREEN files from evergreen_video_1825day to video_4to14day.
• home-mixer/candidate_hydrators/ai_trend_feedback_context_hydrator.rs — 2026-08-20, new AI trend feedback context (157 lines).
• home-mixer/util/urt/post_marshaller.rs — 2026-08-20, fallback rendering for topic feedback / AI trend.
• home-mixer/models/candidate.rs — 2026-08-20, served_slate_context and ai_trend_name/ai_trend_id fields, and a From<xai_recsys_proto::SlateContext> conversion impl.
• home-mixer/scorers/phoenix_scorer.rs — 2026-08-20, wired it to extract and use candidate_slate_context().
• home-mixer/scorers/ranking_scorer.rs — 2026-08-20, served_slate_contexts() and a 3-way fallback.
• home-mixer/params/param.rs — 2026-08-20, new params UseServedSlateContext and EnableAiTrendFeedbackContext.
• phoenix/crates/serving/xai-recsys-engine/src/emb_table.rs — 2026-08-20, split out the load_tensor_into core (+189/−169).
• grox/flows/ptos/classifier.py — 2026-08-20, Grok 4.6 dial 0.1 → 0.3.
• grox/libs/grok_sampler/llm.py — 2026-08-20, introduced stop_predicate and aclosing.
• grox/core/data_loaders/data_types.py — 2026-08-20, introduced the User.profile_image field.
• phoenix/xrex/optimizers/recsys/muon.py — 2026-08-19, new Muon optimizer (240 lines, wraps optax.contrib._muon).
• phoenix/xrex/optimizers/recsys/dense_optim.py — 2026-08-19, exposed the Muon config via RecsysDenseOptimConfig.
• phoenix/xrex/configs/xrecsys.py — 2026-08-19, promoted Muon in _home_direct_packed_base() and introduced the xrecsys_search config.
• phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto — 2026-08-19, RewardOutputs, rewardRerankSlotProb, link-open P10-P90, and SlateContext.sid* fields.
• home-mixer/filters/following_viewer_muted_keyword_filter.rs — 2026-08-19, muted-keyword filter for the Following pipeline only (65 lines).
• home-mixer/candidate_hydrators/quoted_post_text_hydrator.rs — 2026-08-19, new quoted post text hydrator (106 lines).
• grox/flows/ptos/classifier.py — 2026-08-19, introduced the Grok 4.6 internal 10% dial.
• grox/flows/ptos/prior_nsfw.py — 2026-08-19, new Manhattan lookup cache (34 lines).
• visibility-filtering/dark_traffic_setup.rs — 2026-08-19, mirroring connection that sends traffic to xai-vf-shadow (109 lines).
• home-mixer/models/brand_safety.rs — 2026-08-18, removed strip_v1_grok_written, redesigned the V2_WRITTEN_LABELS gate, added the v2_defers_to_v1_when_v2_has_not_ruled test.
• home-mixer/candidate_hydrators/vf_following_candidate_hydrator.rs — 2026-08-18, new VF hydrator for the Following pipeline only (95 lines).
• home-mixer/scorers/vm_ranker.rs — 2026-08-18, connected 7 additional SID fields to the VMRanker request signature.
• phoenix-rankall-strato/stream_forwarders/postCreationEventForwarder.strato — 2026-08-18, Phoenix event stream v3 migration, Kafka cluster move, KeyValue mapping.
• grox/flows/reply_spam/generators.py — 2026-08-18, deleted PostStreamTaskGenerator; ReplyRankingTaskGenerator absorbed the spam plan.
• grox/flows/reply_spam/task_filter.py — 2026-08-18, follower threshold 40k → 60k.
• home-mixer/models/brand_safety.rs — 2026-08-17, ads brand safety V2 verdict function, label set, strip_v1_grok_written helper, and the v2_mirrors_v1_across_tier_matrix verification test (redesigned within a day).
• grox/flows/ptos/task_safety_ptos_adult_content_cross_validation.py — 2026-08-17, calls the Grok 4.5 judge when safemodel and PtOS disagree and issues a demotion verdict.
• grox/flows/ptos/classifier.py — 2026-08-17, introduced the SafetyPtosAdultContentCrossValidationJudge class and a dedicated circuit breaker for EAPI_GROK_4_5_X_ALGO.
• phoenix-rankall-strato/columns/phoenix_rank_all/phoenixRankAllCandidateProcessor.strato — 2026-08-17, removed the favoriteCount>=1 gate from build1FavIndex and buildMMEmbMetadataDump, introduced the 1fav_uec_count_race counter.
• home-mixer/filters/brazil_2026_election_filter.rs — 2026-08-14, filter that hardcodes 665 accounts subject to Brazilian electoral court action and makes an exception for Following.
• Brazilian electoral court candidatos-2026 dataset — regulatory basis that Brazil2026ElectionFilter references.
• @XBR announcement — 2026-08-14, X's announcement of the Brazil 2026 filter (quoted from the README).
• bdsm/runtime/score_results_sink_focal.py — 2026-08-14, partial release of the enforcement note generator.
• home-mixer/params/param.rs — 2026-08-13, production cron mirror (header sync time 2026-08-12T04:09:22Z).
• visibility-filtering/rules/registry.rs — 2026-08-13, 3-tier policy, 54 rules.
• docs/BIDIRECTIONAL_BOOST_CHANGE.md — 2026-08-13, template for documenting parameter diffs (introduction and adjustment of the mutual-follow boost in 2026-07).
• x.com/i/under_the_hood — 2026-08-13, pilot of an account-level label transparency tool.
• twitter/the-algorithm — 2023 initial release (ef4c5eb) and 2025-09-03 refresh (c54bec0), public source code.
• twitter/the-algorithm-ml — 2023, public source code. projects/home/recap/README.md is the only source of numeric weights (as of 2023-04-05).
• Twitter Engineering, "Twitter's Recommendation Algorithm" — 2023, official blog post. States explicitly that model weights and training data are undisclosed.

↑ Contents

Change log (per-commit detail)

This is a record, in date order, of what changed in each commit of X's public repository after the Gen4 release. The earlier topic sections describe the current behavior of X For You; this log shows which commit introduced each change.

Commit summary table