In one lineX fixed the cold-start experiment so that the control and treatment arms leave identical bucket observation records. In the treatment arm, cold-start candidates are narrowed to Phoenix MoE candidates.
1Both arms of the cold-start experiment now leave the same observation records — X
Sourcexai-org/x-algorithm@902a06f — GitHub, 2026-09-04 21:03 UTC, code. This is a regular cron mirror of Open-source X Recommendation Algorithm, and 25 files changed (+472/−179). These notes number X commits sequentially after the Gen4 release, and this commit is Gen4+16.
Background X's open-source mirror consists of commits in which a cron regularly pushes out the production canon. Yesterday's (2026-09-05) edition reflected up to Gen4+15 (9b0dc31), and this commit (902a06f) is the next edition. Phoenix is the Transformer ranking model of the X For You feed, and Phoenix MoE (Mixture of Experts) is an architecture that serves the search and ranking backbone by allocating it to different expert subnetworks. VF is a visibility-filtering service that decides, for each post, whether it may be shown to this viewer. This commit centers on reworking the A/B infrastructure of Phoenix MoE; incidentally, it changed the Phoenix serving request contract and the VF dark-traffic wiring, and it revealed a hint of an upcoming muted-keyword VF rule.
What we learned
A. Reordered execution so that both arms of the cold-start experiment leave the same bucket observation records
Now both the control and treatment arms leave identical experiment bucket observation records (impression) verified. codivert appears to be an X-internal term inferred. It means a pattern that aligns the bucket observation point in time across the two arms so that the A/B analysis is not biased, and "co-divert" means that the two arms each diverge from the same point inferred. This commit made the pattern concrete in code.
The values of the two params come from the bucket membership of the moe_codivert_viewer experiment. The YAML rules of the two arms are also symmetric with each other:
The key change is that the execution order of AuthorColdStart::apply was rearranged verified — home-mixer/scorers/author_cold_start.rs. The previous version (through Gen4+3) ran in this order:
Looks up ColdStartTrackedIds and calls record_tracked_ids.
If EnableViewerColdStart is false, returns early here.
Calls viewer_arm(query). It actually looks up the experiment bucket, and the first impression is left at this point.
Calls author_corpus(...). It actually looks up the author bucket, and the author-side impression is left at this point.
Runs the cold-start logic.
The problem was the early return at step 2: the control arm with EnableViewerColdStart=false never goes through steps 3 and 4. As a result, the impression logs of the two arms differ in shape in the experiment analysis: treatment counts only users for whom both the viewer bucket and the author bucket were observed, and control has no observations at all. The definitions of triggered-analysis and intent-to-treat then diverge, and pipeline-entry bias contaminates the lift estimate. Triggered-analysis compares only users who actually entered the pipeline, and intent-to-treat compares all assigned users.
This commit changed the order as follows:
ColdStartParams::read(query) reads all params at once and stores the arm as params.arm. viewer_arm is also called at this point, so the viewer bucket impression is left.
If params.arm is Control or Treatment, that is, if either arm belongs to codivert, it calls author_corpus(...) regardless of whether the feature is enabled and leaves the author-side bucket impression.
Then, if !params.enabled, it returns early with the original scores unchanged. The impressions have already been left at that point, so the observation logs are symmetric across the two arms.
Runs the cold-start logic.
Two new regression tests pin this symmetry verified. Whether the arm is control or treatment, each must leave exactly 3 observations of the moe_codivert_viewer bucket and exactly 2 observations of the moe_exp (author-targeted) bucket.
#[test]
fn both_arms_log_the_same_codivert_impressions() {
...
assert_eq!(control_impressor.impression_count(CODIVERT_EXPERIMENT), 3);
assert_eq!(treatment_impressor.impression_count(CODIVERT_EXPERIMENT), 3);
}
#[test]
fn both_viewer_arms_log_the_same_author_impressions() {
...
for query in cases {
let impressor = Arc::new(SpyingBucketImpressor::new());
let author_cold_start =
cold_start_with_author_impressor(vec![2], vec![1], impressor.clone());
author_cold_start.apply(&query, &candidates, &scores);
assert_eq!(impressor.impression_count("moe_exp"), 2);
}
}
The assertions pin the symmetry that the number of observations must be the same regardless of arm.
Why it matters This commit fixes the A/B observation methodology of cold start in code for the first time. The previous version of the canon (the notes in this repository that hold the current understanding per service) knew up to the Thompson sampling wiring in Gen4+1 (2026-08-14) and the count_tracked_ids simplification in Gen4+3 (2026-08-18), and both were evolutions of the cold-start logic. It also reveals a test utility called SpyingBucketImpressor (an impression instrumentation spy exposed by the experiment SDK). A cultural observation is added to the canon: X enforces bucket observation symmetry in A/B experiments with regression tests.
B. Cold-start candidates for treatment viewers were narrowed to Phoenix MoE candidates
The cold-start candidates of a treatment viewer are now limited to candidates that come from the Phoenix MoE source verified. Even a post by a treatment corpus author is excluded from the cold-start boost if it did not come from the Phoenix MoE source.
The Phoenix MoE condition was added to the treatment branch of the function that decides candidate eligibility verified — home-mixer/scorers/author_cold_start.rs.
The previous ViewerArm::Treatment branch checked only the corpus == AuthorCorpus::Treatment condition. This commit added && is_phoenix_moe(c).
The cold-start candidates of a treatment viewer are now only the candidates that are posts by treatment corpus authors and have served_type == ForYouPhoenixRetrievalMoe.
fn cold_start_corpus_eligible(arm: ViewerArm, c: &PostCandidate, corpus: AuthorCorpus) -> bool {
match arm {
ViewerArm::Holdout => !is_phoenix_moe(c),
ViewerArm::Control => corpus == AuthorCorpus::Control && !is_phoenix_moe(c),
ViewerArm::Treatment => corpus == AuthorCorpus::Treatment && is_phoenix_moe(c),
}
}
This commit: treatment_viewer_cold_starts_treatment_moe_only
The fixture changing from cold_start_candidate(...) to moe_candidate(...) means the same thing.
Why it matters This change ties Phoenix MoE and cold start into one codivert experiment so that the two are not confounded with each other. For treatment viewers it activates the MoE wiring and applies cold start only to MoE candidates, which isolates the "effect of MoE" from the "effect of the cold-start boost" so that they do not overlap. enable_viewer_cold_start_boost being fixed to true for treatment and false for control in the YAML points the same way. Cold start has now been relocated alongside the Phoenix MoE live-traffic experiment.
C. Engagement counts hydration was decoupled from the cold-start switch
The cold-start switch and the cache condition were removed from the condition that enables engagement counts hydration verified. Cold start and engagement counts hydration are now independent policy axes.
The !query.has_cached_posts gating is gone, so the counts are always refreshed even when a cache exists.
The policy axes were narrowed to the pure hydration flag and shadow traffic.
Why it matters This decoupling lets the rollouts of the two pipelines proceed without blocking each other. The previous wiring was a strong coupling: "if cold start is on, engagement counts must be hydrated." It appears to be a relic of the early Gen4+1 wiring in which cold-start Thompson sampling depends on view_count_on_home and fav_count. Combined with the codivert symmetry (subsection A), it allows a clear separation of experiment axes: "Treatment has cold start on and engagement counts on, and Control has cold start off and engagement counts according to its own flag."
D. Phoenix serving requests gained an experiment parameter override field and two slate metrics
Three new fields were added to the Phoenix serving request contract verified. One is a map that overrides experiment parameters per request, and the other two are metrics attached to the slate (the group of candidates shown together) context. The new fields are experiment_overrides, exact_k, and exact_gap.
The field is PredictNextActionsRequest.experiment_overrides = 21, and its type is map<string, string>.
Previously, Phoenix serving fixed experiment parameters with the config and feature switches at service deployment time.
This field appears to be a wiring in which the requester, home-mixer, passes its already-decided A/B assignment to Phoenix as is, so that the experiment assignment in downstream Phoenix flows consistently with upstream home-mixer inferred.
As the codivert pattern (subsection A) took hold upstream, a delivery channel was needed to carry the same experiment scope downstream.
Two fields, SlateContext.exact_k/exact_gap, were added to the slate context verified. Both are changes in the same two proto files.
SlateContext.exact_k = 16 and SlateContext.exact_gap = 17 are new Option<uint32> fields.
The earlier SlateContext fields and when each was added:
SID (semantic ID) family sid_known (field 6), sid_k_l1/2/3 (7–9), sid_gap_l1/2/3 (10–12): Gen4+3 (2026-08-18), the SID wiring of VMRanker (the scorer that reorders the slate)
recon_cos_milli (13): Gen4+9(2026-08-26)
recon_count_above (14), recon_gap_above (15): around Gen4+11
exact_k/exact_gap extend this series. If sid_k/sid_gap are metrics based on semantic ID codebook matches, exact_k/exact_gap are likely metrics based on exact matches, as the names suggest inferred.
Reading exact_k as the number of neighboring candidates in the slate that match exactly on some signal (post_id? author? media hash?) and exact_gap as the slot gap at which exact matches begin to appear corresponds naturally to the SID-family fields.
The actual definition is not public. The place in Phoenix serving code that fills this field is not in the open source.
Two fields, exact_k: Option<u32> and exact_gap: Option<u32>, were added to both the SlateContext struct and the From<xai_recsys_proto::SlateContext> conversion impl.
Why it matters Phoenix serving keeps adding observation metrics to the slate context. It started with the SID family (Gen4+3), widened to the recon family (Gen4+9–+11), and this time the exact family was added. The consumption path (the RankingScorer fallback) is already connected to the UseServedSlateContext gate from Gen4+5, so new metrics are absorbed automatically. experiment_overrides is a contract-level response to the problem of experiment scope consistency between upstream and downstream, and its timing overlapping with the codivert wiring (subsection A) is no coincidence.
E. Added a devel deployment to VF mirroring so that per-PR preview deployments also receive production traffic
VF dark-traffic is mirroring that duplicates production traffic to a separate deployment and only observes it, without returning a response. This commit widened the mirroring targets from staging to the devel app_env as well verified.
The parse_staging_listener of the previous version (where the shadow→staging rename was completed in Gen4+6) recognized only the .staging.visibility:grpc suffix.
This commit renamed the function to parse_mirror_listener.
Examples of listener names that pass are pinned in the regression test:
The channel factory chooses a different TLS domain depending on whether the listener is devel or staging verified. This is MirrorChannelFactory in the same file.
If the xds_dest of the target listener contains .devel., it uses the channel factory for the devel TLS domain (visibility.xai-vf-service.devel.{dc}.s2s.twttr.net).
Otherwise, it uses the channel factory for staging (visibility.visibility-filtering-service.staging.{dc}.s2s.twttr.net).
A listener name of the form xai-vf-service-pr-123.devel.visibility:grpc suggests the convention of preview deployments (Kubernetes preview environments) generated automatically from GitHub PR numbers inferred.
Similar names are rejected, and only exactly the two app_envs staging and devel are allowed verified. The reject-case regression test in the same file was also narrowed.
Lookalike names such as development or staging-devel are rejected.
— the parse_rejects_out_of_scope_listeners test in the file above
Why it matters The verification infrastructure of the VF rewrite now extends to per-PR verification. This infrastructure is an audit stack that ran from Gen4+4 dark_traffic through the Gen4+6 shadow→staging rename to Gen4+7 reference_compare. When a developer opens a PR that changes a VF rule, a production traffic mirror flows into the devel deployment for that PR, so verdict differences can be checked in advance at real production scale. VF is fail-closed (caveat list #1 in the canon x.md), so rule changes carry high risk, and widening the mirroring stack from the staging deployment to PR deployments is a natural evolutionary path.
F. VF left a post text field in place for an upcoming muted-keyword rule
VF kept a post text field that no rule currently uses, for the muted-keyword rule that will be added verified.
CoreFeature.text got an attribute that suppresses dead-code warnings, and the attribute states explicitly that the field is "kept for use in the upcoming muted-keyword rule" verified — visibility-filtering/models/tweet.rs.
The VF service already fills each tweet's text in the hydration stage, but no rule currently consumes it.
The new test assemble_hydrates_text_from_core_data checks by regression that text is actually filled on the hydration path.
FollowingViewerMutedKeywordFilter, newly added in Gen4+4 (2026-08-19, aad7179), was specific to the Following pipeline (65 lines). This signal is a preview that a muted-keyword rule will soon be attached to the VF service itself as well.
Since VF covers the whole For You pipeline, the policy is widening toward filtering on the viewer's muted-keyword list across all recommendations, not only Following.
Why it matters The point to watch in the next commit is whether the muted keyword filters, implemented separately in the two pipelines, will be unified (the Following filter migrated to the VF service) or remain side by side as two layers.
G. Other changes
The remaining changes touch Phoenix training code and VF and home-mixer code verified.
The new function _unwrap_with_backoff retries with jittered exponential backoff on transient errors (429, 502, 503, 504, and the "KMS transport" marker), 4 times with a total cap of 2 seconds.
publish_tree_dek, adopt_tree_dek, and _unwrap_header_master are all called through this wrapper.
assert_all_enveloped always checks that the _DEK master file exists before finalize (encrypted save left no master _DEK at {path}; not committing).
This is follow-up work that stabilizes the envelope-encrypted Orbax writer newly introduced in Gen4+13.
In retrieval_dataset.py, the code that builds the SID snapshot window paths was consolidated into one helper verified — phoenix/xrex/data/retrieval_dataset.py.
The helper is _sid_window(filename) -> (primary, backup).
The actual path rule is unchanged (post_sid_v8_256x6_snapshots and _backup).
This is a computation layer meant to align the serving precision (presumably served as bfloat16 inferred) with the numerics observed in training.
It is numerics-alignment wiring that pairs with the backbone scores request semantics newly introduced in Gen4+14.
Structured logging that automatically appends extra fields to the end of training log lines was added verified — phoenix/xrex/utils/log_util.py.
log_util takes the extras left after removing the standard attributes from LogRecord.__dict__, joins them as a key=value series, and appends them after the log line.
The event=X extra goes first. Bools are written as true/false, and the rest with json.dumps(default=str).
This change makes training logs greppable and supports downstream parsers.
VF narrowed the modules and functions it exposed externally to crate-internal use verified — visibility-filtering/lib.rs.
The clients, hydration, models, rules, safety_label_source, and twemcache modules were narrowed from pub to pub(crate), that is, crate-internal.
Several pub functions and constants in config.rs, params.rs, server.rs, and twemcache/{client,mod}.rs were also changed to pub(crate) or removed as dead code.
The direction is the same as the net reduction of 800 lines in Gen4+15 (9b0dc31). After the rewrite stabilized, it narrows APIs that no longer need to be exposed externally.
Canon update The following was folded into services/x.md:
Added a Gen4+16 row to the Gen4 generation table.
In the cold-start section, added the codivert impression symmetry wiring, the restriction of treatment to Phoenix MoE, and the separation of engagement counts gating.
Mentioned the devel app_env extension under the Gen4+4 dark traffic section.
Added SlateContext.exact_k/exact_gap to the SlateContext evolution family and the experiment_overrides map to the PredictNextActionsRequest field family.
Created a new section "### Gen4+16 (2026-09-04, 902a06f)" between Gen4+15 and the canon caveat list.
Why it matters The cultural observation from this commit is that X is currently in a phase of reworking the A/B infrastructure for the Phoenix MoE rollout. The evidence is that the following three axes moved together in a coherent, interlocking way:
Codifying the A/B observation methodology: codivert impression symmetry was pinned by regression tests.
Vertical delivery of experiment scope: through the experiment_overrides map, the upstream assignment flows consistently down to downstream.
Merging cold start into the Phoenix MoE experiment axis: the cold-start candidates of treatment viewers apply only to MoE candidates.
The VF stack is evolving on its own toward extending its verification infrastructure (dark_traffic) to PR deployments, and a hint of the upcoming muted-keyword rule is stated explicitly in a dead-code attribute. In the Phoenix training stack, stabilization follow-ups continue for the envelope encryption, StableHLO bundle, and H100 remat newly introduced in Gen4+13–+15.
Corrections
Nothing was overturned. This commit is consistent with the direction of the canon. However, what was observed in Gen4+3 as "the actual wiring of record_tracked_ids callers and params after the count_tracked_ids logic simplification" becomes clear in this commit.
record_tracked_ids consumes the ColdStartTrackedIds param as is.
It is always called regardless of the EnableViewerColdStart gating (a result of the reordering in subsection A).
It is confirmed that the tracked ID observation hook has become an always-on observation channel unrelated to A/B.
Radar
No sources were added to the radar. 30 arXiv cs.IR items were parsed, and industry blogs were skipped because it was Sunday. After triage, 0 of the 30 passed.
The 9 items on recommendation, CTR, and rerankers are all mainly by academic authors and have no production deployment or A/B claims. These 9 were rejected.
CORE (MLLM embedding compositional)
EPIC(SID diffusion recommendation)
LLM4AIGQ(e-commerce guidance query)
HypRQ-VAE(hyperbolic long-tail generative rec)
SelfDR (LLM-based rec reasoning distillation)
DoPR (LLM reranker compressed document prefix)
UniCon(CTR unified context)
Recommender System as Slow and Fast Thinkers
Training seeds and model-selection stability, among others
The other 21 were also rejected. They fall into three groups:
Benchmark and evaluation methodology: SHELF, MultiGhostBench, Incremental Pooled Eval
Domain-restricted: POI, legal, medical, science, baseball, music, and finance QA
LLM application methods such as RAG, private retrieval, and essay grading
What to check next
The open questions are how the codivert experiment, the new fields, the announced-only VF muted-keyword rule, and the PR deployment mirroring from this commit will actually be used.
Initial traffic share and activation time of the moe_codivert_viewer experiment — this diff fixed only the YAML rules and the param default (false), and when live traffic attaches is the next checkpoint to watch.
Also look at the relationship between the two experiments moe_exp and moe_codivert_viewer (the former is author-side assignment, the latter viewer-side assignment).
Also look at the orthogonality of the two assignments (independent randomization or layered).
Consumption path of experiment_overrides — the question is how Phoenix serving parses this map and which layer's experiment params it overrides.
Also look at the build point in home-mixer that fills this field (observe the hydrator name).
The actual definition of exact_k and exact_gap — the question is which metric Phoenix serving fills these two fields with.
Verify the interpretation that they are an exact family corresponding to the SID family.
Also look at how the RankingScorer fallback in home-mixer consumes these fields.
When the VF muted-keyword rule actually appears — the question is how many commits later the dead-code attribute of Gen4+16 is implemented as an actual rule.
Also look at how it integrates with FollowingViewerMutedKeywordFilter (Gen4+4) of the Following pipeline. The question is whether it remains a duplicate layer or the Following filter is migrated.
Actual usage history of VF devel deployment mirroring — the questions are what scale of traffic the production traffic mirror attached to per-PR preview deployments handles, and whether the resource budget of the preview deployments is designed to bear the mirror load.
The open source has no way to observe this. Only code constants about the scale of the dark_traffic targets can be observed.