2026-08-23

1Phoenix gained 20 ad conversion actions and a delayed feedback training branch — X

The canon (this repository's X canon document) from Gen4 through +5 has covered three threads. This commit points in the same direction as these three, but differs from Gen4 through +5 in that it adds a new axis: breaking down the ad revenue prediction head.

• First is the expansion of signal channels in the ranking pipeline (topic feedback → AI trend feedback, SID 3 layers, RewardOutputs·rewardRerankSlotProb, SlateContext).
  • SID (semantic ID) is an ID that represents a post as a codeword sequence over coarse → fine layers.
• Second is the generation change of the PTOS judge layer (safemodel↔PtOS disagreement → Grok 4.5 judge → 4.6 dial 0.1 → 0.3).
  • PTOS (PtOS) is the layer in the grox safety pipeline that decides violation categories on a policy basis.
• Third is VF and grox resource tuning (visibility-filtering shadow, grox reply_spam follower threshold).
  • visibility-filtering (VF) is a service that decides, post by post, whether a post may be shown to this viewer.
  • grox is a Grok-based nearline annotation pipeline.

The six new concepts, in brief:

• MACT (Mobile App Conversion Tracking) is a measurement framework that attributes in-app events in mobile app ads to ad impressions and clicks.
  • In-app events include in-app purchases, tutorial completion, and sign-ups.
  • When an advertiser logs an in-app event through the SDK, the X ads system matches which impression or click led to that event.
  • The existing X ranker proto had only one, ADS_ATTRIBUTED_MACT_CLICK_INSTALL, which attributes an app install to a click. This time it grew to 12: 6 in-app events times 2 kinds, click and view.
• View-through conversion refers to the case where a user saw an ad but did not click it, and later reached a conversion (purchase, sign-up, etc.) through another path.
  • It contrasts with click-based attribution (click-through), and is a measurement approach that credits an ad with an effect from the impression alone.
  • This time the X ranker introduced 8 view-through actions: purchase·add_to_cart·checkout_initiated·sign_up·site_visit·session·landing_page_view·upper_funnel.
• Delayed feedback refers to the phenomenon in ad conversion where the time of the impression or click and the time of the actual conversion (e.g., sign-up, in-app purchase) are separated by anywhere from hours to days.
  • At training time it is hard to tell whether a label that has arrived means "not yet happened" or "will not happen", which distorts the loss.
  • In the ranker literature, delayed conversion prediction has become an established research area of its own (since Chapelle 2014).
  • In this commit, X added an is_delayed_feedback column to each training sample. It also split the loss mask so that ad conversion heads train only on delayed samples and engagement heads only on fresh samples.
• Muon optimizer is an optimizer that approximately orthogonalizes the gradient momentum matrix with Newton-Schulz orthogonalization.
  • It spread through the Keller Jordan 2024 original text and the LLM training of the Moonshot Kimi K2 family.
  • It was introduced in Gen4+4 (aad7179) as the default dense-parameter optimizer of the X Phoenix ranker, but TRAINING.md still stated that the export Python code used AdamW.
  • In this commit the docs, config, and reference training script were rewritten to state explicitly that flagship and nano are trained with Muon, which narrowed the gap between the export and production.
• NVIDIA GB300 (Grace Blackwell 300) is NVIDIA's 2025–2026 datacenter GPU. It belongs to the Blackwell generation and succeeds the B200.
  • Each GB300 node supports large-capacity HBM3e and attention kernels based on CUTLASS/CUTe DSL.
  • X exposed hardware-specific config (batch size, expert parallelism, remat policy, attention impl) explicitly in code for the first time.
• The xai-o2 object storage is S3-compatible object storage presumed to be an xAI internal storage system.
  • In this commit the client crate interface was released, but the actual implementation was left out of the open-source build.
  • This is a signal suggesting that production checkpoint and data storage go through xai-o2 inferred.
  • The open-source build must substitute the standard object_store crate.

20 new ad conversion actions were added at proto numbers 171–190

The new actions are 12 that split MACT in-app events into click and view, and 8 view-through conversions verified.

• 20 ad conversion actions were newly defined in the action list of the Phoenix proto verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto.

  • They are defined in enum ActionName.

    ADS_ATTRIBUTED_MACT_PURCHASE = 171;
    ADS_ATTRIBUTED_MACT_ADD_TO_CART = 172;
    ADS_ATTRIBUTED_MACT_LEVEL_ACHIEVED = 173;
    ADS_ATTRIBUTED_MACT_TUTORIAL_COMPLETE = 174;
    ADS_ATTRIBUTED_MACT_SIGN_UP = 175;
    ADS_ATTRIBUTED_MACT_CUSTOM = 176;
    ADS_ATTRIBUTED_MACT_VIEW_PURCHASE = 177;
    ADS_ATTRIBUTED_MACT_VIEW_ADD_TO_CART = 178;
    ADS_ATTRIBUTED_MACT_VIEW_LEVEL_ACHIEVED = 179;
    ADS_ATTRIBUTED_MACT_VIEW_TUTORIAL_COMPLETE = 180;
    ADS_ATTRIBUTED_MACT_VIEW_SIGN_UP = 181;
    ADS_ATTRIBUTED_MACT_VIEW_CUSTOM = 182;
    ADS_PURCHASE_CONVERSION_VIEW_THROUGH = 183;
    ADS_ADD_TO_CART_CONVERSION_VIEW_THROUGH = 184;
    ADS_CHECKOUT_INITIATED_CONVERSION_VIEW_THROUGH = 185;
    ADS_SIGN_UP_CONVERSION_VIEW_THROUGH = 186;
    ADS_SITE_VISIT_CONVERSION_VIEW_THROUGH = 187;
    ADS_SESSION_CONVERSION_VIEW_THROUGH = 188;
    ADS_LANDING_PAGE_VIEW_CONVERSION_VIEW_THROUGH = 189;
    ADS_UPPER_FUNNEL_CONVERSION_VIEW_THROUGH = 190;
    

  • The first group is 12 (171–182): the 6 MACT in-app events times click/view.

    • The events are purchase, add_to_cart, level_achieved, tutorial_complete, sign_up, and custom.
    • purchase is an in-app purchase, level_achieved is reaching a game level, tutorial_complete is tutorial completion, sign_up is account registration, and custom is an advertiser-defined custom event.
    • Each event has two families, MACT_<event> and MACT_VIEW_<event>.
    • The former is click-attributed: an event that occurred in the app after the user clicked the ad. The latter is view-attributed: an event that occurred in the app after the user only viewed the ad, without a click.
    • Previously only ADS_ATTRIBUTED_MACT_CLICK_INSTALL (=140) existed, so only app installs were predicted. Now the 6 post-install engagement events are predicted by 12 heads, one each for click and view.

  • The second group is 8 view-through conversions (183–190).

    • They cover purchase / add_to_cart / checkout_initiated / sign_up / site_visit / session / landing_page_view / upper_funnel.
    • Each uses the name <conversion>_VIEW_THROUGH.
    • View-through counterpart heads were introduced to pair with the existing click-attributed conversions (ADS_PURCHASE_CONVERSION = 200, etc.).
    • The 8 web conversions (advertiser site visit, purchase, sign-up, etc.) are now also predicted as view-through.

ads_slim and the ads_conversion_view_through engagement group mapping

12 MACT entries were added to the ad conversion mapping, and a new metric group that the ads ranking config can pick was created verified.

• A new mapping was added to constants.py verified — phoenix/xrex/data/recsys/constants.py.

  • 12 new MACT mappings were added to the existing ads_conversion_engagement_to_action_types.

    "IsAttributedMactPurchase": ["AdsAttributedMactPurchase"],
    "IsAttributedMactAddToCart": ["AdsAttributedMactAddToCart"],
    ...
    "IsAttributedMactViewCustom": ["AdsAttributedMactViewCustom"],
    "IsPurchaseConversionViewThrough": ["AdsPurchaseConversionViewThrough"],
    "IsAddToCartConversionViewThrough": ["AdsAddToCartConversionViewThrough"],
    ...
    "IsUpperFunnelConversionViewThrough": ["AdsUpperFunnelConversionViewThrough"],
    

  • It also created a metric group called ads_slim. In addition to 7 signals, it includes only the non-primary entries filtered from the preceding conversion click mappings.

    ads_slim_engagement_to_action_types = {
        "IsOpenLink": ["ClientTweetOpenLink"],
        "IsExternalLinkLongDwelled": ["ClientTweetExternalLinkLongDwelled"],
        "IsVideoQualityViewed": ["ClientTweetVideoQualityView"],
        "IsNotInterestedIn": ["ClientTweetNotInterestedIn"],
        "IsBlockAuthor": ["ClientTweetBlockAuthor"],
        "IsReported": ["ClientTweetReport"],
        "IsMuteAuthor": ["ClientTweetMuteAuthor"],
        **{
            eng: actions
            for eng, actions in ads_p_conv_click_engagement_to_action_types.items()
            if eng not in primary_engagement_to_action_types
        },
    }
    

  • That is, the ad ranker's "slim" line trains only 3 positive signals, 4 negative signals, and non-primary conversion clicks.

    • The 3 positive signals are open link, external link long dwell, and video quality view.
    • The 4 negative signals are not interested, block, report, and mute.

  • "ads_slim": ads_slim_engagement_to_action_types was also registered in the metric group routing dictionary, so that the ads ranking config can pick this group.

• A list of conversion heads to be used in the delayed feedback branch below was introduced.
  • The name is SOURCE_SPLIT_CONVERSION_HEAD_INDICES.
  • It is a list that concatenates the click-conditioned action index list and the view-through action index list.
• When a user is dissatisfied with search results and rewrites the query, this is included in the negative feedback heads. Search surface ranking will learn this signal.
  • CLIENT_TWEET_SEARCH_QUERY_REFORMULATED was added to NEGATIVE_FEEDBACK_HEAD_INDICES.
  • This signal connects to the xrecsys_search config expansion in Gen4+4.

A setting that branches delayed feedback out of the training loss was added

With the new setting on, delayed samples train only the conversion heads and fresh samples train only the engagement heads verified.

• A new sample_source field was added to the batch verified — phoenix/xrex/data/recsys/recsys_batch.py:980.

  sample_source=(
      record_batch.column("is_delayed_feedback")
      .fill_null(False)
      .to_numpy(zero_copy_only=False)
      .astype(np.bool_)
      .reshape(-1, 1)
      if "is_delayed_feedback" in record_batch.schema.names
      else np.zeros((batch_size, 1), dtype=np.bool_)
  ),
  

  • If the kafka/parquet record batch has an is_delayed_feedback column, its value goes into the batch as sample_source; otherwise the field is filled with all False.
  • True means this sample is a delayed feedback re-observation.
  • Adding the two columns is_delayed_feedback and conversionKeepMask to the required columns list in feature_config.py confirmed the upstream indicator.

• The loss mask splits by sample state verified — phoenix/xrex/models/recsys_model.py.

  • The branch is in the loss() function.

    if self.config.split_head_training_by_source and delayed_mask is not None:
        conv_head_split_mask = (
            jnp.zeros(num_actions).at[jnp.array(SOURCE_SPLIT_CONVERSION_HEAD_INDICES)].set(1.0)
        )
        eng_head_mask = 1.0 - conv_head_split_mask
        is_delayed = delayed_mask[:, :, None]
        loss_mask = loss_mask * (1 - is_delayed * eng_head_mask)      # engagement head loss is 0 for delayed samples
        loss_mask = loss_mask * (1 - (1 - is_delayed) * conv_head_split_mask)  # conversion head loss is 0 for fresh samples
    

  • Each training sample is in one of two states, fresh or delayed. Fresh is the original impression, and delayed is the re-observation after the conversion arrived.

  • Fresh samples train only the engagement heads (favorite·reply·click, etc.) and set the conversion heads to mask=0.

  • Delayed samples train only the conversion heads and set the engagement heads to mask=0. Here the conversion heads are the 20 new ones above plus the existing click-conditioned actions.

• Metric slices were introduced. A branch was added to build_metric_masks so that the three slices can be observed.

  • fresh: delayed=0
  • delayed_clicked: delayed=1 & clicked=1
  • delayed_non_clicked: delayed=1 & clicked=0

• Because condition_conversion_on_click is a required precondition (assert), even delayed samples keep the conditional training that predicts conversion under the condition "was there a click".

• mact_in_app_loss_weight is a loss weight multiplied only into the 12 MACT actions, and its default is 1.0.

  • It is exposed as a config parameter, so the training strength of the 12 ad conversion heads can be adjusted.
  • It is observed through stats["mact-in-app-loss-weight"].

• train_view_through_heads is a gate that decides whether to train the 8 view-through heads.

  • When set to False, it sets the loss mask of the view-through heads to 0 so they are not trained.
  • That is, view-through prediction has to be turned on as an opt-in in the config.

• conversion_keep_mask / conversionKeepMask is a mask that marks, for each candidate, whether to exclude it from conversion training.

  • from_record_batch reads the kafka conversionKeepMask column and replaces null rows with True (= keep).
  • In loss(), this mask is ANDed with target_padding_mask to exclude entries from conversion training.
  • One caution: sequence packing allows only all(True), and the not-yet-connected case explicitly fails with an assert.

The Muon optimizer is now formally included in the export

The export docs now state that the flagship ranking config and nano train with the production Muon recipe verified.

• TRAINING.md was rewritten verified — phoenix/TRAINING.md.

  • The previous wording described the export's dense-parameter optimizer as standard Optax AdamW.

    The dense-parameter optimizer in this export is standard Optax AdamW. The internal deployment uses a tuned RMS-normalized-Adam derivative in that optimizer slot.

  • The new wording lists two families, Muon and AdamW. Muon is used by the flagship ranking config and home_direct_packed_nano, and AdamW by the remaining configs.

    Two dense-parameter optimizer families ship in this export:

    • Muon (xrex/optimizers/recsys/muon.py): the production home-ranker recipe — consistent-RMS scaling with decoupled weight decay on the matrix and embedding partitions. The flagship ranking configs and home_direct_packed_nano select it (optim="muon"), so the nano trains with the same dense-optimizer recipe production runs.
    • Standard Optax AdamW: the slot used by the remaining shipped configs (two-tower retrieval, gen-recs, and the legacy ranking presets). ...

• README.md was likewise rewritten toward Muon.

  • The previous wording was "one training-recipe exception ... standard AdamW rather than production's tuned internal variant".
  • The new wording is "The flagship ranking configs and the nano twin train the production Muon recipe, which ships in full".
  • The train_step.py entry in reference/README.md was also generalized to "dense optimizer + rowwise-AdaGrad", and it states explicitly that flagship and nano use Muon.

• The embedding optimizer was also folded into the canon. TRAINING.md states explicitly that flagship and nano use three options together in embedding rowwise AdaGrad.

  • The three options are accumulator half-life decay, lazy per-row decay, and decoupled weight decay.
  • In effect, the Gen4+4 emb_optim_config parameters (LR 0.28, half-life 2500 step, lazy_decay, weight_decay 2.8e-4) were confirmed as canon.

GB300 hardware-specific overrides appeared for the first time

The config defines GB300 hardware-specific overrides verified. The nano config included in the export also inherits these overrides.

• _GB300_OVERRIDES is defined verified — phoenix/xrex/configs/xrecsys.py:330-345.

  _GB300_OVERRIDES = {
      "bs_per_device": 512,
      "ep": 64,
      "attn_impl": "cutedsl_ranker_varlen_attn",
      "remat_policy": RematType.SAVE_GB300_RECSYS,
      "unroll_layer_stack": True,
      "learning_rate": 5e-4,
      "checkpoint_every_n": 300,
      "optim_config": RecsysDenseOptimConfig(
          optim="muon",
          muon_consistent_rms=0.2,
          muon_matrix_weight_decay=0.01,
          muon_split_fused="qkv:128",
          adam_embedding_weight_decay=0.01,
          b1=0.95,
          b2=0.98,
          ...
      ),
      ...
  }
  

  • bs_per_device=512 is 8× larger than the previous _home_direct_packed_nano value of bs_per_device=64. It reflects the ample HBM capacity of the GB300.
  • ep=64 sets expert parallelism to 64-way. It is MoE (mixture-of-experts) style sharding.
  • attn_impl="cutedsl_ranker_varlen_attn" is a variable-length attention kernel based on NVIDIA CUTLASS/CUTe DSL. It is a new attention implementation optimized for the GB300.
  • remat_policy=RematType.SAVE_GB300_RECSYS is a rematerialization policy tuned to the GB300 memory hierarchy. It decides what to checkpoint and what to recompute.
  • unroll_layer_stack=True compiles the Transformer layer stack unrolled (listed explicitly instead of looped). This gives XLA more room to optimize.
  • learning_rate=5e-4 is slightly lower than the Gen4+4 flagship base LR of 7.1e-4. It appears to be an adjustment matched to the 8× larger bs_per_device inferred.
  • checkpoint_every_n=300 saves a checkpoint every 300 steps.

• _home_direct_packed_nano inherits this override.

  • Previously it set learning_rate=2e-3 separately.
  • In this commit it inherits the LR, optim_config, and emb_optim_config of _GB300_OVERRIDES as they are.
  • In other words, even the "nano" model included in the export is tuned on the assumption of GB300 training.

The interface of the xai-o2 storage crate was released

An interface crate for an S3-compatible object storage client was newly added verified. In the open-source build, trying to create this client always returns an error.

• The xai-o2 crate was introduced verified — phoenix/crates/storage/xai-o2/.

  • The description in Cargo.toml reads:

    Client interface for the xai-o2 (S3-compatible) object storage backend

  • There are 3 files, 134 lines in total.

    • base_client.rs (12 lines) is the BaseO2Client async trait. It is a minimal interface that defines only put(key, data) -> Result<()>.
    • O2ClientBuilder in o2_client_builder.rs (115 lines) exposes 15 builder methods. The first eight are endpoint·bucket·prefix·access_key_id·secret_access_key·allow_anonymous·timeout·io_timeout. The other seven are retry_max_times·retry_min_delay·retry_max_delay·write_chunk_size_mib·write_chunk_concurrency·max_concurrent_write_requests·trace_always.
    • lib.rs (7 lines) contains only pub mod declarations.

• In the open-source build, O2ClientBuilder::build() always returns the following error:

  pub const BACKEND_UNAVAILABLE: &str =
      "the xai-o2 backend is not included in this build; use the default object_store backend";
  
  pub fn build(&self) -> Result<Arc<dyn BaseO2Client>> {
      Err(anyhow::anyhow!(BACKEND_UNAVAILABLE))
  }
  

  • The error message returned by build() is "the xai-o2 backend is not included in this build; use the default object_store backend".
  • That is, only an interface stub was released, and the actual implementation is not included in the open-source build.
  • The xai-recsys-engine crate does declare the dependency as xai-o2 = { workspace = true }, but using it in practice requires substituting another backend.

SlateContext is also carried in NextActionDistribution

A slate context field was added to the proto message that holds the per-action prediction distribution for each candidate verified.

• SlateContext was added to the NextActionDistribution message as field 9 verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto.

  message NextActionDistribution {
     map<uint32, float> indexToContinuousValues = 7;
     RewardOutputs rewardOutputs = 8;
  +  SlateContext slateContext = 9;
  }
  

  • This value was introduced in Gen4+4 as ScoreInfo.slateContext. It is the context carried with the final decision of the re-ranking slate.

grox reply_spam follower threshold rose from 60k to 80k (4th increase, 1.33×)

The follower threshold used for spam verdicts and for reply ranking rose from 60,000 to 80,000 in both places verified.

• The two threshold constants in task_filter.py changed together verified — grox/flows/reply_spam/task_filter.py.

  -    FOLLOWER_COUNT_THRESHOLD_FOR_SPAM_DETECTION = 60000
  +    FOLLOWER_COUNT_THRESHOLD_FOR_SPAM_DETECTION = 80000
  ...
  -    FOLLOWER_COUNT_THRESHOLD_FOR_REPLY_RANKING = 60000
  +    FOLLOWER_COUNT_THRESHOLD_FOR_REPLY_RANKING = 80000
  

• The threshold went through five stages over 8 days starting 2026-08-13 verified.

  • 15,000 (before 2026-08-13)
  • 30,000(2026-08-14, Gen4+1)
  • 40,000(2026-08-17, Gen4+2) — task_filter.py@b089ce6
  • 60,000(2026-08-18, Gen4+3)
  • 80,000(2026-08-21, Gen4+6)

• The previous three steps were 2×, 1.33×, and 1.5× (15→30, 30→40, 40→60), and this 1.33× is a smaller increase than the one before.

visibility-filtering dark_traffic target changed from shadow to staging

The VF dark_traffic mirroring target changed from the production shadow workload to the staging environment verified.

• In dark_traffic_setup.rs, the shadow workload constant was removed and a staging destination was added verified — visibility-filtering/dark_traffic_setup.rs.

  -const SHADOW_WORKLOAD: &str = "xai-vf-shadow";
  +pub const STAGING_XDS_DEST: &str = "xai-vf-service.staging.visibility:grpc";
  +const FORWARDER_NAME: &str = "staging";
  +
  +pub fn staging_tls_domain(dc: &str) -> String {
  +    format!("visibility.visibility-filtering-service.staging.{dc}.s2s.twttr.net")
  +}
  ...
  -struct StaticShadowDiscovery;
  +struct StaticStagingDiscovery;
  

  • The new destination is xai-vf-service.staging.visibility:grpc.

• The dark_traffic infrastructure was introduced 2 days earlier, in Gen4+4 (aad7179, 2026-08-19). This time it was renamed from the production shadow workload to the staging environment.

  • The target changed from xai-vf-shadow.prod.visibility to xai-vf-service.staging.visibility.
  • The TLS domain also moved from visibility.visibility-filtering-service.prod.{dc}.s2s.twttr.net to .staging..
  • The StaticShadowDiscovery struct was also renamed StaticStagingDiscovery.

1. An infrastructure cleanup: the staging environment is used like a shadow instead of keeping a separate shadow workload.
2. A judgment that a shadow of production traffic is useful for verifying staging deployments.

Either way, VF experiments now run on the staging resource budget rather than production shadow resources. The canon through Gen4+5 said "the shadow deploy foundation is wired", and that foundation is now redefined as staging mirroring.

Grok 4.5 internal removed entirely and PTOS judge switched to 100% Grok 4.6 internal

Code related to Grok 4.5 internal was deleted from the PTOS classifier, and the two calls that had a probability branch now always use the 4.6 internal family of models verified.

• Removed the 4.5 internal code from classifier.py verified — grox/flows/ptos/classifier.py (+6/−63).
  • Deleted items:
    • _EAPI_4_5_INTERNAL_BREAKER_CONFIG (circuit breaker config)
    • _eapi_4_5_internal_breaker (breaker instance)
    • _GROK_4_6_INTERNAL_DIAL constant (it was 0.3 the day before)
    • Removed self.eapi_4_5_internal initialization from two classes
    • Removed _sample_4_5_internal method
  • Removed the probability branch in _cross_model_validate_with_4_5. It now always calls 4.6 internal directly.
  • Removed the probability branch in the deluxe 4.3 reclassification of classify_policy_for_violation. It now always uses deluxe-4.6-internal.

Retrieval post-embedding computation is now processed in 65k chunks on even shards

Retrieval post-embedding computation splits a shard into chunks of at most 65,536 rows only for even shards; otherwise it processes the whole shard at once as before verified.

• Refactored maybe_build_retrieval_post_embeddings and introduced a _forward_chunked helper verified — phoenix/xrex/train/trainer_recsys.py.
  • Previously, the whole shard was computed in a single candidate_tower_forward_jit call.
  • Now it enters chunking with _chunk_rows = min(_shard_rows, 65536) only in the even-shard case where total_samples % data_world_size == 0.
  • Otherwise it processes the whole shard at once (_chunk_rows = _shard_rows).
  • For each chunk, it computes in the order jax.make_array_from_process_local_data → _lookup → candidate_tower_forward_jit → local shard concatenate, and concatenates at the end.
  • Even when head_dataset_mapping has several heads, it calls _forward_chunked for each head and chunks in the same way.

--hotswap_malloc_trim CLI added to inference hotswap

An option was added that calls glibc's memory trim function at the end of every inference hotswap cycle, and its default is True verified.

• A new option entered the inference code verified — phoenix/xrex/inference/model_runner.py, launch_inference.py.
  • The InferenceModelRunner field hotswap_malloc_trim: bool = True was added.
  • The CLI argument --hotswap_malloc_trim was added, and its default is True.
  • The _malloc_trim_after_swap() method calls _glibc_malloc_trim() (= libc malloc_trim(0)) at the end of every hotswap cycle.
  • Observation is done with metrics_publisher.checkpoint_reload_step_seconds.labels(step="malloc_trim").observe(secs).

The CLI help states this problem explicitly.

glibc malloc_trim(0) on the coordinator thread after each applied hotswap; returns the cycle's freed pages (glibc slots otherwise ratchet ~196 MiB/cycle). No-op under jemalloc.

Three options added to the nano config of two-tower retrieval

Three attention and positional-encoding options were added to the small training config (nano) of retrieval two-tower verified. A stabilization technique common in LLM training has carried over to retrieval two-tower.

• They were added to the xrecsys_two_tower_nano entry of MODEL_CFGS verified — phoenix/xrex/configs/xrecsys_two_tower.py:429-431.
  • _xrecsys_two_tower_combined_base() in the same file did not change in this commit and kept qk_norm=False and attn_logit_cap=80.0 (xrecsys_two_tower.py:273-274).
  • "qk_norm": True turns on attention query/key normalization.
  • "attn_logit_cap": -1 turns off attention logit capping (-1 = unused).
  • "right_anchored_rope": True sets RoPE positional encoding to right-anchored. That is, it anchors relative to the end of the sequence.

EVERGREEN retrieval dataset shrank from 5 years to 4–14 days

The range of videos referenced by the EVERGREEN retrieval dataset shrank substantially verified.

• The parquet file that the EVERGREEN entry in retrieval_dataset.py points to changed verified — phoenix/xrex/data/retrieval_dataset.py:255-258.

  EVERGREEN = (
      5,
  -    _idx("post_sid_v5_256x6_snapshots/evergreen_video_1825day.parquet"),
  -    _idx("post_sid_v5_256x6_snapshots_backup/evergreen_video_1825day.parquet"),
  +    _idx("post_sid_v5_256x6_snapshots/video_4to14day.parquet"),
  +    _idx("post_sid_v5_256x6_snapshots_backup/video_4to14day.parquet"),
  )
  

• The name is still EVERGREEN, but what it actually references shrank sharply from 1825 days (5 years) to recent videos within 4–14 days.

• It appears to be an experiment putting videos from the most recent 4–14 days into the evergreen slot inferred. The name and the substance do not match, so it may be a temporary change.

Other small patches

Small patches went into request metrics, checkpoint loading, the kafka loader, and the grox config.

• Fixed double counting in request_metrics.rs. A request shed by deadline_admission was then also counted as inflight_cap because of the gRPC ResourceExhausted status code; the guard.recorded_reject flag now prevents this.
  • Added two new tests: deadline_shed_does_not_double_count_as_inflight_cap, resource_exhausted_without_prior_reject_counts_inflight_cap.
• Made checkpoint metadata handling more robust. It skips checkpoints that have only the completion marker (COMPLETED_FILENAME) but no actual orbax tmp-dir commit (rename).
  • Renamed _ORBAX_TMP_PREFIX to the public constant ORBAX_TMP_DIR_SUFFIX. The new name matches the suffix semantics.
  • The _has_committed_payload and _is_loadable_checkpoint helpers make the skip decision.
  • It prevents reading a corrupted checkpoint when a crash happened midway through a checkpoint.
• Added exclude_required_columns to kafkaloader. It lets a particular kafka feed exclude some required columns.
  • When an expected column is missing, it skips both the warning log and the schema-mismatch error path.
• Removed 1 line from grox config.py. It is a trivial cleanup, so the concrete diff is omitted.

• Added paragraphs on MACT/view-through 20 actions, the ads_slim metric group, and the delayed feedback split to the ads section.
• Added paragraphs on the Muon export promotion, GB300 override, and xai-o2 stub to Phoenix training·index (introduced in Gen4).
• Reversed the shadow → staging statement for dark_traffic in the safety subsection.
• Folded the statement finalized in Gen4+6 (complete removal of 4.5 internal) into the PTOS judge section.
• Added a ### Resolved by the Gen4+6 release / new (2026-08-23) section to open questions.
• Added the 28e414f commit and related file links to sources.

• Introduced 20 proto ActionNames, greatly increasing the total number of actions.
• Formally handled delayed feedback as a training loss branch. X ranker training is now explicitly aligned with the literature on ads delayed conversion prediction.
• The ads_slim metric group separates the signal combination of the ads ranking line.
• The mact_in_app_loss_weight and train_view_through_heads configs expose a way to tune the training intensity of the ads heads.

At the same time, the Muon training recipe was formally included in the export and a GB300 hardware-specific config appeared for the first time, so this cycle delivered reproducible training infrastructure and confirmation of the hardware generation side by side. The release of the xai-o2 stub states explicitly, for the first time, which storage layer of the infrastructure holds xAI's own system. The redefinition of dark_traffic from shadow to staging matters because it is not just an observation to leave in the documentation but a reinterpretation of Gen4+4 that the canon must reflect.

Corrections

• VF dark_traffic was redefined from production shadow to staging mirroring. When Gen4+4 (2026-08-19) was folded into the canon, the statement was "mirror production traffic to xai-vf-shadow.prod.visibility". After 2 days, the shadow workload reference was deleted and redirected to the staging environment. The statement in the Gen4+4 dark_traffic section of ../services/x.md was rewritten in this Gen4+6 update.

Radar

None. Triaged 30 new arXiv cs.IR entries from 8/19–8/20, and 0 passed. Most are academic papers (dense retrieval RL, sequential rec benchmarks, financial RAG, etc.), and none contained a production claim. There are no new posts on Meta engineering or the Netflix tech blog (Meta's latest post is 2026-08-12, and Netflix's 2026-08-21 post covers the Flink autoscaler and is pure infrastructure).

What to check next