2026-09-10

There were no new sources for Meta and Netflix.

1Per-user ranking weight perturbation and a search-only embedding appeared — X

• A new mechanism lets the ranking scorer perturb engagement-head weights deterministically on every serving request.
• grox (X's nearline LLM annotation pipeline) gained a new multimodal embedding stack dedicated to the search surface.
• Phoenix's (Grok-based Transformer ranker) serving proto gained ContentFeatures, a structured content feature.
• PTOS (policy-based violation category verdicts) gained a deep rescan plan that re-scans only the beginning of a video for adult content.
• Phoenix's FA4 (flash attention kernels written in CuTeDSL) was released at large scale, both forward and backward, for two GPU architectures.

A. A mechanism was added that perturbs ranking weights in a fixed direction per user and head

The ranking scorer can now raise or lower the weights of the 26 engagement heads in a fixed direction per user and per head. Because the default strength is 0.0, default behavior remains deterministic as before.

• home-mixer (the service that serves X home timeline feed requests) gained two new parameters that perturb weights verified — home-mixer/params/param.rs.

  • The perturbation strength is WeightPerturbationSigma (f64, default 0.0).
  • The salt that determines the sign is WeightPerturbationSalt (String, default "").
  • The sync time in the file header was also updated from 2026-09-04T16:22:24Z → 2026-09-08T16:24:06Z.

• The scorer applies this perturbation right after reading the weights from the parameters verified — home-mixer/scorers/ranking_scorer.rs.

  • .perturbed(query) was appended after ScoringWeights::from_params(&query.params).

  • The logic of perturbed() itself is shown below:

    pub(crate) fn perturbed(mut self, query: &ScoredPostsQuery) -> Self {
        let sigma = query.params.get(WeightPerturbationSigma);
        if sigma <= 0.0 {
            return self;
        }
        let salt = query.params.get(WeightPerturbationSalt);
        for (head, weight) in self.weights_mut() {
            *weight *= (sigma * perturbation_sign(&salt, query.user_id, head)).exp();
        }
        self.recompute_sums();
        self
    }
    
    pub(crate) fn perturbation_sign(salt: &str, user_id: u64, head: &str) -> f64 {
        let digest = md5::compute(format!("{salt}:{user_id}:{head}"));
        if digest[0] & 1 == 1 { 1.0 } else { -1.0 }
    }
    

• The perturbation targets all 26 engagement heads verified — the two files above.

  • The target heads are favorite·reply·retweet·photo_expand·video_open·click·open_link·profile_click·vqv·share·share_via_dm·share_via_copy_link·dwell·quote·quoted_click·quoted_vqv·dwell_time·click_dwell_time·active_secs_5m_residual_norm·follow_author·post_unexplored·not_interested·block_author·mute_author·report·not_dwelled.
  • All 26 are targets: the Gen4 ranker's 17 core positive heads, 5 negative heads, and 3 continuous heads, plus 1 post_unexplored. The first three groups correspond to engagement, user feedback, and dwell·click_dwell·active_secs, respectively.

• After the weights are perturbed, the positive, negative, and total sums are also recomputed verified. So downstream gating logic that uses the sums reflects the perturbed values as is.

  • The sum fields are positive_sum·negative_sum·total_sum, and recompute_sums() handles the recomputation.
  • An example of downstream gating is the total_sum scaling of the not_dwelled penalty.

• The sign is determined by md5(salt:user_id:head), so under the same salt the result is fixed per user verified.

  • The same salt and the same user give the same sign. It is deterministic across requests.
  • Even with the same salt and the same user, the sign is independent per head. The 26 heads are perturbed in different directions.
  • Changing the salt reshuffles every per-user assignment. This preserves reproducibility while letting exploration restart by cycling the salt (salt cycling).

• The perturbation multiplier is an exponential function of σ verified. The coefficient multiplier is exp(sigma × sign) = exp(±sigma).

  • At σ=0.1 the weight is multiplied by ×1.105 or ×0.905, and at σ=0.3 by ×1.35 or ×0.74.
  • Because the scale is exponential, multiplicative symmetry is guaranteed. Up and down at σ=0.1 are exact multiplicative inverses, producing a symmetric distribution in log-space.

The purpose of this mechanism can be read three ways. The contract itself is verified, and the purposes are inferred.

1. It can be read as gradient-free infrastructure for weight A/B tests. It is a wobble experiment that measures the sensitivity of weight combinations with production user traffic. Setting a different WeightPerturbationSalt per experiment arm gives each arm an independent noise profile. The contract is verified, and its use for A/B is inferred.
2. It can also be read as on-policy exploration or a bootstrap-style bandit. The interpretation is that perturbing the scores themselves before VMRanker (a remote gRPC service that acts as the slate reranker in home-mixer) forces impressions to diversify inferred. As a bandit technique, it is closer to Thompson sampling than to UCB, but it uses fixed noise instead of a posterior and is deterministic per user-head pair inferred.
3. It can also be read as a tool for model bias auditing or robustness measurement. The interpretation is that it observes which heads the model score stably depends on even when the weights are perturbed inferred.

• Since the default is 0.0, the production default is still deterministic verified — param.rs.
• The mere existence of this contract as an option is a signal. So far Gen4 has introduced stochasticity through individual gates. EnableViewerColdStart's Thompson and ValueModelMode's dwell_regret_sigmoid are examples. This time it added a unified stochastic layer that targets all engagement-head weights.
• This mechanism follows the same trend as the Gen4+16 change that made codivert impressions remain symmetric. The A/B infrastructure itself keeps getting thicker in Gen4.

The following is not public in the open source code verified:

• In which experiment framework the salt value is actually set
• The actual production σ value
• Which surfaces (For You / Following / RankedFollowing) it applies to
• Where result logging flows as a side_effect

B. grox gained a new multimodal post embedding stack dedicated to the search surface

grox built a third pipeline dedicated to the search surface next to the two existing embedding pipelines. It also gained a backfill path that embeds past posts after the fact, and a mechanism that automatically tunes, to match throughput, the fav threshold used to select backfill targets.

• There are 17 new or modified files verified — grox/flows/mm_emb/.
  • 11 new files, 1,026 lines in total, were added. The embedder and renderer are embedder_v8_search.py (113) and renderer_v8_search.py (461).
  • The plan files are plan_post_embedding_v8_search.py (34) and plan_post_embedding_v8_search_for_reply.py (37). The backfill plans are plan_post_embedding_v8_search_backfill.py (43) and plan_post_embedding_v8_search_backfill_batch.py (16).
  • The backfill-side files are task_backfill_fanout.py (85), task_backfill_gate.py (60), and backfill_consumer_dedup.py (49). The fav-related files are batched_fav_hydrator.py (65) and adaptive_min_fav.py (63).
  • The modified files are generators.py (+82), constants.py (+9), and task_multimodal_post_embedding.py (+52). task_write_mm_embedding_sink.py (+90), task_rate_limit.py (+15), and task_embedding_pub.py (+5) were also changed.
• A third pipeline was wired in alongside the two existing embedding pipelines verified.
  • The existing pipelines are legacy v5_1 and current v8_2, and the new pipeline is v8_search.
  • The four new plans are mm_emb_v8_search, mm_emb_v8_search_backfill, mm_emb_v8_search_backfill_batch, and mm_emb_v8_search_for_reply.
• There are separate embedding models for real-time and backfill verified. A separate backfill model being attached means that volume the real-time embedder cannot handle is absorbed by another instance.
  • The two embedding model keys RECSYS_V8_SEARCH_EMBED and RECSYS_V8_SEARCH_BACKFILL_EMBED look up endpoints from grox_config.
• The state key and the version tag are separated, and the new embedding occupies a vector slot independent of v5 and v8.2 verified.
  • The tags are V8_SEARCH_STATE_KEY = "v8_search" and V8_SEARCH_MODEL_VERSION = "v8_search_1".
  • The slot is MultimodalPostEmbeddingState.embeddings["v8_search"].
• Each of the three kafka streams is wired to its corresponding Plan verified.
  • POST_EMBEDDING_V8_SEARCH_STREAM uses the grox-multimodal-embedding-requests-with-summary topic.
  • POST_EMBEDDING_V8_SEARCH_FOR_REPLY_STREAM uses content-understanding-...-min-traction-for-grox-multi-modal.
  • POST_EMBEDDING_BACKFILL_STREAM uses grox-multimodal-embedding-backfill.

The renderer turns one post into one document for embedding. The renderer function is V8SearchEmbedPostRenderer.render_for_embedding.

• An instruction prefix stating the recommendation purpose is attached at the very start of the document verified. This is a common pattern in E5·GTE-family instruction-tuned embeddings.

  • The original prefix text is this single line:

    INSTRUCTION_PREFIX = "Represent this X post (formerly Twitter) for recommendation: capture its topic, intent, key entities, and likely engagement."
    

  • When reproducing, a prompt in the Represent this X post ... recommendation: style induces embeddings aligned to a search-ranking-purpose vector space.

• An author block precedes the post text verified. The block holds the author bio, verified, follower count, following count, and account join month.
  • The follower count is attached in two representations. One is the value that _humanize_followers abbreviates into a human-readable form in K/M/B units, and the other is the bucket from _tier_label (<10·10-100·100-1K·…·>100M).
  • This setup lets the encoder see both signals at once, the "exact number" and the "scale bucket".
• Each text field is truncated at a length cap verified.
  • The body is truncated to MAX_POST_TEXT = 4000 characters, and the quoted post to MAX_QUOTED_TEXT = 1000 characters.
  • The ASR transcript is MAX_ASR = 2000, the card title is MAX_CARD_TITLE = 100, and the article title is MAX_ARTICLE_TITLE = 200.
  • The dangling media URL (https://twitter.com/user/status/N/photo/1) in the body is removed with a regex.
• For media, frames of the original post and the quoted post share one budget verified.
  • The frame budget is MAX_MEDIA = 16. The embedder's actual frame cap (default) is MAX_FRAMES = 4.
  • For video, is_video: True and the transcript are rendered too.
• A post with neither text nor media is discarded downstream verified.
  • The renderer returns document and the embedding becomes an empty list. The warning log Post {post.id} has no text or media content is emitted.
  • The downstream task TaskMultimodalPostEmbeddingV8Search discards it with TaskStopExecution.

The embedder receives the vector through an OpenAI-compatible API and then normalizes it again. The embedder class is MultimodalPostEmbedderV8Search.

• The embedding request sends text and images together through the OpenAI-compatible embed API verified. The timeout is 60 seconds.
  • It calls embed_openai_async(text=text, images=images) with the XaiEmbeddingClientHttp client.
• It looks up the author's core information and passes it to the renderer verified. The lookup uses a StratoUserCore fetch from Strato (the X internal system that code goes through to read or store data).
  • If the lookup fails, the post_embedding_v8_search.core_user_fetch_error counter increments.
• The vector returned by the API is re-normalized to L2 norm 1 verified. This forces dot-product similarity in search ranking to equal cosine similarity.
  • The re-normalization function is _renormalize.
• The observability metrics are three histograms and two counters verified.
  • The histograms are render_duration_ms, encode_duration_ms, and total_duration_ms, and the counters are image_count and core_user_hit.

Separately from the real-time pipeline that processes new posts, a new backfill path embeds past posts after the fact.

• The backfill path splits the post_id list received from kafka into batches, runs dedup and a fav lookup per batch, and sends them to per-instance plans verified. Within the plan, they must pass two gates before being embedded and written to the sink.

  • The diagram below shows the backfill 5-DAG architecture.

    KafkaMmBackfillRequestLoader(TOPIC_BACKFILL="grox-multimodal-embedding-backfill")
      ↓ (splits the byte-packed list<i64> of post_id into batches of size BATCH_SIZE)
    TaskBackfillBatchFanout
      ↓ (BackfillConsumerDedup: removes post_id values already observed within the batch)
      ↓ (BatchedFavHydrator: looks up the fav count per batch)
      ↓ (asyncio.Semaphore(FANOUT_CONCURRENCY=240): cap on parallel execution)
    PlanPostEmbeddingV8SearchBackfill (runs per instance)
      ↓
    TaskBackfillFavGate + TaskBackfillPreloadGate + TaskMultimodalPostEmbeddingV8SearchBackfill + TaskWriteV8SearchSink
    

• The interesting part here is the adaptive fav threshold (AdaptiveMinFav). The fav threshold that selects backfill targets is not a fixed value; it moves with the actual pass QPS verified.

  • The target QPS and the starting threshold are set as below.

    _TARGET_QPS = float(os.environ.get("MM_EMB_BACKFILL_TARGET_QPS", "5"))
    _min_fav = AdaptiveMinFav(start=MIN_FAV_COUNT, target_qps=_TARGET_QPS)
    

  • Like a PI (proportional-integral) controller, it observes backfill's actual pass QPS and adjusts the fav threshold dynamically.

  • If the gate pass QPS is higher than the target (default 5 QPS), it raises the threshold, and if lower, it lowers it. The threshold starts at fav 20k.

  • deadband=0.1 is the range within which no adjustment is made, up to ±10%.

  • warmup_s=60 makes it only observe during the first 60 seconds.

  • gain=0.3 is the adjustment strength relative to the error.

  • floor=0·cap=20000 are the bounds.

• Such a mechanism is needed because backfill traffic must be tuned precisely. It must not exceed the embedding server's GPU/CPU budget and the downstream kafka sink's processing capacity.

  • A hardcoded threshold (e.g., MIN_FAV_COUNT = 20_000) cannot react to kafka replay speed or server queue pressure. AdaptiveMinFav throttles this automatically.
  • What is new is that it applies a rate limiter pattern, common in infrastructure code, to selecting embedding targets.

• The backfill entry filter has two layers, TaskBackfillFavGate and TaskBackfillPreloadGate verified.

  • The first layer, TaskBackfillFavGate, lets in only posts whose like count is at or above the dynamic threshold. The condition is post.counts.likes >= min_fav, and filtered posts are observed via task.backfill_gate.dropped.count{reason=low_favs|fav_count_missing}.
  • The second layer, TaskBackfillPreloadGate, skips posts that already have an embedding. If StratoPostMultimodalEmbeddingMhSearchAi().fetch(post_id, V8_SEARCH_MODEL_VERSION) is non-null, the post is skipped, and it passes only when none exists.

• The PI controller reflects in the target only the QPS that leads to actual embedding verified.

  • It calls _min_fav.observe_pass() when a post passes TaskBackfillPreloadGate.
  • In other words, posts skipped because an embedding already exists (preload hits) do not enter the rate calculation.

This change is the first time a vector space dedicated to the search surface has been separated out.

• Until now, X's grox multimodal embeddings reused a single v5 vector space and a single v8.2 vector space across ranking, retrieval, and search. This was observed in Gen4+13 on 2026-09-01.
• It corresponds to Phoenix expanding to the search ranking surface with the xrecsys_search config in Gen4+4 (2026-08-19). The search document embedding pipeline that pairs with the search ranking model is now complete.

The model and consumer of the search embedding are not public verified.

• The actual models behind the two endpoints RECSYS_V8_SEARCH_EMBED and RECSYS_V8_SEARCH_BACKFILL_EMBED (dimension, base architecture, training data) are not in the open source code.
• How the search surface consumes them (retrieval index name, ANN parameters) is also not public.
• The Kafka topic grox-multimodal-embedding-v8-search was registered, but the consumer-side surface code did not come with it verified.

C. Text structure signals of the post were added to the Phoenix serving request

The Phoenix serving proto gained a new message carrying seven content features for the post and the quoted post, and the home-mixer ranker gained a builder that produces the values. Among them, text length, newline count, and URL presence are text characteristics exposed for the first time.

• Two content feature fields were added to the post info message verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto, phoenix/python/common/xai-proto/proto/recsys.proto.

  • Two fields, contentFeatures and quotedContentFeatures, were added to TweetInfo, and both are of the new message type ContentFeatures. Quoted posts were wired in the same way.

  • The new proto message is shown below:

    message TweetInfo {
      // ...
      ScoreInfo scoreInfo = 35;
      ContentFeatures contentFeatures = 36;
      ContentFeatures quotedContentFeatures = 37;
    }
    
    message ContentFeatures {
      bool hasVideo = 1;
      uint32 maxVideoDurationMs = 2;
      bool hasPhoto = 3;
      uint32 mediaCount = 4;
      uint32 weightedTextLen = 5;
      uint32 newlineCount = 6;
      bool hasUrl = 7;
    }
    

• Of the seven signals, the media signals were previously individual fields and are now bundled into one struct, and the other three are newly exposed text characteristics verified.

  • The media signals are has_video·has_photo·media_count·max_video_duration_ms.

• The text length weighted_text_len is not a simple character count but follows Twitter's own text length counting rules as is (URL=23, CJK 2×, etc.) verified.

  • URLs are always counted as 23 characters. This matches the t.co shortener policy.
  • CJK characters and emoji count as 2 characters. Every character outside the single-weight ranges (SINGLE_WEIGHT_CODEPOINT_RANGES) counts as 2 verified — content_features.rs:72-85.
  • ASCII and basic Latin characters count as 1 character. Punctuation is split by range: ASCII punctuation and general punctuation such as U+2010–U+201F count as 1, and punctuation outside the ranges counts as 2.
  • Variation selectors (0xFE0E·0xFE0F), skin-tone modifiers (0x1F3FB..0x1F3FF), and tag characters (0xE0020..0xE007F) count as 0. They are absorbed into the base emoji.
  • A media attachment URL is not an external URL. If there is a media attachment (attachment_urls=1), the first URL is treated as a media URL and subtracted from the external URL count.

• The newline count newline_count is the number of \n characters verified. It is a proxy for post formats such as tweets that build lists with line breaks or poem-style posts.

• has_url indicates whether there is at least one external URL verified.

• The builder that produces the features exports only two pure functions (build·build_quoted), in Gen4's code style verified — home-mixer/models/content_features.rs (185 new lines).

  • In the builder (home-mixer/models/content_features.rs), the actual computation is handled by TextMetrics::of(text, attachment_urls).

  • Five regression tests pin Twitter's text weighting rule in code.

    assert_eq!(weighted(""), 0);
    assert_eq!(weighted("hello world"), 11);
    assert_eq!(weighted("日本語"), 6);  // 3 CJK characters × 2
    assert_eq!(weighted("see https://t.co/abc123 now"), 3 + 1 + 23 + 1 + 3);  // URL=23
    assert_eq!(weighted("ok 👍🏽"), 2 + 1 + 2);  // skin-tone modifier counts as 0
    assert_eq!(weighted("日本\u{3000}語"), 4 + 2 + 2);  // ideographic space also counts as CJK
    

In Gen4+17, the structural signals of the text itself were promoted to a first-class field of the serving wire format.

• Until now, the content signals the Phoenix ranker consumed were of two kinds. One is hashed features looked up from the embedding table, and the other is scattered booleans and integers such as has_media·has_video·view_count.
• In particular, weighted_text_len is directly tied to the post's 280-character cap (by weight) verified — X developer docs Counting Characters. So the ranker can explicitly learn "how full this post is / whether it is a pass-through that only drops in a URL".

However, how the ranking heads consume this signal is not in this commit verified. This is because there is not yet any code in phoenix/xrex/models/recsys_model.py that reads content_features verified. It is on the wire, but whether training consumes it is something to observe in the next commit.

D. PTOS gained a path that crops only the first 10 seconds of long videos to re-judge adult content

A new path crops the opening portion of videos 30 seconds or longer that already passed PTOS once but carry no NSFW tag, and re-judges them. Only popular posts enter, and a separate rate limit caps the cost.

• This path consists of a new plan, an entry filter, and added constants verified — grox/flows/ptos/plan_safety_ptos_adult_content_leading_frames.py (56 new lines), grox/flows/ptos/task_safety_ptos_adult_content_leading_frames_filter.py (81 new lines), grox/flows/ptos/constants.py (+15).
• To become a rescan target, a post must pass all six gates verified. The gates are in TaskSafetyPtosAdultContentLeadingFramesFilter._eligible_with_post.

1. The post must have an author (post.user). Anonymous users and deleted users are excluded.
2. The fav count must be at or above the deluxe-tier threshold (post.get_fav_count() >= deluxe_fav_count_threshold). This threshold was introduced in Gen4+12 for cost control, and only popular posts enter.
3. post.media or post.quoted_post.media must contain a video of 30 seconds or longer. The criterion is ADULT_CONTENT_LEADING_FRAMES_MIN_DURATION_SECONDS = 30.0.
4. A prior PTOS verdict must exist. StratoSafetyPostAnnotationsResultDirectMh.fetch(post_id) must be non-null. That is, posts that already passed through the PTOS pipeline once are the targets.
5. The prior verdict must not be NSFW. If prior.safetyBoolMetadata.isNsfw == True, the post was already handled and is skipped.
6. The prior verdict must have a violated policy in the AdultContent category, or the fav count must be 512 or more (ADULT_CONTENT_LEADING_FRAMES_UNCONDITIONAL_FAV). Meeting either one is enough.

• A video that passes is cropped to its first 10 seconds and re-judged verified.
  • video.crop_seconds = 10.0 (ADULT_CONTENT_LEADING_FRAMES_CROP_SECONDS) is set, and downstream media hydration crops and judges only that length.
  • The video then goes through the 8-stage DAG again: category detection → policy detection → safemodel sex/nudity verdict → cross-validation.

This path is needed for the following reasons:

• Videos longer than 30 seconds have a high media hydration cost (decoding, frame extraction, model calls). So the real-time pipeline extracts and judges only predetermined sample frames.
• However, if an evading user fills the first 15 seconds or more with normal content and places adult content later, the real-time verdict may miss it.
• PlanSafetyPtosSpecialVideo, introduced in Gen4+12, handles content hidden by static overlays such as captions or watermarks. The path in Gen4+17 handles time-axis evasion, which disguises the first few seconds as safe.
• Despite the name leading-frames, it actually "crops the first 10 seconds and re-judges", so it appears aimed especially at cases where adult content appears briefly at the start inferred.
• Some posts have an AdultContent violated policy in the prior verdict but are not isNsfw. This is because the Grok 4.5 judge in the Gen4+2 adult content cross-validator allows only demotion. So even when the primary is AdultContent, it may not harden into an NSFW tag.
  • This path acts as a second judge that can overturn a demoted verdict.

The implication of the 512 fav entry threshold is also worth reading.

• The deluxe threshold (deluxe_fav_count_threshold) is likely a much higher value (the Gen4+7 log suggests deluxe = 5,000 or more inferred). So unconditionally admitting only cases with 512 or more favs and no prior AdultContent is meant to cover low-intensity signal cases where the prior verdict did not catch AdultContent.
• 512 counts as a moderately popular post on Twitter. At this scale, user reactions to false positives or misses (reports, mutes) have likely already occurred.

The operating figures and the judge model are not public verified.

• How often this path is actually triggered in production
• The rate at which re-judging actually overturns verdicts (precision and recall)
• Which judge model is used
  • The plan DAG shows that the regular category, policy, safemodel and cross-validation stages are all run again verified.

However, a separate rate limit (TaskRateLimitSafetyPtosAdultContentLeadingFrames) is configured, so there is a cost cap verified.

E. Phoenix FA4 attention kernels were released with forward and backward for both SM80 and SM90

Phoenix's FA4 attention kernels were released with both forward and backward for two GPU architectures (SM80 Ampere, SM90 Hopper). This is the first time the original backward pass has been released.

• Six files were added and five files were modified verified — phoenix/xrex/cutedsl/ranker_fa4/.

• The new code totals 7,080 lines, and the modifications are also large verified. This is the strand with the most lines in this commit verified.
• The released kernels are a fork of the original Dao-AILab/flash-attention code, modified by X.AI verified. The file headers state the source and license.

  • The header text is copied as is.

    # Derived from flash-attention (https://github.com/Dao-AILab/flash-attention);
    # modified by X.AI Corp.
    # SPDX-License-Identifier: Apache-2.0
    # Copyright 2026 X.AI Corp.
    

  • The license is Apache-2.0. Relicensing from the original BSD 3-Clause is allowed.

  • A copyright notice for the flash-attention subcomponent was also added to the phoenix/NOTICE file (+198 lines).

• The kernel classes (FlashAttentionForwardBase, FlashAttentionBackwardSm80) reveal the main characteristics verified.
  • They are written in CuTeDSL (the cutlass Python DSL), a new approach to metaprogramming NVIDIA's CUTLASS kernels in Python. The code loads it with import cutlass and import cutlass.cute as cute.
  • They support asymmetric attention, where the hidden dim of Q·K and V may differ (head_dim != head_dim_v). This is a trick in ranking models to reduce the memory footprint by lowering the value dimension.
  • They support Grouped Query Attention (pack_gqa). Several Q heads share one KV head, so an optimization common in LLMs is also applied to ranking attention.
  • They support is_causal, is_local, score_mod and mask_mod. In ranking, causal masking is needed for sequence history attention, and score_mod and mask_mod are the user-defined modifier hooks of Flex Attention.
  • The has_aux_tensors option is a path for feeding extra sideband tensors into attention. It is likely connected to the backbone scores introduced in Gen4+14 inferred.
  • Two schedulers, SingleTileScheduler and SingleTileVarlenScheduler, handle fixed-length and variable-length seqlen respectively. The signature extension of build_dense_block_sparse_layout introduced in Gen4+15 is now complete through varlen backward.
• Some characteristics exist only in the backward kernel (FlashAttentionBackwardSm80) verified.
  • The tile sizes (m_block_size=64, n_block_size=128, num_stages_Q=2, num_stages_dO=2, num_threads=256) are matched to the register and shared memory budget of the SM80 generation.
  • The JIT function _bs_bwd_seq_at performs the index mapping among three iterator domains (mask axis, full axis, diag axis) with constexpr subtiles. This is the core of block-sparse backward.
  • Preprocess and postprocess are separated. Using the softmax_LSE (log-sum-exp) saved in forward, pre-bwd precomputes dO·row_sum, and this stage is the 329 lines of flash_bwd_preprocess.py.
  • The main bwd kernel uses these values, and post-bwd accumulates the partial sums of dQ (flash_bwd_postprocess.py).
  • The dual-block iteration from the Flash Attention 2 paper was reimplemented in CuTeDSL.

This release means three things:

• On the open-source side, this is the first release of the original backward pass. Gen4+15 (see the 2026-09-05 daily) released the rewrite of FA4 forward varlen support. Backward is essential for training, so until now it existed only in X's internal repository, and the open-source mirror had only forward. Now the training and serving attention kernels of the X ranker are fully open source.
• The SM90 connection is confirmed. When the SAVE_H100_RECSYS = 32 remat policy (a memory-saving approach that recomputes activations instead of storing them during backpropagation) was introduced in Gen4+15, we only observed it as "the first code signal that training runs on H100 as well".
  • This commit explicitly includes both the forward and backward files for SM90 (H100), so the H100 training connection is now complete. What the SAVE_H100_RECSYS remat policy actually uses is now confirmed in code.
  • The SM90/H100 alternative path to _GB300_OVERRIDES in Gen4+6 also became clear. GB300 is Grace Blackwell 300, the SM100 generation.
  • X maintains attention kernels optimized for each of two hardware generations, GB300 and H100.
• X reused the flash-attention upstream code. It chose to fork and modify the Dao-AILab original rather than write CuTeDSL from scratch. This choice signals that the attention kernel is standardized infrastructure verified by the LLM community, and that X has not departed from that standard.

Performance figures and the actual sparse pattern are not public verified.

• Actual benchmarks of the SM90/SM80 kernels (GB/s, TFLOPS, speedup over baseline) are not in the open source.
• block_sparse_utils.py was substantially rewritten, but which sparse pattern is actually used is not stated in the model config verified.

F. The other changes center on serving infrastructure cleanup and the connections that support B–D

The remaining changes are collected one per line: serving and training infrastructure cleanup, connections that support B–D above, and small edits with no semantic change.

• Replaced the device blob of the CutlassCall custom_call with a SHA256 digest to stabilize the AOT cache key verified — phoenix/xrex/utils/aot.py (+54/−1).
  • _sanitize_cutlass_calls walks the stablehlo.custom_call{target=CuteDSLRT_NvJaxCutlassCall} ops in StableHLO. It replaces the module field (ELF/fatbin bytes) of backend_config with the SHA256 digest device_digest:<hex>.
  • cutlass_module_device_digest parses the ELF header and hashes only the device blob section.
  • AOT_CACHE_VERSION was also updated from "20260705" to "20260904".
  • As a result, the JAX AOT cache has a stable cache key for the same CUDA kernel binary and avoids recompilation hell. This connection is essential for cases with many embedded kernels, such as ranking models.
• The serving loader (copy_port_client.rs) checks for the existence of MANIFEST.json and picks only checkpoints whose bundle is complete verified — phoenix/crates/serving/xai-recsys-engine/src/copy_port_client.rs (+207/−9).
  • full_prefixes_newest_first was split out as a helper that returns the prefixes present in all channels, newest first.
  • select_bundle_checkpoint picks the newest checkpoint for which export/MANIFEST.json exists non-empty in every channel. The newest checkpoint that is still unbundled is skipped with a warn log.
  • download_named_files is new and atomically downloads a list of specific file names. It randomly picks a channel in which all the files exist.
  • The .mlirbc bundle system (StableHLO export) from Gen4+13 is now fully connected to the serving loader. Only checkpoints whose bundle is complete are loaded for serving.
• The serving sequence_len is now computed automatically from the training config verified — phoenix/xrex/train/recsys_bundle_export.py (+48).
  • _serving_sequence_len back-computes the prefix token length from the training config. The values used are sequence_len, history_seq_len, candidate_seq_len, num_negatives_per_example and num_global_negatives_per_example.
  • At serving time, it overrides the attention kernel size with serving_seq_len = prefix + history + candidate.
  • If it disagrees with num_user_prefix_tokens, it fails closed with a ValueError.
  • model_sequence_len was also added to the MANIFEST.
• quote_hydrator.rs was expanded substantially and now also hydrates the media info of quoted posts verified — home-mixer/candidate_hydrators/quote_hydrator.rs (+142/−12).
  • MediaInfoCacheClient is injected as a new dependency. get_quoted_content looks up the core data and media info of the quoted post in parallel within a 200ms timeout, and falls back on a cache miss.
  • This is an upstream condition for the ContentFeatures.quotedContentFeatures connection in Gen4+17.
• Refactored the media info lookup (media_info_hydrator.rs +70/−15) verified. The path that hydrates max_video_duration_ms and similar values needed for the content features computation was widened here.
• The PTOS reclassification trigger was widened verified. The policy detection in task_safety_ptos_policy.py (+4/−1) also handles re-judging on the leading-frames path.
  • TaskRateLimitSafetyPtosAdultContentLeadingFrames was added to task_rate_limit.py.
• The whole file home-mixer/bin/pipeline_components_json.rs (76 lines) was deleted verified. It appears to have been a JSON dump utility for offline debugging, and we presume it could no longer be maintained inferred.
• Eight candidate-pipeline files were lightly edited (+103/−38) verified. The signatures of the pipeline core were tidied, with no separate semantic change.
• safety_label_source/proto.rs of VF (visibility-filtering, a service that decides for each post whether this viewer may see it) changed by +34/−38 verified. The proto was slightly extended, with no semantic change.
• Lightly edited video_tools to improve video hydration accuracy verified. The changes are image.py +42/−19, subtitles.py +3/−1 and video_frames.py +7/−7.
• A new API was added to the VF client (visibility-filtering-client/vf_client.rs, +119/0) verified. Whether it is a semantic extension or a refactor needs separate confirmation.
• Made minor cleanups to the multi-step reply spam pipeline introduced in Gen4+15 verified. In grox/flows/reply_spam, −17 lines were removed from task_write.py and −9 lines from task_multi_step_reply_spam.py.
• The five pipeline edits in grox/flows/mm_emb finish wiring the v8-search stack from B verified.
  • The v8-search sink was connected in task_write_mm_embedding_sink.py (+90).
  • The v8-search and v8-search-backfill tasks were added to task_multimodal_post_embedding.py (+52).
  • The rest are task_rate_limit.py (+15), task_embedding_pub.py (+5) and constants.py (+9).

1. A/B and exploration infrastructure is being built up in layers. Following the codivert impression in Gen4+16 and the backbone scores request in Gen4+14, this weight perturbation is the first unified contract at the ranking scorer layer that can inject stochasticity into real user traffic.
   • It does not exactly match any line of the bandit literature, whether Thompson sampling, bootstrap or Rademacher perturbation. It combines salt-based deterministic reproducibility with independence across user-head combinations obtained via md5.
   • The production default is off, but the connection itself is complete.
2. The separation of search from ranking is deepening. The search surface now has a vector space, model and backfill pipeline completely separate from For You.
   • The flow is Phoenix xrecsys_search (Gen4+4) → the non-padding connection of search-query embeddings (Gen4+13) → this v8-search embedding pipeline.
   • How much thicker X search becomes than in earlier generations is the point to watch.
3. The layers of content understanding and safety are rising. In content verdicts and ranker inputs, defense layers against text, video, time and overlay evasion are each growing as separate paths.
   • ContentFeatures: text-structure signals as ranking input
   • leading-frames rescan: response to time-axis evasion
   • Special Video motion-reveal: response to static-overlay evasion (Gen4+12)
   • PTOS cross-validator: 3 categories (Gen4+7)
   • Adult content cross-validator (Gen4+2)

Some things also remain unknown from this commit alone.

• The actual production σ value of the weight perturbation, the salt assignment logic, and how to observe the results
• The dim, base model and retrieval index connection of the v8-search embeddings
• In which ranking head and how ContentFeatures is actually used (to observe in the next commit)
• Benchmark figures (GB/s, TFLOPS) of the FA4 SM80/SM90 kernels
• The actual reclassification rate (precision and recall) of the leading-frames path
• The details of what the MediaInfoCacheClient connection in quote_hydrator.rs hydrates for the first time in Gen4+17

Corrections

No conclusions were overturned. This commit added new evidence to the thread and does not conflict with existing conclusions. We note only that the Gen4+15 "first code signal that training runs on H100" is now fully confirmed by the original FA4 SM90 forward and backward in this commit.

Radar

No new entries were added to the radar. We parsed 30 arXiv cs.IR papers, and 0 passed the acceptance criteria (platform-company authors or a production A/B claim). The candidates that stood out are as follows:

• SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching models sequences of 100K length in recommendation. It was rejected because it has no production A/B.
• EAGER: Enrich-and-Align Generative Query Recommendation from Clicked Items in E-commerce Search recommends queries generatively in e-commerce search. It has neither a platform affiliation nor production results.
• Task-Blind No MORE: Multi-Task Information Flow in Unified Ranking Backbones deals with multi-task information flow in unified ranking backbones. It is an academic increment and was rejected.
• Exploring Bottom-Up Clustering for Creating Semantic IDs is a methodology for generating Semantic IDs. It has no evidence to attach to the thread already tracked in topics/semantic-ids.md.

What to check next