0x00 한

Research / Daily reports

  • Sources read: 1
  • Canon files updated: 1
  • Radar entries: 0
In one lineX made it possible to pass a seed post into the diversity re-ranking (DPP) of the For You slate. Candidates similar to this post are more likely to be dropped from the result.

1Diversity re-ranking can now push out candidates similar to a seed post — X

Source xai-org/x-algorithm@6bb4594 — public source code, 2026-09-12, 24 files, +1397/−338. These notes number X commits after the Gen4 release (2026-08-13) in sequence, and this commit is Gen4+20.

Background Since the Gen4 release on 2026-08-13, X has mirrored the For You pipeline source code to xai-org/x-algorithm once or twice a day. The preceding Gen4+19 (fee1d0f, 2026-09-10) unified the hydration timeout. This commit (2026-09-12, 6bb4594) is a mid-sized diff that continues that line and touches all five axes accumulated in the canon so far: ranking reordering, logging and training pipeline, infrastructure migration, wire extension of the ranking model Phoenix, and cleanup and policy in the impression-verdict service VF. The five axes are independent, but landing together in one commit is a signal that they are roughly contemporaneous projects.

A. If a request supplies a seed post, candidates similar to it are more likely to be dropped from the DPP result

For You re-orders the slate (the bundle of candidates sent out at once) with DPP just before final batching. DPP (determinantal point process) is a method that looks at the whole candidate set so that similar items are not picked together. Until now this stage computed only cosine repulsion between candidates within the candidate set. Now, when a request carries a seed post in seed_tweet_id, it also computes the similarity between the other candidates and the seed. The seed itself is filtered from the final result, so the top_k budget is unchanged. The diversity axis can now extend to comparison against a reference tweet, not just within the set.

  • The stage is handled by VMRanker DPP (θ=0.65, K=150).

  • The embedding of the tweet that seed_tweet_id points to enters the kernel as pinned.

  • The rescore signature in vm-ranker/dpp.rs was extended to take a new seed argument.

    pub fn rescore(
        inputs: &[DppInput],
        seed: Option<&DppInput>,
        config: &DppConfig,
        _viewer_id: u64,
    ) -> Vec<DppResult> { ... }
    
  • The new path works in the following four steps:

    1. Grow the kernel by one for the seed. Add one seed to the existing top-K candidates (m = min(inputs.len(), max_selected_rank)) to build a kernel of size total = m + 1. The seed has no score, so its q value is arbitrarily fixed at 1.0.
    2. Pin the seed to the first slot of greedy DPP. greedy_dpp previously picked the index with the largest kernel diagonal as the starting point. Now, if the new argument pinned_first: Option<usize> is given, that index is forced as the first selection, and this index is the seed's position.
    3. Increase the budget. If pinned is present, raise it to max_items = (k + 1).min(n) so the seed does not consume top_k.
    4. Filter the seed from the result. Before the final return, the seed index is excluded with dpp_result.selected.iter().filter(|&(ki, _)| ki < m).
  • The net effect of these four steps is two contracts:

    • Candidates close to the seed's embedding get larger off-diagonal kernel entries, so the determinantal gain shrinks and they are more likely to be excluded from the result.
    • The seed itself does not appear in the result and does not use up top_k.
  • Two new tests verify these contracts.

    #[test]
    fn seed_suppresses_near_duplicate() {
        let seed = make_input(999, 0.0, &[1.0, 0.0]);
        let inputs = vec![
            make_input(1, 10.0, &[1.0, 0.0]),   // exactly the same direction as the seed
            make_input(2, 5.0, &[0.0, 1.0]),    // orthogonal
        ];
        // Without a seed, rescore puts both 1 and 2 in ids
        // After rescoring with the seed, 1 is excluded, only 2 survives, and 999 is not in the result either
    }
    
    #[test]
    fn seed_does_not_consume_top_k_budget() {
        let seed = make_input(999, 0.0, &[0.0, 0.0, 1.0]);
        let inputs = vec![
            make_input(1, 10.0, &[1.0, 0.0, 0.0]),
            make_input(2, 5.0,  &[0.0, 1.0, 0.0]),
        ];
        let config = DppConfig { top_k: 2, ... };
        let result = rescore(&inputs, Some(&seed), &config, 0);
        assert_eq!(result.len(), 2, "seed must not eat top_k budget");
    }
    
  • If the seed's embedding is not in the serving-time store, it flows through the existing path without pinning.

    • The place that fills the new parameter is rank in vm-ranker/scoring/dpp_model.rs.

      fn build_seed_input(req: &RankRequest, ctx: &DppContext) -> Option<DppInput> {
          if req.seed_tweet_id == 0 { return None; }
          match ctx.store.client.get(req.seed_tweet_id) {
              Some(embedding) => {
                  DPP_SEED_CONTEXT.with_label_values(&["pinned"]).inc();
                  let norm = l2_norm(&embedding);
                  Some(DppInput {
                      id: req.seed_tweet_id,
                      score: 0.0,
                      embedding,
                      norm,
                      embedding_missing: false,
                  })
              }
              None => {
                  DPP_SEED_CONTEXT.with_label_values(&["embedding_missing"]).inc();
                  None
              }
          }
      }
      
    • The counter vm_ranker_dpp_seed_context_total{result="pinned"|"embedding_missing"} separates hits and misses for observation.

  • If the embedding norm is near 0 and the kernel diagonal is at or below EPSILON, it gives up the pin and returns to the existing greedy path.

    pinned = pinned_first.filter(|&p| kernel[p * n + p] > EPSILON); — vm-ranker/dpp.rs

Where this path will be used cannot be confirmed from this commit alone. The home-mixer (the service that assembles the For You feed) pipeline that would carry seed_tweet_id is not in this diff. Natural consumers might include the following inferred:

  1. Related tweets on a permalink page: a slate "dissimilar to this tweet"
  2. An alternative collection in a "more like this" flow for a specific tweet
  3. A scenario in thread content where the parent tweet is the seed and similar replies are suppressed

Whether the For You home timeline itself fills seed_tweet_id is something to watch. In theory For You generates a slate without a seed, but an extension that passes a recently viewed tweet as the seed to reduce repeated exposure is logically possible.

B. Phoenix candidate score logs go out on both the existing Thrift stream and a new proto stream

home-mixer gained one more channel for sending Phoenix candidate scores to Kafka. The existing channel uses Thrift format, and the new channel publishes in proto format to a different topic in parallel. When downstream moves to proto is outside this diff.

  • Until now there was one channel, which sent Thrift ScoredCandidate to PHOENIX_SCORES_TOPIC (cluster Aiml). This payload carries 28 engagement head scores and candidate metadata.

  • This commit added a second channel that publishes proto LoggedScoredCandidate in parallel to LOGGED_SCORED_CANDIDATES_TOPIC (cluster Phoenix).

  • Now, for each candidate, two payloads are built and written to two Kafka publishers at the same time.

    • The file is home-mixer/side_effects/phoenix_experiments_side_effect.rs, and tokio::join! is used for the concurrent writes.
  • The first payload is the existing Thrift ScoredCandidate. This stream has been maintained since the v3 schema in Gen4+3 (11a71f8, 2026-08-18).

    • The field is a sorted collection, as in prediction_scores: Option<BTreeSet<Box<PredictionScore>>>.
    • The Kafka topic is PHOENIX_SCORES_TOPIC and the cluster is Aiml.
  • The second payload is the proto LoggedScoredCandidate that is new in this commit.

    // (the proto definition is in xai_proto::recsys_logging)
    message LoggedScoredCandidate {
        int64 post_id = ...;
        int64 viewer_id = ...;
        int64 author_id = ...;
        int64 request_join_id = ...;
        int64 request_time_ms = ...;
        int64 source_post_id = ...;
        bool  is_in_network = ...;
        int64 in_reply_to_post_id = ...;
        int64 quoted_post_id = ...;
        int64 quoted_user_id = ...;
        map<string, double> prediction_scores = ...;
        int64 fav_count = ...;
        int64 reply_count = ...;
        int64 retweet_count = ...;
        int64 quote_count = ...;
        bool  has_media = ...;
        int32 language_code = ...;
        int32 video_duration_ms = ...;
    }
    
  • The Kafka topic name of the new payload is the new constant LOGGED_SCORED_CANDIDATES_TOPIC. The cluster changes to the new KafkaCluster::Phoenix, separate from the existing Aiml.

    • The production assembly code of PhoenixCandidatePipeline assigns the sixth Kafka publisher slot to this topic.

      async {
          Arc::new(
              ProdKafkaPublisherClient::new(
                  LOGGED_SCORED_CANDIDATES_TOPIC,
                  KafkaCluster::Phoenix,
              ).await,
          ) as Arc<dyn KafkaPublisherClient>
      },
      
  • The two payloads are re-sorted into different data structures. The pipeline builds a HashMap<String, f64> internally and sorts and wraps it into a BTreeSet only for the Thrift payload. For the proto payload, the HashMap is carried as is as a map.

    let mut prediction_scores = HashMap::new();
    for (cluster_id, result) in &experiment_results {
        ...
        insert_cluster_scores(&mut prediction_scores, &cluster_name, &scores);
    }
    
    let thrift_scores: BTreeSet<Box<PredictionScore>> = prediction_scores
        .iter()
        .map(|(name, value)| { Box::new(PredictionScore::new(...)) })
        .collect();
    
  • The key naming convention (phoenix.<cluster_name>.<action_name>) and the score field set are kept as they are. The field set is the 28 engagement heads and the pdwell alias.

  • Tests were also added for the new helpers insert_cluster_scores and build_logged_scored_candidate, respectively.

Why it matters The map representation of prediction_scores gains on two axes. On wire size, the sort index disappears; on consumer-side lookup, a map gives O(1) lookup instead of a BTreeSet. The Thrift stream is kept because downstream consumers cannot switch immediately inferred. How long the transition period of running both streams in parallel will last is not in this commit.

C. VF moved how it finds TES and Gizmoduck servers to go through xDS

The way the visibility-filtering (VF) service connects to TES and Gizmoduck changed. TES (Tweet Entity Service) is X's core tweet lookup service, and Gizmoduck is the user profile service. The connection path was migrated from datacenter-templated hostnames to an xDS LDS listener. The service discovery layer moved to Envoy-family xDS.

  • Previously, the bundle of parameters needed to assemble the client was isolated in a struct defined inside the service crate. The service name was built by filling a hostname template with a datacenter string.

    • The file is visibility-filtering/server_deps.rs.

    • The struct is xai_core_entities::tweet_entity_service_client::TESClientConfig, and the value that fills the hostname template is datacenter.

      // Before (~fee1d0f)
      let tes_client = Arc::new(
          init_client_with_retry("tes", init_deadline, || {
              ProdTESClient::new_with_config(
                  None,
                  datacenter,
                  None,
                  tes_client_config(deterministic_aperture),
              )
          }).await
      );
      // tes_client_config returns a TESClientConfig struct:
      // aperture_size, deterministic_aperture, lb_policy, readiness_probe_port,
      // and fills fields such as request_timeout_ms and enable_serve_within.
      
  • This commit assembles the TES client as follows:

    // New (6bb4594)
    let tes_client = Arc::new(
        init_client_with_retry("tes", init_deadline, || async move {
            let strato = build_xds_strato(
                XdsStratoParams {
                    name: "tes-xds",
                    xds_listener: "tweet-entity-service.prod.tweet-entity-service:fed-grpc",
                    tls_domain: format!(
                        "tweet-entity-service.tweet-entity-service.prod.{datacenter}.s2s.twttr.net"
                    ),
                    lb_policy: LbPolicy::penalized_peak_ewma(),
                    client_id: S2S_CLIENT_ID.clone(),
                    retry_config: Some(RetryConfig::for_idempotent()),
                    max_batch_size: TESRpcConstants::max_batch_size(),
                    enable_serve_within: true,
                },
                deterministic_aperture,
            ).await?;
            anyhow::Ok(ProdTESClient { grpc_client: Arc::new(strato) })
        }).await
    );
    
  • The new helper build_xds_strato assembles the gRPC channel with xai_x_rpc::grpc_client::ChannelBuilder.

    async fn build_xds_strato(
        params: XdsStratoParams,
        deterministic_aperture: bool,
    ) -> anyhow::Result<StratoGrpc> {
        let mut builder = ChannelBuilder::new(params.name)
            .tls(TlsMode::mtls_from_env()?.with_domain_override(params.tls_domain))
            .request_timeout(STRATO_REQUEST_TIMEOUT)     // = HYDRATION_TIMEOUT (390 ms)
            .connect_timeout(STRATO_CONNECT_TIMEOUT)     // 400 ms
            .xds(StartFrom::Lds(params.xds_listener.to_string()))
            .await?
            .eager_resolution(XDS_EAGER_RESOLUTION_TIMEOUT)  // 5 s
            .aperture(STRATO_APERTURE_SIZE)              // 12
            .readiness_probe(READINESS_PROBE_PORT, "/ready")
            .buffer_max_wait(DEFAULT_BUFFER_MAX_WAIT)
            .total_timeout(DEFAULT_TOTAL_TIMEOUT);
        if deterministic_aperture { builder = builder.deterministic(); }
        let channel = builder.build_load_balanced(params.lb_policy).await?;
        let mut strato = StratoGrpc::from_load_balanced_channel(
            channel, None, Some(params.client_id),
            params.retry_config, params.max_batch_size,
        );
        if params.enable_serve_within {
            strato = strato.with_op_context(xai_strato::strato_proto::OpContext {
                serve_within: Some(xai_strato::strato_proto::ServeWithin {
                    duration_microseconds: STRATO_REQUEST_TIMEOUT.as_micros() as i64,
                    round_trip_allowance_microseconds: 10_000,
                }),
                ..Default::default()
            });
        }
        Ok(strato)
    }
    
  • The key is the single line .xds(StartFrom::Lds(params.xds_listener)). The client talks to the xDS control plane, starting from a listener name (such as "tweet-entity-service.prod.tweet-entity-service:fed-grpc"), and receives the endpoint list.

    • xDS is the x Discovery Service protocol, and the starting point of this code is LDS (Listener Discovery Service) among them.
    • This is the standard way in Envoy-family service meshes for a client to dynamically receive the target endpoint list from the control plane.
  • Two options attach only to the TES path:

    • enable_serve_within: true attaches a deadline hint on the server side. The hint tells the server how long a request may wait in server queueing. duration_microseconds is the same value as the client timeout (HYDRATION_TIMEOUT = 390 ms), and round_trip_allowance_microseconds = 10_000 is a 10 ms round-trip allowance.
    • retry_config: Some(RetryConfig::for_idempotent()) automatically retries idempotent requests.
  • On the Gizmoduck path, lb_policy is least_request() and both options above are off verified — server_deps.rs.

  • This migration changed four things:

    1. The service discovery layer moved from hostname templates to going through the xDS control plane.
    2. The separate client-config structs the service crate used to export (TESClientConfig, GizmoduckClientConfig) are gone, and the client assembly logic was absorbed into the VF service's own server_deps.rs.
    3. The Strato (X's internal system that code goes through to read or store data) client of the L2 cache warmer now also takes the deterministic_aperture flag.
    4. Because TES_READINESS_PROBE_PORT and GIZMODUCK_READINESS_PROBE_PORT were both 8081, they were merged into the unified constant READINESS_PROBE_PORT: u16 = 8081.

It can also be read as the Gen4+19 HYDRATION_TIMEOUT unification having propagated this far. This time, the timeout constant in the client assembly code is referenced as STRATO_REQUEST_TIMEOUT: Duration = crate::hydration::HYDRATION_TIMEOUT.

D. Phoenix proto gained LLM-judged search relevance actions and a new logit response format

The Phoenix proto gained two new actions that represent search relevance judged by an LLM. A response format that returns the logits of the requested actions as a positional dense array instead of a sparse HashMap was also added. The new actions are read as a signal that the xrecsys_search search ranking config, which appeared in Gen4+4 (aad7179, 2026-08-19), is starting to prepare to supervise search relevance prediction with LLM judgments as well.

  • The new actions are the pair LLM_JUDGE_RELEVANT_TO_SEARCH/_NOT_RELEVANT_TO_SEARCH, and the new response format is returnLogitsList.

    • The numbers are LLM_JUDGE_RELEVANT_TO_SEARCH = 107 and LLM_JUDGE_NOT_RELEVANT_TO_SEARCH = 108.
  • Two fields and two enum values were added to both recsys.proto files (Rust and the Python mirror).

  • The two new actions went into the action enum.

    enum ActionName {
      ...
      CLIENT_TWEET_SEARCH_QUERY_REFORMULATED = 104;
      CLIENT_TWEET_RELEVANT_TO_SEARCH = 105;
      CLIENT_TWEET_NOT_RELEVANT_TO_SEARCH = 106;
      LLM_JUDGE_RELEVANT_TO_SEARCH = 107;
      LLM_JUDGE_NOT_RELEVANT_TO_SEARCH = 108;
      ADS_WEB_CONVERSION = 128;
      ...
    }
    
  • The search relevance axis had only client-side signals until now. These are signals showing whether the user actually opened a result and whether they rewrote the query. In this commit, the LLM judge's verdict was incorporated as a separate action axis.

  • xrecsys_search, which appeared in Gen4+4 (aad7179, 2026-08-19), is a config with a separate vocab and search-specific settings such as condition_search_relevance_on_prompt=True; this incorporation is read as preparation to put LLM judgments in as a supervision signal on the action axis that config uses as a training target inferred.

  • A pipeline in which Grox (a Grok-based nearline classifier sub-repo) or a dedicated LLM judge feeds relevant/not labels to (query, tweet) pairs must exist somewhere. However, the code for that pipeline is not in this diff.

  • The proto definition of the new response format returnLogitsList is below, and the Python mirror file is identical.

    message PredictNextActionsRequest {
      ...
      bool returnBackboneScores = 20;
      map<string, string> experiment_overrides = 21;
      bool returnLogitsList = 22;
    }
    
    message NextActionDistribution {
      ...
      repeated float backboneTopLogProbs = 10;
      repeated float backboneContinuousValues = 11;
      repeated float requestedActionLogits = 12;
    }
    
  • On the Rust engine side, a return_logits_list: bool field was added to PredictRequestItem. Inside PredictRequestBatch::forward, when this flag is on, the response takes the following path:

    if item.return_logits_list {
        dist_entry.requested_action_logits = item
            .requested_action_indices
            .iter()
            .map(|&idx| {
                dist.get(idx as usize).copied().unwrap_or(f32::NEG_INFINITY)
            })
            .collect();
    } else if item.return_log_map {
        for idx in item.requested_action_indices.iter() {
            if let Some(v) = dist.get(*idx as usize) {
                dist_entry.index_to_logits.insert(*idx, *v);
            }
        }
    }
    
  • The two formats have mutually exclusive priority. If both are on, return_logits_list wins.

  • Continuous action index handling is now shared by both formats. Previously it was only inside the return_log_map path.

Why it matters The new format has guaranteed order and stores no indices, so the wire size is smaller. The existing return_log_map is a sparse map that carries HashMap<u32, f32> (action_index → logit) on the wire. This representation is good when the data is sparse, but when the requested action indices are a densely fixed set, the map key overhead is pure loss. returnLogitsList is a positional dense list that returns a float array in requested index order, and it fills with -inf if a requested index falls outside the model vocab. It is likely optimized for consumers whose action indices are already known, such as cascades or real-time rankers inferred.

E. A representation that packs author features into 8 bytes is ready but nothing uses it yet

VF prepared a new representation that packs author features into 8 bytes, but no code in this commit uses it. AuthorFeatures was shrunk to 16 bytes in Gen4+19, and the new representation CompactAuthorFeatures packs it into a single 8-byte NonZeroU64.

  • Gen4+19 changed the label representation AuthorLabelSet(HashSet<LabelValue>) → AuthorLabelSet(u64), shrinking it to a 64-bit bitset, and fixed all of AuthorFeatures as a 16-byte Copy struct.

  • This commit prepared a representation that shrinks it again to 8 bytes.

    #[allow(dead_code)]
    pub(crate) struct CompactAuthorFeatures(NonZeroU64);
    
    const COMPACT_PRESENT_BIT: NonZeroU64 = match NonZeroU64::new(1 << 63) {
        Some(bit) => bit,
        None => unreachable!(),
    };
    const COMPACT_LABEL_SHIFT: u32 = 8;
    const COMPACT_LABEL_MASK: u64 = (1 << (63 - COMPACT_LABEL_SHIFT)) - 1;
    
  • The bit layout is as follows:

    Bit position Content
    63 present marker (always 1)
    0–6 7 account status flags (is_suspended, is_deactivated, is_protected, is_nsfw_user, is_nsfw_admin, is_erased, is_offboarded)
    7 unused
    8–62 55-bit label bitset (11 bits actually used, 1:1 with the 11 AuthorLabel variants of Gen4+19)
  • The to_compact and from_compact round trip are defined together.

  • The tests exhaustively verify all 128 flag combinations and all 2048 label combinations. The two numbers are 1 << 7 and 1 << 11 respectively, and combined they give 262,144 combinations in total.

  • It also asserts that the size of Option<CompactAuthorFeatures> is exactly 8 bytes. The niche optimization of NonZeroU64 represents None as all-zero, so no separate tag is needed.

  • Since #[allow(dead_code)] is attached, CompactAuthorFeatures has no consumer in this commit. Candidates for where it could be connected include the following:

    1. The value type of clock_cache.rs (32-way × 64-shard CLOCK cache), introduced in Gen4+19. This cache requires V: Copy and is optimized for 8-byte values inferred.
    2. The hot-path cache of the Gizmoduck hydrator
    3. The wire representation of AuthorFeatures itself
  • We will watch which consumer it connects to in the next commit.

Why it matters This is read as a signal that the VF service is running two parallel projects to lower real latency. If the Gen4+19 HYDRATION_TIMEOUT increase (80/100 → 390 ms) expanded the latency budget, this axis works on cache hit rate and on shrinking hot-path data structures. Raising the timeout increases the partial-verdict success rate, and narrowing the label and feature representations increases cache density.

F. Two takedown branches carrying only a worldwide-scope code are no longer dropped automatically

VF's worldwide takedown rules became finer-grained. Bystander reports carrying only worldwide scope and legal-copyright codes were removed from the automatic-drop set. The refined file is visibility-filtering/rules/context.rs, and there are two refined pieces.

  • First, a legal request marked with a world-wide copyright code no longer triggers DropLegalTakendownPostRule.

    • legal_takedown_country now explicitly returns None for LegalRequest { country_code: "xy" }.

      fn legal_takedown_country(reason: &TakedownReason) -> Option<&str> {
          match reason {
              TakedownReason::LegalRequest { country_code }
                  if country_code.eq_ignore_ascii_case(WORLDWIDE_COPYRIGHT_COUNTRY_CODE) =>
              {
                  None
              }
              TakedownReason::LegalRequest { country_code }
              | TakedownReason::UnspecifiedReason { country_code } => Some(country_code),
              TakedownReason::Dmca => Some(WORLDWIDE_COPYRIGHT_COUNTRY_CODE),
              _ => None,
          }
      }
      
  • DMCA still returns "xy", so DMCA-based world-wide copyright takedowns are kept. In other words, the "xy" code now triggers a legal takedown only when it arrives as a DMCA reason, and is ignored when it arrives as a LegalRequest.

  • Second, a bystander report with only world-wide scope no longer triggers DropLocalLawsTakendownPostRule.

    • local_laws_takedown_country now returns None for BystanderReport { country_code: "xx" } or "xy".

      fn local_laws_takedown_country(reason: &TakedownReason) -> Option<&str> {
          match reason {
              TakedownReason::BystanderReport { country_code } if !is_worldwide_code(country_code) => {
                  Some(country_code)
              }
              _ => None,
          }
      }
      
  • Only bystander reports with an actually specified country code are consumed by the local laws rule.

  • Three new tests pin this rule into the golden corpus.

    • The test names are legal_takedown_copyright_code_allows_without_viewer_country, unspecified_takedown_worldwide_drops_without_viewer_country, and local_laws_takedown_worldwide_allows_any_viewer.

Why it matters This is the first substantive policy relaxation since Gen4+12 (6384ca7, 2026-09-01) introduced country-scope handling in the first place. UnspecifiedReason is still dropped even when worldwide, so the relaxation applies to only two branches.

  • LegalRequest{"xy"} was previously dropped and is now allowed. Since "xy" is reserved as a copyright code, even if a copyright-related request arrives through the legal channel, the actual basis for a verdict now holds only when a DMCA reason arrives separately.
  • BystanderReport{"xx"} or BystanderReport{"xy"} was previously dropped and is now allowed. A bystander report is caught by the rule only when the country code is explicit.

These two relaxations suggest there was a recognition that the previous rules were too broad for world-wide scope. This code alone does not show how often these conditions occurred in production or which creators were affected.

G. The Under-The-Hood report gained a monthly cap parameter and an employee role

Operators can now set a cap on the month shown by the Under-The-Hood (UTH, X's label transparency tool) report. The employee role was added to the roles that can view the report.

  • under-the-hood/strato/lib/params.strato exports a new int parameter.

    val underTheHoodReportMaxMonthBucket =
      fs.intParam("under_the_hood_report_max_month_bucket", 0)
    
  • "employee" is added to userRoles.

    val userRoles = Set(
      "birdwatch_team",
      "employee",
    )
    
  • The reportMonthBucket signature in underTheHoodReport.User.strato was extended to apply this cap. With the default 0, it returns the most recent completed calendar month as before; if a positive number is given, it clamps to that value as the cap.

    def reportMonthBucket(now: Time.T, maxMonthBucket: Int): Int = {
      val candidate = calendarMonthBefore(now)
      val calendarMonth =
        if (monthPublishCalendarReady(candidate, now)) { candidate }
        else { monthBucketBefore(candidate) }
      if (maxMonthBucket > 0 && calendarMonth > maxMonthBucket) {
        maxMonthBucket
      } else {
        calendarMonth
      }
    }
    

Why it matters This change altered two things:

  1. Operators now hold a safety valve. When the data pipeline has not finished backend preparation for a particular month (for example, an incident, a delay, or a rollout halt), the UTH report can be forcibly pinned to the last safe month.
  2. The "employee" role is added to who can view the report. This is read as meaning the report UI can now be verified and debugged internally.

This safeguard is paired with the UTH country-level account takedown batch job introduced in Gen4+19.

H. The Phoenix training pipeline now picks the Kafka authentication approach automatically and can connect with mTLS

The Phoenix training pipeline now chooses the authentication approach for connecting to Kafka based on the environment. It can now connect with mTLS as well as SASL.

  • A new dataclass that holds the authentication settings was added to phoenix/xrex/data/streaming/kafkaconsumer.py.

    @dataclasses.dataclass(frozen=True)
    class KafkaAuth:
        mode: str                       # "mtls" or "sasl"
        ca_path: str | None = None
        cert_path: str | None = None
        key_path: str | None = None
        sasl_mechanism: str | None = None
        sasl_username: str | None = None
        sasl_password: str | None = None
        verify_broker: bool = False
    
        @classmethod
        def mtls(cls, ca_path, cert_path, key_path, verify_broker=False): ...
        @classmethod
        def sasl(cls, mechanism, username, password): ...
    
        def ssl_context(self) -> ssl.SSLContext: ...
        def consumer_kwargs(self) -> dict[str, Any]: ...
    
  • auto_detect_auth in kafkaloader.py decides the authentication approach in the following order.

    1. If the KAFKA_AUTH env var is "sasl", SASL is forced. The mTLS certificate check is skipped in this case.
    2. If all three files exist in KAFKA_MTLS_CERTS_DIR (default /root/client-certs), mTLS is used. The three files are ca-bundle.crt, client.fullchain, and client.key.
    3. If only the client certificate and key exist and the CA bundle is missing, mTLS is used with a different truststore. This truststore is KAFKA_MTLS_PLATFORM_CA_BUNDLE (default /etc/ssl/internal-ca/ca-bundle.crt) or the system truststore. The system truststore is ssl.get_default_verify_paths().cafile.
    4. If SASL_PLAIN_PASSWORD is set, it falls back to SASL/SCRAM.
    5. If all three fail, it raises an error.
  • The previous code hardcoded ssl.create_default_context() and the SASL_SSL protocol separately in each of three consumer paths. This commit moved all three paths to the new auth.consumer_kwargs().

    • The three paths are discover_partition_count, _consumer_poll_loop, and _consume_multi_consumer.

Why it matters The recsys action stream Kafka that the Phoenix training pipeline consumes can now be reached with mTLS as well. This Kafka has connected with SASL/SCRAM since the v3 schema in Gen4+3. The cluster operating policy appears to be moving from password management infrastructure to certificate management infrastructure inferred. This code alone does not show which of the two has actually been adopted in production.

I. Other changes

The remaining four changes are maintenance with no policy change.

  • Cleaned up the hardcoded account list of the Brazil election filter. This is routine maintenance, and there is no policy change.
    • The filter is Brazil2026ElectionFilter, and the file is home-mixer/filters/brazil_2026_election_filter.rs (+334/−42).
    • The list grew from 2,554 to 2,707 entries, a net increase of 153 verified — test assert in brazil_2026_election_filter.rs. The patch adds 163 // @handle entries and removes 10.
    • The inserted accounts include @VladimirAlves, @merisio, @LorenLuniere, @capitaoassumcao, @annavalente, @ademarmago, @henriquecatan, @jabscres, @yarahaquim, @ElySilveira, @MarceloFaro30, @freire_roberto, @felipececilio, and @claudiojacoski.
    • Replaced the account @rafaelcampelo → @RobertoOshiro.
    • Added and deleted several "no live account found" comments.
  • Made small changes to two VF client files to match the xDS migration in C above. There is no substantive policy change.
    • The changed files are visibility-filtering-client/discovery.rs (+11/−8) and vf_client.rs (+1/−1).
  • In the home-mixer feature-switch defaults mirror file, only the sync time changed. No parameter values changed.
    • The mirror timestamp in home-mixer/params/param.rs was updated from 2026-09-10T16:21:03Z → 2026-09-11T16:22:45Z.
  • Made small cleanups to the Grox motion-reveal image builder. This follows up on the ECC and Farneback alignment pipeline introduced in Gen4+19, and there is no policy change.
    • The changed file is grox/libs/video_tools/image.py (+60/−55).

Canon update Created a new ### Gen4+20 (2026-09-12, 6bb4594) section in services/x.md and wrote up the nine axes A–I. Added an extension note on the seed_tweet_id of VMRanker DPP to the ### Diversity section. Added 6bb4594 as the 21st entry in the Gen4 commit list of the ## Sources section and corrected the total commit count from 19 → 21 (Gen4+19 was also added this time). Moved the last updated date from 2026-09-12 → 2026-09-13.

Why it matters This commit is the second iteration of the VF stabilization and reduction project that started in Gen4+19, and it is the first sign of the search relevance supervised labels (LLM_JUDGE_*) appearing in the proto. On the diversity axis, the introduction of the seed anchor is the substantive new technique. The earlier statement in these notes that "DPP optimizes only diversity within the set" has to be extended to "when a seed is given, diversity against the reference tweet is also computed," and this extension was folded into the ### Diversity note in services/x.md.

  • On the log stream axis, the migration from Thrift to proto has started.
  • On the infrastructure axis, the service discovery migration to xDS has started in VF.
  • On the policy axis, the world-wide scope takedown was relaxed in two places.

⚠ Claim changed: none. This diff overturns no existing canon statement. The statement in the diversity section was extended but not overturned. The behavior without a seed is unchanged.

Corrections

Nothing was overturned.

Radar

None of the 30 arXiv cs.IR papers triaged in this batch passed the acceptance criteria. The criteria are platform-company authors, production A/B, a new mechanism, and large-scale results. There is also no new material on the Meta engineering blog (engineering.fb.com, ai.meta.com/blog) since 2026-09-12. The same holds for the Netflix TechBlog (netflixtechblog.com).

What to check next