- Sources read: 1
- Canon files updated: 1
- Radar entries: 0
1Diversity re-ranking can now push out candidates similar to a seed post — X
Source xai-org/x-algorithm@6bb4594 — public source code, 2026-09-12, 24 files, +1397/−338. These notes number X commits after the Gen4 release (2026-08-13) in sequence, and this commit is Gen4+20.
Background Since the Gen4 release on 2026-08-13, X has mirrored the For You pipeline source code to xai-org/x-algorithm once or twice a day. The preceding Gen4+19 (fee1d0f, 2026-09-10) unified the hydration timeout. This commit (2026-09-12, 6bb4594) is a mid-sized diff that continues that line and touches all five axes accumulated in the canon so far: ranking reordering, logging and training pipeline, infrastructure migration, wire extension of the ranking model Phoenix, and cleanup and policy in the impression-verdict service VF. The five axes are independent, but landing together in one commit is a signal that they are roughly contemporaneous projects.
A. If a request supplies a seed post, candidates similar to it are more likely to be dropped from the DPP result
For You re-orders the slate (the bundle of candidates sent out at once) with DPP just before final batching. DPP (determinantal point process) is a method that looks at the whole candidate set so that similar items are not picked together. Until now this stage computed only cosine repulsion between candidates within the candidate set. Now, when a request carries a seed post in seed_tweet_id, it also computes the similarity between the other candidates and the seed. The seed itself is filtered from the final result, so the top_k budget is unchanged. The diversity axis can now extend to comparison against a reference tweet, not just within the set.
The stage is handled by VMRanker DPP (θ=0.65, K=150).
The embedding of the tweet that
seed_tweet_idpoints to enters the kernel as pinned.The
rescoresignature invm-ranker/dpp.rswas extended to take a newseedargument.pub fn rescore( inputs: &[DppInput], seed: Option<&DppInput>, config: &DppConfig, _viewer_id: u64, ) -> Vec<DppResult> { ... }The new path works in the following four steps:
- Grow the kernel by one for the seed. Add one seed to the existing top-K candidates (
m = min(inputs.len(), max_selected_rank)) to build a kernel of sizetotal = m + 1. The seed has no score, so itsqvalue is arbitrarily fixed at1.0. - Pin the seed to the first slot of greedy DPP.
greedy_dpppreviously picked the index with the largest kernel diagonal as the starting point. Now, if the new argumentpinned_first: Option<usize>is given, that index is forced as the first selection, and this index is the seed's position. - Increase the budget. If pinned is present, raise it to
max_items = (k + 1).min(n)so the seed does not consume top_k. - Filter the seed from the result. Before the final return, the seed index is excluded with
dpp_result.selected.iter().filter(|&(ki, _)| ki < m).
- Grow the kernel by one for the seed. Add one seed to the existing top-K candidates (
The net effect of these four steps is two contracts:
- Candidates close to the seed's embedding get larger off-diagonal kernel entries, so the determinantal gain shrinks and they are more likely to be excluded from the result.
- The seed itself does not appear in the result and does not use up top_k.
Two new tests verify these contracts.
#[test] fn seed_suppresses_near_duplicate() { let seed = make_input(999, 0.0, &[1.0, 0.0]); let inputs = vec![ make_input(1, 10.0, &[1.0, 0.0]), // exactly the same direction as the seed make_input(2, 5.0, &[0.0, 1.0]), // orthogonal ]; // Without a seed, rescore puts both 1 and 2 in ids // After rescoring with the seed, 1 is excluded, only 2 survives, and 999 is not in the result either } #[test] fn seed_does_not_consume_top_k_budget() { let seed = make_input(999, 0.0, &[0.0, 0.0, 1.0]); let inputs = vec![ make_input(1, 10.0, &[1.0, 0.0, 0.0]), make_input(2, 5.0, &[0.0, 1.0, 0.0]), ]; let config = DppConfig { top_k: 2, ... }; let result = rescore(&inputs, Some(&seed), &config, 0); assert_eq!(result.len(), 2, "seed must not eat top_k budget"); }If the seed's embedding is not in the serving-time store, it flows through the existing path without pinning.
The place that fills the new parameter is
rankinvm-ranker/scoring/dpp_model.rs.fn build_seed_input(req: &RankRequest, ctx: &DppContext) -> Option<DppInput> { if req.seed_tweet_id == 0 { return None; } match ctx.store.client.get(req.seed_tweet_id) { Some(embedding) => { DPP_SEED_CONTEXT.with_label_values(&["pinned"]).inc(); let norm = l2_norm(&embedding); Some(DppInput { id: req.seed_tweet_id, score: 0.0, embedding, norm, embedding_missing: false, }) } None => { DPP_SEED_CONTEXT.with_label_values(&["embedding_missing"]).inc(); None } } }The counter
vm_ranker_dpp_seed_context_total{result="pinned"|"embedding_missing"}separates hits and misses for observation.
If the embedding norm is near 0 and the kernel diagonal is at or below EPSILON, it gives up the pin and returns to the existing greedy path.
pinned = pinned_first.filter(|&p| kernel[p * n + p] > EPSILON);—vm-ranker/dpp.rs
Where this path will be used cannot be confirmed from this commit alone. The home-mixer (the service that assembles the For You feed) pipeline that would carry seed_tweet_id is not in this diff. Natural consumers might include the following inferred:
- Related tweets on a permalink page: a slate "dissimilar to this tweet"
- An alternative collection in a "more like this" flow for a specific tweet
- A scenario in thread content where the parent tweet is the seed and similar replies are suppressed
Whether the For You home timeline itself fills seed_tweet_id is something to watch. In theory For You generates a slate without a seed, but an extension that passes a recently viewed tweet as the seed to reduce repeated exposure is logically possible.
B. Phoenix candidate score logs go out on both the existing Thrift stream and a new proto stream
home-mixer gained one more channel for sending Phoenix candidate scores to Kafka. The existing channel uses Thrift format, and the new channel publishes in proto format to a different topic in parallel. When downstream moves to proto is outside this diff.
Until now there was one channel, which sent Thrift
ScoredCandidatetoPHOENIX_SCORES_TOPIC(clusterAiml). This payload carries 28 engagement head scores and candidate metadata.This commit added a second channel that publishes proto
LoggedScoredCandidatein parallel toLOGGED_SCORED_CANDIDATES_TOPIC(clusterPhoenix).Now, for each candidate, two payloads are built and written to two Kafka publishers at the same time.
- The file is
home-mixer/side_effects/phoenix_experiments_side_effect.rs, andtokio::join!is used for the concurrent writes.
- The file is
The first payload is the existing Thrift
ScoredCandidate. This stream has been maintained since the v3 schema in Gen4+3 (11a71f8, 2026-08-18).- The field is a sorted collection, as in
prediction_scores: Option<BTreeSet<Box<PredictionScore>>>. - The Kafka topic is
PHOENIX_SCORES_TOPICand the cluster isAiml.
- The field is a sorted collection, as in
The second payload is the proto
LoggedScoredCandidatethat is new in this commit.// (the proto definition is in xai_proto::recsys_logging) message LoggedScoredCandidate { int64 post_id = ...; int64 viewer_id = ...; int64 author_id = ...; int64 request_join_id = ...; int64 request_time_ms = ...; int64 source_post_id = ...; bool is_in_network = ...; int64 in_reply_to_post_id = ...; int64 quoted_post_id = ...; int64 quoted_user_id = ...; map<string, double> prediction_scores = ...; int64 fav_count = ...; int64 reply_count = ...; int64 retweet_count = ...; int64 quote_count = ...; bool has_media = ...; int32 language_code = ...; int32 video_duration_ms = ...; }The Kafka topic name of the new payload is the new constant
LOGGED_SCORED_CANDIDATES_TOPIC. The cluster changes to the newKafkaCluster::Phoenix, separate from the existingAiml.The production assembly code of
PhoenixCandidatePipelineassigns the sixth Kafka publisher slot to this topic.async { Arc::new( ProdKafkaPublisherClient::new( LOGGED_SCORED_CANDIDATES_TOPIC, KafkaCluster::Phoenix, ).await, ) as Arc<dyn KafkaPublisherClient> },
The two payloads are re-sorted into different data structures. The pipeline builds a
HashMap<String, f64>internally and sorts and wraps it into aBTreeSetonly for the Thrift payload. For the proto payload, the HashMap is carried as is as a map.let mut prediction_scores = HashMap::new(); for (cluster_id, result) in &experiment_results { ... insert_cluster_scores(&mut prediction_scores, &cluster_name, &scores); } let thrift_scores: BTreeSet<Box<PredictionScore>> = prediction_scores .iter() .map(|(name, value)| { Box::new(PredictionScore::new(...)) }) .collect();The key naming convention (
phoenix.<cluster_name>.<action_name>) and the score field set are kept as they are. The field set is the 28 engagement heads and thepdwellalias.Tests were also added for the new helpers
insert_cluster_scoresandbuild_logged_scored_candidate, respectively.
Why it matters The map representation of prediction_scores gains on two axes. On wire size, the sort index disappears; on consumer-side lookup, a map gives O(1) lookup instead of a BTreeSet. The Thrift stream is kept because downstream consumers cannot switch immediately inferred. How long the transition period of running both streams in parallel will last is not in this commit.
C. VF moved how it finds TES and Gizmoduck servers to go through xDS
The way the visibility-filtering (VF) service connects to TES and Gizmoduck changed. TES (Tweet Entity Service) is X's core tweet lookup service, and Gizmoduck is the user profile service. The connection path was migrated from datacenter-templated hostnames to an xDS LDS listener. The service discovery layer moved to Envoy-family xDS.
Previously, the bundle of parameters needed to assemble the client was isolated in a struct defined inside the service crate. The service name was built by filling a hostname template with a datacenter string.
The file is
visibility-filtering/server_deps.rs.The struct is
xai_core_entities::tweet_entity_service_client::TESClientConfig, and the value that fills the hostname template isdatacenter.// Before (~fee1d0f) let tes_client = Arc::new( init_client_with_retry("tes", init_deadline, || { ProdTESClient::new_with_config( None, datacenter, None, tes_client_config(deterministic_aperture), ) }).await ); // tes_client_config returns a TESClientConfig struct: // aperture_size, deterministic_aperture, lb_policy, readiness_probe_port, // and fills fields such as request_timeout_ms and enable_serve_within.
This commit assembles the TES client as follows:
// New (6bb4594) let tes_client = Arc::new( init_client_with_retry("tes", init_deadline, || async move { let strato = build_xds_strato( XdsStratoParams { name: "tes-xds", xds_listener: "tweet-entity-service.prod.tweet-entity-service:fed-grpc", tls_domain: format!( "tweet-entity-service.tweet-entity-service.prod.{datacenter}.s2s.twttr.net" ), lb_policy: LbPolicy::penalized_peak_ewma(), client_id: S2S_CLIENT_ID.clone(), retry_config: Some(RetryConfig::for_idempotent()), max_batch_size: TESRpcConstants::max_batch_size(), enable_serve_within: true, }, deterministic_aperture, ).await?; anyhow::Ok(ProdTESClient { grpc_client: Arc::new(strato) }) }).await );The new helper
build_xds_stratoassembles the gRPC channel withxai_x_rpc::grpc_client::ChannelBuilder.async fn build_xds_strato( params: XdsStratoParams, deterministic_aperture: bool, ) -> anyhow::Result<StratoGrpc> { let mut builder = ChannelBuilder::new(params.name) .tls(TlsMode::mtls_from_env()?.with_domain_override(params.tls_domain)) .request_timeout(STRATO_REQUEST_TIMEOUT) // = HYDRATION_TIMEOUT (390 ms) .connect_timeout(STRATO_CONNECT_TIMEOUT) // 400 ms .xds(StartFrom::Lds(params.xds_listener.to_string())) .await? .eager_resolution(XDS_EAGER_RESOLUTION_TIMEOUT) // 5 s .aperture(STRATO_APERTURE_SIZE) // 12 .readiness_probe(READINESS_PROBE_PORT, "/ready") .buffer_max_wait(DEFAULT_BUFFER_MAX_WAIT) .total_timeout(DEFAULT_TOTAL_TIMEOUT); if deterministic_aperture { builder = builder.deterministic(); } let channel = builder.build_load_balanced(params.lb_policy).await?; let mut strato = StratoGrpc::from_load_balanced_channel( channel, None, Some(params.client_id), params.retry_config, params.max_batch_size, ); if params.enable_serve_within { strato = strato.with_op_context(xai_strato::strato_proto::OpContext { serve_within: Some(xai_strato::strato_proto::ServeWithin { duration_microseconds: STRATO_REQUEST_TIMEOUT.as_micros() as i64, round_trip_allowance_microseconds: 10_000, }), ..Default::default() }); } Ok(strato) }The key is the single line
.xds(StartFrom::Lds(params.xds_listener)). The client talks to the xDS control plane, starting from a listener name (such as"tweet-entity-service.prod.tweet-entity-service:fed-grpc"), and receives the endpoint list.- xDS is the x Discovery Service protocol, and the starting point of this code is LDS (Listener Discovery Service) among them.
- This is the standard way in Envoy-family service meshes for a client to dynamically receive the target endpoint list from the control plane.
Two options attach only to the TES path:
enable_serve_within: trueattaches a deadline hint on the server side. The hint tells the server how long a request may wait in server queueing.duration_microsecondsis the same value as the client timeout (HYDRATION_TIMEOUT = 390 ms), andround_trip_allowance_microseconds = 10_000is a 10 ms round-trip allowance.retry_config: Some(RetryConfig::for_idempotent())automatically retries idempotent requests.
On the Gizmoduck path, lb_policy is
least_request()and both options above are off verified —server_deps.rs.This migration changed four things:
- The service discovery layer moved from hostname templates to going through the xDS control plane.
- The separate client-config structs the service crate used to export (
TESClientConfig,GizmoduckClientConfig) are gone, and the client assembly logic was absorbed into the VF service's ownserver_deps.rs. - The Strato (X's internal system that code goes through to read or store data) client of the L2 cache warmer now also takes the
deterministic_apertureflag. - Because
TES_READINESS_PROBE_PORTandGIZMODUCK_READINESS_PROBE_PORTwere both 8081, they were merged into the unified constantREADINESS_PROBE_PORT: u16 = 8081.
It can also be read as the Gen4+19 HYDRATION_TIMEOUT unification having propagated this far. This time, the timeout constant in the client assembly code is referenced as STRATO_REQUEST_TIMEOUT: Duration = crate::hydration::HYDRATION_TIMEOUT.
D. Phoenix proto gained LLM-judged search relevance actions and a new logit response format
The Phoenix proto gained two new actions that represent search relevance judged by an LLM. A response format that returns the logits of the requested actions as a positional dense array instead of a sparse HashMap was also added. The new actions are read as a signal that the xrecsys_search search ranking config, which appeared in Gen4+4 (aad7179, 2026-08-19), is starting to prepare to supervise search relevance prediction with LLM judgments as well.
The new actions are the pair
LLM_JUDGE_RELEVANT_TO_SEARCH/_NOT_RELEVANT_TO_SEARCH, and the new response format isreturnLogitsList.- The numbers are
LLM_JUDGE_RELEVANT_TO_SEARCH = 107andLLM_JUDGE_NOT_RELEVANT_TO_SEARCH = 108.
- The numbers are
Two fields and two enum values were added to both recsys.proto files (Rust and the Python mirror).
The two new actions went into the action enum.
enum ActionName { ... CLIENT_TWEET_SEARCH_QUERY_REFORMULATED = 104; CLIENT_TWEET_RELEVANT_TO_SEARCH = 105; CLIENT_TWEET_NOT_RELEVANT_TO_SEARCH = 106; LLM_JUDGE_RELEVANT_TO_SEARCH = 107; LLM_JUDGE_NOT_RELEVANT_TO_SEARCH = 108; ADS_WEB_CONVERSION = 128; ... }The search relevance axis had only client-side signals until now. These are signals showing whether the user actually opened a result and whether they rewrote the query. In this commit, the LLM judge's verdict was incorporated as a separate action axis.
xrecsys_search, which appeared in Gen4+4 (aad7179, 2026-08-19), is a config with a separate vocab and search-specific settings such ascondition_search_relevance_on_prompt=True; this incorporation is read as preparation to put LLM judgments in as a supervision signal on the action axis that config uses as a training target inferred.A pipeline in which Grox (a Grok-based nearline classifier sub-repo) or a dedicated LLM judge feeds relevant/not labels to (query, tweet) pairs must exist somewhere. However, the code for that pipeline is not in this diff.
The proto definition of the new response format
returnLogitsListis below, and the Python mirror file is identical.message PredictNextActionsRequest { ... bool returnBackboneScores = 20; map<string, string> experiment_overrides = 21; bool returnLogitsList = 22; } message NextActionDistribution { ... repeated float backboneTopLogProbs = 10; repeated float backboneContinuousValues = 11; repeated float requestedActionLogits = 12; }On the Rust engine side, a
return_logits_list: boolfield was added toPredictRequestItem. InsidePredictRequestBatch::forward, when this flag is on, the response takes the following path:if item.return_logits_list { dist_entry.requested_action_logits = item .requested_action_indices .iter() .map(|&idx| { dist.get(idx as usize).copied().unwrap_or(f32::NEG_INFINITY) }) .collect(); } else if item.return_log_map { for idx in item.requested_action_indices.iter() { if let Some(v) = dist.get(*idx as usize) { dist_entry.index_to_logits.insert(*idx, *v); } } }The two formats have mutually exclusive priority. If both are on,
return_logits_listwins.Continuous action index handling is now shared by both formats. Previously it was only inside the
return_log_mappath.
Why it matters The new format has guaranteed order and stores no indices, so the wire size is smaller. The existing return_log_map is a sparse map that carries HashMap<u32, f32> (action_index → logit) on the wire. This representation is good when the data is sparse, but when the requested action indices are a densely fixed set, the map key overhead is pure loss. returnLogitsList is a positional dense list that returns a float array in requested index order, and it fills with -inf if a requested index falls outside the model vocab. It is likely optimized for consumers whose action indices are already known, such as cascades or real-time rankers inferred.
E. A representation that packs author features into 8 bytes is ready but nothing uses it yet
VF prepared a new representation that packs author features into 8 bytes, but no code in this commit uses it. AuthorFeatures was shrunk to 16 bytes in Gen4+19, and the new representation CompactAuthorFeatures packs it into a single 8-byte NonZeroU64.
Gen4+19 changed the label representation
AuthorLabelSet(HashSet<LabelValue>) → AuthorLabelSet(u64), shrinking it to a 64-bit bitset, and fixed all ofAuthorFeaturesas a 16-byte Copy struct.This commit prepared a representation that shrinks it again to 8 bytes.
#[allow(dead_code)] pub(crate) struct CompactAuthorFeatures(NonZeroU64); const COMPACT_PRESENT_BIT: NonZeroU64 = match NonZeroU64::new(1 << 63) { Some(bit) => bit, None => unreachable!(), }; const COMPACT_LABEL_SHIFT: u32 = 8; const COMPACT_LABEL_MASK: u64 = (1 << (63 - COMPACT_LABEL_SHIFT)) - 1;The bit layout is as follows:
Bit position Content 63 present marker (always 1) 0–6 7 account status flags ( is_suspended,is_deactivated,is_protected,is_nsfw_user,is_nsfw_admin,is_erased,is_offboarded)7 unused 8–62 55-bit label bitset (11 bits actually used, 1:1 with the 11 AuthorLabelvariants of Gen4+19)The
to_compactandfrom_compactround trip are defined together.The tests exhaustively verify all 128 flag combinations and all 2048 label combinations. The two numbers are
1 << 7and1 << 11respectively, and combined they give 262,144 combinations in total.It also asserts that the size of
Option<CompactAuthorFeatures>is exactly 8 bytes. The niche optimization of NonZeroU64 representsNoneas all-zero, so no separate tag is needed.Since
#[allow(dead_code)]is attached,CompactAuthorFeatureshas no consumer in this commit. Candidates for where it could be connected include the following:- The value type of
clock_cache.rs(32-way × 64-shard CLOCK cache), introduced in Gen4+19. This cache requiresV: Copyand is optimized for 8-byte values inferred. - The hot-path cache of the
Gizmoduckhydrator - The wire representation of
AuthorFeaturesitself
- The value type of
We will watch which consumer it connects to in the next commit.
Why it matters This is read as a signal that the VF service is running two parallel projects to lower real latency. If the Gen4+19 HYDRATION_TIMEOUT increase (80/100 → 390 ms) expanded the latency budget, this axis works on cache hit rate and on shrinking hot-path data structures. Raising the timeout increases the partial-verdict success rate, and narrowing the label and feature representations increases cache density.
F. Two takedown branches carrying only a worldwide-scope code are no longer dropped automatically
VF's worldwide takedown rules became finer-grained. Bystander reports carrying only worldwide scope and legal-copyright codes were removed from the automatic-drop set. The refined file is visibility-filtering/rules/context.rs, and there are two refined pieces.
First, a legal request marked with a world-wide copyright code no longer triggers
DropLegalTakendownPostRule.legal_takedown_countrynow explicitly returnsNoneforLegalRequest { country_code: "xy" }.fn legal_takedown_country(reason: &TakedownReason) -> Option<&str> { match reason { TakedownReason::LegalRequest { country_code } if country_code.eq_ignore_ascii_case(WORLDWIDE_COPYRIGHT_COUNTRY_CODE) => { None } TakedownReason::LegalRequest { country_code } | TakedownReason::UnspecifiedReason { country_code } => Some(country_code), TakedownReason::Dmca => Some(WORLDWIDE_COPYRIGHT_COUNTRY_CODE), _ => None, } }
DMCA still returns
"xy", so DMCA-based world-wide copyright takedowns are kept. In other words, the"xy"code now triggers a legal takedown only when it arrives as a DMCA reason, and is ignored when it arrives as aLegalRequest.Second, a bystander report with only world-wide scope no longer triggers
DropLocalLawsTakendownPostRule.local_laws_takedown_countrynow returnsNoneforBystanderReport { country_code: "xx" }or"xy".fn local_laws_takedown_country(reason: &TakedownReason) -> Option<&str> { match reason { TakedownReason::BystanderReport { country_code } if !is_worldwide_code(country_code) => { Some(country_code) } _ => None, } }
Only bystander reports with an actually specified country code are consumed by the local laws rule.
Three new tests pin this rule into the golden corpus.
- The test names are
legal_takedown_copyright_code_allows_without_viewer_country,unspecified_takedown_worldwide_drops_without_viewer_country, andlocal_laws_takedown_worldwide_allows_any_viewer.
- The test names are
Why it matters This is the first substantive policy relaxation since Gen4+12 (6384ca7, 2026-09-01) introduced country-scope handling in the first place. UnspecifiedReason is still dropped even when worldwide, so the relaxation applies to only two branches.
LegalRequest{"xy"}was previously dropped and is now allowed. Since"xy"is reserved as a copyright code, even if a copyright-related request arrives through the legal channel, the actual basis for a verdict now holds only when a DMCA reason arrives separately.BystanderReport{"xx"}orBystanderReport{"xy"}was previously dropped and is now allowed. A bystander report is caught by the rule only when the country code is explicit.
These two relaxations suggest there was a recognition that the previous rules were too broad for world-wide scope. This code alone does not show how often these conditions occurred in production or which creators were affected.
G. The Under-The-Hood report gained a monthly cap parameter and an employee role
Operators can now set a cap on the month shown by the Under-The-Hood (UTH, X's label transparency tool) report. The employee role was added to the roles that can view the report.
under-the-hood/strato/lib/params.stratoexports a new int parameter.val underTheHoodReportMaxMonthBucket = fs.intParam("under_the_hood_report_max_month_bucket", 0)"employee"is added touserRoles.val userRoles = Set( "birdwatch_team", "employee", )The
reportMonthBucketsignature inunderTheHoodReport.User.stratowas extended to apply this cap. With the default 0, it returns the most recent completed calendar month as before; if a positive number is given, it clamps to that value as the cap.def reportMonthBucket(now: Time.T, maxMonthBucket: Int): Int = { val candidate = calendarMonthBefore(now) val calendarMonth = if (monthPublishCalendarReady(candidate, now)) { candidate } else { monthBucketBefore(candidate) } if (maxMonthBucket > 0 && calendarMonth > maxMonthBucket) { maxMonthBucket } else { calendarMonth } }
Why it matters This change altered two things:
- Operators now hold a safety valve. When the data pipeline has not finished backend preparation for a particular month (for example, an incident, a delay, or a rollout halt), the UTH report can be forcibly pinned to the last safe month.
- The
"employee"role is added to who can view the report. This is read as meaning the report UI can now be verified and debugged internally.
This safeguard is paired with the UTH country-level account takedown batch job introduced in Gen4+19.
H. The Phoenix training pipeline now picks the Kafka authentication approach automatically and can connect with mTLS
The Phoenix training pipeline now chooses the authentication approach for connecting to Kafka based on the environment. It can now connect with mTLS as well as SASL.
A new dataclass that holds the authentication settings was added to
phoenix/xrex/data/streaming/kafkaconsumer.py.@dataclasses.dataclass(frozen=True) class KafkaAuth: mode: str # "mtls" or "sasl" ca_path: str | None = None cert_path: str | None = None key_path: str | None = None sasl_mechanism: str | None = None sasl_username: str | None = None sasl_password: str | None = None verify_broker: bool = False @classmethod def mtls(cls, ca_path, cert_path, key_path, verify_broker=False): ... @classmethod def sasl(cls, mechanism, username, password): ... def ssl_context(self) -> ssl.SSLContext: ... def consumer_kwargs(self) -> dict[str, Any]: ...auto_detect_authinkafkaloader.pydecides the authentication approach in the following order.- If the
KAFKA_AUTHenv var is"sasl", SASL is forced. The mTLS certificate check is skipped in this case. - If all three files exist in
KAFKA_MTLS_CERTS_DIR(default/root/client-certs), mTLS is used. The three files areca-bundle.crt,client.fullchain, andclient.key. - If only the client certificate and key exist and the CA bundle is missing, mTLS is used with a different truststore. This truststore is
KAFKA_MTLS_PLATFORM_CA_BUNDLE(default/etc/ssl/internal-ca/ca-bundle.crt) or the system truststore. The system truststore isssl.get_default_verify_paths().cafile. - If
SASL_PLAIN_PASSWORDis set, it falls back to SASL/SCRAM. - If all three fail, it raises an error.
- If the
The previous code hardcoded
ssl.create_default_context()and theSASL_SSLprotocol separately in each of three consumer paths. This commit moved all three paths to the newauth.consumer_kwargs().- The three paths are
discover_partition_count,_consumer_poll_loop, and_consume_multi_consumer.
- The three paths are
Why it matters The recsys action stream Kafka that the Phoenix training pipeline consumes can now be reached with mTLS as well. This Kafka has connected with SASL/SCRAM since the v3 schema in Gen4+3. The cluster operating policy appears to be moving from password management infrastructure to certificate management infrastructure inferred. This code alone does not show which of the two has actually been adopted in production.
I. Other changes
The remaining four changes are maintenance with no policy change.
- Cleaned up the hardcoded account list of the Brazil election filter. This is routine maintenance, and there is no policy change.
- The filter is
Brazil2026ElectionFilter, and the file ishome-mixer/filters/brazil_2026_election_filter.rs(+334/−42). - The list grew from 2,554 to 2,707 entries, a net increase of 153 verified — test assert in
brazil_2026_election_filter.rs. The patch adds 163// @handleentries and removes 10. - The inserted accounts include
@VladimirAlves,@merisio,@LorenLuniere,@capitaoassumcao,@annavalente,@ademarmago,@henriquecatan,@jabscres,@yarahaquim,@ElySilveira,@MarceloFaro30,@freire_roberto,@felipececilio, and@claudiojacoski. - Replaced the account
@rafaelcampelo → @RobertoOshiro. - Added and deleted several "no live account found" comments.
- The filter is
- Made small changes to two VF client files to match the xDS migration in C above. There is no substantive policy change.
- The changed files are
visibility-filtering-client/discovery.rs(+11/−8) andvf_client.rs(+1/−1).
- The changed files are
- In the home-mixer feature-switch defaults mirror file, only the sync time changed. No parameter values changed.
- The mirror timestamp in
home-mixer/params/param.rswas updated from2026-09-10T16:21:03Z → 2026-09-11T16:22:45Z.
- The mirror timestamp in
- Made small cleanups to the Grox motion-reveal image builder. This follows up on the ECC and Farneback alignment pipeline introduced in Gen4+19, and there is no policy change.
- The changed file is
grox/libs/video_tools/image.py(+60/−55).
- The changed file is
Canon update Created a new ### Gen4+20 (2026-09-12, 6bb4594) section in services/x.md and wrote up the nine axes A–I. Added an extension note on the seed_tweet_id of VMRanker DPP to the ### Diversity section. Added 6bb4594 as the 21st entry in the Gen4 commit list of the ## Sources section and corrected the total commit count from 19 → 21 (Gen4+19 was also added this time). Moved the last updated date from 2026-09-12 → 2026-09-13.
Why it matters This commit is the second iteration of the VF stabilization and reduction project that started in Gen4+19, and it is the first sign of the search relevance supervised labels (LLM_JUDGE_*) appearing in the proto. On the diversity axis, the introduction of the seed anchor is the substantive new technique. The earlier statement in these notes that "DPP optimizes only diversity within the set" has to be extended to "when a seed is given, diversity against the reference tweet is also computed," and this extension was folded into the ### Diversity note in services/x.md.
- On the log stream axis, the migration from Thrift to proto has started.
- On the infrastructure axis, the service discovery migration to xDS has started in VF.
- On the policy axis, the world-wide scope takedown was relaxed in two places.
⚠ Claim changed: none. This diff overturns no existing canon statement. The statement in the diversity section was extended but not overturned. The behavior without a seed is unchanged.
Corrections
Nothing was overturned.
Radar
None of the 30 arXiv cs.IR papers triaged in this batch passed the acceptance criteria. The criteria are platform-company authors, production A/B, a new mechanism, and large-scale results. There is also no new material on the Meta engineering blog (engineering.fb.com, ai.meta.com/blog) since 2026-09-12. The same holds for the Netflix TechBlog (netflixtechblog.com).
What to check next
- Find the home-mixer pipeline that carries the reference post — observe in the next commit which candidate pipeline in home-mixer fills
RankRequest.seed_tweet_id. The key question is whether For You itself extends this by carrying recently viewed tweets as seeds to reduce repeated impressions, or whether it attaches only to permalink and related paths. - Check the downstream consumers of
LOGGED_SCORED_CANDIDATES_TOPIC— observe whether the Phoenix training pipeline starts consuming this new proto topic, or whether a separate ranking-log consumer attaches. Examples of a separate consumer are a real-time offline evaluation pipeline and feature store snapshot generation. - Find the pipeline that produces the
LLM_JUDGE_RELEVANT_TO_SEARCH/_NOTlabels — check whether a flow appears in the grox sub-repo that feeds relevant/not labels to (query, tweet) pairs. Also check whether thexrecsys_searchconfig adds these two actions as training targets. - Check the actual consumer of
CompactAuthorFeatures— the key question is which of theclock_cachein Gen4+19, the wire representation ofAuthorFeatures, and the Gizmoduck hydrator hot-path becomes the first consumer. - Check the actual consumer of
return_logits_list— see when home-mixer or another ranker moves to this new response format, and when the existingreturn_log_mappath is marked deprecated. - Find the actual registration definition of the xDS LDS listener — see in which xDS control plane repository a listener name such as
tweet-entity-service.prod.tweet-entity-service:fed-grpcis registered. Also consider the possibility that the service mesh source is released together in this repo. - Watch for when the UTH report monthly cap is actually used — observe when this parameter appears with a non-zero value in the feature-switch defaults mirror (
param.rs).
Lookup failed: none.