2026-09-24

Other changes reviewed today:

• In the same commit, X's impression verdict service added 4 reply-restriction rules and a reverse-follow lookup, completing the reply-restriction slot reserved yesterday.
• The same commit also added a pipeline that picks key frames for video safety verdicts based on scene cuts.
• Spotify's paper on training an LLM reranker passed the radar criteria, together with online A/B results.

1The VM-Ranker service took over value model computation — X

• First, the home-mixer VMRanker scorer moved beyond a score-only DPP wrapper and gained two parameters that send candidates, user context, and 26 PhoenixScores fields. However, there was still no code for what the receiving side would compute or how.
  • The two parameters are VMRankerSendValueModelInputs and VMRankerComputeValueModel.
• Second, two new slots were reserved in the verdict tree of VF (visibility-filtering, a service that decides for each post whether it may be shown to this viewer). However, no rule filled these slots (#[expect(dead_code)]).
  • The two slots are TombstoneReason::LocalRegulations and LimitedEngagementReason::ConversationControl.
• Third, grox (the Grok-based nearline classifier sub-repo) did not yet have the frame sampling utility needed for adult content verdicts.

This commit (2026-09-23, 1b3fec20, Gen4+27) fills these three slots with real code.

A. The VM-Ranker service started computing the value model directly (4 new files, 56 parameters)

• VM-Ranker's value model computation is pure Rust code that does not call a separate model server verified.
  • The computation is handled by vm-ranker/scoring/value_model.rs (191 new lines) in the vm-ranker service. This file calls the shared crate's xai_value_model::compute_value_scores() inline.
  • There are three inputs.
    • ValueModelWeights: 38 action-weight parameters
    • QueryScoringContext: OON (out-of-network, posts from accounts the user does not follow) multipliers per user cohort
    • CandidateScoringInputs: per-candidate Phoenix engagement probabilities
  • The output is two vectors. scores is for the final ordering, and weighted is the weighted sum before DPP adjustment.
• The weights were released together with production values verified — vm-ranker/params.rs.
  • It is a new file with 56 parameters in 211 lines.
  • The key gating flag ComputeValueModel is false by default. This means it is still at the experimental stage.
  • The weights of the positive signals are FavoriteWeight=0.5, ReplyWeight=5.0, RetweetWeight=1.0, QuoteWeight=5.0, ShareWeight=2.0, FollowAuthorWeight=4.0, ClickWeight=0.4, VideoOpenWeight=0.07, and PhotoExpandWeight=0.05.
  • The weights of the negative signals are NotInterestedWeight=-43.2, BlockAuthorWeight=-31.2, MuteAuthorWeight=-58.8, and ReportWeight=-234.0.
  • The DPP settings are DppEnabled=true, DppTheta=0.65, and DppMaxSelectedRank=150.
  • The OON multipliers are OonWeightFactor=0.75, TopicOonWeightFactor=0.5, and NewUserOonWeightFactor=0.00001.
• The author-diversity parameters are also exposed in VM-Ranker verified. The VM-Ranker process reads these values on its own, separately from the home-mixer param mirror.
  • The parameters are EnableAuthorDiversity=true, AuthorDiversityDecay=0.5, and AuthorDiversityFloor=0.25.
  • An example of a value on the home-mixer param mirror side is AuthorSoftmaxOutputDecayFactor=0.5.
• Parameters are updated without a process restart by polling the git remote (hot-reload) verified — vm-ranker/config_sync.rs.
  • The ConfigSync task in the new file (235 lines) polls a sparse-cloned git remote every 30 seconds with git fetch → reset. The interval setting is config_sync_interval_secs, and its default is 30.
  • The files updated by polling are features/home-mixer/main/rust_home_mixer.yml and abdecider/abdecider.yml.
  • FeatureSwitches reads the new values from the next match_recipient() call.
  • Values do not change within a request in progress (immutable).
• An explicit fallback logic was added verified. If any one of the three conditions below applies, it returns the upstream score (the weighted sum passed by home-mixer) unchanged and emits a throttled warning log.
  • The three conditions are unresolved params, missing viewer context, and ComputeValueModel=false.
  • This logic is compute_or_fallback (around vm-ranker/scoring/value_model.rs:108).
  • The checkpoints to watch are three metrics.
    • vm_ranker_value_model_requests_total{mode=passthrough|value_model|fallback}
    • vm_ranker_value_model_fallback_total{reason=...}
    • vm_ranker_value_model_stage_total{stage=heads|cached_weighted}
• A new path skips recomputation for cached candidates verified.
  • If the new home-mixer param CachedPostsReuseWeightedScore (default false) is true, the cached candidate's weighted_score is reused as is and head recomputation is skipped.
  • The gating function RankingScorer::reuses_cached_weighted_score() and the helper unoffset_score(), which inverts the offset transform, are new in home-mixer/scorers/ranking_scorer.rs.
• A new channel sends the debias payload verified — home-mixer/scorers/vm_ranker_debias_payload.rs.
  • It is a new file (52 lines). If the new home-mixer param VMRankerSendDebiasInputs (default false) is true, it sends per-candidate content features and the request context in the experiment_payload proto field.
  • The content features are has_video, max_video_duration_ms, has_photo, media_count, weighted_text_len, newline_count, has_url, and the same features of the quoted tweet.
  • The request context is engine_time_hint_ms, the columnar user-sequence ArrowIPC bytes, product surface, and client_app_id.
  • The name is debias, but the code alone does not show what is being debiased; it is likely a reserved slot for an experimental payload inferred.
• The vm_ranker scorer on the home-mixer side became thinner verified — home-mixer/scorers/vm_ranker.rs.
  • 15 lines were added and 31 lines were deleted in the existing file.
  • The inline branching that selected among 4 modes was encapsulated in the new struct OptionalInputs { value_model, compute_value_model, cached_weighted_score, debias }. OptionalInputs::from_query() reads the home-mixer params and sets the 4 flags.
  • write_request()/write_candidate() branch into helpers.
  • The score response also widened from (tweet_id → score) to a (tweet_id → (score, weighted_score)) tuple, and this value is carried in candidate.weighted_score.

B. VF filled the reply-restriction slot reserved yesterday with 4 rules

• Four new rules that decide conversation control (a setting where the author restricts who can reply) were added verified — visibility-filtering/rules/tweet_rules.rs:328-383.
  • The four rules are grouped in LIMIT_REPLIES_CONVERSATION_RULES.
  • The result of LimitRepliesByInvitationConversationRule (:338-346) is LimitedEngagementReason::ConversationControl. Its condition is HasConversationControl(ByInvitation) AND NOT_LOGGED_OUT AND NOT_RETWEET AND NOT_CONVERSATION_ROOT_AUTHOR AND NOT_INVITED_TO_CONVERSATION.
  • LimitRepliesCommunityConversationRule (:347-359) adds NOT ViewerIsFollowedByConversationRootAuthor to the same base.
  • LimitRepliesSubscribersConversationRule (:360-372) adds NOT ViewerSuperFollowsConversationRootAuthor to the same base.
  • The condition of LimitRepliesVerifiedConversationRule (:373-382) is HasConversationControl(Verified) AND ... AND NOT ViewerHasVerifiedBadge.
  • The LimitedEngagementReason::ConversationControl slot is filled by these 4 rules, as reserved yesterday.
• A new hydrator (a stage that fills in the data needed before a verdict) fetches data from both TES and socialgraph verified — visibility-filtering/hydration/conversation_control_hydrator.rs.
  • It is a new file (310 lines), and the flow has three steps.
    1. TES (Tweet Entity Service, X's internal tweet metadata service) get_conversation_controls() returns the arm, the root author, and the invited user IDs.
    2. If the arm is Community, socialgraph batch_check_followed_by(viewer_id, root_authors) checks whether the root author follows the viewer (reverse edge).
    3. If the arm is Subscribers, socialgraph batch_check_super_follows(viewer_id, root_authors) checks whether the viewer super-follows the root author.
  • The result is held in the new type ConversationControlFeatures { control, root_author_follows_viewer: Option<bool>, viewer_super_follows_root_author: Option<bool> }. This type is in visibility-filtering/models/conversation_control.rs.
• The socialgraph client gained a reverse-edge lookup verified — visibility-filtering/clients/socialgraph_client.rs.
  • The change is +130/−67, and the new method batch_check_followed_by(viewer_id, user_ids) was added.
  • This method looks up reverse edges of FOLLOWS_GRAPH with FlockDB 500-ID chunk pagination and returns a membership dict.
  • The edge_membership_query() signature was generalized to take direction: EdgeDirection.
• The value representing a logged-in viewer now carries the profile as well verified — visibility-filtering/models/viewer.rs.
  • Viewer::LoggedIn carries the profile inline. Viewer::LoggedIn(u64) was expanded to Viewer::LoggedIn { id, profile: ViewerProfile }.
  • ViewerProfile has 4 fields: allows_sensitive_media, viewer_age, has_verified_badge, and account_country_code.
  • has_verified_badge is true when Gizmoduck (X's user information service) reports VerifiedType::Business/Government or is_blue_verified. The LimitRepliesVerifiedConversationRule seen above consumes this value.
• Hydration was reorganized to be lazy, and this is the core of this refactor verified — visibility-filtering/hydration/mod.rs. As a result, hydration latency depends only on the set of rules each surface actually references.
  • The change is +360/−77. The new enum Hydrator and the bitset Hydrators(u8) were introduced.
  • The enum has 7 variants: ViewerProfile, TweetSafetyLabels, Tweet, ExclusiveContent, Author, Relationship, and ConversationControl.
  • Each predicate declares the hydrators it needs with const fn hydrators(). RuleClause::hydrators() takes the union of all predicates in a clause to determine the required hydrator set.
  • RuleEngine::hydrators_for(safety_level) runs only the hydrators that the active policy requires.
  • For example, TweetPredicate::HasConversationControl(_) requires Hydrator::ConversationControl. This hydrator runs only when a SafetyLevel (a per-surface policy bundle) enables a rule that references this predicate.
• The large deletion of TES hydration code is a reaction to the change of direction verified.
  • The deletions are tes_hydrator.rs -333 and models/mod.rs -223.
  • The 11 fields were migrated to the Strato (the X internal system that code goes through to read or store data) column TweetForVisibility in the 2026-09-18 commit (Gen4+25, 8b25829).
  • Since then, TES handles only the pure PureCoreData (author_id, text, source_tweet_id).
  • In today's commit, the PureCoreHydration wrapper struct and the cache-recovery logic were both removed.
  • Renaming fetch_pure_core() → fetch_author_ids() simplified the return type to TweetHydrationBatch<AuthorId>.
  • resolve_candidates() was also compressed into an inline fallback that no longer takes a wrapper struct.

C. grox video safety verdicts gained a pipeline that extracts key frames

• It finds scene cuts with PyAV and OpenCV and extracts representative frames verified — grox/libs/video_tools/key_frames.py.
  • KeyFramesExtractor.extract_key_frames() in the new file (206 lines) does this work.
  • It resizes frames to 32×32 grayscale to build a normalized signature. It judges a cut when the L2 distance between adjacent frames exceeds max(6.0, 4.0 × median_delta).
  • For each shot, it ranks by how far the mid-frame signature is from the video median. Blank shots with stddev<8.0 come last.
  • The top max_key_frames=3 frames are decoded exactly by PTS.
  • The scan window is window_sec=3 and max_fps=15 (default).
• Key frame extraction is attached as a new task to the PTOS (grox's policy-violation verdict stack) safety pipeline verified — grox/flows/ptos/plan_safety_ptos_key_frames.py.
  • The DAG in the new file (52 lines) is rate limit → key frames filter → media hydration → adult-only init → policy detection → safemodel sex/nudity → adult content cross-validation → result sink.
  • To pass the filter (task_safety_ptos_key_frames_filter.py:26-83), the media must be a video and have at least HIGH_FAV_THRESHOLD=128. In addition, it must have an existing adult label or have fav of at least KEY_FRAMES_UNCONDITIONAL_FAV=512.
  • A rate limit (60-second TTL, 10,000-entry dedupe) suppresses duplicate processing.
• Media storage moves from blobstore to O2 verified — grox/core/data_loaders/media_reference_bundle.py.
  • The change is +51/−18. The new class S3Store wraps O2 storage (https://o2-https.o2-prod.atla-prod-storage.cluster.kube.int-x.ai) with boto3.
  • s3://<bucket>/<prefix> URI parsing replaces the existing blobstore://namespace/subpath.
  • In Gen4+6 (2026-08-21), only the trait interface in phoenix/crates/storage/xai-o2/ was released, and the implementation was missing from the build. That backend is actually used in the grox media path through the boto3 S3 protocol.
• The media loader gained an optional key frame path verified — grox/core/data_loaders/media_loader.py.
  • The change is +39, and _construct_convo_video_from_key_frames() is new.
  • If the config MediaHydrationConfig.use_key_frames (default False) is turned on, it returns a ConvoVideo composed of 3 key frames instead of the original video.
  • The 3-second window, 15 fps, and maximum of 3 frames are exposed as config.

D. The cold-start options of Phoenix MoE changed

• The cold-start parameter names of Phoenix MoE (mixture of experts, a model structure that selects and uses some of several expert subnetworks) are being migrated for experiments verified — home-mixer/params/param.rs.
  • The default value layout of PhoenixMOEMaxResults and PhoenixMoeColdStartMaxResults changed.
  • PhoenixRetrievalMOEInferenceClusterId was rotated to a different cluster constant.
  • This appears to be swapping parameters while an experiment on the cold-start MoE slice is running inferred.

pub compute_value_model: bool = false; pub weights: ValueModelWeights = ValueModelWeights { favorite: 0.5, reply: 5.0, retweet: 1.0, quote: 5.0, share: 2.0, follow_author: 4.0, click: 0.4, video_open: 0.07, photo_expand: 0.05, not_interested: -43.2, block_author: -31.2, mute_author: -58.8, report: -234.0 }; — Summary, vm-ranker/params.rs, 2026-09-23

1. The statement that home-mixer param.rs is the only weight mirror is partially updated. Until now, x.md treated home-mixer home-mixer/params/param.rs as the production cron mirror (a mirror file that syncs production values daily and publishes them). Today a second channel, vm-ranker/params.rs, opened.
   • The home-mixer mirror holds the weights that the home-mixer process consumes, and the VM-Ranker mirror holds separate values that the VM-Ranker process consumes. The relationship between the two is still unclear.
   • With ComputeValueModel=false by default, VM-Ranker's values appear to run only as shadow (for observation and comparison) inferred.
2. The statement that home-mixer ranking is "weighted sum → filter → DPP → ads blend" is kept, but the description "vm-ranker is a score-only wrapper" is retired. Which of the three modes VM-Ranker runs in is now decided at run time.
   • The three modes are a mode that only does DPP reordering, a mode that only sends inputs, and a mode that computes the value model inline.
   • The commit alone does not show whether the send_inputs mode is always on in home-mixer or when compute_value_model is turned on in the experiment group.
3. The statement about the reserved slots in the VF verdict tree is corrected. Yesterday's daily recorded "the LimitedEngagementReason::ConversationControl slot has only wiring and no rule yet", but 4 rules filled this slot within a day. TombstoneReason::LocalRegulations is still reserved.

Needs a deep dive The 4 items below need separate investigation, so they are handled in a separate human-initiated session.

• What exact function form the inline computation in vm-ranker/scoring/value_model.rs takes (the aggregation of the weighted sum, and the score combination rule before DPP)
• Whether the xai_value_model crate is released in this repository
• Through what editing workflow the split weights of home-mixer and VM-Ranker are synchronized
• What the intent of SendDebiasInputs is

2Of 5 new arXiv papers, 1 Spotify paper passed the radar criteria — arXiv

Of the 5 new arXiv cs.IR papers since yesterday, scanned against the radar requirements, 1 passed. Today's batch has 30 new cs.IR entries (30 parsed <entry> items, submitted 2026-09-19–09-22), and those new since yesterday are 5 submitted on 2026-09-22. The requirements are three: platform-company authors, a production A/B statement, and novelty in mechanism or scale.

• Spotify published a training method that reduces over-reliance in an LLM reranker for personalized search, together with online A/B results verified — arXiv 2609.25825.
  • The paper title is Robust Fusion of Semantic and Behavioural Signals for LLM Reranking in Personalised Search (2026-09-22, accepted at USRW Workshop at RecSys 2026).
  • In the HTML version, we confirmed that the email domain of all 7 authors is @spotify.com.
  • The problem is shortcut learning that arises when Query Slice Stats (QSS, a summary of past success rates per query-candidate pair) are put into the prompt of an LLM cross-encoder reranker in personalized search. The model over-relies on QSS and becomes weak on rare/unseen queries.
  • To mitigate this, it trains with deterministic dual-sample feature-dropout. Each example is fed to training twice: once with QSS and once without QSS.
  • Offline, ranking quality rose +13.3% with QSS, and on the QSS-removed slice it improved +4.0% over naive QSS.
  • In the online A/B test, both QSS-aware variants had search success of ≈+2%.
  • Added to radar.md.

The other 4 were rejected, and none is on the watch list either. All four are academic research.

• SpeakerMem-R1: multi-party dialogue memory
• Discovery-Driven Table Integration: data integration
• Knowledge-as-Skill LLM Agents: benchmark
• ARAFA: Arabic fact-checking dataset

The industry blog scan is done only on Mondays, and today (Thursday) is not a target.

The latest Meta engineering blog posts are two from 2026-09-21 (Rebalancer open source, Petal subsea cable). Both are infrastructure posts, so we skip them. The latest Netflix Tech Blog post is "Leave the Class Path in the Rearview Mirror" (JVM infrastructure) from 2026-09-18, and we skip this one too.

Radar

• Spotify Robust Fusion: An LLM reranker using QSS and dual-sample dropout, with a production A/B test. Offline +13.3%, online A/B search success ≈+2%. Accepted at USRW Workshop @ RecSys 2026.

What to check next