2026-09-04

1Phoenix ranker's backbone and re-ranking head layers first appear in code — X

• home-mixer started turning on the backbone-score request flag as a hardcoded value at all five feed endpoints verified — home-mixer/server.rs.

  • The line that turns it on is query.return_backbone_scores = true.
  • Each of the five gRPC endpoints turns on this flag right before handing the request to Phoenix.
    • ScoredPostsService (internal scored posts)
    • ForYouFeedService (For You main)
    • The polling variant of ForYouFeed
    • RankedFollowingFeedService
    • FollowingFeedService
  • There is no parameter gating. That is, the production default is on.

  let RequestContext {
      b3_info,
      mut query,
      root_span,
  } = ctx;
  query.return_backbone_scores = true;
  let output = self.get_for_you_feed(query).instrument(root_span).await?;
  

  — home-mixer/server.rs, 2026-09-02

• A new bool field that decides whether to request backbone scores was added to the request struct verified — home-mixer/models/query.rs.

  • The struct is ScoredPostsQuery, and the field is return_backbone_scores: bool.
  • The default constructor (ScoredPostsQuery::new) initializes it to false, and only the server entry point sets it to true.
  • In other words, it is off by default in internal pipeline and test paths, and only the five feed requests entering through the gRPC server require backbone scores.

• The Phoenix request builder passes this flag through unchanged verified — home-mixer/util/phoenix_request.rs.

  • One line, return_backbone_scores: query.return_backbone_scores, carries the value from the request context to the proto field.

• The Phoenix proto added backbone fields to both the request and the response verified — phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto.

  • One request field and two response fields form a set.
    • Request: returnBackboneScores (#20)
    • Response: backboneTopLogProbs (#10, repeated float)
    • Response: backboneContinuousValues (#11, repeated float)
  • Top appears to refer to the top log probabilities of the discrete head (for example, top-K logits over the action vocabulary), and Continuous to the dense regression output inferred.
  • The evidence is the naming convention, and that it corresponds to topLogProbs (the discrete values the reranker head outputs) already present in NextActionDistribution inferred.
  • The structure in which the reranker head is a fine-tuned head sitting on this backbone is consistent with the rerankerHeadTag field that the earlier canon version observed inferred.

  message PredictNextActionsRequest {
    ...
    PageDecodeParams pageDecode = 19;
    bool returnBackboneScores = 20;
  }
  
  message NextActionDistribution {
    ...
    SlateContext slateContext = 9;
    repeated float backboneTopLogProbs = 10;
    repeated float backboneContinuousValues = 11;
  }
  

• A backbone score field was added to the candidate object, and the scorer parses the response in two layers and stores them verified — home-mixer/models/candidate.rs, home-mixer/scorers/phoenix_scorer.rs.

  • The new field is PostCandidate's backbone_scores: Option<PhoenixScores>.
  • The scorer parses the same prediction response twice. candidate_scores is the reranker head output (the existing phoenix_scores), and candidate_backbone_scores is the backbone output newly exposed this time.
  • In other words, a candidate leaves for the next stage carrying the scores of both layers from one prediction round.

  // candidate.rs
  pub struct PostCandidate {
      ...
      pub reranker_head_tag: Option<u32>,
      #[serde(default, skip_serializing_if = "Option::is_none")]
      pub backbone_scores: Option<PhoenixScores>,
      ...
  }
  
  // phoenix_scorer.rs
  .map(|c| PostCandidate {
      phoenix_scores: predictions.candidate_scores(&c.get_original_tweet_id()),
      backbone_scores: predictions.candidate_backbone_scores(&c.get_original_tweet_id()),
      ...
  })
  

• The scorer only stores them, and the downstream consumers lie outside this diff inferred.
• However, the fact that the backbone output keeps being carried on the candidate means at least one of the following inferred. The evidence is that backbone_scores is a field of PostCandidate and, judging by the scaffolding, no path drops the value.
  • The re-ranking and diversity stages consume backbone logits as additional features.
  • The ads/feed mixer records backbone output as a payload for debugging/attribution.
  • Another scorer uses backbone output as input, for example VMRanker's slate reordering or the ads blender MultiRiskAdsBlender.
  • It is an observability channel that lets offline analysis and A/B verdicts see "how much the backbone moved, not the reranker head."
• New consumers are likely to appear in the next commits inferred.

• The default for how Phoenix retrieval aggregates user history changed from short dwell to long dwell verified — home-mixer/params/param.rs.

  • The sync timestamp in the file header was updated from 2026-09-01T16:42:25Z to 2026-09-02T16:03:45Z, so the production feature switch actually changed within a day and the source mirror caught up with it.
  • The comment at the top of the file reads: "mirrored from config feature-switch defaults; last sync ...".
  • Long-dwell aggregation merges only the candidates the user stayed on for a long time into the history embedding, which biases retrieval's signal further "toward high-quality engagement" inferred. The evidence is that this is consistent with the trend since Gen4+6 (2026-08-21), when X greatly expanded the ads conversion head and shifted weight toward dwell-based positive signals.

  param!(
      PhoenixRetrievalAggregationType,
      String,
      "rust_home_mixer_phoenix_retrieval_aggregation_type",
      "DENSE_WITH_LONG_DWELL"   // was "DENSE_WITH_SHORT_DWELL"
  );
  

• Phoenix serving's batch preparation code added a defense in two places that zero-initializes the buffer before filling it verified — phoenix/crates/serving/xai-recsys-engine/src/python.rs.

  • The places initialized are the candidate embedding slice and the search-query embedding slice.
  • Previously, values from the previous batch remained in the reused buffer, and when copy_len < embedding_row.len() (the filled length is smaller than the slot size), the leftover bytes leaked.
  • A copy_len > 0 condition is also applied, so the copy is skipped when the length to copy is 0. The commit does not state a reason.
    • Both sides are sliced to the same copy_len, so a zero-length copy does not panic either verified — Rust slice::copy_from_slice docs. This function panics only when the two slice lengths differ.
  • As a follow-up to Gen4+11 (2026-08-28, bc8e5f0), which cleaned up the emb_table gRPC receive pipeline, it closes a potential correctness issue at the batch-reuse boundary.

  .par_chunks(chunk_size)
  .zip(candidate_embeddings_slices.par_iter_mut())
  .for_each(|(items_chunk, shard_slice)| {
      shard_slice.fill(f16::ZERO);
      shard_slice
          .par_chunks_exact_mut(row_size)
          ...
  });
  ...
  if search_query_embedding_dim > 0 && let Some(sq_slice) = ... {
      sq_slice.fill(0.0);
      sq_slice
          .par_chunks_exact_mut(candidate_seq_len * search_query_embedding_dim)
          ...
  }
  

• After the Gen4 release, the canon knew only the top-level semantics: "Phoenix is a Transformer that emits 24 probabilities per candidate." The rerankerHeadTag field in Gen4+11 let us observe that "several re-ranking heads exist," and Gen4+14 confirms that "beneath those heads there is a backbone layer whose output is a separate response field." This means the standard form of a multi-task ranker (a shared bottom and task-specific heads) is present in X production as is.
• return_backbone_scores is forced on in the five feeds without parameter gating. This is a strong signal that this response addition is already served as a production default, not at an A/B experiment stage. It means the Phoenix serving side is ready to compute the backbone output and put it in the response, and that the latency budget has room.
• "Carrying the backbone score on the candidate" is a reservation that lets the next layer consume it. This commit shows only the storage, but code in which downstream components such as MultiRiskAdsBlender, VMRanker, and DPP re-rank (re-ranking that reflects candidate diversity) use backbone logits as additional features is likely to follow soon. Paired with Gen4+11's rerankerHeadTag, this makes it something to watch whether code for "switching, ensembling, and attribution among hydra heads" opens up in later commits.
• That the Phoenix retrieval dwell default moved to long is consistent with the trend of expanding the ads conversion head since Gen4+6. In effect, in the two-layer ranker, the signal the backbone learns and the user-history aggregation retrieval consults are aligned in the same direction (long-dwell-heavy).

↑ Contents

2scarecrow (formerly Botmaker) gained a path to file automatic reports with XReview — X

• Two new function units send reports from scarecrow to XReview verified — scarecrow/legacy/XReviewIntakeRegistry.scala (179 lines, new).

  • The two functions are CreateXReviewReportProd and CreateXReviewReportStaging. They can be called from scarecrow's rule DSL and send reports to production and staging XReview respectively.
  • A scarecrow rule passes seven arguments when it calls this function.
    • entityType: "post" or "profile"
    • entityId: tweet id or user id
    • userId: the reported user
    • reportType: an XReview allowlist value (for example, bystander_impersonation)
    • botId: the automatic detection bot that created this report
    • Optional note
    • Optional victimId: the victim's user id
  • The self-description attached to the CreateXReviewReportProd class is the same as the quote below.

  "Submits a report to production XReview Intake over gRPC. entityType must be post or profile. reportType must be an XReview-allowlisted value. Impersonation bots should use bystander_impersonation (lane tags key on that type, not generic impersonation). Each evaluation includes detection_timestamp_ms so intake's content-hash report id is unique; evidence rollup then applies." — XReviewIntakeRegistry.scala, description of the CreateXReviewReportProd class

• The XReview intake was registered in the downstream services list verified — scarecrow/legacy/DownstreamServices.scala.

  • The registered name is XReviewIntake.

  case object XReviewIntake extends DownstreamService {
    override val name = "XREVIEW_INTAKE"
  }
  

• A separate client exists for staging, and the staging function is marked as experimental with no real action verified — XReviewIntakeRegistry.scala.

  • CreateXReviewReportStaging sets the action level to NO_ACTION.
  • In scarecrow, the action level is metadata that indicates whether this function is a real action or a dry run. The staging version is marked NO_ACTION, which states explicitly that it is experimental and the rule is not applied in practice.
  • The production version is PROD_AGENT_WORKFLOW_ACTION. The phrase "agent workflow" means it is part of an automatic action pipeline.

• Developer local environments and personal-account instances do not call a real XReview server verified — XReviewIntakeRegistry.scala.

  • In localMode or personalAccountMode, it is replaced with a noop.

  if (localMode() || personalAccountMode()) {
    XReviewReportSubmitter.noop(statsReceiver)
  } else {
    ...
  }
  

• scarecrow connects to XReview intake over mTLS verified.

  • The flag xreview.tls.ca-cert takes the CA certificate path, and SharedXReviewIntakeClientModule.provideXReviewIntakeClient consumes it to create the mTLS connection.
  • The client comes from a shared module in the com.twitter.useng.common.xreview package. useng is X's internal shared-library namespace.
  • In other words, this suggests that systems other than scarecrow also access XReview intake programmatically.

• The report payload fills in XReview's SubmitReportRequest gRPC proto as is verified — scarecrow/legacy/XReviewReportBuilder.scala (59 lines, new).

  • A few keys directly show the nature of this pipeline.
    • report_surface = "x_app": the report arrives through the same surface as user reports.
    • report_source = "proactive": tells XReview that the report comes from automatic detection ("proactive"), not from a user report ("user"). In other words, XReview uses this field to distinguish which reports were created by a person's click and which were put in by a bot.
    • subject = "Botmaker report": the top line of the report is fixed.
    • reporter_id = "0": a bot report has no reporting user id. Instead, detection_bot_id holds which bot created the report.

  val ReportSurfaceValue = "x_app"
  val ReportSourceValue = "proactive"
  val Subject = "Botmaker report"
  val ReporterIdValue = "0"
  ...
  val reportFields = Seq(
    pb(XReviewReportKeys.Subject, Subject),
    pb(XReviewReportKeys.ReportSurface, ReportSurfaceValue),
    pb(XReviewReportKeys.ReportType, reportType),
    pb(XReviewReportKeys.ReportSource, ReportSourceValue),
    pb(XReviewReportKeys.ReportedEntityType, entityType),
    pb(XReviewReportKeys.ReportedEntityId, entityId.toString),
    pb(XReviewReportKeys.ReportedUserId, userId.toString),
    pb(XReviewReportKeys.DetectionBotId, botId.toString),
    pb(XReviewReportKeys.DetectionTimestampMs, detectionTimestampMs.toString)
  ) ++ note.filter(_.nonEmpty).map(n => pb(DescriptionKey, n)).toSeq
    ++ victimId.filter(_ != null).map(id => pb(XReviewReportKeys.VictimUserId, id.toString)).toSeq
  

• The entityType whitelist is limited to "post" and "profile" verified.

  • The normalizeEntityType function throws IllegalArgumentException on any other value, which keeps scarecrow rules from putting arbitrary entity types into XReview.

• The uniqueness of a report id is determined by a content hash together with a timestamp verified.
  • The quote above reads: "intake's content-hash report id is unique; evidence rollup then applies". It means the XReview server builds the report id from a content hash and detection_timestamp_ms.
  • Even if the same bot detects the same content twice, the reports are separate if the timestamps differ, and XReview rolls up (aggregates) the evidence afterward.
  • That is, duplicate detections naturally occur many times in bot reports, and XReview handles those duplicates.
• Impersonation detection bots must use bystander_impersonation verified.
  • The self-description reads: "Impersonation bots should use bystander_impersonation (lane tags key on that type, not generic impersonation)."
  • The lane tag here appears to mean XReview queue routing (which human reviewer team and SLA a report goes to).
  • That is, XReview has a "lane" concept that routes reports to reviewer teams, and the report_type value decides the lane.
  • If a bot uses the wrong report_type, the report is routed to the wrong lane, and review is delayed or ignored.

• There are five observability (metrics) counters verified — XReviewReportSubmitter.scala (91 lines, new).
  • attempted: number of calls
  • submitted: successes
  • skipped: skipped because there is no prodClient/stagingClient
  • unsupported_entity: entityType outside the whitelist
  • rpc_failed: gRPC failure
  • Failures are logged at warn and the exception is rethrown.

• The safety layers the earlier canon version knew were two.
  • visibility-filtering is the layer that drops candidates inline in the scoring and serving path or attaches an interstitial (an action that overlays a warning screen).
  • grox PtOS is the content labeling layer in which the Grok LLM summarizes tweets and attaches labels.
• This commit (Gen4+14) exposes a third layer in code: scarecrow → XReview. Automatic detection bots push reports into the XReview queue through intake as if they were user reports, and XReview then handles evidence rollup and routing to human reviewers (lane tag). This is an asynchronous report and aggregation pipeline, not an inline drop.
• The report_source = "proactive" field suggests that user reports ("user") and bot reports ("proactive") enter XReview in the same queue with the source tagged. This means the XReview dashboard or reviewer UI shows "cases with user pressure" and "cases detected by bots" separately, and presumably the evidence rollup logic also weights the two sources differently inferred. Confirming this requires the XReview server-side logic to be released.
• The existence of the bystander_impersonation type means the impersonation response line is divided into finer categories. The supporting wording is: "bystander_impersonation (lane tags key on that type, not generic impersonation)". An impersonation case found by a bot is labeled as "impersonation reported by a third party" and routed to a separate lane. The reason it states explicitly not to use "generic impersonation" is that lane tag routing uses report_type itself as the discriminator. A wrong tag routes the report to the wrong team, SLA, and severity.
• The XReview client scarecrow uses lives in the useng.common.xreview shared module, that is, the com.twitter.useng.common.xreview package. This means the intake is not scarecrow-only but is called in common by several X internal backends. For example, the grox PtOS pipeline could also put reports into XReview through this library inferred. These usages are likely to become clearer in the next commits inferred.
• ⚠ Claim strengthened: the earlier canon understood X's handoff from automatic detection to human review as roughly "grox PtOS attaches a label and the policy service takes action." This commit confirms in code that one branch of that handoff shares the XReview queue with user reports through the same channel. In other words, XReview is not a queue that handles only user reports; it is also one of the destinations of automatic detection.

↑ Contents

3Visibility rule policies move from heap assembly to static tables — X

• The rule trait and the heap-allocated rule vector are gone from the code verified — visibility-filtering/rules/mod.rs.

  • What disappeared is the Rule trait and Vec<Box<dyn Rule>>.
  • Three items were removed along with them, for −155 lines in mod.rs:
    • the Rule trait definition (two methods, name and evaluate)
    • the evaluate_rules function, which iterated over the Box<dyn Rule> collection to build the verdict
    • 5 related unit tests

• The three policies are now carried as static tables in the rodata section of the program image verified — visibility-filtering/rules/registry.rs.

  • The table type is &'static [&'static [RuleSpec]].
  • All three policies (FILTER_ALL_POLICY, TIMELINE_HOME_POLICY, TIMELINE_HOME_RECOMMENDATIONS_POLICY) are static.
  • RuleEngine::new() no longer allocates on the heap and only wraps a NsfwGatingCountries reference.

  pub(super) struct Policy {
      rules: &'static [&'static [RuleSpec]],
      additional_rules: &'static [&'static [RuleSpec]],
  }
  
  static TIMELINE_HOME_SHARED_RULES: [&[RuleSpec]; 9] = [
      author_rules::AUTHOR_STATE_DROPS,
      author_rules::SOCIALGRAPH_DROPS,
      tweet_rules::TWEET_LABEL_DROPS,
      tweet_rules::NULLCAST_DROP,
      tweet_rules::TES_HOME_DROPS,
      tweet_rules::SENSITIVE_VIEWER_DROPS,
      tweet_rules::EXCLUSIVE_TWEET_DROP,
      tweet_rules::NSFW_MEDIA_INTERSTITIALS,
      tweet_rules::NSFW_AUTHOR_INTERSTITIAL,
  ];
  
  static TIMELINE_HOME_RECOMMENDATION_ONLY_RULES: [&[RuleSpec]; 5] = [
      tweet_rules::RECS_MEDIA_DROPS,
      author_rules::OON_NSFW_AUTHOR_DROPS,
      tweet_rules::OON_TWEET_FLAG_DROPS,
      tweet_rules::OON_TWEET_LABEL_DROPS,
      author_rules::OON_USER_LABEL_DROPS,
    ];
  
  static TIMELINE_HOME_POLICY: Policy = Policy::new(&TIMELINE_HOME_SHARED_RULES);
  static TIMELINE_HOME_RECOMMENDATIONS_POLICY: Policy = Policy {
      rules: &TIMELINE_HOME_SHARED_RULES,
      additional_rules: &TIMELINE_HOME_RECOMMENDATION_ONLY_RULES,
  };
  

• Policies was renamed to RuleEngine verified.

  • The reason is that this struct no longer owns a rule vector. The old name no longer reflected its state (a collection).
  • The new name makes explicit that it is a dispatch layer.

• A policy evaluates rules in order from the front verified — visibility-filtering/rules/registry.rs.

  • The evaluation function is Policy::evaluate, which iterates over the rule iterator in order.
  • The semantics are unchanged.
    • Drop short-circuits immediately.
    • Even if Interstitial comes first, Drop wins if a later rule returns Drop.
    • If Allow holds until the end, decided_by = None.

  pub(super) fn evaluate(&self, context: &RuleContext<'_>) -> Verdict {
      let mut action = VfAction::Allow;
      let mut decided_by = None;
      for rule in self.rules() {
          match rule.evaluate(context) {
              VfAction::Drop(reason) => {
                  return Verdict {
                      action: VfAction::Drop(reason),
                      decided_by: Some(rule.name()),
                  };
              }
              VfAction::Interstitial(reason) if matches!(action, VfAction::Allow) => {
                  action = VfAction::Interstitial(reason);
                  decided_by = Some(rule.name());
              }
              VfAction::Allow | VfAction::Interstitial(_) => {}
          }
      }
      Verdict { action, decided_by }
  }
  

• Added a new test that verifies the uniqueness of rule names across all 15 slices verified — visibility-filtering/rules/tweet_rules.rs.
  • wired_rule_names_are_unique_and_nonempty guarantees that rule names in all 15 slices, 11 tweet-axis slices and 4 author-axis slices, are non-empty and have no duplicates.
  • This assertion matters because rule names are referenced as literals in audits, metrics and the golden corpus.

• The whole rule graph can now be dispatched at program start without any allocation. The VF server can evaluate immediately after fork, and the rule list remains data that can be observed directly in the program image with readelf.
• The composition of 9 common home slices and 5 recommendation-only slices is the same as in Gen4+13, and the rule counts are also unchanged. However, RuleEngine::rule_counts() now calls the static Policy::len() instead of counting vector lengths.
• The Policies → RuleEngine rename makes clear the direction of this refactor, which ran across four commits from Gen4+11 to Gen4+14.
  • Move rules from code to data.
  • Move policies from the heap to rodata.
  • Name things from the dispatch viewpoint.
• The next step is likely to load rule literals from a config file (TOML/JSON), which would allow changing rules without a code deployment inferred. However, that would give up the benefits of Rust static dispatch (inlining, jump tables), so whether it actually goes this way depends on later commits.

↑ Contents

4Training checkpoints can be decrypted with a key from the file header, without a sidecar key file — X

• The function that reads the DEK uses the sidecar file if it exists and otherwise unwraps the key from the envelope header verified.

  • The function is named adopt_tree_dek.
  • The same function reads both checkpoints in the _DEK sidecar format created in the 2026-09-01 commit (Gen4+13) and checkpoints in the future in-envelope derived format.
  • The migration path is backward-compatible.

  def adopt_tree_dek(path: pathlib.Path, kms_client) -> bytes:
      ...
      if dek_path.exists():
          entry = _read_wrapped_dek(dek_path) or {}
          raw = bytes(xai_kms.nfs.unwrap_shared_dek(kms_client, dek_path))
          key_id = entry.get("key_id")
      else:
          raw = _unwrap_header_master(kms_client, path)
          key_id = "header"
      rank_logger.info("KMS unwrap OK for %s (key_id=%s)", path, key_id)
      return raw
  

• The envelope header of a checkpoint file is 4096 bytes and consists of the magic XAIENC01 and the wrapped_len/context_len fields verified — phoenix/python/training/xai-checkpointing/xai_checkpointing/dek.py.

  • The header structure is as follows.
    • The first 8 bytes are the magic XAIENC01.
    • At byte offset 28–32 is the length of the wrapped field (big-endian u32).
    • At byte offset 32–36 is the context length (u32).
    • The wrapped field and the context JSON follow within the 4096 bytes.
  • If the wrapped field has the form xai-dek1:<salt>:<master>, it is treated as the "derived" format; otherwise it is judged to be legacy per-file wrapping and an exception is raised.

  _ENVELOPE_HEADER_SIZE = 4096
  _ENVELOPE_FIXED_LEN = 36
  _DERIVED_PREFIX = "xai-dek1:"
  
  def _header_wrapped_master(path: pathlib.Path) -> tuple[str, dict]:
      envelope = _first_envelope(path)
      head = envelope.read_bytes()[:_ENVELOPE_HEADER_SIZE]
      if len(head) < _ENVELOPE_FIXED_LEN or head[:8] != b"XAIENC01":
          raise ValueError(f"not an envelope: {envelope}")
      wrapped_len = int.from_bytes(head[28:32], "big")
      context_len = int.from_bytes(head[32:36], "big")
      start = _ENVELOPE_FIXED_LEN
      field = head[start : start + wrapped_len].decode()
      context = json.loads(head[start + wrapped_len : start + wrapped_len + context_len])
      if not field.startswith(_DERIVED_PREFIX):
          raise ValueError(
              f"envelope at {envelope} carries a legacy per-file wrapped key, "
              "not the xai-dek1 derived form"
          )
      _salt, master = field[len(_DERIVED_PREFIX) :].split(":", 1)
      ...
  

• The read path changed to use the tensorstore driver xai_encrypted directly verified — phoenix/python/training/xai-checkpointing/xai_checkpointing/load.py.

  • Previously (2026-09-01 commit, Gen4+13), the KMS-proxy KvStore (xai_kms.KvServe) called KMS on every IO. Now the DEK is unwrapped once, and the xai_encrypted driver performs the symmetric decryption directly.
  • This approach removes the KMS round-trip latency from the IO path. The DEK is unwrapped only once, and symmetric decryption is then done locally.
  • Both the previous and the next file formats are supported.

  raw = adopt_tree_dek(path, kms_client)
  kvstore_base = {
      "driver": "xai_encrypted",
      "base": {"driver": "file", "path": path.as_posix() + "/"},
      "dek_b64": base64.b64encode(raw).decode(),
  }
  metadata_json = json.loads(_read_encrypted_file(path, "_METADATA", kvstore_base, ts_context))
  return kvstore_base, metadata_json, _names_from_tree_metadata(metadata_json), ts_context
  

• At save time, encryption_context requires two keys verified — phoenix/xrex/train/checkpoint_write.py.

  • The two keys are domain and run.
  • Judging by the names, domain is the system domain this checkpoint belongs to (for example "recsys", "grox"), and run is the training run id.
  • These two values are carried as the encryption_context of the KMS call, so the audit log records "which run of which domain unwrapped which checkpoint, and when".
  • In other words, this code enforces the KMS audit flow.

  missing_context = {"domain", "run"} - self.checkpoint_config.encryption_context.keys()
  if missing_context:
      raise ValueError(
          "encrypted checkpoint context is missing " + ", ".join(sorted(missing_context))
      )
  

• At save time, mint_nonce is added to encryption_context automatically verified.

  • The value is the sha256 hash of the checkpoint path, inserted automatically as the mint_nonce value of encryption_context. As a formula, mint_nonce = sha256(path).
  • This uses the standard AWS-KMS pattern in which KMS checks the context match at unwrap time, so an attempt to unwrap a different checkpoint with the same KMS key fails.
  • For example, if an attacker copies a checkpoint file to a different path and tries to unwrap it, the mint_nonce does not match and KMS rejects the request.

  encryption_context = dict(self.checkpoint_config.encryption_context)
  encryption_context["mint_nonce"] = hashlib.sha256(os.fsencode(path)).hexdigest()
  kms_client = xai_kms.KmsClient.from_cluster_env(
      key_id=self.checkpoint_config.encryption_key_id,
      encryption_context=encryption_context,
  )
  

• The encryption option and the tensorstore save approach were separated verified — phoenix/xrex/train/trainer.py.

  • _uses_tensorstore_save no longer forces encryption.
  • Previously, encrypt=True turned on the tensorstore save approach automatically.
  • Now checkpoint_write.save_checkpoint takes kms_client as an argument and assembles orbax_encrypted.get_encrypted_checkpointer by itself.

  def _uses_tensorstore_save(self) -> bool:
      return self.checkpoint_config.save_method == "tensorstore"
  

• The error message that prevents downgrading to plaintext saves became more specific verified.

  • The previous message was "restored from an ENCRYPTED checkpoint but orbax writer cannot produce encrypted saves".
  • The new message is "restored from an ENCRYPTED checkpoint but checkpoint_config.encrypt is false — refusing to downgrade to plaintext saves; set checkpoint_config.encrypt=true with encryption_key_id/encryption_context".
  • The new message states explicitly why it failed and what to do.

• _AsyncCheckpointer was extended to wait for wait_until_finished of the orbax_encrypted module as well verified — phoenix/xrex/utils/checkpointing.py.

  • When the flush of a checkpoint save finishes, all encrypted IO is also guaranteed to be finished.

• This commit completes write-path verification (enforcing domain/run and inserting mint_nonce automatically) and two-format support on the read path. However, whether CheckpointConfig.encrypt = True is on by default in the actual training config is not in this diff. This is still a checkpoint to watch.
• The file format is now documented. The 4096-byte header, the XAIENC01 magic and the wrapped_len/context_len offsets remain in the code, so future external tools (checkpoint viewers, audit tools) can parse this format.
• The xai-dek1: prefix indicates a key derivation scheme inferred. The phrase "derived master" suggests a KDF-based approach in which a master key derived from a salt is placed in the header and the wrapping is placed in the wrapped field inferred. This contrasts with the legacy per-file wrapped key (a separately stored KMS-wrapped DEK for each file) inferred. The details of X.AI's own KDF spec ("xai-dek1") are not in the code. They are presumably in the xai_kms.nfs module inferred.
• Removing the KMS-proxy KvStore matters for performance. Previously every IO (one ocdbt operation) required a KMS call, but now the DEK is unwrapped once and decrypted symmetrically in local code. KMS round trips drop from one per IO to one in total, so checkpoint restore latency is likely to fall as well inferred.
  • The path that unwraps once is handled by adopt_tree_dek verified — dek.py.

↑ Contents

5The default two-tower training config uses QK-norm instead of a logit cap — X

• Two attention-related defaults changed in the base config of the composite two-tower verified — phoenix/xrex/configs/xrecsys_two_tower.py.

  • qk_norm: False → True: adds normalization (L2) to the attention query and key vectors. QK-norm is one of the standard techniques for stabilizing Transformer training.
  • attn_logit_cap: 80.0 → -1: the cap (80) that clipped logit values before the attention softmax changed to -1 (= off). With QK-norm on, the QK dot product is already bounded, so a separate logit cap becomes unnecessary.

  def _xrecsys_two_tower_combined_base() -> dict:
      return {
          ...
          "learning_rate": 2e-3,
          "emb_learning_rate": 0.1,
          "qk_norm": True,          # was False
          "attn_logit_cap": -1,     # was 80.0
          "primer_norm": True,
          ...
      }
  

↑ Contents

6Of 30 new papers, 0 passed the radar criteria — arXiv cs.IR

Of 30 new arXiv cs.IR papers (parsing succeeded, lookup normal), 0 passed the radar acceptance criteria. The criteria are platform-company authors or production deployment/A/B claims, novelty in feed/ranking mechanisms, and large-scale results. There was no new material from Meta or Netflix that day.

• ReST (From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs) fell in this triage window, but it was already added to radar.md new this month in yesterday's 2026-09-03 daily. It is not registered twice.
• hLLM is an interesting decoding optimization for generative reranking, but it did not pass the radar criteria and was rejected.
  • The title is "Single Pass Decoding for Generative Reranking", and the authors are Emil Laftchiev, Prachi Agrawal, Moe Kayali and others.
  • It is a decoding scheme that uses LoRA fine-tuning and the Hungarian algorithm to produce a ranking of N items in an O(1) forward pass, and it claims to reach 28ms end-to-end.
  • It is a methods paper with no production deployment claim.
  • The author affiliation would need to be checked through the open-review author profile, but the abstract alone has no platform-company signature.
  • It is a candidate to review again if a production claim in this direction appears from a platform company.
• SPAR is a Generative POI Recommendation paper. It was rejected in principle because the research is limited to POI (radar domain restriction).
• GenCAR was also rejected. It is an academic experiment on OOD recommendation.
• All the rest were rejected as well. The reasons we confirmed fall into three groups.
  • benchmark/RAG methodology (no platform production claim)
  • domain restriction (POI, legal, medical, science)
  • author affiliations mostly academic institutions

↑ Contents

Corrections

None.

↑ Contents

Radar

None.

↑ Contents

What to check next

↑ Contents