2026-08-10

Other changes reviewed today:

• Three of the four LLaTTE (the user model of Meta ads) quotes we put into the canon yesterday were not in the original Meta blog post.
• Of the 30 new arXiv cs.IR submissions, we added Yandex Music's Gryphon-v2 and the Twitch·Amazon live-streaming multi-objective ranking paper to the radar.

1GenRec's serving layer was published in a separate blog post — Netflix

This structure matters because of serving cost. GenRec does not run the LLM once per candidate; it processes the whole candidate set in one pass. So the decode stage that generates tokens is effectively unnecessary, and only the prefill stage, which reads the prompt and builds the internal state, is used. This post explains which engine Netflix chose for this prefill-only workload and what the bottleneck was in that engine.

• GenRec is explicitly named as one of the four core workloads Netflix used as criteria when choosing its serving stack verified.
  • The blog states directly that this workload was one axis of the workload expansion that moved Netflix from TensorRT-LLM to vLLM: "prefill-only inference for ranking and retrieval".
  • The engine switch happened in the summer of 2025, and the V1 migration was carried out in Q4 2025.

• Batch scaling of constrained decoding set the upper bound of GenRec batch serving verified. Constrained decoding is processing that keeps the model from picking tokens outside a fixed candidate list.

  • In vLLM V0 this processing was implemented as a custom logits processor (a function that adjusts the per-token scores output by the model before sampling), but because of the Python GIL it ran sequentially per request.

  • So the CPU time spent on logits processing grew in proportion to batch size.

    "CPU time in logit processing therefore grows linearly with batch size" — In-House LLM Serving at Netflix (blog), 2026-07-17

  • In V1 this processing was rebuilt with batch-level handling, a C++ implementation of the hot path, and multi-threading. As a result, logits processing time did not grow as batches got larger: "logits processing time staying flat as batch size grows".

  • This explains, on the hardware and engine side, why GenRec's "whole candidate set in one pass" scoring holds even as batch scale grows.

• Under a "no special snowflake" principle, Netflix does not treat LLM serving as an exception and integrates it into the existing stack verified. Every model, whether an XGBoost ensemble or a large LLM, is scored through the same gRPC call.

  "Every model — XGBoost ensemble or large-scale LLMs — is scored via the same gRPC call" — Same blog post, 2026-07-17

  • The OpenAI-compatible API was added only as a separate frontend.
  • Small models are handled in-process on CPU, and large models are sent to a remote MSS (Model Scoring Service).

• We confirmed the architecture components verified.
  • It uses MSS, Triton, and a Java control plane.
  • It applies FSx pre-materialization, which unpacks the model weights onto the filesystem in advance.
  • Deployment uses two approaches, Red-Black and Versioned.

• In services/netflix.md, we created a new subsection, "GenRec's serving layer — published in a separate blog post (2026-07)", under the GenRec section. We also added this blog post to the sources section.
  • The subsection covers the engine switch date, the list of four core workloads, MSS, Triton, the Java control plane, and the "no special snowflakes" principle.
  • It also covers FSx pre-materialization, the batch-level restructuring of constrained decoding, and the contrast between Red-Black and Versioned.
• We attached a one-paragraph footnote to the Netflix case in topics/llm-in-recsys.md. It says GenRec is explicitly listed among the four core workloads behind the vLLM adoption.

Also, behind the statement "we put in an LLM ranker" lies a reimplementation at the engine layer inferred. The basis is that keeping performance at larger batch scale first required rewriting logits processing at the batch level. However, the blog does not disclose whether vLLM upstream or Netflix did that reimplementation, so we do not go as far as writing "Netflix fixed the engine."

↑ Contents

2GEM training could not use the LLM training stack as is — Meta (ads)

MFU (Model FLOPs Utilization) is a metric showing what percentage of theoretical performance the GPUs actually use. In large-scale training, a low value means idling the GPUs that were bought, so raising MFU becomes the primary goal of the infrastructure team. This post explains how GEM is trained and why the LLM training stack could not be reused.

• Over 12 months, MFU went up 2× and total FLOPs went up 4× verified.
• The current E2E MFU is 20–25% verified.

• Meta itself states that LLM infrastructure does not transfer directly to recommendation verified.

  "AI infrastructure optimized for LLM training ... does not directly transfer, requiring significant innovation and hardware/software co-design" — Training GEM at LLM Scale (blog), 2026-08-03

  • What it actually built new are custom kernels (JFA, GDPA, BlockAttention), MXFP8 low-precision training, 5D parallelism, and SM-free collectives.
  • It means that the sparse embeddings, jagged tensors (sequence tensors of varying length), and heterogeneous attention that recommendation workloads handle are fundamentally different from the LLM training stack.

• It was stated explicitly for the first time that DHEN remains inside GEM as an expert module verified. This phrase appears in the body: "Deep Hierarchical Ensemble Network experts".
  • Until now we understood the GEM entry in the lineage table only as "Wukong + InterFormer unified". In fact, DHEN is also absorbed into it.

• In the Lineage B table of topics/sequence-transformer-ranking.md, we added a "2026-08 GEM training" row and created a new section on GEM training infrastructure. We also added this blog post to the sources.
  • The new section covers custom kernels, MXFP8, 5D parallelism, and the fact that DHEN remains as an expert inside GEM.

Also, the precision of the lineage table went up one level. The understanding of "Wukong·InterFormer unification" remains valid, and the fact that DHEN is also inside it has been added.

↑ Contents

Overturned in verification: 3 LLaTTE quotes from yesterday are not in the blog text

observed On 2026-08-10 we re-fetched the original, saved it to /tmp/llatte-blog.html, and checked it with grep. Of the 4 quotes we put into the canon yesterday, 3 are not in the blog text.

• ❌ "Deployed as the largest user model at Meta": not in the original.

• ❌ "4.3% conversion uplift on Facebook Feed and Reels": not in the original.

  • The A/B statement the blog actually wrote is: "a cumulative lift of 6% in conversions on Instagram, 3% in conversions on Facebook and 3.5% in ad clicks on Facebook".
  • This statement carries "cumulative" and "our broader model innovations", so it cannot be seen as the effect of LLaTTE alone.

• ❌ "semantic features bend the scaling curve: they are a prerequisite for scaling": the original text does not contain the word "prerequisite".

  • The corresponding sentence is much milder in wording.

    "Semantic content features from foundation models complement traditional collaborative filtering... especially helpful in cold-start scenarios" — From User Sequences to Scaling Laws (blog), 2026-08-05

• ✓ "a two-stage architecture that offloads the heavy computation...": we confirmed that this one is in the original.

We fixed the canon as follows. In the LLaTTE section of topics/sequence-transformer-ranking.md, we attached a re-verification failure mark and replaced the quotes with the A/B figures the blog actually states (6% / 3% / 3.5%, cumulative). We put the formulations of dense tokenization and target-aware attention back in as quotes from the original.

The 3 items above likely came from the LLaTTE paper (arXiv 2601.20083), and we infer that yesterday's canon labeled sentences from the paper as blog quotes inferred. Confirming this requires checking against the paper's text, so we leave it in deep-dive-needed status.

There are two lessons from this.

1. Adversarial verification and re-fetching the original are different checks. A quote that passed two rounds of adversarial verification yesterday showed a defect in today's re-fetch of the original. Adversarial verification checks whether "what is written agrees with itself", while re-fetching checks whether "it is actually in the original".
2. From now on we state (blog) / (paper) next to every quote. This rule is in the link rules of CLAUDE.md. There are two reasons.
   • Meta blog posts can be edited even when the URL stays the same inferred. The basis is that there appears to be a practice of adjusting wording after publication.
   • In lineage sections that cite both papers and blog posts, the sources of quotes are easily mixed up.

↑ Contents

Radar

Of the 30 new arXiv cs.IR submissions screened, 2 passed the acceptance criteria. We added both to "New this month" in radar.md and set the last updated date to 2026-08-10.

• Yandex Music's Gryphon-v2 replaces retrieval, rerank, and rank with a single model (arXiv 2608.06213, 2026-08-06).
  • The structure encodes the user history once, generates candidates with Semantic-ID (a token sequence that represents items as hierarchical codes), and then continues to an item-level ranker that reuses the shared encoder state.
  • It states that it replaces the existing cascade in a production A/B test, and active users rose by +1.41% verified. The original wording is: "replaces the production cascade of 15+ candidate generators, pre-ranking and final ranking".
  • This is the second production statement that corresponds head-on to GenRec. GenRec handles only the ranking path with an LLM, whereas this one also unifies retrieval.
• Twitch·Amazon's Multi-Objective Ranking for Live-Streaming (arXiv 2608.04455, 2026-08-05) addresses the problem that immediate and delayed signals in live streaming have different time scales.
  • The immediate signals are click and initial watch, and the delayed signals are session length and retention.
  • It trains them separately with delayed window collection and a multi-model architecture, and then combines them.
  • It states that DAV and ARPU improved meaningfully in a mobile live feed online A/B test verified.
  • It is adjacent to Netflix's live cold start (RecSys 2025), but this one covers multi-objective ranking for general live feeds.

The remaining 28 were rejected because they were purely academic increments or limited to a single domain.

↑ Contents

What to check next

↑ Contents