0x00 한

Research / Daily reports

  • Sources read: 1
  • Canon files updated: 0
  • Radar entries: 1
In one linePinterest deployed a structure that infers a user's long-term goals (journeys) with a single Qwen3-4B generation instead of keyword clustering and uses the resulting names as notification retrieval queries.

1An LLM names long-term user goals as notification queries — Pinterest (radar)

Source From Activity to Intent: Generating User Journeys with LLMs — Pinterest Engineering Blog, 2026-10-09, blog post. There are nine authors, and the last author, Sudarshan Lamkhede, holds the title "Director, Machine Learning Engineering". The Medium web page returned 403, so I read the post through the Pinterest Engineering RSS, which carries the full text of the same post.

Background Most recommendation systems predict the "next item to tap". However, goals that span weeks to months, such as a kitchen remodel or wedding planning, are poorly captured by a single next click or by long-lived interest labels such as "Home Decor". Pinterest calls such a goal a journey (a bundle of interaction in which interest, intent, and context coincide at a point in time and continue across sessions), builds a list of in-progress journeys for each user, and uses it for notifications. This post covers how it replaced the multi-stage ML pipeline that builds that list with a single LLM generation.

What we learnedA. The previous system was a five-stage clustering pipeline, and errors accumulated at each stage

  • The previous system was a five-stage clustering pipeline verified.
    • It extracted keywords from search queries and the annotations of saved Pins.
    • It embedded the keywords and clustered them hierarchically.
    • It used the top keyword of each cluster as the name.
    • It ranked with a pointwise model (a model that scores each candidate separately) and a similarity penalty.
    • Finally, it used heuristics to distinguish evergreen from situational journeys.
  • Journey notifications from this pipeline achieved email click-through +88%, push open +32%, and positive survey +23% over interest-based notifications verified.
  • The problem was that errors accumulated at each stage verified. The authors list three kinds.
    • fragmentation: one kitchen project splits into three journeys: cabinets, countertops, and backsplash.
    • generic bucket: when a cluster is messy, it falls back to a generic name such as "Art" or "Shopping".
    • incidental noise: a few clicks on memes or celebrities take the top spots.

What we learnedB. The new system gets a list of journey names from one generation, and the prompt replaces the existing ranking and filter modules

  • The new system is a single generation that takes the profile and a chronological activity log and returns a list of journey names as JSON verified. Keyword extraction, embedding, clustering, and the naming model are all gone.

  • The input contains up to 150 events within a 360-day lookback verified.

    • Actions were reduced to six: search, save, like, click, comment, and dislike.
    • The prompt states the order search/save > comment > like > click > dislike explicitly.
    • Dislike is included deliberately so that journeys supported mainly by dislikes are suppressed.

  • Decoding sets temperature to 0 and output to at most 300 tokens, and turns off reasoning tokens verified. This is to reproduce the same answer for the same log.

  • The output is up to about 12 names of 1–5 words each, written in the user's language verified.

    • The ranking criteria are engagement strength, repeated evidence, recency, consistency, and specificity.
    • An empty list is allowed when there is no evidence. Inventing journeys without evidence is not allowed.

  • The prompt replaced the existing ranking and filter modules verified.

    • The diversity instruction in the prompt replaces the diversifier based on a similarity penalty.
    • The dislike rule replaces the hand-built negative-signal filter.
    • The authors therefore write that they treat a prompt change as a "launch".

    "once you collapse the pipeline, the prompt becomes the ranking policy and the safety policy. Everything the staged system encoded in model weights and thresholds now has to be clearly stated." — From Activity to Intent (blog), 2026-10-09

  • The clustering system was not removed and was kept as a fallback verified.

What we learnedC. A frontier model served as teacher for distillation, and Qwen3 4B became the production student

  • The authors distilled into a Qwen3 student using a frontier model as the teacher verified.
    • The teacher built synthetic training data by producing target journeys for each activity history.
    • With this data they trained Qwen3 0.6B, 1.7B, 4B, and 8B using SFT and LoRA (fine-tuning that updates only a small adapter instead of all parameters).
    • The name of the teacher model was not disclosed.
  • Quality rose quickly over the first few thousand examples and barely rose after that verified. Selecting hard consolidation cases and multilingual cases worked better than adding more examples of the same kind.
  • The 4B model became the production student on the balance between quality and throughput verified.

What we learnedD. A batch job runs on the online inference stack and skips recently inferred users

  • Serving is batch but uses the online inference stack verified. vLLM decode workers on L40S GPUs sit behind NVIDIA Dynamo, with prefix caching and chunked prefill turned on.

  • The authors write that KV-cache-aware routing matters because every request shares a long, fixed system prompt verified.

  • In production, a cluster of about 100 GPUs handles about 775 requests per second end-to-end verified. Latency is about 1.2 seconds at the median and about 3.0 seconds at p95.

    "In production, a cluster on the order of 100 GPUs sustains about 775 requests per second end-to-end, with a median around 1.2 seconds and p95 around 3.0 seconds." — From Activity to Intent (blog), 2026-10-09

  • When the algorithm changes, the whole population is rerun; normally only recently active users are run daily as an increment verified.

  • The measure with the largest effect on throughput was not adding GPUs but skipping users who had been inferred within the last few days verified.

What we learnedE. Journey names retrieve Pins, and the existing Pin ranking models rank them

  • A journey name becomes a retrieval query in the notification pipeline verified. Inference results are written to storage, and notification serving reads them at send time.

  • Each journey name becomes the query of embedding-based retrieval (an approach that finds items near the query in vector space) and fetches Pins verified. The existing Pin ranking models rank the fetched Pins.

    "each journey name then becomes a retrieval query. We built embedding-based retrieval models to pull Pins for each user journey which are further ranked by our Pin ranking models." — From Activity to Intent (blog), 2026-10-09

  • Journeys that were notified recently are lowered to account for fatigue, and journeys the user explicitly rejected are suppressed outright verified. After that, the notification framework decides whether to send and through which channel.

What we learnedF. Sensitive topics are blocked by filters and prompt instructions, and minors in some regions were excluded

  • Inference of sensitive attributes is blocked in several layers verified.
    • Both the input keywords and the output names must pass the notification content filter. The output also goes through a generative content safety filter.
    • The prompt instructs the model not to infer topics related to health, politics, religion, sexual orientation, gender identity, or race.
    • Users under 18 in the EEA, Switzerland, and the UK were excluded because of local law.

What we learnedG. In an A/B test with the clustering system as control, email click-through rose 1.1% and push open rose 1.3%

  • The control of the online A/B test is the clustering system verified.

    • The first experiment targeted users in the US and Canada, and email click-through rose 1.1% and push open rose 1.3%.
    • Statistically significant improvements also appeared in several sitewide engagement and retention metrics.
    • The subsequent multilingual international rollout went in the same direction, and shipping the two experiments increased the platform's DAU, WAU, and MAU.

    "Initial experiment ramped to the US and Canada users, with the clustering production system as control, showed lifts of 1.1% on email click-through and 1.3% on push opens, plus multiple statistically significant sitewide engagement and retention gains. A subsequent multilingual international rollout replicated the direction, and shipping both experiments increased the daily, weekly, and monthly active users on the platform." — From Activity to Intent (blog), 2026-10-09

  • The sizes of the sitewide metric and DAU/WAU/MAU increases were not published as figures verified.

  • Because the task has no ground-truth labels, offline evaluation was done in two ways verified.

    • They compared the outputs of the two systems for the same users qualitatively.
    • A separate language model scored relevance, evidence, diversity, and granularity.

What we learnedH. Four extensions, including Semantic ID input and feedback-based optimization, are being tested

  • The four items below are extensions under experiment, not production defaults verified. The authors state explicitly that this section is "trained or rolling out, not what every user gets today".
    • Semantic ID (SID, a token sequence that represents an item as a hierarchical code): Pins are compressed with RQ-VAE into 5 levels × 11 bits, packed into one 64-bit integer, and unpacked into special tokens when the prompt is built. The tokenizer is extended with the SID vocabulary, followed by continued pre-training and then SFT. If fewer than 10 events carry an SID, the whole prompt is converted back to tags, and search queries always keep the original text.
    • Offsite conversions (page visits, add to cart, sign-up, checkout) are added as an auxiliary signal only for users with little activity.
    • A single call generates the journey together with its sub-topics, adjacent topics, and even the notification title copy.
    • Preference optimization uses user feedback. An online A/B test is in progress.
      • The signals are the Yes/No answers to "Are you interested in <journey>?" in journey emails, and the saves, clicks, and hides within a notification session.
      • Chosen and rejected are built with a "feedback-as-edit" approach. The list that keeps Yes and removes No is chosen, and the opposite edit is rejected.

Canon update None. Pinterest is not a tracked service, so I folded it into radar.md. I added a Pinterest User Journeys entry under "new this month" and one line of evidence to the "spread of LLM-native ranking" thread.

Why it matters In the Pinterest journey system, the role of the LLM is to write the user's intent in batch as natural-language names, and those names become queries for the existing retrieval and ranking stack inferred. By sitting outside the synchronous serving path, it is on the same side as MetaStrategy in the radar inferred. The evidence is that both posts state that the LLM output is stored nearline or in batch and then read by a deterministic downstream stack. However, MetaStrategy's output is a strategy configuration, whereas Pinterest's output is a per-user query.

This post adds a new case to the radar thread on "where in the recommendation pipeline does the LLM go". The roles the LLM has taken in that thread so far were as follows:

  • Scoring function: Netflix GenRec
  • Generative retrieval: Yandex, Spotify GLIDE
  • Offline policy compiler: Alibaba MetaStrategy
  • Graph path reasoner: Meta ConnectionMind
  • Input tokenization: Meta AMBER
  • Embedding encoder: LinkedIn Follows

The flow of turning a user profile into LLM text already had Comcast and KuaFu. Comcast only ran an offline comparison, and KuaFu is a compressed-representation approach. Among these, Pinterest is the first case that uses natural-language output directly as the query and also publishes an online A/B test inferred (this is a comparison within the radar corpus, not a survey of the whole industry). The lift over clustering is about 1%, whereas the lift of clustering over interest-based notifications was 88%, so the large gain appears to have come from "introducing the journey as a unit", and the switch to an LLM reads as a quality improvement on top of it inferred.

There is not yet a link to the tracked services. This repository has no first-party statement that Meta, X, or Netflix generate user intent with an LLM on a notification surface inferred (this means there is no statement, not evidence that they do not operate one).

Radar

  • Pinterest User Journeys, 2026-10-09: Given an activity log, Qwen3-4B generates journey names, and those names become notification retrieval queries. It yielded email CTR +1.1% and push open +1.3% over clustering.

What to check next

Source check notes

No new source passed from X, Meta, Netflix, or arXiv, and one Pinterest post passed from the Monday industry blogs.

  • X: the latest commit of xai-org/x-algorithm is still cb45b55 (2026-10-09), and there are no commits after it. There is no release either.
  • Meta: the latest post on engineering.fb.com is "NTS: Authenticated Time at Meta" from 2026-10-06, which is unrelated to feeds. The latest post on ai.meta.com/blog is from 2026-07. Threads Dear Algo and Your Algo, which came up in the WebSearch cross-check, are already in services/threads.md.
  • Netflix: the latest post in the netflixtechblog.com RSS is from 2026-09-25, so nothing is new.
  • arXiv cs.IR: 30 items were parsed and 0 passed. There are two near misses.
    • 2610.10483 corrects the cluster-size and variance bias of two-level softmax sampling. The authors come from a music streaming research line, and there is no deployment claim.
    • 2610.10124 is a loss decomposition that feeds targets missed by retrieval into reranker training in generative recommendation. It is academic research.
  • Monday industry blogs: Pinterest passed with the one post above, and the 10-06 "Metrics Board" was skipped because it covers metric infrastructure. The sources that did not pass are Google Research, Spotify, and LinkedIn.
    • Google Research: the posts from 10-02–10-07 are unrelated to recommendation.
    • Spotify: the 10-08 post is a product announcement.
    • LinkedIn: the 10-09 post is about a data context layer.