Personalization in the Era of LLMs - Shivam Verma, Spotify
Spotify is personalizing open-weight LLMs without full fine-tuning by combining three key components: foundational user embeddings from streaming history, 'Semantic IDs' that tokenize its 100M+ item catalog, and a 'soft tokenization' layer that projects a user's embedding directly into the LLM's context. This allows the model to autoregressively generate the next song or podcast as the next token in a sequence.
Spotify is evolving its personalization strategy, moving away from traditional, siloed recommender systems towards a single, unified generative framework powered by Large Language Models (LLMs). This modern approach is built on three core pillars: foundational user modeling, deep catalog understanding, and a novel method for injecting user context into a frozen LLM at inference time.
1. Foundational User Modeling: Understanding Taste as a Sequence
The foundation of Spotify’s personalization is a deep understanding of its 750 million users. The team is shifting from older methods like autoencoders to sophisticated Transformer-based sequential models.
- Sequential Modeling: Instead of just compressing a user’s features into a static vector, the new approach treats a user’s entire listening history as a sequence. This allows the model to capture the nuances and evolution of a user’s taste over time.
- Shared Embedding Space: This Transformer model generates rich user embeddings that exist in the same vector space as content embeddings (for tracks and podcast episodes). A visualization of this space shows how a user’s embedding is located near the content they listen to, enabling “cross-content” recommendations and exploration of a user’s “neighborhood” of taste.
This user embedding serves as the primary context for all downstream personalization tasks.
2. Catalog Understanding: Teaching LLMs about 100M+ Items with Semantic IDs
A key challenge is teaching a general-purpose LLM, which possesses broad “world knowledge,” about Spotify’s specific and massive catalog of over 100 million tracks and millions of podcasts. The solution is a technique called Semantic IDs.
- From Vectors to Tokens: A Semantic ID is a compressed representation of a content item’s vector embedding. Using vector quantization, a high-dimensional vector (e.g., 1000+ dimensions) is converted into a short sequence of discrete tokens (e.g., four to six tokens).
- Hierarchical Representation: These tokens are hierarchical. For example, artists Ariana Grande and Bruno Mars might share their first two tokens, representing their common identity as “pop artists,” but the subsequent tokens will diverge to capture their unique stylistic niches.
- Recommendation as Generation: This tokenization reframes the recommendation task into a classic next-token prediction problem. The LLM can be trained to autoregressively generate the Semantic ID of the next song or episode a user is likely to enjoy, based on the sequence of their previous listening history (also represented as Semantic IDs).
This process involves continually training open-weight LLMs on Spotify’s data, adapting them to understand and “speak” the language of Semantic IDs.
3. Assembling the System: Personalized Generation with Soft Tokens
With a model for users (embeddings) and a language for content (Semantic IDs), the final piece is to combine them for real-time, personalized recommendations without fine-tuning the LLM for every single user. This is achieved through soft tokenization.
- The Challenge: The base LLM is trained on a vast but general dataset. It has collaborative filtering abilities but lacks specific knowledge of the user currently making a request.
- The Solution: At inference time, the user’s pre-computed embedding vector is projected directly into the LLM’s token embedding space. This creates a “soft token”—a virtual, user-specific token that the model can attend to.
- Personalized Prompting: This soft token is inserted into the prompt alongside the user’s recent listening history (as Semantic IDs). The LLM now has rich, specific context about the current user, allowing it to generate a recommendation that is highly personalized, even though the model’s core weights remain frozen.
The Future: A Steerable, Generative Recommendation Engine
This three-part system—user embeddings, Semantic IDs, and soft tokenization—forms a powerful, sequential modeling framework that is replacing traditional multi-stage recommender pipelines (candidate generation, ranking, etc.).
This new stack is already in production, powering next-episode recommendations for podcasts. It also enables new user-facing features like the “Taste Profile,” which allows users to see and edit what Spotify knows about their preferences, giving them direct steerability over future recommendations. The goal is to expand this unified generative model to all recommendation surfaces across Spotify.