CAG vs Long Context: How AI Models Use and Remember Information

Introduction: Alternatives to RAG

Large Language Models (LLMs) are limited to the knowledge within their training data. To reason over private or real-time information, they need access to external knowledge at inference time. While Retrieval-Augmented Generation (RAG) is a common solution using vector databases, Long Context and Cache-Augmented Generation (CAG) offer two alternative methods that build upon each other.

Long Context: The Brute-Force Approach

The simplest method is to use a long context window, which involves “stuffing” all relevant documents directly into the prompt along with the user’s query.

The Growth of Context Windows

Context window sizes have expanded dramatically, making this approach increasingly feasible:

• 2020 (GPT-3): 2,000 tokens (~10 pages of text).
• 2023 (GPT-4 Turbo): 128,000 tokens (~300 pages).
• 2024 (Google Gemini 1.5 Pro): 2 million tokens (~20 full-length novels).

This trend allows developers to potentially bypass complex retrieval pipelines entirely. The model receives all necessary information at once, eliminating the risk of a retrieval step pulling incorrect or incomplete documents.

Drawbacks of Long Context

Despite its simplicity, this method has significant disadvantages:

• Cost: Most LLM API pricing scales with the number of tokens. Sending hundreds of thousands of tokens with every single query can become prohibitively expensive.
• Latency: Processing a very large context takes more time, leading to slower responses.
• “Lost in the Middle” Effect: LLMs exhibit a performance degradation when trying to retrieve information buried in the middle of a long context. They attend more effectively to information at the beginning and end of the prompt.
• Redundancy: The most significant issue is that the model must re-process all the provided documents from scratch for every query. Ten questions about the same set of documents mean the model reads them ten separate times.

Cache-Augmented Generation (CAG): Read Once, Remember Forever

CAG addresses the redundancy of the long context approach with a simple but powerful idea: what if the model could read the documents just once and then remember them for subsequent queries?

The Role of the Key-Value (KV) Cache

Understanding CAG requires understanding the Key-Value (KV) Cache. During text processing, each layer of a transformer model computes key and value matrices. These matrices are essentially the model’s “working memory,” representing its encoded understanding of everything it has read so far. Normally, this KV cache is computed fresh for every request. CAG’s innovation is to compute it once and reuse it.

The Three Phases of CAG

1. Knowledge Preparation: The relevant documents (e.g., company policies, product documentation) are formatted to fit within the model’s context window.
2. Pre-computation: The model processes these documents once to generate the KV cache. This cache, containing the internal representation of the documents, is then saved and persisted (to disk or memory).
3. Inference: When a user query arrives, the model loads the pre-computed KV cache, appends the new question, and generates the answer.

Since the heavy computational work was done in the pre-computation phase, the inference step is significantly faster. Research has shown speedups of 10x on small datasets and up to 40x on larger ones compared to reprocessing the full context every time.

Limitations of CAG

• The entire knowledge base must still fit within the model’s context window.
• If the source documents change, the entire KV cache must be recomputed. This makes CAG best suited for stable knowledge bases where data does not change frequently, as the cost of constant re-caching would negate the latency savings.

Long Context vs. CAG: A Comparison

The primary difference lies in when the computation occurs.

Feature	Long Context	Cache-Augmented Generation (CAG)
Computation	Performed on every query.	Performed once during pre-computation.
Cost & Latency	High cost and latency on every inference.	First query is expensive; subsequent queries are fast and cheap.
Best Use Case	One-off queries, such as analyzing a single document.	Repeated queries against a stable knowledge base (e.g., an HR chatbot).
Setup	Simple, with nothing to manage.	Requires a pre-computation step and cache management.

Practical Application: Prompt Caching as “CAG as a Service”

Major LLM providers now offer Prompt Caching as a feature in their APIs, effectively turning CAG from a research concept into a practical tool for any developer.

With prompt caching, a developer sends a long system prompt containing all the documents. The provider handles the KV cache management behind the scenes. Subsequent API calls that use the same prompt prefix skip the expensive document processing step and leverage the cache. This comes with significant economic benefits, as cached reads can be discounted by as much as 90% compared to processing fresh tokens. This feature makes the power of CAG accessible without requiring developers to build and manage their own cache infrastructure.

• Tags:
• LLM
• AI Models
• Feature
• Long Context
• Prompt Caching
• CAG
• KV Cache