Build Hour: Prompt Caching

Prompt caching is a powerful technique for optimizing AI applications, enabling them to be faster and cheaper with no negative impact on intelligence. It functions as a form of compute reuse: when multiple requests share an identical prefix, the underlying key-value (KV) tensors from the attention mechanism are cached, avoiding redundant processing.

How Prompt Caching Works

• Mechanism: Caching is not on raw text but on the intermediate state computations (key-value tensors) within the transformer’s attention layers. This is why it has no effect on the model’s output.
• Activation Threshold: Caching begins automatically for prompts exceeding 1024 tokens. The cache is then managed in blocks of 128 tokens.
• Core Requirement: The most critical rule is the need for a contiguous prefix. The input tokens must be identical and in the exact same order as a previous request for a cache hit to occur. Even a single changed character or extra space will break the cache.
• Scope: Caching is multimodal (text, images, audio), automatic, and applies to both the Chat Completions and Responses APIs.
• Cache Lifetime: By default, the cache is ephemeral and held in-memory for 5-10 minutes. An extended 24-hour cache is available via an API parameter.

The Impact: Cost and Latency

Prompt caching offers substantial benefits, particularly for applications with long contexts.

• Cost Reduction: Discounts on cached tokens are significant and increase with newer model families.
  • GPT-4 family: 50% discount
  • GPT-4.5 family: 75% discount
  • GPT-5 family: 90% discount
  • Real-time audio models: ~99% discount
• Latency Improvement: For long prompts, caching dramatically reduces the time to first token. Latency becomes proportional to the length of the newly generated output rather than the entire input sequence. For very long prompts (e.g., 200,000 tokens), this can result in a 67% reduction in latency.

Developer Playbook: Maximizing Cache Hit Rate

To achieve the highest cache hit rates, developers should implement several key strategies.

1. Use the prompt_cache_key

This optional API parameter is crucial for influencing how requests are routed. OpenAI hashes the first 256 tokens of a prompt to select a compute engine. If you send many requests with the same prefix, they will be load-balanced across multiple engines, reducing cache hits.

• Function: The prompt_cache_key is combined with the prefix hash, allowing you to intentionally group related requests onto the same engine.
• Use Case (Branching Chat): If a chat application has different initial paths (e.g., “Support” vs. “Styling”) that share a common system prompt but diverge afterward, using a separate prompt_cache_key for each path ensures that their respective contexts are cached effectively on the correct engines.
• Use Case (Coding Assistants):
  • Per-User Key: Useful if a user frequently works across different codebases, preserving cache for that user’s session.
  • Per-Conversation Key: Scales better if users have many unrelated threads.
  • A coding customer saw their cache hit rate jump from 60% to 87% by implementing this parameter.

2. Strategic Context Engineering

Context engineering (dynamically curating the prompt) and prompt caching are inherently at odds. Modifying the context invalidates the cache. The goal is to balance the intelligence gains from a curated context against the cost of a cache miss.

• Truncation: Instead of truncating small amounts of context on every turn (which constantly breaks the cache), make larger cuts less frequently. The retention_ratio parameter in the Real-time API is an example of this, saving up to 70% on 30-minute sessions.
• Summarization (Compaction): For long-running agents, use compaction to summarize the conversation history. While this invalidates the cache at the point of compaction, it prevents the context from growing indefinitely. It’s often cheaper to take a one-time cache miss than to pay for an ever-increasing number of input tokens. OpenAI now offers both server-side and standalone compaction endpoints.

3. API and Endpoint Selection

• Responses API vs. Chat Completions: For reasoning models, always use the Responses API. These models generate hidden “chain-of-thought” tokens that are crucial for their reasoning process. The stateful Responses API persists these tokens between turns, preserving the cache. The stateless Chat Completions API drops them, breaking the cache on every turn. Switching can increase cache rates from 40% to 80% and improve intelligence.
• Flex Processing vs. Batch API: For asynchronous, latency-insensitive workloads, consider using service_tier="flex". It provides the same 50% discount as the Batch API but on a per-request basis, offering greater control and compatibility with prompt_cache_key and extended caching.

4. Advanced Techniques

• allowed_tools: If you have a large set of tools defined in your system prompt, changing which tools are available for a given turn will break the cache. The allowed_tools parameter lets you specify a subset of tools for the model to use on a specific request without modifying the prefix and invalidating the cache.
• Extended Prompt Caching: Use the prompt_cache_retention="24h" parameter to extend the cache lifetime from minutes to a full day. This offloads the cache from in-memory to GPU local storage. It’s ideal for keeping caches “warm” for applications that handle long, recurring contexts, such as coding assistants.

Customer Spotlight: Warp

Warp, an agentic development environment, leverages prompt caching to make its coding agents faster and more efficient. Their agentic loops naturally create long, incremental contexts perfect for caching.

• Static Prefixes: They ensure system prompts and tool definitions are static. Dynamic, user-specific content is placed in a separate context message after the static prefix to maximize the initial cache hit.
• Scoped Caching: They think of caching in layers:
  1. Global: System prompts and tools are cached across all users.
  2. User: User-specific configurations are cached across their parallel agent sessions.
  3. Task: The agent’s turn-by-turn conversation sees the highest reuse.
• Results: By implementing a task-scoped prompt_cache_key, Warp more than doubled their cache hit rate.

• Tags:
• Latency
• Context Management
• LLM Optimization
• Warp
• Prompt Caching
• Cost Reduction
• API Best Practices