Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

Kuba Rogut from Turbopuffer presented a benchmark comparing the effectiveness of different code search methods for Large Language Model (LLM) agents like Claude Code, specifically focusing on semantic code retrieval. He highlights that while Claude Code by default relies on agentic search (essentially grep), early versions did use local vector databases, which were later deprioritized for simpler grep-like approaches.

The Case for Semantic Search: Cursor’s Experience

Turbopuffer’s customer, Cursor, a prominent AI coding assistant, leverages semantic code search with Turbopuffer’s vector database. Cursor reports significant performance gains: a 24% relative improvement in answer accuracy (12.5-13% across all models before Composer 2) and a 2.6% increase in code retention in large codebases, alongside a 2.2% decrease in dissatisfied user requests from an online A/B test. These seemingly small percentages are significant because not all queries necessitate or benefit from semantic search.

Embeddings as Cached Compute

Rogut argues that “embeddings are cached compute.” Traditional grep-based agentic search requires repeated computation for every session and agent, even for the same codebase or question, leading to high token usage (e.g., 6,000 tokens for a single trace). In contrast, semantic search involves an upfront cost for chunking, embedding, and indexing the codebase. This creates a cache of semantic meaning, allowing agents to quickly query and retrieve relevant chunks with significant token savings over time, especially when running multiple agents.

TurboGrep: The Benchmarking Tool

To test this hypothesis, Turbopuffer developed TurboGrep, a simple CLI tool. It utilizes a tree splitter library to parse codebases, chunk them, embed them using Voyage’s code model, and upload them to Turbopuffer. A v1 version is open-source, with v2 forthcoming. A demonstration showed Claude Code calling the TPufferSearch tool to efficiently retrieve content for a password_reset_token_generator query in a Django repository.

ContextBench Methodology

Rogut used a modified version of Cursor’s internal ContextBench, a publicly available paper benchmark. This benchmark doesn’t just assess if an agent solves a problem but critically examines how it solves it—specifically, whether it identifies the correct files, lines, and symbols along the way. This relies on a human-labeled dataset indicating the golden context required for each task. Three conditions were tested:

1. Raw Claude Code: Out-of-the-box behavior.
2. Windowed Reads (50 lines max): Claude Code with limited file reading to reduce noise from overly long files.
3. Windowed Reads + Turbopuffer Search Tool: Combining the above with semantic search.

Benchmark Results: Precision and Recall

Precision

Precision measures the proportion of relevant files read out of all files the agent accessed. Rogut translated this into

• Tags:
• Feature
• Semantic Search
• Claude Code
• Vector Databases
• LLM Agents
• Turbopuffer
• Code Retrieval
• ContextBench
• Precision
• Recall
• Voyage AI