Turbocharge Your Agent's Retrieval with TurboQuant - Shashi Jagtap, Superagentic AI

The talk, presented by Shashi Jagtap, founder of Superagentic AI, addresses the significant memory overhead in AI agents, particularly concerning the KV cache and embeddings. This issue leads to performance degradation as context grows, especially when running models on local devices where RAM is a shared resource. By default, embeddings are stored at 32-bit precision, far exceeding the 3-4 bits often needed for effective search, resulting in up to a five-fold waste of memory.

The Problem: Hidden RAM Bill in Agent Retrieval

Traditional approaches to mitigate memory issues—such as model quantization (to 4-bit, 8-bit), context compaction, using smaller embeddings, or CPU/disk offloading—often entail trade-offs in quality, speed, or require specialized hardware. The core problem is that embeddings consume excessive memory, with KV cache growing to potentially exceed the model size itself, leading to resource contention on devices like Macs with shared RAM pools.

Introducing TurboQuant: Training-Free Compression

TurboQuant, a method developed by Google Research and presented at ICLR 2026, offers a training-free compression technique to store embeddings and KV cache vectors at 3-4 bits instead of 32-bits without sacrificing search quality. The fundamental insight behind TurboQuant is that search primarily cares about which vectors are closest to a query, not their full precision representation.

How TurboQuant Works

TurboQuant operates through a multi-stage process, distillable into two or three key steps:

1. Data Evenness: It shuffles the vector data to even out its distribution.
2. Scalar Quantization: The shuffled data is then rounded into buckets using a technique called scalar quantization, effectively compressing the vector.
3. Error Correction (QJL) & Reranking: A crucial final stage, known as QJL (Quantized JPEG-Like), fixes any remaining errors with just one bit of information. This process, combined with a tiny reranking step, is highlighted as the “magic” that maintains search quality and accuracy despite aggressive compression. The reranking step is essential for preserving the integrity of the search results, ensuring that the most relevant items are still identified correctly after quantization.

The speaker emphasizes that while the underlying mathematics (e.g., Polar Quant, QJL) can be complex for non-ML specialists, the user experience is streamlined. Users only need to determine their desired bit budget (e.g., 3.5-4 bits is a common sweet spot), and the library handles the rest without requiring manual tuning.

Benefits and Applications

TurboQuant delivers significant benefits:

• Memory Reduction: Achieves up to a 5x reduction in memory usage for embeddings.
• Quality Preservation: Maintains retrieval accuracy, crucial for agent performance.
• Training-Free: No additional model training is required.
• Versatile Use: Applicable to both KV cache compression for inference and RAG (Retrieval Augmented Generation) vector stores.

Integration with Existing Stacks

TurboQuant is designed for seamless integration:

• Inference Engines: It is being adopted by popular local inference engines such as llama CPP, MLX, Ollama, and LM Studio, meaning users may soon benefit automatically.
• RAG Systems: For RAG, users can keep their existing agent frameworks and vector databases (e.g., Pinecone, Chroma DB, Lance DB, SurrealDB). The integration involves simply swapping the retrieval layer with a TurboQuant-enabled retriever.

Superagentic AI’s Turbo Agent

Superagentic AI has developed Turbo Agent, an open-source library that allows users to leverage TurboQuant for their agent’s retrieval without altering their core agent framework or vector database. The demonstration showed an agent achieving the same answers with an index memory 5x smaller (e.g., 1.6 KB vs. 8 KB for float 32 baseline) by using a TurboQuant retriever.

Other Compression Approaches

While TurboQuant focuses on 3-4 bits for both KV cache and vectors, other methods exist:

• Rabbit and You: Available in Milvus, uses 1-4 bits, strong for large static indexes but only supports vector search, not KV cache.
• F8 bit: Used in VLLM and SGLine, offering 8-bit compression.

Key Takeaways

The central message is to “change your mindset and only compress for ranking,” recognizing that search cares about proximity, not full precision. Developers are encouraged to:

1. Try TurboQuant by swapping out their current retriever.
2. Start small and benchmark recall and latency on their own data.
3. Explore the Turbo Agent GitHub repository for further information and practical examples with databases like Lance DB and SurrealDB.

• Tags:
• AI Agents
• RAG
• Vector Search
• Quantization
• KV Cache
• Machine Learning Infrastructure
• TurboQuant
• Embedding Compression
• Memory Optimization
• Google Research
• ICLR 2026