The RAG Mistake Almost Every Team Is Making (with Pete Johnson)

Pete Johnson, Field CTO of AI at MongoDB, shares insights from advising over 100 companies on their AI strategies, focusing on practical approaches to achieving AI ROI and optimizing AI systems for production.

The AI ROI Gap: Why Many Organizations Struggle

Johnson highlights a significant gap: 83% of organizations have AI steering committees and success metrics, yet only 19% see a return on investment. This disconnect stems from two main issues:

• Picking the Wrong Problems: Many companies fail to identify appropriate use cases where AI can deliver measurable value.
• Lack of Clear Metrics: Without established metrics, it’s impossible to quantify the impact of AI solutions.

Strategies for Success and ROI: Johnson observed that successful companies initially focus on employee-facing use cases with a “human in the loop.” This approach offers:

• Lower Data Security/Quality Bar: Internal data has less stringent requirements than customer-facing data.
• Existing Metrics: Employee performance metrics are already established, allowing for easy measurement of AI’s impact. Examples include improving call center efficiency (cost per call, call volume) or accelerating Software Delivery Life Cycles (SDL). The key is to measure business outcomes (e.g., shipping code faster) rather than vanity metrics (e.g., lines of code, token count).

Johnson cites the “token maxing” incidents at Uber and Meta as cautionary tales, demonstrating how vanity metrics can lead to inefficiencies and wasted resources. He emphasizes that AI amplifies existing data and security challenges rather than solving them.

Jevons Paradox and the Future of Work

Johnson applies Jevons Paradox (where increased efficiency of a resource leads to increased consumption) to the impact of AI on jobs, contrasting bank tellers with toll booth workers:

• Bank Tellers: When ATMs and direct deposit automated redundant tasks, bank tellers transitioned to higher-value roles like relationship banking and small business loans, leading to an increase in their numbers and bank branches. The job evolved to require more human interaction and value-added services.
• Toll Booth Workers: Automation (RFID, license plate readers) left no higher-value role for these workers, leading to job displacement.

Johnson argues that for skilled professionals in areas like software engineering and data science, AI is more akin to the bank teller scenario. It augments capabilities, making individuals more impactful and driving demand for their expertise, rather than replacing them.

From SQL to NoSQL: Database Evolution and Denormalization

Tracing the history of databases, Johnson explains E.F. Codd’s 1970 paper, which laid the foundation for SQL and normalization. In an era of expensive disk storage, normalization minimized data redundancy, but required multiple “joins” (disk reads) for data retrieval.

As disk storage became cheaper and retrieval speed more critical, denormalization (duplicating data for faster, single-read access) became viable. MongoDB’s NoSQL document model inherently supports this by storing data in flexible JSON structures, optimizing for fast retrieval and developer flexibility. This aligns with the shift in scarce resources: from disk space in 1970 to developer time and query speed today.

Vector Search: The New Frontier for AI Applications

Johnson identifies vector search as a critical component in modern AI application architectures, particularly for overcoming limitations of Generative LLMs:

• LLM Limitations: LLMs are typically trained on public data and have a knowledge cutoff, making them unsuitable for proprietary or real-time information.
• Retrieval Augmented Generation (RAG): RAG addresses these limitations by:
  1. Embedding proprietary documents into vectors and storing them in a vector database.
  2. At query time, embedding the query and performing a vector (similarity) search to retrieve relevant documents.
  3. Feeding these retrieved documents along with the query into the LLM’s context window, providing it with up-to-date and proprietary information without fine-tuning.

Common Mistakes in RAG: The biggest mistake is assuming the embedding model is a commodity. The choice of embedding model directly impacts retrieval quality.

MongoDB’s Approach to Vector Search: MongoDB acquired Voyage AI to enhance its embedding capabilities, with their models scoring up to 14% higher on retrieval benchmarks like RTB (Retrieval Embedding Benchmark).

Three Criteria for Good Vector Search: Johnson outlines:

1. Quality: Achieved through superior embedding models, ensuring accurate and relevant retrieval.
2. Storage Efficiency: Balancing vector dimension size with performance needs.
3. Developer Friction: Minimizing the effort for developers.

MongoDB Features for Developer Friction:

• Matryoshka Embeddings: Named after Russian nesting dolls, these ordered embeddings allow developers to test different vector dimensions (e.g., from 1024 to 512) by simply “lopping off” the end of larger embeddings, avoiding costly re-embedding of the entire corpus.
• Shared Embedding Spaces: Different sizes of Voyage models (e.g., Nano, Small, Medium, Large) generate compatible embeddings, allowing developers to use a local, free “Nano” model for development and then deploy with a larger model without re-embedding.
• Auto Embeddings: MongoDB automatically manages the embedding pipeline, updating embeddings and indexes as data changes.

MongoDB’s vector search integrates natively into its document model, treating vectors as just another array of floats within a JSON document. This allows it to leverage existing database features like replication, sharding, and security, while offering flexible design choices (e.g., storing data “by value” within the document or “by reference” as a pointer to external data).

Agentic Memory: The Next Frontier in AI Architecture

Agentic memory has evolved beyond simple short-term and long-term session tracking. Johnson describes a more sophisticated approach designed to combat token maxing and improve consistency:

1. An agent receives a query, sends it to a generative LLM, and stores the response in a rich agentic memory.
2. For subsequent similar queries, instead of immediately calling the generative LLM, the agent performs a vector search against its memory for candidate answers.
3. A lower-cost evaluative LLM (e.g., Quin, Llama) is used to determine if the retrieved answer is sufficient.
4. If the answer is good enough, it’s returned, avoiding an expensive generative LLM call and saving token costs.
5. If not, the query is passed to the generative LLM, and the process cycles.

This method not only reduces costs but also significantly improves the consistency of responses, a critical factor for business applications where probabilistic, varying LLM outputs are unacceptable.

• Tags:
• RAG
• Enterprise AI
• Vector Search
• LLM Limitations
• Jevons Paradox
• MongoDB
• Embedding Models
• AI ROI
• Agentic Memory
• NoSQL
• Matryoshka Embeddings
• Retrieval Quality
• Denormalization
• Software Delivery Life Cycle
• Productivity Metrics