Building Advanced Agents Over Complex Data // Jerry Liu

The Limitations of Naive RAG

A typical naive Retrieval-Augmented Generation (RAG) pipeline is easy to prototype but often fails to meet production-level quality. This basic approach usually involves simple parsing, naive splitting of documents (e.g., by paragraph or a fixed number of sentences), top-k dense retrieval, and stuffing the retrieved chunks into a prompt.

While this works for simple, fact-based questions over a small set of simple documents, it breaks down under more complex, real-world conditions. The primary failure modes include:

• Complex Data: Even simple questions fail when posed over documents containing tables, charts, images, or complex layouts.
• Large Data Volume: Performance degrades when querying over a large corpus of documents (hundreds, thousands, or millions).
• Complex Questions: Vague, multi-part, or analytical questions tend to break the system, leading to hallucinations or incorrect answers.

The top priority for developers should be improving response quality over a representative set of questions. This can be achieved by focusing on two main areas: improving data quality and implementing more sophisticated agentic architectures. This talk focuses on the former, as data quality is a critical and often underrated component.

The First Principle: Garbage In, Garbage Out

The age-old machine learning principle of “garbage in, garbage out” is just as true for LLM application development. Good data quality is a non-negotiable prerequisite for any production-grade LLM application. Without a clean, well-structured data processing layer, even the most powerful LLMs will struggle to provide high-quality responses. The core components of this layer are parsing, chunking, and indexing.

Advanced Data Processing for High-Quality RAG

1. Parsing Complex Documents

Many enterprise documents are complex, containing not just text but also embedded tables, charts, images with spatial layouts, and important metadata in headers and footers. Naive RAG indexing pipelines fail on these because standard parsers (like PyPDF) extract a messy, unstructured blend of text and numbers, leading to hallucinations.

For instance, when a financial table is poorly parsed, models like GPT-4o or Opus cannot correctly answer questions about specific values within it, even if the text is technically present.

To address this, specialized parsers like LlamaParse are designed to handle complex documents (PDFs, PowerPoints, etc.). They can properly extract and format tables, understand natural language parsing instructions, and even pull out images for multimodal RAG applications.

Good parsing alone can dramatically improve performance. When a document’s structure, like a train schedule table, is preserved in the text representation with proper spacing and alignment, modern LLMs can understand and query it far more effectively than if it were a jumbled block of text.

2. Hierarchical Indexing and Retrieval

Combining advanced parsing with more sophisticated indexing strategies is key. For documents with heterogeneous data (text, tables, images), a hierarchical indexing approach is superior to flat chunking.

The pipeline looks like this:

1. Parse: Break the document down into its constituent multimodal elements (e.g., text chunks, tables, images).
2. Represent: For each element, extract one or more text representations. For a table or image, this could be a detailed summary or multiple descriptions.
3. Index: Embed and index these text representations (the summaries), which then link back to the original, underlying object (the raw table or image).
4. Retrieve: Use a two-step “recursive retrieval” process. First, retrieve the relevant indexed summaries based on the user’s query. Then, follow the links to fetch the full source elements (the complete table, the image itself) to provide the LLM with the complete context.

This method ensures that the most relevant and complete information is surfaced, significantly reducing hallucinations on complex financial documents.

3. Chunking and Indexing Best Practices

Beyond parsing, several best practices have emerged for chunking and indexing:

• Chunking Strategy:

  • Page-level chunking often serves as a strong baseline for documents like PDFs and PowerPoints, as pages naturally contain semantically related information. It’s often more effective than worrying about optimal chunk size.
  • In the future, as long-context models become more efficient, document-level chunking will become more viable, further simplifying the process.
  • Always try to preserve semantic coherence. Avoid breaking tables, lists, or sections in the middle.

• Metadata Extraction: This is a powerful and underrated technique. Extracting rich metadata (e.g., section titles, dates, authors) creates a semi-structured representation of your data. This allows for more precise retrieval through filtering and enables a wider range of queries beyond just vector search.

• Indexing Multiple Representations: A single text chunk shouldn’t be limited to a single vector. Indexing multiple representations (e.g., a summary, key sentences, hypothetical questions) linked to the same underlying object allows the system to match a wider variety of user queries to the correct source data.

• Use a Document Store: A vector database alone is insufficient. A document store (essentially a key-value store) is crucial for:

  • Storing hierarchical information and the original source documents.
  • Enabling caching and incremental syncing by storing document hashes to track changes efficiently.
  • Storing conversation history and long-term memory for agents, potentially using knowledge graphs.

Future Challenges and Directions

Data processing for LLMs is still evolving, and several interesting challenges remain:

• Multimodality: As multimodal models improve, the native representation for a document chunk might become an image of the page rather than extracted text. This would natively capture diagrams, charts, and layout without information loss.
• Long Context Windows: While RAG will remain essential for very large document corpuses, long context windows will likely eliminate the need for fine-grained chunking decisions like setting the perfect chunk size.
• Unified Storage Systems: The field is moving beyond a singular focus on vector databases. The future requires unified storage systems that can handle multiple query types (vector, SQL, graph) and seamlessly manage unstructured, structured, and multimodal data in one place.

• Tags:
• LLM
• RAG
• Feature
• Vector Database
• Agents
• Data Quality
• LlamaIndex
• Parsing
• Indexing