Grep or Embeddings? Agentic Search Over Company Documents — George He, LlamaIndex
George He, head of engineering at LlamaIndex, explores when agents need pre-indexed vector search and when plain file traversal with grep is enough. He contrasts Claude Code's embedding-free approach on code with the harder reality of large, messy, multimodal, permissioned company documents, and proposes a harness of hybrid retrieval, metadata filtering, grep, and reading tools backed by LlamaParse and LiteParse. The talk closes with a live demo of an agent building a cash-flow table from 135 Alphabet filings, grounded in source files.
Local file traversal versus pre-indexed retrieval
George He, head of engineering at LlamaIndex, opens with a question he wants the audience to sit with: when and why do agents need vector context, as opposed to just traversing files? His answer starts with an admission that models have gotten much better at orchestration, instruction following, and working with file systems, skills, and MCP servers. Still, large corpora and complex documents keep pushing teams toward some form of pre-indexing. He frames the choice as two evolving approaches: local file search and execution (grep, glob, listing, reading) versus embeddings and pre-retrieval indexing. The talk is less about picking a winner and more about knowing when to use one, the other, or a hybrid of both.
Why Claude Code skipped embeddings
He uses Claude Code as the case study. According to him, Boris, the Claude Code creator, saw the team experiment with a vector database and decide against it. The reasons he highlights are practical scaling and maintenance rather than raw retrieval performance. Keeping a vector index in sync with constantly changing code is hard. Anthropic’s answer was to rely on the local filesystem with glob and grep, and this works because of several properties of code. Code is small, semi-structured, and already text. It also ’leaves breadcrumbs’ in the form of imports, tests, and references, which give an agent a natural hierarchy to walk. Because the code is already on disk, adding a retrieval layer buys little.
Why company data breaks that model
Company data differs from a code repository on almost every axis. It is huge, messy, and multimodal, mixing PDFs, slides, images, scanned documents, and manufacturing schematics. It has no folder structure an agent can grip, and it is not curated the way well-formed code is. Permissions matter far more, since who can see what is a first-class concern. Scale changes the calculus. His rough rule of thumb is that with a hundred or perhaps a thousand files, local traversal is fine because the agent can hold the file names in context without burning many tokens. Beyond that, especially at thousands or millions of files, traversal alone is too expensive. You need a compass that points roughly toward the right region of the corpus, a softened version of finding a needle in a haystack.
He also pushes back on the idea that bigger context windows solve the problem. Context has grown from 100k to 200k to a million tokens, but the real question is what you are willing to pay. Sending binary documents through a model’s ingestion pipeline, or having the model render and extract data from them on every query, is extremely costly. Sub-agents do not clean this up either. They often use cheaper models, but their environments are still token-heavy and latency-heavy, and documents get reprocessed each time. That is his argument for pre-indexing: do the expensive work once and avoid paying for it repeatedly.
Let the agent choose: a harness of five tools
Rather than hard-coding a decision between RAG and file traversal, He argues for a harness that gives the agent tools and lets it decide. The core idea is that semantic search down-scopes the problem, while traversal and reading ground the answer in source files, reducing hallucination. He lays out five tools. The first is hybrid search, which acts as the compass and answers ‘where should I look?’ The next three are listing files, filtering by metadata, and grep, which together let the agent expand and narrow within the subcorpus. The fifth is reading complex documents, which requires both parsed text and rendered page screenshots.
On hybrid retrieval, he recommends it for enterprise use cases. Keyword search such as BM25 still has its place for developers and for agents hunting for specific terms, while semantic similarity captures meaning. Fusing keyword and vector scores, weighted appropriately, consistently beats either alone on benchmarks. Re-ranking the top 100 results with an LLM or a deeper model improves things further. He also mentions graph RAG and traditional RAG as options in the same space.
The traversal tools handle the rest. Listing and metadata filtering let an agent discover files that did not appear in the first search and then narrow down. Grep can confirm that multiple keywords co-occur, which makes a particular document more desirable. A read operation with an offset and a maximum length lets the agent pull a specific window out of a remote file without downloading the whole thing. He notes that these primitives matter most in shared, distributed file systems that you do not want to stream into context on every call.
Parsing complex documents and schematics
He argues that chunking applies not only to text but to rendering. Charts, scans, and schematics are not human-readable lines of text. They are a different representation, and they break traditional indexing and chunking workflows. His recommendation is to pre-parse documents with a structured parsing flow so the agent does not have to reread binary content each time. LlamaParse is the hosted product, and LiteParse is the open-source tool for extracting structure such as tables, charts, and segmented content into markdown or into formats that preserve spatial layout. Spatial reasoning is essential for multimodal data.
He also recommends persisting page screenshots as they are generated during rendering. Text and visual content are often intertwined, and an agent benefits from seeing both. For evaluating parse quality, he suggests asking practical questions of OCR pipelines such as Textract or Tesseract. Are tables captured as tables? How faithful is the output to the actual layout? His warning is that errors compound. An LLM that cannot understand location and spatial structure may look fine early on but falls apart later in the pipeline.
Production concerns: multi-tenancy, permissions, freshness
Once real customers depend on the system, several questions always come up. Multi-tenancy asks whether one user can see another user’s data. Permissioning asks how metadata from sources like Google Drive or SharePoint gets pulled in and kept current without grinding the system to a halt or reindexing everything. Freshness is the hardest part, and it is the reason he thinks many companies have given up on pre-indexing. Keeping complex documents and their metadata up to date is difficult.
He breaks synchronization into three steps: ingest, parse, and index. Ingestion brings documents into a synchronized system. Parsing converts them into a standardized format, usually markdown or text, though other formats can serve purely multimodal representations. Indexing chunks the content and exports it into a pre-index state for the tools to use. On storage, he recommends persisting data to disk with a faster memory layer on top. He says LlamaIndex has operated this at the multi-tenant level across millions to tens of millions of documents. Vector stores such as turbopuffer can handle permissioning, but only if metadata is tagged at ingestion time, which pushes filtering down into the storage layer.
Demo: a cash-flow table from Alphabet filings
The demo corpus contains 135 files derived from Alphabet’s financial reports, a mix of HTML and PDF documents. Metadata such as company and year is attached so the agent can filter down. He runs a balanced hybrid search for cash flow statements. The results show chunks alongside the page screenshots they came from, which lets the agent zero in on a chunk and request surrounding context in a cleaner way. He then issues a grep for a regex pattern and reads a window from a given offset, for example 500 characters in and the next thousand, to show that remote storage can be treated almost like a local filesystem without downloading everything.
The chat request asks for a cash-flow table from 2021 to 2025. As the agent works, it searches the corpus, reviews screenshots and context, re-scopes its queries, and reads specific files. The final answer draws on multiple documents, and He emphasizes that the process matters as much as the output. The agent can check its own findings against source files, which gives more confidence that it found the right component.
Takeaway
His closing line sums up the argument: ‘search helps you find information and the harness gives you the results that you can trust.’ The message is that retrieval quality for enterprise document collections depends less on any single technique and more on giving the agent a well-designed set of primitives to combine.