Automating document work with Long-Horizon AI Agents at LlamaIndex
Jerry, CEO of LlamaIndex, argues that as AI agents move from answering questions to running long-lived workflows, documents become their most important context source. He walks through the evolution from simple RAG to agent harnesses, explains why PDFs are hard to parse, and presents the company's approach to OCR, extraction with grounding, and document search. He closes by outlining unsolved problems: agent-native document formats, in-place editing, versioning, distilling frontier workflows into cheaper models, and long-horizon background agents.
Document context as the missing layer for long-running agents
Jerry, co-founder and CEO of LlamaIndex, opens by reframing the company. Three years ago it was known as a RAG framework, starting as a popular open source toolkit around the time GPT-3 appeared. Its focus has since moved to building the best models and harness layer for document understanding, so that general-purpose agents can access the context locked inside documents. The company says it has processed over a billion pages, has more than 30 million monthly downloads across its open source packages, and serves a few hundred thousand users, including AI startups and large enterprises.
From stateless RAG to agent harnesses
He traces the arc of the field. The first wave of LLM applications was simple chat over your data: chunk a document, embed it, store it in a vector database, retrieve, and generate. The model was effectively a one-shot call with no tool use, no decomposition of questions, and no multi-step reasoning, so he calls it ‘pretty stupid’ by today’s standards. The chat interface has stayed roughly the same, but agents have moved from answering questions to automating end-to-end work. Today’s typical architecture is an agent harness, which is essentially an LLM looping on itself with a core set of abstractions: tool use standardized through MCP and skills, sub-agents, and code interpreters. Computer-use agents extend this by treating the whole computer as another tool.
This shift changes where the engineering effort goes. Retrieval strategy is now largely handled by the harness, which can pick keywords for even a simple BM25 or grep tool. Developers spend less time hand-coding pipelines and more time specifying tasks in natural language, writing runbooks, defining evals, and setting guardrails. He anticipates a further move toward goal-level specification, where agents self-learn implementation details and humans mainly define metrics and constraints.
Context as the bottleneck
If the model is a blank slate that transforms input tokens into output tokens, then the main lever is controlling what goes into those tokens. He lists several kinds of context: task context (prompts, goals, questions), external interfaces (MCP and skills), web search, structured data warehouses such as Snowflake or Databricks, and document context. Documents are the focus of his company. He estimates that roughly 90% of unstructured enterprise data sits in PDFs, slide decks, and Word files, spread across file systems, SharePoint, and Box. He argues that more documents are now read by agents than by people, which suggests a growing explosion in both the volume of documents produced and consumed.
Why parsing documents is hard
The talk argues that an agent cannot natively read a PDF because PDFs are built for display, not machine consumption. A PDF stores glyphs and shapes at coordinates. It has no concept of a table, a paragraph, or a reading order, so even when the text is digitized, it may not be stored in a usable sequence. OCR has existed for about 20 years, and earlier solutions relied on traditional ML models trained per document template, such as a specific invoice or tax form. Vision-language models changed this by offering general models that can read a screenshot of nearly any document without task-specific training.
LlamaIndex combines VLM capabilities with classical document-parsing heuristics. Its approach uses post-trained specialized models, including ones for tables, charts, and handwriting, together with an agentic harness that routes between frontier models and its own models. He is candid that frontier models are expensive at scale. Running Gemini or Claude over a billion PDFs would be prohibitively costly, so task-specific, post-trained open-weight models offer a better blend of accuracy and cost. He also notes that the company customizes a PDF engine fork to extract the signals it needs.
Benchmarks and the cost-accuracy frontier
Measuring parsing quality is hard because it depends on human interpretation. The team built an open source benchmark, published on Hugging Face and Kaggle, covering text, tables, graphics, diagrams, charts, strikethroughs, redlines, and bounding-box annotations. They have benchmarked about 120 solutions and rerun evals whenever a major model ships, such as the recent Gemini release. The benchmark serves two purposes: it is an external measure of progress and an internal target for hill-climbing their own models.
He describes two operating regimes. The high-accuracy regime, such as claims automation or extracting from financial data rooms, requires something close to 99.5% correctness and justifies paying more per page. The low-cost regime, such as indexing a 10-million-document SharePoint for RAG, can tolerate around 70% accuracy as long as retrieval finds the right document. For this second case the company offers a free, open source parser that uses no models at all.
Just-in-time OCR and two-stage document QA
He describes how tools like Claude Cowork or Codex handle an uploaded set of documents. A fast, cheap pass converts the files to plain text with a library such as pypdf. Keyword or grep search then locates relevant sections. If a region is complex, such as a large table or a chart, the system screenshots those pages and sends them to a VLM. Each stage has a clear role, and LlamaIndex aims to provide tooling for both: a light parser that beats pypdf and PyPDF, and VLM-based OCR for accurate offline or on-demand processing.
Extraction with grounding and calibration
Extraction takes a document and a user-provided schema and returns structured values. He argues that one-shot frontier-model extraction from complex documents misses and hallucinates values. The company instead uses a tuned agent harness with post-trained models that iteratively extracts each value, checks for drops and hallucinations, and attaches bounding boxes and citations back to the source. In his demo, the system caught almost every value, and he acknowledges that a few misses remain and that grounding is an area for improvement. The output also includes calibrated confidence scores. A corresponding extraction benchmark treats the task as a reasoning benchmark and compares one-shot frontier models with agent-harness setups built on Codex, Claude Code, and other models, plotting accuracy against cost per page.
Search: vectors versus keywords
He questions the default reliance on vector databases. Semantic search remains strong for broad, imprecise queries at large scale, but in many domains, including code and targeted questions, BM25 and keyword search work well. The key point is that capable agents can reason about tool inputs, so simple tools with good agents often perform as well as complex retrieval stacks. LlamaIndex provides a toolbox of keyword, semantic, and other search operations over document collections.
Open problems
He closes with several unsolved gaps. First, agent-native document formats: Markdown is readable and easy to diff but visually limited, and HTML is interactive but hard to edit reliably. The ideal format would be easy for both humans and agents to review and edit, with versioning and permissions. Second, document editing: form-filling a PDF without AcroForm fields, or editing a Word or PDF file in place while preserving the original style, rather than overlaying arbitrary text. Third, versioning and checkpointing, so that agent edits can be rolled back when they diverge from human intent.
Fourth is what he calls hill-climbing as a service, or distillation. The pattern is to run a task end to end with a frontier model or coding agent, define the target quality through benchmarks, and then distill that workflow into a cheaper open-weight model, post-trained model, or optimized harness. He says he was once skeptical of this approach but is now warming to it, since coding agents are making post-training more accessible to non-researchers. The Jevons-style decision models he mentions fit the same optimization pattern, trading a small accuracy difference for much lower cost and latency on decision tasks.
Fifth is long-horizon agents. Many document workflows, such as contract redlining, negotiations, and loan underwriting, take weeks and involve multiple parties or external blockers. He sees a trend toward persistent, always-on agents that accumulate context and run continuously rather than restarting each session. Even as agents become more autonomous, he maintains that context will remain the central constraint, and that unlocking large volumes of document data is the core capability these agents will need.