Building an Agentic Platform — Ben Kus, CTO Box

Box’s journey into generative AI began with a common enterprise challenge: extracting structured data from the 90% of enterprise data that is unstructured. Historically, this was the domain of brittle, specialized ML models that were difficult to build and maintain.

The Initial Approach: Pure LLM Extraction

In 2023, with the advent of powerful Large Language Models (LLMs), the initial approach seemed straightforward and highly effective. The process involved:

1. Pre-processing a document, including Optical Character Recognition (OCR).
2. Making a single-shot call to a generic, off-the-shelf LLM with a prompt asking it to extract the desired fields.

This method was a significant leap forward. It was flexible, worked across various document types, and outperformed many specialized models from the past. For simple documents and a limited number of fields, the performance was “amazing.”

The “Despair Moment”: Hitting the Limits of Simplicity

This initial success quickly met its limits when confronted with the complexity of real-world enterprise use cases. Several key challenges emerged:

• Document Complexity: The single-shot approach failed on long, complex files like a 300-page lease document with hundreds of distinct fields to extract. The models would “lose track,” similar to a human.
• Task Complexity: The system struggled with more nuanced tasks beyond simple extraction, such as performing risk assessments based on a document’s content.
• Fragile Pre-processing: OCR is an inherently difficult problem. Scanned documents with handwritten notes, crossed-out text, and various PDF formats would lead to OCR errors, which cascaded into poor extraction results.
• Accuracy and Confidence: LLMs do not provide reliable confidence scores. While implementing an “LLM as a judge” could flag potential inaccuracies, enterprise customers require high accuracy, not just a notification that the system is unsure.

This led to a “despair moment” where simply waiting for the next, more powerful model was not a viable strategy. The fragility of the architecture itself needed to be addressed.

The Solution: An Agentic-First Architecture

The team’s key insight was to re-architect the system around an agentic framework, even for a non-interactive, backend process like data extraction. An AI agent at Box is defined by its ability to use a directed graph to orchestrate a series of steps, equipped with:

• Instructions and objectives
• Access to a set of tools
• Memory and context
• A reasoning framework for planning and execution

This shift was initially met with internal skepticism, with suggestions to revert to older methods like fine-tuning or adding more regex-based post-processing. However, the agentic approach preserved the generality of using LLMs while adding the necessary robustness.

The agentic routine for data extraction follows a multi-step, reflective process:

1. Prepare and Group Fields: The agent intelligently groups related fields (e.g., all fields related to the “parties” in a contract) to be processed together, maintaining context.
2. Multi-Query Execution: It breaks down the extraction into multiple, targeted queries against the document rather than one massive request.
3. Verification and Tool Use: The agent uses tools to validate its findings. This can involve using multiple LLMs from different vendors and having them “vote” on the correct answer. For OCR’d documents, it might double-check an extraction by looking at an image of that section of the page.
4. Reflective Judgment: An “LLM as a judge” is used not just to score the final output, but to provide iterative feedback. It can tell the agent, “Here’s some feedback, keep trying,” allowing the system to self-correct before producing a final answer.

Advantages and Key Lessons

This architecture proved to be highly effective and extensible. The same framework that solved metadata extraction was later used to build Box’s “Deep Research” capability, which performs complex analysis across a corpus of documents by searching, outlining, and synthesizing information.

The key lessons learned from this journey are:

• Clean Architectural Abstraction: An agentic framework provides a clean separation between the “intelligent workflow” (the directed graph) and the underlying distributed systems needed for scale.
• Ease of Evolution: The system is easy to modify. If a new problem arises, a new node or tool can be added to the graph without a complete redesign. For example, to improve sloppy output, a final node was added to “summarize and format the result.”
• The Case Against Fine-Tuning: Box is currently “anti-fine-tuning” for their use cases. The operational overhead of fine-tuning and re-tuning models from multiple vendors (Gemini, OpenAI, Anthropic, Llama) for every new version is immense. They have found that sophisticated prompting and agentic structures yield better results and are more maintainable, especially since the next generation of base models often surpasses previous fine-tuned versions.
• Build Agentic Early: The primary takeaway is that if a problem seems solvable by a set of AI models, it is crucial to build an agentic architecture early. This foundation allows for the continuous and efficient integration of new capabilities and provides the robustness required for complex, enterprise-grade challenges.

• Tags:
• Agentic AI
• LLM
• Feature
• Enterprise AI
• System Design
• Metadata Extraction
• AI Architecture