How We Solved Agent Building — Andrew Qu, Vercel

Vercel’s Chief of Software, Andrew Qu, recounts the iterative development of an internal data science agent, a project initiated by identifying key pain points across Vercel’s departments. The most compelling need came from the data team, which was a bottleneck in answering routine product and customer inquiries, requiring data scientists to drop everything to write custom SQL queries. This led to a year-long “crazy experiment” aiming to put an agent on every desk, similar to Bill Gates’ vision for computers.

Evolution of the Data Science Agent (D0)

The journey began with several architectural iterations:

1. Version 1: The Mega Prompt (03:39)
   • Initial approach involved a single, large prompt incorporating the Snowflake schema and a user’s question.
   • SQL generation was promising, but execution and reporting were manual, requiring a data scientist to copy/paste and run the SQL.
   • This “harnees engineering” provided confidence in the models’ ability to generate valid SQL given structure.

2. Version 2: Chained Scoped Agents (04:51)

   • Recognizing the multi-step process of a data scientist (processing, semantic exploration, execution, reporting), the team broke down the task into a chain of narrowly scoped agents:
     • Query Agent: Understands the user’s question.
     • Planning Agent: Maps out the query strategy (e.g., using read entity YAML and search schemas tools).
     • Execution Agent: Runs the SQL.
     • Reporting Agent: Visualizes data and provides insights.
   • Each agent had a dedicated system prompt and tools, enabling an end-to-end question-to-answer loop without manual intervention. This improved efficiency but still faced limitations.

3. Version 3: Single Agent Managing Internal State (05:58)

   • The realization emerged that one agent with “mega context” and internal memory management was needed.
   • This agent could reflect on its actions, manage its state (planning, building, executing, reporting), and re-evaluate if errors occurred during execution or joining.
   • Despite confidence in this system, initial rollout to trusted users was met with the verdict: “it was awful.” The agent only cleared 30% of evals, highlighting a lack of scalability in manually mapping scenarios.

The File System Unlock and Claude Code Inspiration (07:17)

A major breakthrough came with the emergence of Claude Code and Opus 4.5:

• Observation: These models excelled at answering complex questions using a minimal toolset: list file, read file, run bash.
• Insight: The advantage was not a clever tool set, but a plain file system environment that agents are inherently good at navigating and manipulating. This allows for emergent behavior rather than prescriptive tool usage.
• Rebuild: The data science agent (D0) was rebuilt in a sandbox environment, dumping the entire semantic layer into it. The agent was given basic file system operations (bash, read file, write file) and a few Vercel-specific tools.
• Results: This led to a significant “leap,” doubling the evaluation score and dramatically improving performance.

Distilling Queries into Reusable Skills (10:14)

To further enhance the agent, Vercel introduced the concept of “skills”:

• Problem: Every new agent run starts from scratch, lacking pre-established context beyond the semantic layer.
• Solution: A recurring job distills common, successful queries into approximately 100 reusable “skills” (e.g., aggregations, product lookups, billing info).
• Impact: When an agent starts, it can leverage these skills, beginning with accumulated contextual knowledge instead of nothing, making it far more efficient. Vercel also developed “Skillsh” for discovering and running these skills.

Eve: The Next.js for Agents (12:04)

This journey culminated in the release of Eve, an agent framework designed to simplify agent development:

• Inspiration: Next.js’s file-system-defined infrastructure, where conventions (e.g., pages, api folders) dictate deployment.
• Eve’s Approach: Agents are built by defining folders for skills, tools, and channels, allowing the framework to automatically compose the agent.
• Architecture: Eve provides a runtime (durability, isolated execution, model calls, connections) and integrates with Vercel’s infrastructure (Workflows, Sandboxes, Connect) for seamless deployment, though it supports open-source adapters for various databases and services.
• Impact: The D0 agent was rewritten in Eve, showcasing a simplified file system structure and improved iteration. Early beta users, like Aura, saw significant success, building more effective agents with fewer steps and better insights compared to off-the-shelf solutions. Eve also offers out-of-the-box observability for agent runs.

The Case for Narrow, Company-Specific Agents (14:18)

Andrew Qu argues strongly for specialized agents over general ones:

• Company-Specific Knowledge: The true power of an agent lies in embedding deep, proprietary company knowledge (e.g., Vercel’s web-based specificities). Off-the-shelf agents, while a good starting point, cannot capture this nuance.
• Vercel’s Experience: Vercel now has about 20 “product-market fit” agents in use, ranging from marketing retrospectives and legal contract redlining to the data science agent.
• Productivity Gains: The data team is more productive, freed from routine queries to focus on performance improvements and new data sources.
• Future Vision: Agents can automate mundane tasks across HR, finance, and sales, making companies of all sizes more efficient. Eve is presented as the optimal tool to build these agents.

• Tags:
• AI Agents
• Prompt Engineering
• Feature
• Claude Code
• Tool Use
• Semantic Layer
• Software Architecture
• Agent Frameworks
• LLM Applications
• Vercel Eve
• Internal Tools
• Data Science Automation
• File System Agents
• Next.js for Agents