How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs
Milind Sunil Shah of Brainbase Labs presents a six-step framework for improving back-office agents, illustrated with a Domino's project that tracks competitor pizza prices across websites and delivery apps. Accuracy rose from 41% to 97%, cost per source fell from $2 to $1, and run time dropped from 129 to 71 minutes, all through instructions, skills, and playbooks rather than a new model. The talk covers kickoff, ontology, benchmarks and evals, a narrow first agent, iteration, and handing off the full loop.
Starting from a concrete business question
Domino’s operates more than 20,000 stores across 90-plus markets, and its marketing office kept asking the same question: what does a pizza cost at the competitor down the street, on their delivery app, and on their website, with and without promotions? The deliverable was a spreadsheet listing the lowest price for a set of items. An analyst did this by hand, spending about two hours a day. Every step looks easy in isolation: open the competitor’s site, pick the right store, compare delivery and takeaway prices, find the applicable promotions, and record the cheapest price. The real risk is that one wrong step still produces a cell that looks finished but is wrong. The agent had to read the items, find the sites, interpret prices, fill the spreadsheet, hand it off for review, and flag anything it was unsure about.
Kickoff: agreeing on what a win looks like
The first step is to agree with the customer or end user on what the job is, what the human signs off on at the end, how long and how much it costs today, how accurate the agent must be, who judges the output, and what the agent must never do alone. For Domino’s, the answers produced a brief with a clear target: a faster, repeatable way to collect local competitor pricing without losing the judgment that edge cases require. The win was defined in numbers, with the highest accuracy, the lowest cost per source, and the shortest run time. Before any work began, the customer was not even sure the discount search could be done at all.
Ontology: defining every term
The second step is writing down every term the agent needs in order to avoid assumptions. That covers what to collect, where each answer comes from, where it goes, what each term means to the analysts doing the work by hand, and what the agent should do when it cannot resolve something. The core principle is that every term should have a definition, because anything left unwritten gets guessed, and guesses produce errors.
Brainbase keeps this knowledge in a memory layer that behaves like a Postgres database the agent can work with directly. A concrete example is item 14: a large pizza from brand B, delivered through app B. The agent browses the site and finds 24 struck through and 19 as the sale price. A rule was added to record the price the user actually pays at checkout, which is 19 and not the crossed-out 24. This rule came from a real failure. The system stores each observation in a price observations table, recording the item ID, the price seen, the full price, and the source. The output spreadsheet starts as placeholder item IDs, and the agent only has to look up each ID and fill in the price. When a cell is wrong, it points to the exact page and the rule that led to the choice.
Other rules came from traced failures. A large pizza is the item itself or a bundle containing it. Delivery is not the same as takeaway, and the agent had confused the two. Promotion math comes first: a 30% discount is applied before the price goes into the spreadsheet. If a price cannot be verified, it is logged as a gap, and the agent does not guess.
Benchmarks and evals
The next step is deciding what can be evaluated offline and what needs online checks. For offline evaluation, the best starting point is the customer’s past cases with known answers. Domino’s analysts had spreadsheets they had already completed, so those inputs could be rerun and compared against the final outputs. Recreating the historical environment was harder. Archive.org snapshots of competitor websites worked poorly for delivery apps, which do not serve data the same way. The fix was to store browser screenshots of everything the agent did from version one onward, which became the internal benchmark environment.
Scoring combines three methods. Deterministic code checks handle objective questions: whether every ID cell was replaced with a price, whether the discount map was computed correctly, and whether a price that differs more than 20% from the previous run was flagged. An LLM as a judge reviews the trace to confirm the right item was selected and the right promotion applied. An agent as a judge is more involved. When an output artifact needs verification, it takes multiple turns and tool calls to check it, and in this case it re-verified every flagged price. Analysts still review the final flags and investigate what went wrong when something fails. The guiding idea is to score as much as possible with these three methods.
Shipping a narrow first agent
The first version is meant to be measured, not good. It should cover the smallest slice worth scoring, so the team can learn how accurate it is, how long it runs, what it costs, and what it gets wrong. Scope expands only after a baseline is established. Domino’s version one reached 41% accuracy and cost $2 per source, and it ran for a long time. It was still useful as a baseline. Scope then grew step by step: one app and one item in one city, then burgers and takeaway options, then another app, and finally other websites. Starting narrow made it possible to find and fix problems one at a time.
Iteration: score, fix, ship
Iteration means going back through the earlier steps repeatedly: refining the ontology, the evals, and the benchmarks as failures appear. The most expensive and most frequent failures come first. Every fix should be weighed against its effect on run time and cost, and the team should decide whether that tradeoff is acceptable. Failures should also feed back into new benchmark cases and eval criteria. The practice is to change one thing at a time and to fix the instructions before changing the underlying model.
Two examples from Domino’s show this. In the first, the agent was given a link to a large pizza on brand B’s own site. It landed on a section that was already highlighted, clicked the item, took the price, and stopped. An analyst later found a $21 deal further down the same page that the agent had missed. The fix was a simple rule: review the entire page, not just the section the link points to. The second example was harder. On brand D’s promotions page, all the prices were inside images, not text. The agent went through 26 promo images and zoomed in 4x to 6x. It found the right card at minute 13 but kept zooming to verify it and got stuck in a loop until it was stopped manually at minute 15. Reviewing the trace showed that the HTML DOM, including image titles and alt text, would have answered the question much faster. The updated rules tell the agent to check the DOM first and to cap zoom levels. The result was a third as many image reads and a cost reduction of more than 50%. Some problems still need engineers to read traces and browser tool calls directly, even with automated checks in place.
Across these changes, the system went from version 1 at 41% accuracy and $2 per source with a 129-minute run time to version 11 at 97% accuracy, $1 per source, and 71 minutes per run. The improvements came from instructions, skills, and playbooks rather than from a new model.
Handing off the whole loop
The final step is handing off the entire loop, not just the agent. The team needs answers to a few questions in advance: who runs the agent, who reviews the outputs, who turns corrections into rules and benchmark cases, and who decides when a new version is ready. Where possible, the customer should be able to make small changes themselves. At Domino’s, the agent now runs on the customer’s schedule, daily or on demand through an email trigger. Analysts review only the exceptions. Each correction they make is automatically added as a benchmark case, and the Brainbase team stays available for anything beyond that. The deployment is now expanding to more markets and more areas. The framework has also worked on other problems with messy data sources, where the right answer has to be worked out rather than looked up.