Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa
Ryan Ahern of Exa explains how version-controlled search through Exa Snapshot lets you view the web as it existed on a past date, enabling backtests of AI agents while avoiding look-ahead bias. A hands-on acquisition-prediction exercise illustrates feature tuning and the dangers of overfitting without holdout sets.
Ryan Ahern, a full stack engineer at Exa, opens with a question that frames the whole talk: how do you know whether your agent would have made the right call? His answer is version-controlled search, which lets you see the web as it existed on a past date and then backtest an agent’s predictions against what actually happened. He organizes the talk into three parts: why agentic web search matters, why version control matters, and a hands-on exercise.
Why AI is becoming the main searcher
Ryan’s starting observation is that AI systems now search far more than humans do. Humans tend to type a few keywords and look at page one, while agents keep going, issuing multiple queries, paging through results, and refining until they find what they need. Exa provides search for several large coding agents and agentic applications. Cursor, for example, uses it to fetch documentation it lacks, and Gamma uses it to pull historical financial figures. He also places Exa within finance, where hedge funds, sovereign wealth funds, asset managers, and VCs use it for alpha generation, risk forecasting, and early startup discovery.
Quality, cost, and latency
Ryan frames Exa’s value around a triangle of quality, cost, and latency. On quality, Exa built its own continuously refreshed benchmark, because a static public benchmark gets published, and search providers and labs then train against it. His argument is that users should build benchmarks for their own use cases and keep them current, since the reason to use a search provider instead of answering from model weights is access to the latest information. On cost, he notes that search results consume context-window tokens, so Exa has developed ways of returning only the tokens an agent needs across all results, reducing input-token spend. On latency, responses can come back in under 300 milliseconds, while deeper research tasks can run for around 20 minutes. He cited an evaluation example in which a long-running task found 3,212 video game publishers and the developers they work with.
Versioning the whole web
The new capability is called Exa Snapshot. Exa’s index covers roughly 100 billion documents and trillions of URLs. Whenever it discovers a new version of a page, or encounters a page on a given date, it stores that version. Two modes are offered. Content versioning takes a known URL and a date and returns the version served on or nearest before that date. Version search runs a full search as it would have been executed at a past date.
Ryan illustrates with two examples. The first concerns OpenAI’s annualized revenue, where figures around 70 billion and 50 billion were debated depending on whether revenue was counted gross or net, a distinction that could itself be a valuable trading signal. The second concerns adverse media around Ed Sheeran, where earlier context could inform counterparty due diligence, KYC, or KYB processes in insurance or partnerships.
Avoiding look-ahead bias
The central methodological point is look-ahead bias. A live search is kept current, which is exactly what you want in production, but when it is used to evaluate past predictions it leaks the future. A system that reads news saying a company was acquired will appear to have a perfect true positive rate, which tells you nothing about its predictive power. By setting the snapshot date just before an outcome is announced, you constrain what information can inform the system. You can then tune which features, such as hiring signals, earnings call analysis, or CEO podcast interviews, actually improve predictions.
Patterns for backtesting
Ryan describes three patterns. In the first, a traditional model uses snapshot only to gather historical features that cannot be reconstructed from live pages. Pages often don’t retain versions, and publish dates are unreliable because articles are updated for SEO. Exa is not involved in the prediction itself here, but the approach is still a valid way to backtest.
In the second pattern, an agentic feature is added to a traditional model. The live system uses search to turn unstructured web content into a structured signal, which then feeds a conventional ML classifier. Each agentic feature can be backtested and tuned one at a time.
In the third pattern, the whole system is agentic. The model decides at runtime when to search and combines internal and external knowledge. Using a coding example, Ryan suggests freezing the environment just before a pull request was resolved in an external repository. If the model then resolves the task, you can be more confident that the gain comes from your model improvements rather than from future information leaking in.
Hands-on: predicting acquisitions
Attendees were given about 30 minutes to find the best one or two yes/no features that predict a company will be acquired. Snapshot backtests each feature at 1, 7, and 30 days before an acquisition, and against random dates for companies that were not acquired. Scoring awarded +1 for a correct prediction, -1 for a false alarm, and no penalty for misses.
Ryan demonstrated a feature asking whether a company had lost its CEO or CFO or appointed interim leadership. It correctly flagged Delivery Hero, which was acquired, but also flagged Qiagen, which was not, a confounding case. Adding a feature for having been reported to explore a sale improved detection but also produced more false alarms. Combining the two features with AND rather than OR raised the score, because both conditions had to hold on the same day before the system predicted an acquisition.
Results and overfitting
The top winners illustrated the risk of overfitting. The first-place feature was an overly specific exclusion list that effectively named the companies in the test set. Ryan jokingly chose to treat this as a feature rather than a bug. He admitted that omitting a holdout set kept the exercise simple, but the result showed exactly why one is needed. Once you know the evaluation set, you can tune to it. With versioned search, you can instead build fresh holdouts, such as companies with known outcomes that were not used during tuning, to check performance beyond the visible data. Other winning features combined conditions such as an active investor publicly calling for a sale, or a share price at least 60% below its five-year high.
Takeaways
Ryan’s recap has three points. Exa is strong agentic search that is cost-effective and fast enough for production. Snapshot lets you measure whether Exa actually improves your model, rather than relying on intuition. Any competition or evaluation should include a holdout set to avoid overfitting. Enterprise access is available by contacting him directly.