How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver
A look at how Optiver, a proprietary trading firm, runs an on-prem AI platform where every model, agent, and harness decision has to show results in P&L within hours. Topics include the inference gateway, same-day adoption of frontier models, the meta harness design, and efforts to bring AI spend back under control.
Trading at speed, measured in P&L
Optiver is a market maker and proprietary trading firm. It trades its own capital, has no clients, makes more than 10 million trades a day across over a million instruments, processes petabytes of data daily, and operates from more than 10 locations worldwide. For AI work, the defining feature is the feedback loop. A change rolled out to traders shows its P&L impact within hours, because the balance sheet is reset at the start of each day. A consumer tech company, by contrast, measures engagement over days and weeks and revenue over quarters. The speed cuts both ways: a missed opportunity, such as a major market event like a SpaceX AI IPO, can be costly, so decisions have to be made quickly, measured efficiently, and taken with a bias for action. Trading also scales differently from a service business. The more it trades, the more it earns, while a consumer product competes for a finite amount of user attention.
Platform principles
The platform rests on four principles:
- Move fast while staying modular. New models, agent loops, swarms, and graph-based paradigms appear constantly, so promising approaches get measured and then adopted.
- Measure at every layer. External benchmarks matter less than internal ones, because the data distribution of Optiver’s tasks differs from what is publicly available.
- Offer self-service with good defaults. Traders are high-agency people, but unconstrained self-service becomes chaotic. AI engineers forward-deploy into trading teams, learn how they trade, and feed what they learn back into the platform.
- Keep everything on-prem. In a highly regulated industry where IP is valuable, Optiver runs its own GPUs and infrastructure. The only external calls are to third-party model providers, under zero data retention agreements. Noyan describes the company as AI native, with everyone using AI in different ways.
The stack
At the bottom, third-party models and self-hosted open-source models sit behind an inference gateway. The gateway handles routing, per-model cost tracking, budgets, and failover. Failover matters because any downtime directly means lost money. Above it, an agent platform lets agents register skills, plugins, and tools, and gives each team its own knowledge and memory systems. An agent gateway works as a single ledger of all agents in the company. It handles communication, orchestration of humans and agents, and shared identity management. The user-facing layer includes a web interface for sessions, an agent workbench for shareable persona management, and a company-wide personal assistant.
Shipping frontier models the same day
When a frontier model is released publicly, the goal is to use it in production flows that same day, provided it adds value. That requires a mature evaluation framework. Optiver partners with trading teams who contribute their own benchmarks, which become golden sets. Every new model runs against these sets immediately, and the results are shared so teams can judge whether the model fits their use case. Onboarding a model also means exposing it through the gateway with the right APIs, interfaces, and cost controls, and often updating the harnesses that wrap it. Because the company is global, support runs in rotations across regions to keep uptime high. Recommendations on how to use each model are communicated back to the company.
Staying model and harness agnostic
Optionality is the core idea. Any model that adds value gets integrated. The platform also has to withstand regional model availability problems and cloud provider outages. Some workloads must run in a fully isolated DMZ on-prem zone, for both cost and isolation reasons. Every model interaction goes through the inference gateway, is logged, and flows automatically into the evaluation system. Several harnesses are supported, and a meta harness layer lets each team define its own configuration and gain portability across harnesses. Team knowledge bases plug into this layer, and knowledge is harvested over time from traces and used to derive additional context. Agents can be deployed and managed on the platform, or accessed through the web portal and IDE.
Getting AI spend under control
Spend rose sharply in December with the Claude Code moment, then jumped again with the large frontier model releases over the summer. The response was optimization rather than restriction. A key finding was that on a fresh session, people often started with around 200k tokens of context by default. Self-service had led everyone to dump MCPs, knowledge bases, and documents into the prompt. The changes included:
- Adding MCP tool search, and replacing some MCP calls with a CLI that wraps several MCPs more efficiently.
- Examining cache invalidation behavior and whether cache parameters should be set statically or dynamically.
- Working with individual teams to move material out of massive system prompts and into knowledge bases.
These gains scale with token price, so they matter more as costs rise. Other measures included:
- Visibility and alerting. Showing teams what they spend and letting them set their own limits. Most teams chose reasonable ones.
- Open-source models. Many are competitive on token cost and serve as value replacements for some use cases.
- Model routing. Sending prompts dynamically to cheaper models, or having a frontier model plan while smaller models implement the steps.
- Knowledge base hygiene. Retrieving only the right amount of context is itself a major cost control.
The goal is not to spend fewer tokens for its own sake, but to get more return from the same tokens. Noyan argues that single-session optimization still leaves value on the table, and that work on routing and sub-agent efficiency continues.
Meeting users where they are
Before a formal solution existed, people built bespoke setups: different harnesses, dev boxes, and hacky web UIs. Optiver built one web portal to manage sessions, with chat windows and artifacts. Vendor products offer similar features, but the in-house version fits the infrastructure and can be iterated at the team’s own pace. Traders now contribute code, sending pull requests that the team merges. This has produced collaborative document editing and multiplayer rooms, where a user’s agents and other deployed agents share one workspace. A Slack client also exists. The underlying philosophy is that highly agentic users will build what they need if the platform doesn’t, so the team builds a better version first.
Build versus buy, evaluation, and agent design
On knowledge bases, frontier labs often rely on large hierarchical markdown files, while vector search has its own trade-offs. Optiver uses a hybrid approach and is interested in graph databases, but Noyan is candid that the current memory solution is not satisfying.
On building in-house rather than adopting tools like Codex, the main reason is that most vendor solutions assume managed infrastructure and try to lock teams into their model or stack. Codex only recently offered managed agents that run on a company’s own infrastructure, and Optiver is now working with them on it. Dependence on one vendor’s roadmap also cost time. One inference gateway provider handled urgent requests through its go-to-market process, which was too slow. Optiver now finds itself ahead of the curve, with frontier labs asking how it built its managed agent capabilities.
On evaluation, every tool call, input, and output is logged. The hard problem is replaying tool calls whose results depended on the state of the world at the time. A trading backtest database can be queried point-in-time easily, but a system like Slack cannot. The team assigns a confidence level to how accurately each tool invocation can be reconstructed, and uses a proxy around the MCPs to snapshot state. Noyan admits this remains imperfect, and that recording everything may be the only fully reliable answer.
On scaling agents, the temptation is to give one agent every capability. That bloats the agent, creates tech debt, and makes quality uneven, since some functions improve while others regress. The preferred approach is to scope each agent to a task it does well, version it, and let agents communicate through the agent gateway. Each agent stays measurable and easier to manage, and the problem becomes one of multi-agent orchestration.