Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI
Blackstone CTO John Stecher and Norm AI CTO Scott Worland discuss how to build AI agent systems that remain useful as models improve. They cover why data strategy and straight-through processing come first, how to weigh ROI against waiting for a better model, how evals anchor financial-services work, and why Norm rebuilt its platform after a large hand-labeled eval set exposed the cost of over-engineered code.
Data and workflow fundamentals come before agents
John Stecher argues that the most common failure point for companies going all in on AI agents is the data, not the models. As a large asset manager with proprietary, public, and third-party data woven together, Blackstone has focused on moving information out of transactional systems into a form agents can access, manage, and recall easily. Without a cohesive data strategy, he says, agent initiatives will fail.
He also argues that straight-through processing still matters, even though it gets overshadowed by enthusiasm for agents. His approach is to use coding agents to map how people actually perform workflows, then write deterministic code that optimizes those flows end to end. Agents come in afterward, injecting intelligence only where humans used to make judgment calls. Agents also take on the work engineers have long disliked, such as writing functional and performance tests, managing versioning, and upgrading libraries, which frees investment and finance professionals for higher-value work.
A heavily structured first generation
Norm was founded in 2023, when context windows were growing quickly but frontier progress briefly slowed. To get outputs that were accurate and consistent enough for regulated clients, the team built a heavily structured system: careful context assembly, sequenced model calls, consistency checks, and distillation into a final output. Scott Worland describes it as a large map-reduce problem with a lot of scaffolding, more workflow than agent.
The fortunate part was the evaluation set. To regression test that system, Norm had tens of thousands of expert hand-labeled examples. When agentic approaches matured, meaning an LLM given tools, a goal, and a way to verify itself against that goal, the team ran them against the benchmark right away. Many engineers realized they were staring at hundreds of thousands of lines of carefully written code that was largely obsolete, and in places actively in the way. The team then rebuilt the platform from a blank slate. The engineering leadership team rented a van, drove to upstate New York, and asked what they would build today knowing what they know now. The result was a set of specs for a platform that could deliver agentic experiences to clients and work alongside human engineers as AI co-developers.
Evals are the foundation in financial services
John stresses that in financial services the bar for accuracy is very high, and evals are how you meet it. Changes to model versions, harnesses, or underlying data all produce different answers, sometimes with skew in accuracy. A well-maintained eval set lets a team run tests consistently and measure the health of a system over time. Scott adds that Norm’s law firm works directly with lawyers to build benchmarks. Rather than a traditional roadmap of agent capabilities, the team uses a scorecard that defines success across dimensions. As the system improves, those scores should rise. They also found that removing upfront structure often produced better performance than the scaffolding they had built.
The bitter lesson and where to invest
Both speakers return to Rich Sutton’s bitter lesson: hand-crafted systems built by clever people tend to be overtaken by approaches that scale with compute. Scott traces the idea back to the early days of neural networks and a remark from an IBM translation lab leader about firing linguists. Norm’s founders were already aware of the trap in 2023 and tried to avoid it. Scott now sees the choice as a gradient rather than a binary one. Teams should invest heavily in durable foundations, such as data capture, state management for models, benchmarks, and agent primitives like the harness and sandbox. They should invest more conservatively in components that will change as the frontier moves. The goal is to build conduits, not dams, so that new intelligence from frontier labs can flow into the stack without being blocked by earlier design choices.
ROI, timing, and rapid prototyping
John weighs the ROI of each request against the true cost of building it today. If a model will be good enough and cheap enough in six months, waiting can be the better decision, and Blackstone has made that call before. But he warns against waiting too long. Current development tools let teams move from concept to a clickable proof of concept very quickly, without cybersecurity review or production concerns. He encourages engineers to build POCs, skip the PowerPoints, and think in terms of value produced rather than tokens consumed. Iterate quickly and throw away what isn’t needed.
Scott describes how Claude Code changed his thinking. The key was not the tool itself but the realization that an agent that writes code and executes it in a verifiable environment could be generalized across most knowledge work. That moment convinced him the ingredients for broadly capable agents were in place.
Production is still the hard part
John notes that building prototypes is now easy, but production remains hard. That means infrastructure that runs inside an enterprise’s environment, cybersecurity review, and supportability. Agent-written software must be traceable and feed into SRE platforms so it can be monitored in production. Norm keeps its core primitives under strict human review, since data security is as important as delivery speed when handling highly sensitive data. Teams have to resist getting ahead of their stack when a demo looks impressive but relies on capabilities they cannot yet support safely.
Rebuild instead of maintain
Ten years ago, building software took so much time, effort, and capital that the decision felt almost irreversible. John now sees many internal applications as disposable. Build them, let them serve a purpose at a point in time, and expect them to be replaced sooner than before. Engineers should constantly ask whether rewriting an application costs less than maintaining it. A decade ago a CFO would have considered that question absurd. Today it is routine, and new applications may create far more business value than the old ones.
John also sees AI bringing experienced engineers back into hands-on building. Engineers who have spent decades understanding system design can now express that knowledge through agents instead of being slowed by scaffolding.