Your Company Should Train Its Own AI Models - Will Brown, Prime Intellect
Will Brown of Prime Intellect argues that organizations should train models on their own traces, feedback, and workflows so that agents learn institutional knowledge the way human coworkers do. The talk covers evals as test cases, decision models and routing, widget-style interfaces that turn human decisions into training data, and nightly training runs toward self-improving organizations.
The organization as a learner
Agents keep rediscovering the same things because most of what keeps a company running never gets written down. Much of an organization’s knowledge lives in people’s heads and in the spaces between emails, meeting notes, and who actually talks to whom. The gap between the official org chart and the real one, the friction points, and the unwritten rules of how change gets adopted are all forms of implicit knowledge. Individual employees often see what is broken but lack the vantage point to change it, and nobody is looking across the whole organization. Agents are well suited to that work: they can comb through unstructured text, surface recurring patterns, and point to workflows that waste time.
Context versus weights
Agents currently learn on the job mostly through in-context learning, usually by reading a skill file or exploring a codebase. That approach forces them to relearn things, burn input tokens, and reprocess large contexts every time a workflow runs. Cached input is becoming the bottleneck in many agent costs, and even cheaper caching still means repeatedly paying to re-read the same institutional context.
The proposed split is between code and data. Rules and skills behave more like code, while preferences, friction points, heuristics, and trends are data, and organizations run on large amounts of it. Training is how frontier models have improved: each generation gets better because it was trained more, not only because prompts got better. The open question is what belongs in context and what belongs in the weights, and the answer is that knowledge has to be earned before it moves into the weights.
The current route by which expertise reaches models is inefficient. A practitioner who knows how a set of enterprise tools is used may share that knowledge through an expert network, pass through a company that turns it into RL environments, get resold to a researcher who is not a domain expert, and finally enter a training pipeline. The person with the knowledge rarely supervises how the model is trained. The proposal is to close that gap so that the people who correct their coding agents day to day can teach models directly, and so that contextual preferences (“in this setting, this was not what I wanted, and I would have preferred this”) compound into durable learning rather than piling up in an AGENTS.md file that must be refactored constantly. Training suits this kind of data because its shape already resembles what a training pipeline consumes.
There are several layers of learning: what people do, what the harness holds in skills and memory, what lives in files and databases retrieved through tools, and the models’ own broad world knowledge. That last layer decays. Models with stale knowledge cutoffs must rebuild a world model in context each time, so organizations will want models that understand their own operations intuitively.
Evals as test cases
A practical path into this is to treat a model like a codebase: something with a lifecycle, to which test cases, features, and assets can be added. Software engineering already has mature practices for testing and maintaining code over time, and the same discipline can apply to models. Traces from everyday use, including prompts and feedback on model outputs, accumulate the raw material for evals. Writing evals by hand has historically been painful and organizations have resisted it, but agents are increasingly able to curate evals, especially when trained on best practices for building robust ones.
Routing and decision models
The interest in decision models and routing is driven partly by cost. Cloud spend can be very large, so teams want to know when a cheaper model will do. Routing belongs in the harness, yet it resembles a decision model. Tasks with clear right answers and a chance for a human or another agent to give feedback are well suited to this. A large and a small model can perform the same task, and a judge can determine whether the small model was good enough. Each such judgment becomes data that justifies routing more traffic to the small model. The same logic applies to decision models that learn to mimic how a particular person would judge a situation. Approving tool calls is a simple example, though organizations will also have bespoke rules and preferences that can be compounded into a decision model. This form of continual learning is one that many teams could adopt immediately.
More ambitious projects go further, turning production traces from high-volume use cases into rich environments that blend real and synthetic data and capture the complexity of the tasks long-running agents are meant to perform. Orchestration, tooling, and humans in the loop make this possible, and it is becoming more automated with each generation of models.
Widgets as training data
As models get better at writing code on the fly, dynamic interfaces become practical. Widget-style cards are a personalized, specific version of tool-call approval, and they can elicit human decisions directly. They also address a real problem with bot sprawl: a person may run twenty bots and lose track of which is doing what, and the industry oscillates between a single stream and many threads. The better pattern is to hand agents an idea, let them run, and have them surface a decision only when one is needed, through an interface built for that decision. That is easier on humans than parsing a long thread, and the human response is clean data with an explicit target. This sidesteps some of the reward hacking problems that come with less structured feedback, though those still need attention. The interface can sit at the orchestration layer or in the decision models, and it does not need to cover every model in the stack.
Economics and nightly training
One recent example is a company called Parainstrument, a friend of Prime Intellect that used their tooling to train a model for about $300K. The model is roughly Pareto-frontier for knowledge work such as spreadsheets and slides. That is an employee-scale investment, comparable to hiring a software engineer, and it can be spread over time as knowledge compounds. Demand for this kind of training is growing month over month as open models get strong enough to be specialized through an organization’s domain, harness, and environment.
A prototype of nightly continual learning illustrates the loop. Traces are collected, experiments are proposed, flagged issues are reviewed, and a training run starts overnight. Inference capacity that serves agents during the day can be repurposed for training at night, when employees are less active. This is a concrete route to continual learning in model weights.
The overall arc runs from discovering a workflow, to creating skills, building evals, and adding routing, to eventually capturing the economies of scale that come from company data. The result is an organization that improves itself over time.