An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases
Tim Sweeney, principal engineer at Weights & Biases, demos ARIA, W&B's agent for automated ML research. ARIA launches and monitors GPU training jobs on Karpathy's autoresearch project, surfaces patterns across 200+ experiments, and builds reports and dashboards. The talk then covers how the team built ARIA with Weave observability, LLM judges, YAML-based tasks as unit tests, and nightly evals, and closes with practical tips for productionizing agents.
Opening and context
Tim Sweeney, principal engineer at Weights & Biases (W&B), opens by polling the room across three personas: ML researchers, applied engineers, and AI managers. He frames the talk in three parts: what ARIA does, how the team built it on W&B and CoreWeave, and practical advice for productionizing agents. The transcript spells the agent ‘Arya’, but it is ARIA throughout. Sweeney notes that W&B has been around for nine years, joined CoreWeave about a year ago, and is known for training, inference, and Weave, the observability layer that collects data on AI development workflows and makes it actionable.
ARIA inside the W&B workspace
The demo starts in a W&B project with over 200 training jobs, built on Andrej Karpathy’s autoresearch project. Sweeney chose it because it is a minimal LLM training codebase that is easy to improve iteratively. ARIA opens from a button in the workspace and presents a familiar chat interface. After a quick self-introduction, Sweeney points to a long-running chat in which ARIA has already run over 200 experiments. It downloaded the code, configured launch jobs and GPUs, and autonomously modified both the code and the hyperparameters.
For the live portion, Sweeney asks ARIA to run another batch of experiments aimed at finding the best model on stage. ARIA opts for hyperparameter changes rather than a risky architectural swing and executes the loop through a shell call. Behind the scenes, W&B Launch manages a queue connected to a Kubernetes cluster on GPUs. Sweeney shows live terminal output from that cluster and stresses that this is not a recording.
The demo then covers analysis. ARIA can summarize the highest-performing runs, which is useful for a new team member or for someone catching up after time off. When asked to find patterns, it reported that a new family of models emerged during the autoresearch process, that batch size is a high-leverage parameter, and that a promising architectural recipe had surfaced. Sweeney notes these are insights that would have taken him hours or days to find manually. ARIA can also generate W&B Reports, which Sweeney describes as ‘markdown on steroids’ with embedded plots. In one report it used the parameter importance chart to show correlations among training parameters. ARIA is also tuned to build workspaces and plots using W&B’s native chart types.
Recap and the vision for automated research
Sweeney positions ARIA as a data science companion that surfaces insights that become harder to find as experiments and teams grow. It also addresses the friction of reporting and plotting, since the team wants insights to become shareable visual communication. ARIA became available in the W&B iOS app this week, so users can steer hyperparameter sweeps from a phone.
Sweeney frames this as a step toward a fully automated end-to-end research platform. He is explicit that the goal is not to replace RL researchers but to complement them. ARIA handles orchestration, GPU workloads, responding to events in the W&B ecosystem, looking up arXiv papers, and collaborating on hypotheses. This leaves researchers free to focus on new ideas, architectures, and parameters.
How ARIA was built
Sweeney sketches a standard agent architecture: web and iOS clients talk to an API server, which writes to a database that a worker harness consumes. The harness connects to several dependencies. A sandbox executes arbitrary shell calls, Python, and data science code, and Sweeney suggests CoreWeave’s sandbox for this. An LLM provider can be W&B Inference, or a fine-tuned model. Long-running training jobs, which can last days, run outside the agent loop through W&B Launch on CoreWeave GPUs. Finally, an observability layer is critical. The team logs 100% of traces to Weave, covering sessions, turns, tool calls, and errors.
The architecture contains two complementary loops. An offline production loop uses Weave to surface behaviors, turn them into tasks, and run evals. A second, adversarial improvement loop forms hypotheses, implements candidate agents, and analyzes eval results. A shared dashboard in Weave supports go/no-go decisions, the best model is promoted through a registry, and the loop closes the data flywheel. Model artifacts live in a repository that can be backed by W&B Artifacts.
Observability in Weave
Sweeney walks through the agent dashboard in Weave, which shows span volume, conversation volume, and token tracking. He prefers the conversations view, a live feed of conversations filtered to internal employees, and the middle spans view, which visualizes trace topology. Different colors and shapes distinguish tool calls, LLM calls, and thinking blocks. Opening a conversation exposes the system prompt, user messages, shell calls, and reasoning. Sweeney’s research lead and PM use this view to annotate, leave feedback, and identify behavioral nuances that can become tasks.
ARIA is also embedded throughout the W&B UI. A summarize button opens a new chat contextualized to the object being viewed. Sweeney points out that this means ARIA is analyzing ARIA’s own conversations to recommend improvements.
Signals and LLM judges
Weave includes an integrated LLM judge experience that runs live against production traffic. Signals include user frustration, low-quality responses, and ask-user events. In the demo, a judge flagged a conversation in which the user said they were not satisfied with a loss curve, and the judge’s reasoning was visible. These clusters of behavior feed the following week’s iteration.
Tasks as unit tests and nightly evals
Production traces are converted into tasks, which Sweeney describes as YAML files that act as unit tests for the model. His example asks ARIA to compare two runs that both give good results and to explain the difference. Each task defines three judges: an LLM judge for correctness, an LLM judge for whether the insights are interesting, and a rule-based check that the result was produced within six tool calls, which rewards expedience. Roughly 200 such tasks are grouped into a nightly eval suite tracked in Weave. Two nights before the talk, the candidate model scored 73% against 72% for the production model, which the team planned to promote that Friday. Weave also provides a performance plot over time.
Tips for productionizing agents
Sweeney closes with four recommendations. First, invest in agent-oriented observability: log sessions, turns, tools, and feedback so that you can catch behavioral bugs, which are distinct from exceptions or performance problems. Second, treat tasks and evals as the new CI. Researchers and engineers should sit on the same scrum team, and eval metrics should drive go/no-go decisions. Third, keep humans in the loop. LLM judges miss behavioral nuances, so the team should review the best and worst traces weekly on a shared board. Fourth, add value through context and tools before over-engineering the harness. Much low-hanging fruit comes from giving the agent knowledge of the business domain, the available primitives, and the company’s data, rather than from elaborate memory schemes.
Live result
At the end, Sweeney checks the workspace. The previous best loss was about 5.831, and the live batch reached 5.833. Since lower loss is better, this was very close to a new record but did not beat it. The batch ran 12 experiments, and more were scheduled to run overnight. Sweeney invites the audience to try ARIA, scan the QR codes, and read the docs.