Fully Connected keynote: Building tools for agents at Weights & Biases

In the rapidly evolving AI landscape, creating long-term roadmaps is a difficult task. Instead of promising future features, it’s more telling to examine what has been built recently. The guiding principle for Weights & Biases has always been to create useful tools for the developers building AI systems. Over the last three months, significant progress has been made across the W&B product suite, reflecting the changing needs of the industry, from foundational model training to the complexities of agentic systems.

Recent Innovations in W&B Models

The W&B Models product, the core of experiment tracking, has received substantial updates focused on performance, usability, and lifecycle management in response to the massive scale of modern AI development.

• Performance and Scalability: The most consistent user request has been for faster performance. In the last three months, long-tail query times (P90 and P99) for the largest customers have been reduced by two orders of magnitude. This backend enhancement now enables full-fidelity charts where users can zoom in and render millions of data points, handling the unexpected scale of users running hundreds of millions of experiments.
• Enhanced Training Lineage: Modern foundation model training is no longer linear. To support this, W&B now has first-class support for forking runs. Users can stitch together multiple branched runs into a single, cohesive view, allowing them to rewind and understand the complete lineage of an experiment, even if it spans months and multiple distinct processes.
• Observability for Distributed Training: With the rise of distributed training across thousands of nodes, W&B now offers unified console logs. This feature aggregates logs from all nodes into a single, searchable interface, simplifying the debugging process for large-scale training jobs.
• Model Lifecycle Management: The W&B Registry has become a critical tool, especially for larger customers in regulated industries. It addresses the human coordination problem of tracking not just experiments, but the thousands of models moving into and out of production. Recent improvements include more sophisticated access controls, global search for better discoverability, and robust API-based access for programmatic interaction.

Advancing Agent Development with W&B Weave

W&B Weave is designed for developers who are building applications and agents on top of AI models. The focus is on speeding up the experimental cycle and providing deep insights into complex agentic behavior.

• Deeper Insights and Customization: The Weave playground now supports custom LLMs, allowing developers to seamlessly import models they’ve trained with W&B Models and test them in an interactive environment. Tracing capabilities have also been significantly improved to handle the complexity of modern agents, which can involve thousands of nested LLM and tool calls.
• Built-in Safety and Guardrails: To help developers build responsible AI, Weave now includes a set of high-quality, pre-built guardrails to address common concerns like PII leakage, jailbreaking, and toxic responses. These guardrails are built on open models, allowing users to inspect, fine-tune, or replace them with their own custom solutions.
• Comprehensive Evaluation and Monitoring: Weave now features flexible evaluation APIs, including a UI-based builder for creating “LLM-as-a-judge” evaluations. This empowers all stakeholders, including non-engineers, to define and run evaluations. Furthermore, these evaluations can be configured as monitors that score traces live in production, providing real-time feedback on agent performance.

The W&B and CoreWeave Partnership: An Integrated AI Cloud

The acquisition of Weights & Biases by CoreWeave marks a significant step towards creating an essential, integrated cloud for AI. CoreWeave provides an AI-native infrastructure stack—from liquid-cooled, high-density data centers to managed Kubernetes and Slurm—designed to maximize the value of expensive GPU resources.

• Integrated Observability: A key integration allows W&B Models customers running on CoreWeave to see infrastructure health metrics from CoreWeave’s “Mission Control” system directly alongside their training runs. This helps differentiate between model-level issues and underlying hardware failures, drastically simplifying debugging.
• High-Performance AI Object Storage: CoreWeave has built an object storage system from the ground up for AI workloads, featuring high throughput (up to 7 Gbps per GPU) and a simplified pricing model. A critical feature is the elimination of egress fees, giving users the freedom to move data without financial penalty. This is integrated with W&B Models via the “bring your own bucket” feature.
• New Joint Offerings: The partnership has already yielded new products, including W&B Inference, a simple and powerful inference service, and Serverless RL, which abstracts away the complexity of reinforcement learning infrastructure, making it accessible to all developers. The recent acquisition of Marimo aims to bridge the gap between the exploratory nature of notebooks and the reliability required for production software.

Case Study: Building a State-of-the-Art SWE Agent

To stay close to the developer experience, the W&B team undertook a project to build an autonomous software engineer agent, targeting the Swe-bench benchmark—a challenging test based on real-world GitHub issues.

• The Experimental Loop: The process involved nearly 1,000 experiments, each a costly and time-consuming endeavor. The team focused on optimizing the three phases of the experimental loop:
  1. Research: Analyzing agent failures by building custom visualization tools on top of Weave.
  2. Implementation: Modifying the agent’s harness (the code surrounding the LLM, including prompts, tools, and memory).
  3. Run: Executing the agent on the benchmark within Docker containers.
• Automating the Research Process: The team experimented with an approach inspired by the “Darwin-Girdle Machines” paper to automate the experimental loop. This “researcher” agent analyzed failures and proposed the next iteration. Remarkably, this automated process achieved the same level of performance as the months-long human-led effort.

Future Directions: Towards Self-Improving AI

While the SWE-agent project was successful, it highlighted key limitations and future directions for the field. The next frontier is to move beyond simply improving the agent’s harness and begin improving the core model itself through Reinforcement Learning. RL is uniquely suited for agentic systems because it can provide rewards based on final outcomes, allowing the model to learn complex, multi-step reasoning without prescriptive guidance.

The ultimate vision is a “researcher agent” that can automate the entire AI development lifecycle. By combining a programmer agent with the W&B and CoreWeave APIs, such an agent could analyze past experiments, provision infrastructure, launch new training jobs, and iterate towards better models. As benchmarks like MLE-bench show, we are still in the early days (around a 20% solve rate for ML engineering tasks), but progress here will accelerate innovation across the entire field of AI.

• Tags:
• AI Agents
• Reinforcement Learning
• LLMOps
• AI Infrastructure
• W&B
• CoreWeave
• SWE-Agent
• Model Registry