W&B Models end-to-end demo
W&B Models is the system of record for the entire model development lifecycle. This guide explores how to monitor training, tune hyperparameters, track artifacts and lineage for reproducibility, and automate MLOps workflows like evaluation and deployment using a central platform.
Weights & Biases (W&B) Models is a comprehensive platform designed to be the central system of record for the entire machine learning development lifecycle. With minimal code integration, it enables teams to track experiments end-to-end, manage assets, and automate workflows, allowing data scientists, ML engineers, and researchers to focus on model development rather than operational overhead.
End-to-End Experiment Tracking
W&B automatically captures and visualizes all critical aspects of a training run, regardless of the ML framework (PyTorch, Keras, Hugging Face Transformers) or infrastructure (cloud or on-prem).
Training and System Metrics
The platform automatically logs and visualizes key metrics out-of-the-box.
- System Metrics: GPU utilization and memory allocation are tracked to help teams understand hardware usage and identify performance bottlenecks.
- Training Metrics: Loss curves and other custom metrics are automatically plotted, saving teams the effort of manual chart setup. These visualizations are highly customizable; for instance, you can apply smoothing to a reward curve to better understand its overall trajectory instead of focusing on noisy, step-by-step data.
Alerts and Monitoring
To enable proactive monitoring, users can set up alerts on any tracked metric. By defining a condition, such as a 50% spike in a value, teams can trigger notifications to Slack channels or specific users, allowing for rapid investigation and debugging of training anomalies.
Media and Visual Feedback
W&B provides powerful tools for qualitative analysis by tracking media generated during training. For a reinforcement learning task, this could be a video simulation of the agent’s behavior. A synchronized step slider allows you to jump to any point in the training process and visually inspect the model’s performance, directly correlating it with the quantitative metrics. This capability extends to any visual or text-based output, such as image generation or language model outputs, providing invaluable intuition.
Scaling Experiments and Evaluation
Hyperparameter Tuning with Sweeps
For exploring a large parameter space, W&B Sweeps provides a structured way to organize and visualize hyperparameter tuning experiments. It offers clear visualizations, like parameter importance plots, that help you quickly understand how different hyperparameter combinations affect model outcomes, such as cumulative reward.
Integrated Evaluation
W&B closes the loop between training and validation by allowing evaluation results to be logged back into the same workspace. This creates a unified view where you can seamlessly navigate from a real-world evaluation metric back to the corresponding training run, its media simulations, and its specific configuration.
Reproducibility, Lineage, and Collaboration
W&B is built to ensure that every experiment is reproducible and auditable.
Run Details
For every run, W&B captures a complete snapshot of the training environment:
- The exact parameters passed to the training script.
- The Git commit hash and repository used to launch the run.
- System logs and package requirements (
requirements.txt).
Artifacts and Lineage
The core of W&B’s reproducibility features lies in Artifacts. Artifacts are versioned collections of files, such as datasets, models, or evaluation results. The platform automatically tracks the flow of these artifacts, creating a complete Lineage Graph. This graph visually details which input artifacts (e.g., dataset version) were used by which run to produce an output artifact (e.g., a model file), and where that model was subsequently consumed (e.g., in an evaluation run).
Model Governance and MLOps Automation
Central Model Registry
While artifacts are local to a project, they can be linked to a central, organization-wide Model Registry. The registry serves as a single source of truth for production-ready models and datasets. It features granular access control, allowing you to define admins who can manage protected aliases (e.g., production, staging) and users who have read-only access.
Automations for CI/CD
Automations connect W&B to external CI/CD systems like GitHub Actions via webhooks, enabling powerful MLOps workflows. Automations can be triggered by various events:
- Adding a tag to an artifact: For example, adding a
testtag to a model artifact can automatically trigger a CI/CD pipeline that runs an evaluation script and logs the results back to W&B. - Linking a model to the registry: This can trigger a more resource-intensive hardware validation test.
- A run finishing: This can trigger a script to automatically generate a detailed performance report, complete with charts and media, using the W&B SDK.
By integrating these components, Weights & Biases acts as a central hub that streamlines the entire ML lifecycle, from initial experimentation to production deployment, fostering collaboration and ensuring robust, reproducible results.