Tokenless

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

An experiment by Snorkel AI reveals that in agentic AI training, the quality of tasks is paramount. Using the same model and compute, fine-tuning on high-quality tasks yielded a 6% performance improvement, a 5x greater uplift compared to the 1% gain from low-quality tasks. The key difference lies in the nature of the tasks: high-quality tasks are genuinely harder, featuring more tool calls and cleaner failure modes that provide a meaningful learning signal. In contrast, low-quality tasks often fail due to ambiguity and environmental noise, hindering effective model improvement.

3 min read
Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel
source: https://www.youtube.com/watch?v=YYH0DMQr30A

The foundational thesis of Snorkel AI is that data quality is the most critical factor in training powerful models. This principle extends directly to the domain of agentic AI, where the quality of a task dictates the effectiveness of training. An empirical study was conducted to validate this intuition, demonstrating that high-quality tasks lead to substantially better training outcomes than their low-quality counterparts.

Defining and Measuring Task Quality

The experiment focused on agentic, TerminalBench-style tasks within containerized environments to ensure reproducibility and isolation. A task was defined as “high-quality” if it met four core criteria:

  1. Achievable: The task is solvable.
  2. Non-trivial: It presents a genuine challenge.
  3. Functionally Correct: The internal logic of the task operates as expected.
  4. Reliable Environment: The containerized environment is stable and consistent.

Tasks that passed automated tests for all four criteria were placed in an “accepted” (high-quality) bucket. Those that failed one or more were placed in a “rejected” (low-quality) bucket.

High-Quality vs. Low-Quality Tasks: A Tale of Two Buckets

Analysis revealed significant differences between the accepted and rejected tasks when executed by models like Codex and a claude-3-opus-20240229 precursor.

High-Quality (Accepted) Tasks:

  • Greater Complexity: Averaged twice as many tool calls, indicating more steps and interaction were required.
  • Higher Difficulty: Showed a lower pass rate, confirming they were intrinsically harder.
  • More Reasoning: Required more output tokens, suggesting more complex reasoning from the model.
  • Cleaner Failures: When models failed, it was for meaningful reasons, such as logic errors or incomplete execution. These “clean failures” provide a valuable signal for what the model needs to learn, allowing it to “hill climb” effectively.

Low-Quality (Rejected) Tasks:

  • Noisy Failures: These tasks tended to fail due to degenerate cases, such as environmental issues or a fundamental mismatch between the prompt’s request and the evaluation’s criteria.
  • Under-specification: A common issue was ambiguity or implicit dependencies. The task might not provide the model with the necessary context or information to pass tests that had unstated requirements.

A key takeaway is that ambiguous specifications do not produce harder tasks; they produce noise. These noisy failures do not offer a useful learning signal and can mask true model performance.

The Experiment: A 5x Uplift from Quality Alone

To measure the impact on training, two identical Reinforcement Learning (RL) runs were conducted using the same base model, compute budget, and number of tasks. The only variable was the quality of the training data.

  • Run 1: Trained on low-quality tasks from the “rejected” bucket.
  • Run 2: Trained on high-quality tasks from the “accepted” bucket.

The results were striking and confirmed the initial hypothesis:

  • Training on low-quality tasks improved the base model’s performance by approximately 1%.
  • Training on high-quality tasks improved the base model’s performance by approximately 6%.

This represents a 5x difference in performance uplift attributable solely to the quality of the training tasks.

Conclusion: Prioritize Quality with Expert-in-the-Loop

This experiment provides strong empirical evidence that a focus on data and task quality is essential for achieving significant gains in model performance. The most effective way to ensure this quality is by incorporating human expertise. An “expert-in-the-loop” approach is critical for designing tasks that are well-defined, functionally correct, and provide the clean failure signals necessary for models to learn and improve efficiently. For more complex or less easily verifiable domains, this extends to developing detailed rubrics and ensuring high inter-annotator agreement to maintain quality at scale.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

How Cursor Trained Composer on Fireworks: Distributed Infrastructure for High-Performance RL

How Cursor Trained Composer on Fireworks: Distributed Infrastructure for High-Performance RL

Cursor's Federico Cassano and Fireworks' Dmytro Dzhulgakov detail their collaboration on Composer 2, a specialized foundation model for software engineering. They discuss their top-down training strategy, the infrastructure challenges of large-scale distributed Reinforcement Learning on sparse models, and how model specialization achieves frontier performance with superior efficiency.

End-to-End Foundation Models for the Energy Industry — with Jazmia Henry

End-to-End Foundation Models for the Energy Industry — with Jazmia Henry

Jazmia Henry details the end-to-end process of building specialized foundation models for the energy industry. She covers the four key stages from data curation of unstructured, handwritten documents to optimizing inference, and introduces her Grounded Continuous Evaluation (GCE) framework to combat reward hacking in reinforcement learning.