Tokenless

Stop Making Models Bigger, Make Them Behave — Kobie Crawdord, Snorkel

Snorkel.ai's research demonstrates how a 4-billion-parameter model, fine-tuned with Reinforcement Learning for under $500, significantly outperformed a 235-billion-parameter model on financial analysis tool-use tasks. The key was cultivating 'tool discipline' and error correction capabilities, rather than relying on sheer model size or deeper reasoning. Single-table training generalized effectively to harder multi-table problems, emphasizing the importance of targeted behavioral fixes identified through detailed evaluation rubrics.

2 min read
Stop Making Models Bigger, Make Them Behave — Kobie Crawdord, Snorkel
source: https://www.youtube.com/watch?v=TNwJ1LMiENk

Kobe Crawford from Snorkel.ai, a “Frontier AI Data Lab,” introduces research challenging the conventional wisdom of constantly increasing model size for improved performance. Snorkel’s core mission is to provide high-quality datasets, ensuring expert involvement in the data generation and validation process. This research, a collaboration with UC Berkeley’s RLLM team and the Agentic Project, aimed to demonstrate that a 4-billion-parameter model could outperform a 235-billion-parameter model on a financial analysis tool-use task.

The Problem: Over-reliance on Larger Models for Tool Use

Crawford highlights that for enterprise AI applications, priorities often shift towards reliability, cost-efficiency, speed, security, and the ability for on-premise deployment. A common but often misguided solution to performance issues is to simply

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

An experiment by Snorkel AI reveals that in agentic AI training, the quality of tasks is paramount. Using the same model and compute, fine-tuning on high-quality tasks yielded a 6% performance improvement, a 5x greater uplift compared to the 1% gain from low-quality tasks. The key difference lies in the nature of the tasks: high-quality tasks are genuinely harder, featuring more tool calls and cleaner failure modes that provide a meaningful learning signal. In contrast, low-quality tasks often fail due to ambiguity and environmental noise, hindering effective model improvement.

End-to-End Foundation Models for the Energy Industry — with Jazmia Henry

End-to-End Foundation Models for the Energy Industry — with Jazmia Henry

Jazmia Henry details the end-to-end process of building specialized foundation models for the energy industry. She covers the four key stages from data curation of unstructured, handwritten documents to optimizing inference, and introduces her Grounded Continuous Evaluation (GCE) framework to combat reward hacking in reinforcement learning.