Going In Deep On Data | YC Paper Club

Three experts discuss the evolving landscape of data in AI, covering its critical role in model performance, challenges in sourcing and evaluating expert data, the development of advanced data generation techniques for diffusion models, and the complexities of multilingual pre-training.

August 20, 2026 9 min read

Going In Deep On Data | YC Paper Club — source: https://www.youtube.com/watch?v=IfoPg2QefF8

The YC Paper Club event highlighted the increasing criticality of data in the current AI landscape, challenging the historical perception of data as a commodity.

Francois Chaubard: The Data Bottleneck and Its Economic Impact

Chaubard initiated the discussion by asserting that data, not architecture or computational power, has become the primary bottleneck in AI development. He noted a significant shift in venture capital perception since 2016, moving from an initial dismissal of data businesses as having “zero terminal value” to acknowledging over $100 billion in market cap creation today.

He illustrated this shift with a common interview question: given a model achieving 85% F1 score, what’s the next step? The correct, and often overlooked, answer is to deeply analyze the data—specifically, false positives and negatives—to identify and address the root causes of errors, such as environmental factors (e.g., fog on a fridge for object detection). This emphasizes that model architecture becomes less critical when data quality and real-world distribution mismatches are the core problems, especially with highly expressive models like Transformers. Chaubard referenced Andrej Karpathy’s early work on “human level” ImageNet performance, highlighting inherent ambiguities and biases within datasets. He suggested that while PhDs historically focused 95% on architecture, production environments now demand 97% focus on data.

The speaker stressed that real-world deployment frequently encounters test distributions vastly different from curated training sets, leading to model failure. To automate economic processes, a dual need exists: expert data for verifiable rewards (e.g., in RL environments) and preference-style data for subjective tasks (e.g., design preferences, coding style, medical diagnoses where expert agreement is low). He characterized datasets and RL environments as “products” requiring meticulous craftsmanship, citing examples like ArcGI games and Focal Systems’ stock management. Furthermore, data is not static; dynamic environments, such as UI changes in software like Salesforce, render even vast quantities of existing data obsolete, necessitating continuous data acquisition.

Chaubard concluded by advocating for specialized data companies, drawing an analogy to the app ecosystem: just as Apple doesn’t deliver groceries, major AI labs shouldn’t be expected to curate highly specialized data for every domain (e.g., medical, legal). Instead, dedicated companies that excel in curating, annotating, and managing expert data and RL environments for specific verticals are essential for unlocking AI’s full potential across diverse applications.

Vincent Sunn Chen: Scaling Expert Supervision for Benchmarking Agents

Vincent Sunn Chen, from Snorkel, elaborated on the challenge of scaling expertise to build effective datasets, a core thesis for Snorkel’s decade-long research. He described the evolution of data-related problems:

Data 1.0 (Basic Labeling): Focused on simple tasks like thumbs-up/down labels. This approach was unscalable (O(N) cost per data point), not robust to noise without extensive redundancy, lacked adaptability to changing specifications, and provided no provenance for rationale. This became particularly intractable in expert-heavy domains like medical imaging, where specialist time is limited.
Data 2.0 (Agent Evaluation): Represents the current frontier, involving the creation of complex environments, rubrics, and verifiers for evaluating advanced AI agents. This can require hours to days of expert human effort per evaluation, driven by increasing complexity in input, output, environment dynamics, and sequence length.

For Data 1.0, Snorkel developed “data programming,” aiming to encode expert supervision in software for scalability, adaptability, and auditability. This involved:

Modeling expertise as labeling functions: Expressing expert rationale (e.g., rules for spam detection) as programmatic heuristics.
Weak supervision: Addressing the inherent overlap and inaccuracy of these signals by developing theory to estimate true quality without ground truth.
Noise-aware end models: Training models that can generalize beyond the initial coverage of labeling functions while accounting for noise. The underlying “label model” estimates each “voter’s” (labeling function’s) accuracy in an unsupervised manner, akin to grading students without an answer key, then uses Bayesian inference to determine ground truth.

For Data 2.0, Chen introduced Senior SWEBench, a benchmark designed to evaluate coding agents as if they were senior engineers. Existing benchmarks often saturate quickly and evaluate agents on junior-level tasks. Senior SWEBench features:

Realistic Task Design: Natural language instructions (e.g., Slack messages, log dumps, user stories) to simulate real-world, higher-level abstractions a senior engineer would encounter.
Nuanced Reward Phase: Going beyond mere correctness to evaluate “taste”—alignment with codebase practices and architectural decisions, minimizing code bloat.

To address the trade-off between reliability (unit tests) and flexibility (LLM judges) in evaluation, Snorkel developed a validation agent. This agent takes high-level expert-provided “validation specs” (user stories, functional requirements) and a code patch, then dynamically implements deterministic test scripts. This approach scales expert intuition to generate robust, executable tests. A new metric, “Tasteful Pass,” was introduced to combine correctness with quality aspects like code practices. Chen announced that Fable, Opus, and Soul models currently achieve top performance on this benchmark, pushing the Pareto frontier. He concluded by emphasizing the need for continued data research, particularly in benchmarks, and highlighted Snorkel’s “Open Benchmarks Grants” to support community efforts.

Volo Kuleshov: Diffusion Language Models for Real-time Production AI

Volo Kuleshov presented Inception’s work on a new generation of language models powered by diffusion, designed for high-speed, real-time AI applications. Unlike traditional autoregressive models that generate tokens sequentially, diffusion models start from a noisy initial sequence and refine all tokens in parallel over multiple steps. This parallel generation offers a significant speed advantage, with Inception’s Mercury 2 models achieving over 1,000 tokens per second, substantially faster than current autoregressive models.

A key application area is real-time voice agents (e.g., for customer support or education). In these cascaded systems (speech-to-text -> LLM -> text-to-speech), the LLM often bottlenecks overall latency. Faster LLMs enable more seamless interactions, the deployment of larger models, and extended reasoning capabilities. Kuleshov showcased Mercury 2’s performance on voice benchmarks, demonstrating a new Pareto frontier of quality and latency. Notably, Mercury 2 achieved higher quality and speed compared to a 120-billion parameter autoregressive model running on specialized Cerebras hardware, emphasizing that intelligent software for diffusion models can achieve superior performance on more widely available GPUs.

Kuleshov underscored the critical role of data for both training and evaluating diffusion models. He critiqued existing benchmarks like TabBench, arguing they fail to capture the full complexity and range of real-world scenarios, often being “benchmaxed” with models scoring high on academic tests but significantly lower and with higher variance on production logs.

To address this, Inception developed Dowo Forge, an agentic system for synthesizing a wide range of realistic RL environments similar to TabBench, directly from real-world data sources (user interactions, company partnerships, or a “business knowledge graph” of crawled internet data). Dowo Forge orchestrates multiple agents through several steps:

Initial Environment Synthesis: Generating policies, user profiles, and tools specific to a domain (e.g., banking), optionally conditioned on real usage data.
Task Generation: Creating simulated user interactions based on a library of abstract scenarios (“seeds”) and personas, specialized by agents for the target domain.
Iterative Task Refinement: Filtering and reconstructing tasks to ensure they are neither too easy (lacking learning signal) nor too hard. This involves “hardening traps” to incrementally increase task difficulty (e.g., removing information, increasing complexity) or filtering tasks until an optimal balance for learning and realism is achieved.

The system’s output is continuously inspected and refined using real-world feedback. Kuleshov reported that Mercury 2.5, when trained on Dowo Forge-synthesized tasks, improved performance by over 23% on diverse unseen environments, matching state-of-the-art models and correlating strongly with real-world production performance. He highlighted Mercury 2’s availability for latency-sensitive tasks in voice, search, and code, and offered YC startups $500,000 in credits.

Shayne Longpre: ATLAS – Practical Scaling Laws for Multilingual Models

Shayne Longpre presented ATLAS, his PhD work on understanding cross-lingual transfer, synergies, and interferences in multilingual pre-training. He argued that current scaling and mixing law research is overwhelmingly English-centric, neglecting the unique challenges and opportunities of the “rest of the world” where data scarcity is a major constraint. He noted that even a language like Thai, with 80 million speakers, has less than 0.6% of the tokens available compared to English in common corpora.

Longpre highlighted the limitations of existing approaches for low-resource languages:

Massive Multilingual Models: Suffer from poor performance on low-resource languages due to their tiny proportion in the overall training mixture.
Monolingual Training: Leads to rapid overfitting and necessitates much smaller models, still resulting in suboptimal performance.

The proposed solution involves a mixed training approach: leveraging substantial amounts of target language data while strategically incorporating high-quality, synergistic data from other languages (e.g., English, Indonesian, Malay) before encountering diminishing returns. Crucially, “synergistic” is not solely about language families but an empirical measure of helpful, non-noisy data.

To quantify these relationships, ATLAS introduced a method for measuring language synergies:

By comparing learning curves of monolingual training with mixed-language training, an “efficiency gap” metric is derived. A smaller gap indicates higher synergy.
This leads to a cross-lingual transfer matrix, where red signifies high synergy and blue indicates high interference. For example, Spanish showed high synergy with Portuguese and French but strong interference with Japanese.
The study found that model size significantly impacts transfer: larger models exhibit more synergy, accommodating multiple languages, while smaller models show more interference.
Asymmetry of Transfer: A key finding was that transfer is not symmetric (e.g., Portuguese helping Spanish does not imply Spanish helping Portuguese), necessitating empirical measurement.
Script vs. Language Family: Both factors contribute, but shared script tends to be more influential, potentially due to tokenization artifacts.

ATLAS extended Chinchilla-style scaling laws to multilingual settings by decomposing the data parameter (‘D’) into components representing the monolingual source, empirically identified close transfer languages, and a general “everything else” bucket. This refined scaling law accurately predicts loss and optimizes language mixtures. Longpre concluded by detailing practical implications for practitioners, including guidance on when to fine-tune versus pre-train from scratch based on computational resources, and insights into the “curse of multilinguality”—quantifying how much more data or model size is needed to maintain performance when increasing the number of languages a model supports.