State of Data — Sean Cai, Independent / State of Data

Sean Cai opens by stating his intention to reveal the “alpha” in data markets and the murky realities behind data production by large companies like Remarque and Handshake. He reframes the perception of data markets, moving beyond the 2019 image of annotators in Manila, emphasizing that while that work exists, the real value lies in data that elevates generalist models to “real expertise.” Cai likens data to the “coal and iron” of the white-collar revolution, suggesting its supply chain is undergoing an industrializing unbundling.

Two years ago, vertically integrated giants like Scale AI had to do everything, but unit economics have changed. Today, specialists increasingly outcompete these giants in various steps of the data pipeline: sourcing people, building environments, designing rewards, and running evaluations. This fragmentation is seen as permanent, not transitional, as quality no longer scales linearly with quantity. This leads to a “cottage industry” where labs diversify vendors (20-30) due to inherent distrust in scaling quality.

Cai introduces key concepts:

• Antikythera mechanisms: Bespoke systems that translate messy business contexts into evaluations (Evals), crucial for labs seeking real-world seed data.
• Type One vs. Type Two Data:
  • Type One: Pure capture of real workflows (e.g., GitHub commits, session replays) with minimal non-expert reward shaping. This data provides realism, helping models improve from 20% to 80% performance. The only durable supply comes from partnering with live businesses, not acquiring defunct startup codebases.
  • Type Two: Contrived data, manufactured by experts in arbitrary settings. Useful for models at a “first-grade level.” Cai highlights the industry’s “dirty secret”: much Type Two data is sold as Type One.

He then delves into Verifier’s Law, inspired by Jason Wei, which states that the ease of training a model for a task is proportional to its verifiability. Cai breaks verifiability into three axes:

1. Asymmetry of verification: How easily can a task be decomposed into checkable steps?
2. Veracity of verification: The degree of consensus on what “correct” means.
3. Proliferation of verification: How often does the real world provide fresh examples of verified work?

Coding, as the first mature AI application layer, scores high on all three axes (unit tests for objective correctness, community consensus, GitHub commits for free reasoning traces). Cai uses this framework to classify application layer maturity: Code -> Search -> Finance -> Healthcare/Law -> Cyber, Biological, Scientific Discovery, and Taste (the least verifiable). Domains with low verifiability often require enterprise workflow data (e.g., Slack logs, Jira logs).

Cai critiques the industry’s benchmarks, calling them “snake oil” and “quietly fake.” He describes a common, perverse “Type Two recipe”: experts generate tasks, solve them, cherry-pick model failures, package them as a benchmark, and then sell data to “hill-climb” that same benchmark. This is described as “Goodhart’s law with a profit motive,” where measures become targets, losing real meaning. This creates a “fog of war” where nobody truly knows which data improves models. Contrived benchmarks test isolated, in-distribution questions, failing to assess sustained reasoning across complex, dependent episodes. He notes that cross-harness and cross-infrastructure differences are significant causes of benchmark divergence and high false positive/negative rates.

He offers a practical example with three finance tasks (ARR waterfall reconciliation, LBO valuation memo, long-short pair trade). Through agnostic benchmarking, he observed that Opus 4.8 performed worse than 4.7 on some rubrics due to “over-engineered self-reflection.” GPT 5.5 and Opus 4.8, while scoring similarly, failed in opposite ways (GPT nailing arithmetic, Opus nailing methodology but failing arithmetic), revealing different post-training directions. This underscores that a single benchmark number is just “one sample from a distribution nobody measured,” leading to “benchmark psychosis.”

To predict future AI trends, Cai suggests using data markets as an “upstream indicator.” He notes Anthropic’s spending on cybersecurity data (January) followed by biological data (March/April) accurately predicted their subsequent product releases (Metis, Cybor, Claude Bio). He provides a checklist for identifying next domains: classify professional tasks by the three verifiability axes, apply to long-horizon tasks with specific characteristics (enforced step length, heterogeneous tool calls, state transitions, mandatory failure recovery), and look for five signals in raw data (sequential decisions against one entity, inferable expert action per step, independently recorded outcomes, economically available high-wage work). Robotics is presented as a counterexample due to unsettled modalities and unsophisticated data vendors.

Cai concludes that data companies must evolve into “neo-labs” because durable value accrues to services and the application layer of real work. He notes that successful data companies are pivoting to enterprise (e.g., Remarque/Handshake’s significant enterprise revenue). As enterprises aim to own their intelligence, an abstraction layer is needed for:

1. Serving and routing small models based on latency/performance profiles.
2. Managing RL datasets for automatic post-training retraining upon base model migration.
3. Antikythera mechanisms.

He argues that the “automobile has already arrived” for model layer decoupling, citing examples like GLM 5.2 surpassing GPT on some rubrics, indicating that models are not fungible like electricity, preventing durable lock-in. His two key takeaways are for researchers to stop outsourcing their definition of realism to vendors and for builders to recognize that their moat is the pipeline into real-world work plus the infrastructure for continuous retraining. Cai ends by announcing his new venture: building Antikythera mechanisms and offering RL-as-a-service to enterprises to monetize data assets.

• Tags:
• Benchmarks
• Feature
• LLM Evaluation
• Enterprise AI
• Data Quality
• AI Data
• Model Performance
• Data Markets
• AI Supply Chain
• Verifier's Law
• Type 1 Data
• Type 2 Data
• Antikythera Mechanisms
• RL as a Service