Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI

Ari Morcos, CEO and co-founder of DatologyAI, argues that data quality is the most underinvested part of model training and acts as a “compute multiplier.” He highlights a growing crisis of compute scarcity and skyrocketing token usage, especially for reasoning models, leading to situations like Google capping Meta’s Gemini usage and OpenAI selling “token futures.” In this environment, improving data quality becomes paramount.

Morcos illustrates that enhancing data quality effectively “steepens the learning curve,” enabling dramatically better model performance for the same compute budget, or equivalent performance with significantly less compute (e.g., 10 times less). The core objective is to maximize “signal per token and per batch” by focusing on the marginal information gain of each data point. Key characteristics of optimal data include relevance to specific use cases, diversity (to improve robustness and reduce brittleness), informativeness, and correct mixing of various data sources.

DatologyAI operates as an “oil refinery for data,” taking existing tokens from public, proprietary, and licensed datasets and enhancing their quality through a “Four Cs” process:

• Clean: Involves heuristic filtering (like GoPher), removing low-quality or short documents, and rigorously decontaminating training data against all downstream benchmarks to prevent data leakage and ensure fair evaluation.
• Curate: Employs quality classifiers, taxonomy balancing, redundancy reduction (identifying and removing semantically similar but not identical data points), upsampling/downsampling based on quality and relevance, and task distribution matching.
• Create (Synthetic Data): Utilizes a proprietary “rephrasing” technique (pioneered by DatologyAI) as an advanced form of data augmentation. This involves converting high-quality source documents into hundreds of templated formats (e.g., true/false questions). This approach avoids “model collapse” because information is rooted in the source, increasing diversity and allowing the model to learn more efficiently. Critically, only high-quality documents are rephrased.
• Compose: Focuses on strategically combining these refined datasets and sequencing them across different training stages, often involving multi-phase curricula, to optimize overall model performance.

Morcos presents compelling empirical evidence and customer success stories:

• Foundational Research: DatologyAI’s work, stemming from the NeurIPS best paper “Beyond Scaling Laws,” demonstrated that judicious data selection can “bend the scaling laws” by eliminating redundant or unnecessary data, making training more efficient.
• Vision-Language Models (VLMs): Through data curation alone on a relatively small 25-billion-token dataset, DatologyAI’s models achieved a 14 absolute percentage point improvement, surpassing existing Pareto frontiers. They matched the performance of models like Quen 3.5 4B while using 145 times less training compute. Furthermore, curated data led to more concise answers and improved inference efficiency, achieving similar quality to Quen 35 with 35 times fewer flops per correct answer.
• Multilingual Text Models: To address the English-centric bias of internet data, curated multilingual datasets achieved better MMLU performance using only 8% multilingual tokens (max 6 billion tokens per non-English language). This resulted in performance exceeding models like Quen 3 with 8 times less compute. The research also revealed strong cross-lingual benefits, where curating English data improved non-English performance, correlated with language similarity.
• Thomson Reuters: Partnered with DatologyAI for mid-training legal models. By mid-training on 100 billion tokens (a mix of proprietary and public data, less than 1% of the original pre-training budget), they saw a 5 percentage point increase in legal capabilities without catastrophic forgetting of general abilities. Crucially, post-training gains almost tripled when applied to the mid-trained model, highlighting the synergistic impact of data quality across all training stages.
• Arcee (Trinity Model): Arcee, with DatologyAI, trained an “open frontier” model on 17 trillion curated public tokens (no proprietary data or closed API usage). This model became competitive with leading models like GLM5 and Kimmy, and even outperformed Claude on certain tasks. The entire endeavor, including all R&D, compute, and salaries for this and other models, cost less than $20 million, demonstrating that highly performant custom models are achievable without “hundreds of millions of dollars.”

In conclusion, Morcos emphasizes that focusing on “signal per token” rather than just more tokens is critical, as repeating high-quality data is superior to introducing low-quality data. Data quality remains the most “underleveraged compute multiplier,” and investing in it is a “frontier research engineering problem” that yields tremendous leverage, enabling robust data understanding at petabyte scale.

• Tags:
• Synthetic Data
• LLMs
• Vision Language Models
• Data Curation
• Scaling Laws
• Data Quality
• Model Training
• Compute Multiplier
• Multilingual Models
• Inference Efficiency
• DatologyAI
• Thomson Reuters
• Arcee
• Rephrasing
• Token Scarcity
• Beyond Scaling Laws