The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Poolside’s presentation, led by Mara Abdin and Robert McHardy, delves into their approach to scaling large language models, focusing on sophisticated synthetic data generation and rigorous pre-training methodologies. They recently shifted from enterprise-only models to releasing open-weight models like Laguna M and Laguna XS, with a new 118B parameter model, Laguna S, on the horizon. The core message revolves around the necessity of a holistic approach encompassing data, training code, and architectural correctness.

Synthetic Data Generation for LLM Pre-training

Poolside emphasizes that synthetic data is not a replacement but a crucial complement to organic data. It serves to extract implicit rationale, planning, and structure, filling gaps and regularizing both token presentation and the teaching process for the model. For their Laguna XS.2 model, synthetic data constituted 13% of the pre-training mix, contributing to a continuously growing corpus now exceeding six trillion tokens.

The team identified limitations when scaling from Laguna M.1 to XS.2, particularly hitting repetition saturation with high-quality organic data. Synthetic data, specifically through rephrasing, addressed this “token uniqueness problem,” preventing early model saturation. While adopting common multi-mode rephrasing pipelines, Poolside also developed specialized pipelines for converting natural language to code/text and processing scan data, leveraging the cost-effectiveness and scalability of synthetic data generation.

Inside the Synthetic Data Pipeline

Poolside views every synthetic data pipeline as modular, comprising six key components:

• Seeds: Initial input data.
• Primary Inputs: Main data for generation.
• Metadata: Contextual information.
• Secondary Inputs: Additional supporting data.
• Generator: An LLM acting as an agent with tools or driven by prompt templates.
• Supplementary Functions: Filters and validators to ensure quality.

This modularity allows for the composition of pipelines ranging from simple, cheap, and scalable (using smaller models and being “seed-heavy” like rephrasing) to more complex and orchestrated workflows for generating high-value educational data. A critical “rule of thumb” is to break down complex tasks for the model, ensuring the task is “neither trivial nor so hard the model learns nothing from it.” This prevents the model from losing correctness or diversity.

Poolside showcased various “shapes” of synthetic data generation:

• Rephrasing: Generating diverse phrasings of existing content.

• Multi-stage Pipelines/Workflows: Breaking down complex generations into sequential steps. An example given is generating a novel by first defining settings, characters, plots, and twists, then generating chapters iteratively, leading to a “better novel.”

• Cross-domain Porting: Translating concepts or data from one domain to another, such as converting math problems into code or translating code between languages.

• Multi-turn Role: Involving iterative processes, like multi-turn chats between agents or task evolution pipelines where a “judge” and an “evolver” refine tasks over several iterations.

Orchestration and Policing Generations

Poolside’s Hive infrastructure facilitates configurable generation workflows. It manages queues of agents, each with defined prompts, parameters, and models. Crucially, orchestrators police these generations, providing flexibility and hierarchical control over LLMs, dynamically changing instructions, or selecting subsequent agents. A “supervisor” component offers a global view, policing the orchestrator itself.

Pre-training: Trust Nothing, Crash on Mismatch

Robert McHardy emphasized a “trust nothing” philosophy in pre-training. When scaling models to billions of parameters across thousands of GPUs, numerous issues can arise. Poolside employs model replica hash checks during distributed training. The invariant is that weights should always be identical across all replicas. Periodically, hashes of these weights are compared, and if any mismatch occurs, training is immediately crashed. This proactive approach helps surface “messy failures.”

Failures at Scale and Their Resolution

Several critical failures observed during large-scale pre-training were highlighted:

• Broken GPUs: A broken GPU led to silent data corruption, manifest as spiky loss curves and unusually large gradient norms. This was effectively caught by the replica hash checks, as computations diverged between replicas.

• Numerical Precision Issues: During Laguna M.1 training, activations before the LM head grew to a scale where the default BF16 accumulation for tensor parallel operations lacked sufficient numerical precision. This caused “exploding gradients,” flattening the loss curve and halting model learning. The fix involved switching this specific accumulation to FP32, immediately restoring convergence and decreasing gradient norms.

• Silently Corrupted Gradients (Race Condition): When incorporating FP8 training (using DeepChem FP8 kernels), a race condition was discovered, leading to illegal memory accesses and NaNs. Debugging revealed about 0.5% of gradients were silently corrupted (replaced by random values). This was a “blind spot” for standard hash checks, as real training runs lack the redundancy to compare forward/backward passes across replicas. Poolside developed a fix (available as a PR) and is working on a dry-run hash checker to address such elusive issues.

Laguna S: Scaling to 118 Billion Parameters for Agentic Coding

Poolside integrated all these insights into Laguna XS.2 and further scaled them into Laguna S, a 118 billion parameter model (with 8B active parameters) trained on 30 trillion tokens across 4,000 GPUs. This model is designed specifically for agentic coding.

Early results for Laguna S are promising:

• It surpasses Laguna XS.2 and the larger Laguna M.1 on coding benchmarks like MultiPL-E, CodeBench, and Big Code Bench.
• It outperforms GLM 4.5 Air and is competitive with more recent models like Nemotron 3 360 and Deepseek V4 Flash Max on general benchmarks like Big Bench Hard and Eval+.
• Crucially, for its intended purpose, Laguna S performs “much better than all the other models we tested here” on Sweep Bench (agentless multilingual), which serves as a proxy for agentic performance during pre-training.
• The team acknowledges areas where Laguna S is not top-tier (e.g., MMLU pro knowledge benchmark), attributing this to a deliberate data focus on coding rather than broad knowledge, which “could be plugged if we wanted to.”

The key takeaway is that their “recipe,” combining advanced synthetic data generation, robust training validation, and numerical stability fixes, “held as it scaled.” Laguna S will also be released as an open-weight model in the future.

• Tags:
• LLM
• Synthetic Data
• Large Language Models
• Pre-Training
• Model Evaluation
• Agentic Coding
• Distributed Training
• Numerical Stability
• Data Generation Pipelines
• GPU Failures
• Race Conditions
• Deep Learning Scaling