How Cursor Trained Composer on Fireworks: Distributed Infrastructure for High-Performance RL

The Rationale for Specialization

The core insight behind building Composer, Cursor’s own foundation model, is the concept of finite model capacity. A model’s weights can be thought of as a storage drive with a limited number of bits. The goal was to allocate all of these bits to the singular task of software engineering specifically within the Cursor environment, rather than general programming or other tasks. By focusing the model’s entire capacity on this narrow domain, it becomes not only better at the task but also far more efficient. This specialization allows a smaller, less expensive model to deliver performance comparable to or better than much larger, general-purpose models like GPT-4o. This approach represents a trade-off in the three-dimensional space of quality, speed, and cost, where specialization pushes the frontier of what’s possible.

A Top-Down Training Strategy

Instead of a traditional bottom-up approach (pre-training from scratch, then mid-training, then instruction tuning), the team adopted a “top-down” strategy to deliver a useful model to users as quickly as possible. The process was:

1. Start with a Strong Base: They began with Qwen1.5-32B, a powerful open-source model.
2. Continual Pre-Training (Mid-Training): The first phase involved extensive mid-training on vast amounts of code tokens, operating at a scale close to pre-training. This step imbues the model with world knowledge, familiarity with specific code patterns, libraries, and a broad distribution of coding knowledge. It learns how to write code, but not necessarily correct code.
3. Large-Scale Reinforcement Learning (RL): The checkpoint from the mid-training phase was then subjected to a massive RL run. In this stage, the model actively interacts with the Cursor environment (the “harness”). This is where it learns to act correctly, call tools properly, navigate its environment, and differentiate between correct and incorrect code. The RL phase sharpens the broad knowledge from mid-training, tuning the model to be an expert that consistently produces correct and useful output.

Overcoming Large-Scale RL Infrastructure Hurdles

Reinforcement Learning infrastructure is inherently more complex than that for pre-training. It requires orchestrating three major components simultaneously: the training cluster, the inference workers for model “rollouts” (simulated agent sessions), and the environments that mimic a user’s computer.

To maximize the use of expensive GPUs, they implemented an asynchronous, pipelined system. The trainer continuously updates the model weights using results from the rollout workers, which are always running inference with the latest available model version. This introduces a “staleness” problem—by the time a rollout completes, the model may have already been updated—but the massive efficiency gain from keeping all hardware constantly utilized far outweighs the minor algorithmic trade-off.

Key infrastructure innovations included:

• Globally Distributed Clusters: To circumvent the scarcity of large, contiguous GPU clusters, they distributed the workload. The core training ran on one main cluster, while the inference and environment simulations were spread across multiple smaller clusters worldwide, even opportunistically using idle production servers.
• Efficient Weight Synchronization: A major challenge in this distributed setup was synchronizing the multi-terabyte model weights across the globe without introducing significant delays. The solution was a sophisticated compression algorithm that identified and transmitted only the delta (the small subset of weights that changed in a training step) rather than the entire model. This reduced the data transfer size by over 20x, making near-real-time synchronization feasible.

Solving Numerical Instability in Sparse Models

The base model’s sparse Mixture of Experts (MoE) architecture introduced a significant challenge: numerical mismatch. Floating-point arithmetic is non-deterministic; the order of operations can lead to tiny differences in results. In an MoE model, these minute variations can be amplified by the gating network, causing it to route a token to a different set of experts during the training rerun than it did during the initial inference rollout. This breaks the training signal, as the trainer attempts to update experts that weren’t even involved in the original generation.

To combat this, the team implemented several solutions:

• Custom GPU Kernels: They wrote kernels that enforce a deterministic order of operations to minimize variance.
• Router Replay: The inference process was modified to pass a small piece of metadata back to the trainer, explicitly stating which experts were activated for each token. This ensures the trainer and the inference rollout are perfectly aligned, eliminating the mismatch.

Advanced RL Techniques for Long-Horizon Agency

Training agents for long-horizon tasks introduces two primary difficulties:

1. Credit Assignment: The longer the sequence of actions (the trajectory), the harder it is for the model to determine which specific actions led to the final positive or negative reward.
2. Finite Context Windows: Models eventually run out of space to store the history of their work.

Cursor’s solution is “self-summarization,” a technique integrated directly into the RL loop. The model learns not only to perform the coding task but also how to summarize its own progress. When it approaches its context limit, it generates a summary, effectively resetting its context window while retaining the essential information needed to continue its task. This allows the model to operate over millions of tokens, far beyond its nominal context window, by jointly optimizing for task completion, summary generation, and the ability to follow its own summaries.

Offline vs. Online RL

The team utilizes a two-pronged approach to RL:

• Offline RL (Simulated): The bulk of the training occurs in a simulated environment. This allows for massive parallelism, where the model can attempt a single task hundreds of times simultaneously (e.g., 128 rollouts from the same prompt). This provides a much richer, more precise learning signal and allows the model to experiment with “off-policy” actions without affecting real users. This phase is crucial for bootstrapping the model’s core reasoning and tool-use capabilities to a level high enough for production.
• Online RL (Real-Time RL): Once the model is performing well, it’s deployed to users. User interactions that signal satisfaction or dissatisfaction are then used as a reward signal to continuously update the live model every few hours. This acts as a “cherry on top,” further refining the model’s behavior based on real-world usage.

• Tags:
• Agentic AI
• Foundation Models
• Reinforcement Learning
• Feature
• AI Infrastructure
• Mixture-of-Experts
• Distributed Training
• Model Specialization