Introducing serverless reinforcement learning: Train reliable AI agents without worrying about GPUs

The Journey from SFT to RL

Supervised Fine-Tuning (SFT) is effective for model distillation—training a smaller, faster, or cheaper model to mimic a larger one. However, it has significant drawbacks. SFT is a “heavyweight” process that tends to “clobber” model weights, severely degrading performance on tasks outside its specific training domain. This lack of generalizability means that if your task changes or a new customer has a slightly different use case, you often need to retrain the model entirely.

Reinforcement Learning (RL) offers a more effective alternative. In practice, RL achieves high performance on a specific task with far fewer or smaller weight updates compared to SFT. This has two key benefits:

1. Reduced Performance Degradation: Because the model’s behavior is altered less overall, it maintains its capabilities on out-of-domain tasks.
2. Better Generalization: Models trained with RL generalize better to tasks that are on the edge of their training domain. For example, an RL model trained to support two customers for a voice shopping service might be able to handle a third customer with a new product catalog just by updating the prompt, whereas an SFT model would likely require a full retraining with the new customer’s data.

A Concrete Example: The ART-E Research Agent

To demonstrate the power of RL, the team benchmarked an agentic task called ART-E, where an agent must search an email inbox to answer a user’s question.

• Baseline Models: Top-tier models like GPT-4 achieved around 90% accuracy. While decent, a 10% error rate is often unacceptable for a production application.
• RL-Trained Model: A much smaller 14-billion parameter model (Qwen 1.5 14B) was trained using RL. It started with a poor 41% accuracy but, after training on a few hundred examples, its performance soared.
• Results:
  • Accuracy: Reached 96%, representing a 60% reduction in errors compared to the best off-the-shelf model.
  • Latency: Achieved a ~1-second response time, compared to 3-5 seconds for the larger models.
  • Cost: The run cost was an order of magnitude cheaper than even the most efficient baseline models.

Solving Core RL Challenges

Two major barriers have historically made RL difficult to implement: defining rewards and managing infrastructure.

The Reward Problem and Ruler

A core requirement of RL is a “reward” signal that tells the agent whether it performed a task well or poorly. This is straightforward for tasks with clear outcomes, like math problems or code with unit tests, but is very difficult for ambiguous, real-world tasks like customer support.

To solve this, the team developed Ruler (Relative Universal LM Elicited Rewards), an open-source, parameterized “LLM as a judge.”

Instead of trying to assign an absolute “good” or “bad” score, Ruler works by running the same task multiple times in parallel (e.g., four times) and then using an LLM to provide a relative ranking of the four different outcomes.

This constrained, comparative approach has proven to be highly effective, producing models whose performance matches or even exceeds those trained with meticulously hand-written reward functions.

The Infrastructure Problem and Serverless RL

RL training loops are notoriously complex and resource-intensive. They require generating numerous “rollouts” (running the agent), scoring them, and then using those results to update the model weights. This creates significant infrastructure headaches:

• Managing separate GPUs for training and inference.
• Inefficient GPU utilization, as the GPU sits idle while waiting for external API calls or other processes.
• The fragility of long-running training jobs, which can fail overnight.

Serverless RL was created to eliminate this friction. It’s a managed service that offloads the most complex parts—model training and inference—to CoreWeave’s optimized infrastructure. The user’s environment, including proprietary tools, APIs, and reward logic, remains on their local machine or server. This hybrid approach provides maximum flexibility while abstracting away the infrastructure pain. Because the service can multiplex jobs from many users, it achieves much higher GPU utilization, making it not only more convenient but also typically cheaper than running it yourself.

Playbook for Adopting RL

For teams considering RL, the following approach is recommended for the highest chance of success:

1. Start with Prompting, Not RL: First, build your agent using a prompted foundation model. Focus on setting up the harness, defining the tools, and integrating the agent into your workflow. Get this version working as well as possible.
2. Push to Production (If Possible): Deploy the prompted agent to understand its real-world performance and limitations.
3. Apply RL to Enhance Performance: Once you’ve hit the ceiling of what prompting can achieve, use RL to improve the agent’s reliability, speed, or accuracy.

The best initial use cases for RL are often in latency-sensitive applications like real-time voice agents. In these scenarios, you are already forced to use a smaller, faster model. RL is the best tool available to take that small model and make it perform even better than a large, slow, API-based one.

• Tags:
• Fine-Tuning
• Reinforcement Learning
• LLM Agents
• RL
• Serverless RL
• SFT
• Ruler