SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

Nubank, a leading digital bank in Latin America serving 135 million customers, faces a significant challenge in rapidly developing and deploying AI agents for customer support. The core problem, as articulated by Shreya Rajpal (CEO, Snowglobe) and Aman (Principal ML Engineer, Nubank), is that agent evaluation (evals)—specifically the generation of robust evaluation data—has become the primary bottleneck, hindering high-throughput experimentation and fast release cycles.

The Agent Evaluation Bottleneck

Traditionally, AI agent development has been slow due to the complexities of evaluation data:

• Data Complexity: Unlike earlier ML tasks with structured, single-turn data, modern AI agents require multi-turn, stateful conversational data with numerous internal tool calls. Each “data point” is a complex trajectory, making its generation and annotation extremely expensive and time-consuming.
• Existing Data Generation Methods:
  • Manual Authoring: Painfully slow and resource-intensive, requiring hand-planning state updates and consistent trajectories.
  • Production Traces: While “almost free,” they involve testing on live users, making parallel experimentation difficult and feedback cycles excessively long (weeks to months for A/B tests) due to sparse and noisy customer feedback.
• Impact on Release Cycle: Without effective evals, changing agent components (harness, prompt, tools) can take hours, offline evals days, and production A/B tests “forever” to confirm improvements.

Simulation as the Accelerator

Nubank’s experience, in collaboration with Snowglobe, demonstrates that generating evaluation data through simulations rather than solely relying on production data can accelerate agent shipping by 20x. This approach has enabled Nubank to achieve significantly higher customer satisfaction (TNPS) for its AI agents, with many now approaching, and some even exceeding, human quality.

How Snowglobe’s Grounded Simulations Work

Snowglobe’s simulation framework for agent evals mechanically involves three key steps:

1. Agent Integration: The Snowglobe SDK wraps the agent without requiring code changes, allowing the system to identify and mock necessary tools.
2. Persona and Use Case Definition: Developers define personas (e.g., “Maria Souza,” a 34-year-old designer, first-time credit card customer with a specific tone and intent) and the use cases to be simulated.
3. Grounded Data Generation: For each persona, the system generates “essential grounding data” (e.g., fake addresses, credit card numbers, account context) that remains consistent throughout the simulated conversation. This ensures realistic and verifiable interactions for the agent’s tool calls.
4. Execution and Output: Snowglobe runs thousands of multi-turn conversations against the real agent, complete with mocked tools and consistent state. The output includes the conversation dataset and per-turn metrics, which are then fed into the existing evaluation pipeline.

The Self-Improvement Loop and Tangible Results

This simulation-driven approach integrates into a continuous self-improvement loop: Ship → Observe → Create Robust Evals → Simulate → Pipe Sim/Real Data → Optimize Agent Harness → Ship. This iterative process, leveraging automated prompt optimization and other techniques, allows for rapid experimentation and validation.

Key results and benefits observed by Nubank:

• Dramatic Improvement in TNPS: Customer satisfaction for AI agents has significantly increased, with some exceeding human performance.
• High Sim-to-Real Correlation: Evaluations from simulated data show a high correlation with real production data. Human expert review confirms that 80% of simulated conversations are comparable to real ones, making the data highly usable for both new (“greenfield”) and mature agents.
• Effective Derisking: Simulations caught regressions and issues that could have lowered self-service rates, preventing negative impacts in production.
• Improved Self-Service Rate (SSR): Beyond TNPS, simulations have contributed to a 4% improvement in SSR in one instance, enhancing operational efficiency without compromising customer satisfaction.
• Accelerated Model Testing: The ability to quickly generate evaluation data allows Nubank to rapidly test various open-source models and agent variants, saving weeks of effort and enabling A/B testing of different model versions (e.g., open-source vs. frontier models) with high confidence before production deployment. The team now runs multiple ideas through simulation and only launches A/B tests when satisfied with the simulated output.

Core Takeaways

The path to building continuously self-improving AI agents in an enterprise setting hinges on three critical factors:

1. Simulated Evaluation Data: Generating evaluation data in simulation, rather than solely relying on production signal, is essential to short-circuit release bottlenecks and accelerate development.
2. Closing the Sim-to-Real Gap: Trusting simulation results requires establishing strong offline/online human review processes and metrics to ensure that simulations accurately reflect real-world performance.
3. Aligned Metrics and Reliable Data Generation: The foundation of continuous improvement lies in having metrics that accurately capture desired signals and a dependable method (like simulation) for generating the data needed to compute those metrics.

• Tags:
• AI Agents
• Synthetic Data
• Generative AI
• Evaluation
• Feature
• Testing
• Simulation
• Large Language Models (LLMs)
• Machine Learning Operations (MLOps)
• Nubank
• Snowglobe
• Customer Support AI
• Continuous Improvement