Why AI Agents Shouldn't Replace Your Fraud Models
Varant Zanoyan, original author of the Chronon feature platform, introduces 'agentic experimentation'—a pattern where AI agents improve high-stakes ML systems without making live decisions. He explains how Chronon solves key challenges like infrastructure sprawl, safety, and reproducibility through a semantic API, branch-based isolation, and compute reuse, enabling agents to safely create production-ready pipelines for human review.
High-stakes systems—such as fraud detection, credit underwriting, and search ranking—cannot tolerate the risks associated with full, autonomous agentic decision-making. These systems demand auditability, low latency, and high QPS, which are often at odds with the nature of current agentic models. An effective alternative is agentic experimentation: using AI agents to improve the underlying rules and machine learning models rather than replacing them entirely.
In this model, an agent’s role is to iterate on the ML system by creating new features, training new model versions, and deploying a complete, end-to-end pipeline to a development environment. A human then reviews these changes, decides whether to launch an A/B test, and ultimately promotes the changes to production. For this workflow to be effective, the agent’s output must be production-ready, meaning it is reviewable, reproducible, and safe.
The Origin and Evolution of Chronon
The Chronon open-source project was born at Airbnb to solve the iteration speed bottleneck in their payments fraud detection system. The core challenges were the difficulty of building streaming features, scattered feature logic across different systems (Spark, Flink), and inconsistencies between online and offline data.
Chronon’s solution was to provide a single, declarative API for defining features. From this definition, an automated engine orchestrates both training data generation (offline backfills) and production inference pipelines (online streaming, serving), guaranteeing consistency between them. This reduced the time to ship a new feature from months to days.
Its success led to adoption across Airbnb for trust and safety, search ranking, and even non-ML use cases like real-time metric calculations. This journey culminated in a partnership with Stripe, where Chronon now powers 100% of its charge path models. Today, it is also used by OpenAI for Sora 2 personalization and Netflix for content ranking.
Key Challenges in Agentic Experimentation
Building agents that can safely modify production ML systems presents three primary challenges:
1. Infrastructure Sprawl
Without the right abstractions, an agent would need to interact with a complex web of infrastructure—Spark, Flink, Kafka, Kubernetes, Airflow—to perform an end-to-end experiment. This approach is:
- Error-prone: Can lead to silent failures and incorrect data.
- Unreviewable: A human cannot reasonably review agent-generated code across multiple infrastructure components and repos.
- Unsafe: The agent could interfere with production workloads, add excessive load, or rack up enormous costs.
Solution: Infrastructure Automation Chronon provides a high-level, semantic API where the agent defines what it wants to do (e.g., “take this data, transform it this way, and add it to that model”). Chronon’s engine then automates the underlying infrastructure management, creating production-ready training and serving pipelines while ensuring consistency.
2. Safety and Efficiency
Agent experiments must not interfere with live production systems. At the same time, recomputing thousands of features for every minor change is computationally expensive and slow.
Solution: Branch-Based Isolation & Compute Reuse Chronon implements two key mechanisms to solve this:
- Resource Isolation: Agent experiments are tied to Git branches. Chronon automatically routes all compute and storage for a given branch to isolated resources, guaranteeing that the agent’s work cannot impact production infrastructure.
- Compute Reuse: To avoid wasteful recomputation, Chronon uses partial aggregate caching. When an agent adds a new feature (e.g., a 7-day windowed aggregation), the system reuses pre-computed partial aggregates from the raw data that were cached when calculating existing 30-day or 60-day features. Unchanged features are simply copied from production to the development environment. This makes experimentation both safe and highly efficient.
Because Chronon acts as a shared feature repository, any compute performed by one agent can be reused by other agents or humans across the organization, creating powerful economies of scale.
3. Reproducibility
For an agent’s output to be reviewable and trustworthy, its results must be perfectly reproducible. An experiment run today must yield the exact same data as the same experiment run tomorrow.
Solution: Semantic Hashing Chronon employs a semantic hashing mechanism under the hood. It understands the semantics of the data pipelines and guarantees that any job, when rerun, will produce a bit-for-bit identical output. This provides the necessary foundation for a human to confidently review and ship an agent’s proposed changes.
By providing this data foundation built on infrastructure automation, safe experimentation, and reproducibility, it becomes possible for agents to effectively and safely accelerate iteration on the most critical, high-stakes AI systems.