Fast & Asynchronous: Drift Your AI, Not Your GPU Bill // Artem Yushkovskiy
Delivery Hero presents "Asya", an open-source framework that replaces traditional AI pipelines with a distributed, asynchronous actor model. This paradigm shift dramatically lowers GPU costs and improves scalability by treating each processing step as an independent, auto-scaling microservice on Kubernetes.
The conventional approach to serving AI, often centered around synchronous POST /predict endpoints and rigid orchestrators like Kubeflow, fails to scale efficiently for complex, multi-step GenAI workloads. This model leads to significant challenges, including API rate-limiting, cascading failures, and high operational costs, with engineering teams spending up to 80% of their time just maintaining the system. While self-hosting models can reduce vendor costs, the underlying architectural issues of traditional pipelines remain a primary bottleneck.
A more robust paradigm is to dismantle the concept of a monolithic pipeline and embrace a distributed system of stateless, asynchronous actors. This talk introduces Asya, an open-source framework developed at Delivery Hero that embodies this paradigm shift, moving from brittle DAGs to a resilient and highly scalable actor-based architecture.
The Asya Framework: An Asynchronous Actor Model
Asya’s core principle is the complete decoupling of processing steps. There is no central orchestrator; instead, the system consists of independent actors that communicate via messages. This design is highly effective for complex AI workflows where steps have vastly different computational requirements and execution times.
The architecture is built around a few key concepts:
- Message-Driven Core: The “pipeline” logic is not defined in code but is carried within the message itself. Each message contains:
- Route: A list of actors the message must visit.
- Payload: A data blob that is progressively enriched by each actor. Actors add information rather than modifying or deleting existing data, ensuring downstream actors have full context.
- Independent, Auto-Scaling Actors: Each actor is a microservice that scales independently from zero to N based on the length of its input queue (managed by KEDA). This “zero-to-infinity” autoscaling ensures that expensive resources like GPUs are only allocated when there is active work, leading to dramatic cost savings and optimal resource utilization.
- Dynamic Routing: The framework includes a special “Router” actor that can dynamically modify a message’s future route. This enables sophisticated conditional logic, allowing an LLM judge, for example, to decide the subsequent processing path for a given request. The framework prevents actors from rewriting a message’s history to avoid unintended loops.
- Built-in Resilience: Error handling is a first-class citizen. If an actor fails, the message is automatically routed to a dead-letter queue (DLQ) or a dedicated error-handling actor, which can implement retry logic or persist the message for manual review.
Developer Experience: Bridging Data Science and Platform Engineering
Asya is designed with a clear separation of concerns to cater to both data scientists and platform engineers:
- For Data Scientists: The experience is simplified to writing pure Python functions. An actor is just a function that accepts a dictionary (the payload) and returns a dictionary. This abstracts away all underlying infrastructure, including Kubernetes, dependencies, and scaling logic.
- For Platform Engineers: Management is done declaratively through a Kubernetes Custom Resource Definition (CRD). Defining an
AsyncActoris straightforward, and the framework automatically injects the necessary sidecars and configurations to integrate the actor into the system.
To bridge the gap between the asynchronous backend and synchronous clients, Asya provides a stateful HTTP gateway. This allows external applications to interact with the system through a simple request-response API, while the underlying execution remains fully async and scalable.
Use Cases and Comparisons
Asya is not a replacement for all AI serving patterns. It shines in near-real-time scenarios where end-to-end latency is measured in seconds or minutes, not milliseconds. It is ideal for large-scale, complex batch operations like the image enhancement service at Delivery Hero, which involves dozens of steps from pre-processing and analysis to model inference.
The framework is fundamentally different from orchestrators like Kubeflow or Metaflow, as it eliminates the central point of control and failure. While it shares concepts with other actor-based systems like Dapr or Ray Serve, Asya is specifically tailored for GenAI workloads on Kubernetes, providing a lightweight and focused solution. It is also seen as complementary to tools like KServe, with plans to integrate by injecting sidecars into existing deployments to bring them into the Asya ecosystem.