The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Byung-Gon Chun, founder and CEO of FriendliAI, discusses the evolving landscape of AI inference, emphasizing the shift from chat-based applications to complex agentic workflows. His talk highlights how FriendliAI, a company born from research that invented continuous batching, has rebuilt its inference cloud specifically to address the unique challenges and opportunities presented by AI agents.

The Rise of Agents and Open-Weight Models

Chun asserts that 2026 will be the year agents enter massive production, driven by two key trends: the exponential growth of AI agents across software, operations, and knowledge work, and the maturation of open-weight models. He notes that open-weight models have now reached “frontier” capabilities, rivaling closed frontier models, but at a significantly lower token cost.

To illustrate this, FriendliAI conducted a concrete experiment: building a tower defense game using a coding agent with two different models. A closed frontier model (Anthropic’s Opus 4.8) cost approximately $1.50 for the task, while an open-weight model (GLM 5.2 on FriendliAI) completed the same usable-level task for 27 cents, representing a 5.6 times cost reduction. This demonstrates the economic promise of open-weight models for agentic applications.

Agentic Inference: A Different Problem

Chun argues that agentic inference is fundamentally different from traditional chat inference. In chat, the basic unit is a single request, and latency is measured by how fast one response is obtained. For agents, the basic unit is a task, which involves many model calls and tool calls, often running autonomously for minutes or hours. Users care about the completion of the entire task, not the latency of individual requests.

Key characteristics of agentic workflows include:

• Loop-based execution: Agents typically operate in a plan -> act (tool call) -> observe -> repeat loop, alternating between LLM inference and non-LLM tool executions.
• Growing context: Every observation is appended to the context, leading to much longer prompts and completions that grow as the task progresses.
• Shared Prefixes: Consecutive agent steps often share a substantial prefix in their input. Recomputing this identical prefix repeatedly is a significant waste of computational resources.
• Interleaved tool work: Non-LLM operations are integrated between model calls.
• Unpredictable call patterns: The number of model calls depends on the input, making fixed request rate planning difficult.
• Metric: The real metric is end-to-end task latency, not single request latency.

FriendliAI’s Rebuilt Inference Cloud for Agents

FriendliAI has engineered its inference stack specifically to optimize for end-to-end task latency. This specialized cloud is built on four core pillars:

1. Prefix Caching: Given that agent steps share large prefixes, the key-value (KV) for this prefix is computed once and cached. Subsequent steps reuse this cached KV, processing only the new suffix. This significantly improves time-to-first-token and reduces redundant compute, with benefits escalating as tasks run longer.

2. KV Cache Management: To ensure efficient caching, FriendliAI employs:

   • Frugal memory management to maximize active context on GPU memory.
   • KV contigation to reduce memory footprint.
   • Hierarchical caching across GPU memory, host memory, and disks to extend beyond GPU limits.
   • Distributed caching, allowing a single prefix to be served across multiple replicas.

3. Cache-Aware Routing: At a global cluster scale, a naive load balancer might destroy cache locality by distributing requests evenly. FriendliAI’s cache-aware router intelligently sends a request to a replica that already has the required prefix cached. This transforms a cold prefill into a cache hit while still balancing overall load.

4. Agent-Aware Optimization: Recognizing that most existing systems schedule each LLM call independently, FriendliAI’s optimizer understands the agent-level context. This enables better decisions, such as:

   • Preempting the right work.
   • Speculatively prefilling context for likely next steps.
   • Making more informed cache eviction decisions based on the agent’s overall program.

Impact and Adoption

By integrating these innovations, FriendliAI has achieved significant performance gains. In a test using their GLM 5.2 model with kilo code to create a simple mobile game, FriendliAI completed the same end-to-end task two times faster than another well-known inference provider. Furthermore, a customer testimonial from Hilo Code, a popular AI coding tool, reported that in a split test of GLM5 usage, FriendliAI was consistently seven times faster with a significantly lower error rate compared to other third-party providers and direct usage from the model lab G.A.I..

FriendliAI offers flexible deployment options, including serverless Model APIs, dedicated endpoints for isolated deployments with guaranteed SLAs, and Bring Your Own GPU (BYOG) for running their stack on user infrastructure.

Conclusion

In summary, three core messages resonate from FriendliAI’s work:

1. Frontier open-weight models are making production agents economically scalable.
2. Agentic inference is distinct from chat, demanding optimization for end-to-end task latency.
3. FriendliAI provides a purpose-built inference cloud for agents, delivering fast, reliable, and cost-effective performance.

• Tags:
• AI Agents
• Production AI
• LLM Serving
• Open-Weight Models
• KV Cache
• Inference Optimization
• Prefix Caching
• Continuous Batching
• Agentic Inference
• Cache-Aware Routing
• Agent-Aware Optimization
• Task Latency
• FriendliAI
• Byung-Gon Chun