Baseten CEO Tuhin Srivastava on Custom Models, and Building the Inference Cloud

The AI market is rapidly expanding as companies realize AI can be embedded everywhere, leading to a massive demand for inference. This growth is fueled by the increasing capability of open-source models, which have crossed a chasm in baseline performance, and the mainstream adoption of post-training techniques for specialization. This has led to a “longtail of models” where customers own and customize their intelligence, with Baseten’s 30x growth serving as an index on the entire application layer’s expansion.

Why the Application Layer Wins

The independent application layer will persist and thrive due to its ability to capture unique user signals that are inaccessible to frontier model labs. The core defensibility for companies lies not just in a model, but in encoding this user signal into complex workflows.

A key example is Abridge, an ambient medical scribe integrated deeply into clinician workflows within hospitals. The value is created by the clinicians’ edits to notes and the subsequent actions within the EMR system. This feedback loop generates a rare and valuable reward signal. Companies with access to such signals can post-train models to create long-horizon, agentic systems that are highly specialized and difficult for a general-purpose frontier model to replicate. This principle also applies to areas like customer support, where a single ticket can trigger a dozen actions, creating a complex workflow ripe for a specialized model.

The Dominance of Custom Inference

The vast majority of inference workloads are not on vanilla open-source models. Over 95% of the tokens served on Baseten come from “dedicated inference,” where customers have made modifications to the model. This customization happens in two primary ways:

1. Performance: Compiling models differently or applying techniques like quantization to improve speed and cost.
2. Quality: Post-training models on proprietary data and user signals to specialize them for a specific use case.

Recognizing this trend, Baseten acquired a research team that was previously a customer specializing in post-training. This move accelerates the market’s shift toward custom models and highlights the deep link between inference and post-training. The way a model is trained directly affects how it must be quantized and optimized for inference, making them two sides of the same coin. The ideal state is a continuous loop: inference generates data and user signals, which are used for evaluation, which then feeds a post-training process to improve the model.

Navigating the GPU Supply and Capacity Crunch

The narrative around the GPU supply crunch understates its severity. There is “very, very little slack compute available,” with Baseten running its own large clusters at uncomfortably high mid-90s utilization.

To manage this, Baseten has built a single runtime fabric that spans 18 different clouds and 90 clusters globally. This technology, initially designed for reliability and low latency, has become a critical strategic advantage for sourcing supply. It allows Baseten to onboard a new provider in any country and integrate them into the fabric in less than half a day.

The challenge is not just a supply crunch but also a “supplier and operationally crunched” market. Many new providers lack the experience to run data centers at the SLA required for high-stakes inference workloads.

The market dynamics have also shifted contract terms significantly. Securing a large cluster of next-generation GPUs (like B200s) now requires a 3-to-5-year contract with a 20-30% TCV (Total Contract Value) prepayment. This makes a low cost of capital a significant advantage when acquiring capacity.

The Winning Formula: Software, Compute, and Operations

While GPUs-as-a-service is a commodity, an inference platform with a strong software layer is incredibly sticky. The software that manages custom models, ensures reliability, and optimizes performance creates a deep moat.

However, in a world of constrained compute, the primary strategic advantage is access to the compute itself. The ability to secure and own GPU capacity is fundamental, a strategy being pursued even by the major labs.

The future of hardware is likely a “multi-chip world,” with specialized chips for different parts of the inference process, such as decode-specific chips (like Groq’s LPU). Despite this, NVIDIA’s dominance is hard to overstate due to its robust supply chain, the CUDA ecosystem, and the speed at which developers can move on its platform.

Scaling Lessons and Future Outlook

Scaling a company 30x in a year reveals that all theoretical edge cases become real, recurring problems. The team constantly deals with systems-level and kernel-level issues that only manifest at extreme scale. This requires building a strong “operations culture” where reliability is paramount, embodied by a “pager culture” where the entire team is on call to ensure inference never goes down.

Looking forward, the increasing efficiency and decreasing cost of inference will not lead to less consumption. It will trigger Jevons paradox: as intelligence becomes cheaper, developers will embed much more of it into applications. Agents will run for longer, and user experiences will become richer and more capable. This ultimately leads to a “concierge everything” future, where every consumer has access to personalized, intelligent tools for everything from healthcare to education, and companies that fail to integrate intelligence into their workflows face an “extinction moment.”

• Tags:
• Large Language Models
• Supply Chain
• Inference
• AI Infrastructure
• GPU
• Open Source Models
• Baseten