The CEO Behind the Fastest-Growing AI Inference Company | Tuhin Srivastava

Baseten’s journey began in 2019, long before the current AI boom. The initial focus was on serving data scientists with small models for internal use cases. The market shift in 2022, catalyzed first by ChatGPT setting new standards for AI products and then more significantly by Stable Diffusion, created a massive expansion. This shift wasn’t a pivot for Baseten but a “refocusing.” The core infrastructure had already been built, but the market dynamics changed in three key ways:

1. Models: Shifted from small models to large, computationally demanding models.
2. Use Cases: Moved from internal tools to production-grade applications where SLAs and reliability are critical.
3. Users: Transitioned from data scientists to engineers with the agency to build and modify infrastructure.

A key lesson from the early years was the importance of not scaling the team too quickly. By remaining a small team (only 18 people when they raised their Series B), Baseten retained the agility to kill product lines and refocus the entire company in a matter of weeks when the market opportunity became clear.

Differentiating in the Inference Market

While the inference of generic, open-source models on shared endpoints is becoming a commodity, the real challenge and opportunity lie in dedicated deployments for custom and fine-tuned models. Every customer workload is slightly different, with unique models, inputs, outputs, SLAs, and compliance requirements. Baseten differentiates across three main pillars:

1. Infrastructure: Providing reliability and scale is paramount. This involves ensuring high uptime (“many nines”), fault tolerance, and elasticity. The infrastructure is designed around customer needs, not cloud constraints, enabling complex requirements like multi-cloud deployments to acquire sufficient capacity (e.g., thousands of B200s) or routing traffic to specific regions to minimize hops.
2. Performance: This is about speed—specifically metrics like Time to First Token (TTFT), time per output token, and overall throughput. While the best-performing runtimes will likely become open source over time, being best-in-class on performance is a crucial requirement.
3. Developer Experience & Platform: Inference is treated as a software problem. Customers often manage a portfolio of 3 to 20 models, each with different hardware, scaling needs, and traffic patterns. Baseten provides the software layer to manage this complexity, similar to a CI/CD tool for model deployment.

The Technical Challenges of LLM Inference

Modern inference presents a two-layer problem: the infrastructure level and the runtime level.

• Infrastructure-Level Problems: These involve scaling a workload across potentially thousands of GPUs. Key challenges include stateful routing to reuse KV caches (ensuring a user’s subsequent requests hit the same GPU), managing capacity across different cloud providers, and handling global traffic routing.
• Runtime-Level Problems: This is about optimizing how fast a model runs on a given GPU. This is where frameworks like vLLM, TensorRT-LLM, and SGLang come into play, employing techniques like quantization, flash attention, and continuous batching. The talent pool for this research-intensive work is extremely limited.

Comparing Open-Source Runtimes

Different runtimes offer trade-offs between usability, control, and raw speed:

• TensorRT-LLM (TRT-LLM): Backed by NVIDIA, it offers the highest performance at scale, especially once a model has been out for a few months (“day 90”). It’s the lowest-level option, providing maximum control but requiring more expertise.
• vLLM: Prioritizes usability and ease of use. It allows developers to get started with inference quickly but may come with a performance hit unless it’s expertly configured.
• SGLang: Sits somewhere in between vLLM and TRT-LLM, balancing usability and performance.

The existence of multiple frameworks reflects a nascent market that has not yet converged, similar to the early days of front-end web development.

The Hardware and Model Ecosystem

While Baseten experiments with various chips, NVIDIA GPUs remain the standard for production workloads due to the unmatched reliability, versatility, and developer ecosystem of CUDA. New hardware providers face a significant challenge: the market is moving so fast (with a ~60-day planning horizon) that the 6-9 month chip development cycle makes it difficult to keep up. Even so, customers with high-volume workloads often demand the latest hardware like B200s, as the performance gains (e.g., a 40-50% speedup for video) can create a fundamentally better user experience.

On the model front, customers are shifting from closed-source APIs (like OpenAI and Anthropic) to open-source models for several reasons:

• Cost: Open-source can be more economical at scale.
• Control & Reliability: Dedicated infrastructure provides more control over uptime and performance.
• Data Privacy: Companies want to prevent their data from being used to train third-party models.
• Customization: The need for purpose-built or fine-tuned models for specific tasks.

Ultimately, the future is likely a hybrid one, with companies using a mix of both closed and open-source models.

The Future of Inference

Inference is arguably the most important and foundational layer of the AI stack. As models become integrated into every application, the demand for inference will continue to grow exponentially. In a future with AGI, inference might be the “last market” that exists, as it’s the fundamental requirement for any model to run. This market is also deepening, as concepts like tool use, code execution sandboxes, and reinforcement learning are increasingly being framed as inference problems.

• Tags:
• LLMs
• Feature
• Inference
• VLLM
• AI Infrastructure
• GPU
• Baseten
• Model Deployment
• TensorRT-LLM