How Open Source Became AI's Backbone | Inferact with a16z
Simon Mo, CEO of Inferact and lead maintainer of vLLM, discusses how open-source AI, exemplified by vLLM, transformed into critical infrastructure. The conversation highlights the technical complexities of serving LLMs, the evolving economics and licensing of open-weight models, the need for control over guardrails, and the rapidly disappearing capability gap between open and proprietary AI.
Simon Mo, co-founder and CEO of Inferact, alongside Matt Bornstein and Elena Burger, explores the pivotal role of open-source AI, particularly vLLM, in shaping the modern AI landscape. They discuss how open-source AI evolved from a research curiosity into critical infrastructure, driven by the unique challenges of serving large language models (LLMs) and the growing demand for control among developers.
The Rise of Open-Source AI and vLLM’s Critical Role
Serving LLMs is fundamentally different from traditional ML workloads, primarily due to their computational intensity, reliance on accelerators like GPUs, and the need to manage non-deterministic outputs and complex batching/scheduling. This spurred the development of specialized inference engines like vLLM. Initially a project to accelerate slow open-source demos, vLLM evolved to address a “pile of unsolved problems” in LLM serving.
The transition of open-source AI into critical infrastructure became evident around 2023, coinciding with the widespread adoption of tools like GitHub Copilot and ChatGPT. While closed-source solutions initially led, application companies realized that to build truly innovative AI, beyond mere wrappers, they needed control over their models for fine-tuning, custom guardrails, and optimized inference. vLLM sits at the core of this, acting as an “inference engine” that translates raw GPU power into functional intelligence endpoints. It supports over a thousand model architectures, often on “day zero” of their release, and works closely with hardware vendors like Nvidia, AMD, Google, and Amazon, frequently serving as a benchmark for their newest chips. This co-design process ensures models are not just released but are operationalized effectively for diverse use cases, from edge devices to large-scale voice and coding agents.
Economics, Licensing, and Sustainability of Open-Weight Models
The “Open Weights Letter,” signed by Inferact and many other industry leaders, underscores the commitment to protecting open AI development against a world dominated by proprietary APIs. The discussion highlights a crucial dynamic: the need for control versus cost. While cost has become a significant factor more recently due to high token spending, control has always been paramount. Companies desire control over system performance, data retention, security, compliance, and the ability to meet specific SLAs, which proprietary APIs often cannot guarantee.
Funding for open-weight model development is a growing concern. Historically, open-source software relied on donations, but training frontier AI models requires billions of dollars in compute resources. This has led to an evolution in licensing terms. Moving beyond the permissive Apache 2, models like Llama, Minimax, and Kimi K3 now include commercial clauses, requiring agreements for high-volume or specific derivative uses. This is seen as a necessary step for economic sustainability, akin to the pharmaceutical industry’s R&D model, ensuring continued investment in groundbreaking research and development.
Guardrails, Moderation, and the Challenge of Proprietary APIs
The conversation touches on the “Hugging Face incident,” where a Chinese open-source model was used to contain a cyberattack originating from a rogue, unboxed proprietary model. This event illuminated the inherent limitations of proprietary API guardrails: their arbitrary nature, potential for false positives, and difficulty in enforcement. Simon shares an example where Anthropic models, due to their strict guardrails, blocked legitimate frontier AI research by Inferact’s developers, causing them to switch to models like Kimi K3, which offers similar quality with more controllable guardrails. This underscores a critical takeaway: for trusted and specific use cases, open-weight models allow users to define and manage their own guardrails, preventing legitimate applications from being unjustly blocked.
The Future: Closing the Gap Between Open and Proprietary Models
Looking five years ahead, Simon Mo argues that the capability gap between open-weight and frontier proprietary models is rapidly disappearing, if not already non-existent. He contends that differentiation is increasingly about distribution and go-to-market strategies rather than fundamental model capabilities. Key drivers of progress are the quality of training data, the environments built for model self-improvement (e.g., Moonshot AI’s front-end coding environment for K3), and the expertise of brilliant researchers globally. He cites the example of Kimi K3 removing rotary positional embeddings, with the inventor of the technique himself contributing to its removal, as evidence of profound research leading to open advancements.
The discussion also briefly touches on the “distillation debate,” concluding that distillation (copying knowledge from larger models) is likely incidental rather than a critical cornerstone of current progress. Instead, the real progress stems from dedicated research teams, innovative algorithms, diverse data, rich learning environments, and ample compute resources.
Building a Company from Open Source
Inferact’s journey from an open-source project (vLLM) to a company reflects a philosophy of “open-source first.” The company aims to build necessary tools in the open and then focus on closing the “last mile” to deliver value, ensuring customers and partners achieve the best outcomes from these critical technologies.