Dedicated Inference
A walkthrough of CoreWeave's three inference paths (Serverless, Dedicated, and Inference on CKS), covering when each fits, how a multi-agent research assistant uses OpenAI-compatible endpoints across model types, and live demos of gateways, deployments, and Kubernetes-based vLLM, Dynamo, and LLM-D serving.
The speaker frames the inference landscape as a parallel to the software development lifecycle. Teams usually start by experimenting with general-purpose proprietary models, where control is limited to inference hyperparameters. As they move toward production, priorities shift to controlling scale and cost, and to accuracy and business-specific responses, which typically means custom models, a choice of runtime and APIs, and much more control over the stack. CoreWeave maps this progression onto three products: Serverless Inference, Dedicated Inference, and Inference on CKS. The speaker stresses that they are layered, since Serverless Inference is built on Dedicated Inference, and Dedicated Inference is built on Inference on CKS, so customers can move between them without rebuilding.
Multi-agent research assistant use case
To explain why the three paths matter, the speaker describes a multi-agent research assistant. Such a system has several agents: some ingest documents, some reason across them to answer complex questions, and a router sends each query to the right sub-agent. Each role can call for a different kind of inference. A simple out-of-the-box open-source model can handle routing. A fine-tuned or specialized model can ingest business-specific documents and generate grounded responses. A multimodal model can handle images and video. The key architectural point is that all of these endpoints share a common interface, the OpenAI-compatible endpoint, so the agent framework only changes the URL it calls and not the client code.
Serverless Inference
Serverless Inference is pay-per-token access with no cluster management, aimed at shipping AI applications quickly. It is built with Weights and Biases in mind and includes tracing, evaluations, and observability, along with autoscaling and high availability out of the box. It also supports custom LoRA models. The speaker positions it for teams that want SaaS-level simplicity but a clear path to deeper control later, such as AI engineers who need tracing and evals to iterate with confidence as traffic and requirements grow.
In the demo, the Weights and Biases model garden lists model cards, and a playground lets users test a model with simple queries. The more useful part, in the speaker’s view, is the trace view. It shows the chat output, the call history with inference hyperparameters, and a summary of tokens generated, latency, and cost. The speaker considers these metrics essential during experimentation. The same endpoints can also be reached through the standard OpenAI SDK.
Dedicated Inference
Dedicated Inference is presented as the control path for production. Customers deploy their own model weights on a managed runtime, with explicit GPU selection and a choice of open runtimes. CoreWeave handles autoscaling, routing, and high availability. The product offers both real-time and batch endpoints and uses infrastructure-aligned pricing, so costs track the underlying hardware. The speaker identifies three audiences: teams shipping custom models who want managed operations without giving up transparency, enterprises that need predictable SLOs, governance boundaries, and integration with existing tools, and organizations scaling to multi-node inference who want explicit GPU and runtime control without replatforming.
The demo runs in a Marimo notebook using the Python SDK. The workflow has four steps. First, check which zones are available for deployment. Second, create a gateway, which the speaker describes as a single-tenant, isolated endpoint that handles authentication, authorization, traffic splitting, and request routing. An optional reserved capacity claim can be attached at this stage. Third, create a deployment, which sits behind a gateway. One gateway can front multiple deployments, and all of them can be called through it. The example deploys a GPT-OSS 120B model and specifies the runtime and its version, the number of GPUs per replica, and minimum and maximum replica counts. The speaker then scales the maximum from 4 to 8 replicas to show that deployments can be updated in place. Last, the endpoint is called through the OpenAI SDK, and the notebook includes delete and cleanup operations.
Inference on CKS
Inference on CKS (CoreWeave Kubernetes Service) gives customers a fully managed environment for running their own distributed inference stack on dedicated, single-tenant GPU nodes with Kubernetes-native control. It supports explicit GPU selection, multi-node distributed inference, and runtimes such as LLM-D, NVIDIA Dynamo, and custom runtimes. CoreWeave publishes tutorials for deploying LLM-D and Dynamo with single commands. Infrastructure-level tuning, network optimization, and isolated boundaries come with the platform. The speaker says this path fits ultra-high-scale and latency-critical workloads, very large models, long-context deployments, platform teams building shared inference foundations, and organizations with strict regulatory or tenancy requirements, such as fintech, telecom, insurance, and healthcare.
The Dynamo demo starts from an already running Dynamo installation. A small Qwen3 model is deployed with vLLM by applying a Kubernetes YAML manifest. The speaker then shows the running pods, including the frontend, the gateway, and the vLLM workers, and calls the resulting OpenAI-compatible endpoint with either the SDK or curl. The transcript gives the model size as 6 billion parameters, which is probably a transcription error for the 0.6B variant. The LLM-D demo follows the same pattern. A YAML file specifies the number of GPUs, the GPU type, and a GPT-OSS 120B model. After the manifest is applied, the pods come up and LLM-D exposes an OpenAI-compatible endpoint.
Closing remarks
The speaker’s main point is portability. Because the products are built on each other, a customer can start with one and move to another with relatively little effort. Customers cited as using CoreWeave inference products include Goodfire, Perplexity, Method.ai, and Clay (the transcript says ‘client’, which is most likely Clay). Further resources are linked from a QR code and the CoreWeave Inference page.