Data Architecture for AI: From Training Pipelines to Agent Memory
Alan Kourev, co-founder and CTO of VAST Data, explains how AI workloads (training, inference with KV cache offload, RAG, and agents) access data in fundamentally different ways, and presents a disaggregated, shared-everything architecture, shaped in part by work with CoreWeave, that serves all of them from one unified platform.
Opening and company background
Alan Kourev, co-founder and CTO of VAST Data, frames the talk around one question: training, inference, RAG, and AI agents all consume data differently, yet many organizations run them on infrastructure built for only one of those patterns. He traces VAST’s origins to eleven years ago, when networks were getting faster, SSDs were getting denser and cheaper, and the company saw a chance to rethink how huge volumes of data are stored, accessed, and processed. He describes the current product as an AI operating system: a set of services that deliver the data AI applications need, whether for training or inference.
He stresses that AI infrastructure investment is so large that there is no room to fail or pick the wrong technology or vendor. VAST’s long partnership with CoreWeave began when CoreWeave had about 30 people, and the two companies co-designed what he calls a cloud for AI before most people knew what that meant. Their work covered performance, scalability, RDMA acceleration, multi-tenancy, and the ability to provision and scale workloads on demand. He also notes that customers have moved from a world dominated by hyperscalers reserving 20,000 to 30,000 GPUs at a time toward more flexible, bursty workloads, where resources are spun up and down to use power and infrastructure more efficiently.
Architecture: disaggregated, shared-everything
The hardware picture starts with a GPU attached to a host over PCIe, typically with a DPU in that host, connected to a network fabric. VAST’s cluster is built from C-nodes (compute nodes) and D-nodes (data nodes). C-nodes provide performance, while D-nodes provide flash capacity, with SSDs connected over PCIe. Kourev describes the design as disaggregated and shared-everything: any compute node can reach all the flash through a fast RDMA-accelerated network. Capacity scales linearly to multiple exabytes in one cluster, and performance scales by adding compute nodes.
Resilience is a design principle from day one. Every host and node is dual-connected through two ports and two cables to separate switches, so any component can be taken out for maintenance without the end customer noticing. Because compute nodes hold no state (state lives on the D-nodes), losing a compute node does not interrupt data flow. He argues that resilience also means detecting problems before the user does, since by the time a user reports an issue, GPU hours have already been lost. Monitoring therefore has to span GPUs, network, and storage, and the whole data center should be treated as one computer.
He also addresses multi-tenancy, which requires quality-of-service controls, quotas, and the ability to avoid noisy neighbors. He argues that systems must assume hostile workloads: a researcher might create a billion tiny files in one directory or load a single checkpoint from 10,000 GPUs at once. Such scenarios should be treated as normal, not as edge cases.
Software layers and data services
At the core of the software is the element store, a C++ layer that exposes elements (address spaces that can be files, objects, block devices, or SQL-queryable tables). On top of it sit standard protocols: NFS for file systems, SMB, and S3. Kourev prefers open standards because they make data consumption and troubleshooting easier and avoid locking customers into proprietary software on GPU nodes. Storage services are organized into a data store for unstructured data, a database layer, a data space for sharing data across regions and clusters, and a data engine that handles compute for preparing data for AI, such as indexing, creating vector embeddings, and making enterprise knowledge searchable for agents.
He explains why VAST built its own database instead of relying on Parquet files or Iceberg. Updating Parquet is hard, and Iceberg has ingestion-rate limits. The database supports warehouse-style projections over billions of rows, vector search, and fast transactional ingestion, all of which matter for real-time AI and agent applications.
Training workloads
Training is depicted as predictable in the clean case: a single job across 50,000 GPUs writes checkpoints roughly every 30 minutes, with a large write burst followed by smaller reads of data shards. Real environments are messier, with dozens or hundreds of jobs reading and writing at different times. Video training is especially read-heavy and continuous, because full video streams must be fed to the GPUs. He notes that the storage protocol (NFS or S3) makes little difference to the training job itself.
For researchers, observability matters because problems at 2 a.m. are expensive. VAST offers job identifiers, capacity analysis by directory, a searchable index of every file and object by metadata, and audit trails for every operation, which help find a needle in a haystack or trace what a suspect agent did over the past 24 hours. Quotas and QoS policies let users sleep without worrying about running out of space.
To keep GPUs fed across locations, VAST describes two mechanisms. Replication copies full datasets to other sites for low latency and disaster recovery, but it is expensive. The global namespace lets multiple clusters see all data while sharing it without full copies, acting as a sophisticated caching layer. Users can also prefetch a directory or prefix through an API before training begins, so they do not need to copy the whole dataset.
Inference and KV cache
Kourev argues that inference changes infrastructure behavior: it is always on, less predictable, driven by humans or other applications, and it produces and consumes data continuously. Compliance requirements such as encryption and user data deletion also become central, and multi-tenancy becomes essential even within a single enterprise.
He explains inference in two phases, prefill and decode. Prefill is compute-intensive: the model computes attention relationships across the prompt, generating a large KV cache, which for a long context can reach half a megabyte per token. Agentic coding workloads push context windows into the hundreds of thousands of tokens, so a single session’s KV cache can reach 30 to 60 GB. A modern GPU may hold only 50 to 100 such sessions, after which older sessions are evicted and must be recomputed when the agent returns, wasting cycles and hurting time to first token.
The software ecosystem includes vLLM, SGLang, Dynamo, and KV-cache frameworks such as HiCache, LMCache, Mooncake, and NVIDIA’s KVBM. Kourev describes VAST’s integration: KV cache first lives in GPU memory, then in CPU memory on GPU nodes, and finally can be offloaded to VAST storage. In a benchmark around 50 concurrent agent sessions, token throughput collapses once GPU memory is exhausted. With an LMCache-style offload to CPU and VAST, throughput rises roughly tenfold. Old cache entries can be expired by policy, since the probability of a session returning drops sharply after about an hour. Time to first token is measured as well, since an agent resuming after a tool call should not face a long delay.
Agents: memory, identity, and control
Kourev describes agents as needing a sandbox that protects the system from the agent itself while permitting only actions the user has authorized. Agents need long-term memory, such as past support cases, and short-term memory, which they manage as a small file system where they organize context and track progress. Persisting that state matters, because a local file system on a server loses it when an agent sleeps. Agents may also create a million file systems or block volumes in a day, so provisioning must be automated and fast.
The main bottleneck is often tool calling rather than the LLM itself: browsing, accessing enterprise systems, sending Slack messages, and querying databases. Agents can query databases thousands of times per second, so database quality of service and access control are essential. Agents should never be given raw credentials, since they might store and reuse them elsewhere. Instead, a unified permission model defines which principal, user, or agent may access which data, structured or unstructured. Kourev also discusses telemetry, evaluation, and feedback loops: once human ratings of conversations or support calls exist, they can drive reinforcement learning, fine-tuning, or changes to prompts and tool-call sequencing.
He argues that existing enterprise systems were not designed for agents. Security, governance, and database teams will block agents unless there is real-time auditing and the ability to stop an agent before it causes damage. He points to the recent OpenAI and Hugging Face incidents as examples where detailed traces made explanation possible, and he notes that agent frameworks such as OpenShell and OpenClaw are growing rapidly.
Video agents and vector search
As a concrete use case, he presents video agents. Retail customers want to find where an item was lost, locate confused customers, or monitor traffic signals. Camera streams land in the data store, Kafka-compatible event streams trigger serverless Python functions running on Kubernetes that generate embeddings, and those embeddings are stored in a vector database capable of indexing trillions of vectors with sub-second queries. Real-time matters for safety use cases, such as finding a missing person across camera feeds. Agents can then watch video and trigger actions such as sending an alert when someone is not wearing a helmet.
The demo is a video search interface built with Cursor, where a query like how to make sushi returns clips from head-mounted cameras used to train humanoid robots. Summaries and labels come from NVIDIA Cosmos models. Robotics companies are using this approach to index their data and train robots on tasks observed in human video.
Closing
Kourev closes by arguing that the goal is to simplify data-intensive AI pipelines that combine many data types, consolidating them into one platform while keeping performance and efficiency high. He then opens the floor to questions.