Inference at Scale:Breaking the Memory Wall

Sid Sheth, CEO of d-matrix, details their memory-centric approach to AI inference hardware, focusing on their Digital In-Memory Compute (DIMC) architecture. He explains how DIMC, an augmented SRAM technology, minimizes data movement to solve the memory bottleneck, delivering significant gains in latency and energy efficiency, particularly for the 'decode' phase of large language models.

February 12, 2026 5 min read

Inference at Scale:Breaking the Memory Wall — source: https://www.youtube.com/watch?v=5WqMSwdSs7k

The Bet on Cloud Inference and Memory-Centric Design

Founded in 2019, before the rise of ChatGPT, d-matrix made a contrarian bet on data center and cloud inference. While many startups focused on edge computing or the highly competitive training market dominated by NVIDIA, d-matrix identified a gap for a dedicated, efficient inference solution in the cloud.

The founding team anticipated that AI models, particularly transformers like BERT and the emerging GPT-3, would continue to grow in size, making memory access the primary bottleneck. Their first-principles analysis of the inference workload revealed it to be a repetitive, parallel compute problem heavily dependent on memory access. This led to their core strategy: integrating memory and compute as closely as possible to build a fundamentally more efficient architecture.

The Memory Bottleneck: HBM vs. SRAM

The choice of memory technology is critical for AI hardware. Sid Sheth provides a clear breakdown of the trade-offs:

High-Bandwidth Memory (HBM): Originally developed for High-Performance Computing (HPC) and later adopted for AI training, HBM acts like a “highway with many lanes,” providing high-bandwidth access to a processor. While effective for the massive, parallel data needs of training, HBM is a poor fit for mainstream inference due to three key factors:
- Cost: It remains an expensive technology.
- Energy: It is very power-hungry.
- Bandwidth Limits: The pace of AI model growth is outstripping HBM’s ability to scale its bandwidth, making it “not fast anymore” for cutting-edge inference needs.
SRAM (Static RAM): d-matrix, along with other early players like Grok and Cerebras, initially focused on SRAM for its speed. However, on-chip SRAM capacity is limited. Recognizing that models would quickly outgrow a single chip, d-matrix designed its system with a two-tiered memory approach from the start, using a large on-chip SRAM tier and a second, larger LPDDR memory tier to accommodate extremely large models and the exploding KV-cache sizes associated with long contexts.

Prefill vs. Decode: The Two Phases of Generative Inference

Generative AI models operate in two distinct phases, which have different hardware requirements:

Prefill (The “Thinking” Phase): When a model receives a prompt, it processes the input and generates the internal context (KV cache). This phase is compute-intensive.
Decode (The “Speaking” Phase): The model then generates the response token by token. Each new token requires accessing the entire KV cache. This phase is memory-intensive and highly sensitive to latency. A slow decode phase results in a poor user experience, with long delays between words.

d-matrix’s architecture is particularly well-suited for accelerating the memory-bound decode phase, where low latency is paramount.

Digital In-Memory Compute (DIMC): The Core Innovation

d-matrix’s key technology is Digital In-Memory Compute (DIMC). It’s a novel architecture that turns memory itself into a compute fabric.

How it Works: A traditional SRAM cell uses six transistors (6T) to store one bit of data. d-matrix augmented this design, creating a ten-transistor (10T) cell that can both store a bit and perform a single-bit multiplication.
The Benefit: By embedding compute directly within the memory array, model parameters (weights) can be stored and used for matrix math calculations without being moved. This minimization of data movement is the key to efficiency. It saves a tremendous amount of time and energy, directly addressing the three most precious resources: money, time, and energy.

This approach allows all rows of the SRAM to be activated simultaneously, creating a dataflow engine with much higher throughput than a traditional SRAM.

System, Scale, and Performance Trade-offs

The d-matrix solution is built like a set of Lego blocks, scaling from the chip to the rack level.

A single PCIe card contains 248 DIMC blocks.
A server rack holds 64 of these cards.
A single rack can run models up to 100 billion parameters entirely from its fastest SRAM/DIMC tier, a capacity 5-10 times greater than other solutions on the market.

Sheth is candid about the engineering trade-offs. The d-matrix architecture is optimized for low-latency, medium-throughput workloads, such as reasoning models or interactive video generation. It is not designed to compete with GPUs on extremely high-batch, high-throughput applications, where latency is less of a concern.

Software, Portability, and Go-to-Market

d-matrix acknowledges NVIDIA’s powerful software moat with CUDA, which is the default for researchers and developers experimenting with new models. D-matrix targets the next phase: production deployment and optimization.

Their strategy involves two primary engagement models:

Provider-Led Porting: For customers like neo-clouds or sovereigns, d-matrix takes a list of their 20-30 most important models and writes the optimized low-level kernel libraries to ensure they run efficiently on the hardware.
Customer-Led Porting: For sophisticated hyperscalers with proprietary models, d-matrix provides the tools and access for them to write their own kernels and port their models themselves.

The company’s hardware is designed to be “plug-and-play,” fitting into standard server form factors from partners like Supermicro. This collaborative, ecosystem-focused approach differs from more vertically integrated players and is why d-matrix has chosen not to build its own cloud.

The Future of Inference

Looking ahead, Sheth highlights several key areas to watch:

Agentic and Multimodal AI: These workflows will significantly increase the demand for low-latency, efficient inference.
Scale-Up Interconnects: As systems grow, the network connecting servers within a rack becomes critical. While NVIDIA has NVLink, the rest of the ecosystem is coalescing around Ethernet-based standards like Broadcom’s EAN.
Openness: d-matrix has embraced open standards from the beginning, using PyTorch and MLIR on the software side and standard interfaces like PCIe and Ethernet on the hardware side.