The Limits of Today’s AI Models
Karan Goel, CEO of Cartesia, discusses the fundamental limitations of Transformer architectures, arguing they behave more like retrieval systems than learning systems. He explains how State Space Models (SSMs) enable compression and abstraction, and why Cartesia is tackling multimodal intelligence by first solving for voice AI, aiming to develop a transferable 'recipe' for end-to-end representation learning.
The Ceiling of Transformer Architectures
Current AI progress, largely built on the Transformer architecture, may be approaching a fundamental ceiling. While Transformers excel at processing information within a given context, they are inherently limited by their design. Karan Goel, CEO of Cartesia, characterizes them as “retrieval-oriented machines.”
The core mechanism of self-attention, with its keys, values, and queries, allows a model to access its entire context or prompt in a raw, uncompressed format. This is excellent for tasks requiring high-fidelity recall of specific facts but prevents the model from forming compressed, abstract representations. This inability to compress information is a significant bottleneck for achieving more human-like intelligence, which requires reasoning over vast amounts of information and experiences accumulated over long periods.
State Space Models and Intelligence as Compression
As an alternative, Goel points to State Space Models (SSMs), a class of recurrent models that operate on a different principle. The core idea underpinning SSMs is that a fundamental primitive for intelligence is compression. To reason over large amounts of information—whether it’s text, audio, or visual data—an agent must abstract and consolidate its understanding into a more compressed form.
SSMs and Transformers occupy opposite ends of a spectrum:
- Transformers: Offer perfect, high-fidelity recall of a limited context window. They are retrieval-focused.
- SSMs: Compress information, which results in a “fuzzier” representation of the world. This loss of fidelity is a feature, not a bug, because the pressure to compress forces the model to build abstractions.
This tension has led to the emergence of hybrid architectures that combine the strengths of both approaches, attempting to balance high-fidelity retrieval with long-term, abstract reasoning. The ultimate goal is to discover the ideal architecture for true multimodal learning over extended timescales.
Cartesia’s Strategy: Voice AI as a Recipe for Multimodality
While often perceived as a “voice AI company,” Cartesia’s focus on audio is a strategic wedge into the much larger problem of multimodality. Multimodality is not just about combining images and text; at its core, it’s about mapping a continuous signal to discrete symbols (e.g., audio waves to text).
Cartesia’s core belief is that solving the audio-text problem in the right way creates a transferable “recipe” for other multimodal domains, such as video, robotics, and other sensor data. The fundamental challenges are shared across these fields.
Rethinking Tokens and End-to-End Representation Learning
A critical, shared problem in multimodality is tokenization—how to represent a continuous signal in discrete tokens for a model to process. Traditionally, this involves hand-engineered pipelines (e.g., converting 16 kHz audio signals into 50 Hz representations).
Cartesia’s research aims to eliminate this step entirely. The goal is to build models that learn representations directly from raw signals in an end-to-end fashion. The model itself should learn the necessary hierarchies of abstraction internally, without relying on a predefined tokenization scheme. This approach of “getting rid of tokens” is the key part of the recipe that, if successful, would be transferable to any other signal-based modality, just as Transformers work on different types of tokens.
From Research Lab to Product-Driven Startup
Transitioning from an academic lab to a research-driven startup introduces a crucial shift in mindset. While academic research thrives on exploring many different visions, a startup requires a singular vision that is ruthlessly executed.
Product development serves as a “truth serum” for research. Customers don’t care about the novelty of the underlying architecture; they only care about performance. This imposes a level of intellectual honesty, forcing the team to prove that a new approach is genuinely better for the end-user, not just more interesting from a research perspective. This discipline prevents the premature deployment of novel models for their own sake and ensures that research is grounded in real-world impact. Ultimately, even research-heavy companies are governed by the fundamental “laws of startup gravity.”