The AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO

Engram’s Vision: Building AI That Truly Learns From You

Dan Biderman, co-founder and CEO of Engram, outlines the company’s ambitious vision for AI that goes beyond current limitations by deeply learning and internalizing knowledge. Drawing from his background in computational neuroscience and a philosophy of “doing more with less,” Engram focuses on creating highly efficient, continuously updated AI systems.

The Shortcomings of Current AI Approaches

Biderman argues that prevalent methods like Retrieval-Augmented Generation (RAG) and ever-larger context windows eventually break down, particularly at enterprise scale:

• Context Rot: As context windows grow, models become increasingly confused and less accurate. Even at 10 million tokens, this phenomenon persists.
• Cost and Inefficiency: Treating every query as a fresh start, especially with massive corporate data (potentially trillions of tokens), is prohibitively expensive and token-inefficient. Frontier models consuming thousands of dollars per query for common tasks are unsustainable.
• RAG’s Limits: While useful, RAG relies on textual chunks and lacks genuine intuition. It’s like reading a cookbook repeatedly without developing the chef’s ingrained understanding and ability to innovate.
• Compaction’s Lossiness: Context compaction, while improving, is inherently lossy and can lead to forgetfulness over long horizons.

Engram’s Solution: Knowledge Cartridges and Continual Learning

Engram’s core innovation lies in training models to compress and internalize knowledge directly into their weights, rather than relying solely on external retrieval:

• Gradient-Based Learning: By allowing models to “study” a corpus in advance through gradient descent (similar to pre-training), they develop compact representations—called knowledge cartridges—that can be loaded in and out.
• Intuition Over Rote Retrieval: These cartridges enable models to go “beyond notes and recipes” to develop an “intuition” akin to an experienced chef, allowing for extrapolation and innovation. This can be 1000x more compressed than raw text.
• “Destroying Prefill”: Engram’s approach shifts the computational burden from real-time prefill (where models repeatedly read context) to a prior, efficient training phase. This is an application of test-time training or test-time compute, where information is deeply embedded into the model’s parameters, making subsequent inference faster and less confused.
• Neural Memory Trace: Engram aims to create a form of neural memory trace, a lossy process that embeds knowledge into weights, distinguishing it from purely textual compaction.

Enterprise Applications and Holistic Queries

Engram targets enterprises dealing with vast amounts of proprietary data, such as law firms or investment banks. They aim to solve “ambient hard questions” that require holistic understanding across disparate documents, not just keyword searches.

• Example: Answering “which M&A deals haven’t been completed this year?” requires reading and synthesizing information across many client matters, understanding the absence of completion—a task difficult and expensive for current frontier models using RAG.
• This approach mirrors the benefits of pre-training on the entire web, where learning associations within data leads to powerful generalization and inference capabilities.

The Vision for Personal AI and Autonomous Memory

Engram envisions a future where every individual possesses a personalized AI model, or a set of weights, that continually learns from their interactions and data.

• “Tamagotchi Weights”: These models improve the more users engage with them, becoming better for them specifically, rather than for a general audience.
• User Control: Users retain control over these personalized weight sets.
• Long-term Hardware Trends: The increasing inference capabilities of personal device hardware are crucial for this vision.
• Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA, combined with Engram’s cartridges and memory layers, are key to enabling this personalized, continual learning.

What Belongs in Weights vs. Text: An Open Research Question

Determining what knowledge should be internalized (in weights) versus externalized (in text/RAG) is a fundamental open problem, mirroring debates in human memory research.

• Autonomous Decision-Making: The goal is for models to autonomously decide what information to store internally based on salience, frequency, and utility, without explicit manual supervision.
• User Feedback Loop: While models should learn from user feedback (thumbs up/down), the loop must be tight and specific to that user’s model, ensuring their input directly contributes to its improvement, fostering trust.

Token Efficiency and the Future of Intelligence

Biderman emphasizes that intelligence and efficiency are inseparable. “Getting smarter means exerting less energy to solve increasingly harder problems.”

• “Doing More with Less”: While the current AI paradigm has focused on “doing more with more” (scaling compute and data), the next paradigm will involve “doing more with less” to tackle longer-horizon and more complex problems.
• Model Routing: Engram sees routing as a critical component, with their highly trained models acting as intelligent guides, discerning when to use cheaper models or engage powerful frontier models for complex, high-stakes tasks.

Team, Culture, and Hiring

Engram’s team comprises specialized researchers in memory and continual learning, with diverse inclinations spanning mathematics, systems, and AI leadership. They pair experienced PhDs with rising talent.

• Product-Centric Research: Despite its research-heavy foundation, Engram maintains a sober product-oriented culture, recognizing the need to build and distribute products people love.
• Hiring Needs: Engram is actively seeking:
  • Research Engineers: To solve open problems in continual learning, cost-efficient updates, and data learning without model destruction.
  • Infrastructure Engineers: To build systems for millions of endpoints, manage efficient read/write from disk to HBM, and handle the swapping and updating of models across massive, distributed compute footprints.

• Tags:
• RAG
• Feature
• Enterprise AI
• Personal AI
• Long Context
• Continual Learning
• Token Efficiency
• Knowledge Cartridges
• Parameter-Efficient Fine-Tuning (PEFT)
• Neural Memory
• Test-Time Training