Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

Engram, co-founded by Dan Biderman and Jessy Lin, posits that memory and continual learning represent the true bottleneck in making AI models more useful, rather than raw intelligence. Their core premise, dubbed “always training,” advocates for deeply baking specific knowledge into a model’s weights, analogous to how pre-training imprints general knowledge, instead of relying heavily on context windows or Retrieval-Augmented Generation (RAG).

Internalizing Knowledge vs. Externalization

The current industry standard for providing models with new information involves extensive context engineering, such as crafting large prompts or using multi-turn conversations. While recognizing the utility of such tools, Engram argues this approach falls short of true memory and continual learning. They highlight two main issues with externalized memory:

1. Cost and Complexity: The sheer volume of tokens generated daily (potentially tens of millions per individual) makes searching and re-reading computationally expensive and confusing for models without significant breakthroughs.
2. Lack of Deep Understanding: Simply providing context at test time doesn’t allow models to internalize information. An experienced employee understands company-wide initiatives, processes, and culture implicitly, a level of understanding that superficial context provision cannot achieve.

Engram’s Approach and Architecture

Engram’s solution involves training

• Tags:
• Large Language Models
• Feature
• Computational Neuroscience
• Memory
• AI Architecture
• Continual Learning
• Personalized AI
• Model Weights
• Adapter Fine-Tuning
• RAG Killer
• Token Efficiency