Olive from MiniMax walks through MiniMax M3, an open-weight model built for agentic coding, 1M token context, and native multimodality from the first training step. The talk centers on MSA, a GQA-aware block-level sparse attention design co-developed with GPU infrastructure in mind, and on why training text and vision together from scratch beats adding vision later.
Eric Nguyen, CEO of Radical Numerics, details the journey from early genome language models to Omni, a system that excels in DNA reading, writing, and reasoning. Omni outperforms specialized models in disease-variant prediction, particularly in previously challenging non-coding regions. The discussion also covers the critical need for AI-native biosecurity defenses, addressing how models can design biological sequences that evade current detection systems, and the broader vision for General Biological Intelligence.
Kevin Madura introduces Recursive Language Models (RLMs), a paradigm shift where an LLM treats its context as an interactable object within a REPL, rather than mere tokens. This allows for direct computation, recursive decomposition of problems by delegating to sub-models, and significantly mitigates context rot, outperforming traditional RAG, agents, and tool calls. Use cases range from complex data analysis and long document processing to code auditing and harness optimization, hinting at a future where models are natively RLM-aware.
Engram co-founder Dan Biderman discusses building AI that learns from users, critiquing long context, RAG, and compaction's limitations. He introduces Engram's approach of compressing knowledge into "cartridges" and model weights via continual learning and gradient-based updates, aiming for "intuition" over retrieval. The vision extends to personal, "Tamagotchi" AI models and addresses the critical need for token efficiency and "doing more with less" in both enterprise and personal AI.
Martin Keen explains how Long Context and Cache Augmented Generation (CAG) serve as powerful alternatives to RAG for providing external knowledge to LLMs. This summary details the mechanics of each approach, the role of the KV cache, the practical application through prompt caching, and the trade-offs in performance, cost, and latency for real-world AI workloads.
Martin Keen compares Retrieval Augmented Generation (RAG) with the emerging long context window approach in LLMs. He analyzes the pros and cons of each, from infrastructure simplicity and retrieval accuracy to computational costs and the 'needle in the haystack' problem, providing guidance on when to use each solution.