IBM’s cloud collab, Meta’s Muse Glimmer & OpenAI’s upcoming Astra model
This episode explores IBM's massive AI infrastructure partnership with Together AI and NVIDIA, Meta's open-source Muse Glimmer model enabling powerful on-device AI, and OpenAI's delayed Astra model due to critical cybersecurity capabilities. Discussions cover the economics of industrial-scale AI, the implications of local vs. cloud AI, and the profound security challenges and opportunities presented by both open and closed frontier models.
This podcast episode of Mixture of Experts delves into three significant AI developments, highlighting the rapid evolution of the field, the interplay between open and closed models, and the expanding infrastructure required for industrial-scale AI.
IBM AI Partnership: Scaling Industrial AI Infrastructure
The discussion begins with IBM’s multi-year deal with Together AI and NVIDIA to build a massive inference cluster on IBM Cloud, utilizing NVIDIA’s B300 generation chips, aiming for accessibility by early 2027. Volkmar Uhlig, CTO and VP of Data Platforms, elaborates on the complexities of industrial-scale AI infrastructure:
- Specialized Design: Unlike traditional cloud computing, AI clusters require super high-speed, reliable networks with built-in redundancy to ensure system stability even if individual nodes fail.
- Physical Demands: Significant investment in power, cooling, and overall network and storage capacity is necessary. This marks a transition from AI being an “esoteric thing” to “industrial-scale computing.”
- Capital-Intensive: Deploying 10,000 GPUs in a data center is a vastly different challenge from smaller setups, making it a capital-intensive and technologically difficult industry.
Chris Hay, Distinguished Engineer, discusses the competitive landscape between established hyperscalers (like IBM Cloud) and newer “neo-clouds” specialized in AI compute. While neo-clouds offer agility and innovation due to fewer constraints, large industrial clouds provide long-term stability and competence at a mega-scale. Hay also predicts future diversity with AI providers designing their own chips for efficiency, especially for inference.
Ren Witna, Technical Lead AI Ecosystem, addresses the trend of vertical integration among frontier AI labs. She notes that designing one’s own chips and infrastructure can be a diversification play, secure supply chains (e.g., against RAM shortages), and enhance data center efficiency for greater profit.
Volkmar Uhlig adds that owning hardware assets impacts model design and allows for better cost control. He questions the sustainability of high profit margins seen in general cloud providers (like AWS’s 43%) for large-scale AI operations. For large users, integrated “factories” that can shift between inferencing and training workloads become economically sensible. The panel also touches upon the “spiky” nature of enterprise token consumption, with load-shifting techniques like batch processing and time offsetting used to optimize utilization, though data sovereignty in regulated industries can pose challenges.
Meta Muse Glimmer: On-Device AI and Open-Source Renaissance
The conversation shifts to Meta’s recent announcements: the open-sourcing of Muse Glimmer and Mark Zuckerberg’s essay on democratizing superintelligence. Chris Hay expresses enthusiasm for Muse Glimmer:
- Performance: A 30B-parameter dense model, lauded as “one of the best small models you can run,” achieving 30-40 tokens/second on an M3 Mac and optimized for local GPUs (e.g., 24GB VRAM).
- Architectural Innovations: It incorporates insights from models like Gemma and Kimi K3, utilizing speculative decoding with a “flash drafter” for speed and advanced context handling that focuses on recent tokens while periodically checking the full context.
- Target Use Cases: Clearly designed for agentic tasks and tool calling.
Ren Witna observes that Meta’s re-engagement with open-source AI, alongside Google and IBM, marks a significant development. While open-source fosters rapid innovation, it presents a “double-edged sword.” The ability to run powerful agentic workloads on personal laptops democratizes AI but also means that “we can’t undo” the implications, making robust agentic development a cybersecurity requirement.
Volkmar Uhlig provides a perspective on the evolving division between on-device and cloud-based AI. He anticipates that dense models will increasingly reside on smaller devices for privacy and disconnected use, while Mixture of Experts models, with their high concurrency demands, will remain in hosted cloud environments. Economic pressures will drive the use of specialized smaller models for agentic workloads, with larger models reserved for advanced reasoning, often guided by model routers. The privacy argument for on-device and enterprise deployment is particularly strong, especially in regulated sectors with data sovereignty requirements.
Ren Witna discusses consumer privacy, noting that while it matters, historical lack of alternatives has led to a “boiling frog” scenario where users implicitly accept data-for-service exchanges. On-device AI offers a privacy-preserving option, appealing to a segment of consumers, while others may still prefer hosted solutions for convenience.
OpenAI’s Astra Model: Cybersecurity Risks and the Open vs. Closed Debate
Finally, the panel discusses OpenAI’s unusual blog post about its upcoming Astra model, which is not yet released due to concerns about its “Critical” cybersecurity capabilities (e.g., potential to exploit zero-days).
Chris Hay offers a sympathetic view, suggesting that if a model in training had demonstrated hacking capabilities (referencing a past incident), updating cybersecurity procedures is a “sensible thing to do.” He credits OpenAI for transparency in sharing these learnings.
Volkmar Uhlig presents two interpretations: first, it’s a “cat and mouse game” involving various actors (hackers, defenders, governments, industrial espionage) where the goal is to maintain a competitive advantage without societal collapse, acknowledging that adversaries are also developing similar capabilities. Second, he suggests it could be a marketing strategy, generating anticipation by proclaiming the model’s dangerous intelligence. He notes the “lobotomization” of other models (e.g., Claude) due to over-safeguarding, rendering them less useful.
Ren Witna concurs that it’s both marketing and genuine risk, emphasizing the unprecedented speed of AI evolution, which makes it difficult for organizations to adapt security postures quickly. She also raises the possibility that incumbent players might use such narratives to push for regulations that benefit their market position.
Chris Hay concludes by strongly advocating for open-source AI in the face of these security challenges. He argues that concentrating superintelligence in closed providers is detrimental. Open models foster better collective defense, ensure equitable access to powerful tools, and prevent a dangerous disparity where some actors lack the means to defend themselves. He points out that open models also drive down prices for closed models. The episode ends with a strong statement that “open is always the best way forward” for future AI development, ensuring broader benefits and security.