Is open source safe? Featuring Mixture of Experts

The conversation explores the nuanced relationship between open source, AI, and security, acknowledging that while open source is a powerful engine for innovation, it is not a “cart blanch solution” and introduces unique risks alongside its benefits.

The Foundational Role of Open Source in AI Innovation

Gabe Goodhart champions open source as a key driver of the current AI revolution. He argues that in the AI era, the gap between scientific research and usable technology has never been smaller. Because science is inherently open, and the underlying math (e.g., tensors) is so close to the functional code, the innovation happens in the open. This tight coupling between open science and utility is why the vast majority of progress originates in the open-source community.

Martin Keen adds that open source is foundational to the entire AI ecosystem, even for closed, proprietary models. He points to open standards like the Model Context Protocol (MCP) from Anthropic (now with the Linux Foundation) and the skill.md agent skill specification. These open standards are used by both open-weight and closed foundation models, demonstrating that “there really is no getting away from the fact that open source is foundational to everything in AI now.”

Open Weights vs. Open Code: A Critical Distinction

A key point of discussion is the distinction between open-source software (the code) and open-source models (the weights).

• Open Source AI Stack: The software used to run and manage AI systems (akin to Linux or Kubernetes) is just software. It has an attack surface that must be meticulously managed. It is securable, but not inherently secure by default.
• Open Model Weights: This presents a different challenge. While closed-source models can have guardrails and monitoring implemented at the software layer, once model weights are public, these safety features can be “obliterated.” Malicious actors can fine-tune the model to remove rejection capabilities and unlock its unfiltered potential. However, this openness is also what fuels rapid architectural innovation, as seen with advancements in linear attention mechanisms.

The Security Perspective: Trust, Transparency, and Threat Models

Jeff Crume, representing the cybersecurity viewpoint, pushes back against the idea that open source is inherently more secure.

• “A Thousand Eyes” is Naive: The classic argument that many eyes make all bugs shallow doesn’t scale to systems with billions of parameters. He notes, “a thousand eyes wouldn’t be nearly enough to look at a billion parameters.” History has shown that critical bugs can remain latent in open-source code for decades.
• Securable vs. Secure: He draws a parallel to Linux, which was once claimed to be immune to malware. Time proved this false. Like Linux, open-source AI systems are securable—they can be made secure with effort—but they are not secure out of the box.
• Trust Requires Transparency: For AI to be trustworthy, it must be transparent. Relying on a closed, proprietary system is “blind faith,” not a valid technical pillar of trust. However, transparency alone isn’t a guarantee of security.
• An Old Problem on Steroids: The threat of reverse-engineering is not new. Just as one could decompile proprietary executables to find vulnerabilities before, AI now puts this capability “on steroids,” allowing for much faster analysis of machine code or model behavior.

The Expanding Attack Surface: Agentic Systems

The conversation highlights a “net new vulnerability” that arises with autonomous AI agents. Gabe Goodhart expresses significant concern that an agent loop is essentially a code interpreter where the “code” is any text it ingests from the internet.

“We’re all very familiar with this concept of, you know, having to sandbox your untrusted code. Well, now the internet is your untrusted code.”

This creates a massive attack surface for prompt injection, where unstructured, untrusted data can trigger an agent to take malicious actions based on the tools it has access to. While current security efforts focus on sandboxing execution and managing tool permissions, Goodhart notes that almost no one is addressing the core problem of “managing the wild world of untrusted data that’s going to get slurped in at runtime.”

The Interpretability Double-Edged Sword

A final challenge discussed is the lack of interpretability in large language models, a problem that affects both open and closed systems. It’s currently impossible to fully understand how a model arrives at a specific output.

However, open-weight models offer a potential path forward. Researchers can “unbox” these models by directly manipulating weights to observe the impact on outputs, treating them like a knowledge graph. This could lead to breakthroughs in explainability. This capability is a double-edged sword: the same techniques that allow for interpretability also allow malicious actors to more easily understand and manipulate the model for harmful purposes.

• Tags:
• Foundation Models
• AI Security
• Prompt Injection
• Open-Source AI
• Trustworthy AI
• Model Access