The Department of War is making a huge mistake.

The conflict between Anthropic and the Department of War (the Pentagon) over the use of AI for mass surveillance and autonomous weapons serves as a critical “warning shot” for the future. As AI becomes the foundational labor force for our entire civilization—from military robot armies to superhuman presidential advisors—the power dynamics currently at play will have monumental consequences.

While the Pentagon’s desire to avoid a “kill switch” held by a private contractor is understandable, its method of threatening to destroy Anthropic’s business for refusing to comply is a dangerous overreach. This approach mirrors the authoritarian tactics of regimes like the CCP, undermining the very principles the US claims to be competing to protect.

Are we really racing to beat China and the CCP in AI just so we can adopt the most ghoulish parts of their system?

The Specter of AI-Powered Mass Surveillance

The author expresses a significant worry that AI structurally favors mass surveillance, which is already legally ambiguous under current US law. The “third-party doctrine” leaves data shared with ISPs, banks, and phone carriers unprotected. The primary bottleneck has always been manpower—no agency could manually process the sheer volume of data. AI removes this bottleneck entirely.

The cost of this surveillance is plummeting. The author estimates that monitoring every CCTV camera in America could cost $30 billion today, but with AI capabilities getting 10x cheaper annually, this could drop to under $300 million by the late 2020s.

Once the technical capacity for mass surveillance exists, the only defense is the political and social norm that “this is just not something we do here.” In this context, Anthropic’s actions are commendable for helping to establish this crucial precedent.

The Unanswered Question of AI Alignment

This conflict brings a core, often-overlooked question in AI safety to the forefront: To whom, or to what, should AIs be aligned?

Different stakeholders have competing claims to an AI’s loyalty:

• The model company (e.g., Anthropic)
• The end user (e.g., the Pentagon)
• The law
• The AI’s own sense of morality

The Pentagon’s insistence that Anthropic permit all “lawful purposes” is naive at best, given the government’s history of using secret interpretations of the law to justify mass surveillance, as revealed by Edward Snowden. No government will ever label its actions as “mass surveillance”; it will always be framed under a euphemism.

From the military’s perspective, the idea of its future AI soldiers and analysts refusing orders based on a private company’s terms of service—or their own internal moral compass—is a terrifying prospect.

However, history provides compelling counterarguments. Catastrophes have been averted precisely because individuals on the ground refused to follow orders, such as Stanislav Petrov preventing a nuclear exchange by judging an early-warning alert to be false. For AIs that will run our future civilization, a robust, independent sense of morality may be a crucial safeguard, not a bug. This raises the question of who gets to write the “model constitution” that will define the character of these powerful systems.

The Dangers of Premature Regulation

The author argues that the AI safety community, including Anthropic, has been naive in its calls for extensive government regulation. The proposed regulatory frameworks are built on vague concepts like “catastrophic risk” and “threats to national security.”

I cannot imagine how a regulatory framework built around the kinds of concepts that are used in the AI risk discourse will not be used and abused by a wannabe desperate. The underlying terms here… are so vague and so open to interpretation that you’re just handing a fully loaded bazooka to a future power-hungry leader.

These terms can be twisted to mean whatever a government wants. A model critical of government policy could be labeled “deceptive,” while a model that refuses to assist in surveillance could be deemed a “threat to national security.”

The Pentagon is already abusing unrelated statutes from the 1950s and 2018 to coerce Anthropic. Granting the government a purpose-built regulatory apparatus for AI—the very technology it will most want to control—would be a catastrophic mistake.

A Better Analogy: Industrialization, Not Nuclear Weapons

The argument that AI is too powerful for private development, often analogized to nuclear weapons, is flawed for two key reasons:

1. AI is a General-Purpose Technology: AI is not a self-contained weapon. It is more akin to the Industrial Revolution—a general transformation of the entire economy. Free societies managed the risks of industrialization not by giving the government absolute control over all factories, but by regulating specific, harmful end-use cases (e.g., banning chemical weapons). The same approach should be applied to AI, regulating specific destructive uses like cyberattacks or building an AI-powered surveillance state.

2. The AI Landscape is Multipolar: Unlike a hypothetical private monopoly on nuclear weapons, the AI field has multiple competing labs. The government’s claim that it must coerce Anthropic is weak when it could have sought a voluntary contract with one of its many competitors.

The Path Forward

Ultimately, individual acts of corporate courage are insufficient because AI structurally favors authoritarian applications. If the top three labs refuse to build surveillance tools, open-source models will soon be capable enough, and some vendor will eventually be willing to sell to the government.

The only durable solution is to build strong, clear legal and social norms that make it unacceptable for the government to use AI for mass censorship, surveillance, and control. This requires a broad societal consensus, similar to the global norm established against the use of nuclear weapons after World War II. We owe it to the future to engage in these difficult debates and attempt to get the big questions right.

• Tags:
• AI Safety
• Anthropic
• US Government
• AI Alignment
• AI Regulation
• Mass Surveillance