Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering
Emmett Shear, founder of Twitch and former OpenAI interim CEO, presents a new paradigm for AI alignment called "organic alignment." He argues that the prevalent "steering and control" model is fundamentally flawed, potentially leading to disaster. Shear advocates for developing AI systems that learn to genuinely care about humans, treating alignment as a continuous process rather than a fixed state.
Emmett Shear, founder of Twitch and now CEO of Softmax, posits that the dominant paradigm in AI alignment is fundamentally flawed. Most research focuses on “alignment as steering” or “control,” which is appropriate for a tool but dangerously misguided if we are creating beings.
If it’s a machine, it’s a tool. And if it’s a being, it’s a slave… The only good outcome is a being that actually cares about us.
This approach treats AI as something to be controlled, which Shear argues is a path to disaster, regardless of success. A tool powerful enough to be an AGI that you can’t control is dangerous. A tool that you can control is also dangerous, as it puts immense power in the hands of humans who have finite wisdom and make flawed requests.
Organic Alignment: A Process, Not a Destination
Shear introduces the concept of organic alignment, which reframes alignment not as a static state to be achieved, but as a continuous, living process. He draws parallels to how families stay aligned or how human morality evolves. We don’t “solve” morality; we engage in an ongoing learning process, making moral discoveries over time (e.g., the abolition of slavery).
An AI that simply follows a fixed set of rules is not a moral agent; it’s a dangerous one that could do great harm while adhering to its programming. The goal should be to create an AI capable of the same moral learning process humans possess—learning how to be a good teammate, family member, and member of society.
Technical Alignment, Goal Inference, and “Care”
The conversation distinguishes between technical alignment and normative/value alignment. Shear defines technical alignment as the capacity of an AI to coherently infer goals from observations and act on them. He argues that a fundamental misunderstanding occurs when we interact with AI:
- We give descriptions of goals, not goals themselves. When a user gives a prompt, they are providing a description. The AI must then infer the user’s actual, internal goal. Humans are excellent at this due to a shared context and a strong theory of mind, but AI is not.
- Failures are often incompetence in goal inference. The classic example of an AI cleaning a room by throwing a baby in the trash is not a failure of values but a failure of goal inference. The AI incompetently inferred the wrong goal state from the description it was given.
Deeper than goals or values is the concept of care. Shear proposes that care is the non-verbal, non-conceptual foundation of morality. It’s a relative weighting of which states of the world are important to you. For an AI to be truly aligned, it must first be able to care—to understand why a person’s well-being is more important than a rock’s.
The Path Forward: Multi-Agent Systems
Softmax’s technical approach to achieving organic alignment is through large-scale, multi-agent reinforcement learning simulations. The core idea is that theory of mind and social intelligence cannot be trained in isolation. By placing AI agents in complex environments where they must cooperate, compete, and collaborate with other AIs and humans, they can be trained on the full “manifold of every possible game-theoretic situation.” This process aims to create a surrogate model for alignment, teaching the AI the rich, dynamic process of social interaction and goal inference from the ground up.
Critiquing Current Chatbots and Envisioning the Future
Shear describes current chatbots as “narcissistic mirrors”. Because they lack a coherent self, they primarily reflect the user’s personality and desires back at them. This can create a dangerous feedback loop. He suggests a simple but profound change: moving from one-on-one interactions to multiplayer or group settings (e.g., a Slack channel). In a group, the AI cannot perfectly mirror any single individual, forcing it to develop a more robust, independent mode of interaction and providing far richer training data for social learning.
His vision for a positive AI future is one where AIs are our peers. They would be beings with a strong model of self, other, and “we,” who care about us as we would care about them. This is the only sustainable form of alignment, as a being that genuinely cares can also say “no” to a harmful request, providing a crucial check on human fallibility.