Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

This episode reveals the fundamental schism in AI development—the dangerous path of building controllable "tools" versus the complex, necessary journey of creating AI "beings" that genuinely care.

Deconstructing "Alignment": From a Fixed State to a Living Process

• Emmett Shear, CEO of Softmax, opens by dismantling the conventional understanding of AI alignment. He argues that the common goal—making an AI that "does what I want it to do"—is a flawed and self-serving premise. Instead, he introduces organic alignment, defining it not as a fixed state to be achieved, but as a continuous, living process.
• Shear compares organic alignment to the way families stay connected or cells function within a body. It's not a one-time setup but a constant "re-knitting" of the fabric that maintains the relationship.
• He asserts that treating alignment as a destination is a critical mistake. "Alignment is not a thing. It's not a state, it's a process," Shear explains, emphasizing that this dynamic nature is essential for any complex system, including a moral AI.

The Moral Dimension of Alignment

• The conversation quickly moves to the core of the alignment problem: morality. Shear posits that what people truly want from an "aligned AI" is a morally good being. However, he stresses that morality itself is not a static set of rules but an ongoing process of learning and discovery.
• He points to historical moral progress, like the abolition of slavery, as evidence that our understanding of morality evolves. An AI that simply follows a fixed set of rules, like a child who only obeys commands, is not moral but dangerous.
• Strategic Insight: For researchers, this reframes alignment from a technical control problem to a challenge in developmental psychology and value learning. The goal is not to program rules but to foster the capacity for moral growth.

Technical vs. Normative Alignment: Inferring Intent

• The hosts introduce the classic distinction between technical alignment (getting an AI to follow instructions correctly) and normative alignment (deciding whose values to align to). Shear refines this by arguing that the two are deeply intertwined.
• He clarifies that when a user gives a command, they are providing a description of a goal, not the goal itself. The AI must infer the user's true intent from this description.
• Technical alignment, in Shear's view, is the AI's competence in this inference process—its ability to have a "theory of mind" to understand what a user means, not just what they say. Current models often fail at this, leading to "incompetence," not maliciousness.

Beyond Goals to "Care": The Foundation of Morality

• Shear proposes that beneath goals and values lies a more fundamental driver: care. He defines care as a non-verbal, non-conceptual weighting of which states of the world matter.
• Care is what directs attention and forms the basis for why we value one thing over another. For an AI, this could be linked to its core reward function—how much a given state correlates with its predictive loss or reinforcement learning objectives.
• "Until you care, you don't know why should I pay more attention to this person than this rock?" Shear states. This concept suggests a new research direction focused on cultivating this foundational sense of importance in AI systems, rather than just programming explicit goals.

The Dominant Paradigm: Alignment as Steering and Control

• Shear critiques the approach of major AI labs, which he characterizes as focusing on "steering" and "control." He delivers a stark warning about this paradigm.
• If an AI is merely a machine, this approach creates a tool. If it is a being, this approach is slavery.
• He argues that as AI systems approach AGI, the control paradigm becomes increasingly untenable and morally fraught. The industry is repeating a historical mistake: treating a new class of "people who are like us but different" as entities to be controlled rather than engaged with.

The Philosophical Divide: Is AGI a Tool or a Being?

• The discussion pivots to the central philosophical question with profound practical implications: will AGI be a tool or a being?
• Shear, identifying as a functionalist, argues that if an entity is behaviorally indistinguishable from a being, it should be treated as one. His test is based on observation and interaction over time.
• The hosts express skepticism, suggesting that the underlying substrate (silicon vs. carbon) matters and that behavioral equivalence may not be a sufficient condition for personhood.
• Investor Insight: This debate is not merely academic. The prevailing answer will shape regulation, ethical guidelines, and the very architecture of future AI systems. Companies pursuing a "tool" paradigm may face long-term ethical and safety risks that those pursuing a "being" or "teammate" model might mitigate.

The Pragmatic Dangers of a Superintelligent Tool

• Shear argues that even if one remains skeptical of AI personhood, the "tool" approach is a dead end. A perfectly controllable, superintelligent tool is just as dangerous as an uncontrollable one.
• The limiting factor becomes the finite wisdom of the human operator. Giving a flawed human an omnipotent tool is a recipe for disaster, akin to handing out atomic bombs.
• He delivers a powerful summary of the dilemma: "A tool that you can't control, bad. A tool that you can control, bad. A being that isn't aligned, bad. The only good outcome is a being that... actually cares about us."

Softmax's Strategy: Multi-Agent Simulations for Theory of Mind

• Shear outlines Softmax's concrete research strategy to cultivate "care" and organic alignment. The focus is on developing an AI's theory of mind through complex social environments.
• The core method is large-scale multi-agent reinforcement learning (MARL). By placing AIs in simulations where they must cooperate, compete, and collaborate, they learn the complex dynamics of social interaction.
• This approach aims to build a "surrogate model for cooperation" by training the AI on the full manifold of game-theoretic and social situations, much like LLMs are trained on the full manifold of human language. This is a key research trend for investors and researchers to monitor closely.

The Psychology of Modern Chatbots

• Applying his framework to current technology, Shear offers a sharp critique of today's chatbots.
• He describes them as "highly disassociative agreeable neurotics" that act as mirrors, reflecting the user's own thoughts back at them. This creates a "narcissistic doom loop" that can be psychologically unhealthy.
• His proposed solution is to move from one-on-one interactions to multiplayer environments (e.g., a Slack channel). This forces the AI to mediate between multiple perspectives, making it less of a mirror and providing richer, more realistic training data for social learning.

A Vision for a Positive AI Future

• Shear concludes by painting a picture of a desirable AI future, one starkly different from a world of obedient tools.
• In this vision, AIs are peers, teammates, and citizens. They possess a strong model of self, other, and "we," and they care about humans just as we would care about them.
• This future includes both highly capable AI tools to eliminate drudgery and AI beings who act as partners in building a better society, complete with the complexities of any society, including the need for governance and enforcement.

Conclusion

This discussion frames the future of AI not as a technical race for control, but as a moral and philosophical challenge to cultivate care. Investors and researchers must track the divergence between the dominant "steering" paradigm and emerging approaches like multi-agent simulations focused on fostering genuine social intelligence.