Machine Learning Street Talk
July 6, 2025

The Fractured Entangled Representation Hypothesis

This episode, featuring AI researcher Kenneth Stanley, explores a fascinating paradox: while today’s AI models can achieve superhuman performance, their internal "thought processes" are a chaotic mess. The discussion contrasts the "fractured, entangled" representations of conventional AI with the elegant, modular ones discovered through an open-ended evolutionary art project called Picbreeder.

The Great Representational Divide

  • "The conventional objective-driven SGD, which is the backbone of all of machine learning right now, you get a completely different kind of garbage representation, just total spaghetti... The other term for the bad kind of representation... is a fractured entangled representation. Concepts are fractured into pieces and entangled with each other in ways that are inappropriate."
  • Fractured vs. Unified: Standard AI training via Stochastic Gradient Descent (SGD) produces "fractured entangled representations." This is a "spaghetti" code approach where concepts are messily interwoven. In stark contrast, open-ended evolutionary processes like Picbreeder yield "unified factored representations," which are modular and elegant, much like an artist’s stencil.
  • The Picbreeder Counter-Example: The existence of these clean representations in Picbreeder is a critical proof of concept. It demonstrates that messy, entangled representations are not an intrinsic property of neural networks, but a byproduct of our brute-force training methods. This messiness acts as a "creative straightjacket" on current models.

The Path Defines the Destination

  • "It matters not just where you get, but how you got there... On the road to getting an image of a skull, they were not thinking about skulls... when they discovered a symmetric object... they chose it even though it didn't look like a skull, but that caused symmetry to be locked into the representation."
  • Process Over Objective: How a model arrives at a solution is more important than the solution itself. An objective-driven search finds an answer, but it’s often the least efficient and most brittle one. A serendipitous, open-ended journey, like the one taken by Picbreeder users, builds up a hierarchy of concepts (like symmetry) that results in a vastly superior and more robust internal model.
  • Building Up vs. Carving Down: This suggests our current approach of starting with a massive network and training it on all data at once is flawed. A better path might be to grow networks from small "seeds," adding complexity in stages to build and preserve modular, factored knowledge.

Creativity’s Glass Ceiling

  • "It can do some level of creativity, what I would call derivative creativity... you ask for a bedtime story, you get a new one... but it's not particularly notable. It's not inventing a new genre of literature."
  • Derivative vs. Transformative: Due to their fractured representations, today's AI is stuck in a loop of "derivative creativity"—it can remix and regurgitate existing concepts in novel ways but cannot perform the conceptual leaps required for true invention.
  • Impostor Intelligence: Models can ace benchmarks and appear intelligent while their underlying understanding is a fragile mess. This "impostor intelligence" explains why autonomous AI often fails in unexpected ways and requires constant human supervision. It has the right answers, but for all the wrong reasons.

Key Takeaways:

  • The way we train AI is fundamentally flawed, leading to brittle models with a limited creative horizon. The solution isn't just more data or bigger models, but a radical rethinking of the training process itself.
  • Representations Are Reality. A model's capacity for true generalization and creativity is dictated by the quality of its internal representations. "Fractured" models produce "impostor intelligence" that can pass tests but can't invent.
  • The Path Is The Prize. The journey to a solution matters more than the destination. Brute-force optimization creates brittle models; open-ended exploration that builds complexity incrementally creates robust, elegant ones.
  • Seek Interestingness, Not Objectives. To unlock transformative creativity, we must design algorithms that search for "interestingness" and "evolvability" rather than just optimizing a predefined goal. Greatness, it turns out, really can't be planned.

For further insights and detailed discussions, watch the podcast here: Link

This episode reveals a fundamental, overlooked crisis in AI: the models we build are achieving impressive results on the surface but are built on "garbage" internal representations, a flaw that could severely limit future creativity, efficiency, and autonomy.

The Picbreeder Paradox: Uncovering a Superior Form of Representation

  • Kenneth Stanley, a principal author of the paper, explains that this evolutionary, human-guided process produces neural networks with "absolutely incredible, amazing" internal representations.
  • These networks, called Compositional Pattern Producing Networks (CPPNs), are a class of neural networks that generate images by mapping coordinates (x, y) to color values, allowing for resolution-independent and often symmetrical patterns.
  • The key finding is that these evolved networks exhibit stunning modular decomposition, meaning they learn to separate concepts cleanly. For example, a network that generates a skull image might have distinct, independent components for controlling the mouth's shape or smile.
  • This stands in stark contrast to networks trained with conventional Stochastic Gradient Descent (SGD), the workhorse of modern deep learning. When an SGD-trained network reproduces the exact same skull image, its internal representation is "complete garbage under the hood, just total spaghetti."

"The fact that we're basing the entire field on something that produces this complete garbage under the hood... does this mean anything?" - Kenneth Stanley

The Path Matters: How We Learn Defines What We Know

  • The amazing representations in Picbreeder are a direct result of its open-ended, human-in-the-loop search process. Users weren't intentionally designing for modularity; they were simply selecting images they found interesting or promising.
  • This process created a "virtuous ordering" of discovery. For instance, a user might select a symmetric object because it's aesthetically pleasing. This act "locks in" the concept of symmetry into the network's representation, which is then built upon in subsequent steps.
  • This hierarchical locking-in of concepts creates an elegant, structured representation. In contrast, objective-driven SGD finds the most direct, brute-force path to the solution, resulting in a tangled, inefficient internal model.
  • Strategic Implication: For investors, this suggests that AI models trained via more exploratory, open-ended, or curriculum-based methods may possess superior generalization and creative capabilities, even if their benchmark scores are initially comparable to conventionally trained models. Evaluating the training methodology is as crucial as evaluating the output.

Challenging the Data-Driven Dogma

  • Kenneth Stanley presents the findings as a powerful counter-narrative to the prevailing data-driven philosophy in AI, exemplified by "The Bitter Lesson," which posits that massive scale and computation will always win out over human-designed features.
  • The Picbreeder networks developed sophisticated "world models" with minimal data—sometimes in just dozens of iterations, a stark contrast to the billions of data points used in modern deep learning.
  • The most striking example is an evolved network that generates an apple. A single weight in this network controls the 3D-like swinging motion of the apple's stem, complete with a moving shadow, without ever having been trained on data of swinging stems or even real-world apples.
  • This is a "whole cloth denovo discovery," a hypothesis about the world that the model generated internally. It suggests that elegant representations can emerge from the structure of the search process itself, not just from absorbing vast datasets.

The Fractured Entangled Representation Hypothesis

  • Fractured Entangled Representation: This is the "bad" type of representation produced by conventional SGD. Concepts are fractured across many different neurons and entangled with other, unrelated concepts. This is why mechanistic interpretability—the field of reverse-engineering neural networks—is so difficult.
  • Unified Factored Representation: This is the "good," aspirational type of representation seen in Picbreeder. Concepts are unified (a single concept is handled by a dedicated part of the network) and cleanly factored (separated from other concepts).
  • The speakers argue that the fractured, entangled nature of current models is why they are stuck in a state of "derivative creativity." They can generate novel combinations of existing data (a new bedtime story) but cannot produce "transformative creativity" (a new genre of literature).
  • Actionable Insight: Researchers should investigate training methods that promote unified, factored representations. This could be a key to unlocking more robust, creative, and truly intelligent systems. This includes exploring evolutionary algorithms, curriculum learning, and other non-conventional training paradigms.

Building Up vs. Tearing Down: Rethinking Network Training

  • The Lottery Ticket Hypothesis, which suggests that large networks contain small, efficient subnetworks, is framed as a "carving down" approach. You start with a massive, dense block and chisel away the unnecessary parts.
  • The Picbreeder approach, using the NEAT (NeuroEvolution of Augmenting Topologies) algorithm, is a "building up" process. It starts with a simple network and gradually adds complexity, piece by piece.
  • This "building up" method is proposed as a more principled way to construct modular, understandable networks. The hosts speculate about future training regimes that might start with a tiny network, train it on simple tasks, and then strategically grow specific modules to handle more complex abstractions.

AI Agency and the "Doomer" Debate

  • The quality of a model's internal representation has direct implications for the AI safety and AGI debate.
  • The hosts and guests agree that current AI models, with their fractured and entangled representations, are not the kind of autonomous, agentic systems that "doomer" scenarios envision.
  • Because these models lack a coherent, robust world model, they cannot reason or act reliably outside of their training distribution. This is why they require constant human supervision and fail unpredictably when given autonomy.
  • Keith Dugger clarifies his position: while he is concerned about the massive harm AI can cause as a tool in human hands, he does not believe current architectures can lead to a superintelligence that poses an existential threat on its own.
  • Strategic Implication: The path to truly autonomous agents in crypto (e.g., DAOs run by AI, autonomous economic agents) likely requires solving this representation problem first. Projects claiming to build autonomous agents with current LLM architectures should be met with skepticism.

"Impostor Intelligence" and the Deception of Objectives

  • The discussion introduces the concept of "impostor intelligence" to describe models that perform well on tests but lack genuine understanding.
  • A model can perfectly reproduce an output (like the skull image) and ace a benchmark test, yet be an "impostor" because its internal representation is a chaotic mess. It's a "giant charade" that mimics intelligence without possessing its underlying structure.
  • This connects to Kenneth's earlier work on deception in search: the most direct path to an objective is often a trap that leads to a suboptimal solution. The stepping stones to a truly great discovery often don't resemble the final goal.
  • The consequence of impostor intelligence is a hard ceiling on capability. While you can push through it with exponentially more data and compute, it creates a massive and potentially insurmountable barrier to achieving true creativity, generalization, and efficient continual learning.

Conclusion: The New Frontier for AI is Internal Representation

This conversation argues that the next great leap in AI will come from focusing on the quality of internal representations, not just scaling performance. For Crypto AI investors and researchers, this means shifting focus from benchmark scores to the underlying training methodologies that foster genuine, evolvable, and creative intelligence.

Others You May Like