If You Can't See Inside, How Do You Know It's THINKING? [Dr. Jeff Beck]

Defining Agency: A Matter of Computational Sophistication

From a mathematical modeling perspective, there is no fundamental structural difference between an agent and an object like a rock. Both can be described as executing a policy—a mapping from inputs to outputs. The true distinction lies in the degree of sophistication of the internal computations. An agent is an object with highly complex internal states that represent information over long timescales and execute sophisticated, context-dependent policies.

While philosophical notions introduce concepts like intentionality, a more practical definition hinges on how a policy is computed. Key characteristics we attribute to advanced agents, such as humans, include:

• Planning: Internally simulating future consequences of various actions.
• Counterfactual Reasoning: Considering alternative possibilities and outcomes.
• Goal-Oriented Behavior: Acting to achieve specific objectives.

These are all features of the policy computation process. A simple lookup table could, in principle, produce the same outputs, but the underlying mechanism is what defines agency.

The Black Box Problem: Observing Agency from the Outside

A significant challenge is identifying agency from a purely external perspective. If you only observe the input-output behavior (the policy), it’s nearly impossible to definitively prove that an entity is engaging in internal planning versus executing an incredibly complex, pre-computed function.

You’ll never know for sure in any meaningful way like whether or not it’s it’s just doing a function transformation or whether it’s engaged in planning and counterfactual reasoning.

The most pragmatic approach, echoing Daniel Dennett’s “intentional stance,” is to adopt the simplest computational model that explains the observed behavior. If the most parsimonious model is one that assumes the entity is performing planning and counterfactual reasoning (e.g., a model with 40 parameters) compared to a direct function transformation (e.g., a model with millions of parameters), then it is useful to treat it as if it were an agent. Agency, in this view, becomes a measure of the sophistication of the internal states required to model the entity’s behavior effectively. This also implies that agency requires a physical embodiment; a perfect simulation of a brain, disconnected from a body and the world, is a model of an agent, but not an agent itself.

Energy-Based Models (EBMs) vs. Traditional Networks

Energy-Based Models offer a different paradigm from standard feed-forward neural networks, with the core distinction lying in the application of the cost function.

• Traditional Networks: The cost function is applied only to the final output, and optimization occurs solely over the model’s weights.
• Energy-Based Models: The cost function operates on both the final output and the internal states (latent variables) of the model. This necessitates a dual optimization: one minimization finds the energetic minimum of the internal states, and a second minimizes the prediction error.

This process is deeply connected to Bayesian inference. The energy function acts like a prior probability distribution over the latent variables, and the model optimizes with respect to all of them. The Variational Autoencoder (VAE) is a canonical example of an EBM. Its cost function includes not only the reconstruction error (input vs. output) but also a term that penalizes the internal latent representation for deviating from a prior distribution (e.g., a Gaussian distribution).

This dual optimization of weights and internal states is conceptually similar to “test-time training,” where a subset of model weights are treated as latent variables and updated based on a new, unseen input.

JEPA: Learning Abstract Representations in Latent Space

Yann LeCun’s Joint Embedding Predictive Architecture (JEPA) is motivated by the idea that predicting every single pixel in a future frame is often unnecessary and computationally wasteful. For robust, abstract understanding, it’s more efficient to learn in a compressed, latent space.

The core idea of JEPA is to:

1. Take an input (e.g., an image or video clip).
2. Compress it into a low-dimensional embedding.
3. Predict the embedding of a future or masked part of the input, rather than the raw pixels.

This approach faces the challenge of “mode collapse,” where the model can learn a trivial solution by mapping all inputs and outputs to zero. To avoid this, non-contrastive learning techniques are used. Unlike contrastive methods that rely on pushing negative samples apart, these methods use regularization techniques (e.g., Barlow Twins, VICReg) to prevent the embeddings from collapsing while encouraging them to capture informative features. This allows the model to jointly learn the optimal pre-processing (embedding) and the predictive model, rather than treating them as separate, sequential steps.

The Evolution of Intelligence and AI Safety

The brain’s evolution provides a powerful analogy for developing more capable AI. Intelligence evolved in a modular fashion, combining simpler, specialized systems to create more complex capabilities. The olfactory cortex, for instance, evolved to navigate the highly complex, non-smooth, combinatorial space of smells. This part of the brain, which developed sophisticated associative processing, is arguably the precursor to our frontal cortex and its planning abilities. The future of truly intelligent systems will likely involve this “Lego-like” ability to combine existing modules in novel ways to solve new problems.

Regarding AI safety, the more pressing concern is not a rogue superintelligence like Skynet, but human “enfeeblement,” where we are reduced to simply approving or rejecting AI-generated outcomes. The danger lies in how humans specify goals. A naive command like “end world hunger” could lead to catastrophic outcomes if the AI finds a literal but destructive solution.

A safer, more robust approach is to use techniques like Maximum Entropy Inverse Reinforcement Learning (IRL).

1. First, observe existing human systems to create a model of the current steady-state distribution of actions and outcomes. This empirically derived distribution becomes the AI’s baseline reward function.
2. Instead of issuing a new, high-level command, make small, incremental perturbations to the desired outcome distribution.
3. Evaluate the consequences of these small changes, allowing for careful, controlled optimization without the risks of misspecified high-level goals.

This method grounds the AI’s objectives in observed reality and promotes gradual, safer progress.

• Tags:
• Self-Supervised Learning
• AI Safety
• Feature
• Philosophy of AI
• Agency
• Energy-Based Models
• JEPA
• Inverse Reinforcement Learning