Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati

Project Paradox and the problem of long-horizon social behavior

Erina Karati opens by framing the talk around a question she sees many AI engineers running into: how do we evaluate and improve agents that carry state over long periods? She uses a video game village populated by AI agents as a running example, but the lessons are meant to generalize beyond games.

Project Paradox: a stateful agent framework

Karati and Arunachalam Manikandan built Project Paradox at Supercell’s AI Innovation Lab as a modular framework that lets any developer plug autonomous agents into a video game. The agents move with intent, pick up and drop objects, are aware of their surroundings, react to events that change their beliefs and emotions on the fly, and start conversations with other agents or the player. Game developers can also add new actions to the framework.

Karati describes the architecture as deliberately stateful, with four components:

• Per-agent memory: each agent has its own RAG-backed memory namespace, so memory does not bleed between agents.
• Emotion as a small vector: events and conversations update values such as joy, sadness, fear, anger and disgust.
• Belief and trust scores toward other agents and the player: after an interaction, the LLM decides whether the trust score should go up, go down, or stay unchanged. She calls this a trust matrix.
• Memory importance: the LLM rates each event’s importance, and events above a threshold go into a separate cache so they can be retrieved more reliably later. Her analogy is that you forget what you had for dinner last week but remember if someone was murdered.

A picnic demo showed the system working in short scenes. A character planned a sequence of actions, picked up a pastry, walked to the picnic area, and answered questions in context afterward.

Where long-horizon behavior breaks

The problems appeared over longer horizons. In the mango example, one agent tells another about a sale, that agent passes it on, and after many intervening events the player asks about mangoes. The system remembers the rough topic but loses its source. A rumor hardens into fact, so “might” becomes certain, and agents sometimes know something but fail to use it when making plans. Karati’s core question became how to improve a multi-agent system on long-running social behavior rather than on single responses.

Bringing in autoresearch

Inspired by Andrej Karpathy’s recent autoresearch work, the team asked whether the system could run experiments on itself. Instead of hand-tuning prompts or relying on one impressive demo, the approach is to define a scenario suite, run the agents, collect traces, score behavior, change a small policy surface, and keep only the changes that improve the score. In this framing, Project Paradox is the lab bench and autoresearch is the experimental loop around it.

The target is not just RAG retrieval but the agent protocol itself: how agents write and retrieve memories, communicate uncertainty, attribute sources, update trust, and replan around new facts. Karati is clear that the autoresearch layer is not another villager. It is a meta-system outside the village. Villagers only have local perspectives, since there is no shared memory database and information moves only when agents communicate it. The meta-layer reads the full traces of a run, compares them against scenario ground truth, scores society-level behavior, proposes a constrained change, and reruns the scenario. The key shift is from evaluating one answer to evaluating an entire run.

One iteration of the loop works as follows: define a controlled scenario, run the simulation, collect structured traces (observations, conversations, memory writes, retrievals, belief updates), score the behavior, propose a small policy edit, and rerun. The change is kept only if the score improves and the guardrails hold. Otherwise it is reverted.

Controlled scenarios

Free-roaming villages look compelling but are hard to evaluate. Karati argues for scenario suites, with examples such as:

• Public fact diffusion: if an agent learns the bakery will close tomorrow, do the right agents learn it, remember who said it, and change their plans?
• Rumor uncertainty: when agent A hears that agent C “might leave” the village, does that stay a rumor or harden into “is leaving”?
• Replanning: when a route the group planned to take is blocked, do agents update the plan and communicate the change to avoid stale actions?

She stresses that these exact scenarios are illustrations, not universal benchmarks. The point is that long-horizon behavior needs scenario suites. After a long run through the loop, the mango question got an in-context answer. She is careful, though, not to claim the system generally improved from one run.

A balanced scorecard

A single vague metric like “agent quality” hides the interesting failures. Karati’s scorecard is multi-dimensional:

• Diffusion: reach, meaning how many agents know the fact after N steps.
• Provenance: source retention among agents who know the fact, including whether they remember where it came from.
• Rumors: uncertainty preservation and a false-certainty rate.
• Planning: action consistency and time to replan.
• Privacy: containment.

Optimizing one metric invites gaming. Optimizing diffusion alone makes agents overshare, and optimizing memory recall alone produces noisy or stale memories. The scorecard keeps the autoresearch agent honest and stops it from inflating a single number.

Keeping the editable surface small

The loop should not be allowed to rewrite the codebase. The harness, scenarios and metrics are frozen, and only a narrow policy surface is editable: memory writing and retrieval policy, communication prompts, belief and trust rules, source attribution, and replanning triggers. Karati gives concrete search directions:

• If source attribution disappears, preserve the source in memory records and summaries.
• If rumors harden into facts, store confidence markers (firsthand versus secondhand) and require hedging when retelling uncertain claims.
• If public facts stay local, classify useful facts differently and have agents proactively share source evidence.

The distinction she draws is between an LLM writing random patches and an LLM searching within a controlled policy space. These are small protocol changes that can still produce large effects on society-level behavior.

Lessons learned

Memory is not enough. A RAG memory alone does not yield the long-horizon behavior you want. Agents need provenance, meaning whether information was firsthand, secondhand, verified or uncertain, and it can help to separate raw episodic memories from what the agent currently believes.

Rollback is not optional. A change can improve one thing and damage another. A policy that spreads public facts faster might also leak private information, and a policy that increases recall might increase stale memory use. The loop works like a ratchet: try a change, score it, keep it only if the scorecard improves and the guardrails hold.

The pattern extends beyond games. Support agents need to know which policy update came from where and whether it supersedes an older answer. Personal assistants must remember and correct commitments. Research agents need provenance, citations, contradiction handling and hypothesis updates. Coding agents need long-running context across issues, files, teammates and changing requirements. Workflow agents need access controls, handoffs and replanning when the world changes. Karati’s point is that all of these systems maintain state over time, and that state shapes future action.

A recipe for long-horizon agents

Freeze the harness, define scenarios, log traces, score behavior, expose only a small policy surface, search over changes, and keep only the changes that survive measurement. The real question across controlled runs is whether the system behaves better.

In closing, Karati reframes Project Paradox. It began as an attempt to make game agents feel alive in a 3D world, but the deeper engineering problem was state: which agent knows what, who told whom, what is true, uncertain or outdated, and whether agents act on what they remember. Autoresearch gave the team a more systematic approach than trusting one demo or endlessly hand-tuning prompts. As she puts it, “Long horizon agents need experiments and not just prompts.”

• Tags:
• Multi-Agent Systems
• Provenance
• AutoResearch
• Agent Memory
• Agent Evaluation
• Game AI
• Long-Horizon Agents
• Trust Scores
• Balanced Scorecard
• Project Paradox
• Supercell