Self-Improving Harnesses, Local Personal AI And YC's Agent For Work | YC Paper Club
A deep dive into the evolving world of LLM harnesses, demonstrating how they've transformed from simple prompt engineering into sophisticated self-improving systems. The discussion covers architectural advancements (e.g., Prime Agent's context as cache, Turing vs. Von Neumann), practical applications (auto-researchers, on-device personal AI like OpenJarvis, YC's QM for work), and the challenges and future directions of agentic AI. It emphasizes that harness innovation is key to unlocking significant performance gains and enabling LLMs to operate autonomously and adaptively.
The discussion introduces the critical role of “harnesses” in advancing Large Language Models (LLMs) and agentic systems, challenging the perception that they are mere scaffolding or prompt engineering. The host highlights that better harnesses can dramatically improve LLM performance, citing examples like ARC-AGI scores jumping from 30% to 95% or even 100% (with Nvidia’s AVO) purely through harness innovation. The evolution is traced from basic static harnesses to sophisticated self-improving ones, emphasizing the shift from measuring perplexity to leveraging test-time experience for rapid adaptation to new problem distributions.
A Brief History and Evolution of LLM Harnesses
The presentation outlines a historical progression of harness capabilities:
- V0 (Early LLMs like GPT-2): Featured a basic
end-of-sequenceloop, top-P sampling, and simple system prompts (e.g.,You are a math teacher). Interactions were token-based with minimal external functionality. - Static Harness Era (V1): Significant functionality was added without the harness itself learning.
- Context Innovations: In-context learning (few-shot learners, e.g., July 2020 paper) and Chain-of-Thought (CoT) where logical steps are spread across more tokens for better reasoning.
- Action Space Innovations: Tool-calling (ToolFormer, WebGPT) allowed LLMs to invoke external functions (e.g., Python
sub(5,3)). MemGPT introduced CRUD operations on the context itself, creating explicit memory. - Skill Chaining: Voyager demonstrated chaining tools to achieve tasks and distilling these into
skills.mdfor reuse (e.g., in Minecraft). - Code Generation: Intercode allowed agents to output code directly, creating on-the-fly tools/skills.
- Multi-Agent Reflection: React, Self-Refine, and Reflection enabled agents to take action, evaluate internally, and improve their output by looping or getting reward signals from the environment.
- Recursive Language Models (RLM): Allowed spawning persistent sub-agents recursively, orchestrated by a main agent, forming complex multi-agent systems.
- Self-Improving Harnesses: The latest frontier, where the harness itself learns and adapts.
- DSPY: Learns optimal system prompts by iterating, merging, and evaluating candidates through genetic programming.
- Darwin Machines: Extends this by allowing the agent to modify its own harness code, creating a