Continual Learning: How AI Agents Get Better With Every Use | Arjun Karanam, Trajectory

Arjun Karanam, co-founder of Trajectory, delivered a talk on the critical need for continual learning in the agent ecosystem, addressing what he terms the “experience gap.” While current AI models are rapidly improving in ‘IQ’—their raw intelligence and capabilities—they consistently lack ’experience,’ often feeling like it’s their “first day on the job.” This analogy posits that even a genius like Terrence Tao would be a poor accountant on day one without practical experience, highlighting that experience, not just raw intelligence, is crucial for effective performance. The core problem is that trillions of tokens representing agent interactions, which contain valuable real-world experience, are currently discarded.

Trajectory’s Vision and Approach

Trajectory aims to close this experience gap by building a platform for continual learning. Their approach involves:

• Traceability: Capturing all agent interactions and thrown-away experience.
• Model Specification: Developing methods to define precise agent behaviors from user interactions, converting traces into exact specifications that agents should learn against. This involves research into extracting rewards from user interactions.
• Learning Mechanisms:
  • Model Improvement: Applying advanced Reinforcement Learning algorithms, such as SDPO and DPO, to long interaction traces to continuously improve core models.
  • Harness Optimization: Researching how to effectively route feedback, distinguishing between knowledge that should be trained into the model (e.g., general patterns) and factual context that should be handled by the harness (e.g., a company being delisted).

Four Wishes for the Agent Ecosystem

Karanam outlined four key areas where improvements are needed to accelerate the adoption and effectiveness of continual learning in the agent ecosystem:

1. Enhanced Traceability

• Trace the entire tree: Beyond main actions, capture the full execution tree, including all sub-agents and tool calls. Discarding these details makes comprehensive learning difficult.
• Capture corrective behavior: Move beyond simplistic “thumbs up/down” feedback, which is often noisy. Focus on eliciting and capturing explicit user edits, undos, and retries as these provide much clearer signals for learning.

2. Real-World Evals

• Evals from production traffic: Evaluations should be drawn directly from how users interact with the product and the frontier requests they attempt, even if currently impossible.
• Make every task rolloutable: Implement infrastructure that allows for replaying user actions, providing an invaluable resource for evaluation and improvement.
• Grade through the real harness: Conduct evaluations within the actual production harness environment, ensuring relevance and accuracy, rather than simplified or varied setups.

3. Smarter Harnesses

• Let the agents cook: Shift the harness design philosophy from constraining agents (a necessity with older, less reliable models) to enabling them to orchestrate primitives. The harness should define the available tools and information, allowing the agent to utilize them creatively.
• Agent interface mirroring user interface: Ideally, every action a user can perform in the UI should also be accessible to the agent via a tool call, simplifying training and continual learning.
• Informative tool responses: Tool calls should return detailed, substantive responses rather than generic confirmations (e.g., “Done”). Richer feedback provides better signal for agents to learn from.

4. Flexible Models

• Embrace open weights: Companies should become comfortable running open-weight models. This is crucial for gaining ownership over model weights and enabling continuous, targeted improvements, despite associated security and safety considerations.
• Experiment with model routers: Leverage model routers to direct specific tasks to the most capable models or components, optimizing resource usage and performance across diverse capabilities.

Owning Your Intelligence and Trajectory’s Product

Karanam emphasized that the ability to own and continually improve one’s intelligence layer is a strategic imperative that should not be outsourced. Trajectory’s product aims to democratize this capability, allowing any company to manage and improve their models. The demo showcased an easy-to-use interface for importing benchmarks, training models (abstracting away complex fine-tuning knobs), evaluating performance, comparing models, and deploying them—all within about 15 minutes of active work.

Q&A Insights

The Trainable Object in Continual Learning

Continual learning should optimize across the entire system—model weights, harnesses, tools, and application layers—rather than solely focusing on weights. The system should intelligently determine what needs updating (e.g., model weights for general knowledge, harness context for specific facts), abstracting this complexity away from the user, similar to how modern file systems abstract memory management.

Continual Learning without Customer Data

To address privacy concerns, Trajectory employs methods that avoid direct training on customer data. This involves sampling distributions from customer interactions, synthetically generating new data based on these distributions, and comparing them to ensure consistency without exposing sensitive information.

Episodic Memory and Feedback Hierarchy

Feedback signals have different qualities and hierarchical relevance:

• Signal Type: Distinguish between general indications of failure (“something went wrong” like a thumbs-down) where only negative behavior is penalized, and precise corrections (e.g., retries, explicit edits) where there’s high confidence in the correct output.
• Pertinence Hierarchy: Some information is globally relevant (e.g., a tool repeatedly failing) and should be trained into the model. Other information, like user-specific preferences (“never use this sub-agent”), is best handled contextually by the harness on a per-org or per-customer basis.

When Continual Learning Matters Most

While generally beneficial, continual learning is most impactful for “frontier” tasks—those at the edge of what models can currently achieve. It allows models to learn from user attempts at challenging tasks, pushing the boundaries of what’s possible and enabling users to accomplish things they couldn’t before, creating a compounding effect on model capabilities.

• Tags:
• Agentic AI
• LLMs
• AI Models
• Machine Learning
• Deep Learning
• Evaluations
• Continual Learning
• Traceability
• Open Weights
• AI Harnesses
• Experience Gap
• Model Routers
• Trajectory