Tokenless

Too much lock-in for too little gain: agent frameworks are a dead-end // Valliappa Lakshmanan

Lak Lakshmanan presents a robust architecture for building production-quality, framework-agnostic agentic systems. He advocates for using simple, composable GenAI patterns, off-the-shelf tools for governance, and a strong emphasis on a human-in-the-loop design to create continuously learning systems that avoid vendor lock-in.

5 min read
Too much lock-in for too little gain: agent frameworks are a dead-end // Valliappa Lakshmanan
source: https://www.youtube.com/watch?v=HUHjSFWr9pg

Building agentic systems requires an architecture that is flexible, production-capable, and avoids lock-in to any specific LLM, cloud, or agent framework. The key is to balance the rapid development promised by frameworks with the control and quality needed for production environments. The recommended approach is to use simple, composable patterns for agent logic, leverage commercial off-the-shelf tools for infrastructure and monitoring, and build the core orchestration to align with specific business workflows and Key Performance Indicators (KPIs).

Core Principles for Modern Agentic Systems

When developing agentic systems, several critical goals must be met to ensure long-term viability and success:

  • LLM and Location Agnostic: Avoid tight coupling to a single LLM provider or cloud. This allows for flexibility in choosing models based on latency, cost, or accuracy requirements, and accommodates legal or data residency constraints.
  • Flexible and Controllable Orchestration: The agentic space is evolving rapidly. Rigid frameworks can quickly become obsolete as capabilities like tool calling and Chain of Thought are integrated directly into base models. Orchestration should be explicitly coded to match business workflows, allowing developers to maintain control and rip out components as they are replaced by native LLM features.
  • Traceability and Governance: Every action taken by the system must be traceable. This is essential for creating audit trails, generating metrics on adoption and correctness, enabling explainability, and performing evaluations.
  • A Path to Autonomy: Systems should not be expected to be fully autonomous from day one. A “co-pilot” or human-in-the-loop approach is crucial. The system should be designed as a learning path where human inputs are captured to train the system, allowing it to become more autonomous over time on easier tasks.
  • Mitigate Business Risk: Avoid vendor lock-in where possible. While some lock-in is inevitable, it’s critical to be deliberate about what risks are acceptable.

A Composable System Architecture

The proposed architecture is built on modularity, reusability, and continuous learning. It consists of several key components:

1. Composable Agents

Agents should be designed as independent, reusable executors.

  • Role-Based: Each agent is given a specific role and goal via a system prompt.
  • Stateful Context: Agents operate on a stateful context that is passed between steps in a workflow. Prompts are dynamically populated with data from this state.
  • Grounded: Agents are grounded with business-specific context, such as available tools and knowledge bases.
  • Structured Outputs: Agents must produce structured outputs (e.g., JSON) to enable reliable evaluation and integration with other components.
  • LLM Agnostic: Use lightweight libraries like Pydantic or LlamaIndex to abstract away the specific LLM, making it easy to swap models.
  • Separation of Concerns: Keep prompts separate from the application code. Using templating engines like Jinja allows for easier prompt optimization and maintenance without code changes.

2. Multi-Agent Orchestration

The orchestration logic is where domain knowledge is encoded and should be fully controlled by the developer, not a framework.

  • Human-Powered “Co-pilot” Workflow: Start by building a system where a human makes key decisions and edits outputs at each step. This approach is vital for capturing the “unwritten rules” of a business process that are often missed by standard operating procedures.
  • Avoid Deep Recursion: Currently, agentic systems perform well with a “plan and execute” pattern. However, allowing agents to re-plan based on execution results can lead to them going “haywire.” It’s recommended to design workflows that are “one deep and no deeper” to maintain stability.

3. Governance and Infrastructure

Leverage battle-tested, standard tools for infrastructure to ensure scalability and reliability.

  • Deployment and Monitoring: Treat the system as just another application. Deploy agents as microservices on serverless platforms (e.g., AWS Lambda, Google Cloud Run) and use standard, off-the-shelf monitoring and security tools.
  • Persistence Primitives: Use standalone services for memory, vector databases, and caching. However, start with the simplest solution first. For example, pass short-term memory in-state between function calls and only introduce a persistent database when state becomes too large or needs to span sessions.
  • Guardrails: Implement guardrails as configurable prompts. To avoid adding latency, run guardrail checks asynchronously in parallel with the main task. In most cases, the guardrail will pass, and no time is lost. If it fails, the main task can be preempted.

4. The Continuous Learning Pipeline

A core feature of this architecture is its ability to improve over time.

  • Capture All Human Feedback: The user interface must be designed to be fully editable. Every change a user makes—from selecting a different agent to editing a piece of text—is valuable feedback that must be captured.
  • Plan for Post-Training: Use the captured feedback data for post-training techniques like Direct Preference Optimization (DPO) or instruction tuning to create smaller, fine-tuned models that are more accurate and efficient for specific tasks.
  • Augment with a Data Program: As a system improves, users provide less feedback, a phenomenon known as “automation fatigue.” To counteract this, a dedicated data program is needed to proactively source the specific kinds of feedback required to continue improving the system.

Handling Multi-Agent Conflicts

In workflows where multiple agents act in parallel (e.g., a review panel with conflicting viewpoints), a specific pattern is needed for resolution.

  • Append-Only Memory: Shared memory should be treated as an append-only log. Multiple agents can write to it without conflict. The responsibility of resolving conflicting information lies with the agent that consumes the memory, not the writers.
  • Conflict Resolution Agent: To prevent parallel agents with opposing goals from escalating conflicts, introduce a dedicated “secretary” or “resolver” agent. This agent’s job is to synthesize the outputs from the parallel agents, resolve contradictions, and produce a single, coherent summary or set of instructions for the next step in the workflow.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Building an Agentic Platform — Ben Kus, CTO Box

Building an Agentic Platform — Ben Kus, CTO Box

Ben Kus, CTO of Box, outlines the technical evolution of their AI platform, detailing the transition from a promising but fragile LLM-based metadata extraction system to a robust, scalable agentic architecture. He explains why this shift was necessary to handle enterprise-level complexity and the key lessons learned.

Why Language Models Need a Lesson in Education

Why Language Models Need a Lesson in Education

Stephanie Kirmer, a staff machine learning engineer at DataGrail, adapts her experience as a former professor to address the challenge of evaluating LLMs in production. She proposes a robust methodology using LLM-based evaluators guided by rigorous, human-calibrated rubrics to bring objectivity and scalability to the subjective task of assessing text generation quality.