Agents need more than a chat - Jacob Lauritzen, CTO Legora

The nature of complex, end-to-end work is being reshaped by AI, shifting the primary bottleneck from the execution of tasks to their planning and review. Actually doing the work has become cheap and easy, but defining the requirements and, more importantly, verifying the output is now the most significant challenge. Reviewing large, AI-generated outputs is as painful as reviewing a massive pull request on GitHub.

The Verifier’s Rule and Its Impact on Vertical AI

A useful mental model for this new landscape is the Verifier’s Rule, which states that if a task is solvable and its output is easy to verify, it will eventually be solved by AI. This applies not just to foundational models but to agents as well; if a task is verifiable, an agent can be run in a loop with feedback until it succeeds.

Different industries and even different tasks within the same vertical lie at various points on this verifiability spectrum.

• Legal: Checking definitions in a contract is easy to verify. However, drafting a full contract is extremely difficult to verify, as its true quality is only tested when it’s scrutinized in court. Litigation strategy is nearly impossible to verify, as there is no single objective truth.
• Coding: Linting or writing simple functions is easy to verify. Building a commercially successful consumer application is very difficult to verify.

The goal for vertical AI companies is to involve humans where it matters most, focusing on two key aspects of human-agent collaboration: Trust and Control.

Increasing Trust in Agent Output

Trust dictates how much a human needs to review an agent’s work. Low trust requires scrutinizing every step, while high trust allows the agent to work autonomously. Trust can be systematically increased in several ways:

1. Make Tasks More Verifiable: Introduce mechanisms that make verification easier. In coding, this can be achieved with test-driven development. In legal tech, while direct verification is hard, a proxy can be used, such as comparing a newly generated contract against a set of “golden” precedent contracts to ensure similarity.
2. Decompose Tasks: Break down a large, hard-to-verify task into smaller, more manageable sub-tasks. A human can handle the subjective parts (e.g., choosing a risk profile for a contract), while the agent performs the easily verifiable work (e.g., applying formatting, checking for defined terms).
3. Add Guardrails: Increase trust by limiting the agent’s possible actions. By restricting an agent to read or write only specific files or search a limited set of websites, you reduce the chance of unexpected behavior and build confidence in its operation.

Enhancing Human Control Over Agent Execution

Control is the measure of how effectively a human can instill their knowledge and judgment into the agent’s workflow. The evolution of control mechanisms can be seen as a progression:

1. Basic Chat (Low Control): The initial approach of giving a prompt and waiting for a final output offers almost no control. If a correction is needed, the agent often suffers from context loss (“compaction”) and produces a flawed result.
2. Upfront Planning: The agent proposes a step-by-step plan for human approval before execution. This offers more control by aligning on the approach upfront. However, it’s inefficient and brittle, as the agent cannot account for “progressive discovery”—unexpected issues that arise only after the work has begun.
3. Skills: A more robust method is to encode human judgment into reusable “skills.” These are specific instructions for how to handle certain sub-tasks (e.g., “Here is how you should always review a confidentiality clause”). This allows the agent to handle contingencies and special cases discovered during execution.
4. Elicitation: When an agent encounters a situation not covered by its skills, it can elicit input from the human. The ideal implementation for this is non-blocking: the agent makes a decision to unblock itself, logs that decision, and continues. The human can later review the decision log and override it if necessary.

Beyond Chat: The Need for High-Bandwidth Interfaces

These sophisticated control mechanisms cannot be effectively managed within a traditional chat interface. Chat is a one-dimensional, low-bandwidth medium that collapses a complex, tree-like structure of work into a linear, hard-to-follow conversation.

The future of human-agent collaboration lies in high-bandwidth artifacts. These are rich, persistent, and domain-specific interfaces where humans and agents can work together on the task itself.

• Example 1: The Document: Instead of a chat box, the collaboration happens directly within the document. A user can highlight a specific clause and instruct an agent to change only that part, add comments, or tag specialized agents to handle different sections.
• Example 2: Tabular Review: For a contract review task, an agent can generate a structured, tabular interface. It reviews all documents autonomously but flags specific items that require human judgment. This gives the human high control and makes the review process efficient, allowing them to quickly provide input and kick off the rest of the agent’s work.

While natural language is a universal interface for input, we should not constrain agents to human modes of communication for collaboration. Agents are not human, and by leveraging richer, more structured interfaces, we can unlock a more powerful and effective partnership between humans and AI.

• Tags:
• AI Agents
• LLM
• Feature
• User Experience
• Vertical AI
• Human-in-the-Loop
• Collaboration Tools