Perception Agents — Antje Barth, Amazon AGI Lab
Antje Barth of Amazon AGI Lab discusses the architectural gap in current AI agents, which excel at individual tasks but fail at complex, end-to-end workflows due to a lack of reliability and contextual understanding. She introduces "Perception Agents"—AI systems that see, reason, and act on computers like humans, using visual and multimodal input to enable reliable collaboration and close the perception-action loop, highlighting new open-source tools for annotation and verification.
Antje Barth, a member of technical staff at Amazon AGI Lab, presented on the evolution of AI agents, highlighting the critical gap between agents that perform individual tasks and those that can reliably execute “real work” in complex, multi-application environments. While agents have advanced from finding buttons to driving browsers and desktop apps, the speaker emphasized that “clicking was actually the easy part. What we didn’t solve is the actual work.” This “actual work” often lives “within the seams of all of those different applications” and requires navigating ambiguity and dynamic interfaces.
The core problem lies in reliability and trust. Current agents, despite being capable of tool use and workflow chaining, typically achieve only 60-80% success rates in end-to-end workflows. Such low reliability is unacceptable for critical tasks; as Barth noted, “If your agent one in four times deletes a database, you will never touch that agent again.” This contrasts sharply with the rapid progress and trust gained in coding agents, which succeeded because code is verifiable through running tests and checks. However, most “messy” knowledge work lacks such clear, programmatic verification (e.g., “Did the report I created land? Is the design on brand?”).
To address this, Barth proposed Perception Agents, which are designed to “see and use computers the way people do,” reason about what they see, and act with clicks and keystrokes. This approach mirrors how humans handle messy work: by sharing context (e.g., looking at the same screen) and engaging in a Perceive, Plan, Act loop, much like robotics. Current agents act and move on without observing the consequences or recovering from failures. A perception agent, however, “takes in the screen the way you do, not scrape the code behind the page, but what’s actually rendered, the layout, the state, what just changed the work,” and keeps up in real-time.
Key advantages of Perception Agents include:
- Completing the loop: They can “read the rendered screen so it can confirm its own output instead of just firing off those actions and then hoping.”
- API-agnostic operation: They work directly off the rendered interface, observing the same pixels and structure as a human, making them effective even when no APIs are available.
- Precise, visual input: Users can point to elements on screen to give instructions (e.g., “this heading needs to change,” “update this section”), providing a “much more precise signal and less lossy than text” than verbose text prompts.
To enable this, Amazon AGI Lab has open-sourced the first two pieces of their Perception Agent Harness:
- Annotation Tool: A Chrome extension allowing users to visually select and mark elements on a screen (e.g., drawing a box around a heading) and describe desired changes. This captures precise visual context for the agent, eliminating ambiguity.
- Verification Component: This allows defining design rules (e.g., in a “design MD file”) against which the agent can check its own work. It performs visual checks (e.g., ensuring brand compliance, correct layout) and user flow checks (simulating user interactions like adding or deleting tasks). The agent then generates a report, detailing what passed and what didn’t, reducing manual review.
Barth also extended the concept of perception beyond just screens to multimodal perception. An experiment demonstrated using B devices (wearable transcribers) in a meeting to capture a discussion, generate a detailed summary, and extract insights. These insights were then sent to an agent, which applied the suggested changes to a website (e.g., changing background color, heading color, emoji) and immediately kicked off the verification process. This illustrates how “perception can also be listening in the room to what you’re discussing.”
The presentation concluded with a call to action, encouraging the community to try out the open-source tools, provide feedback, and contribute to building these new agent patterns, emphasizing the goal to “build AI that makes all of us smarter together.”