The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs
Giedrius Šteimantas discusses the often-overlooked infrastructural layer for AI agents operating on the open web. He highlights how traditional browser automation leads to high costs and unreliability, particularly due to CAPTCHAs and invalid content. Drawing from ten years of web scraping experience at Oxylabs, he presents a multi-stage solution that prioritizes cost-efficiency and content validation, using specialized APIs and hardened headless browsers only when absolutely necessary, thus significantly reducing token waste and improving agent performance.
The speaker, Giedrius Šteimantas from Oxylabs, identifies a crucial “missing layer” in agentic AI: the robust infrastructure required for agents to effectively and cost-efficiently interact with the open web. He illustrates this by detailing the shortcomings of a friend’s personal shopping agent, which, despite sound model quality, proved slow, expensive, and unreliable due to an over-reliance on basic browser automation and a failure to properly validate retrieved content.
The Problem with Naive Agent Implementation
The friend’s agent, designed to converse with users about their style and then use a second agent to purchase selected items, was built entirely using browser automation. This led to significant issues:
- Performance and Cost: It was inherently slow and expensive, frequently encountering CAPTCHAs and website blocks.