Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data
Yohan Raju of Bright Data shows how an AI agent can combine a live Amazon listing with Reddit sentiment to produce a purchase recommendation, arguing that agents need fresh, structured web data rather than relying only on pre-trained knowledge or pre-indexed retrieval.
Building AI agents that reach live web data
Yohan Raju, a solutions architect at Bright Data, builds his talk around one claim: an agent making a decision needs fresh, structured evidence at that moment, and pre-trained weights or a pre-indexed RAG store cannot always supply it. His demo is a product intelligence agent. It pulls an Amazon listing for a tumbler cup, searches Reddit for discussion of the same product, and asks Gemini to turn both sources into a buy or don’t-buy recommendation.
Company background and scope
Raju describes Bright Data’s trajectory as a progression. The company started as a proxy infrastructure provider with one of the largest residential IP networks, then expanded into a data collection platform that he says now handles petabytes of data per month, second only to Google. The talk has three parts: agents versus retrieval-augmented generation, the infrastructure needed to run agents on live web data, and a hands-on notebook exercise.
Why scraping gets harder as coverage grows
He traces the usual trajectory of a scraping customer. It starts small, with a handful of target sites, typically Amazon and Walmart for ecommerce, since they offer broad product coverage for price, review and catalog data. Building a scraper for one site takes a developer roughly three to five days once development, QA and end-to-end testing are included. Ten sites are manageable; a thousand are not.
The larger cost is maintenance. A single selector change, such as a price field moving, can silently break the downstream pipeline. Teams then have to diagnose the change, repair the scraper, backfill lost records, and check whether SLAs and data quality have been affected. In the past, scaling meant adding engineers in proportion to the number of sites. Raju argues that agents can now discover pages and generate scrapers with much less human effort, though he does not go deep into how reliable that generation is.
Every site is also structured differently and defends its data differently. Amazon and Walmart both expose price, title and description but use different layouts, so each needs its own extraction logic. Protection ranges from sites that accept data-center IPs, to systems such as Cloudflare or Akamai that require residential IPs, real browsers and human-like behavior. Scrolling or clicking too quickly, or blocking images, can trigger anti-bot defenses.
The traditional pipeline
Raju walks through a conventional architecture in stages:
- Trigger: a scheduler, cron job or fixed list of URLs starts the crawl.
- Proxy layer: masks the requester using static data-center or ISP addresses, rotating residential IPs drawn from consumer devices that look authentic, or managed tools such as Bright Data’s Web Unlocker and Browser API, which handle blocks, retries and IP selection internally.
- Parsing and validation: extracts fields and checks completeness, for example whether price comes back in about 90 percent of requests, and whether ratings and reviews are present.
- Ingestion and post-processing: deduplication, enrichment and batch processing, moving data through bronze and silver layers of a medallion architecture before it reaches a database.
A failure at any stage, whether an anti-bot block or a changed selector, can force the entire process to restart and create SLA misses that need backfilling.
Agents, RAG and the sync versus async question
Raju uses the behavior of AI assistants to show when live retrieval matters. Asked ‘when’s the best time to travel to San Francisco,’ a model can answer from pre-trained knowledge, perhaps saying summer. Add ’this year’ and the assistant needs current information about events, weather or conditions, so it has to search the web. He calls the first path async, relying on pre-trained or indexed data, and the second sync, fetching live data during the interaction. Each needs a different setup.
He also notes that caching is worthwhile for some queries. A day’s weather forecast for New York, for example, can be fetched once and served to many users, which saves tokens and cost. He points out that many websites block AI crawlers such as ClaudeBot and GPTBot through robots.txt, which can leave assistants unable to reach specific sources. Bright Data presents its value as unlocking access to those sources.
Bright Data’s MCP server
Raju introduces the Model Context Protocol (MCP) server as the interface between a reasoning model, such as one run in Claude Code or LangChain, and Bright Data’s library of more than 60 tools. These cover search engine results, page extraction as markdown or HTML, site-specific collectors for Amazon, Reddit, LinkedIn and Instagram, and browser automation for actions like clicking, scrolling, placing a food order or booking travel. He also mentions financial, business intelligence and app store tools. The agent chooses which tool to call, which he presents as the streamlined alternative to hand-built scrapers.
As an example, he describes a social listening case. An agent searched for a person’s name, located their Instagram, TikTok and LinkedIn profiles, retrieved follower counts through Bright Data’s pipeline, and returned the results in context.
Glassdoor access demo
The side-by-side demo runs the same prompt in Claude: read the contents of a Glassdoor page. With Bright Data’s MCP server, the agent retrieves the page. Without it, the request is blocked. Raju uses Glassdoor as an example of a heavily protected site and as evidence that agents need a managed access layer.
In the Q&A, he attributes this capability to scale: a large proxy network and anti-bot logic that changes its approach to each site over time. A few days of scraping through a data center IP might succeed, but once a site detects it and adds layers of protection, a bot or scraper alone will not hold up. He also observes that teams building in-house often spend more engineering time working out what a site changed than they spend maintaining the scraper.
Job data as a continuous use case
Job data is his second example. Openings, companies, salaries and locations change constantly, so a resume-matching model needs continuous updates. It also needs historical records to show trends, such as salary growth for software engineers over several years, and to understand shifts in demand. Consistent structured output matters, because downtime propagates through the whole pipeline and can distort the market view a model is built on.
Live notebook exercise
Participants need a Bright Data API token and a Gemini API key. The notebook calls Bright Data’s Amazon collection endpoint to retrieve product-level data for a Stanley-style tumbler, including price, images, ratings, alternative sellers and features. It then runs a Reddit keyword search for the same product, collecting posts about whether it is worth buying. The raw responses go to Gemini for analysis.
The live results included Reddit commenters describing the cup as embarrassing and noting that people were obsessed with it. The model’s analysis reported a decent discount, overwhelmingly positive reviews with a few mixed opinions, and high demand alongside low stock on Amazon. It recommended buying.
Takeaway
Raju closes by arguing that the pattern extends well beyond ecommerce. Whenever an agent needs live, structured data from the web, the loop of collection, validation and analysis applies. In his view, the main bottleneck is not model reasoning but reliable access to web data at scale.