Parallel’s Parag Agrawal: Building a New Web for AI Agents
Parag Agrawal, CEO of Parallel Web Systems, outlines his vision for an agent-centric web where AI agents query the internet 1000x more than humans. He details how Parallel is reinventing search infrastructure, dismissing human click data as a "bug," and tackling the economic crisis of the ad-supported internet with a novel monetization model based on Shapley values to pay content creators.
Parallel Web Systems: Reinventing Search for the Agentic Era
Parag Agrawal, former Twitter CEO and founder of Parallel Web Systems, posits that the future of web search will be dominated by AI agents, not humans. His core bet is that agents will query the web “a thousand times more than humans ever have,” necessitating a fundamental rethinking of existing search infrastructure and business models. Parallel’s foundational belief is that “human click data is a bug,” and agentic search should rely on agent feedback and the ability of large models to compress information, rather than human interaction patterns.
Distinguishing Human vs. Agentic Search
Traditional web search, designed for humans, focuses on crawling, indexing, and ranking to serve often-lazy, keyword-based queries. Humans expect quick, easily digestible answers, leading to content optimized for SEO and readability (e.g., “AI slop”). For agents, this paradigm is inefficient. Agents, unlike humans, are not “lazy”; they can formulate precise, longer queries with fewer typos. They don’t need user-friendly summaries or to “click around and fumble around a PDF.” Instead, they require authoritative, high-signal excerpts delivered directly into their context window. This difference allows Parallel to avoid the trade-offs of human-centric search, directly providing the most relevant information to models.
Incremental Index Building and Agent-First Product Strategy
Addressing the challenge of competing with giants like Google in web-scale indexing, Parallel adopted an incremental approach. Rather than building a full index upfront, they first launched a search agent product. This agent could crawl the web after a query arrived, leveraging the agent’s patience (minutes) for deep research. Early use cases included insurance underwriting, claims processing, sales data enrichment, and financial data curation—all areas where agents could replace human outsourced work by curating data from the web. This strategy allowed Parallel to “incrementally go build our index” based on empirical agent feedback and real-world problems, rather than theoretical evaluations or massive upfront infrastructure investment.
Technical Architecture and Optimization
Parallel’s search API processes agent queries through a sophisticated multi-stage system:
- Query Enrichment: Models analyze and enrich incoming queries.
- Indexed Routing: Queries are routed to various specialized indexes (e.g., big, fresh, knowledge graphs, structured data).
- Multi-layer Retrieval and Ranking: A cascade of models, with increasingly larger and more sophisticated architectures, reduces billions of URLs or documents to thousands, then to specific excerpts and paragraphs, ultimately aiming for “a thousand tokens from a trillion web pages on the web” with the highest signal.
The optimization problem revolves around quality, cost, and latency. While initial products prioritized quality and cost, Parallel recently shipped “Turbo,” reducing agentic search latency to 200 milliseconds. Their research focuses on curating training data, optimizing models for specific architectures, and efficiently allocating compute within the memory hierarchy to meet these demanding thresholds.
The Economic Problem: Ad-Supported Internet vs. Agentic Web
Agrawal highlights a critical economic challenge: the ad-supported internet, reliant on human attention and clicks, collapses when agents replace people. Content owners, traditionally monetizing via ads or subscriptions driven by human traffic, find it difficult to monetize agent visits. Existing fixed-fee contracts with large language model (LLM) providers are also unsustainable, as model inference scales exponentially (e.g., 50x year-on-year) while deal sizes remain static. This creates a disincentive for content owners to allow agents access to their data, leading to a “broken business model.”
Parallel’s Solution: Shapley Values for Content Monetization
To address this, Parallel proposes a new business model centered on Shapley values, a game-theoretic concept for fairly distributing collaboration gains.
- Attribution Model: Instead of clicks, Parallel assesses the incremental value a piece of content contributes to an agent’s output. This is done by simulating agent performance with and without specific content, identifying how much a source contributes to the quality of the answer.
- Scalable Estimation: While theoretically expensive, Parallel uses models trained on extensive simulation data to estimate Shapley values efficiently.
- Differential Pricing: This system pays content owners more for unique, differentiated data and for content used in high-value work (e.g., a banker’s query vs. a casual user’s query).
This approach aims for incentive alignment, ensuring content owners are compensated proportionally to the value their data provides. Agrawal predicts this model will deliver “meaningful dollars for a very wide range of content owners on the web” within 12 to 24 months, significantly exceeding current web data business models.
The Future: 1000x Queries and the “Parallel Web”
Agrawal describes how agents dramatically multiply web queries: a single human prompt can lead to 5-20 agent searches, while sophisticated background agents can generate hundreds or thousands. He envisions a shift from the web being “pull” (humans querying) to “push” (agents proactively monitoring and triggering actions). This “parallel web for agents” will see AI systems constantly watching for relevant changes (e.g., in satellite imagery, customer commentary, or other agent outputs) and initiating work. This means content creators will increasingly be “dual publishing” for both human and agent audiences. Parallel’s API documentation, for instance, is primarily tested with agents, reflecting this agent-first audience. The company recently partnered with Google Cloud to provide search and grounding for their enterprise agent APIs, further solidifying its role in this evolving landscape.