Parag Agrawal on AI Search & the Future Web

Parag Agrawal, former CEO of Twitter and founder of Parallel Web Systems, discussed the evolving landscape of web search and the internet's future in a recent conversation on the Training Data podcast. Agrawal articulated a vision where AI agents, not humans, will be the primary drivers of web interaction, necessitating a fundamental rethink of search technology and business models.

Rethinking Search for AI Agents

Agrawal's core thesis is that the internet's architecture and search mechanisms, built for human interaction, are no longer sufficient for the scale and nature of AI agents. "Our view at Parallel is that human click data is a bug and agent doing work with search should rely on agent feedback not human feedback," he stated. He believes that large language models are adept at compressing information, and this capability can be applied to search indexing and ranking to serve AI agents more effectively.

Parallel Web Systems is developing technology to enable agents to search and utilize the web. Agrawal explained that this is driven by the expectation that agents will interact with the web at a volume a thousand times greater than humans. This necessitates a reinvention of the underlying search technology and the business models that support it.

Lessons from Twitter and the Pre-AI Era

Agrawal noted that building Parallel has required him to unlearn many lessons from his time at Twitter. "When I was at Twitter in leadership roles, Twitter was a post-product market fit, extraordinarily scaled business where your feedback loops were from the hundreds of millions of customers using the product for 30-plus minutes every day. In that world, you operate differently than a pre-product market fit company," he said. He contrasted this with Parallel's mission to build for the "not yet here customer", the AI agents of the future.

The Problem of Web Search: A Billion-to-Billion Match

The fundamental problem of web search, Agrawal explained, is matching user queries to the vast and unstructured information available on the internet. This involves crawling the web, indexing URLs, and ranking results to provide the most relevant answer. "One way to think about it is it's a billion to billion matching problem," he described. He highlighted that traditional search engines like Google and Bing have tackled this with massive infrastructure, but the advent of AI agents presents new challenges and opportunities.

Agentic Feedback and Information Compression

Agrawal emphasized the shift from human feedback loops, like click data, to agent feedback. "We believe that these models are really good at compressing information," he said, "and we can benefit from a lot of the research that have gone into building models and apply it to search indexing and ranking." This compression is key to efficiently serving the high volume of queries expected from AI agents.

Overcoming Infrastructure Hurdles

The cost and complexity of building a full-scale web crawl and index have historically been barriers for new entrants. Agrawal explained that Parallel's approach circumvents this by not requiring massive upfront infrastructure investment. Instead, they are building an incrementally larger and more sophisticated index by solving problems for customers and using agentic loops to gather evaluation data. "Our view at Parallel is that human click data is a bug. An agent doing work with search should rely on agent feedback, not human feedback," he asserted.

The Rise of the Agentic Web

Agrawal sees the future internet as one increasingly dominated by AI agents. "The interface actually changes when you think about serving agents versus humans," he noted. While humans rely on keyword search, agents can provide more complex and nuanced queries. He also touched upon the challenge of deciphering human-generated content, such as SEO-optimized affiliate marketing pages, stating that agents may be less susceptible to this "slop" because Parallel aims to extract information directly from authoritative sources.

The Economics of the Internet are Shifting

The economic underpinnings of the internet, largely built on advertising models driven by human attention, are being disrupted by the rise of AI agents. "If humans don't show up and their agents show up on the web, like what does this mean? How does the business work?" Agrawal questioned. He highlighted the difficulty in monetizing agent visits under existing subscription models and the potential for old business models to break if new ones aren't developed.

Shapley Values and Incentive Alignment

Agrawal discussed the concept of Shapley values as a framework for fairly distributing value and incentivizing content owners to collaborate in an AI-driven ecosystem. "The biggest property of it all is incentive alignment," he stressed, "at what point do people want to collaborate into this enterprise?" He believes that efficient differential pricing, paying for high-value content and work, will be crucial for sustainability.

Parallel's Vision: A Parallel Web for AI

Agrawal concluded by outlining Parallel's ambitious vision: the emergence of a "parallel web built for AI agents." He sees a future where agents are ubiquitous, driving significant computational work and where the web transitions from a "pull" to a "push" model, with agents being proactively notified of relevant changes. "Imagine like agents are everywhere and they can do a lot of things," he stated. "If there is something you can do today and it's worth doing and we still have GPUs available, we'll just go do it." This future, he believes, is only 12 to 24 months away.