Oxylabs: Web Data Infrastructure Powers AI's Future

In the rapidly evolving AI landscape, the underlying infrastructure that feeds models with data is becoming as critical as the models themselves. Patricija Žemaitytė, Product Manager at Oxylabs, a web intelligence and premium proxy provider, shared insights at the AI Engineer World's Fair on how Oxylabs builds the infrastructure layer essential for companies to extract and utilize public web data at scale.

The Evolving Role of Web Data in AI

Žemaitytė highlighted a significant shift in the industry, moving away from relying solely on static training data. She explained that for AI models to remain relevant and useful, they require access to fresh, live information. "Training alone is no longer enough, and to stay useful, models need to get access to fresh information, live search, real external data," she stated. Without this, even the most advanced models are limited by their knowledge cutoff.

Innovation Born from Pressure: The Video API Suite

Žemaitytė recounted her experience at Oxylabs, where a client's urgent request for video API capabilities for AI training, with a two-week deadline and a demand for 5 petabytes per month, spurred significant development. This wasn't just about downloading videos; it was about building a complete pipeline for collection, transfer, storage, and delivery, all with the reliability needed for AI training. This initial challenge led to the development of a comprehensive video API suite, including transcript support, subtitle functionality, and search capabilities, demonstrating that "innovation never comes as a neat road map. It comes as a pressure, as a deadline, and sometimes, and quite often, as a trip report from San Francisco."

The Quest for Speed: SERP Data and Low Latency

The conversation then shifted to the critical role of SERP (Search Engine Results Page) data in AI systems, particularly for retrieval pipelines, assistants, and agents that need to interact with live information. Žemaitytė shared the challenge of delivering SERP data with sub-second latency. Oxylabs' traditional SERP scraper averaged around 4 seconds, necessitating a complete redesign to meet the new demand.

This led to the development of a 'Fast Search API' that focuses on essential AI data points like organic results and top stories, stripping away unnecessary page elements. The journey involved significant architectural changes, including starting from scratch to achieve performance targets. "When your baseline is at 4 seconds, we are not talking about optimization. We are talking about redesign," she explained.

Scaling Challenges and the Reality of Production

Žemaitytė also touched upon the complexities of scaling infrastructure. She described an instance where Oxylabs had to scale its web unblocker from 10,000 requests per second to 60,000 requests per second in under two months. This massive increase highlighted the need for architectural shifts, reliable central components, and robust observability. The most significant bottleneck, she noted, was not generating synthetic traffic but the difficulty of organic data testing, which mimics real client usage.

She emphasized the critical difference between systems that work in development versus those that survive real-world conditions. "There is a difference between a system that works in development, a system that works in a test, and a system that actually survives reality," Žemaitytė remarked.

The Future: Infrastructure as the Foundation for AI

Concluding her presentation, Žemaitytė reiterated Oxylabs' core mission: to build the essential infrastructure layer that enables companies to extract and operate public web data at scale. She summarized the key takeaways, emphasizing that the next generation of AI will be powered by this underlying infrastructure. "The next generation of AI will not be powered by better models. It will be powered by better infrastructure around it," she asserted.

This infrastructure, she explained, must be capable of connecting models to reality, delivering fresh web data directly into AI workflows, and scaling to handle billions of requests. The overarching lesson from Oxylabs' journey is that innovation is a continuous process of adaptation under pressure. As Žemaitytė put it, "What innovation is, is the ability to keep adapting fast enough that changing requirements becomes a new infrastructure."