Artificial Intelligence

Oxylabs: Web Data Infrastructure Powers AI's Future

Patricija Žemaitytė of Oxylabs discusses how web data infrastructure is crucial for AI, detailing challenges in scaling, latency, and the iterative nature of innovation.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

4 min read

Patricija Žemaitytė presenting on stage at AI Engineer World's Fair
Credit: AI Engineer
Visual TL;DR
AI Needs Live DataDriver
From the article 2 mentions"Training alone is no longer enough, and to stay useful, models need to get access to fresh information, live search, real external data," she stated.
Beyond Static TrainingContext
From the articleŽemaitytė highlighted a significant shift in the industry, moving away from relying solely on static training data.
Oxylabs: Web Data InfraCore
From the article 4 mentionsConcluding her presentation, Žemaitytė reiterated Oxylabs' core mission: to build the essential infrastructure layer that enables companies to extract and operate public web data at scale.
Innovation Under PressureContext
developing a Video API Suite to meet specific, high-demand data needs
From the articleThe overarching lesson from Oxylabs' journey is that innovation is a continuous process of adaptation under pressure.
SERP Data SpeedDriver
quest for low latency in search engine results page data delivery
From the articleThe conversation then shifted to the critical role of SERP (Search Engine Results Page) data in AI systems, particularly for retrieval pipelines, assistants, and agents that need to interact with live information. Žemaitytė shared the challenge of delivering SERP data with sub-second latency.
Scaling ChallengesDriver
addressing the reality of production and scaling web data infrastructure
From the article 4 mentionsThis initial challenge led to the development of a comprehensive video API suite, including transcript support, subtitle functionality, and search capabilities, demonstrating that "innovation never comes as a neat road map.
Foundation for AIOutcome
infrastructure is the critical underlying layer powering AI's future
AI's Future PoweredEffect
web data infrastructure crucial for AI's continued evolution and utility
From the article 3 mentionsShe summarized the key takeaways, emphasizing that the next generation of AI will be powered by this underlying infrastructure.
Contents(5)

In the rapidly evolving AI landscape, the underlying infrastructure that feeds models with data is becoming as critical as the models themselves. Patricija Žemaitytė, Product Manager at Oxylabs, a web intelligence and premium proxy provider, shared insights at the AI Engineer World's Fair on how Oxylabs builds the infrastructure layer essential for companies to extract and utilize public web data at scale.

Oxylabs: Web Data Infrastructure Powers AI's Future - AI Engineer
Oxylabs: Web Data Infrastructure Powers AI's Future, AI Engineer

The Evolving Role of Web Data in AI

Žemaitytė highlighted a significant shift in the industry, moving away from relying solely on static training data. She explained that for AI models to remain relevant and useful, they require access to fresh, live information. "Training alone is no longer enough, and to stay useful, models need to get access to fresh information, live search, real external data," she stated. Without this, even the most advanced models are limited by their knowledge cutoff.

Innovation Born from Pressure: The Video API Suite

Žemaitytė recounted her experience at Oxylabs, where a client's urgent request for video API capabilities for AI training, with a two-week deadline and a demand for 5 petabytes per month, spurred significant development. This wasn't just about downloading videos; it was about building a complete pipeline for collection, transfer, storage, and delivery, all with the reliability needed for AI training. This initial challenge led to the development of a comprehensive video API suite, including transcript support, subtitle functionality, and search capabilities, demonstrating that "innovation never comes as a neat road map. It comes as a pressure, as a deadline, and sometimes, and quite often, as a trip report from San Francisco."

The Quest for Speed: SERP Data and Low Latency

The conversation then shifted to the critical role of SERP (Search Engine Results Page) data in AI systems, particularly for retrieval pipelines, assistants, and agents that need to interact with live information. Žemaitytė shared the challenge of delivering SERP data with sub-second latency. Oxylabs' traditional SERP scraper averaged around 4 seconds, necessitating a complete redesign to meet the new demand.

This led to the development of a 'Fast Search API' that focuses on essential AI data points like organic results and top stories, stripping away unnecessary page elements. The journey involved significant architectural changes, including starting from scratch to achieve performance targets. "When your baseline is at 4 seconds, we are not talking about optimization. We are talking about redesign," she explained.

Scaling Challenges and the Reality of Production

Žemaitytė also touched upon the complexities of scaling infrastructure. She described an instance where Oxylabs had to scale its web unblocker from 10,000 requests per second to 60,000 requests per second in under two months. This massive increase highlighted the need for architectural shifts, reliable central components, and robust observability. The most significant bottleneck, she noted, was not generating synthetic traffic but the difficulty of organic data testing, which mimics real client usage.

She emphasized the critical difference between systems that work in development versus those that survive real-world conditions. "There is a difference between a system that works in development, a system that works in a test, and a system that actually survives reality," Žemaitytė remarked.

The Future: Infrastructure as the Foundation for AI

Concluding her presentation, Žemaitytė reiterated Oxylabs' core mission: to build the essential infrastructure layer that enables companies to extract and operate public web data at scale. She summarized the key takeaways, emphasizing that the next generation of AI will be powered by this underlying infrastructure. "The next generation of AI will not be powered by better models. It will be powered by better infrastructure around it," she asserted.

This infrastructure, she explained, must be capable of connecting models to reality, delivering fresh web data directly into AI workflows, and scaling to handle billions of requests. The overarching lesson from Oxylabs' journey is that innovation is a continuous process of adaptation under pressure. As Žemaitytė put it, "What innovation is, is the ability to keep adapting fast enough that changing requirements becomes a new infrastructure."

Mentioned in this story

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsOxylabsPatricija ŽemaitytėAIWeb DataInfrastructureData ExtractionWeb ScrapingAI Training

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.