Artificial Intelligence

AI scrapers are taking libraries offline

AI scraping bots are overloading library servers like a denial of service attack, but no one has proved who runs most of them.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

University library systems are buckling under remote, high-volume AI scraping that needs no login, only endless complex queries. CTV News reported new University of Edinburgh research on rising costs and lost access, with Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University, saying the load can mimic a denial of service attack.

AI scrapers are taking libraries offline
AI scrapers are taking libraries offline

The obvious reading is that greedy labs are looting public knowledge. I am not convinced the villain story explains the failure, because no one has shown who runs most of the bots.

Lewis said one library logged more than two million bot queries in four weeks linked to firms including Apple, OpenAI, Anthropic, Google, Tencent and Meta. He cited more than 11,000 searches in a single day in 2024 for niche music holdings at the University of North Carolina at Chapel Hill Library. The fair concession is that libraries run lean systems built for students, not industrial harvest, so heavy queries can knock human users offline.

Attribution still stops at brand names on traffic logs.

In October, Lewis said OpenAI, Google, Microsoft and Anthropic agreed to respect a policy token for training opt-out, a voluntary industry fix. Lewis argued four volunteers do not make a rule when labs are racing for higher quality data and niche expertise, the two drivers he said are pushing crawlers toward libraries, archives and museums. His preferred fix is platform charging agreements that set load and payment up front, with per-query micro-charges as a weaker technical cousin.

CP24 separately described the incidents as outages at university library websites, a harder failure than slow service. If you want library data, negotiate load and payment before you crawl, because free riding at this scale reads as an attack.

Mentioned in this story

Markets

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsOpenAIGoogleAnthropicAppleMetaMicrosoftUniversity of EdinburghOntario Tech University

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.