AI bots disrupt university libraries with 11,000-search surge

Academic libraries worldwide are seeing AI scraping bots slow or crash online collections, CBC News reported in a Nov. 5 article, citing one university's 11,000 Finnish-music searches in one day.

The pressure matters because libraries, museums and cultural institutions host open digital archives that can also feed large language models; researchers found reports began in late 2024 across North America, Europe and Oceania.

University libraries told CBC that persistent automated traffic is knocking some collections offline. Their archives include academic, historical and government records meant for students, researchers and public users.

"They're just out there on the internet, just bombarding us with trying to grab up as much material as possible," Jill Strykowski, cataloguing and government documents lead at San Jose State University, told CBC News.

Strykowski said the traffic is "causing an incredible amount of work and headache for every institution that does digitization work and every institution that is trying to make information openly available."

Dean Seeman, director of collections and digital infrastructure at University of Victoria Libraries, told CBC that the automated requests can become severe enough to disable services temporarily. "Sometimes the bot traffic gets so bad that it causes the sites to go down completely for a little while," he said.

The bots are used by AI companies and third-party operators to collect web data, often for large language models such as Claude, Gemini and ChatGPT, CBC reported. Cultural institutions are exposed because much of their material is digitized, curated and accessible without payment or login barriers.

Eric Hellman, executive director of Project Gutenberg, told CBC the issue has worsened since he wrote in 2025 that poorly programmed AI bots were damaging open access. He said search engines and archives can use crawlers that follow robots.txt, a voluntary website file telling automated systems what they may access.

Hellman said some AI scraping bots do not follow those instructions and can seek thousands of documents a second, including duplicate downloads in different formats. The University of North Carolina at Chapel Hill reported last year that more than 11,000 searches for Finnish music came from thousands of internet addresses on the same day, periodically shutting students, faculty and staff out of the library catalogue.

Libraries have responded with CAPTCHA checks and other verification tools, CBC reported. Michael Weinberg, co-director of GLAM-E, an academic initiative at NYU Law School and the University of Warwick, said those tools can also block useful bots, while login requirements create barriers for people seeking open material.

Major AI firms have argued that using public web data to train models is allowed under fair use and serves the public interest, CBC reported. The article said Anthropic, OpenAI, Google and Microsoft expressed support last week for a standard robots.txt opt-out signal, but compliance would be voluntary and would not cover smaller AI developers or third-party scrapers.

Weinberg said the current scraping wave risks discouraging institutions from publishing collections online. "I think it's been a huge benefit to society that libraries and museums and other cultural institutions have put a huge amount of work into making their collections available," he told CBC.