Tag
Wellcome Collection describes how a surge of AI-related crawler traffic and bot floods strained its digital collections during mid-2026, and how it responded via robots.txt updates, CDN rate limiting, and traffic compression. The post details escalating waves of automated traffic, including botnet-like floods, and the engineering work needed to keep its open catalogue and API online.
The article discusses the challenges of blocking old browser user-agents to reduce crawler load and provides specific notes for feed readers and archival services affected by these measures.
The article describes how reputable sites blocking AI crawlers forces AI models to rely on low-quality, AI-generated content, leading to issues like post-hoc citation and infrastructure-level confirmation bias, and offers advice for better source verification.
Cloudflare announces a new setting that allows website owners to stay discoverable in search engines while opting out of AI training, with Apple, Google, and Microsoft committing to honor it.
Konstantin Ryabitsev discusses how abusive web crawlers are consuming excessive CPU resources at git.kernel.org, impacting legitimate access and raising concerns for services like Datasette.