Creepy crawlies

Simon Willison's Blog News
web-crawlers linux-kernel cpu-usage scrapers datasette tech-news

Summary

Konstantin Ryabitsev discusses how abusive web crawlers are consuming excessive CPU resources at git.kernel.org, impacting legitimate access and raising concerns for services like Datasette.

No content available
Original Article
View Cached Full Text

Cached at: 09/08/26, 12:21 AM

# Creepy crawlies Source: [https://simonwillison.net/2026/Sep/7/creepy-crawlies/](https://simonwillison.net/2026/Sep/7/creepy-crawlies/) 7th September 2026 \- Link Blog **[Creepy crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies)**\([via](https://news.ycombinator.com/item?id=49491791)\) Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of[git\.kernel\.org](https://git.kernel.org/), the official Git repository for the Linux kernel: > TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones\. At any one time, across 5 geo\-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html\. I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages\.

Similar Articles

Creepy crawlies

Lobsters Hottest

AI crawlers are causing significant CPU load on kernel.org by inefficiently scraping git commits for training data, consuming more resources than all legitimate access combined.

Caught a .git/config crawler

Lobsters Hottest

The author describes how their honeypot website caught a .git/config crawler by serving fake git repository data, and analyzes Apache logs showing heavy crawling activity from a single IP address.