Each AI agent crawls website completely differently. Here's what 3 mons of 11 million event logs actually show.

Each AI agent behaves completely differently when it visits your website. Here's what we found after 3 months of tracking 11 millions of real crawler logs across 34 websites. * **GPTBot:** It crawls relentlessly, all day every day, and it barely checks the rules. in 280k crawls across 23 sites, it pulled up robots.txt just 9 times. It just takes. * **Google's bot:** the opposite of GPTBot. it re-fetched robots.txt 8,765 times, checking the rules over and over. 25 years of crawling taught it manners the new AI bots never learned. * **ClaudeBot:** across the sites we track, its crawling went from 7.3k (Apr) → 64k (May) → 168k in the first ten days of June. It is racing to read as much of the web as it can, and that race is the whole story (more below). * **The live ones:** when someone asks an AI about your business, it skips your whole site and grabs the single page that answers. on Claude's live bot, 75% of those visits are one page. it ignores everything else you ever published. the page an AI picks to represent you is the whole game now. * **Bytespider:** the heaviest crawler we logged all quarter belongs to the company that owns TikTok. on one site, it made 1.2 million visits, more than Google and every OpenAI crawler combined. even the familiar names are repurposed now. * **Microsoft's Bing:** still crawls like the search engine it always was, but everything it indexes now also feeds Copilot. * **MetaBot** almost never checks robots.txt, yet it is one of the only crawlers that reads an llms.txt file at all. Every one of these companies is building its own copy of the web. its own crawler, its own index, its own answer. Anthropic is not crawling that hard for fun. they all want to be the place people ask, which means they all want to stop depending on Google. my bet: Google's ranking matters a little less every quarter from here. when this many AIs read your site their own way to build their own index, "rank #1 on Google" stops being the thing to optimize for, being the page each AI picks is.