Tag
Meta's updated commercial terms, effective October 30, may grant it permission to crawl and scrape content from websites associated with businesses using its platforms—including news publishers—to train its AI, potentially overriding robots.txt directives and bypassing licensing agreements, though Meta insists it is merely consolidating existing terms.
博主分享如何成功让自己的网站被 Google 除名（deindex），以应对 AI 抓取带来的内容被肆意爬取的问题，并解释了 robots.txt 与 meta robots 标签配合使用的技巧。
A study of robots.txt files from top 10,000 sites reveals that most block AI training crawlers like GPTBot while largely ignoring answer-time bots such as OAI-SearchBot, highlighting a blind spot in current web governance for AI.
Analysis of 11 million crawler logs across 34 websites reveals distinct behaviors: GPTBot crawls relentlessly ignoring robots.txt, Google's bot checks rules frequently, ClaudeBot's crawling is rapidly accelerating, and Bytespider is the heaviest crawler. The findings suggest a shift from Google-centric SEO to optimizing for AI agent page selection.
The article discusses how AI coding assistants make large-scale web scraping accessible to ordinary people, raising ethical concerns about ignoring robots.txt and rate limits, and questions the responsibility of AI providers.
A commentary on the ethical challenges of AI agents ignoring website rules like robots.txt when generating scrapers, and the responsibility of AI providers to implement guardrails without hindering product usability.
A developer optimized their website for AI bots by fixing robots.txt, adding llms.txt, improving semantic HTML, and more, resulting in a 12x increase in AI traffic the next day.
Amazonbot, Amazon's web crawling bot, now respects robots.txt directives, marking a change in its previous behavior.