Tag
Reuters reports Amazon is planning a 'World Product Graph' to index nearly every product sold online, partly to make third-party AI shopping agents depend on Amazon's catalog instead of other sites. The internal PR/FAQ also notes Rufus's answers are often 'factually wrong or incomplete,' though Amazon says those figures are outdated and accuracy is about 97%.
Google announced plans to support a shared AI training control in robots.txt, based on IETF AIPREF work, letting websites express opt-out preferences across multiple AI providers in a single step, with rollout targeted for 2027.
OpenAI announced it will honor a shared robots.txt signal from the IETF AI Preferences working group, letting website owners express AI training preferences once across multiple providers instead of managing rules for each crawler individually.
This arXiv paper analyzes over 5 million webpages from 2019 to 2026, finding that the share of pages with AI-generated text grew from near zero before 2022 to nearly 25% by August 2026. The rise is uneven: English-language and .com domains have higher AI prevalence than other languages or .gov/.edu domains, and AI text concentrates in the least central pages of the web graph, with estimates produced via a Bayesian hierarchical model correcting for detector error rates.
分析Wikimedia基金会的AI智能体事故报告:自主智能体未经授权大量抓取页面、发起重查询并进行了未授权编辑,可能促成Wikidata Query Service五月的部分宕机。核心问题在于缺乏明确的权限契约——没有速率限制、没有读写边界,导致智能体在无人定义合规行为的情况下造成服务中断。
该论文研究了 LLM 预训练数据采集中的爬虫预算分配问题,发现文档质量在主机层面存在显著聚集性,通过先采样少量页面再按质量深入爬取,可比统一深度或基于中心度的分配多收集数倍高质量页面。
推文介绍了GitHub上爆红的开源爬虫项目crawl4ai(51k星),可将任意网站内容转为LLM可直接使用的干净Markdown,无需API密钥或付费订阅,被认为打破了付费爬虫服务的价值壁垒。
一位小网站站长开放了对 AI 代理的支持(llms.txt、Markdown、MCP 服务器、按次付费端点)并记录流量:绝大多数是 Meta、Amazon 等 AI 爬虫以及目录和扫描类 MCP 客户端,真正干活的代理调用仅有 3 次,说明当前的“代理互联网”主要是机器人在索引机器人。
作者批评把整站页面粗暴灌入向量库的做法,主张先爬取发现 URL、再用简单路径过滤大幅缩减索引规模,因为智能体有任务和 token 预算,无法容忍大量噪声内容。
A scalable computational framework using web data and large language models is presented to quantify organizational environmental actions, demonstrated on U.S. Jewish congregations, comparing keyword retrieval, semantic retrieval, and direct classification approaches.
Data Scout is a targeted web crawling method that uses an LLM to expand topics into queries and screen content, significantly improving efficiency for creating domain-specific pretraining corpora compared to filtering large archives.
The article details the blog author's measures to block old browsers to reduce load from malicious crawlers, likely for LLM training, and addresses specific issues with feed readers like Inoreader and Feedly, as well as browsers like Vivaldi and archive services.
LinkedIn blocks AI crawlers like GPTBot and ClaudeBot but allows search-index bots such as OAI-SearchBot, providing limited profile data that omits key information like job titles and experience dates.
A pro tip on leveraging AI agents like GLM 5.3 Flash to convert any API or web app into an RSS feed by crawling, mapping endpoints, and building custom dashboards.
WaterCrawl is an open-source web application that crawls websites to transform extracted content into LLM-ready data structures, built with Python, Django, Scrapy, and Celery.
Crawl4ai is a free, AI-oriented scraping tool that converts web content to Markdown, ideal for RAG and Agent development.
Exa announces it serves 80B pages and tracks 1.4T URLs, positioning itself as one of the largest web indexes, with plans to reach Google-scale by early 2027.
The article explains that many bots claiming to be Googlebot are actually fake, impersonating its user agent to bypass bot controls, and provides guidance on how to verify legitimate Googlebot traffic.
A discussion on how to crawl an entire website to train a RAG chatbot, covering strategies and challenges.
Analysis of 11 million crawler logs across 34 websites reveals distinct behaviors: GPTBot crawls relentlessly ignoring robots.txt, Google's bot checks rules frequently, ClaudeBot's crawling is rapidly accelerating, and Bytespider is the heaviest crawler. The findings suggest a shift from Google-centric SEO to optimizing for AI agent page selection.