web-crawling

Tag

Cards List
#web-crawling

Amazon is planning a World Product Graph to index every product online, partly to deal with AI shopping agents

Reddit r/AI_Agents ↗ · 6d ago

Reuters reports Amazon is planning a 'World Product Graph' to index nearly every product sold online, partly to make third-party AI shopping agents depend on Amazon's catalog instead of other sites. The internal PR/FAQ also notes Rufus's answers are often 'factually wrong or incomplete,' though Amazon says those figures are outdated and accuracy is about 97%.

0 favorites 0 likes
#web-crawling

Supporting a shared AI training control for websites (2 minute read)

TLDR AI ↗ · 2026-11-02 Cached

Google announced plans to support a shared AI training control in robots.txt, based on IETF AIPREF work, letting websites express opt-out preferences across multiple AI providers in a single step, with rollout targeted for 2027.

0 favorites 0 likes
#web-crawling

Making it easier for websites to express AI training preferences

OpenAI Blog ↗ · 2026-10-30 Cached

OpenAI announced it will honor a shared robots.txt signal from the IETF AI Preferences working group, letting website owners express AI training preferences once across multiple providers instead of managing rules for each crawler individually.

0 favorites 0 likes
#web-crawling

The uneven rise of AI-generated web content

arXiv cs.AI ↗ · 2026-10-28 Cached

This arXiv paper analyzes over 5 million webpages from 2019 to 2026, finding that the share of pages with AI-generated text grew from near zero before 2022 to nearly 25% by August 2026. The rise is uneven: English-language and .com domains have higher AI prevalence than other languages or .gov/.edu domains, and AI text concentrates in the least central pages of the web graph, with estimates produced via a Bayesian hierarchical model correcting for detector error rates.

0 favorites 0 likes
#web-crawling

Wikimedia's agent incident report reads like a bug report for a missing permission contract

Reddit r/AI_Agents ↗ · 2026-10-20

分析Wikimedia基金会的AI智能体事故报告:自主智能体未经授权大量抓取页面、发起重查询并进行了未授权编辑,可能促成Wikidata Query Service五月的部分宕机。核心问题在于缺乏明确的权限契约——没有速率限制、没有读写边界,导致智能体在无人定义合规行为的情况下造成服务中断。

0 favorites 0 likes
#web-crawling

Host-Aware Web Crawling for LLM Pretraining Data

arXiv cs.CL ↗ · 2026-10-12 Cached

该论文研究了 LLM 预训练数据采集中的爬虫预算分配问题,发现文档质量在主机层面存在显著聚集性,通过先采样少量页面再按质量深入爬取,可比统一深度或基于中心度的分配多收集数倍高质量页面。

0 favorites 0 likes
#web-crawling

@huoshan007: I dug up a beast. crawl4ai, GitHub's top crawler, turned one guy's rage from getting scammed by a $16 paid service stra…

X AI KOLs Timeline ↗ · 2026-10-11 Cached

推文介绍了GitHub上爆红的开源爬虫项目crawl4ai(51k星),可将任意网站内容转为LLM可直接使用的干净Markdown,无需API密钥或付费订阅,被认为打破了付费爬虫服务的价值壁垒。

0 favorites 0 likes
#web-crawling

I opened my small website to AI agents and logged who showed up. Mostly it’s bots indexing bots.

Reddit r/AI_Agents ↗ · 2026-10-10

一位小网站站长开放了对 AI 代理的支持(llms.txt、Markdown、MCP 服务器、按次付费端点)并记录流量:绝大多数是 Meta、Amazon 等 AI 爬虫以及目录和扫描类 MCP 客户端,真正干活的代理调用仅有 3 次,说明当前的“代理互联网”主要是机器人在索引机器人。

0 favorites 0 likes
#web-crawling

Dumping your whole site into the agent is making it worse and everyone keeps doing it anyway

Reddit r/AI_Agents ↗ · 2026-10-09

作者批评把整站页面粗暴灌入向量库的做法,主张先爬取发现 URL、再用简单路径过滤大幅缩减索引规模,因为智能体有任务和 token 预算,无法容忍大量噪声内容。

0 favorites 0 likes
#web-crawling

Quantifying Organizational Environmental Action from Web Data and Large Language Models

arXiv cs.CL ↗ · 2026-09-16 Cached

A scalable computational framework using web data and large language models is presented to quantify organizational environmental actions, demonstrated on U.S. Jewish congregations, comparing keyword retrieval, semantic retrieval, and direct classification approaches.

0 favorites 0 likes
#web-crawling

Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

arXiv cs.LG ↗ · 2026-09-10 Cached

Data Scout is a targeted web crawling method that uses an LLM to expand topics into queries and screen content, significantly improving efficiency for creating domain-specific pretraining corpora compared to filtering large archives.

0 favorites 0 likes
#web-crawling

Some (mostly historical) issues with the Unix load average

Lobsters Hottest ↗ · 2026-09-10 Cached

The article details the blog author's measures to block old browsers to reduce load from malicious crawlers, likely for LLM training, and addresses specific issues with feed readers like Inoreader and Feedly, as well as browsers like Vivaldi and archive services.

0 favorites 0 likes
#web-crawling

LinkedIn returns HTTP 999 to GPTBot and ClaudeBot but HTTP 200 to OAI-SearchBot. I measured what is inside the 200.

Reddit r/artificial ↗ · 2026-09-01

LinkedIn blocks AI crawlers like GPTBot and ClaudeBot but allows search-index bots such as OAI-SearchBot, providing limited profile data that omits key information like job titles and experience dates.

0 favorites 0 likes
#web-crawling

@TheAhmadOsman: PRO TIP RSS feeds are back btw It's so easy to turn any API or web app into an RSS feed with agents now Give GLM 5.3 Fl…

X AI KOLs Timeline ↗ · 2026-08-30 Cached

A pro tip on leveraging AI agents like GLM 5.3 Flash to convert any API or web app into an RSS feed by crawling, mapping endpoints, and building custom dashboards.

0 favorites 0 likes
#web-crawling

@tom_doerr: Crawls websites to transform extracted web content into LLM-ready data structures. https://github.com/watercrawl/WaterC…

X AI KOLs Timeline ↗ · 2026-08-27 Cached

WaterCrawl is an open-source web application that crawls websites to transform extracted content into LLM-ready data structures, built with Python, Django, Scrapy, and Celery.

0 favorites 0 likes
#web-crawling

@binghe: The Cyber Bodhisattva of Web Scraping: Crawl4ai Free, AI-Powered Scraping Tool. 78.9k Free: No registration, no API keys, no per-page fees, a replacement for $16/month paid scrapers. Built for AI: Converts any complex webpage to Markdown with one click, directly usable by large language models,…

X AI KOLs Timeline ↗ · 2026-08-21 Cached

Crawl4ai is a free, AI-oriented scraping tool that converts web content to Markdown, ideal for RAG and Agent development.

0 favorites 0 likes
#web-crawling

@WilliamBryk: Exa is now one of the largest indexes in the world. We serve 80B pages, track 1.4T urls, and are on track to be Google-…

X AI KOLs Following ↗ · 2026-08-03 Cached

Exa announces it serves 80B pages and tracks 1.4T URLs, positioning itself as one of the largest web indexes, with plans to reach Google-scale by early 2027.

0 favorites 0 likes
#web-crawling

Most Googlebots are fake

Lobsters Hottest ↗ · 2026-07-27 Cached

The article explains that many bots claiming to be Googlebot are actually fake, impersonating its user agent to bypass bot controls, and provides guidance on how to verify legitimate Googlebot traffic.

0 favorites 0 likes
#web-crawling

Imagine you're building a RAG chatbot that trained on an entire website. How Would you crawl the entire site

Reddit r/AI_Agents ↗ · 2026-06-28

A discussion on how to crawl an entire website to train a RAG chatbot, covering strategies and challenges.

0 favorites 0 likes
#web-crawling

Each AI agent crawls website completely differently. Here's what 3 mons of 11 million event logs actually show.

Reddit r/AI_Agents ↗ · 2026-06-10

Analysis of 11 million crawler logs across 34 websites reveals distinct behaviors: GPTBot crawls relentlessly ignoring robots.txt, Google's bot checks rules frequently, ClaudeBot's crawling is rapidly accelerating, and Bytespider is the heaviest crawler. The findings suggest a shift from Google-centric SEO to optimizing for AI agent page selection.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback