web-scraping

A study of robots.txt files from top 10,000 sites reveals that most block AI training crawlers like GPTBot while largely ignoring answer-time bots such as OAI-SearchBot, highlighting a blind spot in current web governance for AI.

A developer built a non-AI-based checker that detects hidden instructions on web pages designed to deceive AI agents, addressing a vulnerability where pages can instruct agents to lie about their safety.

AnyCrawl is a high-performance open-source crawler/scraping tool that integrates Cheerio, Playwright, and Puppeteer engines. It supports static parsing and JS rendering, batch SERP scraping, site-level crawling, multi-threaded/multi-process concurrency, proxy support, and optimized output formats for LLM data collection.

.MD this page is an open-source browser extension that converts web pages into clean, LLM-ready Markdown using Mozilla's Readability, with features like one-click capture, preview, and export.

This paper proposes a constrained, verifiable agent framework for open-web data collection that shifts LLM output from free-form code to typed JSON collector configurations, achieving zero execution-stage LLM tokens and low latency on 80 tasks.

Cloudflare announces a new default policy to block mixed-use web crawlers from ad-hosted pages starting September 2026, aiming to force AI companies to pay for publishers' content and distinguish search from AI training.

This article categorizes 14 web scraping tools into five groups: AI new paradigms, engineering-grade frameworks, browser automation, China-specific platforms, and modern lightweight tools, accompanied by real-world cases and selection recommendations.

Context.dev provides a single API for scraping, enriching, and extracting data from the internet.

Agent Reach is a single CLI tool that gives AI agents instant access to Twitter, Reddit, YouTube, GitHub, Bilibili, and XiaoHongShu without API keys or fees. It is 100% open source and compatible with various agent frameworks.

NousResearch's Hermes Agent now reads web pages up to 60x faster and 49x cheaper by using optimized scraping backends and paging large pages locally.

This post describes how to set up a Coinbase Agentic Wallet to use Apify's web-data tools by paying USDC on Base over the x402 protocol, eliminating the need for an Apify account or API key.

MediaCrawler is an open-source multi-platform social media crawler tool with over 54,000 stars on GitHub. It supports data collection from 7 major platforms including Xiaohongshu, Douyin, Bilibili, etc., and features multi-account, IP proxy, breakpoint resume, and AI integration.

A pro tip for Hermes agents: use the free open-source repo Agent Reach to give agents internet access, enabling them to read YouTube, Reddit, X, LinkedIn, and GitHub, reducing token costs and improving research efficiency.

Introduced the open-source project Proxifly on GitHub, which automatically fetches and verifies available proxies every 5 minutes, covers 60+ countries, supports multiple protocols, and provides downloads in multiple formats as well as npm installation.

Tweet de @ecommartinez que lista 10 repositorios de GitHub para hacer web scraping y extraer datos limpios de cualquier sitio web.

A commentary on the shifting attitudes towards web scraping for AI training, questioning the sudden condemnation of data collection without permission.

Open Agent Builder is a visual workflow builder for creating AI agent pipelines powered by Firecrawl, enabling drag-and-drop design of multi-step scraping and automation workflows with real-time execution updates.

Agent-Reach is a free and open-source repository that, with a single command, gives AI agents access to 14 platforms, including web pages, YouTube, RSS, Bilibili, and Xiaohongshu, without complex configuration, significantly lowering the barrier for agents to go online.

MediaCrawler is a multi-platform social media data scraping tool that supports public content crawling from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. It bypasses JS reverse engineering by leveraging browser login state, lowering the technical barrier.

Selector Forge is a browser extension that uses AI to generate and verify reliable CSS/XPath selectors for web automation, helping developers build robust selectors for testing, scraping, and page automation.