web-scraping

webclaw is an open-source tool that converts websites into clean Markdown, JSON, and LLM-ready context, available as a CLI, MCP server, REST API, and SDKs for AI agents and RAG pipelines.

Firecrawl, the well-known web scraping tool, can now be called directly without an API Key. It's very useful for AI agents and is an open-source project.

Introducing the open-source project Firecrawl: a web data API that converts any URL into clean Markdown/JSON, supports AI interactions and whole-site crawling, designed specifically for LLMs and Agents, with 25k+ GitHub stars.

WebRover is an MIT-licensed open-source AI agent that uses natural language to drive the browser for web automation, cross-site data scraping, and deep research, with support for local deployment.

A SearXNG-style metadata search engine written in Rust, fanning out queries to multiple search engines concurrently, deduplicating results, and ranking via Reciprocal Rank Fusion.

PullMD is a self-hosted URL-to-Markdown service for humans and AI agents, converting web pages, PDFs, and media files into clean Markdown, with support for an MCP server and Claude Code skill.

Reddit's DMCA lawsuit against SerpApi and Perplexity AI over Google search scraping continues, with a judge allowing discovery despite dismissing some claims. The case highlights ongoing tensions between AI scraping and copyright law.

Website to Markdown API is a developer tool that converts any website into clean, LLM-ready Markdown, making it easy to feed web content into AI models.

A static analysis scanner tested five DTC brands and found that despite good structured data, their custom JavaScript (e.g., size pickers using <div> instead of <select>) makes them inaccessible to AI shopping agents, potentially costing $82K–$491K monthly in lost revenue from AI-referred traffic.

Google and Reddit lost a court case where they used the DMCA to sue web scraper SerpApi; the judge dismissed the lawsuit, ruling Google lacked standing under the DMCA because it did not own copyrighted content in search results.

Monid is a new API that lets AI agents read social media platforms without logins or subscriptions, priced at $0.0015/request as a cheaper alternative to Apify.

The author solved the problem of Claude Code being unable to read WeChat public account articles by faking User-Agent and Referer to bypass WeChat verification, and using Python standard library to extract the title, author, and body text. He packaged this functionality into an open-source skill (rn-wechat-extract), supporting one-command extraction of full text, usable in Agent workflows.

A commentary highlighting the irony of Big Tech companies complaining about model distillation as an existential threat, given that their own empires were built on web scraping.

Agent-Reach is an open-source tool that, with a single command, connects AI agents to 14 online platforms (such as Twitter, Bilibili, YouTube, etc.), solving the pain points of web scraping and API payment, allowing agents to directly obtain online information.

The article criticizes Anubis, an HTTP proof-of-work proxy meant to block AI scrapers, showing it is trivially bypassed by AI while imposing a regressive burden on human users, especially those with weak devices or non-JavaScript browsers.

An update on the escalating problem of AI scraper bots overwhelming websites, discussing residential proxy networks and their impact on the open web.

Chris Siebenmann explains his anti-crawler measures that block old browsers due to a surge in high-volume crawlers collecting data for LLM training, causing confusion for feed readers and archival services.

Fudge MCP is a tool that gives AI agents design taste by learning from existing websites.

Crawl4AI is an open-source web crawler tool that converts web content into clean Markdown format, designed for LLM's RAG, Agents, and data pipelines. Zero API Key, fast output of structured data.

Context.dev is a YC-backed API that allows developers and AI agents to scrape, crawl, and extract structured data from any website, with features like markdown, HTML, sitemaps, screenshots, and brand intelligence, aiming to simplify web data integration.