crawler

A user praises a data scraping tool, saying that aside from the occasional need to update the Xiaohongshu API, it basically meets all scraping needs.

The author describes how their honeypot website caught a .git/config crawler by serving fake git repository data, and analyzes Apache logs showing heavy crawling activity from a single IP address.

AnyCrawl is a high-performance open-source crawler/scraping tool that integrates Cheerio, Playwright, and Puppeteer engines. It supports static parsing and JS rendering, batch SERP scraping, site-level crawling, multi-threaded/multi-process concurrency, proxy support, and optimized output formats for LLM data collection.

MediaCrawler is a multi-platform social media data scraping tool that supports public content crawling from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. It bypasses JS reverse engineering by leveraging browser login state, lowering the technical barrier.

This article introduces 10 open-source GitHub repositories for web scraping, including Firecrawl, Crawl4AI, etc., which can extract clean data from websites and support AI-ready formats.

A viral open-source web crawling tool called Crawl4AI offers free, LLM-friendly scraping with features like JavaScript rendering, async crawling, and clean structured output, contrasting with paid services like Firecrawl.

datasette-ip-rate-limit 0.1a0 is a configurable rate-limiting plugin for Datasette, built to block aggressive crawlers.

MediaCrawler是一个开源的多平台自媒体数据采集工具，支持小红书、抖音、快手、B站、微博、贴吧、知乎等主流平台的公开信息抓取，基于Playwright浏览器自动化实现，无需JS逆向。