又在 GitHub 挖到一款高性能爬虫/抓取利器：AnyCrawl，把数据采集这件事做得更省心、更高效。

它把 Cheerio、Playwright、Puppeteer 三种引擎打包到一起：静态页面秒解析，复杂 JavaScript 渲染也能稳稳拿下，还支持批量抓取 Google 等搜索结果。

GitHub：http://github.com/any4ai/anycrawl

核心能力一览：

• 多引擎切换：静态解析与 JS 渲染按需切换
• SERP 抓取：批量采集 Google 等搜索引擎结果
• 站点级爬虫：单页抓取到整站智能遍历，一套搞定
• 高性能并发：多线程/多进程架构，任务量上来照样跑得快
• 代理支持：内置 HTTP / SOCKS 代理，适配更多网络环境
• 面向 LLM：为大模型数据处理优化输出格式，减少二次清洗成本

同时支持 Docker 一键部署，配置轻量，上手快，并配有清晰的使用文档。

any4ai/anycrawl

Source: https://github.com/any4ai/anycrawl

AnyCrawl

Sponsors

Swiftproxy(https://www.swiftproxy.net/?ref=AnyCrawl) — High-performance residential proxies built for scraping, automation, and large-scale data collection. Access 80M+ rotating residential IPs across 195+ countries with stable connections, high anonymity, and developer-friendly integration. Ideal for AI agents, crawlers, browser automation, and anti-bot bypass workflows. Free trial available. Use code PROXY90 for an exclusive 10% discount.

📖 Overview

AnyCrawl is a high‑performance crawling and scraping toolkit:

• SERP crawling: multiple search engines, batch‑friendly
• Web scraping: single‑page content extraction
• Site crawling: full‑site traversal and collection
• High performance: multi‑threading / multi‑process
• Batch tasks: reliable and efficient
• AI extraction: LLM‑powered structured data (JSON) extraction from pages

LLM‑friendly. Easy to integrate and use.

🚀 Quick Start

📖 See full docs: Docs

Generate an API Key (self-host)

If you enable authentication (ANYCRAWL_API_AUTH_ENABLED=true), generate an API key:

pnpm --filter api key:generate
# optionally name the key
pnpm --filter api key:generate -- default

The command prints uuid, key and credits. Use the printed key as a Bearer token.

Run Inside Docker

If running AnyCrawl via Docker:

• Docker Compose:

docker compose exec api pnpm --filter api key:generate
docker compose exec api pnpm --filter api key:generate -- default

• Single container (replace <container_name_or_id>):

docker exec -it <container_name_or_id> pnpm --filter api key:generate
docker exec -it <container_name_or_id> pnpm --filter api key:generate -- default

📚 Usage Examples

💡 Use the Playground to test APIs and generate code in your preferred language.

If self‑hosting, replace https://api.anycrawl.dev with your own server URL.

Web Scraping (Scrape)

Example

curl -X POST https://api.anycrawl.dev/v1/scrape \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "url": "https://example.com",
  "engine": "cheerio"
}'

Parameters

More parameters: see Request Parameters.

Cache details (self-host / S3 / map index): see docs/cache.md.

Browser Runtime

The public scrape and crawl engine values remain cheerio, playwright, and puppeteer. For self-hosted browser engines, playwright and puppeteer are launched through CloakBrowser by default; callers should not send a cloakbrowser engine value.

CloakBrowser requires Node.js 20 or newer. Docker images pre-install its browser binary during image build. For local or custom deployments, set CLOAKBROWSER_CACHE_DIR to a stable writable path and CLOAKBROWSER_AUTO_UPDATE=false to avoid browser downloads during worker startup. If you manage the binary yourself, set CLOAKBROWSER_BINARY_PATH.

LLM Extraction

curl -X POST "https://api.anycrawl.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_ANYCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "json_options": {
      "schema": {
        "type": "object",
        "properties": {
          "company_mission": { "type": "string" },
          "is_open_source": { "type": "boolean" },
          "employee_count": { "type": "number" }
        },
        "required": ["company_mission"]
      }
    }
  }'

Atlas Cloud Provider

AnyCrawl supports Atlas Cloud as an OpenAI-compatible LLM provider for extraction and summarization workloads.

• Official site: Atlas Cloud
• LLM base URL: https://api.atlascloud.ai/v1
• Recommended env model format: atlascloud/deepseek-v3

ATLASCLOUD_BASE_URL=https://api.atlascloud.ai/v1
ATLASCLOUD_API_KEY=your-atlascloud-api-key
DEFAULT_LLM_MODEL=atlascloud/deepseek-v3
DEFAULT_EXTRACT_MODEL=atlascloud/deepseek-v3

If you prefer file-based AI config, add an atlascloud provider entry in ai.config.json and map it to any Atlas Cloud model exposed through the OpenAI-compatible chat API.

Site Crawling (Crawl)

Example

curl -X POST https://api.anycrawl.dev/v1/crawl \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "url": "https://example.com",
  "engine": "playwright",
  "max_depth": 2,
  "limit": 10,
  "strategy": "same-domain"
}'

Parameters

More parameters and endpoints: see Request Parameters.

Search Engine Results (SERP)

Example

curl -X POST https://api.anycrawl.dev/v1/search \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "query": "AnyCrawl",
  "limit": 10,
  "engine": "google",
  "lang": "all"
}'

Parameters

Supported search engines

• Google

❓ FAQ

1. Can I use proxies? Yes. AnyCrawl ships with a high‑quality default proxy. You can also configure your own: set the proxy request parameter (per request) or ANYCRAWL_PROXY_URL (self‑hosting).
2. How to handle JavaScript‑rendered pages? Use the Playwright or Puppeteer engines.

🤝 Contributing

We welcome contributions! See the Contributing Guide.

Backers

Support us with a monthly donation and help us continue our activities. [Become a backer]

📄 License

MIT License — see LICENSE.

🎯 Mission

We build simple, reliable, and scalable tools for the AI ecosystem.