14 Web Scraping Tools, Categorized into 5 Types with Real Cases – No Need to Look Elsewhere After Reading This

A fellow developer asked me privately: “You’ve shared several scraping-related projects recently, what’s the difference between them, and which one should I use?”

I counted – including this time – it’s 14. Just so you don’t have to hunt around. I’ve done the heavy lifting for you.

Today I’ve gathered them all in one place, added a scenario-based classification, and explained everything at once.

Category 1: New AI-era Species – Turning Web Pages into Something LLMs Can Read Directly

The core of these tools isn’t “grabbing data,” it’s “feeding web pages to AI.”

Firecrawl · 130,000 Stars · YC Incubated

Give it a domain, and it crawls the entire site, cleans it, and converts it to Markdown, with API support. The biggest feature is the hosted service – no need to build infrastructure yourself. YC-backed, the most mature in commercialisation. Suitable for teams that need quick integration and don’t want to mess with infrastructure. (Fun fact: there are API resellers on Xianyu that are cheap enough to make you swear.)

→ github.com/firecrawl/firecrawl

Crawl4AI · 70,000 Stars

Runs locally, data never leaves, complete data sovereignty. The top choice for LLM integration. Asynchronous, fast, can handle many pages simultaneously. Suitable for scenarios where you have your own server and need data to run locally, or you just don’t want to send data to third parties.

→ github.com/unclecode/crawl4ai

ScrapeGraphAI · 20,000 Stars

The most unique one – you say in natural language, “Help me grab all product names and prices from this page,” and the LLM automatically generates the scraping flow for you, executed as a graph structure. No need to write selectors, no XPath knowledge required. Suitable for people who can’t write scraping code but just want results, or for quickly validating ideas.

→ github.com/ScrapeGraphAI/Scrapegraph-ai

Real Cases

• Firecrawl — LangChain, LlamaIndex, and Dify official docs all have integration tutorials; “full-site to RAG knowledge base” is already a standard component; API resellers on Xianyu offer it at insane prices.
• Crawl4AI — The go-to for indie developers building local knowledge bases. Batch pulling industry articles into Obsidian or self-built vector stores is one of the most common personal AI workflow combos today.
• ScrapeGraphAI — Competitive intelligence scenarios: let the LLM directly “read prices / read specs” from competitor pages without maintaining selectors, even if the site redesigns.

How to Choose Within This Category

• Team project, don’t want to run a service → Firecrawl (hosted version)
• Run locally, data stays in-house → Crawl4AI
• Describe requirements and let AI scrape, quick validation → ScrapeGraphAI

Category 2: Engineering-Grade Scraping Frameworks – Production Environment Ready

These are “proper scraping frameworks” for engineers who need stability, long-term operation, and large-scale crawling.

Scrapy · 62,000 Stars · Python

The elder statesman from 2008, still the default choice for production Python scraping. Three-layer architecture: Spider + Pipeline + Middleware. Includes scheduling, rate limiting, proxy rotation, and breakpoint resume. Mature ecosystem, all pitfalls have been discovered. Downside: no built-in JS rendering (needs extra Playwright plugin). Suitable for engineers doing e-commerce price monitoring, news aggregation, data ETL that needs to run stably long-term.

→ github.com/scrapy/scrapy

PySpider · 16,000 Stars · Python

Comes with a Web UI visualisation platform. You can write scraping code, debug, and view results right in the browser, with built-in task scheduling and JS rendering support. More friendly to people who dislike command lines. Suitable for scenarios where you want a visual management interface and need JS rendering but don’t want to use Playwright.

→ github.com/binux/pyspider

Colly · 25,000 Stars · Go

Written in Go, extremely fast, concurrency performance far exceeds Python frameworks. Clean API design, suitable for engineers with Go experience. No JS rendering, pure HTML scraping. Suitable for scenarios with extreme speed requirements, high concurrency, and target pages that don’t rely on JS rendering.

→ github.com/gocolly/colly

Crawlee · 22,000 Stars · Node.js / Python · Made by Apify

Specifically designed for modern JS-heavy sites, with built-in anti-detection mechanisms, seamlessly switching between Puppeteer or Playwright drivers. Apify, a company that started with scraping services, open-sourced this internal tool, and it’s well-polished. Suitable for scenarios where target sites are SPAs, require login/session handling, or have strong anti-scraping measures.

→ github.com/apify/crawlee

Real Cases

• Scrapy — The classic use case is price comparison engines; Zyte (the company maintaining Scrapy) built their business on it and later turned it into a commercial scraping service; a team in China uses it for daily nationwide news aggregation from hundreds of media sources into a database for sentiment analysis.
• Colly — The standard in the Go ecosystem for SEO tools; scraping hundreds of thousands of pages for link analysis – Python frameworks at that scale are either slow or resource-hungry, Go handles it well.
• Crawlee — Underpins the Apify platform; thousands of public scrapers in the Marketplace run on it, fully validated in commercial use.

How to Choose Within This Category

• Python ecosystem, long-term production → Scrapy
• Want a visualisation platform → PySpider
• Go project, extreme performance → Colly
• JS-heavy sites + anti-scraping → Crawlee

Category 3: Browser Automation – The Nuclear Weapon for JS Rendering and Anti-Scraping

Strictly speaking, these aren’t just for scraping – they control real browsers. But they’re often used for data collection.

Puppeteer · 94,000 Stars · Node.js · Maintained by Google

The pioneer of Chrome automation. Maintained by Google, official API, comprehensive docs. Only supports Chromium, exclusive to Node.js. By 2026 there are better options, but Puppeteer has the most resources – you’ll find answers on StackOverflow the easiest.

→ github.com/puppeteer/puppeteer

Playwright · 90,000 Stars · Multi-language · Made by Microsoft

Supports three major browsers: Chromium + Firefox + WebKit, with SDKs for Python, Node.js, Java, and C#. By 2026, the default recommendation for new projects is Playwright – it’s more feature-complete than Puppeteer, and Microsoft is actively investing.

→ github.com/microsoft/playwright

Selenium · 33,500 Stars · Multi-language

The oldest browser automation framework, with enterprise-grade multi-language support, the top choice for the Java ecosystem. Many company legacy systems use it, ecosystem is extremely mature. However, it’s relatively slow, and modern projects generally prefer Playwright for new builds.

→ github.com/SeleniumHQ/selenium

Real Cases

• Playwright — Standard in modern SaaS CI/CD; Supabase has an official Playwright e2e test repo, Microsoft uses it in their own products; in scraping scenarios, simulating user behaviour to collect login-required data is also Playwright-based.
• Selenium — Banks and insurance companies still run massive automation tests on it because those projects are Java-based, built ten years ago, and can’t be replaced.
• Puppeteer — Appears in many Node.js full-stack projects for web page screenshot services, PDF report generation, or scraping SPA backends that require login.

How to Choose Within This Category

• New project in 2026 → Playwright (multi-language, multi-browser)
• Node.js project with legacy Puppeteer code → Keep using Puppeteer
• Java enterprise project → Selenium
• Pure scraping without testing needs → Consider Crawlee (better encapsulation for scraping scenarios)

Category 4: Domestic Platform Specialised – Can’t Bypass the Encryption

Major domestic platforms (Xiaohongshu, Douyin, Bilibili, etc.) have complex encryption algorithms. Reverse engineering them yourself is extremely costly, and they change frequently.

MediaCrawler · 51,000 Stars

Seven platforms in one codebase: Xiaohongshu, Douyin, Bilibili, Zhihu, Weibo, Kuaishou, WeChat Official Accounts. Includes keyword search, comment scraping, creator profiles, and trending lists. No need to crack platform encryption yourself. Suitable for content operations, competitive analysis, and self-media research.

→ github.com/NanmiCoder/MediaCrawler

InfoSpider · Thousands of Stars

Specifically designed for personal data export – exporting your own data from platforms like WeChat and Alipay. Driven by browser plugins, not for bulk scraping public data, but to help you reclaim your own data. A completely different scenario – it’s a personal privacy data management tool.

→ github.com/kangvcar/InfoSpider

Real Cases

• MediaCrawler — Content marketing teams use it to automatically pull trending Xiaohongshu posts in target categories daily, analysing titles/comment counts/engagement rates to generate daily reports; MCN agencies use it to monitor data trends for their accounts; researchers use it to scrape Weibo topic comments for sentiment analysis.
• InfoSpider — Data migration and backup scenarios: when switching devices or needing to store data in private storage, use it to export chat records, billing records for archiving.

Category 5: Modern Lightweight and Special Scenarios

Scrapling · 28,000 Stars

The killer feature: adaptive positioning – when the target website updates and the HTML structure changes, it automatically relocates previously scraped elements without rewriting selectors. Also includes a built-in MCP Server that can serve as a data source plugin directly for Claude. Suitable for scenarios where you need long-term maintenance of scraping tasks and the target site changes frequently.

→ github.com/D4Vinci/Scrapling

Katana · 14,000 Stars · Go · Made by ProjectDiscovery

Unique positioning: security / penetration testing oriented crawler. CLI tool, written in Go so it’s very fast. ProjectDiscovery is a security tools team (they also make Nuclei). This tool is for penetration testing and attack surface discovery, not ordinary data collection. Suitable for security research, red teaming, and attack surface crawling.

→ github.com/projectdiscovery/katana

Spider-Flow · Thousands of Stars

Graphical no-code scraper, define scraping flows by drag-and-drop. Suitable for operations staff who can’t code but need to regularly scrape data.

→ github.com/ssssssss-team/spider-flow

Real Cases

• Scrapling — The go-to for long-term price/inventory monitoring: tracking competitor pricing pages or certain job listings on recruitment sites. These pages might only change every few months, but when they do, traditional scrapers break. Scrapling’s adaptive positioning automatically recovers, saving maintenance costs.
• Katana — Standard Bug Bounty workflow: first use Katana to quickly crawl all endpoints on the target domain and build an attack surface map, then use Nuclei to batch scan for vulnerabilities. This pair of ProjectDiscovery tools is an industry standard.
• Spider-Flow — Common in small-to-medium enterprises: operations team builds it themselves, regularly scrapes product data from competitor e-commerce platforms, no need to ask developers to write scripts.

One Diagram: Match Your Need

What do you need?
├── Feed data to AI/LLM
│   ├── Don't want to build your own service → Firecrawl
│   ├── Run locally, data self-controlled → Crawl4AI
│   └── Describe requirements in natural language → ScrapeGraphAI
│
├── Large-scale production scraping
│   ├── Python, long-term stable → Scrapy
│   ├── Want a visual interface → PySpider
│   ├── Go language, extreme speed → Colly
│   └── JS-heavy sites + anti-scraping → Crawlee
│
├── Target site has heavy JS rendering
│   ├── New project / multi-language → Playwright
│   ├── Node.js legacy project → Puppeteer
│   └── Java enterprise project → Selenium
│
├── Domestic social media data
│   ├── Xiaohongshu / Douyin / Bilibili etc. → MediaCrawler
│   └── Export your own data → InfoSpider
│
└── Special scenarios
    ├── Target site changes often → Scrapling
    ├── Security research / penetration testing → Katana
    └── Operations staff who can't code → Spider-Flow

Common Combos

• AI Knowledge Base Full Pipeline — Crawl4AI (web) + MarkItDown (local files) → RAG vector database
• Domestic Content Research — MediaCrawler (platform data) + Crawl4AI (external link pages) → LLM analysis
• Production-grade + AI Output — Scrapy (large-scale collection) + Crawl4AI (clean and convert to Markdown) → feed to LLM
• Long-term monitoring — Scrapling (few key pages, adaptive anti-redesign) + Scrapy (batch periodic full crawl) in parallel
• Security scenarios — Katana (attack surface discovery) → Playwright (deep verification of target behaviour)

Tools aren’t good or bad – they’re only suitable or not for your scenario.

If you choose wrong, it’s not a lack of features, it’s a wrong direction.

Send this to a fellow developer who needs it. Save them from stepping into pitfalls. 🚀

#WebScraping #Python #AITools #OldYangShare