AnakinScraper OSS, the open-source web scraping API purpose-built for AI. Instantly convert any website into clean Markdown or structured JSON, ready to feed directly into RAG, AI Agents, and more. Key highlights: anti-detect browser (Camoufox Firefox), intelligent proxy auto-selection (Thompson Sampling), automatic handler chain degradation (HTTP → Browser → External API), one-command startup: make up. Fully self-hosted with zero cloud dependencies. Supports sync/async/batch modes and includes a built-in web dashboard. Lighter and smarter than Firecrawl or Crawlee, optimized specifically for AI scenarios! https://github.com/Anakin-Inc/anakin…

Anakin-Inc/anakin

Source: https://github.com/Anakin-Inc/anakin

AnakinScraper OSS

CI (https://github.com/Anakin-Inc/anakinscraper-oss/actions/workflows/ci.yml) License: AGPL-3.0 Go (https://go.dev) Python (https://python.org) Docker React

The open-source web scraping API for AI. Turn any website into LLM-ready markdown or structured data. Self-host with a single command. No cloud dependencies. Powers RAG pipelines, AI agents, and data extraction at scale.

git clone https://github.com/Anakin-Inc/anakinscraper-oss.git && cd anakinscraper-oss && make up
# Scrape any website — one curl, full result:
curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

Why AnakinScraper?

Features

• Handler chain with fallback — HTTP fetch → anti-detect browser → external API. Each handler tries in order; if one fails, the next picks up automatically. Most pages resolve on the free local HTTP handler — paid APIs are only called for the ~5% that actually need them. Docs →
• Custom API handlers — plug in any third-party scraping service as a chain fallback. Only invoked when local handlers fail — saves 90%+ on API costs vs routing everything through a paid service. Built-in anakin.io (https://anakin.io) handler included. How to add your own →
• Domain configs — per-domain scraping strategies: choose which handlers to use, set timeouts, retries, custom headers, block domains, and validate content with pattern matching. Docs →
• Failure detection — define failure patterns and required patterns per domain. If the scraped content matches a failure pattern (e.g. CAPTCHA page) or misses a required pattern, the job auto-retries with the next handler. Docs →
• Anti-detect browser — Camoufox (https://github.com/daijro/camoufox) (anti-detect Firefox) with realistic fingerprints, not headless Chrome. Docs →
• Proxy auto-select — Thompson Sampling (https://en.wikipedia.org/wiki/Thompson_sampling) picks the best proxy per domain, learning from success/failure in real time. Docs →
• Structured JSON extraction — use Gemini AI to extract structured data from any page (bring your own API key)
• Sync + async + batch API — POST /v1/scrape for instant results, /v1/url-scraper for async with polling, batch up to 10 URLs
• LLM-ready markdown — automatic boilerplate removal, clean content extraction. Feed directly into RAG pipelines, Claude, GPT, or any LLM without preprocessing
• Web dashboard — built-in React UI for scraping, job tracking, domain config management, and proxy monitoring
• Zero-config mode — run with just Go, no database needed. Or use Docker for the full stack
• Self-contained — no Redis, no AWS, no message queues. Single Go binary. Optional PostgreSQL for persistence

Quick Start (no Docker, no database)

Just Go 1.25+. Two commands:

cd server && go run cmd/server/main.go
# In another terminal:
curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

Jobs are stored in memory (lost on restart). For persistence, set DATABASE_URL. For JavaScript-heavy sites, add the browser service via Docker.

Self-Host (Docker — full stack)

Prerequisites

• Docker (https://docs.docker.com/get-docker/) and Docker Compose

Start

git clone https://github.com/Anakin-Inc/anakinscraper-oss.git
cd anakinscraper-oss
make up

That’s it. Three containers start:

Web Dashboard

A built-in web UI is included for visual scraping, job tracking, and configuration:

cd webapp && npm install && npm run dev

Open http://localhost:3000 — the dashboard proxies API calls to the server on port 8080. Pages: Dashboard (health + quick scrape) | Scrape (sync/async/batch with live results) | Jobs (tracked history with status filters) | Domain Configs (CRUD with handler chain management) | Proxy Scores (Thompson Sampling performance)

Scrape a URL

Synchronous (recommended for getting started):

curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .

One request, full result back. No polling. Timeout: 30 seconds by default (configurable via the timeout request field, max 120 seconds).

Asynchronous (for long-running scrapes):

# Submit
curl -s -X POST http://localhost:8080/v1/url-scraper \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
# Poll for result
curl -s http://localhost:8080/v1/url-scraper/JOB_UUID | jq .

With AI-powered JSON extraction (requires GEMINI_API_KEY):

curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "generateJson": true}' | jq .generatedJson

No API keys required for the scraper itself. Just JSON in, results out.

Architecture

┌─────────────────┐
│    Your App     │
│   (cURL / CLI)  │
└────────┬────────┘
         │ HTTP
         ▼
┌─────────────────┐       ┌──────────┐
│     Server      │──────▶│  Gemini  │
│  (Go/Fiber)     │optional│  (JSON)  │
│   Port 8080     │       └──────────┘
└──┬──────┬───┬──┘
   │      │   │
   ┌──────┘   └────────────┐
   │                      ▼
   ▼                      ▼
┌──────────┐       ┌──────────────┐       ┌──────────────┐
│ Storage  │       │   Browser    │       │  API Handler │
│ Postgres │       │   Service    │       │ (anakin.io   │
│ or memory│       │  (Camoufox)  │       │  or custom)  │
│(optional)│       │ (optional)   │       │ (optional)   │
└──────────┘       └──────────────┘       └──────────────┘

The server is a single Go binary that runs with zero dependencies. Optionally add PostgreSQL for persistence, the browser service for JavaScript-heavy sites, and API handlers for hard-to-scrape sites. Workers execute the handler chain (HTTP → browser → API fallback), convert HTML to markdown, and optionally extract structured JSON via Gemini.

API Reference

See docs/API.md for the complete API reference. Quick overview:

Request Fields

Response

{
  "id": "550e8400-...",
  "status": "completed",
  "url": "https://example.com",
  "html": "...",
  "cleanedHtml": "...",
  "markdown": "# Page Title\n\nContent...",
  "generatedJson": {
    "status": "success",
    "data": {"title": "Page Title", "content": "..."}
  },
  "durationMs": 1234
}

generatedJson is only present when generateJson: true and GEMINI_API_KEY is configured.

Handler Chain

Each scrape job goes through the handler chain. On failure, it falls back to the next handler:

HTTP Handler (fast, ~200ms) ──fail──▶ Browser Handler (Camoufox) ──fail──▶ API Handler (optional)

HTTP Handler — direct HTTP GET with a browser user-agent. Handles static HTML, server-rendered pages. No browser overhead. Browser Handler — connects to Camoufox (https://github.com/daijro/camoufox) (anti-detect Firefox) over WebSocket via Playwright protocol. Full JavaScript rendering, network-idle detection, realistic browser fingerprints. Handles SPAs, lazy-loaded content, and sites with anti-bot protection. API Handler (optional) — delegates to an external scraping API when local handlers fail. Set ANAKIN_API_KEY to enable the built-in anakin.io (https://anakin.io) fallback for hard-to-scrape sites (Cloudflare, DataDome, etc.). See Adding Custom API Handlers below.

Adding Custom API Handlers

The API handler pattern makes it easy to integrate any third-party scraping service as a chain fallback. The built-in anakin.io handler (server/internal/handler/api.go) is a working example — copy and modify it for your provider:

1. Copy api.go to my_provider.go
2. Add a constructor like NewAnakinHandler — set your provider’s URL, auth header name, and response format
3. Register in main.go:

if cfg.MyProviderAPIKey != "" {
    handlers = append(handlers, handler.NewAPIHandler(handler.APIHandlerConfig{
        Name:       "my-provider",
        APIURL:     "https://api.my-provider.com/scrape",
        APIKey:     cfg.MyProviderAPIKey,
        AuthHeader: "Authorization", // or "X-API-Key", "Bearer", etc.
    }))
}

4. Add env var to config.go: MyProviderAPIKey: os.Getenv("MY_PROVIDER_API_KEY")

API keys always come from environment variables — never hardcoded. The handler only activates when its key is set.

Extending the Chain

Implement the ScrapingHandler interface:

type ScrapingHandler interface {
    Name() string
    CanHandle(ctx context.Context, req *HandlerRequest) bool
    Scrape(ctx context.Context, req *HandlerRequest) (*ScrapeResult, error)
    IsHealthy() bool
}

See examples/custom-handler/.

Configuration

All configuration via environment variables:

Project Structure

anakinscraper-oss/
├── server/                  # Go server (API + workers)
│   ├── cmd/server/          # Entry point
│   └── internal/
│       ├── config/          # Environment config
│       ├── models/          # Data types
│       ├── worker/          # Channel-based worker pool
│       ├── handler/         # Scraping handlers (HTTP, Browser)
│       ├── converter/       # HTML → Markdown
│       ├── gemini/          # Gemini AI JSON extraction
│       ├── domain/          # Domain configs + failure detection
│       ├── store/           # Job storage (PostgreSQL or in-memory)
│       ├── proxy/           # Proxy pool + Thompson Sampling
│       ├── telemetry/       # Anonymous usage telemetry
│       ├── processor/       # Job processing
│       └── http/
│           ├── handlers/    # API request handlers
│           └── router/      # Route registration
├── browser-service/         # Camoufox anti-detect browser server
├── webapp/                  # React web dashboard (Vite + Tailwind)
├── openclaw-skill/          # OpenClaw skill wrapper
├── examples/                # Usage examples
├── docker-compose.yml       # Full stack (3 containers)
├── scripts/init-db.sql      # Database schema
└── .env.example             # Config template

Development

Running Locally (without Docker)

Minimal (Go only):

cd server && go run cmd/server/main.go

No database, no browser service. HTTP handler scrapes static sites. Jobs stored in memory.

Full local stack (Go + Python + PostgreSQL):

# Terminal 1: PostgreSQL
docker compose up postgres -d

# Terminal 2: Browser Service (for JS-heavy sites)
cd browser-service && pip install -r requirements.txt && python server.py

# Terminal 3: Server with persistence
cd server && DATABASE_URL="postgres://postgres:postgres@localhost:5432/anakinscraper?sslmode=disable" go run cmd/server/main.go

Running Tests

cd server && go test ./...

Building

cd server && go build -o server ./cmd/server

CLI

Use the Anakin CLI (https://github.com/Anakin-Inc/anakin-cli) to scrape from your terminal. It works against both self-hosted and the hosted API (https://anakin.io):

# Install
pip install anakin-cli

# Scrape via your local instance (no API key needed)
anakin scrape "https://example.com" --api-url http://localhost:8080

# Or set it as your default
export ANAKIN_API_URL="http://localhost:8080"
anakin scrape "https://example.com"

# Extract structured JSON
anakin scrape "https://example.com" --format json --api-url http://localhost:8080

# Batch scrape
anakin scrape-batch "https://example.com" "https://httpbin.org/html" --api-url http://localhost:8080

The same CLI also supports AI web search and deep research via the hosted API — get a free API key (https://anakin.io/dashboard) to unlock those features. See the anakin-cli repo (https://github.com/Anakin-Inc/anakin-cli) for full usage.

Integrations

OpenClaw Skill

Use AnakinScraper as an OpenClaw (https://openclaw.ai) skill:

cp -r openclaw-skill ~/.openclaw/workspace/skills/anakinscraper

See openclaw-skill/SKILL.md.

Self-Hosted vs Hosted

This repo gives you the full scraping engine. anakin.io (https://anakin.io) adds the infrastructure you’d otherwise build yourself: