Tag
OpenClaw with Cockroach Crawler transforms an AI agent into a powerful web research machine that can crawl JavaScript-heavy pages, extract structured data, generate PDFs, and take screenshots without separate API keys for many public sources.
Unclecode created Crawl4AI, a free and open-source web crawler that converts web pages to clean Markdown for AI models, after being frustrated with paid alternatives; it has gained 60,000 GitHub stars and over a million monthly downloads.
Crawl4AI is an LLM-optimized open-source crawler that converts web content into clean, structured Markdown. It supports intelligent content filtering, LLM-driven extraction, browser automation, and is ideal for RAG and AI Agent scenarios.
Agentmap is an open-source tool that crawls any website and converts it into a structured map of pages, flows, and data, enabling coding agents (like Claude Code, Cursor) to skip blind exploration and directly perform tasks, saving tokens.
Introduces ScrapeGraphAI, an LLM-based scraping tool that can take natural language descriptions of requirements and automatically generate the scraping workflow, eliminating the need to write selectors or care about HTML structure, supporting local models and multiple integration platforms.
Describes building a web crawler that extracts content from hotel websites, uses an AI agent to generate structured FAQs, and stores them in a vector database for automatic knowledge base creation.
Amazonbot, Amazon's web crawling bot, now respects robots.txt directives, marking a change in its previous behavior.