Tag
The author explains how he secures his VPS and blog against bots and scrapers using fake WordPress logins, endlessh tarpit, blocking AI crawlers, fake phpinfo, infinite fake websites, and prompt injection to poison LLMs.
The author details a fully local, no-cloud-API web research stack for AI agents, using self-hosted SearXNG, a persistent cache, TLS-fingerprinted fetching, headless browser fallback, and a local reranker, inviting community discussion on similar setups.
The article discusses a website using Anubis, a proof-of-work system, to protect against mass scraping by AI companies, highlighting the ongoing battle between content providers and AI data collection.
The BrowserAct team open-sourced a browser automation command-line tool designed specifically for AI agents, providing three layers of anti-blocking mechanisms (fingerprint spoofing, captcha cracking, human takeover), supports multi-browser parallelism and account isolation, and optimizes output format to save tokens.
The author shares their experience of switching from headless browsers to replaying direct requests to scrape websites, reducing block rates and resource usage significantly.
The author explains that their blog is blocking requests from old or suspicious browser user agents to mitigate a surge in high-volume crawlers, likely for LLM training data. Specific instructions are provided for users of Vivaldi and Inoreader to adjust settings or report issues.
The author releases StealthFox, an open-source Firefox fork designed to bypass anti-bot systems by generating unique, consistent browser fingerprints at the C++ level for AI web agents.
GStack v1.28 adds download capabilities and headed configuration mode with anti-bot detection via Xvfb on headless Linux containers, plus llms.txt for agent integration.