What’s your actual agentic web research stack? (fully local, no cloud APIs)

Reddit r/LocalLLaMA Tools

Summary

The author details a fully local, no-cloud-API web research stack for AI agents, using self-hosted SearXNG, a persistent cache, TLS-fingerprinted fetching, headless browser fallback, and a local reranker, inviting community discussion on similar setups.

Been running a fully local web research pipeline for my AI agent setup for a while now and realized I haven't seen much discussion about how others are handling this part. The inference side gets all the attention, but getting an agent to actually browse the real web without everything falling apart is its own problem. My stack ended up as a layered pipeline: self-hosted SearXNG for search, a persistent cache/index layer (Hister) that stores every fetched page, rnet (now wreq) for TLS-fingerprinted HTTP fetches that get past basic anti-bot, camofox (wrapping Camoufox) as a headless browser fallback for JS-heavy pages, and a local qwen3-reranker-4b for relevance scoring. All talking to the agent through an MCP server. No cloud API calls anywhere in the chain. Reddit 403s on Firefox fingerprints from datacenter IPs but Safari passes. Cloudflare managed challenges need full browser rendering regardless of fingerprint. Pages change or disappear between sessions, so having the cached snapshot from when you actually read it matters more than you'd think. The cache layer quietly became the most valuable piece — repeat lookups are instant, and when a page goes down or gets edited, the agent still has what you originally saw. All of it runs on one box alongside the inference models. No external dependencies, no privacy calculus about browsing history hitting a cloud API. Wrote up the full architecture, pitfalls, and config details here: https://kmarble.dev/posts/completely-local-agentic-web-research/ Curious what others are running for this. Is anyone doing fully local web access for their agents, or are most people just pointing at a search API and accepting the tradeoffs? The fingerprinting and anti-bot layer especially feels like something everyone has to solve independently.
Original Article

Similar Articles