stale html and headless browsers kept getting me blocked, so i started replaying the actual requests instead

spent a few months trying to scrape sites for an agent that needed live pricing and docs, and the headless browser route just kept eating me alive. playwright fleet on residential proxies, the whole thing. worked great in dev, then production hit and i was burning IPs in maybe 400 pages, plus one of the target sites pushed a redesign and half my selectors died overnight. felt like babysitting a daycare of chrome instances that all wanted to cry at once. what finally fixed it for me was just opening devtools, watching the network tab, and realizing 80% of the pages i cared about were hydrating from a json endpoint anyway. so instead of rendering, i started replaying the underlying request directly. set the right headers, the right cookie, the right accept-language, and the response comes back clean json. no dom, no selectors to break, no chrome. one site i was pulling went from \~6s per page in a browser to \~180ms as a plain request, and the block rate basically dropped to zero because i looked like the site's own frontend calling its own api. the catch is it's not magic. some stuff i ran into: - sites with signed request params or short-lived tokens need you to grab the token from a cheap warmup request first, then replay - a few endpoints check the referer and origin headers in ways the browser sets silently, so you have to mirror them exactly - anti-bot stacks like the heavier akamai/cloudflare setups still catch you on tls fingerprint, not just headers, so you need a client that doesn't scream "python requests" at the handshake - when the site is genuinely client-side rendered with no backing api (rare but happens), you're back to a browser whether you like it or not the mental shift that helped me most was stopping thinking of the site as "pages to render" and starting to think of it as "an api with a website glued on top." once you see the actual requests, scraping stops being an arms race and starts being boring, which is what you want. anyone else gone full request-replay for their agent's data layer? curious how people are handling the token refresh and tls fingerprint side at scale, because that's where i still feel like i'm duct-taping things.