Our research agent was great on 10 sites and fell apart at 200

The demo was fine, obviously the demo was fine, I picked the ten sites. clean html, agent reads them, pulls the fields, writes rows, everyone's happy. then we pointed it at the real list which is about 200 domains and the failure rate went from basically nothing to somewhere north of 30 percent, and it took me the better part of two weeks to accept that almost none of that was the model's fault. 200 arbitrary websites is just a far more hostile place than 10 curated ones. some are react apps that hand an empty div to anything without a browser, a few sit behind a consent wall that only shows up for eu traffic so it never reproduced locally, two are wordpress installs from roughly 2014 that return a cheerful 200 with an error page in the body, and the agent has no way of telling the difference between a page that genuinely said nothing and a page it simply failed to read. so it writes null, with total confidence, and carries on. That last bit is the real problem and I don't think it's a scraping problem at all. there's no signal anywhere in my pipeline that separates empty from broken. I've been adding one, basically a confidence score on the fetch rather than on the extraction, and it has surfaced more bad rows in a fortnight than every prompt change I made before it.