Building web agents made me realize how much context gets wasted on bad URLs. How do you filter your scrapes?

I’ve been fiddling with some agentic workflows and I have come to notice an issue with how agents handle web scraping. Normally, when you hand an agent a URL, it scrapes the page, and it drops the entire markdown payload into the context window. If the URL was just a login wall, a generic navigation page, or completely off-topic, you still burn the tokens to figure that out. I was checking out a tool that approaches this by returning page metadata (like the page_structure, category, and ranked snippets) alongside the text. The agent can evaluate the structure and category to decide if the page is actually useful before it processes the full payload. How do you handle this in your projects? Do you have a pre-processing step to catch login walls and nav pages, or are you just passing the raw markdown straight to the model?