Are bigger context windows actually the wrong direction for agents?

I've been building a coding agent for a few months, and I keep coming back to the same weird thought. What if we're solving the wrong problem with agent memory? A lot of the effort seems to go into making context windows bigger. More history, more summaries, more replay, more stuff in the prompt. And to be clear, bigger context is useful. But every long-running agent I've worked on eventually starts dragging around junk too. Old debugging attempts, plans that were abandoned hours ago, assumptions that stopped being true, random conversation that no longer matters. At some point it feels less like memory and more like clutter. So lately I've been wondering if the better approach is almost the opposite. Keep the working context small, store memory somewhere else, and only pull in what the agent actually needs right now. Basically treat the model like it's stateless, because it is. Maybe I'm missing something obvious, but my gut says some long-session failures come from accumulated junk, not from lack of context. For people who run agents for hundreds or thousands of turns, where do you think this idea falls apart? What's the first thing that breaks?