Exploring continual learning without replay buffers: Our findings using dynamic task-similarity routing [P]

Hi, I’ve been doing some work in the continual learning space and wanted to share an open-source framework we put together called Coincidex, along with some architectural insights and failure modes we found along the way. Most conventional approaches to sequential task learning rely heavily on replay buffers (which introduce severe memory/privacy overhead) or complex, hand-tuned task masks. We wanted to see if we could bypass both by relying entirely on a context-driven task similarity layer to handle data routing dynamically. The Approach: Instead of caching historical samples to prevent catastrophic forgetting, the framework drops in as a single layer swap. As sequential data streams in, it computes a task-similarity matrix on the fly, routing the data paths based on that context. Research Insights & Trade-offs: We spent a lot of time benchmarking this against baselines, and here is what actually happened in practice: Where it succeeds: The dynamic routing handles clean task boundaries surprisingly well. In small-scale continual vision setups, it achieves graceful transfer without the need for manual mask tuning or storing old data. Where it breaks (The Failure Modes): We aren't going to overpromise here—the similarity layer has distinct limits. On highly chaotic, long-tail task sequences with massive distribution shifts, the routing model struggles to maintain stability compared to a heavy replay-buffer baseline. Why we are sharing it: We built this as a lightweight alternative for setups where memory or privacy constraints make replay buffers impossible. We would love to get the community's eyes on the routing architecture, specifically on how we might tackle the failure modes in rougher task sequences, or thoughts on visualizing the similarity matrix at different checkpoints. You can check out the source code, architecture breakdown, and full benchmark suites here: https://github.com/rakib-nyc/coincidex