Tag
This paper mechanistically analyzes how positional encoding choices (RoPE vs. SWA NoPE hybrids) shape in-context retrieval across 22 open-weight models, showing that PE hybrids shift from positional to semantic retrieval mechanisms, explaining their long-context gains and trade-offs.
This paper systematically studies in-context retrieval at million-token scale, introducing BlockSearch, a 0.6B LM retriever, and analyzing attention dilution. The model matches or outperforms dense retrieval on benchmarks like MS MARCO and NQ, and significantly outperforms on tasks requiring different similarity notions, highlighting the potential of in-context retrieval while emphasizing attention control under extreme context growth.