Tag
This paper reveals that frontier LLMs struggle with exact copying tasks due to limitations in positional encodings, and proposes 2D-RoPE, a method that organizes text into a 2D grid to enable perfect copying, showing advantages in synthetic and large-scale pretraining.
This paper studies functional redundancy in recurrent neural networks by using ordered real Schur coordinates to identify structured ablations that preserve task performance, finding that task-restricted symmetries vary across tasks and trained solutions.