Tag
This paper reveals that frontier LLMs struggle with exact copying tasks due to limitations in positional encodings, and proposes 2D-RoPE, a method that organizes text into a 2D grid to enable perfect copying, showing advantages in synthetic and large-scale pretraining.