Tag
This paper identifies a 'countdown subcircuit' in Llama-3.1-70B-Instruct that allows the model to track remaining tokens across various tasks, and demonstrates that this subcircuit is reused across different contexts and even models.