backpropagation-through-time

A scaling study over ~600 training runs finds that full BPTT on looped transformers outperforms truncated alternatives at matched compute, with two fully trained loops emerging as the compute-optimal setting that beats non-looped models with ~27% fewer parameters.