Tag
The paper proposes Parallel Pretraining (PPT), which concatenates documents with their translations in the same training sequence, and finds that a two-stage PPT-then-standard-pretraining approach consistently improves general language understanding and cross-lingual performance over standard pretraining at matched token budgets.