Tag
The paper curates and releases open datasets and a model for Armenian, demonstrating that continued pretraining with news and STEM data improves performance and addresses data scarcity in low-resource NLP.
This paper systematically studies perturbation-based continued pre-training (CPT) for improving zero-shot dialect robustness in multilingual LLMs, comparing six training conditions across German, Italian, and Arabic. It finds that character-noised CPT is the most effective general strategy and reveals that different perturbation methods induce distinct robustness mechanisms.
This paper introduces Hidden Decoding, a sequence-length scaling method for LLMs that adds internal computation per token by expanding each token into multiple streams with independent embeddings, using Stream-Factorized Attention to keep costs low. Experiments on models up to 617B parameters show consistent improvements over baselines, demonstrating a practical fixed-backbone scaling path.
Riazi-8B is an Urdu large language model fine-tuned for mathematical reasoning, achieving improved performance on MGSM-Urdu through continued pre-training and supervised fine-tuning on Urdu Chain-of-Thought data.