Tag
A developer building a Polish legal domain model describes three stages of continued pretraining on Qwen3.5-9B-BASE and asks the community how to best use their remaining data (legal Q&A, study materials, harness conversations) to add chat ability, chain-of-thought legal reasoning, and function-calling for their in-house AI law harness — specifically where SFT versus RL fits in.
该研究构建了一个关于人类-生物圈-AI耦合福祉的继续预训练语料库，通过一次QLoRA CPT训练在五个开源模型上安装生态推理倾向，显著降低了模型在高压或对抗性场景下做出破坏生态系统决策的比例，且MMLU性能不受影响。
The paper shows that letters-only pre-tokenizer regexes in Llama-3 and Qwen3 impose an irreducible ~2.2x token premium on Nepali, and proposes an English-preserving regex repair that, with 32k new merges, reduces the premium to 1.01x, testing it via continued pretraining experiments.
Apertus 1.5 extends the 8B and 70B Apertus 1.0 models with native image/audio understanding, an optional thinking mode, a 4x longer 262K-token context window, and improved instruction-following and tool use, released as open weights under an Apache 2.0 license.
This paper studies how to specialize large language models for generating executable algorithmic trading code, showing that framework-oriented continued pretraining plus supervised fine-tuning on validated request-to-code pairs greatly improves success on the QuantCode-Bench benchmark and agentic strategy generation, while also revealing a capability-retention failure where domain specialization degrades structured tool calling.
The article describes a project with experiments on training a local LLM using continued pretraining (CPT) and RAG for domain-specific knowledge, featuring comprehensive evaluations and findings.
The paper curates and releases open datasets and a model for Armenian, demonstrating that continued pretraining with news and STEM data improves performance and addresses data scarcity in low-resource NLP.
This paper systematically studies perturbation-based continued pre-training (CPT) for improving zero-shot dialect robustness in multilingual LLMs, comparing six training conditions across German, Italian, and Arabic. It finds that character-noised CPT is the most effective general strategy and reveals that different perturbation methods induce distinct robustness mechanisms.
This paper introduces Hidden Decoding, a sequence-length scaling method for LLMs that adds internal computation per token by expanding each token into multiple streams with independent embeddings, using Stream-Factorized Attention to keep costs low. Experiments on models up to 617B parameters show consistent improvements over baselines, demonstrating a practical fixed-backbone scaling path.
Riazi-8B is an Urdu large language model fine-tuned for mathematical reasoning, achieving improved performance on MGSM-Urdu through continued pre-training and supervised fine-tuning on Urdu Chain-of-Thought data.