I taught my 1B to follow instructions. It got worse at following instructions...
Summary
The author trained 1B, 2B, and 3B models with the same SFT recipe and observed that instruction-following (IFEval) regressed for the 1B and 2B models but improved for the 3B, possibly due to different learning rates or model capacity.
Similar Articles
Finetuned a Early 2023-Era Model on 2 Instruction Following Datasets and it Became Good
A finetuned Pythia-6.9B model on two instruction-following datasets for 550 steps becomes capable in 13 languages, showing significant improvement over the base model.
Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models
This paper studies transferring lessons about supervised fine-tuning (SFT) across alignment training, model organisms, and toy models, showing that techniques like training on reasons for behavior and mixing on-model data can improve generalization and capability preservation.
SEIF: Self-Evolving Reinforcement Learning for Instruction Following
This paper introduces SEIF, a self-evolving reinforcement learning framework that enhances LLM instruction-following capabilities through iterative difficulty adaptation and co-training of instructor and follower components.
@f14bertolotti: Stellar performance from a 3B model. These results were achieved primarily through post-training refinements on Qwen2.5…
This technical report introduces VibeThinker-3B, a 3B parameter model that achieves frontier-level verifiable reasoning performance through post-training refinements on Qwen2.5-Coder, including curriculum-based supervised fine-tuning, multi-domain reinforcement learning, and offline self-distillation, matching or exceeding much larger models like DeepSeek V3.2.
Improving instruction hierarchy in frontier LLMs
OpenAI presents a training approach using instruction-hierarchy tasks to improve LLM safety and reliability by teaching models to properly prioritize instructions based on trust levels (system > developer > user > tool). The method addresses prompt-injection attacks and safety steerability through reinforcement learning with a new dataset called IH-Challenge.