Tag
This paper investigates whether structured synthetic reasoning data can improve arithmetic reasoning in small language models under consumer-hardware constraints. Fine-tuning Qwen3-0.6B and Qwen3-1.7B with LoRA on a synthetic corpus derived from GSM8K improved exact-match accuracy by 12-13 percentage points, with strong transfer to related benchmarks.
This paper investigates whether arithmetic heuristic neurons in LLMs are form-invariant across symbolic arithmetic, natural language word problems, and Python code. Using activation patching, they find a shared circuit of neurons that is necessary and sufficient for arithmetic computation, and that cross-format failures arise from activation states rather than distinct circuits.
This paper identifies a 'positional copying' shortcut where small language models answer arithmetic questions by copying the last number before the answer delimiter, bypassing actual reasoning. This effect explains why shuffling CoT steps retains performance; it accounts for 89-92% of teacher-forcing accuracy in 1-3B models on GSM8K.