This study introduces section-aware compression for reasoning traces, training models to drop filler narration while preserving compute and verification spans, matching or exceeding original accuracy while using 2-3 times fewer tokens.
TLDR: I trained Qwen3.5-4B and gemma-4-12b on self-distilled, compressed reasoning traces; compression was section-aware (compute and verification spans remain, fillers/narration/transitions get dropped/compressed). The models match or beat the originals, often by a large margin, while using 2-3 times fewer tokens. Full study, models and code are available. section-aware compression, visualized Hi! For the past few months I have been exploring reasoning compression: I built datasets, trained (many) LoRAs, benchmarked them, and I finally wrote up what I found. The main results from my study: Flat compression causes greedy decoding to fail. Compressing the whole trace into a compressed style and training on that caused the model to loop on 93% of GSM8K problems at temperature 0 (accuracy .03), often right after reaching the correct answer. The same checkpoint scores .90 at temperature 1.0 on a subset mined from those loop failures, so the model hasn’t forgotten anything; it’s just failing to terminate. Section-aware compression works, and beats uncompressed SFT. If you keep the spans where the model computes and verifies (in its own words) and only compress/delete the narration between them, accuracy goes up: +.15 on GSM8K vs an uncompressed SFT control, at ~1.7x fewer reasoning tokens. The model interprets the computation spans not only as “working memory”, but also as an anchor for termination. What the system prompt "means" depends on training. To the original model, "please reason step by step..." means think harder; to a compression-trained model it acts as an efficiency trigger (.82 @ ~1.5k tokens with it, ~.63 @ ~3.4k without). Training with the prompt also creates a dependency on it (visible on MATH-500), so the recipe that worked best is: train bare (without a system prompt), serve with the prompt. You can make compression a switch. Training with an identity prompt that describes the compressed style binds the behavior to it: prompt on = concise reasoning (.80 @ ~1.9k tokens), prompt off = the model decompresses back to ~4.1k tokens at normal accuracy. A stronger teacher made things worse. Having a bigger model segment/convert the traces produced cleaner data (more lint passes) but a worse model than uncompressed SFT. What matters is that the traces stay close to the model's own distribution. The "code tax" follows the data and NOT the domain. Qwen's code traces are nearly all computation, so section-aware compression barely touched them, and compression training lengthened code thinking and cost ~.13 on HumanEval/MBPP. On Gemma the same recipe compressed code the hardest and HumanEval nearly doubled (.31 -> .57). Models reproduce the per-domain trace lengths they were trained on. It transfers. Full pipeline re-run on gemma-4-12b-it (3x the params, different family/tokenizer): .86 GSM8K @ 1,679 tokens vs .57 @ 3,753 for the original, and the compressed arm wins at every sampling temperature, not just greedy. All of this is small-scale on purpose (322-648 training rows per arm, ~1.5 3090-hours per arm), which is what made a ~15-arm ablation grid affordable, but it also means these numbers are not necessarily optimal. A ~1.5k-row arm is queued and a 25-30k-row build is planned; if you have spare compute and want to see this run at scale, get in touch! I am open to suggestions and discussions, so feel free to comment here or to contact me! You can find everything here: Full write-up 🤗 Models + datasets (main collection) 🤗 Ablation arms Code (pipeline + experiment journal)
This paper proposes compressing reasoning traces before knowledge distillation to reduce computational costs and inference lengths, showing an accuracy-efficiency trade-off where compressed traces retain up to 96% of raw-trace accuracy with up to 18x higher per-token efficiency.
A systematic study of compressing recursive reasoning models for edge hardware finds that aggressive quantization destroys global reasoning while preserving local prediction. The paper introduces per-channel calibrated INT4 to recover reasoning ability and provides deployment recipes fitting 8 MB SoC and 4 MB MCU targets.
This paper introduces Implicit Compression Regularization (ICR), a method to address LLM overthinking during RL post-training by guiding models toward concise yet accurate reasoning trajectories.
This paper proposes D-RPC, a method for distilling reasoning from large language models to smaller ones by compressing reasoning paths into a reusable bank, achieving better performance and consistency on math and commonsense benchmarks.
This paper introduces PUMA, a plug-and-play framework that detects semantic redundancy in chain-of-thought reasoning to enable early exit, achieving 26.2% average token reduction across multiple models and benchmarks while preserving accuracy and reasoning quality.