Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Summary
This paper proposes compressing reasoning traces before knowledge distillation to reduce computational costs and inference lengths, showing an accuracy-efficiency trade-off where compressed traces retain up to 96% of raw-trace accuracy with up to 18x higher per-token efficiency.
View Cached Full Text
Cached at: 06/08/26, 03:16 PM
Paper page - Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Source: https://huggingface.co/papers/2606.05988
Abstract
Post-hoc compression of reasoning traces reduces computational costs and inference lengths while maintaining high accuracy, offering an accuracy-efficiency trade-off in knowledge distillation.
Reasoning models produce longchain-of-thought tracesthat are costly to distill and encourage verbose student outputs. We studypost-hoc compressionof such traces beforeknowledge distillation. Two teachers, Qwen3.5-397B-A17B and gpt-oss-120B, generate about 283k correct traces each; twoinstruction-tuned modelsthen compress them to 8.6-21.0% of their original character length. Across a 48-run main grid plus seven Qwen-teacher truncation ablations, compressed traces reduce training tokens to 12-30% of raw, speed up training by 2.0-7.6x, and shorten inference outputs by 3-19x with smaller reductions under the shorter gpt-oss teacher. However, raw traces retain the highest downstream accuracy at every scale and for both teachers. A length-matched raw-trace truncation ablation shows that compression is not merely benefiting from a smaller token budget: model-compressed traces usually beat or match naive truncation, especially for smaller students, while maintaining shorter inference outputs. Overall, reasoning-trace compression offers an accuracy-efficiency trade-off rather than a free improvement: students retain up to 96% of raw-trace accuracy while gaining up to 18x higher per-token efficiency, and at the 0.8B scale underLoRAcompressed traces narrow the raw-vs-compressed gap but do not exceed raw.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.05988
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.05988 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.05988 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.05988 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
[Study/Models] Flint: Compressing Reasoning Without Breaking It
This study introduces section-aware compression for reasoning traces, training models to drop filler narration while preserving compute and verification spans, matching or exceeding original accuracy while using 2-3 times fewer tokens.
Reasoning Compression with Mixed-Policy Distillation
This paper proposes Mixed-Policy Distillation (MPD), a framework that transfers concise reasoning behaviors from large teacher models to smaller student models, reducing token usage by up to 27.1% while improving performance.
Structural Rationale Distillation via Reasoning Space Compression
This paper proposes D-RPC, a method for distilling reasoning from large language models to smaller ones by compressing reasoning paths into a reusable bank, achieving better performance and consistency on math and commonsense benchmarks.
Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation
This paper investigates a critical disconnect in trace-based knowledge distillation for LLMs, revealing that semantically correct Chain-of-Thought traces are not reliably correlated with correct final answers and that traces optimized for model performance are often least interpretable to end users.
Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training
This paper introduces Implicit Compression Regularization (ICR), a method to address LLM overthinking during RL post-training by guiding models toward concise yet accurate reasoning trajectories.