Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
Summary
Prompt-Level Distillation (PLD) extracts reasoning patterns from teacher models into structured instructions for student model system prompts, improving performance on reasoning tasks without fine-tuning overhead.
View Cached Full Text
Cached at: 06/16/26, 11:31 AM
Paper page - Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
Source: https://huggingface.co/papers/2602.21103
Abstract
Prompt-Level Distillation extracts reasoning patterns from teacher models to enhance student model performance while maintaining interpretability and reducing latency.
Advanced reasoning typically requiresChain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. The standard alternative,fine-tuningsmaller models, often sacrifices interpretability while introducing significant resource and operational overhead. To address these limitations, we introducePrompt-Level Distillation(PLD). We extract explicit reasoning patterns from aTeacher modeland organize them into a structured list of expressive instructions for theStudent model’sSystem Prompt. Evaluated using Gemma-3 4B, PLD improvedMacro F1 scoreson StereoSet (57\% to 90.0\%) and Contract-NLI (67\% to 83\%), while increasingLogiQAaccuracy to 70\%. Similar results on Mistral Small 3.1 demonstratecross-architecture generalizability, enabling these compact models to match frontier performance with negligible latency overhead. These expressive instructions render the decision-making process transparent, allowing for full human verification of logic, making this approach ideal for regulated industries such as law, finance, and content moderation, as well as high-volume use cases and edge devices.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2602\.21103
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2602.21103 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2602.21103 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2602.21103 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Reasoning Compression with Mixed-Policy Distillation
This paper proposes Mixed-Policy Distillation (MPD), a framework that transfers concise reasoning behaviors from large teacher models to smaller student models, reducing token usage by up to 27.1% while improving performance.
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
This paper proposes a novel Chain-of-Thought distillation framework that transfers teacher models' stepwise attention on key information to student models through a Mixture-of-Layers module for dynamic layer alignment. The method achieves consistent performance improvements on mathematical and commonsense reasoning benchmarks by explicitly guiding student models to progressively focus on critical information during reasoning.
One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
PromptSD is a novel on-policy self-distillation method where the teacher differs from the student only by a learnable soft prompt, enabling multi-task knowledge absorption without weight drift. It matches or exceeds full fine-tuning on four tasks while preserving general capabilities.
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
Introduces SPOT, a method for on-policy distillation that uses sparse probing and outcome calibration to improve reasoning performance in smaller student models while balancing solution quality and coverage.
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
Adaptive Teacher Exposure for Self-Distillation (ATESD) improves LLM reasoning by dynamically adjusting how much of the reference reasoning the teacher shows the student during training, using a learnable policy controller and a discounted learning-progress reward. Experiments on math benchmarks show consistent improvements over existing self-distillation and RL baselines.