Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Summary
Researchers propose trace rewriting methods to prevent unauthorized LLM knowledge distillation while preserving answer correctness and embedding detectable watermarks.
View Cached Full Text
Cached at: 04/21/26, 03:38 PM
Paper page - Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Source: https://huggingface.co/papers/2602.15143
Abstract
Techniques for modifying teacher-generated reasoning traces to prevent unauthorized knowledge distillation while maintaining answer correctness and enabling detectable watermarks are presented.
Knowledge distillationis a widely adopted technique for transferring capabilities from LLMs to smaller, more efficient student models. However, unauthorized use ofknowledge distillationtakes unfair advantage of the considerable effort and cost put into developing frontier models. We investigate methods for modifying teacher-generated reasoning traces to achieve two objectives that deter unauthorized distillation: (1)anti-distillation, or degrading the training usefulness of query responses, and (2)API watermarking, which embeds verifiable signatures in student models. We introduce several approaches for dynamically rewriting a teacher’s reasoning outputs while preserving answer correctness and semantic coherence. Two of these leverage the rewriting capabilities of LLMs, while others usegradient-based techniques. Our experiments show that a simpleinstruction-based rewritingapproach achieves a stronganti-distillationeffect while maintaining or even improving teacher performance. Furthermore, we show that our rewriting approach also enables embedding watermarks that can be reliably detected with essentially no false alarms. Our code is available at https://github.com/xhOwenMa/trace-rewriting.
View arXiv pageView PDFGitHubAdd to collection
Get this paper in your agent:
hf papers read 2602\.15143
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2602.15143 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2602.15143 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2602.15143 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
This paper proposes methods for protecting large language models against unauthorized knowledge distillation by rewriting reasoning traces to degrade training usefulness while preserving correctness, and embedding verifiable watermarks in distilled student models. The approach uses instruction-based and gradient-based rewriting techniques to achieve anti-distillation effects without compromising teacher model performance.
Chainwash: Multi-Step Rewriting Attacks on Diffusion Language Model Watermarks
This research paper introduces Chainwash, a multi-step rewriting attack that effectively removes statistical watermarks from diffusion language model (LLaDA-8B-Instruct) outputs, reducing detection rates from 87.9% to 4.86% after five chained rewrites.
Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation
This paper investigates a critical disconnect in trace-based knowledge distillation for LLMs, revealing that semantically correct Chain-of-Thought traces are not reliably correlated with correct final answers and that traces optimized for model performance are often least interpretable to end users.
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
Proposes Self-Verified Distillation, a method where LLMs generate and self-verify candidate solutions from unlabeled seed questions using prompt-based verification, then train on the filtered dataset, achieving significant gains on math, science, and coding benchmarks across Qwen3 models.
Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attacks
This paper introduces open-book benign rewriting (OBBR) as a proactive defense against backdoor attacks on LLMs, showing it neutralizes harmful content by projecting to benign prompts, and improves safety by 51% over state-of-the-art defenses.