SKILL-KD: Contrastive Skill Distillation for LLM Agents

Hugging Face Daily Papers Papers
skill-distillation llm-agents contrastive-learning prompt-engineering knowledge-distillation agent-training

Summary

SKILL-KD is a contrastive skill distillation framework that improves LLM agents by distilling actionable discrepancies between teacher and student trajectories into textual skill patches, with drift-aware consolidation to iteratively refine skills.

Skill-based prompting has become a practical mechanism for improving large language model (LLM) agents, yet existing skill acquisition methods often treat skills as experience summaries, memory entries, or direct summaries of successful demonstrations. This creates a mismatch for weaker student agents: when a student fails because it lacks task knowledge or operational strategy, its failed trajectory may not contain enough evidence to infer the missing behavior, while the teacher trajectory may be too implicit to be internalized as reusable guidance. We propose SKILL-KD, a contrastive skill distillation framework that treats skills as an explicit distillation medium between agents of different capabilities. Given a student failure and the teacher trajectory on the same task, SKILL-KD distills their actionable discrepancy into a textual skill patch, evaluates the patch by re-running the student, and iteratively refines the patch when the student still fails. To prevent repeated local updates from causing skill drift, SKILL-KD further maintains trace-linked edit histories and performs Drift-Aware Skill Consolidation, deciding whether each patch should add a new rule, delete or modify an existing rule, or be skipped. Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines.
Original Article
View Cached Full Text

Cached at: 08/06/26, 05:50 AM

Paper page - SKILL-KD: Contrastive Skill Distillation for LLM Agents

Source: https://huggingface.co/papers/2607.28048

Abstract

Skill-basedpromptinghasbecomeapracticalmechanismforimprovinglargelanguagemodel(LLM)agents,yetexistingskillacquisitionmethodsoftentreatskillsasexperiencesummaries,memoryentries,ordirectsummariesofsuccessfuldemonstrations.Thiscreatesamismatchforweakerstudentagents:whenastudentfailsbecauseitlackstaskknowledgeoroperationalstrategy,itsfailedtrajectorymaynotcontainenoughevidencetoinferthemissingbehavior,whiletheteachertrajectorymaybetooimplicittobeinternalizedasreusableguidance.WeproposeSKILL-KD,acontrastiveskilldistillationframeworkthattreatsskillsasanexplicitdistillationmediumbetweenagentsofdifferentcapabilities.Givenastudentfailureandtheteachertrajectoryonthesametask,SKILL-KDdistillstheiractionablediscrepancyintoatextualskillpatch,evaluatesthepatchbyre-runningthestudent,anditerativelyrefinesthepatchwhenthestudentstillfails.Topreventrepeatedlocalupdatesfromcausingskilldrift,SKILL-KDfurthermaintainstrace-linkededithistoriesandperformsDrift-AwareSkillConsolidation,decidingwhethereachpatchshouldaddanewrule,deleteormodifyanexistingrule,orbeskipped.Acrossfiveagentbenchmarksandtwostudentsettings,SKILL-KDconsistentlyimprovesfrozenstudentagentsoverfixed-modeladaptationbaselines.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.28048

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.28048 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.28048 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.28048 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles