Tag
This paper studies quantization-aware distillation for NVFP4 low-precision LLMs and finds that output-matching with KL loss alone can mask internal representational drift. The authors propose CKA-QAD, which preserves internal geometry via CKA-guided alignment, improving reasoning and coding accuracy in compact models.
Trace2Policy extracts human-readable decision rules from expert behavior traces and iteratively refines them via error-driven skill refinement, outperforming pure LLM baselines on compliance-sensitive tasks in logistics.
This paper reexamines the role of temperature in large language model distillation, revealing that temperature asymmetrically benefits forward KL divergence over reverse KL, allowing simple KL methods to match state-of-the-art distillation approaches at higher temperatures.
This paper introduces bounded behavioral indistinguishability, a formal framework for evaluating black-box LLM distillation beyond semantic similarity. Experiments on Qwen and Llama models show that distillation reduces but does not eliminate adversarial distinguishability, highlighting the need for category-aware evaluation.