Tag
An educational blog post explaining how knowledge distillation works, covering the teacher-student framework, soft labels, temperature, and distillation loss, with real examples.
This paper investigates the calibration gap that arises when temperature scaling, a common post-hoc calibration method relying on one-hot labels, is applied to models trained with soft label distributions reflecting genuine human disagreement. Experiments across vision and language domains show that temperature scaling calibrated on hard labels consistently underperforms direct soft-label calibration, with larger gaps in language tasks, highlighting risks for safety-critical deployments.
Proposes a labeled-data-free meta-learning method that generates tasks by assigning soft labels from pre-trained models to unlabeled data, avoiding computationally expensive model inversion. Achieves up to 104x speedup and 8.4-36.4% accuracy improvements over state-of-the-art DFML methods.
This paper finds that 42.6% of annotator disagreement in HateXplain concentrates at the hate/offensive boundary, demonstrating that majority vote silences minority values and leads to models being wrong but highly confident on contested inputs.