Visual Contrastive Self-Distillation
Summary
VCSD removes the need for external teachers, privileged answers, or visual evidence in on-policy self-distillation by using content-erased control images to produce contrastive signals, consistently outperforming existing methods on vision-language benchmarks.
View Cached Full Text
Cached at: 07/24/26, 05:06 AM
Paper page - Visual Contrastive Self-Distillation
Source: https://huggingface.co/papers/2607.21556
Abstract
On-policyself-distillation(OPSD)ispromisingasitremovestheexternalteacherrequiredbyon-policydistillation(OPD),yetitstillneedsasymmetricinformationbetweenteacherandstudenttoensurethattheself-teacherprovidesastrongerlearningsignalthanthestudent.Existingmethodscreatethisasymmetryeitherthroughprivilegedanswersorvisualevidence.Weaskwhetherbothcanberemoved,yieldingasimplerformofOPSDdrivenpurelybyinputconditioning.Forthispurpose,weproposeVisualContrastiveSelf-Distillation,namelyVCSD,whichconvertsimage-contentremovalintoanon-policyself-distillationsignal.Ateachstudent-generatedresponseprefix,theEMAteacherproducestwonext-tokendistributionsunderthesamepromptandprefix--oneconditionedontheoriginalimageandtheotheronacontent-erasedcontrol.Theirtoken-wiselog-probabilitydifferencehighlightscandidateswhoselikelihoodisspecificallyincreasedbytheinstance-levelvisualcontent.Weusethiscontrasttosharpentheteacher’soriginal-imagedistributionwithinitsplausiblesupport,anddistilltheresultingfull-distributiontargetintothestudent.UsingViRL39Kdataset,VCSDconsistentlyoutperformsmatchedOPSDacrossQwen3-VLandQwen3.5models.Forexample,onQwen3-VL,itimprovestheseven-benchmarkaggregatefrom62.27%rightarrow67.04%at2B,71.30%rightarrow73.16%at4B,and72.51%rightarrow76.26%at8B.Furthermore,VCSDrequiresnoexternalteacher,privilegedanswers,visualevidencesignals,reasoningtraces,oradditionalinference-timecost.
View arXiv pageView PDFProject pageGitHubAdd to collection
Get this paper in your agent:
hf papers read 2607\.21556
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.21556 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.21556 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.21556 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation
Proposes NOPD, a self-distillation method that improves vision-language models without external supervision by leveraging prediction discrepancies between clean and corrupted inputs. Achieves significant gains on visual reasoning tasks, matching or exceeding RL and distillation from external models.
@sheriyuo: Qwen Tongyi Lab proposes RLCSD, a simple but important critique of on-policy self-distillation. Their key observation i…
Qwen Tongyi Lab proposes RLCSD to address the style drift problem in on-policy self-distillation, where the learning signal focuses on style tokens rather than task-critical reasoning tokens. Their method uses contrastive supervision to focus on task-relevant tokens, achieving consistent improvements over prior methods on reasoning benchmarks.
Self-Supervised Visual On-Policy Distillation
The paper introduces S^2VOPD, a self-supervised method that improves vision-language models by distilling from original images into strongly augmented student views, achieving performance surpassing GPT-5.4 on benchmarks.
OPD-V: Visual On-Policy Self-Distillation with Modality Balance
Introduces OPD-V, a visual on-policy self-distillation paradigm for multimodal large language models that leverages positive and negative teachers to exploit modality balance as privileged information, improving reasoning performance across benchmarks while reducing training cost.
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
SCOPD introduces a sparse-context on-policy self-distillation framework that enhances the efficiency of vision-language models by improving the use of pruned visual tokens, achieving higher performance retention without additional inference cost.