Dataset Distillation by Influence Matching

Hugging Face Daily Papers Papers
dataset-distillation influence-matching synthetic-data classification vision-language benchmarks

Summary

This paper introduces Influence Matching (Inf-Match), a dataset distillation method that aligns the final training outcome by learning a compact synthetic set whose effect on converged parameters matches that of the full dataset. It achieves state-of-the-art accuracy on classification benchmarks and outperforms strong baselines on vision-language distillation tasks.

We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5\%, a +4.7\% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5\%. The code will be released via https://github.com/hrtan/infmatch.
Original Article
View Cached Full Text

Cached at: 07/24/26, 05:02 PM

Paper page - Dataset Distillation by Influence Matching

Source: https://huggingface.co/papers/2607.16859

Abstract

Werevisitdatasetdistillationfromanoutcome-centricperspective.Ratherthanaligningprocesssurrogates(per-stepgradientsortrainingtrajectories),InfluenceMatching(Inf-Match)alignsthefinaloutcomeoftraining:itlearnsacompactsyntheticsetwhoseeffectontheconvergedparametersmatchesthatofthefulldataset.Concretely,weintroduceafullydifferentiable,sample-levelinfluenceestimatorthatquantifiesparametershiftsfromaddingorremovingdata,withouttime-consuminginverse-Hessianproductsorconvexityassumptions.Theestimatorrunsinlineartimebyunrollingtheoptimizationdynamicsandapplyingafirst-orderTaylorapproximation.Wethenlearnthesyntheticsetbyminimizingthemismatchbetweenitsinfluenceandthatoftherealdataset,yieldingoutcomealignmentratherthanheuristicprocessimitation.Inf-Matchdeliversthebestaccuracyacrossstandardclassificationbenchmarks.Forinstance,onTiny-ImageNet(IPC=10),Inf-Matchattains31.5\%,a+4.7\%improvementoverNCFM.Beyondclassification,Inf-Matchscalestovision-languagedistillationonFlickr30K,outperformingstrongprocess-matchingbaselines.Forinstance,with200to1000syntheticsamples,ourmethodachievedaleadingimpressiveaverageonimage/textretrievaltasks,higherthanNCFMby2.5\%.Thecodewillbereleasedviahttps://github.com/hrtan/infmatch.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.16859

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.16859 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.16859 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.16859 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Data Unlearning via Inverse Distillation

arXiv cs.LG

The paper introduces Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a multi-step flow-matching or diffusion teacher into an efficient one-step student while suppressing generation of forgotten training data, requiring only the teacher and forget-set samples without access to retained data or auxiliary classifiers.