ICLHerd：以 In-Context Learning 作为代理的 LLM 基于 Herding 的数据选择方法

# ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy
Source: [https://arxiv.org/html/2610.27142](https://arxiv.org/html/2610.27142)
1\]Dept\. Computing Science & Alberta Machine Intelligence Institute \(Amii\), University of Alberta 2\]Canada CIFAR AI Chair\\metadata\[Emails\]

###### Abstract

Data selection plays a central role in modern large language model training, as the quality of training data strongly influences downstream performance\. A common approach is to curate a development set that encodes the desired skills \(e\.g\., reasoning or multilinguality\) and select data that are similar to them\. Existing methods compute similarity using cosine distance over either the model’s hidden states or gradients\. However, we observe that this max\-similarity strategy induces a mode\-seeking bias: the selected data clusters tightly around the development samples\. To address this, we propose ICLHerd, a data selection framework that leverages the von Mises–Fisher kernel to balance mode\-seeking with mode\-averaging\. We further introduce in\-context learning as a validation proxy, enabling efficient hyperparameter search over the balance trade\-off\. Results show that ICLHerd outperforms existing LLM data selection algorithms, highlighting the importance of balancing mode\-seeking and mode\-averaging\.

## 1Introduction

Data selection is the task of choosing training samples from a data pool in order to maximize model performance given a budget of training data\. This is especially important in the large language model \(LLM\) era, where the candidate data pool is potentially the entire Internet\([Wenzek et al\., 2020](https://arxiv.org/html/2610.27142#bib.bib11);[Penedo et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib13);[Li et al\., 2024a](https://arxiv.org/html/2610.27142#bib.bib12)\)\. In modern LLM training, data selection is usually one of the most crucial steps, and researchers rely heavily on intuition and trial\-and\-error to curate an appropriate training set tailored for the desired model capability\. For example, DeepSeek\-V4 incorporated different filtering strategies to remove templated data \(e\.g\., repetitive html scripts\) and mitigate the risk of model collapse\([Xu et al\., 2026](https://arxiv.org/html/2610.27142#bib.bib2)\); Qwen 3 was trained across a selected set of 119 languages and dialects in an attempt to improve data diversity and improve the model’s multilingual skills\([Yang et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib3)\); Llama 4 was trained on what the technical report claimed as “specialized datasets” to improve the model’s long\-context capabilities\([Meta AI, 2025](https://arxiv.org/html/2610.27142#bib.bib4)\)\. It is important to note that the training recipes of these frontier open\-weight models are not publicly available\. This leaves a gap in systematic research on data selection, as the scale of training data \(often trillions of tokens\) makes it infeasible to ablate individual selection strategies, limiting the scientific rigor of these technical reports\.

Existing open research on LLM data selection generally uses a development \(dev\) set to guide the selection process, where the dev samples encode the desired capabilities, e\.g\., math and reasoning\. A promising direction is to embed all the samples and select training data based on the similarity to the dev set\.[Xia et al\. \(2024\)](https://arxiv.org/html/2610.27142#bib.bib1)propose LESS, which embeds training samples via down\-projected LoRA gradients, and at each iteration, selects the training sample that maximizes cosine similarity to any dev sample\.[Ivison et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib5)propose RDS\+, which embeds samples using the model’s own hidden states for efficiency, and employs a round\-robin selection that cycles through dev samples in fixed order, matching each dev sample in turn to promote data diversity\.

Despite the efforts, we observe that these algorithms suffer from themode\-seekingproblem, where the selected dataset is overly crowded around the dev samples\. Here, amoderefers to a peak in the embedding space, and both algorithms tend to select samples that are close to one or a few peaks of the dev distribution: LESS greedily pick the sample that match well against any of the dev samples, whereas RDS\+ attempt to match all dev samples in a round\-robin fashion\. Alternatively, we may also select data based on the mean embedding of the dev set, as opposed to the maximum used in LESS and RDS\+\. This, on the other hand, will lead to themode\-averagingproblem, where the embeddings of the selected dataset will center around the centroid of the dev distribution, losing the diversity of capabilities encoded by different dev samples\.

To this end, we propose ICLHerd, a data selection framework based on the principled data herding scheme\([Welling, 2009](https://arxiv.org/html/2610.27142#bib.bib32);[Chen et al\., 2010](https://arxiv.org/html/2610.27142#bib.bib33)\), where LESS and RDS\+ emerge as special cases\. The general idea is to make the distribution of selected samples close to that of the dev set\. We use the maximum mean discrepancy\([Gretton et al\., 2012](https://arxiv.org/html/2610.27142#bib.bib34), MMD,\)to measure the similarity between two distribution based on the selected samples and those in the dev set\. In particular, we adopt the von Mises–Fisher \(vMF\) kernel\([Hall et al\., 1987](https://arxiv.org/html/2610.27142#bib.bib6)\)on normalized features as the similarity measure, which operates on cosine similarities, since LLM representations are commonly compared by their direction rather than their magnitude\([Banerjee et al\., 2005](https://arxiv.org/html/2610.27142#bib.bib51);[Reimers and Gurevych, 2019](https://arxiv.org/html/2610.27142#bib.bib52);[Gao et al\., 2021](https://arxiv.org/html/2610.27142#bib.bib53)\)\. We also design a fused kernel that combines hidden\-state and gradient representations, allowing us to leverage their complementary signals for data selection\. Overall, the data selection reduces to the problem of looking for similar samples with a diversity penalty\. We apply the herding algorithm\([Welling, 2009](https://arxiv.org/html/2610.27142#bib.bib32);[Chen et al\., 2010](https://arxiv.org/html/2610.27142#bib.bib33)\)for optimization, which greedily selects the best sample at a time and adjusts the diversity penalty accordingly\.

To efficiently tune the balance between mode averaging and mode seeking, we further propose in\-context\-learning\-as\-a\-proxy for our ICLHerd framework\. Typically, hyperparameter tuning on data selection requires repeatedly training the model and validating the model’s performance, which can be prohibitively expensive\. Our proposed ICLHerd circumvents this problem by validating through in\-context learning, where we put the selected data as a prompt and evaluate its loss on the dev set\. Note that this is especially efficient due to prefix caching, as the same prompt is shared among dev samples\. Our results establish a connection between in\-context learning and gradient\-descent\-based learning performance\.

Overall, we propose a ICLHerd framework of data selection for training LLMs\. Our contributions are three\-fold: \(1\) we address the mode\-averaging and mode\-seeking problems in data selection for LLMs; \(2\) we design a fused vMF kernel combining hidden states and gradients to control this balance; and \(3\) we propose in\-context\-learning\-as\-a\-proxy for efficiently tuning it\.

## 2Related Work

Data selection for language model training aims to choose a subset of a large candidate pool to reduce training costs\([Albalak et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib8);[Qin et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib7)\)\. Existing methods span different paradigms\.

Classifier\-based data filtering\.Classifier\-based methods train a lightweight classifier to separate desired samples from the raw pool, and select samples by a threshold on the classifier score, with early studies in machine learning focusing on identifying mislabeled examples and eliminating noise\([Wilson, 1972](https://arxiv.org/html/2610.27142#bib.bib45);[Brodley and Friedl, 1999](https://arxiv.org/html/2610.27142#bib.bib46)\)\. For LLM training, the classifiers are usually trained with high\-quality text as positive data, e\.g\., Wikipedia\([Brown et al\., 2020](https://arxiv.org/html/2610.27142#bib.bib9);[Gao et al\., 2020](https://arxiv.org/html/2610.27142#bib.bib10)\)\. Alternatively, positive samples can also be task\-specific data, such as instruction following\([Xie et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib15);[Li et al\., 2024a](https://arxiv.org/html/2610.27142#bib.bib12)\)\. More recently, researchers also use an LLM to annotate positive samples and ensemble several classifiers for different aspects of data quality\([Penedo et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib13);[Su et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib14)\)\. While these methods are efficient enough for pretraining pools, they score each document independently and do not consider the distribution of the selected set\.

Loss and perplexity scoring\.Another family of data selection methods filter examples by a language model’s loss or perplexity, treating this as a proxy for sample utility\([Lewis and Gale, 1994](https://arxiv.org/html/2610.27142#bib.bib47);[Moore and Lewis, 2010](https://arxiv.org/html/2610.27142#bib.bib31);[Wenzek et al\., 2020](https://arxiv.org/html/2610.27142#bib.bib11)\)\. Existing approaches for LLM training differ on which part of the loss range to preserve: the hardest examples, the middle\([Marion et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib17);[Ankner et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib16)\), or the easiest, such as the most likely response to each instruction\([Zhang et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib20)\)\. For instruction tuning,[Li et al\. \(2024b\)](https://arxiv.org/html/2610.27142#bib.bib19)compare the loss of the response with and without the instruction, ensuring that difficulty reflects the instruction rather than the response alone\. However, a loss score only measures how difficult an example is for the model; since a useful example can be either easy or hard, such methods may overlook other aspects of data quality\([Wettig et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib25);[Sachdeva et al\., 2026](https://arxiv.org/html/2610.27142#bib.bib24)\)\.

Similarity\-based data filtering\.Similarity\-based methods select data that are most similar to a small dev set from the target task\([Shimodaira, 2000](https://arxiv.org/html/2610.27142#bib.bib48);[Huang et al\., 2006](https://arxiv.org/html/2610.27142#bib.bib49);[Koh and Liang, 2017](https://arxiv.org/html/2610.27142#bib.bib26)\)\. Specifically, they embed the pool and dev samples into a representation space and compute similarity between them\. The choice of representation varies, including the gradient of the model’s loss\([Xia et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib1);[Dai et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib27)\), the hidden states of the model being trained\([Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\), or the output of a separate embedding model\([Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5);[Yin and Rush, 2025](https://arxiv.org/html/2610.27142#bib.bib18)\)\. They also differ in their selection scheme:[Xia et al\. \(2024\)](https://arxiv.org/html/2610.27142#bib.bib1)rank examples by their best match to any dev sample;[Ivison et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib5)go through the dev samples in a round\-robin fashion to equally optimize for every dev sample; and[Dai et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib27)iteratively select the sample that most benefits the currently most underrepresented task\. Despite these efforts, there is a lack of systematic study on which features and which selection scheme to use\. These similarity\-based methods also rely on the best matches to the dev samples, whereas our ICLHerd selects a set of samples whose distribution is close to that of the dev set\.

LLM\-as\-a\-judge for data filtering\.LLM\-as\-a\-judge \(LaaJ\) uses a strong language model to evaluate text\([Zheng et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib28);[Liu et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib29);[Gu et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib30)\), and has also been used for data selection by rating each training example and keeping the high\-scoring ones\([Chen et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib22);[Wettig et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib25);[Sachdeva et al\., 2026](https://arxiv.org/html/2610.27142#bib.bib24)\)\. These methods differ in how they define quality, e\.g\., accuracy\([Chen et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib22)\), complexity\([Liu et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib23)\), and multi\-criteria rubrics\([Wettig et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib25)\)\. Researchers also develop different ways to elicit a judgment, e\.g\., a 0–5 rating\([Chen et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib22)\), the probability of a “yes” answer\([Sachdeva et al\., 2026](https://arxiv.org/html/2610.27142#bib.bib24)\), and pairwise comparison\([Wettig et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib25)\)\. However, scoring a whole pool with a large judge is computationally expensive; to address this, some methods distill judge labels into a small trained scorer\([Liu et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib23);[Wettig et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib25)\)\. Since each example is scored independently and may be redundant, a separate mechanism for coverage is often added to the approach, such as a diversity filter and a sampling temperature\([Liu et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib23);[Wettig et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib25);[Sachdeva et al\., 2026](https://arxiv.org/html/2610.27142#bib.bib24)\)\.

Recent work shows that random selection and hidden\-state similarity are strong baselines for LLMs\([Diddee and Ippolito, 2025](https://arxiv.org/html/2610.27142#bib.bib21);[Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\), a finding consistent with our results\. This suggests that data selection for LLMs is still challenging and warrants further investigation\.

## 3Method

### 3\.1Problem Formulation

Consider a standard train, dev, and test split of data, where we have three distinct sets:𝒟\\mathcal\{D\}for model training,𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}for hyperparameter tuning, and𝒟test\\mathcal\{D\}\_\{\\text\{test\}\}for model evaluation\. In the context of LLMs, a model is usually pretrained with massive unlabeled data and then finetuned with in\-domain data to improve performance\. Even for fine\-tuning, the training set𝒟\\mathcal\{D\}is often a gigantic pool of unfiltered texts scraped from the Internet, most of which are noisy and out of distribution with respect to𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}and𝒟test\\mathcal\{D\}\_\{\\text\{test\}\}\([Albalak et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib8);[Penedo et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib13);[Qin et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib7)\)\.

In this paper, we study data selection: our goal is to filter𝒟\\mathcal\{D\}to a much smaller subset, where the selected data are better aligned to the task that we care about, for example, reasoning and instruction following\. Under this formulation, we utilize𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}to guide the selection process, in addition to its role of hyperparameter tuning in standard machine learning\. Formally, given a budget ofNselN\_\{\\text\{sel\}\}training samples, our goal is to find the optimal subset𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}that minimizes a loss on the dev set

minimize𝒟sel⊂𝒟ℒ​\(𝒟dev,argmin𝜽ℒ​\(𝒟sel,𝜽\)\)subject to\|𝒟sel\|=Nsel,\\displaystyle\\operatorname\*\{minimize\}\_\{\\mathcal\{D\}\_\{\\text\{sel\}\}\\subset\\mathcal\{D\}\}\\;\\mathcal\{L\}\\\!\\left\(\\mathcal\{D\}\_\{\\text\{dev\}\};\\;\\operatorname\*\{argmin\}\\limits\_\{\\bm\{\\theta\}\}\\mathcal\{L\}\(\\mathcal\{D\}\_\{\\text\{sel\}\};\\bm\{\\theta\}\)\\right\)\\quad\\text\{subject to\}\\quad\|\\mathcal\{D\}\_\{\\text\{sel\}\}\|=N\_\{\\text\{sel\}\},\(1\)whereℒ⁡\(⋅,𝜽\)\\mathcal\{L\}\(\\cdot;\\bm\{\\theta\}\)computes the loss on a dataset with parameter𝜽\\bm\{\\theta\}, andargmin𝜽ℒ​\(𝒟sel,𝜽\)\\operatorname\*\{argmin\}\_\{\\bm\{\\theta\}\}\\mathcal\{L\}\(\\mathcal\{D\}\_\{\\text\{sel\}\};\\bm\{\\theta\}\)is typically given by a gradient descent algorithm as an approximation\.

Note that this optimization problem is difficult, because it is an integer programming problem with non\-linear loss function that involves an expensiveargmin\\operatorname\*\{argmin\}operator over the entire parameter space\. Brute forcing this problem requires training the model\(\|𝒟\|Nsel\)\\binom\{\|\\mathcal\{D\}\|\}\{N\_\{\\text\{sel\}\}\}times and choosing the best subset, which is not feasible in practice\.

### 3\.2Our ICLHerd Approach

The overall goal of our approach is to select a dataset𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}, whose distribution is close to that of𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}\. However, we only have discrete data samples in𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}and𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}, without an explicit form of their distributions\. Thus, we use the maximum mean discrepancy\([Gretton et al\., 2012](https://arxiv.org/html/2610.27142#bib.bib34)\)to measure distribution similarity based on samples using a kernel function\. With such a formulation, the objective becomes selecting samples that are close to𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}, but with a diversity penalty\. We minimize this objective by the classic herding algorithm, which selects samples greedily while maintaining a running profile of diversity scores\.

In the rest of this section, we will first introduce the MMD formulation, kernel design, and the herding algorithm\. Then, we will present an in\-context\-learning\-as\-a\-proxy method for efficient hyperparameter tuning in the LLM setting\.

MMD formulation\.We assume𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}and𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}are sampled from two underlying distributionsPselP\_\{\\text\{sel\}\}andPdevP\_\{\\text\{dev\}\}, respectively\. Specifically, the MMD betweenPselP\_\{\\text\{sel\}\}andPdevP\_\{\\text\{dev\}\}is defined as

MMD⁡\(Psel,Pdev\)=‖𝔼𝐱∼Psel\[ϕ⁡\(𝐱\)\]−𝔼𝐲∼Pdev\[ϕ⁡\(𝐲\)\]‖ℋ2,\\displaystyle\\operatorname\{MMD\}\(P\_\{\\text\{sel\}\},P\_\{\\text\{dev\}\}\)=\\big\\\|\\mathop\{\\mathbb\{E\}\}\_\{\\mathbf\{x\}\\sim P\_\{\\text\{sel\}\}\}\[\\phi\(\\mathbf\{x\}\)\]\-\\mathop\{\\mathbb\{E\}\}\_\{\\mathbf\{y\}\\sim P\_\{\\text\{dev\}\}\}\[\\phi\(\\mathbf\{y\}\)\]\\big\\\|\_\{\\mathcal\{H\}\}^\{2\},\(2\)whereϕ:𝒳→ℋ\\phi:\\mathcal\{X\}\\to\\mathcal\{H\}is a feature mapping into a reproducing kernel Hilbert spaceℋ\\mathcal\{H\}, and∥⋅∥ℋ\\\|\\cdot\\\|\_\{\\mathcal\{H\}\}is the norm in the Hilbert space\. We estimate the MMD using finite datasets𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}and𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}, given by

MMD≈‖1\|𝒟sel\|​∑𝐱∈𝒟selϕ⁡\(𝐱\)−1\|𝒟dev\|​∑𝐲∈𝒟devϕ⁡\(𝐲\)‖2\.\\displaystyle\{\\operatorname\{MMD\}\}\\approx\\Big\\\|\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\text\{sel\}\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{D\}\_\{\\text\{sel\}\}\}\\phi\(\\mathbf\{x\}\)\-\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\text\{dev\}\}\|\}\\sum\_\{\\mathbf\{y\}\\in\\mathcal\{D\}\_\{\\text\{dev\}\}\}\\phi\(\\mathbf\{y\}\)\\Big\\\|^\{2\}\.\(3\)When expanding the formula, we replaceϕ​\(⋅\)⊤​ϕ​\(⋅\)\\phi\(\\mathbf\{\\cdot\}\)^\{\\top\}\\phi\(\\cdot\)with a general kernel functionk⁡\(⋅,⋅\)k\(\\cdot,\\cdot\)\. This allows us to consider sample similarity without explicitly definingϕ⁡\(⋅\)\\phi\(\\cdot\), ask⁡\(⋅,⋅\)k\(\\cdot,\\cdot\)serves as an inner product in theϕ⁡\(⋅\)\\phi\(\\cdot\)space\.

We further drop the1\|𝒟dev\|2​∑𝐲,𝐲′∈𝒟devk⁡\(𝐲,𝐲′\)\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\text\{dev\}\}\|^\{2\}\}\\sum\_\{\\mathbf\{y\},\\mathbf\{y\}^\{\\prime\}\\in\\mathcal\{D\}\_\{\\text\{dev\}\}\}k\(\\mathbf\{y\},\\mathbf\{y\}^\{\\prime\}\)term, as it is independent of𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}\. The objective becomes

minimize𝒟sel⊂𝒟1\|𝒟sel\|2​∑𝐱,𝐱′∈𝒟selk⁡\(𝐱,𝐱′\)−2\|𝒟sel\|​\|𝒟dev\|​∑𝐱∈𝒟sel,𝐲∈𝒟devk⁡\(𝐱,𝐲\)\.\\displaystyle\\operatorname\*\{minimize\}\_\{\\mathcal\{D\}\_\{\\text\{sel\}\}\\subset\\mathcal\{D\}\}\{\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\text\{sel\}\}\|^\{2\}\}\}\\sum\_\{\\mathbf\{x\},\\mathbf\{x\}^\{\\prime\}\\in\\mathcal\{D\}\_\{\\text\{sel\}\}\}k\(\\mathbf\{x\},\\mathbf\{x\}^\{\\prime\}\)\-\{\\frac\{2\}\{\|\\mathcal\{D\}\_\{\\text\{sel\}\}\|\|\\mathcal\{D\}\_\{\\text\{dev\}\}\|\}\}\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{D\}\_\{\\text\{sel\}\},\\mathbf\{y\}\\in\\mathcal\{D\}\_\{\\text\{dev\}\}\}k\(\\mathbf\{x\},\\mathbf\{y\}\)\.\(4\)
We also convert this to a maximization problem for clarity, and introduce a hyperparameterα\\alphabalancing the two terms instead of using fixed coefficients\. Overall, we have the following objective

maximize𝒟sel⊂𝒟∑𝐱∈𝒟sel,𝐲∈𝒟devk⁡\(𝐱,𝐲\)⏟similarity−α​∑𝐱,𝐱′∈𝒟selk⁡\(𝐱,𝐱′\)⏟diversity penalty\.\\displaystyle\\operatorname\*\{maximize\}\_\{\\mathcal\{D\}\_\{\\text\{sel\}\}\\subset\\mathcal\{D\}\}\\mkern 12\.0mu\\underbrace\{\\sum\_\{\\mathclap\{\\mathbf\{x\}\\in\\mathcal\{D\}\_\{\\text\{sel\}\},\\mathbf\{y\}\\in\\mathcal\{D\}\_\{\\text\{dev\}\}\}\}k\(\\mathbf\{x\},\\mathbf\{y\}\)\}\_\{\\text\{similarity\}\}\\;\-\\;\\alpha\\underbrace\{\\sum\_\{\\mathclap\{\\mathbf\{x\},\\mathbf\{x\}^\{\\prime\}\\in\\mathcal\{D\}\_\{\\text\{sel\}\}\}\}k\(\\mathbf\{x\},\\mathbf\{x\}^\{\\prime\}\)\}\_\{\\text\{diversity penalty\}\}\.\(5\)
Intuitively, the first term aims to maximize the similarity with𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}, the second term encourages selecting diverse data samples, avoiding the selected samples crowding around certain modes\. Note that the diversity objective is naturally derived from the MMD formulation, whereas previous work uses heuristics to select diverse data\([Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\)\.

Kernel design\.A common choice of the kernel is the radial basis function \(RBF\)\. For neural networks, the von Mises–Fisher kernel\([Hall et al\., 1987](https://arxiv.org/html/2610.27142#bib.bib6)\)is often preferred\([Kobayashi, 2018](https://arxiv.org/html/2610.27142#bib.bib50)\), as the representations are often compared by their direction rather than the Euclidean distance\. We empirically analyze different kernels and observe that vMF is indeed superior to other options\.

Specifically, the von Mises–Fisher \(vMF\) kernel scales exponentially with the cosine similarity controlled by a concentration parameter\. This allows us to tune how sharply similarity decays as samples move away from the dev samples\. Formally, we have

kvMF​\(𝐱,𝐲\)=exp⁡\(κ​ψ​\(𝐱\)⊤​ψ​\(𝐲\)\),\\displaystyle k\_\{\\text\{vMF\}\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\exp\\big\(\\kappa\\,\\psi\(\\mathbf\{x\}\)^\{\\top\}\\psi\(\\mathbf\{y\}\)\\big\),\(6\)whereψ⁡\(⋅\)\\psi\(\\cdot\)is a unit\-norm feature map that represents the samples in an embedding space, andκ\\kappais the concentration parameter that balances between mode\-averaging \(mean similarity\) and mode\-seeking \(max similarity\)\.

As shown in Eqn\. \([6](https://arxiv.org/html/2610.27142#S3.E6)\), a highκ\\kappacauses mode seeking, since the influence of distant samples diminishes rapidly, encouraging the selected subset to cluster around the development samples\. On the other hand, a lowκ\\kappacauses mode averaging, as it places greater emphasis on preserving global similarity and results in selected samples crowded at the center of the dev samples \(which may not resemble any dev samples\)\.

For the representationψ⁡\(⋅\)\\psi\(\\cdot\)of a sample, previous studies use either gradients or hidden states of an LLM\([Xia et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib1);[Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\), but we notice that they complement each other\. Therefore, we propose a fused vMF kernel that utilizes signals from both hidden states and gradients

kfused\-vMF​\(𝐱,𝐲\)=β​kvMF​\(ψemb​\(𝐱\),ψemb​\(𝐲\)\)\+\(1−β\)​kvMF​\(ψgrad​\(𝐱\),ψgrad​\(𝐲\)\)\\displaystyle k\_\{\\text\{fused\-vMF\}\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\beta\\,k\_\{\\text\{vMF\}\}\\big\(\\psi\_\{\\text\{emb\}\}\(\\mathbf\{x\}\),\\psi\_\{\\text\{emb\}\}\(\\mathbf\{y\}\)\\big\)\+\(1\-\\beta\)\\,k\_\{\\text\{vMF\}\}\\big\(\\psi\_\{\\text\{grad\}\}\(\\mathbf\{x\}\),\\psi\_\{\\text\{grad\}\}\(\\mathbf\{y\}\)\\big\)\(7\)whereψgrad​\(⋅\)\\psi\_\{\\text\{grad\}\}\(\\cdot\)andψemb​\(⋅\)\\psi\_\{\\text\{emb\}\}\(\\cdot\)compute the unit\-norm feature map based on the model’s gradients and hidden states, respectively, andβ\\betacontrols the strength of each signal\. Specifically, we follow[Xia et al\. \(2024\)](https://arxiv.org/html/2610.27142#bib.bib1)to down\-project the gradients, which reduces the memory cost, and follow[Ivison et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib5)to pool the hidden states over tokens\.

Herding algorithm\.We adopt the herding algorithm\([Bach et al\., 2012](https://arxiv.org/html/2610.27142#bib.bib35)\)to optimize the MMD objective\. Essentially, data herding is a greedy algorithm that chooses samples to maximize Eqn\. \([5](https://arxiv.org/html/2610.27142#S3.E5)\) iteratively while keeping a running selected set for the diversity penalty\.

First, the selected set is initialized asS0=∅S\_\{0\}=\\emptyset\.

For each steptt, we pick one sample𝐱t\\mathbf\{x\}\_\{t\}by

argmax𝐱∈𝒟∖St−1\[∑𝐲∈𝒟devk⁡\(𝐱,𝐲\)−α​∑𝐱′∈St−1k⁡\(𝐱,𝐱′\)\]\\displaystyle\\operatorname\*\{argmax\}\_\{\\mathbf\{x\}\\in\\mathcal\{D\}\\setminus S\_\{t\-1\}\}\\Bigg\[\\sum\_\{\\mathbf\{y\}\\in\\mathcal\{D\}\_\{\\text\{dev\}\}\}k\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\alpha\\sum\_\{\\mathbf\{x\}^\{\\prime\}\\in S\_\{t\-1\}\}k\(\\mathbf\{x\},\\mathbf\{x\}^\{\\prime\}\)\\Bigg\]\(8\)The selected sample𝐱t\\mathbf\{x\}\_\{t\}is added to the selected set, given bySt=St−1∪\{𝐱t\}S\_\{t\}=S\_\{t\-1\}\\cup\\\{\\mathbf\{x\}\_\{t\}\\\}\.

Overall, our ICLHerd generalizes existing similarity\-based methods and better balances between extreme mode averaging and mode seeking\.

In\-context learning proxy\.ICLHerd has several hyperparameters, such as the concentration parameterκ\\kappathat balances mode averaging and mode seeking, the diversity weightα\\alpha, and the fusion weightβ\\beta, which need to be tuned properly\. Traditional hyperparameter tuning for data selection requires repeated model training and subsequent inference to evaluate the selected subset, which is computationally expensive and limits how thoroughly we can balance between mode\-seeking and mode\-averaging\.

To this end, we propose in\-context learning as an efficient validation proxy, where we place the selected data samples in the model’s context and measure the loss on the dev set to gauge the data quality\. Intuitively, higher\-quality training samples are also more informative in\-context learning samples, thus resulting in a lower loss on the dev set when they are used to prompt an LLM\. This allows us to estimate the quality of the data without training on the dataset\.

However, it is noisy and still costly to perform in\-context learning evaluation for each individual samples\. We propose to estimate the quality of𝒟sel\\mathcal\{D\}\_\{\\text\{sel\}\}by repeatedly drawingkksamples and evaluate their quality\.

Let𝐱1,𝐱2,…,𝐱k∈𝒟sel\\mathbf\{x\}\_\{1\},\\mathbf\{x\}\_\{2\},\\ldots,\\mathbf\{x\}\_\{k\}\\in\\mathcal\{D\}\_\{\\text\{sel\}\}bekksamples randomly drawn from the selected dataset such that it fits in a budget ofBBtokens available for in\-context learning\. We build an in\-context learning prompt by concatenation:

𝐜=𝐱1⊕𝐱2⊕⋯⊕𝐱k\.\\displaystyle\\mathbf\{c\}=\\mathbf\{x\}\_\{1\}\\oplus\\mathbf\{x\}\_\{2\}\\oplus\\cdots\\oplus\\mathbf\{x\}\_\{k\}\.\(9\)
We then estimate quality of the in\-context learning samples𝐜\\mathbf\{c\}by computing the loss on the dev set, given by

ℒ\(𝐜\)=1\|𝒟dev\|∑𝐱∈𝒟dev\(−1\|𝐱\|∑t=1\|𝐱\|logp𝜽\(xt∣𝐜,𝐱<t\)\)\.\\displaystyle\\mathcal\{L\}\(\\mathbf\{c\}\)=\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\text\{dev\}\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{D\}\_\{\\text\{dev\}\}\}\\bigg\(\-\\frac\{1\}\{\|\\mathbf\{x\}\|\}\\sum\_\{t=1\}^\{\|\\mathbf\{x\}\|\}\\log p\_\{\\bm\{\\theta\}\}\\big\(x\_\{t\}\\mid\\mathbf\{c\},\\mathbf\{x\}\_\{<t\}\\big\)\\bigg\)\.\(10\)
Although one subset𝐱1,𝐱2,…,𝐱k\\mathbf\{x\}\_\{1\},\\mathbf\{x\}\_\{2\},\\ldots,\\mathbf\{x\}\_\{k\}already gives a decent proxy for data quality, this score may be susceptible to random noise\. To mitigate this, we repeat this processMMtimes with different random samples, each forming a context𝐜m\\mathbf\{c\}\_\{m\}form=1,⋯,Mm=1,\\cdots,M\.

Finally, we aggregate a proxy score as the maximum loss across different draws

ICLProxy⁡\(𝒟sel\)=maxm=1M⁡ℒ⁡\(𝐜m\)\\displaystyle\\operatorname\{ICLProxy\}\(\\mathcal\{D\}\_\{\\text\{sel\}\}\)=\\max\_\{m=1\}^\{M\}\\mathcal\{L\}\(\\mathbf\{c\}\_\{m\}\)\(11\)Note that it is possible to also take the average instead the maximum, but our pilot study suggests that max aggregation is more consistent with actual data quality, potentially because bad outlier samples are especially detrimental to training\.

ICLHerd is efficient, as we use a prefix caching technique when evaluating different samples in𝒟dev\\mathcal\{D\}\_\{\\text\{dev\}\}\. In other words, the cached prefix𝐜\\mathbf\{c\}is computed once and reused for different dev samples, so each dev sample only requires processing its own tokens\. Overall, our proposed method avoids repeated LLM training for hyperparameter tuning and is much faster than the traditional training–validation framework\.

## 4Experiments

### 4\.1Setups

Selection pool dataset\.Following[Ivison et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib5), we use the unfiltered Tülu 2 SFT mixture dataset\([Ivison et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib36)\)as our selection pool𝒟\\mathcal\{D\}, containing approximately 5\.8M samples from sources including Open Orca\([Mukherjee et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib39)\), FLAN\([Chung et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib37)\), and WizardLM Evol\-Instruct\([Xu et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib38)\)\. This pool combines data across various skills, requiring the selection algorithm to identify examples relevant to each target task\.

Table 1:Main results on five tasks with 10K selected samples per task\. Training data are selected using Qwen3\-1\.7B representations and used to finetune Qwen3\-1\.7B, 4B, and 8B\. Due to resource constraints, Top\-PPL and Mid\-PPL are only evaluated at 1\.7B as their performance is relatively low\. The best and second\-best results within each model size areboldandunderlined, respectively\. ICLHerd significantly outperforms RDS\+, the strongest baseline, under a two\-sided binomial test across tasks and model sizes \(p≈0\.007p\\approx 0\.007\)\.Evaluation tasks and datasets\.We evaluate all methods on five tasks in a zero\-shot setting: GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.27142#bib.bib40)\), MLQA\([Lewis et al\., 2020](https://arxiv.org/html/2610.27142#bib.bib41)\), DROP\([Dua et al\., 2019](https://arxiv.org/html/2610.27142#bib.bib42)\), XSum\([Narayan et al\., 2018](https://arxiv.org/html/2610.27142#bib.bib43)\), and OASST1\([Köpf et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib44)\)\. These tasks cover a diverse range of skills, including mathematical reasoning, multilingual question answering, reading comprehension, summarization, and instruction following\. For in\-depth analyses, we focus on GSM8K and MLQA to test two distinct capabilities: reasoning and multilinguality\. Details are presented in Appendix[6](https://arxiv.org/html/2610.27142#S6)\.

Baselines\.We compare ICLHerd against five representative baselines spanning random, loss\-based, and representation\-based selection\.

∙\\bulletRandom: uniform sampling from the pool\. This is a surprisingly strong baseline that often remains competitive\([Diddee and Ippolito, 2025](https://arxiv.org/html/2610.27142#bib.bib21);[Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\)\.

∙\\bulletTop\-PPL\([Ankner et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib16)\): selecting samples with the highest perplexity loss under the base model\.

∙\\bulletMid\-PPL\([Marion et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib17)\): selecting samples around the median perplexity loss under the base model\.

∙\\bulletLESS\([Xia et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib1)\): embedding samples via gradients and ranking them by maximum cosine similarity to any dev sample\.

∙\\bulletRDS\+\([Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\): embedding samples via hidden states and selecting the best match for each dev sample in a round\-robin fashion\.

We provide additional details on the datasets in Appendix[7](https://arxiv.org/html/2610.27142#S7)\.

### 4\.2Results and Analyses

Table 2:Analysis of the concentration parameterκ\\kappausing the fusion kernel \(β=0\.5\\beta=0\.5, no diversity term\)\. Mean\- and max\-similarity on concatenated features serve as the approximate extremes for small and largeκ\\kappa, respectively\. Random and round\-robin \(on fused features\) lie outside theκ\\kappacontinuum and are shown as references\.κ\\kappais given as a multiple of the value from the median heuristic\([Gretton et al\., 2012](https://arxiv.org/html/2610.27142#bib.bib34)\)\.Table 3:Comparison of feature spaces\. RDS\+ and LESS use hidden states and gradients, respectively, while ICLHerd \(fused\) combines both in the kernel\.Main results\.The top half of Table[1](https://arxiv.org/html/2610.27142#S4.T1)shows the main results across the five tasks, where data are selected by Qwen3\-1\.7B and the performance is obtained by finetuning the same Qwen3\-1\.7B model\. As seen, random selection remains a strong baseline\([Diddee and Ippolito, 2025](https://arxiv.org/html/2610.27142#bib.bib21);[Ivison et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib5)\), and is particularly competitive on OASST1, likely because the Tülu 2 pool is rich in instruction\-tuning data\. Loss\-based methods, Top\-PPL and Mid\-PPL, generally underperform random selection\. This is because Top\-PPL favors surprising samples that often have very short answers; while this benefits the short\-span F1 on DROP, it degrades the chain\-of\-thought reasoning required for GSM8K\. Mid\-PPL is less extreme and thus maintains an overall edge over Top\-PPL\. In contrast, RDS\+[Ivison et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib5)outperforms both random and loss\-based selection, whereas LESS\([Xia et al\., 2024](https://arxiv.org/html/2610.27142#bib.bib1)\)barely improves over random on average and even falls behind it on MLQA, as its max\-similarity selection suffers from extreme mode seeking although Table[3](https://arxiv.org/html/2610.27142#S4.T3)shows its gradient\-based features are informative\. Our ICLHerd performs consistently well, achieving the best results on four of the five tasks and on average\. The only exception is DROP, where it ranks second to Top\-PPL because ICLHerd tends to generate more verbose answers that are penalized by the F1 score\.

MLQA per\-language results\.

Table 4:Per\-language F1 on MLQA using Qwen3\-1\.7B with 10k selected samples\. The average matches the MLQA column of Table[1](https://arxiv.org/html/2610.27142#S4.T1)\. The best and second\-best results areboldandunderlined, respectively\.Table[4](https://arxiv.org/html/2610.27142#S4.T4)presents the breakdown results for MLQA in terms of different languages\. As seen, ICLHerd achieves the best F1 in six of the seven languages\. The exception is Hindi, for which Top\-PPL perform surprisingly well, while other methods are only slightly better or even worse than random\. Among other languages, our ICLHerd has large margins in English, Spanish, and German, showing that our gain \(reported as average in Table[1](https://arxiv.org/html/2610.27142#S4.T1)\) is not driven by a few languages\.

Transfer to larger models\.We further test whether the selected data transfer to larger models, where data are selected with Qwen3\-1\.7B and fine\-tuned on Qwen3\-4B and 8B \(bottom half of Table[1](https://arxiv.org/html/2610.27142#S4.T1)\)\. This mimics a practical application scenario, where a small model is used to select high\-quality data for training large models\.

Since Top\-PPL and Mid\-PPL underperform random in the skill\-based selection setting, we drop them from now on to save compute\. As seen, ICLHerd outperforms both RDS\+ and LESS on average and achieves the highest performance in 8 out of 10 settings\. We perform a statistical test against RDS\+, which is the strongest baseline\. Including the 1\.7B results, ICLHerd outperforms RDS\+ under 13 of the 15 setups, which is statistically significant under a two\-sided binomial test \(p≈0\.007p\\approx 0\.007\)\.

These results indicate that a small model with ICLHerd can select data that are effective for significantly larger models, suggesting that the cost of data selection can be paid once with a small model and reused for larger ones\.

Table 5:Comparison of the ICL and Train–Val proxies against test performance\. Train–Val involves fine\-tuning on selected data followed by dev\-set evaluation\. The final row shows the Spearman’sρ\\rhobetween each proxy and the test score across the six methods\.Table 6:Efficiency comparison\. Times are in minutes, with inference measured on 1K samples\. Speedup is the total time of training–validation divided by that of each method, computed from unrounded times\.ρ\\rhodenotes the Spearman correlation relative to the rankings in Table[1](https://arxiv.org/html/2610.27142#S4.T1)\. Experiments use an NVIDIA L40S GPU\.Analyzing the concentration parameter\.Table[2](https://arxiv.org/html/2610.27142#S4.T2)compares different selection criteria and their impact on mode\-seeking and mode\-averaging behaviors\. As a controlled setting, all methods use the same feature space withβ=0\.5\\beta=0\.5\. Max\-similarity, as adopted by LESS, selects samples based on their highest similarity to any dev sample, leading to extreme mode seeking\. On MLQA, for instance, the max operator latches onto purely English samples, so max\-similarity is even worse than random selection\. Conversely, mean\-similarity selects samples based on their overall fit across the dev set; while this can be suboptimal if the average falls into a low\-density region, it nevertheless performs reasonably well on MLQA because the balanced dev set keeps the average close to each language\. Round\-robin, used by RDS\+, mitigates extreme mode\-seeking by matching each dev sample in turn, thereby promoting diversity and performing well on MLQA\. Generally, max\-similarity, mean\-similarity, and round\-robin perform similarly on GSM8K, where the selection should more carefully balance high\-quality chain\-of\-thought samples with diversity\.

Unlike these fixed criteria, ICLHerd controls the balance through the concentration parameterκ\\kappa, where a smallκ\\kappabehaves like mean\-similarity and a largeκ\\kappabehaves like max\-similarity\. Our ICLHerd selectsκ\\kappaby validation based on the in\-context learning proxy\. For the analysis, we manually set different scores forκ\\kappaand evaluate the performance\. In particular, we initializeκ\\kappausing the median heuristic\([Gretton et al\., 2012](https://arxiv.org/html/2610.27142#bib.bib34)\)and tune a multiplicative factor\. The results are reported in Table[2](https://arxiv.org/html/2610.27142#S4.T2)\. In this analysis, we also omit the diversity term \(settingα=0\\alpha=0\) across all runs to isolate the effect ofκ\\kappa\. In this way,κ=0\\kappa=0resembles mean\-similarity selection; ifκ\\kappaapproaches infinity, our method becomes max\-similarity selection\.

Empirical results show thatκ\\kappaindeed balances mode averaging and mode seeking, where small and largeκ\\kappamatches the performance of mean\- and max\-similarity, respectively\. On GSM8K, an intermediateκ\\kappais optimal, as a balancedκ\\kappastrikes a good balance between mode averaging and mode seeking, allowing us to pick diverse, high\-quality chain\-of\-thought reasoning samples\. On MLQA, mode averaging is preferred, potentially because the dev set is well\-balanced across languages\. By contrast, a largeκ\\kappasuffers from the same mode\-seeking issue as max\-similarity\.

Note that Table[2](https://arxiv.org/html/2610.27142#S4.T2)shows results of differentκ\\kappavalues that are set manually\. However, our ICLHerd uses in\-context learning as proxy and automatically selects the bestκ\\kappafor a task, highlighting the flexibility of our approach\.

Comparing feature spaces\.Table[3](https://arxiv.org/html/2610.27142#S4.T3)compares ICLHerd using different feature spaces\. As seen, ICLHerd variants generally outperform RDS\+ and LESS regardless of the features\. Within ICLHerd variants, hidden states and gradients provide complementary signals that suit different tasks, with hidden states performing better on GSM8K and gradients on MLQA\. By combining both, the fused space takes advantage of both worlds and achieves the best performance overall\.

Analyzing the in\-context learning proxy\.We compare the ICL proxy with the standard training–validation approach for hyperparameter tuning\. The latter is computationally inefficient as it requires full training and inference\.

We use Spearman’sρ\\rhoto measure the ranking correlation between the dev performance \(using ICL proxy or standard training–validation\) and test performance for the methods in Table[1](https://arxiv.org/html/2610.27142#S4.T1)\. As seen in Table[5](https://arxiv.org/html/2610.27142#S4.T5), the ICL proxy is comparable to standard training–validation in terms of method ranking: it achieves a perfect ranking on GSM8K, while training–validation does so on MLQA\. Moreover, both methods identify ICLHerd as the top\-performing method, consistent with the test results\. This shows that the ICL proxy is an effective replacement for fine\-tuning and validation\.

Analyzing time efficiency\.We further analyze the efficiency gains of the ICL proxy over training–validation in Table[6](https://arxiv.org/html/2610.27142#S4.T6), where we vary the number of in\-context example draws, i\.e\.,MMin Eqn\. \([11](https://arxiv.org/html/2610.27142#S3.E11)\)\. We see that our ICL proxy yields significant speedup of 68–157×\\timeswhenM=1M=1, although the proxy quality is slightly lower \(shown by Spearman’sρ\\rhocorrelation\)\. Even withM=10M=10, our method remains 7–16×\\timesfaster\. This is understandable, as our ICL proxy eliminates the need for training, which is often the most time\-consuming component of the training–validation framework\.

Comparing kernels\.

Table 7:Comparison of kernels within our herding framework, all using fused hidden\-state and gradient features\.We compare the choice of kernel function in Table[7](https://arxiv.org/html/2610.27142#S4.T7)\. Between Euclidean and cosine kernels, we observe that cosine similarity generally performs better\. This is consistent with existing literature, which prefers using direction rather than magnitude for high\-dimensional embeddings\([Banerjee et al\., 2005](https://arxiv.org/html/2610.27142#bib.bib51);[Reimers and Gurevych, 2019](https://arxiv.org/html/2610.27142#bib.bib52);[Gao et al\., 2021](https://arxiv.org/html/2610.27142#bib.bib53)\)\. Both RBF and vMF kernels offer an exponential decay when samples are moved apart, which enables a balance between mode averaging and mode seeking; they outperform the Euclidean and cosine kernels by a large margin\. Furthermore, the angular similarity of the vMF kernel outperforms Euclidean distance, being the most effective kernel across all tested options\.

Effect of the dev set size\.

Table 8:Effect of the dev set size on ICLHerd\. Full denotes the entire dev set \(7,473 samples for GSM8K and 3,500 for MLQA\)\.We study the effect of the dev set size on our ICLHerd\. We progressively sample 10, 100, and 1,000 dev entries\. We keep the 10K training budget and tune hyperparameters on the full dev set\. As shown in Table[8](https://arxiv.org/html/2610.27142#S4.T8), performance is nearly flat across dev set sizes, varying by less than one point on both tasks\. This is because the dev set enters the objective only through the average kernel similarity, which a few samples already estimate well\. This highlights the robustness of our approach across different dev set sizes\.

## 5Conclusion

In this work, we propose ICLHerd, a data selection framework for LLMs based on data herding, which balances mode averaging and mode seeking with a vMF kernel fused over hidden states and gradients\. We further propose in\-context learning as an efficient proxy for tuning its hyperparameters without training\. Experiments show that ICLHerd outperforms existing data selection methods on average, and that our proxy is a reliable and efficient replacement for fine\-tuning and validation\.

## References

- Albalaket al\.\(2024\)A\. Albalak, Y\. Elazar, S\. M\. Xie, S\. Longpre, N\. Lambert, X\. Wang, N\. Muennighoff, B\. Hou, L\. Pan, H\. Jeong, C\. Raffel, S\. Chang, T\. Hashimoto, and W\. Y\. WangA survey on data selection for language models\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=XfHWcNTSHp)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.27142#S3.SS1.p1.1)\.
- Ankneret al\.\(2024\)Z\. Ankner, C\. Blakeney, K\. Sreenivasan, M\. Marion, M\. L\. Leavitt, and M\. PaulPerplexed by perplexity: perplexity\-based data pruning with small reference models\.arXiv preprint arXiv:2405\.20541\.External Links:[Link](https://arxiv.org/abs/2405.20541)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p5.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.4.1.1),[Table 4](https://arxiv.org/html/2610.27142#S4.T4.7.3.1.1)\.
- Bachet al\.\(2012\)F\. Bach, S\. Lacoste\-Julien, and G\. ObozinskiOn the equivalence between herding and conditional gradient algorithms\.arXiv preprint arXiv:1203\.4523\.External Links:[Link](https://arxiv.org/abs/1203.4523)Cited by:[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p11.1)\.
- Banerjeeet al\.\(2005\)A\. Banerjee, I\. S\. Dhillon, J\. Ghosh, and S\. SraClustering on the unit hypersphere using von Mises–Fisher distributions\.Journal of Machine Learning Research6\(46\),pp\. 1345–1382\.External Links:[Link](https://jmlr.org/papers/v6/banerjee05a.html)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p16.1)\.
- Brodley and Friedl \(1999\)C\. E\. Brodley and M\. A\. FriedlIdentifying mislabeled training data\.Journal of Artificial Intelligence Research11,pp\. 131–167\.External Links:[Link](https://doi.org/10.1613/jair.606)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Chenet al\.\(2024\)L\. Chen, S\. Li, J\. Yan, H\. Wang, K\. Gunaratna, V\. Yadav, Z\. Tang, V\. Srinivasan, T\. Zhou, H\. Huang,et al\.AlpaGasus: training a better Alpaca with fewer data\.InProceedings of the International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FdVXgSJhvz)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.
- Chenet al\.\(2010\)Y\. Chen, M\. Welling, and A\. SmolaSuper\-samples from kernel herding\.InProceedings of the Conference on Uncertainty in Artificial Intelligence,pp\. 117–124\.External Links:[Link](https://proceedings.mlr.press/r8/chen10a.html)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1)\.
- Chunget al\.\(2024\)H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.External Links:[Link](https://jmlr.org/papers/v25/23-0870.html)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p2.1),[§7](https://arxiv.org/html/2610.27142#S7.p2.1)\.
- Daiet al\.\(2025\)Q\. Dai, D\. Zhang, J\. W\. Ma, and H\. PengImproving influence\-based instruction tuning data selection for balanced learning of diverse capabilities\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 7079–7102\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.373/)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p4.1)\.
- Diddee and Ippolito \(2025\)H\. Diddee and D\. IppolitoChasing random: instruction selection strategies fail to generalize\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 1943–1957\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.103/)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p6.1),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p4.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p1.1)\.
- Duaet al\.\(2019\)D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. GardnerDROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 2368–2378\.External Links:[Link](https://aclanthology.org/N19-1246/)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p2.1),[§7](https://arxiv.org/html/2610.27142#S7.p4.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. LeahyThe Pile: an 800GB dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.External Links:[Link](https://arxiv.org/abs/2101.00027)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Gaoet al\.\(2021\)T\. Gao, X\. Yao, and D\. ChenSimCSE: simple contrastive learning of sentence embeddings\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 6894–6910\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.552/)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p16.1)\.
- Grettonet al\.\(2012\)A\. Gretton, K\. M\. Borgwardt, M\. J\. Rasch, B\. Schölkopf, and A\. SmolaA kernel two\-sample test\.Journal of Machine Learning Research13\(25\),pp\. 723–773\.External Links:[Link](https://jmlr.org/papers/v13/gretton12a.html)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p1.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p8.1),[Table 2](https://arxiv.org/html/2610.27142#S4.T2),[Table 2](https://arxiv.org/html/2610.27142#S4.T2.4)\.
- Guet al\.\(2024\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Wang, W\. Gao, L\. Ni, and J\. GuoA survey on LLM\-as\-a\-judge\.arXiv preprint arXiv:2411\.15594\.External Links:[Link](https://arxiv.org/abs/2411.15594)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.
- Hallet al\.\(1987\)P\. Hall, G\. S\. Watson, and J\. CabreraKernel density estimation with spherical data\.Biometrika74\(4\),pp\. 751–762\.External Links:[Link](https://doi.org/10.1093/biomet/74.4.751)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p7.1)\.
- Huanget al\.\(2006\)J\. Huang, A\. J\. Smola, A\. Gretton, K\. M\. Borgwardt, and B\. SchölkopfCorrecting sample selection bias by unlabeled data\.InAdvances in Neural Information Processing Systems,pp\. 601–608\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2006/hash/a2186aa7c086b46ad4e8bf81e2a3a19b-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p4.1)\.
- Ivisonet al\.\(2023\)H\. Ivison, Y\. Wang, V\. Pyatkin, N\. Lambert, M\. Peters, P\. Dasigi, J\. Jang, D\. Wadden, N\. A\. Smith, I\. Beltagy, and H\. HajishirziCamels in a changing climate: enhancing LM adaptation with Tülu 2\.arXiv preprint arXiv:2311\.10702\.External Links:[Link](https://arxiv.org/abs/2311.10702)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p1.1)\.
- Ivisonet al\.\(2025\)H\. Ivison, M\. Zhang, F\. Brahman, P\. W\. Koh, and P\. DasigiLarge\-scale data selection for instruction tuning\.arXiv preprint arXiv:2503\.01807\.External Links:[Link](https://arxiv.org/abs/2503.01807)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p2.1),[§2](https://arxiv.org/html/2610.27142#S2.p4.1),[§2](https://arxiv.org/html/2610.27142#S2.p6.1),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p10.1),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p10.2),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p6.1),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p4.1),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p8.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.13.1.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.17.1.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.7.1.1),[Table 4](https://arxiv.org/html/2610.27142#S4.T4.7.6.1.1),[Table 5](https://arxiv.org/html/2610.27142#S4.T5.5.8.1.1),[§6](https://arxiv.org/html/2610.27142#S6.p1.1)\.
- Kobayashi \(2018\)H\. KobayashiFrustratingly easy model ensemble for abstractive summarization\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 4165–4176\.External Links:[Link](https://aclanthology.org/D18-1449/)Cited by:[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p7.1)\.
- Koh and Liang \(2017\)P\. W\. Koh and P\. LiangUnderstanding black\-box predictions via influence functions\.InProceedings of the International Conference on Machine Learning,pp\. 1885–1894\.External Links:[Link](https://proceedings.mlr.press/v70/koh17a.html)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p4.1)\.
- Köpfet al\.\(2023\)A\. Köpf, Y\. Kilcher, D\. von Rütte,et al\.OpenAssistant conversations – democratizing large language model alignment\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/949f0f8f32267d297c2d4e3ee10a2e7e-Abstract.html)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p2.1),[§7](https://arxiv.org/html/2610.27142#S7.p6.1)\.
- Lewis and Gale \(1994\)D\. D\. Lewis and W\. A\. GaleA sequential algorithm for training text classifiers\.InProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 3–12\.External Links:[Link](https://doi.org/10.1007/978-1-4471-2099-5_1)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1)\.
- Lewiset al\.\(2020\)P\. Lewis, B\. Oguz, R\. Rinott, S\. Riedel, and H\. SchwenkMLQA: evaluating cross\-lingual extractive question answering\.InProceedings of the Annual Meeting of the Association for Computational Linguistics,pp\. 7315–7330\.External Links:[Link](https://aclanthology.org/2020.acl-main.653/)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p2.1),[§7](https://arxiv.org/html/2610.27142#S7.p3.1)\.
- Liet al\.\(2024a\)J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha,et al\.DataComp\-LM: in search of the next generation of training sets for language models\.InAdvances in Neural Information Processing Systems,pp\. 14200–14282\.External Links:[Link](https://doi.org/10.52202/079017-0455)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p1.1),[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Liet al\.\(2024b\)M\. Li, Y\. Zhang, Z\. Li, J\. Chen, L\. Chen, N\. Cheng, J\. Wang, T\. Zhou, and J\. XiaoFrom quantity to quality: boosting LLM performance with self\-guided data selection for instruction tuning\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 7602–7635\.External Links:[Link](https://aclanthology.org/2024.naacl-long.421/)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1)\.
- Liuet al\.\(2024\)W\. Liu, W\. Zeng, K\. He, Y\. Jiang, and J\. HeWhat makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning\.InProceedings of the International Conference on Learning Representations,External Links:2312\.15685,[Link](https://openreview.net/forum?id=BTKAeLqLMw)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.153)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.
- Marionet al\.\(2023\)M\. Marion, A\. Üstün, L\. Pozzobon, A\. Wang, M\. Fadaee, and S\. HookerWhen less is more: investigating data pruning for pretraining LLMs at scale\.arXiv preprint arXiv:2309\.04564\.External Links:[Link](https://arxiv.org/abs/2309.04564)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p6.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.5.1.1),[Table 4](https://arxiv.org/html/2610.27142#S4.T4.7.4.1.1)\.
- Meta AI \(2025\)Meta AIThe Llama 4 herd: the beginning of a new era of natively multimodal AI innovation\.External Links:[Link](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p1.1)\.
- Moore and Lewis \(2010\)R\. C\. Moore and W\. LewisIntelligent selection of language model training data\.InProceedings of the Annual Meeting of the Association for Computational Linguistics: Short Papers,pp\. 220–224\.External Links:[Link](https://aclanthology.org/P10-2041/)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1)\.
- Mukherjeeet al\.\(2023\)S\. Mukherjee, A\. Mitra, G\. Jawahar, S\. Agarwal, H\. Palangi, and A\. AwadallahOrca: progressive learning from complex explanation traces of GPT\-4\.arXiv preprint arXiv:2306\.02707\.External Links:[Link](https://arxiv.org/abs/2306.02707)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p1.1)\.
- Narayanet al\.\(2018\)S\. Narayan, S\. B\. Cohen, and M\. LapataDon’t give me the details, just the summary\! Topic\-aware convolutional neural networks for extreme summarization\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 1797–1807\.External Links:[Link](https://aclanthology.org/D18-1206/)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p2.1),[§7](https://arxiv.org/html/2610.27142#S7.p5.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. WolfThe FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p1.1),[§2](https://arxiv.org/html/2610.27142#S2.p2.1),[§3\.1](https://arxiv.org/html/2610.27142#S3.SS1.p1.1)\.
- Qinet al\.\(2025\)Y\. Qin, Y\. Yang, P\. Guo, G\. Li, H\. Shao, Y\. Shi, Z\. Xu, Y\. Gu, K\. Li, and X\. SunUnleashing the power of data tsunami: a comprehensive survey on data assessment and selection for instruction tuning of language models\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=RJT1baPhdV)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.27142#S3.SS1.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing,pp\. 3982–3992\.External Links:[Link](https://aclanthology.org/D19-1410/)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p16.1)\.
- Sachdevaet al\.\(2026\)N\. Sachdeva, B\. Coleman, W\. Kang, J\. Ni, L\. Hong, E\. H\. Chi, J\. Caverlee, J\. McAuley, and D\. Z\. ChengHow to train data\-efficient LLMs\.InProceedings of the International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yKUbw7q1IA)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1),[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.
- Shimodaira \(2000\)H\. ShimodairaImproving predictive inference under covariate shift by weighting the log\-likelihood function\.Journal of Statistical Planning and Inference90\(2\),pp\. 227–244\.External Links:[Link](https://doi.org/10.1016/S0378-3758(00)00115-4)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p4.1)\.
- Suet al\.\(2025\)D\. Su, K\. Kong, Y\. Lin, J\. Jennings, B\. Norick, M\. Kliegl, M\. Patwary, M\. Shoeybi, and B\. CatanzaroNemotron\-CC: transforming Common Crawl into a refined long\-horizon pretraining dataset\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2459–2475\.External Links:[Link](https://aclanthology.org/2025.acl-long.123/)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Welling \(2009\)M\. WellingHerding dynamical weights to learn\.InProceedings of the International Conference on Machine Learning,pp\. 1121–1128\.External Links:[Link](https://doi.org/10.1145/1553374.1553517)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p4.1)\.
- Wenzeket al\.\(2020\)G\. Wenzek, M\. Lachaux, A\. Conneau, V\. Chaudhary, F\. Guzmán, A\. Joulin, and E\. GraveCCNet: extracting high quality monolingual datasets from web crawl data\.InProceedings of the Language Resources and Evaluation Conference,pp\. 4003–4012\.External Links:[Link](https://aclanthology.org/2020.lrec-1.494/)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p1.1),[§2](https://arxiv.org/html/2610.27142#S2.p3.1)\.
- Wettiget al\.\(2024\)A\. Wettig, A\. Gupta, S\. Malik, and D\. ChenQuRating: selecting high\-quality data for training language models\.InProceedings of the International Conference on Machine Learning,pp\. 52915–52971\.External Links:[Link](https://proceedings.mlr.press/v235/wettig24a.html)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1),[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.
- Wilson \(1972\)D\. L\. WilsonAsymptotic properties of nearest neighbor rules using edited data\.IEEE Transactions on Systems, Man, and CyberneticsSMC\-2\(3\),pp\. 408–421\.External Links:[Link](https://doi.org/10.1109/TSMC.1972.4309137)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Xiaet al\.\(2024\)M\. Xia, S\. Malladi, S\. Gururangan, S\. Arora, and D\. ChenLESS: selecting influential data for targeted instruction tuning\.InProceedings of the International Conference on Machine Learning,pp\. 54104–54132\.External Links:[Link](https://proceedings.mlr.press/v235/xia24c.html)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p2.1),[§2](https://arxiv.org/html/2610.27142#S2.p4.1),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p10.1),[§3\.2](https://arxiv.org/html/2610.27142#S3.SS2.p10.2),[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p7.1),[§4\.2](https://arxiv.org/html/2610.27142#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.12.1.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.16.1.1),[Table 1](https://arxiv.org/html/2610.27142#S4.T1.7.6.1.1),[Table 4](https://arxiv.org/html/2610.27142#S4.T4.7.5.1.1),[Table 5](https://arxiv.org/html/2610.27142#S4.T5.5.7.1.1),[§6](https://arxiv.org/html/2610.27142#S6.p1.1)\.
- Xieet al\.\(2023\)S\. M\. Xie, S\. Santurkar, T\. Ma, and P\. LiangData selection for language models via importance resampling\.InAdvances in Neural Information Processing Systems,pp\. 34201–34227\.External Links:[Link](https://doi.org/10.52202/075280-1482)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p2.1)\.
- Xuet al\.\(2026\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.DeepSeek\-V4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.External Links:[Link](https://arxiv.org/abs/2606.19348)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p1.1)\.
- Xuet al\.\(2024\)C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, Q\. Lin, and D\. JiangWizardLM: empowering large pre\-trained language models to follow complex instructions\.InProceedings of the International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/82eec786fdfbbfa53450c5feb7d1ac92-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2610.27142#S4.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2610.27142#S1.p1.1),[§6](https://arxiv.org/html/2610.27142#S6.p1.1)\.
- Yin and Rush \(2025\)J\. O\. Yin and A\. M\. RushCompute\-constrained data selection\.InProceedings of the International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4es2oO9tw1)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p4.1)\.
- Zhanget al\.\(2025\)D\. Zhang, Q\. Dai, and H\. PengThe best instruction\-tuning data are those that fit\.InAdvances in Neural Information Processing Systems,pp\. 156585–156621\.External Links:[Link](https://doi.org/10.52202/085713-4717)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p3.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.27142#S2.p5.1)\.

\\beginappendix

Table 9:Statistics of the evaluation tasks\. Dev samples define the target skill for data selection, and test samples are used for the reported results\.## 6Implementation Details

We use Qwen3\-1\.7B\([Yang et al\., 2025](https://arxiv.org/html/2610.27142#bib.bib3)\)and set the training budget to 10K, i\.e\., selecting 10K samples for training each task\. Following[Ivison et al\. \(2025\)](https://arxiv.org/html/2610.27142#bib.bib5), we fine\-tune for 2 epochs with a learning rate of 2e\-5, and following[Xia et al\. \(2024\)](https://arxiv.org/html/2610.27142#bib.bib1), we employ LoRA with rank 128\. We set vMF concentrationκ\\kappaby validation \(Section[3](https://arxiv.org/html/2610.27142#S3)\)\.

## 7Dataset Details

We evaluate our method on the following five tasks:

∙\\bulletGSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.27142#bib.bib40)\): grade\-school math word problems, evaluated by exact match of the final answer\.

∙\\bulletMLQA\([Lewis et al\., 2020](https://arxiv.org/html/2610.27142#bib.bib41)\): extractive question answering over passages in seven languages, evaluated by F1 averaged across languages\.

∙\\bulletDROP\([Dua et al\., 2019](https://arxiv.org/html/2610.27142#bib.bib42)\): reading comprehension requiring discrete reasoning \(e\.g\., counting and arithmetic\), also evaluated by F1\.

∙\\bulletXSum\([Narayan et al\., 2018](https://arxiv.org/html/2610.27142#bib.bib43)\): one\-sentence summarization of BBC articles, evaluated byROUGE\-L\.

∙\\bulletOASST1\([Köpf et al\., 2023](https://arxiv.org/html/2610.27142#bib.bib44)\): open\-ended instruction following, evaluated byROUGE\-Lrelative to the reference response\.

Table[9](https://arxiv.org/html/2610.27142#S5.T9)lists the number of dev and test samples for each task\. For tasks without a dedicated dev split, we use samples from the training set as the dev set\. MLQA provides no training set, so we sample 500 dev and 1,000 test samples per language from its validation and test splits, respectively, to keep both sets balanced across languages\. For XSum, we subsample 1,000 dev and 1,000 test samples for efficiency, as summarization requires reading long contexts\. For OASST1, we split 500 dev and 500 test samples from the Tülu 2 pool and remove all OASST1 samples from the selection pool to prevent data leakage\.