Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

# Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks
Source: [https://arxiv.org/html/2607.09739](https://arxiv.org/html/2607.09739)
## Coresets Before Score Sets: Evaluation\-Unsupervised Prompt Subset Selection for LLM Benchmarks

Jihan Yao1,Gantavya Bhatt111footnotemark:1,Arnav Das111footnotemark:1,Peter Jin6,Ke Bao5,Qiaolin Yu5, Khushi Bhardwaj6,Chang Su3,Jialei Wang6,Yikai Zhu5,Sugam Devare6, Damon Mosk\-Aoyama6,Zhen Dong6,Venkat Krishna Srinivasan6,Yineng Zhang4, Oleksii Kuchaiev6,Jiantao Jiao2,6,Banghua Zhu1,5,Jeff Bilmes1

1University of Washington, Seattle2University of California, Berkeley 3Oracle4Together AI5LMSYS6NVIDIA Correspondence:jihany2@cs\.washington\.edu,gbhatt2@uw\.edu,arnavmd2@uw\.edu

###### Abstract

We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite\. In evaluation\-unsupervised benchmark coreset selection \(our approach\), the selection algorithm uses no model evaluation outcomes, and operates on a fine granularity by producing subsets of prompts over multiple benchmarks rather than producing a sub\-collection of entire benchmarks\. We use submodular subset selection, and we develop and evaluate many different submodular functions for this purpose, including determinantal point process \(DPP\) based approaches, submodular mutual information functions, and facility location\-based functions\. On a new large\-scale suite of 35 heterogeneous benchmarks spanning five different capability categories, 18 frontier LLMs, and over 61K prompts, we find that the facility location \(FL\) function operating exclusively on inexpensive semantic prompt embeddings preserves LLM scores better than twelve separate score\-based and diversity\-based baselines, across a range of coreset budgets\. Moreover, we show our proposed objective is not limited to the evaluation\-unsupervised regime: in the setting where only a handful of whole benchmarks must be selected and a large amount of model scores are available, the same objective matches or outperforms state\-of\-the\-art baselines on the MMLU and MTEB leaderboards, while being substantially cheaper to compute\. Together, our results suggest that submodularity, in general, is a strong and reliable tool for benchmark compression\.

Coresets Before Score Sets: Evaluation\-Unsupervised Prompt Subset Selection for LLM Benchmarks

Jihan Yao1††thanks:Equal contribution, random order\., Gantavya Bhatt111footnotemark:1, Arnav Das111footnotemark:1, Peter Jin6, Ke Bao5, Qiaolin Yu5,Khushi Bhardwaj6,Chang Su3,Jialei Wang6,Yikai Zhu5,Sugam Devare6,Damon Mosk\-Aoyama6,Zhen Dong6,Venkat Krishna Srinivasan6,Yineng Zhang4,Oleksii Kuchaiev6,Jiantao Jiao2,6,Banghua Zhu1,5,Jeff Bilmes11University of Washington, Seattle2University of California, Berkeley3Oracle4Together AI5LMSYS6NVIDIACorrespondence:jihany2@cs\.washington\.edu,gbhatt2@uw\.edu,arnavmd2@uw\.edu

## 1Introduction

LLM evaluation is a major but necessary cost in large language model \(LLM\) development\. As models are deployed across increasingly specialized settings, evaluation suites have expanded from a small number of general\-purpose benchmarks to large collections spanning knowledge, mathematics, coding, instruction following, and agentic behavior\(Phanet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib46); Wanget al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib42); Merrillet al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib75); Pyatkinet al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib76); Yaoet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib65)\)\. Typically, LLM evaluation is repeatedly performed throughout the full LLM development cycle as new models are regularly created\. However, repeatedly running evaluations on the full suite of benchmarks is computationally expensive and costly\. Recently, a large body of work has shown that many benchmarks contain substantial redundancy\(DatologyAI Team,[2026](https://arxiv.org/html/2607.09739#bib.bib22); Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73)\), meaning that benchmark evaluation is naturally inefficient, since different parts of different benchmarks might redundantly evaluate the same model ability \(also see Figure[6](https://arxiv.org/html/2607.09739#S4.F6)\)\. This has motivated a growing focus on*benchmark coresets*111We prefer the term “benchmark coresets” or benchmark “subset selection” to avoid overloading the term “compression,” which can also imply bit\-level encoding, entropy, rate–distortion, gzip/JPEG\-style coding, and data distillation, as discussed inBilmes \([2022](https://arxiv.org/html/2607.09739#bib.bib8)\)\.: i\.e\., selecting a much smaller set of examples or benchmarks \(or portions thereof\) that preserves the evaluation results of a full evaluation suite\(Viveket al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib77); Kipniset al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib74)\)\.

In this paper we study LLM benchmark coreset selection: selecting a small subset of benchmark prompts whose induced model scores and rankings approximate those obtained from the complete benchmark suite\. We distinguish between two variants of this problem\.

Inevaluation\-supervised benchmark coreset selection222We consider this a “supervised” rather than an “unsupervised” subset selection procedure, consistent with previous terminologyWeiet al\.\([2014](https://arxiv.org/html/2607.09739#bib.bib7)\)in data subset selection\., the coreset algorithm may use model\-by\-prompt evaluation outcomes, such as previous scores, correctness indicators, responses, or item\-response statistics, as supervision information available for subset selection\. This first case is more informationally rich than the second \(below\), but has two key limitations\. First, it is required that many models have already been evaluated on the full set of benchmarks, and that their responses can be used to guide the benchmark coreset algorithm\(Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73); Viveket al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib77); Kipniset al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib74); Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\. This assumption is reasonable for established benchmarks such as MMLUWanget al\.\([2024](https://arxiv.org/html/2607.09739#bib.bib42)\)\. For a newly released benchmark, however, such responses must be collected from scratch for every model considered\. As the ecosystem of benchmarks grows rapidly \(as it should\), benchmark compression strategies that rely heavily on these responses can become prohibitively expensive\. The second limitation of the supervised approach is the experimental scope: prior methods are typically evaluated on old benchmarks with open leaderboards or large historical model responses, such as Open LLM Leaderboard and MMLU\(Myrzakhanet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib125); Hendryckset al\.,[2020](https://arxiv.org/html/2607.09739#bib.bib43)\)\. However, recent work has shown that these benchmarks suffer from serious data contamination and benchmark\-specific overfitting risks\(Denget al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib78); Xuet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib79)\)\. As a result, conclusions drawn from these benchmarks may carry biases from data contamination, and not serve as true guidance for new benchmark development\. This narrow experimental scope also limits how fidelity is measured\. Since prior studies are often conducted on single or only a few benchmarks, they mainly evaluate whether coresets preserves a single benchmark score or an overall average score\. Such aggregated fidelity can miss capability\-wise failures: errors from different capabilities may cancel each other out, making a compressed subset appear faithful even when it distorts performance on specific capabilities\. Therefore, extensive experiments on recent, heterogeneous benchmark suite spanning multiple capabilities are necessary\.

Inevaluation\-unsupervised benchmark coreset selection, by contrast, \(the approach we introduce in this paper\), the selection algorithm uses no previous model evaluation outcomes and uses information only from the prompts themselves\. Indeed, we hypothesize that a useful benchmark coreset method should operate in an*evaluation\-unsupervised*regime: selecting a subset from a target benchmark should only utilize dataset\-level observable information, such as text, metadata, and optional capability labels, but with no, or at most very limited, access to model responses or evaluation scores\.

Another distinct aspect of our approach is granularity: we produce coresets as fine\-grained subsets of prompts rather than the finding subsets of a set of benchmarks\. This allows us to extract non\-redundant fractional portions of multiple benchmarks to produce a highly tuned benchmark coreset that well represents all of the original benchmarks better than the more coarse\-grained approach of selecting a subset of entire benchmarks\.

Overall, in this paper, we study methods for the production of*evaluation\-unsupervised benchmark coresets*at the fine granularity of prompts\. We evaluate a number of different strategies for this, many under the submodularBilmes \([2022](https://arxiv.org/html/2607.09739#bib.bib8)\)umbrella, a class of techniques that has been successfully applied to many data subset selection problems in the past, including document summarization, sensor placement, and active learning\. In particular, we evaluate many different submodular functions for this purpose, including determinantal point process \(DPP\) based approachesKulesza and Taskar \([2012](https://arxiv.org/html/2607.09739#bib.bib85)\), submodular mutual information functionsBilmes \([2022](https://arxiv.org/html/2607.09739#bib.bib8)\); Iyeret al\.\([2021b](https://arxiv.org/html/2607.09739#bib.bib6),[a](https://arxiv.org/html/2607.09739#bib.bib5)\), and facility location\-based functionsBilmes \([2022](https://arxiv.org/html/2607.09739#bib.bib8)\)\. Our methods first construct a model\-independent semantic representation for each prompt in each benchmark by concatenating the prompt and ground\-truth fields, such as answers, constraints, and patches, and then embedding the concatenated text\. In the DPP case, the method uses these features directly, while in the facility location case, it builds a sparse similarity matrix over items\. In both cases, it maximizes a submodular objective so that the selected subset covers the full evaluation suite without requiring historical scores or responses\.

As part of this work, and to help with evaluations, we introduce a new large\-scale score\-evaluation suite of 35 heterogeneous benchmarks spanning five capability categories, 18 frontier LLMs, \(including Claude–4\.5 Sonnet, GPT–5, Gemini–2\.5 Pro\), and over 61K newly collected prompts\. Unlike previous work, which mostly experiments on legacy benchmarks, our study examines coreset selection algorithms on a more generalizable and up\-to\-date scenario\.

Across budgetsk∈\[70,200\]k\\in\[70,200\], facility location \(FL\) achieves the best overall and capability\-wise fidelity, with the lowest normalized overall mean relative error \(0\.008\) and near\-zero normalized capability mean relative error, outperforming twelve score\-based and diversity\-based baselines\. Our analysis shows that this advantage comes from coverage: score\-based methods tend to select samples that are concentrated within a single benchmark, while IRT\-based embeddings learned from only 18 models are unstable\. In contrast, semantic embeddings do not require model responses or evaluation scores and provide a more reliable basis for selection\.

We further find that the same FL objective remains effective beyond the evaluation\-unsupervised regime\. We evaluate how well FL can select a benchmark\-level coreset that recovers held\-out models’ full scores on the whole leaderboard when rich model evaluation scores are available\. The results show that FL is the strongest method across most budgets\. In this setting, where only a handful of whole benchmarks must be selected and many model scores are available, the same objective matches or outperforms state\-of\-the\-art baselines on the MMLU and MTEB leaderboards, while being substantially cheaper to compute\. Together, our results suggest that submodularity is a strong and reliable tool for benchmark coresets\.

## 2Related Work

#### Benchmark Coresets\.

Given the rapid growth in LLM benchmark datasets, a growing body of work has proposed various methods for reducing benchmarks to accelerate evaluation times\. These methods generally select a subset of samples or subsets of benchmarks\. At the sample level, existing methods mainly rely on observed model behavior: item\-response theory\(Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73)\), confidence\-based anchor examples\(Viveket al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib77)\), model response matrices\(Kipniset al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib74); Huanget al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib1)\), hidden\-state representations\(Zhanget al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib4)\), or a small set of anchor LLMs combined with intrinsic sample features\(Saranathanet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib90)\)\. At the dataset level, existing methods select benchmark subsets using raw evaluation scores\(Subramaniet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib24)\), task\-similarity measures from model response patterns\(Surkov and Yamshchikov,[2024](https://arxiv.org/html/2607.09739#bib.bib127)\), or covariance, imputation, and submodular objectives over benchmark scores\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\. Despite these differences, the majority of existing methods depend on model response or scores from aggregated public leaderboards, which are expensive to obtain and often sparse since not all leaderboards evaluate the same sets of models, limiting their applicability in new or private evaluation settings\. In contrast, our method operates at the sample level in an unsupervised setting, selecting samples using only semantic embeddings and requiring very limited model evaluations, using them solely for calibration and evaluation purposes\.

#### Coreset Selection\.

Benchmark subset selection can largely be viewed as an application of coreset selection, which often has been studied in the context of curating training datasets\. Typically, methods strive to select samples that are informative\(Paulet al\.,[2021](https://arxiv.org/html/2607.09739#bib.bib34)\), diverse\(Abbaset al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib35); Sener and Savarese,[2018](https://arxiv.org/html/2607.09739#bib.bib28); Ashet al\.,[2019](https://arxiv.org/html/2607.09739#bib.bib29); Bhattet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib27)\), aligned with a target\(Daset al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib26); Kothawadeet al\.,[2021](https://arxiv.org/html/2607.09739#bib.bib36)\), or some combination of these objectives\(Bukharin and Zhao,[2024](https://arxiv.org/html/2607.09739#bib.bib30); Daset al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib26); Maharanaet al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib33); Ashet al\.,[2019](https://arxiv.org/html/2607.09739#bib.bib29)\)\. Methods can utilize signals from the model as it is being trained as in active learning\(Mirzasoleimanet al\.,[2019](https://arxiv.org/html/2607.09739#bib.bib31); Sener and Savarese,[2018](https://arxiv.org/html/2607.09739#bib.bib28); Ashet al\.,[2019](https://arxiv.org/html/2607.09739#bib.bib29); Paulet al\.,[2021](https://arxiv.org/html/2607.09739#bib.bib34); Kothawadeet al\.,[2021](https://arxiv.org/html/2607.09739#bib.bib36); Killamsettyet al\.,[2020](https://arxiv.org/html/2607.09739#bib.bib32)\), or can use signals from an extrinsic model \(such as embeddings\)\(Abbaset al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib35); Bhattet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib27); Bukharin and Zhao,[2024](https://arxiv.org/html/2607.09739#bib.bib30); Daset al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib26)\)\. Submodularity has been used for data subset selection sinceLin and Bilmes \([2009](https://arxiv.org/html/2607.09739#bib.bib2)\)\. Many of these methods can be applied directly to benchmark compression, particularly those that rely on extrinsic signals\. In the setting of benchmark reduction, such signals can take the form of semantic embeddings or model evaluation scores\. We find that semantic embeddings are generally more effective despite being significantly cheaper to acquire\.

## 3Methodology

### 3\.1Facility Location \(FL\)

#### Problem Setup\.

The evaluation\-unsupervised setting introduced above can be formalized as follows\. Letℬ\\mathcal\{B\}be the set of benchmarks,VVbe the all benchmark samples \(e\.g\., prompts plus metadata\) and𝒞\\mathcal\{C\}the set of capability categories\. Each itemi∈Vi\\in Vbelongs to exactly one benchmarkb​\(i\)∈ℬb\(i\)\\in\\mathcal\{B\}, and each benchmark belongs to exactly one capability categoryc∈𝒞c\\in\\mathcal\{C\}\. We denote byℬc⊆ℬ\\mathcal\{B\}\_\{c\}\\subseteq\\mathcal\{B\}the subset of benchmarks belonging to capabilitycc, so that\{ℬc\}c∈𝒞\\\{\\mathcal\{B\}\_\{c\}\\\}\_\{c\\in\\mathcal\{C\}\}partitionsℬ\\mathcal\{B\}\. Given a budgetk≪nk\\ll n, the selector must choose a coresetA⊆VA\\subseteq Vwith\|A\|=k\|A\|=kwithout model\-related information\. Thus the selection may only depend on benchmark sample information, represented by an embeddingeie\_\{i\}for eachii, and the induced nonnegative similarity scorewi​j≥0w\_\{ij\}\\geq 0between any two itemsiiandjj\.

#### Objective\.

Givenwi​jw\_\{ij\}, a nonnegative similarity score between embeddingseie\_\{i\}andeje\_\{j\}, the facility location functionffmeasures how well a subsetAArepresentsVV:

f​\(A\)=∑i∈Vmaxj∈A⁡wi​jf\(A\)=\\sum\_\{i\\in V\}\\max\_\{j\\in A\}\\,w\_\{ij\}\(1\)In Equation[1](https://arxiv.org/html/2607.09739#S3.E1), each elementi∈Vi\\in V\(a “client”\) is represented by its most similar elementj∈Aj\\in A\(its “facility”\), i\.e\., thej∈Aj\\in Amaximizingwi​jw\_\{ij\}\. In our task, we seek to select a subset that maximizesff:

A∗=argmaxA⊆V\|A\|=kf​\(A\)A^\{\*\}=\\operatorname\*\{argmax\}\_\{\\begin\{subarray\}\{c\}A\\subseteq V\\\\ \|A\|=k\\end\{subarray\}\}\\,f\(A\)\(2\)
Maximizingffincentivizes selecting subsetsAAwhose elements collectively cover all ofVVwell\. Sinceffis a well\-known submodular function, a simple greedy algorithm that iteratively adds the element with the largest marginal gain achieves a\(1−1/e\)\(1\-1/e\)approximation guarantee\(Nemhauseret al\.,[1978](https://arxiv.org/html/2607.09739#bib.bib72)\)\. We provide a primer on submodular functions in the appendix\. All experiments were performed usingBilmes \([2026](https://arxiv.org/html/2607.09739#bib.bib128)\)\.

#### Semantic Embeddings\.

For eachi∈Vi\\in V, we build a text representation by concatenating \(i\) the problem prompt, and \(2\) ground\-truth and evaluation metadata fields\. The second part is benchmark\-specific: for instruction\-following benchmarks it contains the executable instruction constraints; for knowledge benchmarks it contains the canonical answer text; for math benchmarks it contains the final answer; for coding benchmarks it contains oracle tests or solutions; and for agentic benchmarks it contains tool schema, task state, and expected actions\. Appendix[B](https://arxiv.org/html/2607.09739#A2)lists the source fields used for each benchmark\. We serialize structured fields instead of using plain text so that fine\-grained format differences remain visible in the similarities\. We embed each concatenated string withQwen3\-Embedding\-4Bto getei=LLM​\(texti\)e\_\{i\}=\\mathrm\{LLM\}\(\\mathrm\{text\}\_\{i\}\)\. We left\-pad and truncate to at most 8192 tokens\.

#### Similarity Matrix Construction\.

Choosing the right similarity matrix is a critical design choice for facility location to be effective\. We define the similarity between samplesiiandjjas the \(clipped\) cosine similarity of their embeddings,wi​j=max⁡\(0,cos⁡\(ei,ej\)\)w\_\{ij\}=\\max\\\!\\left\(0,\\cos\(e\_\{i\},e\_\{j\}\)\\right\)\. Cosine similarity matches the training objective of embedding models, and the clipping enforceswi​j≥0w\_\{ij\}\\geq 0which ensures thatffis submodular\.

With a dense similarity matrix, however,ffsaturates quickly\. Specifically, a single well\-placed facility attains nonzero similarity to most ofVV, so the first few selections capture nearly all of the attainable objective value and the marginal gains of subsequent elements collapse by several orders of magnitude\. Beyond saturation, the dense matrix requiresO​\(n2\)O\(n^\{2\}\)memory, which can be substantially expensive for large prompt sets\. To ameliorate both issues, we sparsify the matrix, retaining only the top\-ttentries of each row, witht=⌈α​n/k⌉t=\\lceil\\alpha n/k\\rceiland all other entries set to zero as is standard inBilmes \([2026](https://arxiv.org/html/2607.09739#bib.bib128)\)\. Sparsification also reduces memory fromO​\(n2\)O\(n^\{2\}\)toO​\(α​n2/k\)O\(\\alpha n^\{2\}/k\), allowing us to scale to much larger datasets\.

The parameterα\\alphainterpolates between two failure modes, both visible in Figure[7](https://arxiv.org/html/2607.09739#A3.F7), which shows the marginal gains of the greedy selection across sparsification levels\. Whenα\\alphais too small, each selected facility covers only a handful of clients and these coverage sets rarely overlap, so every candidate yields a nearly identical gain and the selection order becomes largely arbitrary\. Whenα\\alphais too large, the kernel approaches density and saturation reappears, with gains collapsing within the first few dozen selections\. We therefore treatα\\alphaas a hyperparameter and tune it on a validation set, as described in Section[3\.2](https://arxiv.org/html/2607.09739#S3.SS2)\.

### 3\.2Evaluation

We evaluate coreset fidelity by how faithfully the selected subset predicts per\-model benchmark scores relative to the full evaluation suite\.

#### Prediction\.

LetAb=\{i∈A:b​\(i\)=b\}A\_\{b\}=\\\{i\\in A:b\(i\)=b\\\}be the selected items from benchmarkbb\. The predicted score of modelmmon benchmarkbbiss^mb​\(A\)=\|Ab\|−1​∑i∈Abyim\\hat\{s\}^\{b\}\_\{m\}\(A\)=\|A\_\{b\}\|^\{\-1\}\\sum\_\{i\\in A\_\{b\}\}y^\{m\}\_\{i\}, whereyimy^\{m\}\_\{i\}is the score of modelmmon itemii; benchmarks withAb=∅A\_\{b\}=\\emptysetare excluded from all averages\. The overall predicted score aggregates across covered benchmarks:

s^m​\(A\)=1\|ℬ​\(A\)\|​∑b∈ℬ​\(A\)s^mb​\(A\),\\hat\{s\}\_\{m\}\(A\)\\;=\\;\\frac\{1\}\{\|\\mathcal\{B\}\(A\)\|\}\\sum\_\{b\\in\\mathcal\{B\}\(A\)\}\\hat\{s\}^\{b\}\_\{m\}\(A\),\(3\)whereℬ​\(A\)=\{b∈ℬ:Ab≠∅\}\\mathcal\{B\}\(A\)=\\\{b\\in\\mathcal\{B\}:A\_\{b\}\\neq\\emptyset\\\}is the set of covered benchmarks\. The ground\-truthsms\_\{m\}is the same average computed over all items\. For capabilitycc\(one of\|𝒞\|=5\|\\mathcal\{C\}\|\{=\}5categories\), the predicted capability scores^mc​\(A\)\\hat\{s\}^\{c\}\_\{m\}\(A\)is the mean ofs^mb​\(A\)\\hat\{s\}^\{b\}\_\{m\}\(A\)over benchmarksb∈ℬcb\\in\\mathcal\{B\}\_\{c\}that are covered; the ground\-truthsmcs^\{c\}\_\{m\}is defined analogously over all items\.

#### Metrics\.

Our two primary metrics are*mean capability relative error*\(Cap\-MRE\) and*overall mean relative error*\(MRE\):

Cap​\-​MRE​\(A\)\\displaystyle\\mathrm\{Cap\\text\{\-\}MRE\}\(A\)=1\|𝒞\|​∑c∈𝒞1M​∑m=1M\|s^mc​\(A\)−smc\|smc,\\displaystyle\\;=\\;\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\frac\{\\bigl\|\\hat\{s\}^\{c\}\_\{m\}\(A\)\-s^\{c\}\_\{m\}\\bigr\|\}\{s^\{c\}\_\{m\}\},\(4\)MRE​\(A\)\\displaystyle\\mathrm\{MRE\}\(A\)=1M​∑m=1M\|s^m​\(A\)−sm\|sm\.\\displaystyle\\;=\\;\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\frac\{\\bigl\|\\hat\{s\}\_\{m\}\(A\)\-s\_\{m\}\\bigr\|\}\{s\_\{m\}\}\.\(5\)Cap\-MRE measures per\-capability score fidelity averaged across all categories; MRE is the average of the overall benchmark fidelity\. Since the overall score is itself an average over capabilities, the two metrics aggregate the same per\-capability errors very differently\. MRE sums the signed deviations before taking a magnitude, while Cap\-MRE takes the magnitude within each capability first\. The overall error can be smaller, so a subset that over\-represents some capabilities and under\-represents others lets these signed errors cancel, making MRE look near\-perfect even when every capability is badly estimated\. Therefore, it is better to look at both Cap\-MRE and MRE for deciding on the subset\.

## 4Experiments

### 4\.1Experiment Setup

To overcome the scale limitation of previous work, we newly sample and evaluate models responses on over 61,498 prompts from 35 recently developed highly\-valued benchmarks, covering five capability categories: agentic tool use, coding, instruction following, knowledge, and mathematics\. We benchmark 18 frontier LLMs drawn from seven different model families or providers \(Anthropic/Claude, DeepSeek, Google/Gemini, Moonshot/Kimi, OpenAI/GPT, Qwen, and xAI/Grok\)\. We will release the large scale benchmark data for future research\. The detailed benchmarks and models we use are presented in Appendix[B](https://arxiv.org/html/2607.09739#A2)\.

In total, we collect 1,106,964 entries in the \(model, sample\) matrix\. Due to checkpoint unavailability and other force majeure factors, we end up having 47,521 missing entries \(4\.29%\)\. We impute missing entries with NeuMF\(Heet al\.,[2017](https://arxiv.org/html/2607.09739#bib.bib91)\), an item\-response theory \(IRT\) learning model, trained only on observed scores, with logit

zim=b0\+bi\+cm\+ϕi⊤​ψm\+MLP​\(\[ϕi,ψm,ϕi⊙ψm\]\),\\begin\{split\}z\_\{i\}^\{m\}=\{\}&b\_\{0\}\+b\_\{i\}\+c\_\{m\}\+\\phi\_\{i\}^\{\\top\}\\psi\_\{m\}\\\\ &\+\\,\\mathrm\{MLP\}\(\[\\phi\_\{i\},\\psi\_\{m\},\\phi\_\{i\}\\odot\\psi\_\{m\}\]\),\\end\{split\}\(6\)and binary cross\-entropy training objectiveℒNeuMF=∑\(i,m\)∈ΩBCE​\(yim,σ​\(zim\)\)\\mathcal\{L\}\_\{\\mathrm\{NeuMF\}\}=\\sum\_\{\(i,m\)\\in\\Omega\}\\mathrm\{BCE\}\(y\_\{i\}^\{m\},\\sigma\(z\_\{i\}^\{m\}\)\)\.Ω\\Omegais the set of observed model\-sample pairs,yimy\_\{i\}^\{m\}is the observed score,b0b\_\{0\}is global bias,bib\_\{i\}is sample bias,cmc\_\{m\}is model bias, andϕi,ψm\\phi\_\{i\},\\psi\_\{m\}are the learned sample/model embeddings\. The training process also yields sample/model embeddings that can be used for the IRT\-based benchmark compression algorithm baseline\(Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73)\)\. We will refer them as IRT embedding in the following sections\.

We use a 3\-fold cross\-validation over theM=18M\{=\}18models, partitioning them into three equal folds of six\. In each round, two folds serve as the calibration split and the remaining fold as the held\-out test split\. Missing entries for the 12 models are replaced by NeuMF predictions\. The models are used to tune the hyperparameters of the coreset selection algorithm, e\.g\., theα\\alphaparameter for the similarity matrix that is used by FL\. Held\-out models do not have learned model embeddings, so any remaining missing entries for held\-out models are filled with that model’s observed mean only when constructing the fold’s complete evaluation matrix\. We use the average Cap\-MRE on the calibration split for hyperparameter tuning, and report test split performance averaged over the three folds\.

### 4\.2Baselines

We categorize the benchmark compression algorithm baselines into two families: score\-based and diversity\-based methods\.

#### Score\-Based Methods\.

These methods rank samples by a scalar score and select the top\-kkwith no coupling between selected items\.

- •Hardness \(IRT\)\.fits a one\-parameter \(1\-PL\) item\-response theory model to the observed model\-response matrix, obtaining a difficulty parameterβi\\beta\_\{i\}per prompt\(Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73)\), and selects thekkmost difficult prompts\.
- •Point\-biserial correlation \(PBSC\)\.\(Joshiet al\.,[2026](https://arxiv.org/html/2607.09739#bib.bib11)\)computes the pointwise\-biserial correlationrir\_\{i\}between item scores\{yim\}m=1M\\\{y^\{m\}\_\{i\}\\\}\_\{m=1\}^\{M\}and aggregate model scores\{sm\}m=1M\\\{s\_\{m\}\\\}\_\{m=1\}^\{M\}, and selects thekkprompts with the highest correlation\.

Formally, for everyA⊆VA\\subseteq Vanda∈Aa\\in Aletma∈ℝm\_\{a\}\\in\\mathbb\{R\}be a score obtained using either of the approaches, thenf​\(A\)=∑a∈Amaf\(A\)=\\sum\_\{a\\in A\}m\_\{a\}\.

#### Diversity\-based Methods\.

These methods explicitly model interactions between candidate items to promote coverage\.

- •Determinantal Point Processes \(DPP\)\.A DPP assigns probability proportional todet\(KA\)\\det\(K\_\{A\}\)for everyA⊆VA\\subseteq V, whereKKis a positive\-semidefinite kernel matrix\(Kulesza and Taskar,[2012](https://arxiv.org/html/2607.09739#bib.bib85)\)\. We use a greedy algorithm that iteratively adds the item with the highest marginal log\-determinant gain, that is, MAP inference\. Following our method, all DPP variants use sample text embeddingseie\_\{i\}\. We consider four kernels: acovariance kernel\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)Ki​j=e~i⊤​e~jK\_\{ij\}=\\tilde\{e\}\_\{i\}^\{\\top\}\\tilde\{e\}\_\{j\}with mean\-centered embeddingse~i=ei−e¯\\tilde\{e\}\_\{i\}=e\_\{i\}\-\\bar\{e\}\(DPP\-Cov\); acosine kernelKi​j=cos⁡\(ei,ej\)K\_\{ij\}=\\cos\(e\_\{i\},e\_\{j\}\)\(DPP\-Cos\); and adot\-product kernelKi​j=ei⊤​ejK\_\{ij\}=e\_\{i\}^\{\\top\}e\_\{j\}\(DPP\-Dot\)\. For either of the cases, the objective takes the form wheref​\(A\)=log⁡\(det\(KA\)\)f\(A\)=\\log\(\\det\(K\_\{A\}\)\)\. Inspired byKrauseet al\.\([2008](https://arxiv.org/html/2607.09739#bib.bib10)\); Smola \([2026](https://arxiv.org/html/2607.09739#bib.bib18)\); Iyeret al\.\([2021a](https://arxiv.org/html/2607.09739#bib.bib5)\), we also consider a mutual information\-based variant, which isf^​\(A\)=If​\(A;V∖A\)\\hat\{f\}\(A\)=I\_\{f\}\(A;V\\setminus A\), whereIf​\(A;B\)=f​\(A\)\+f​\(B\)−f​\(A∪B\)I\_\{f\}\(A;B\)=f\(A\)\+f\(B\)\-f\(A\\cup B\)\. For our DPP setup, this isf^​\(A\)=log⁡\(detKA\)\+log⁡\(detKV∖A\)−log⁡\(detKV\)\\hat\{f\}\(A\)=\\log\(\\det K\_\{A\}\)\+\\log\(\\det K\_\{V\\setminus A\}\)\-\\log\(\\det K\_\{V\}\), which is an instance of non\-monotone submodular function\. For the Covariance, Cosine, and Dot Product kernels, this yields three additional baselines:DPP\-Cov\-MI,DPP\-Cos\-MI, andDPP\-Dot\-MI, with “MI" referring to mutual information\.
- •kk\-Center\.The greedykk\-Center algorithm iteratively selects the sample farthest from all currently selected items, minimizing the maximum distance from any sample to its nearest selected center\(Sener and Savarese,[2018](https://arxiv.org/html/2607.09739#bib.bib28)\)\.333Please note that this objective is, in fact, a minmax facility location function\.To optimize the above NP\-hard object, we follow the proposed greedy methods, which provide a 2\-multiplicative approximation to the optimal selection\(Cooket al\.,[2009](https://arxiv.org/html/2607.09739#bib.bib9)\)\.
- •nn\-kkmeans Clustering\.We run thekk\-means clustering algorithm and assign each cluster centroid’s nearest sample as its representative\. For bothkk\-Center and Clustering, to remain consistent with TinyBenchmarks\(Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73)\), we evaluate two embedding variants: \(i\)text embeddingseie\_\{i\}fromQwen3\-Embedding\-4B, identical to those used by facility location and DPP; and \(ii\)IRT embeddingsϕi\\phi\_\{i\}, the prompt latent vectors learned by our NeuMF model \(§[3\.2](https://arxiv.org/html/2607.09739#S3.SS2)\), which encode the observed per\-prompt response pattern across the training models\.

![Refer to caption](https://arxiv.org/html/2607.09739v1/x1.png)Figure 1:The lowerxx\-axis reports mean normalized MRE and mean normalized Cap\-MRE, where scores are linearly normalized to\[0,1\]\[0,1\]within each budget and then averaged across all budgetsk∈\[70,200\]k\\in\[70,200\]\. The upperxx\-axis reports the SE\-aware mean average rank jointly over MRE and Cap\-MRE, where a method is ranked above another only when their error intervals do not overlap\.Lower values indicate better performance across all metrics\.\. Methods are sorted top\-to\-bottom by MAR\. The dashed vertical line marks the change in the bottomxx\-axis scale used to include the high\-error baselines without compressing the remaining methods\. Colors indicate method family\.

### 4\.3Main Results

#### FL almost always performs best across all budgets\.

Tables[1](https://arxiv.org/html/2607.09739#S4.T1)report MRE and Cap\-MRE \(mean±\\pmSE across folds\) at four representative budgets:k∈\{80,120,160,200\}k\\in\\\{80,120,160,200\\\}\. Full per\-budget plots across the entire range in increments of 5,k∈\[70,200\]k\\in\[70,200\], are provided in Appendix[C\.1](https://arxiv.org/html/2607.09739#A3.SS1)\(Figures[8](https://arxiv.org/html/2607.09739#A3.F8)and[9](https://arxiv.org/html/2607.09739#A3.F9)\)\. Moreover, Figure[1](https://arxiv.org/html/2607.09739#S4.F1)summarize performance across all coreset budgets up tok=200k=200with the mean normalized score for each method on MRE and Cap\-MRE, respectively\. We generally see a trend that FL almost always performs best across all budgets\. FL achieves the lowest normalized MRE \(0\.03\) and a near\-zero normalized Cap\-MRE, beating all baselines even those using model response information\.

Figure[1](https://arxiv.org/html/2607.09739#S4.F1)also reports the SE\-aware mean average rank \(MAR\) jointly over MRE and Cap\-MRE for all methods\. FL achieves a MAR of 1\.1, close to the theoretical minimum of 1, indicating near\-uniform dominance across budgets and metrics\. DPP\-Cov is the strongest competitor with a MAR of 1\.8, followed by Clustering at 2\.4\. We also observe that other variants of DPP underperform, which could be attributed to their tendency to rely heavily on a single benchmark rather than selection across the benchmark\. Score\-based baselines occupy the bottom two positions \(MAR 11\.3 and 13\.0\)\. This is unsurprising: without an explicit diversity objective, they tend to concentrate selections within a single benchmark, failing to achieve the broad coverage needed for low relative error\.

Table 1:Cap\-MRE and MRE at key budgetsk∈\{80,120,160,200\}k\\in\\\{80,120,160,200\\\};the lower the better\.Boldindicates the best result, andunderlineindicates the second\-best\.Blue backgroundrepresents evaluation\-unsupervised algorithms\. FL almost always performs best across all budgets\.
#### Possible Pitfalls of IRT based embeddings\.

The method proposed in tinyBenchmarks\(Poloet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib73)\)learns prompt embeddings from IRT models: each prompt is assigned a latent difficulty parameter estimated from how models respond to it, and these parameters are used to cluster prompts into representative groups\. The quality of these embeddings, therefore, depends critically on having enough models to reliably estimate the latent item parameters\. In our setting, we have onlyM=18M\{=\}18models shared across all 35 benchmarks making the IRT parameter estimates possibly noisy\. Thus the potential instability of IRT embeddings trained on small models set may explain their failure\.

To demonstrate this concretely, we measure how much the IRT\-induced similarity structure changes across the three model folds\. For each benchmark, we compute the Spearman correlationρ\\rhobetween the pairwise IRT similarity matrices produced by each pair of folds \(three pairs total\) and report the mean and standard error across pairs\. As shown in Table[2](https://arxiv.org/html/2607.09739#A3.T2),ρ\\rhoranges from0\.300\.30\(Nexus Bench\) to0\.920\.92\(Follow Bench\), with many benchmarks below0\.70\.7\. Semantic embeddings are computed from sample text alone and are therefore identical regardless of which models are available, making them more reliable for facility location\. This indicates IRT’s high dependency on large amount of model responses, which is infeasible to collect in reality\.

#### Text vs\. IRT embeddings on facility location\.

Figure[2](https://arxiv.org/html/2607.09739#S4.F2)directly compares FL using text embeddings against FL using IRT embeddings, alongside random selection as a reference\. On MRE, FL \(text\) outperforms FL \(IRT\) substantially across all budgets, with the gap most pronounced at small budgets \(k≤120k\\leq 120\): atk=80k\{=\}80, FL \(text\) achieves0\.050±0\.0150\.050\\pm 0\.015versus0\.141±0\.0220\.141\\pm 0\.022for FL \(IRT\)\. Crucially, the SE bands for FL \(IRT\) on MRE are consistently wider, reflecting the instability in IRT\-derived embeddings: because latent item parameters vary across folds, FL selects different subsets each time, producing high variance in test error\. On Cap\-MRE the two methods converge at larger budgets – FL \(IRT\) reaches0\.107±0\.0130\.107\\pm 0\.013atk=200k\{=\}200versus0\.113±0\.0190\.113\\pm 0\.019for FL \(text\), but the SE bands overlap substantially throughout, and FL \(IRT\) requires access to model response data, making it inapplicable in the evaluation\-unsupervised setting\. Taken together, text embeddings provide a more stable and practically accessible basis for FL without sacrificing capability\-level coverage\.

![Refer to caption](https://arxiv.org/html/2607.09739v1/x2.png)Figure 2:FL with text embeddings vs\. FL with IRT embeddings vs\. random, across all budgetsk∈\[70,200\]k\\in\[70,200\];the lower the better\. Shaded bands show standard error across folds\.
#### Effect of block\-diagonal similarity structure\.

In our FL setup, the pairwise similarity matrix is constrained to be block\-diagonal according to benchmark identity: prompts from different benchmarks have zero similarity, preventing the greedy marginal\-gain computation from trading coverage across benchmarks\. This benchmark stratification is a deliberate design choice to prevent the coreset from concentrating on a single benchmark\. That said, it is not a necessary condition for facility location to work: a well\-tuned FL objective can still select a diverse set of examples even under severe cross\-benchmark size imbalance\. Figure[3](https://arxiv.org/html/2607.09739#S4.F3)compares FL \(ours, with stratification\) against FL without stratification, and confirms that removing the constraint does not catastrophically degrade performance — both variants comfortably outperform random selection across the full budget range\. As future work, one could leverage the unstratified FL solution to quantify benchmark similarity: the overlap in facility coverage between two benchmarks, when the similarity matrix spans all prompts jointly, provides a natural proxy for the mutual information between two benchmarks\.

![Refer to caption](https://arxiv.org/html/2607.09739v1/x3.png)Figure 3:FL with block\-diagonal similarity \(ours\) vs\. FL without benchmark stratification \(no stratification\) vs\. random, across all budgetsk∈\[70,200\]k\\in\[70,200\];the lower the better\. Shaded bands show standard error across folds\. Both the FL variants outperform random\.

### 4\.4Benchmark Level Subset Selection

A separate line of work studies submodular selection not over samples within a benchmark, but over entire*benchmarks*within a leaderboard\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\.

#### Setting\.

Given a score matrix𝒮\\mathcal\{S\}ofMMmodels×\\timesNNbenchmarks, the goal is to select a small setAAofkkbenchmarks, such that an unseen model’s scores on the unselected complementA¯=V∖A\\bar\{A\}=V\\setminus Acan be reconstructed from its scores onAAalone\. That work estimates a benchmark\-level correlation matrixΣ^\\hat\{\\Sigma\}and greedily selectsAAto maximize either differential entropy or mutual information with the complement, which is precisely the case where one uses a DPP with the covariance matrix as a kernel\.

Partitioning the estimated parameters alongAAandA¯\\bar\{A\},

μ^=\[μ^Aμ^A¯\],Σ^=\[Σ^A​AΣ^A​A¯Σ^A¯​AΣ^A¯​A¯\],\\hat\{\\mu\}=\\begin\{bmatrix\}\\hat\{\\mu\}\_\{A\}\\\\\[2\.0pt\] \\hat\{\\mu\}\_\{\\bar\{A\}\}\\end\{bmatrix\},\\qquad\\hat\{\\Sigma\}=\\begin\{bmatrix\}\\hat\{\\Sigma\}\_\{AA\}&\\hat\{\\Sigma\}\_\{A\\bar\{A\}\}\\\\\[2\.0pt\] \\hat\{\\Sigma\}\_\{\\bar\{A\}A\}&\\hat\{\\Sigma\}\_\{\\bar\{A\}\\bar\{A\}\}\\end\{bmatrix\},\(7\)a test modeliiis evaluated only on the selected benchmarks, and its unselected scores are imputed by the Gaussian conditional mean

B^i​A¯=μ^A¯\+Σ^A¯​A​Σ^A​A−1​\(Bi​A−μ^A\),\\hat\{B\}\_\{i\\bar\{A\}\}=\\hat\{\\mu\}\_\{\\bar\{A\}\}\+\\hat\{\\Sigma\}\_\{\\bar\{A\}A\}\\,\\hat\{\\Sigma\}\_\{AA\}^\{\-1\}\\bigl\(B\_\{iA\}\-\\hat\{\\mu\}\_\{A\}\\bigr\),\(8\)We evaluate each method with two complementary metrics: testR2R^\{2\}, which measures how well an unseen model’s held\-out scores are recovered, and the residual variance fraction, which measures how much benchmark variance remains unexplained after conditioning onAA\.

We report testR2R^\{2\}in standardized space, so the constant predictorB^i​A¯=μ^A¯\\hat\{B\}\_\{i\\bar\{A\}\}=\\hat\{\\mu\}\_\{\\bar\{A\}\}attainsR2=0R^\{2\}=0:

R2=1−∑i∑j∈A¯\(B^i​j−Bi​j\)2∑i∑j∈A¯\(Bi​j−x¯j\)2,R^\{2\}=1\-\\frac\{\\sum\_\{i\}\\sum\_\{j\\in\\bar\{A\}\}\\bigl\(\\hat\{B\}\_\{ij\}\-B\_\{ij\}\\bigr\)^\{2\}\}\{\\sum\_\{i\}\\sum\_\{j\\in\\bar\{A\}\}\\bigl\(B\_\{ij\}\-\\bar\{x\}\_\{j\}\\bigr\)^\{2\}\},\(9\)wherex¯j\\bar\{x\}\_\{j\}is the training\-set mean of benchmarkjj\. A highR2R^\{2\}at smallkkimplies that thekkselected benchmarks suffice to recover the full leaderboard profile of an unseen model\. The residual variance fraction, in contrast, is a model\-independent quantity derived directly from the conditional covariance ofA¯\\bar\{A\}givenAA, i\.e\. the Schur complement

Σ^A¯∣A=Σ^A¯​A¯−Σ^A¯​A​Σ^A​A−1​Σ^A​A¯\.\\hat\{\\Sigma\}\_\{\\bar\{A\}\\mid A\}=\\hat\{\\Sigma\}\_\{\\bar\{A\}\\bar\{A\}\}\-\\hat\{\\Sigma\}\_\{\\bar\{A\}A\}\\,\\hat\{\\Sigma\}\_\{AA\}^\{\-1\}\\hat\{\\Sigma\}\_\{A\\bar\{A\}\}\.\(10\)
RV​\(A\)=tr⁡\(Σ^A¯∣A\)tr⁡\(Σ^\)\.\\mathrm\{RV\}\(A\)=\\frac\{\\operatorname\{tr\}\\\!\\bigl\(\\hat\{\\Sigma\}\_\{\\bar\{A\}\\mid A\}\\bigr\)\}\{\\operatorname\{tr\}\\\!\\bigl\(\\hat\{\\Sigma\}\\bigr\)\}\.\(11\)Smaller values indicate that the selected benchmarks explain more of the total leaderboard variance, and1−RV​\(A\)1\-\\mathrm\{RV\}\(A\)is the corresponding explained\-variance fraction\.

#### Experimental Setup\.

We evaluate on two leaderboards, MMLU \(54515451models,5757sub\-benchmarks\) and MTEB \(263263models,5656sub\-benchmarks\), with a70/3070/30train/test split\. The correlation matrixΣ^\\hat\{\\Sigma\}is estimated on the training split and used for both selection and imputation across all methods\. While indeed one can do the text embedding\-based approach as done in Section[3\.1](https://arxiv.org/html/2607.09739#S3.SS1), we keep setup consistent withSmola \([2026](https://arxiv.org/html/2607.09739#bib.bib18)\)for fair comparison\. Moreover, in this particular setup, since the number of models is many\-fold compared to Section[3\.1](https://arxiv.org/html/2607.09739#S3.SS1), the covariance matrix is reliable\. We run FL on the sameΣ^\\hat\{\\Sigma\}, applying the truncationΣ^i​j\+=max⁡\(Σ^i​j,0\)\\hat\{\\Sigma\}^\{\+\}\_\{ij\}=\\max\(\\hat\{\\Sigma\}\_\{ij\},0\)since FL requires non\-negative pairwise similarities, unlike DPP, which requires positive semidefiniteness; imputation still uses the originalΣ^\\hat\{\\Sigma\}, so the truncation affects only which benchmarks are selected, never how scores are predicted\.

![Refer to caption](https://arxiv.org/html/2607.09739v1/x4.png)

Figure 4:Benchmark\-level selection on MMLU and MTEB \(same protocol as\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\), with FL run on the identical score\-derived similarity matrixΣ\\Sigmaused by greedy entropy and greedy MI\. FL matches or beats both DPP\-style objectives almost everywhere, and dominates throughout on MTEB according to mean testR2R^\{2\}\.![Refer to caption](https://arxiv.org/html/2607.09739v1/x5.png)

Figure 5:Benchmark\-level selection on MMLU and MTEB \(same protocol as\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\), with FL run on the identical score\-derived similarity matrixΣ\\Sigmaused by greedy entropy and greedy MI\. FL achieves the lowest residual variance fraction across the entire range on both leaderboards according to mean residual variance fraction\.
#### Results\.

Figure[5](https://arxiv.org/html/2607.09739#S4.F5)reports testR2R^\{2\}as a function of the number of selected benchmarkskkon MMLU and MTEB\. FL is the strongest method across nearly the entire range, and its advantage is largest in the small\-kkregime that matters most for cheap evaluation: on MMLU, it reachesR2≈0\.86R^\{2\}\\approx 0\.86with a single benchmark and≈0\.92\\approx 0\.92byk=4k=4, where Greedy Entropy \(DPP\-Cov\) still trails badly \(R2≈0\.61R^\{2\}\\approx 0\.61atk=1k=1\) since the first element can be arbitrarily bad\. It recovers only for largerkk, converging to FL byk≈18k\\approx 18–2020; greedy MI \(DPP\-Cov\-MI\) is competitive at smallkk, particularly on MTEB, but plateaus below FL and Greedy Entropy \(R2≈0\.83R^\{2\}\\approx 0\.83–0\.850\.85\) at largerkk\. Overall, a small number of FL\-selected benchmarks suffices to recover the full leaderboard profile of an unseen model:k=4k=4benchmarks out of5757already yieldR2\>0\.9R^\{2\}\>0\.9on MMLU\. Figure[5](https://arxiv.org/html/2607.09739#S4.F5)shows scores of residual variance fraction, where FL again achieves the best across the entire range on both leaderboards\. In appendix[C\.4](https://arxiv.org/html/2607.09739#A3.SS4), we expand DPPs on the choice of kernel to Gaussian kernels, and again observe that FL achieves the best across the entire range on both leaderboards\.

#### Benchmark level similarity using Submodular Mutual Information\.

The benchmark\-level selection ofSmola \([2026](https://arxiv.org/html/2607.09739#bib.bib18)\)operates in the*evaluation\-supervised*regime: it estimates a benchmark–benchmark similarity matrix from the covarianceΣ\{\\Sigma\}of a dense model–benchmark score matrix\. This raises a natural question: when such scores are unavailable, how can one construct the similarity matrix? Here we show that the*same*facility\-location based approach can build a benchmark\-level similarity matrix and order the benchmarks by the greedy FL ordering, but in a fully*unsupervised*manner that depends on*prompt text*\. The key idea is to replace the score covariance with the submodular mutual information \(SMI\) of the prompt\-level FL function\. LetAb=\{i∈V:b​\(i\)=b\}A\_\{b\}=\\\{i\\in V:b\(i\)=b\\\}be the set of prompts belonging to benchmarkb∈ℬb\\in\\mathcal\{B\}, and letffbe the prompt\-level facility\-location function induced by the cosine similaritywi​j=max⁡\(0,cos⁡\(ei,ej\)\)w\_\{ij\}=\\max\(0,\\cos\(e\_\{i\},e\_\{j\}\)\)over the prompt embeddings\. For any two prompt subsetsAAandBB, the submodular mutual informationIf​\(A;B\)=f​\(A\)\+f​\(B\)−f​\(A∪B\)I\_\{f\}\(A;B\)=f\(A\)\+f\(B\)\-f\(A\\cup B\)measures how strongly the two sets overlap in the regions of embedding space they cover\. SinceIf​\(Ab;Ab′\)≤min⁡\(f​\(Ab\),f​\(Ab′\)\)I\_\{f\}\(A\_\{b\};A\_\{b^\{\\prime\}\}\)\\leq\\min\\\!\\big\(f\(A\_\{b\}\),f\(A\_\{b^\{\\prime\}\}\)\\big\), we normalize it to obtain a benchmark–benchmark similarity matrixMM,

Mb​b′=If​\(Ab;Ab′\)min⁡\(f​\(Ab\),f​\(Ab′\)\),M\_\{bb^\{\\prime\}\}\\;=\\;\\frac\{I\_\{f\}\(A\_\{b\};A\_\{b^\{\\prime\}\}\)\}\{\\min\\\!\\big\(f\(A\_\{b\}\),\\,f\(A\_\{b^\{\\prime\}\}\)\\big\)\},\(12\)so that a largeMb​b′M\_\{bb^\{\\prime\}\}indicates that benchmarksbbandb′b^\{\\prime\}redundantly cover the same region of embedding space\. Figure[6](https://arxiv.org/html/2607.09739#S4.F6)visualizesMMover the full set of benchmarks\.

We then define a second, benchmark\-level facility\-location functionGGover the ground setℬ\\mathcal\{B\}, usingMMas its similarity matrix,

G​\(S\)=∑b∈ℬmaxb′∈S⁡Mb​b′,S⊆ℬ,G\(S\)\\;=\\;\\sum\_\{b\\in\\mathcal\{B\}\}\\max\_\{b^\{\\prime\}\\in S\}M\_\{bb^\{\\prime\}\},\\qquad S\\subseteq\\mathcal\{B\},\(13\)and order the benchmarks using the greedy order based on the largest conditional gain\. This produces a*conditional*ordering: the first benchmark is the most representative of the entire suite, and each subsequent benchmark is the most representative one*given*those already selected\. Table[5](https://arxiv.org/html/2607.09739#A3.T5)\(in appendix[C\.4](https://arxiv.org/html/2607.09739#A3.SS4)\) reports the full ordering\. The greedy order first selectsMMLU, the broadest and largest knowledge benchmark, and the remaining order tracks the semantic structure of the suite: near\-duplicate benchmarks are deferred to the end once a representative has been chosen\.MMLU\-Pro\(the second\-largest benchmark, with12,03212\{,\}032prompts\) drops to rank2525becauseMMLUalready covers it; likewiseBFCL v4\(rank2929\) followsBFCL\(rank22\),LiveCodeBench v5\(rank2222\) followsv6\(rank33\),AIME 2024\(rank3434\) followsAIME 2025\(rank88\), andτ\\tau\-Bench\(rank3535\) followsτ2\\tau^\{2\}\-Bench\(rank55\)\. This provides an additional application of the facility location function through the lens of submodular mutual information\.

![Refer to caption](https://arxiv.org/html/2607.09739v1/x6.png)Figure 6:Normalized benchmark–benchmark similarity matrixMM\(Eq\.[12](https://arxiv.org/html/2607.09739#S4.E12)\), computed purely from prompt embeddings\. A bright entryMb​b′M\_\{bb^\{\\prime\}\}indicates that benchmarksbbandb′b^\{\\prime\}redundantly cover the same region of embedding space; this matrix is the similarity used by the benchmark\-level facility\-location greedy that produces the ordering in Table[5](https://arxiv.org/html/2607.09739#A3.T5)\.

## 5Conclusion

We studied*evaluation\-unsupervised benchmark coreset selection*, where a coreset is selected using no previous model evaluation outcomes and information only from the prompts themselves, in contrast to prior evaluation\-supervised methods that rely on model responses or evaluation scores\. Our main finding is that facility location using semantic embeddings is a strong and reliable choice in this regime\. It attains the lowest MRE and Cap\-MRE across the full budget rangek∈\[70,200\]k\\in\[70,200\]on a new large\-scale suite of 35 heterogeneous benchmarks, 18 frontier LLMs, and 61K prompts\. Facility location outperforms twelve baselines and random selection, with the covariance\-kernel DPP its closest competitor and difficulty\-based selectors, biserial correlation the weakest\. This indicates that diversity, rather than difficulty, is what matters for high fidelity\. Further analysis reveals that IRT embeddings learned from a sparse model score matrix can be very noisy, whereas text embeddings provide a model\-independent basis\. Beyond the evaluation\-unsupervised setting, we also evaluate facility location under the evaluation\-supervised setting for benchmark subset selection, where abundant model scores are available; there too, facility location outperforms submodular baselines including DPP and its mutual\-information variant\. Lastly, we provide a case where one can construct a benchmark–benchmark similarity matrix using submodular mutual information \(see[C\.4](https://arxiv.org/html/2607.09739#A3.SS4)\) with a facility location function, motivating future work for evaluation\-unsupervised benchmark subset selection\. Overall, these results suggest that facility location is an effective and practical objective for benchmark coreset construction\.

## References

- A\. Abbas, K\. Tirumala, D\. Simig, S\. Ganguli, and A\. S\. Morcos \(2023\)SemDeDup: data\-efficient learning at web\-scale through semantic deduplication\.External Links:2303\.09540,[Link](https://arxiv.org/abs/2303.09540)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- AIME \(2025\)Aime problems and solutions\.Note:[https://artofproblemsolving\.com/wiki/index\.php/AIME\_Problems\_and\_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions)Accessed: 2025\-04\-20Cited by:[5th item](https://arxiv.org/html/2607.09739#A2.I1.i5.p1.1)\.
- Anthropic \(2025\)System card: claude opus 4 & claude sonnet 4\.Technical reportAnthropic\.Note:Accessed: 2025\-05\-23External Links:[Link](https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf)Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- J\. T\. Ash, C\. Zhang, A\. Krishnamurthy, J\. Langford, and A\. Agarwal \(2019\)Deep batch active learning by diverse, uncertain gradient lower bounds\.CoRRabs/1906\.03671\.External Links:[Link](http://arxiv.org/abs/1906.03671),1906\.03671Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Balunović, J\. Dekoninck, I\. Petrov, N\. Jovanović, and M\. Vechev \(2025\)Matharena: evaluating llms on uncontaminated math competitions\.arXiv preprint arXiv:2505\.23281\.Cited by:[5th item](https://arxiv.org/html/2607.09739#A2.I1.i5.p1.1)\.
- V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan \(2025\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- G\. Bhatt, Y\. Chen, A\. Das, J\. Zhang, S\. Truong, S\. Mussmann, Y\. Zhu, J\. Bilmes, S\. Du, K\. Jamieson, J\. Ash, and R\. Nowak \(2024\)An experimental design framework for label\-efficient supervised finetuning of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 6549–6560\.External Links:[Link](https://aclanthology.org/2024.findings-acl.390/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.390)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Bilmes \(2022\)Submodularity in machine learning and artificial intelligence\.arXiv preprint arXiv:2202\.00132\.Cited by:[Appendix A](https://arxiv.org/html/2607.09739#A1.p1.18),[§1](https://arxiv.org/html/2607.09739#S1.p6.1),[footnote 1](https://arxiv.org/html/2607.09739#footnote1)\.
- J\. Bilmes \(2026\)Submarine: SUBModularity for ARtificial INtelligencE and machine learning\.Note:Online Software System[https://submarine\.page](https://submarine.page/)Cited by:[§3\.1](https://arxiv.org/html/2607.09739#S3.SS1.SSS0.Px2.p2.5),[§3\.1](https://arxiv.org/html/2607.09739#S3.SS1.SSS0.Px4.p2.7)\.
- A\. Bukharin and T\. Zhao \(2024\)Data diversity matters for robust instruction tuning\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 3411–3425\.External Links:[Link](https://aclanthology.org/)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Chen, X\. Hao, W\. Liu, X\. Huang, X\. Zeng, S\. Yu, D\. Li, S\. Wang, W\. Gan, Y\. Huang,et al\.\(2025\)ACEBench: who wins the match point in tool learning?\.arXiv e\-prints,pp\. arXiv–2501\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- W\. J\. Cook, A\. Frank, and M\. Jünger \(2009\)Combinatorial optimization\.Oberwolfach Rep\.5\(4\),pp\. 2875–2942\(en\)\.Cited by:[2nd item](https://arxiv.org/html/2607.09739#S4.I2.i2.p1.2)\.
- A\. M\. Das, G\. Bhatt, S\. Verma, Y\. Wang, V\. V\. Muppirala, and J\. Bilmes \(2026\)Matched data, better models: target aligned data filtering with sparse autoencoders\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cgmo3v18sx)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- DatologyAI Team \(2026\)Discriminative, faithful, and efficient VLM evaluations\.External Links:2601\.02316,[Link](https://arxiv.org/abs/2601.02316)Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p1.1)\.
- C\. Deng, Y\. Zhao, X\. Tang, M\. Gerstein, and A\. Cohan \(2024\)Investigating data contamination in modern benchmarks for large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 8706–8719\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p3.1)\.
- M\. Fang, X\. Wan, F\. Lu, F\. Xing, and K\. Zou \(2025\)Mathodyssey: benchmarking mathematical problem\-solving skills in large language models using odyssey math data\.Scientific Data12\(1\),pp\. 1392\.Cited by:[5th item](https://arxiv.org/html/2607.09739#A2.I1.i5.p1.1)\.
- Google \(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.External Links:2507\.06261,[Link](https://arxiv.org/abs/2507.06261)Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024a\)Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.arXiv preprint arXiv:2402\.14008\.Cited by:[5th item](https://arxiv.org/html/2607.09739#A2.I1.i5.p1.1)\.
- X\. He, L\. Liao, H\. Zhang, L\. Nie, X\. Hu, and T\. Chua \(2017\)Neural collaborative filtering\.InProceedings of the 26th international conference on world wide web,pp\. 173–182\.Cited by:[§4\.1](https://arxiv.org/html/2607.09739#S4.SS1.p2.8)\.
- Y\. He, D\. Jin, C\. Wang, C\. Bi, K\. Mandyam, H\. Zhang, C\. Zhu, N\. Li, T\. Xu, H\. Lv,et al\.\(2024b\)Multi\-if: benchmarking llms on multi\-turn and multilingual instructions following\.arXiv preprint arXiv:2410\.15553\.Cited by:[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[4th item](https://arxiv.org/html/2607.09739#A2.I1.i4.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[5th item](https://arxiv.org/html/2607.09739#A2.I1.i5.p1.1)\.
- Z\. Huang, G\. Ling, S\. Zhong, H\. Wu, and L\. Lin \(2025\)MiniLongBench: the low\-cost long context understanding benchmark for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11442–11460\.Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- R\. Iyer, N\. Khargoankar, J\. Bilmes, and H\. Asanani \(2021a\)Submodular combinatorial information measures with applications in machine learning\.InAlgorithmic Learning Theory,pp\. 722–754\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p6.1),[1st item](https://arxiv.org/html/2607.09739#S4.I2.i1.p1.12)\.
- R\. Iyer, N\. Khargonkar, J\. Bilmes, and H\. Asnani \(2021b\)Generalized submodular information measures: theoretical properties, examples, optimization algorithms, and applications\.IEEE Transactions on Information Theory68\(2\),pp\. 752–781\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p6.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2024\)Livecodebench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[2nd item](https://arxiv.org/html/2607.09739#A2.I1.i2.p1.1)\.
- Y\. Jiang, Y\. Wang, X\. Zeng, W\. Zhong, L\. Li, F\. Mi, L\. Shang, X\. Jiang, Q\. Liu, and W\. Wang \(2023\)Followbench: a multi\-level fine\-grained constraints following benchmark for large language models\.arXiv preprint arXiv:2310\.20410\.Cited by:[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2023\)Swe\-bench: can language models resolve real\-world github issues?\.arXiv preprint arXiv:2310\.06770\.Cited by:[2nd item](https://arxiv.org/html/2607.09739#A2.I1.i2.p1.1)\.
- S\. Joshi, H\. Yin, R\. Adiga, R\. Monti, A\. Carranza, A\. Fang, A\. Deng, A\. Abbas, B\. Larsen, C\. Blakeney,et al\.\(2026\)DatBench: discriminative, faithful, and efficient vlm evaluations\.arXiv preprint arXiv:2601\.02316\.Cited by:[2nd item](https://arxiv.org/html/2607.09739#S4.I1.i2.p1.4),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.130.131.1.1)\.
- K\. Killamsetty, D\. Sivasubramanian, G\. Ramakrishnan, and R\. K\. Iyer \(2020\)GLISTER: generalization based data subset selection for efficient and robust learning\.CoRRabs/2012\.10630\.External Links:[Link](https://arxiv.org/abs/2012.10630),2012\.10630Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Kipnis, K\. Voudouris, L\. Schulze Buschoff, and E\. Schulz \(2025\)Metabench\-a sparse benchmark of reasoning and knowledge in large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 31734–31770\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p3.1),[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kothawade, N\. Beck, K\. Killamsetty, and R\. K\. Iyer \(2021\)SIMILAR: submodular information measures based active learning in realistic scenarios\.CoRRabs/2107\.00717\.External Links:[Link](https://arxiv.org/abs/2107.00717),2107\.00717Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Krause, A\. Singh, and C\. Guestrin \(2008\)Near\-optimal sensor placements in gaussian processes: theory, efficient algorithms and empirical studies\.Journal of Machine Learning Research9\(8\),pp\. 235–284\.External Links:[Link](http://jmlr.org/papers/v9/krause08a.html)Cited by:[1st item](https://arxiv.org/html/2607.09739#S4.I2.i1.p1.12),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.130.132.2.1)\.
- A\. Kulesza and B\. Taskar \(2012\)Determinantal point processes for machine learning\.Foundations and Trends® in Machine Learning5\(2\-3\),pp\. 123–286\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p6.1),[1st item](https://arxiv.org/html/2607.09739#S4.I2.i1.p1.12),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.130.132.2.1)\.
- Y\. Li, Z\. Dong, and Y\. Shao \(2025\)DrafterBench: benchmarking large language models for tasks automation in civil engineering\.arXiv preprint arXiv:2507\.11527\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- H\. Lin and J\. A\. Bilmes \(2009\)How to select a good training\-data subset for transcription: submodular active selection for sequences\.InProc\. Annual Conference of the International Speech Communication Association \(INTERSPEECH\),Brighton, UK\.Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, F\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin,et al\.\(2024\)Toolsandbox: a stateful, conversational, interactive evaluation benchmark for llm tool use capabilities\.arXiv preprint arXiv:2408\.04682\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- A\. Maharana, P\. Yadav, and M\. Bansal \(2023\)D2 pruning: message passing for balancing diversity and difficulty in data pruning\.External Links:2310\.07931,[Link](https://arxiv.org/abs/2310.07931)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p1.1)\.
- B\. Mirzasoleiman, A\. Badanidiyuru, A\. Karbasi, J\. Vondrák, and A\. Krause \(2015\)Lazier than lazy greedy\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[Appendix A](https://arxiv.org/html/2607.09739#A1.p1.18)\.
- B\. Mirzasoleiman, J\. A\. Bilmes, and J\. Leskovec \(2019\)Data sketching for faster training of machine learning models\.CoRRabs/1906\.01827\.External Links:[Link](http://arxiv.org/abs/1906.01827),1906\.01827Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Myrzakhan, S\. M\. Bsharat, and Z\. Shen \(2024\)Open\-llm\-leaderboard: from multi\-choice to open\-style questions for llms evaluation, benchmark, and arena\.arXiv preprint arXiv:2406\.07545\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p3.1)\.
- G\. L\. Nemhauser, L\. A\. Wolsey, and M\. L\. Fisher \(1978\)An analysis of approximations for maximizing submodular set functions—i\.Mathematical programming14\(1\),pp\. 265–294\.Cited by:[Appendix A](https://arxiv.org/html/2607.09739#A1.p1.18),[§3\.1](https://arxiv.org/html/2607.09739#S3.SS1.SSS0.Px2.p2.5)\.
- OpenAI \(2025\)OpenAI o3 and o4\-mini system card\.Technical reportOpenAI\.Note:Accessed: 2025\-04\-16External Links:[Link](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf)Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2026\)OpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=2GmDdhBdDk)Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- M\. Paul, S\. Ganguli, and G\. K\. Dziugaite \(2021\)Deep learning on a data diet: finding important examples early in training\.CoRRabs/2107\.07075\.External Links:[Link](https://arxiv.org/abs/2107.07075),2107\.07075Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[4th item](https://arxiv.org/html/2607.09739#A2.I1.i4.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p1.1)\.
- F\. M\. Polo, L\. Weber, L\. Choshen, Y\. Sun, G\. Xu, and M\. Yurochkin \(2024\)TinyBenchmarks: evaluating llms with fewer examples\.arXiv preprint arXiv:2402\.14992\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p3.1),[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2607.09739#S4.I1.i1.p1.2),[3rd item](https://arxiv.org/html/2607.09739#S4.I2.i3.p1.5),[§4\.1](https://arxiv.org/html/2607.09739#S4.SS1.p2.7),[§4\.3](https://arxiv.org/html/2607.09739#S4.SS3.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.37.37.1),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.56.56.1)\.
- V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. Hajishirzi \(2026\)Generalizing verifiable instruction following\.Advances in Neural Information Processing Systems38\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p1.1)\.
- V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. Hajishirzi \(2025\)Generalizing verifiable instruction following\.arXiv preprint arXiv:2507\.02833\.Cited by:[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.
- Y\. Qin, K\. Song, Y\. Hu, W\. Yao, S\. Cho, X\. Wang, X\. Wu, F\. Liu, P\. Liu, and D\. Yu \(2024\)Infobench: evaluating instruction following ability in large language models\.arXiv preprint arXiv:2401\.03601\.Cited by:[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)Gpqa: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,Cited by:[4th item](https://arxiv.org/html/2607.09739#A2.I1.i4.p1.1)\.
- G\. Saranathan, C\. Xu, M\. P\. Alam, T\. Kumar, M\. Foltin, S\. Y\. Wong, and S\. Bhattacharya \(2025\)SubLIME: subset selection via rank correlation prediction for data\-efficient llm evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 30572–30593\.Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Sener and S\. Savarese \(2018\)Active learning for convolutional neural networks: a core\-set approach\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1aIuk-RW)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2607.09739#S4.I2.i2.p1.2),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.37.37.1)\.
- V\. Sirdeshmukh, K\. Deshpande, J\. Mols, L\. Jin, E\. Cardona, D\. Lee, J\. Kritz, W\. Primack, S\. Yue, and C\. Xing \(2025\)Multichallenge: a realistic multi\-turn conversation evaluation benchmark challenging to frontier llms\.arXiv preprint arXiv:2501\.17399\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2),[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.
- A\. Smola \(2026\)Submodular benchmark selection\.External Links:2605\.02209,[Link](https://arxiv.org/abs/2605.02209)Cited by:[Figure 10](https://arxiv.org/html/2607.09739#A3.F10),[Figure 11](https://arxiv.org/html/2607.09739#A3.F11),[§C\.4](https://arxiv.org/html/2607.09739#A3.SS4.p1.5),[§1](https://arxiv.org/html/2607.09739#S1.p3.1),[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1),[Figure 5](https://arxiv.org/html/2607.09739#S4.F5),[1st item](https://arxiv.org/html/2607.09739#S4.I2.i1.p1.12),[§4\.4](https://arxiv.org/html/2607.09739#S4.SS4.SSS0.Px2.p1.9),[§4\.4](https://arxiv.org/html/2607.09739#S4.SS4.SSS0.Px4.p1.10),[§4\.4](https://arxiv.org/html/2607.09739#S4.SS4.p1.1),[Table 1](https://arxiv.org/html/2607.09739#S4.T1.130.132.2.1)\.
- N\. Subramani, A\. Gomez, and M\. Diab \(2025\)SimBA: simplifying benchmark analysis using performance matrices alone\.External Links:2510\.17998,[Link](https://arxiv.org/abs/2510.17998)Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- M\. K\. Surkov and I\. P\. Yamshchikov \(2024\)Vygotsky distance: measure for benchmark task similarity\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 17409–17420\.Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Team, Y\. Bai, Y\. Bao, G\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen, Y\. Chen,et al\.\(2025\)Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- N\. team \(2024\)NexusBench: fc and agent benchmarking suite\.External Links:[Link](https://github.com/nexusflowai/NexusBench)Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- T\. T\. Team \(2025\)Terminal\-bench: a benchmark for ai agents in terminal environments\.External Links:[Link](https://github.com/laude-institute/terminal-bench)Cited by:[2nd item](https://arxiv.org/html/2607.09739#A2.I1.i2.p1.1)\.
- R\. Vivek, K\. Ethayarajh, D\. Yang, and D\. Kiela \(2024\)Anchor points: benchmarking models with much fewer examples\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1576–1601\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p3.1),[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024\)Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[4th item](https://arxiv.org/html/2607.09739#A2.I1.i4.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p1.1),[§1](https://arxiv.org/html/2607.09739#S1.p3.1)\.
- Z\. Wang, Y\. Liu, Y\. Wang, W\. He, B\. Gao, M\. Diao, Y\. Chen, K\. Fu, F\. Sung, Z\. Yang,et al\.\(2025\)OJBench: a competition level code benchmark for large language models\.arXiv preprint arXiv:2506\.16395\.Cited by:[2nd item](https://arxiv.org/html/2607.09739#A2.I1.i2.p1.1)\.
- J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus \(2024\)Measuring short\-form factuality in large language models\.arXiv preprint arXiv:2411\.04368\.Cited by:[4th item](https://arxiv.org/html/2607.09739#A2.I1.i4.p1.1)\.
- K\. Wei, Y\. Liu, K\. Kirchhoff, and J\. Bilmes \(2014\)Unsupervised submodular subset selection for speech data\.In2014 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 4107–4111\.Cited by:[footnote 2](https://arxiv.org/html/2607.09739#footnote2)\.
- B\. Wen, P\. Ke, X\. Gu, L\. Wu, H\. Huang, J\. Zhou, W\. Li, B\. Hu, W\. Gao, J\. Xu,et al\.\(2024\)Benchmarking complex instruction\-following with multiple constraints composition\.Advances in Neural Information Processing Systems37,pp\. 137610–137645\.Cited by:[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.
- xAI \(2025\)Grok 4\.Note:[https://x\.ai](https://x.ai/)Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- C\. Xu, S\. Guan, D\. Greene, M\. Kechadi,et al\.\(2024\)Benchmark data contamination of large language models: a survey\.arXiv preprint arXiv:2406\.04244\.Cited by:[§1](https://arxiv.org/html/2607.09739#S1.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix B](https://arxiv.org/html/2607.09739#A2.SS0.SSS0.Px2.p1.1)\.
- J\. Yao, P\. Jin, K\. Bao, Q\. Yu, K\. Bhardwaj, C\. Su, J\. Wang, Y\. Zhu, S\. Devare, D\. Mosk\-Aoyama,et al\.\(2025\)The measure of all measures: quantifying llm benchmark quality\.InNeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling,Cited by:[Table 1](https://arxiv.org/html/2607.09739#S4.T1.130.131.1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2),[§1](https://arxiv.org/html/2607.09739#S1.p1.1)\.
- X\. Zhang, C\. Li, Y\. Zong, Z\. Ying, L\. He, and X\. Qiu \(2023\)Evaluating the performance of large language models on gaokao benchmark\.arXiv preprint arXiv:2305\.12474\.Cited by:[4th item](https://arxiv.org/html/2607.09739#A2.I1.i4.p1.1)\.
- Y\. Zhang, J\. Hu, S\. Feng, P\. Yuan, X\. Wang, Y\. Li, J\. Shi, C\. Tan, J\. Zhang, B\. Pan,et al\.\(2026\)Learning more from less: unlocking internal representations for benchmark compression\.arXiv preprint arXiv:2602\.00710\.Cited by:[§2](https://arxiv.org/html/2607.09739#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Zhong, Z\. Du, X\. Zhang, H\. Hu, and J\. Tang \(2025\)ComplexFuncBench: exploring multi\-step and constrained function calling under long\-context scenario\.arXiv preprint arXiv:2501\.10132\.Cited by:[1st item](https://arxiv.org/html/2607.09739#A2.I1.i1.p1.2)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[3rd item](https://arxiv.org/html/2607.09739#A2.I1.i3.p1.1)\.

## Appendix AA Primer on Submodular Functions

Submodular functions are set functions that exhibit diminishing returns\. Formally, letVVbe the ground set of elements and letf:2V→ℝf:2^\{V\}\\to\\mathbb\{R\}be a set function\. For two subsetsA⊆B⊆VA\\subseteq B\\subseteq Vand an elementv∈V∖Bv\\in V\\setminus B,ffis*submodular*if

f​\(A∪\{v\}\)−f​\(A\)≥f​\(B∪\{v\}\)−f​\(B\)\.f\(A\\cup\\\{v\\\}\)\-f\(A\)\\;\\geq\\;f\(B\\cup\\\{v\\\}\)\-f\(B\)\.\(14\)The quantitiesf​\(A∪\{v\}\)−f​\(A\)f\(A\\cup\\\{v\\\}\)\-f\(A\)andf​\(B∪\{v\}\)−f​\(B\)f\(B\\cup\\\{v\\\}\)\-f\(B\)are the*marginal gains*of addingvvtoAAand toBB, respectively\. Condition \([14](https://arxiv.org/html/2607.09739#A1.E14)\) states that the marginal gain of an element to a smaller set is at least its marginal gain to a larger superset\. In contrast to score\-based functions, which assign an independent value to each element and evaluate a subset by summing those values \(so thatf​\(A\)=∑v∈Amvf\(A\)=\\sum\_\{v\\in A\}m\_\{v\}for per\-element scoresmvm\_\{v\}\), submodular functions capture interactions among the selected elements and are therefore well suited to modeling notions of diversity and coverage\. A set function is additionally*monotone*iff​\(A\)≤f​\(B\)f\(A\)\\leq f\(B\)wheneverA⊆BA\\subseteq B; the facility location function of Equation[1](https://arxiv.org/html/2607.09739#S3.E1)is both monotone and submodular\. For benchmark coreset selection we maximize such a function subject to the cardinality budgetkk,

A∗=argmaxA⊆V\|A\|=kf​\(A\),A^\{\*\}\\;=\\;\\operatorname\*\{argmax\}\_\{\\begin\{subarray\}\{c\}A\\subseteq V\\\\ \|A\|=k\\end\{subarray\}\}\\,f\(A\),\(15\)recovering the problem in Equation[2](https://arxiv.org/html/2607.09739#S3.E2)\. Although Equation[15](https://arxiv.org/html/2607.09739#A1.E15)is NP\-hard in general, monotone and normalized submodular function maximization under a cardinality constraint admits a simple greedy algorithm that starts fromA=∅A=\\emptysetand repeatedly adds the element of largest marginal gain,argmaxv∈V∖Af​\(v∣A\)\\operatorname\*\{argmax\}\_\{v\\in V\\setminus A\}f\(v\\mid A\), attaining a\(1−1/e\)\(1\-1/e\)approximation of the optimum\(Nemhauseret al\.,[1978](https://arxiv.org/html/2607.09739#bib.bib72)\); accelerated variants such as lazy and stochastic greedy preserve comparable guarantees while scaling to much larger ground sets\(Mirzasoleimanet al\.,[2015](https://arxiv.org/html/2607.09739#bib.bib84)\)\. SeeBilmes \([2022](https://arxiv.org/html/2607.09739#bib.bib8)\)for more details\.

## Appendix BExperiment Details

#### Benchmarks\.

We evaluate 35 benchmarks grouped into five categories:

- •Agentic Tool Use: ACEBench\(Chenet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib60)\), BFCL\(Patilet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib61)\), ComplexFuncBench\(Zhonget al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib62)\), DrafterBench\(Liet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib63)\), MultiChallenge\(Sirdeshmukhet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib58)\), NexusBench\(team,[2024](https://arxiv.org/html/2607.09739#bib.bib64)\),τ\\tau\-Bench\(Yaoet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib65)\),τ2\\tau^\{2\}\-Bench\(Barreset al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib66)\), ToolSandbox\(Luet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib67)\)\.
- •Coding: LiveCodeBench v5/v6\(Jainet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib68)\), OJBench\(Wanget al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib69)\), Terminal\-Bench\(Team,[2025](https://arxiv.org/html/2607.09739#bib.bib70)\), SWE\-bench\(Jimenezet al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib71)\)\.
- •Instruction following: ComplexBench\(Wenet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib53)\), FollowBench\(Jianget al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib54)\), IF\-Bench\(Pyatkinet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib55)\), IF\-Eval\(Zhouet al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib56)\), InfoBench\(Qinet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib57)\), MultiChallenge\(Sirdeshmukhet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib58)\), Multi\-IF\(Heet al\.,[2024b](https://arxiv.org/html/2607.09739#bib.bib59)\)\.
- •Knowledge: MMLU\(Hendryckset al\.,[2020](https://arxiv.org/html/2607.09739#bib.bib43)\), MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib42)\), GPQA\(Reinet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib44)\), SimpleQA\(Weiet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib45)\), HLE\(Phanet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib46)\), Gaokao 2023\(Zhanget al\.,[2023](https://arxiv.org/html/2607.09739#bib.bib47)\)\.
- •Math:AIME 2024/2025\(AIME,[2025](https://arxiv.org/html/2607.09739#bib.bib48)\), HMMT Feb25\(Balunovićet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib49)\), Math 500\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.09739#bib.bib50)\), MathOdyssey\(Fanget al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib51)\), OlympiadBench\(Heet al\.,[2024a](https://arxiv.org/html/2607.09739#bib.bib52)\)\.

#### Models\.

We evaluate on 18 models:DeepSeek\-R1\-0528\(Guoet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib120)\),DeepSeek\-V3\-0324\(Liuet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib119)\),DeepSeek\-V3\.1\(Liuet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib119)\),DeepSeek\-V3\.1 thinking\(Liuet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib119)\),Kimi\-K2\-Instruct\(Teamet al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib122)\),Qwen3\-235B\-A22B\-FP8\(Yanget al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib123)\),Qwen3\-235B\-A22B\-Instruct\-2507\-FP8\(Yanget al\.,[2025](https://arxiv.org/html/2607.09739#bib.bib123)\),Claude\-4\-Sonnet\(Anthropic,[2025](https://arxiv.org/html/2607.09739#bib.bib121)\),Claude\-4\-Sonnet thinking\(Anthropic,[2025](https://arxiv.org/html/2607.09739#bib.bib121)\),Gemini\-2\.5\-Flash thinking\(Google,[2025](https://arxiv.org/html/2607.09739#bib.bib16)\),Gemini\-2\.5\-Pro thinking\(Google,[2025](https://arxiv.org/html/2607.09739#bib.bib16)\),GPT\-4\.1\(Hurstet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib117)\),GPT\-4o\-20240806\(Hurstet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib117)\),GPT\-4o\-mini\(Hurstet al\.,[2024](https://arxiv.org/html/2607.09739#bib.bib117)\),GPT\-5\(OpenAI,[2026](https://arxiv.org/html/2607.09739#bib.bib15)\),Grok\-4\(xAI,[2025](https://arxiv.org/html/2607.09739#bib.bib124)\),O3\-high\(OpenAI,[2025](https://arxiv.org/html/2607.09739#bib.bib118)\), andO4\-mini\-high\(OpenAI,[2025](https://arxiv.org/html/2607.09739#bib.bib118)\)\.

#### Embedding fields\.

Each concat embedding uses the prompt plus the benchmark target fields below\. “Prompt” refers to the original prompt field name used to construct the prompt shown to the model\. “Target” refers to the additional ground\-truth or evaluation\-target field name appended to the prompt before embedding\. When the target field is listed as none, the embedding is prompt\-only for that benchmark\.

- •ACEBench: Prompt:question,function,initial\_config\. Target:rubric\.ground\_truth\.
- •BFCL: Prompt:questionwith tool/function definitions fromfunction,functions, ortools, and task\-state fieldsinitial\_config\. Target:ground\_truth\.
- •ComplexFuncBench: Prompt: user turns inconversations\[\*\]\.content\. Target: none\.
- •DrafterBench: Prompt:Instruction\. Target:Groundtruth\.
- •MultiChallenge: Prompt:CONVERSATION\. Target:TARGET\_QUESTION\.
- •NexusBench: Prompt: task text fromInput,prompt,user\_query,generated\_question, and tool\-schema fields fromjson\_toolsorfc\_definitionwhen present\. Target:Output,outputs\.reference,ground\_truth,answer,fncall, ormodified\_correct\_ground\_truth\.
- •τ\\tau\-Bench: Prompt:instruction\. Target:actions\.
- •τ2\\tau^\{2\}\-Bench: Prompt:user\_scenario\.instructions,initial\_state\. Target:evaluation\_criteria\.
- •ToolSandbox: Prompt: starting messageScenario\.starting\_context, setting\-state fieldscellular,wifi,location\_service,low\_battery\_mode,latitude,longitude, plustool\_allow\_listandtool\_augmentation\_list\. Target: none\.
- •Aider Polyglot: Prompt:instruction\.md\. Target: none\.
- •LiveCodeBench v5/v6: Prompt: v5prompt; v6question\_contentandstarter\_code\. Target: v5verification\_info\.ground\_truth; v6public\_test\_cases\.
- •OJBench: Prompt:prompt\. Target: none\.
- •Terminal\-Bench: Prompt:instruction\.md\. Target:solution/solve\.sh\.
- •SWE\-bench: Prompt:problem\_statement\. Target:patch\.
- •ComplexBench: Prompt:instruction\_en\. Target:scoring\_questions\[\*\]\.question\_enorscoring\_questions\[\*\]\.question\.
- •FollowBench: Prompt:instruction\. Target:target\.
- •IF\-Bench: Prompt:prompt\. Target:instruction\_id\_list,kwargs\.
- •IF\-Eval: Prompt:prompt\. Target:instruction\_id\_list,kwargs\.
- •InfoBench: Prompt:instruction,input\. Target:decomposed\_questions\.
- •Multi\-IF: Prompt:turn\_1\_prompt,turn\_2\_prompt,turn\_3\_prompt\. Target:turn\_1\_instruction\_id\_list,turn\_1\_kwargs,turn\_2\_instruction\_id\_list,turn\_2\_kwargs,turn\_3\_instruction\_id\_list,turn\_3\_kwargs\.
- •MMLU: Prompt:question,choices\. Target:choices\[answer\]\.
- •MMLU\-Pro: Prompt:question,options\. Target:options\[answer\_index\]oranswer\.
- •GPQA: Prompt:Question\. Target:Correct Choice\.
- •SimpleQA: Prompt:problem\. Target:answer\.
- •HLE: Prompt:question\. Target:answer\.
- •Gaokao 2023: Prompt:question\. Target:answer\.
- •AIME 2024/2025: Prompt:problem\. Target: AIME 2024reference\_solutionboxed answer orexpected\_answer; AIME 2025expected\_answer\.
- •HMMT Feb25: Prompt:problem\. Target:expected\_answer\.
- •Math 500: Prompt:problem\. Target:answer\.
- •MathOdyssey: Prompt:problem\_statement\. Target:expected\_answer\.
- •OlympiadBench: Prompt:question\. Target:answer\.

## Appendix CAnalysis \(Cont’\)

### C\.1Full Per\-Budget Results

Tables[1](https://arxiv.org/html/2607.09739#S4.T1)report results at four representative budgets, and Figure[1](https://arxiv.org/html/2607.09739#S4.F1)summarize performance aggregated across budgets\. For completeness, we provide the full per\-budget curves here: Figure[8](https://arxiv.org/html/2607.09739#A3.F8)reports overall relative error \(MRE\) and Figure[9](https://arxiv.org/html/2607.09739#A3.F9)reports capability\-wise relative error \(Cap\-MRE\), each evaluated at every budgetk∈\[70,200\]k\\in\[70,200\]in increments of55\. Every curve is the mean over the three model folds and the shaded bands denote±\\pmSE\. All twelve baselines are shown; the two score\-based baselines \(PBSC and Hardness\) are omitted because their errors are an order of magnitude larger and would compress the visible range \(they are included in Tables[1](https://arxiv.org/html/2607.09739#S4.T1)and[1](https://arxiv.org/html/2607.09739#S4.T1)\)\.

![Refer to caption](https://arxiv.org/html/2607.09739v1/x7.png)Figure 7:Marginal gains of the greedy selection across sparsification levelsα\\alpha\. Whenα\\alphais too small, candidate gains are nearly identical and the selection order becomes arbitrary; whenα\\alphais too large, the kernel approaches density and gains collapse within the first few dozen selections\.![Refer to caption](https://arxiv.org/html/2607.09739v1/x8.png)Figure 8:Per\-budget overall relative error \(MRE\) for all methods across budgetsk∈\[70,200\]k\\in\[70,200\]in increments of55\(lower is better\)\. Curves show the mean over the three model folds; shaded bands denote±\\pmSE\. Score\-based baselines are omitted \(see text\)\. FL \(ours\) attains the lowest MRE across essentially the entire range, with DPP\-Cov the closest competitor\.![Refer to caption](https://arxiv.org/html/2607.09739v1/x9.png)Figure 9:Per\-budget capability\-wise relative error \(Cap\-MRE\) for all methods across budgetsk∈\[70,200\]k\\in\[70,200\]in increments of55\(lower is better\)\. Same protocol as Figure[8](https://arxiv.org/html/2607.09739#A3.F8)\. FL \(ours\) attains the lowest Cap\-MRE across essentially the entire range, while the cosine\-kernel DPP variants are weakest\.
### C\.2Analysis on IRT induced similarity

We measure how much the IRT\-induced similarity structure changes across the three model folds\. For each benchmark, we compute the Spearman correlationρ\\rhobetween the pairwise IRT similarity matrices produced by each pair of folds \(three pairs total\) and report the mean and standard error across pairs\. As shown in Table[2](https://arxiv.org/html/2607.09739#A3.T2),ρ\\rhoranges from0\.300\.30\(Nexus Bench\) to0\.920\.92\(Follow Bench\), with many benchmarks below0\.70\.7\. Semantic embeddings are computed from sample text alone and are therefore identical regardless of which models are available, making them more reliable for facility location\. This indicates IRT’s high dependency on large amount of model responses, which is infeasible to collect in reality\.

Table 2:Per\-benchmark stability of the IRT\-induced similarity structure across the three model folds\. For each benchmark, we report the number of promptsnnand the mean Spearman correlationρ\\rhoover three\-fold pairs between the pairwise IRT similarity matrices\. Benchmarks are sorted byρ\\rho; many fall well below0\.60\.6, indicating that the performance\-matrix\-derived similarity structure may be unstable when onlyM=18M\{=\}18models are available\.
### C\.3Computational Cost Comparison

Beyond performance, FL and DPP are not equally cheap to scale\. FL is much more efficient to compute, because of one doesn’t even need a fullN×NN\\times Nmatrix, and can work with sparse, non\-negative pairwise affinity\. DPP, on the other hand, needs a symmetric, positive\-semidefinite matrix for probabilities to make sense\. While in this current setup, the ground set over which subset selection is performed is relatively small, the computational complexity of DPP\-based methods can become prohibitive as the number of benchmarks or prompts increases\.

### C\.4Extended Results with Gaussian Kernel

WhileSmola \([2026](https://arxiv.org/html/2607.09739#bib.bib18)\)assumed benchmark performances as a joint Gaussian with covarianceΣ\\Sigma, one can go beyond this assumption by assuming an explicit kernel form\. More precisely, we model the similarity between two benchmarksi,ji,jasKi,j=exp⁡\(−γ​‖𝒮:,i−𝒮:,j‖2\)K\_\{i,j\}=\\exp\(\-\\gamma\\\|\\mathcal\{S\}\_\{:,i\}\-\\mathcal\{S\}\_\{:,j\}\\\|^\{2\}\), and compare it against facility location and existing DPP\-based results here; we tuneγ∈\{0\.1,1,10\}\\gamma\\in\\\{0\.1,1,10\\\}\. The evalutaion results are displayed in Figure[10](https://arxiv.org/html/2607.09739#A3.F10)and Figure[11](https://arxiv.org/html/2607.09739#A3.F11)\. Our findings remain consistent with facility location function outperforming all the baselines judged using test\-R2R^\{2\}and residual variance fraction\.

Table 3:TestR2R^\{2\}\(↑\\uparrow\) across budgetskk\(20 splits\)\. Best per columnbold, secondunderlined\. facility location outperforms all the other baselines, even when the DPP variants are provided with a kernel with larger capacity\.Table 4:Residual variance fraction \(↓\\downarrow\) across budgetskk\(20 splits\)\. Best per columnbold, secondunderlined\. facility location outperforms all the other baselines, even when the DPP variants are provided with a kernel with larger capacity\.![Refer to caption](https://arxiv.org/html/2607.09739v1/x10.png)Figure 10:Benchmark\-level selection on MMLU and MTEB \(same protocol as\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\), with FL run on the identical score\-derived similarity matrixΣ\\Sigmaused by greedy entropy and greedy MI\. FL matches or beats both DPP\-style objectives almost everywhere, and dominates throughout on MTEB according to mean testR2R^\{2\}\.![Refer to caption](https://arxiv.org/html/2607.09739v1/x11.png)Figure 11:Benchmark\-level selection on MMLU and MTEB \(same protocol as\(Smola,[2026](https://arxiv.org/html/2607.09739#bib.bib18)\)\), with FL run on the identical score\-derived similarity matrixΣ\\Sigmaused by greedy entropy and greedy MI\. FL achieves the lowest residual variance fraction across the entire range on both leaderboards according to mean residual variance fraction\.Table 5:Evaluation\-unsupervised benchmark\-level greedy ordering\. Benchmarks are ordered by the facility\-location greedy on the normalized submodular mutual information matrixMM\(Eq\.[12](https://arxiv.org/html/2607.09739#S4.E12)–[13](https://arxiv.org/html/2607.09739#S4.E13)\), computed purely from prompt embeddings\. Rankiiis the most novel benchmark given thei−1i\{\-\}1already selected\. That is, it is important to realize that this is not a rank ordering of the individual intrinsic properties of each benchmark individually\. Rather, the rankiibenchmark contains the next most usefulness on top of \(and assuming\) that we already have selected the previousi−1i\-1benchmarks\. For example, if we had to select only a single benchmark, MMLU would be best\. If we have already selected MMLU then the next benchmark, on top of MMLU, that would be best to select is BFCL, and so on\. This is all according to Figure[6](https://arxiv.org/html/2607.09739#S4.F6)\.