Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
Summary
This paper introduces ECC, an algorithm that calibrates semantic embeddings with limited model comparisons to cluster queries by latent capability requirements, improving LLM capability ranking quality by over 17 percentage points over baselines.
View Cached Full Text
Cached at: 05/19/26, 06:39 AM
# Capturing LLM Capabilities via Evidence-Calibrated Query Clustering Source: [https://arxiv.org/abs/2605.17110](https://arxiv.org/abs/2605.17110) [View PDF](https://arxiv.org/pdf/2605.17110) > Abstract:Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability\-aware LLM evaluation\. Existing clustering methods, which primarily rely on semantic taxonomies or embeddings, often fail to capture such latent capability requirements due to a misalignment between surface\-level semantics and actual model performance\. We propose ECC, an algorithm that calibrates prior semantic embeddings using limited posterior model comparisons to bridge the gap between surface\-level semantics and latent capability requirements\. ECC characterizes each cluster through a capability profile parameterized by a Bradley\-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands, jointly learning a flexible, capability\-aware clustering structure that supports query\-specific inference of LLM capabilities\. Extensive quantitative and qualitative evaluations demonstrate that ECC significantly improves LLM capability ranking quality, outperforming human\-labeled and embedding\-based baselines by an average of 17\.64 and 18\.02 percentage points, respectively, and proves effective in downstream tasks such as query routing\. ## Submission history From: Fangzhou Wu \[[view email](https://arxiv.org/show-email/ec52c671/2605.17110)\] **\[v1\]**Sat, 16 May 2026 18:30:37 UTC \(1,188 KB\)
Similar Articles
Calibrating LLMs with Semantic-level Reward
Proposes CSR, a framework that calibrates LLMs directly in semantic space using a novel semantic calibration reward, reducing ECE by up to 40% and improving AUROC by up to 31% over verbalized-confidence baselines across multiple datasets.
Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
This paper introduces a behavioral evaluation framework for calibrating claims about deployment-time memory in LLM test-time training, proposing an evidence ladder and explicit baselines to bridge proxy metrics and behavioral evidence.
CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
CRAFT converts rubric-based evaluation into hierarchical capability diagnosis for LLMs, identifying specific weaknesses and generating targeted fine-tuning data, achieving stronger results on finance and legal benchmarks across four open-source models.
LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction
This paper evaluates LLM-based strategies (embedding, prompt, hybrid) against classical tabular models on an industrial car retrofit prediction dataset with hashed categorical features. It finds that tree ensembles outperform LLMs overall, but embeddings and hybrid approaches remain useful, while direct prompting fails without semantic cues.
Uncertainty-Guided LLM Semantic Augmentation for Heterogeneous Treatment Effect Estimation
This paper proposes CURL, a plug-in adapter that uses estimator uncertainty to allocate pretrained LLM semantic capacity for improving heterogeneous treatment effect (CATE) estimation. It introduces two role-conditioned prompts to construct assignment- and heterogeneity-oriented representations, improving ten host learners on four benchmarks.