Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
Summary
This paper proposes Evidence-Calibrated Query Clustering (ECC), an algorithm that aligns semantic embeddings with latent LLM capability demands using posterior model comparisons and Bradley-Terry modeling, significantly improving capability ranking quality for LLM evaluation.
View Cached Full Text
Cached at: 05/22/26, 02:32 AM
Paper page - Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
Source: https://huggingface.co/papers/2605.17110
Abstract
Query clustering algorithm ECC improves LLM capability evaluation by aligning semantic embeddings with latent capability demands through posterior model comparisons and Bradley-Terry modeling.
Query clusteringorganizes queries into groups that reflect sharedlatent capability demands, enabling capability-aware LLM evaluation. Existing clustering methods, which primarily rely on semantic taxonomies or embeddings, often fail to capture such latent capability requirements due to a misalignment between surface-level semantics and actual model performance. We propose ECC, an algorithm that calibrates priorsemantic embeddingsusing limitedposterior model comparisonsto bridge the gap between surface-level semantics and latent capability requirements. ECC characterizes each cluster through a capability profile parameterized by aBradley-Terry modeland usestrainable mixture weightsto accommodate queries with mixed capability demands, jointly learning a flexible,capability-aware clusteringstructure that supports query-specific inference of LLM capabilities. Extensive quantitative and qualitative evaluations demonstrate that ECC significantly improvesLLM capability rankingquality, outperforming human-labeled and embedding-based baselines by an average of 17.64 and 18.02 percentage points, respectively, and proves effective in downstream tasks such asquery routing.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.17110 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.17110 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.17110 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Thank you :) Swift Qwen 3.8 27B now has 100k+ downloads, is #1 finetune and #9 model on HuggingFace Trending
Jovan from UkisAI thanks the community for the success of Swift Qwen 3.8 27B, an open-source model that reduces token usage by 58.3% and increases speed by 1.95x through efficient thinking patterns. Future improved checkpoints and models are planned.
Humor Arena: Which LLM is the funniest?
The article presents a comparison of 20 LLM versions on humor generation using a fine-tuned judge, with Fable 5 scoring highest in a joke benchmark.
A common bias when building AI agents: optimizing for tone over message efficiency
The author discusses a common bias in AI agent development where conversational tone is prioritized over message efficiency, advocating for hybrid models that minimize turns per resolution.
Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
This paper proposes an Infinite-Parameter LLM architecture that uses a hypernetwork to generate weights from live data via Bayesian updates, enabling continuous adaptation and outperforming in-context learning.
LLM Classification Is Feature Engineering
The article discusses the limitations of using LLMs directly as classifiers and proposes treating LLM outputs as features in traditional machine learning models like logistic regression to achieve better calibration, interpretability, and performance.