DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Summary
DecisionPivot is a diagnostic benchmark with 48 tasks and 40,033 items evaluating decision models in LLM systems, revealing large gaps between aggregate accuracy, contrast-group performance, calibration, and reasoning effort that standard evaluations obscure.
View Cached Full Text
Cached at: 10/21/26, 06:59 AM
Paper page - DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Source: https://huggingface.co/papers/2610.28522
Abstract
DecisionPivot is a diagnostic benchmark that evaluates decision models across controlled variations, revealing gaps between aggregate accuracy, contrast-group performance, calibration, and reasoning effort.
Decision modelssupportrouting,guardrails, andoutput verificationinLLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduceDecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized intocontrast groupswith computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35 public test sets, preserving their original evaluation metrics. We evaluate 40decision modelsand 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy,probability ordering,calibration, andempirical selective coverage. Our results reveal substantial differences obscured by aggregate scores: two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%. Across 2,400 binary contrast pairs, Perplexity Decider correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%, exposing a gap betweenprobability orderingand decision correctness. Models with similar item accuracy also differ in the reliability of their high-confidence predictions. On a separate 1,296-item subset, GLM-5.3 achieves 93.5\% contrast-group accuracy at a higherreasoning setting, compared with 81.5% at its lowest effort. By linking errors to pivotal details and difficulty levels,DecisionPivotprovides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.
View arXiv pageView PDFGitHub5Add to collection
Get this paper in your agent:
hf papers read 2610\.28522
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.28522 in a model README.md to link it from this page.
Datasets citing this paper1
#### startlux-models/DecisionPivot Preview• Updatedabout 1 hour ago • 468 • 3
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.28522 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
This paper introduces DiagnosticIQ, a benchmark for evaluating LLMs in translating industrial symbolic maintenance rules into actionable steps. It highlights that while frontier models perform well on standard tasks, they exhibit brittleness and pattern-matching behaviors under structural perturbations.
ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
ChaosBench-Logic v2 is a large-scale benchmark of 40,886 questions over 165 dynamical systems that evaluates LLMs' logical reasoning abilities, revealing near-random performance on regime transition reasoning and systematic failure modes even in frontier models.
LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
Introduces LingxiDiagBench, a large-scale multi-agent benchmark for evaluating LLMs on Chinese psychiatric consultation and diagnosis. Key findings show high accuracy on binary classification but poor performance on multi-way differential diagnosis, highlighting a decoupling between conversational quality and diagnostic accuracy.
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Researchers introduce MedSP1000, a 1,638-case interactive benchmark derived from standardized patient scenarios to evaluate LLMs as dynamic clinical agents across multi-turn encounters. Results show even the best model (GPT-5.5) completes only 60.4% of expert rubric items, suggesting current LLMs are not yet reliable enough for clinical practice.
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
KlinikeBench introduces a clinician-verified benchmark of 333 interactive clinical tasks where LMs must gather patient history, use clinical tools, and reach a diagnosis within a turn budget. Across 31 models, the best (e.g., GPT-6-astra, Claude Opus 5) succeed on under 30% of tasks despite 90.7% diagnostic accuracy, revealing a large gap between diagnostic knowledge and real-world clinical interaction.