Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
Summary
This paper investigates central tendency bias in multimodal LLMs used for clinical ordinal scoring of the Clock Drawing Test, finding that LLMs compress predictions toward the middle of the scale, disproportionately affecting critical extremes. The study extends the LLM-as-judge bias literature to clinical assessment, highlighting the need for calibration-aware evaluation before deployment.
View Cached Full Text
Cached at: 05/19/26, 06:33 PM
Paper page - Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
Source: https://huggingface.co/papers/2605.16386
Abstract
Large language models exhibit systematic bias toward central tendency when evaluating clinical assessments, particularly affecting critical score extremes important for cognitive impairment screening.
Multimodal large language models (LLMs) are increasingly explored as automated evaluators in clinical settings, yet their scoring behavior onordinal clinical scalesremains poorly understood. We benchmark three frontier LLM families against supervised deep learning models for scoringClock Drawing Test(CDT) images on two public datasets using theShulman rubric. While fully fine-tunedVision Transformersachieve the bestcalibration(MAE 0.52, within-1 accuracy 91%), zero-shot LLMs remain competitive on tolerance-based agreement (GPT-5 MAE 0.67, within-1 accuracy 92%) despite higher absolute error. However, per-score analysis reveals that all three LLM families exhibit a pronouncedcentral tendency effect(systematic endpoint compression): predictions are systematically compressed toward the middle of the scale, with over-prediction at the low end (score 0 to 1) and under-prediction at the high end (score 5 to 4). This effect disproportionately affects the clinically critical extremes where accurate scoring most impacts screening decisions for cognitive impairment. Targeted ablations show that neither few-shot exemplars spanning the full score range nor removing clinical terminology from the prompt eliminates the effect. Our findings extend theLLM-as-a-judge biasliterature from NLP evaluation toclinical assessment, and highlight the need forcalibration-aware evaluation and post-hoccalibrationbefore deploying LLM-based raters in high-stakes screening workflows.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.16386
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.16386 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.16386 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.16386 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies perceptual judgment bias in multimodal LLM judges, where they over-reward fluent but visually wrong responses, and proposes a dataset PPJD and a trained model Perception-Judge using GRPO with batch-ranking reward to mitigate this bias and improve perception-grounded evaluation.
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Researchers introduce MM-JudgeBias, a benchmark that exposes systematic compositional biases in multimodal large language models when used as automatic judges, testing 26 SOTA MLLMs across 1,800 samples.
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
This paper investigates whether standard benchmarks underestimate LLM performance by re-evaluating hallucination detection datasets using an LLM-first, human-adjudicated assessment method. The study finds that incorporating LLM reasoning into the adjudication process improves agreement and suggests that model-assisted re-evaluation yields more reliable benchmarks for ambiguity-prone tasks.
LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation
This paper introduces a psychometric datasheet protocol for evaluating LLM judges as measurement instruments, measuring dark current, positional false preference, stable cross-sensitivity, and target sensitivity. A case study on three open-weight models reveals significant differences in judge quality and behavior.
Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
This paper investigates LLM-based metrics for evaluating clinical significance in radiology report generation. It identifies discrimination bias in existing LLM evaluators and proposes training lightweight interpretable metrics to improve the balance between error detection and tolerance of harmless variations.