PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Summary
PerceptionRubrics introduces a rubric-based evaluation framework for multimodal models that uses atomic auditing and gated scoring to better align benchmark scores with human perception, revealing reliability gaps and open-closed stratification.
View Cached Full Text
Cached at: 07/02/26, 07:47 AM
Paper page - PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Source: https://huggingface.co/papers/2606.28322 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
PerceptionRubrics presents a rubric-based evaluation framework that identifies gaps between benchmark scores and real-world performance through atomic auditing and gated scoring mechanisms.
We introduce PerceptionRubrics, arubric-based evaluationframework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorousatomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived fromgolden captionsconstructed via a novelCircular Peer-Review consensuspipeline and then distilled into adual-stream systemofMust-Right(essential facts) andEasy-Wrong(fine-grained details) rubrics. Crucially, PerceptionRubrics implements aGated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) TheReliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2)Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3)Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.28322
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.28322 in a model README.md to link it from this page.
Datasets citing this paper1
#### M1chaelPeng/PerceptionRubrics Viewer• Updated1 day ago • 1.04k • 10
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.28322 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@yanawei_: Thanks AK for featuring our work! More details: http://weiyana.github.io/PerceptionRubrics…
Introduces PerceptionRubrics, a rubric-based evaluation framework for multimodal AI that shifts from holistic matching to atomic auditing using 1,038 images and over 10,000 instance-specific rubrics, with gated scoring to enforce strict perceptual fidelity.
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
PerceptionBench is a benchmark designed to evaluate atomic visual perception capabilities of Multimodal Large Language Models (MLLMs), using a bottom-up taxonomy of ten atomic perceptual capabilities. Results across 16 frontier MLLMs show no model reaches 60% accuracy, indicating visual perception remains largely unsolved.
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies perceptual judgment bias in multimodal LLM judges, where they over-reward fluent but visually wrong responses, and proposes a dataset PPJD and a trained model Perception-Judge using GRPO with batch-ranking reward to mitigate this bias and improve perception-grounded evaluation.
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
CalibratedRubric is a task-adaptive framework for building compact, measurable rubric banks for open-ended LLM evaluation, using Bayesian measurability filtering and IRT-based selection to improve human-gold agreement and rank fidelity across financial, healthcare, general, and legal benchmarks.
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
C2 proposes a scalable rubric-augmented reward modeling framework that trains a cooperative rubric generator and critical verifier exclusively from binary preferences, eliminating the need for costly rubric annotations while achieving up to 6.5 point gains on RM-Bench.