PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Summary
PerceptionRubrics introduces a rubric-based evaluation framework for multimodal models that uses atomic auditing and gated scoring to better align benchmark scores with human perception, revealing reliability gaps and open-closed stratification.
View Cached Full Text
Cached at: 07/02/26, 07:47 AM
Paper page - PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Source: https://huggingface.co/papers/2606.28322 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
PerceptionRubrics presents a rubric-based evaluation framework that identifies gaps between benchmark scores and real-world performance through atomic auditing and gated scoring mechanisms.
We introduce PerceptionRubrics, arubric-based evaluationframework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorousatomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived fromgolden captionsconstructed via a novelCircular Peer-Review consensuspipeline and then distilled into adual-stream systemofMust-Right(essential facts) andEasy-Wrong(fine-grained details) rubrics. Crucially, PerceptionRubrics implements aGated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) TheReliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2)Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3)Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.28322
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.28322 in a model README.md to link it from this page.
Datasets citing this paper1
#### M1chaelPeng/PerceptionRubrics Viewer• Updated1 day ago • 1.04k • 10
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.28322 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@yanawei_: Thanks AK for featuring our work! More details: http://weiyana.github.io/PerceptionRubrics…
Introduces PerceptionRubrics, a rubric-based evaluation framework for multimodal AI that shifts from holistic matching to atomic auditing using 1,038 images and over 10,000 instance-specific rubrics, with gated scoring to enforce strict perceptual fidelity.
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies perceptual judgment bias in multimodal LLM judges, where they over-reward fluent but visually wrong responses, and proposes a dataset PPJD and a trained model Perception-Judge using GRPO with batch-ranking reward to mitigate this bias and improve perception-grounded evaluation.
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
C2 proposes a scalable rubric-augmented reward modeling framework that trains a cooperative rubric generator and critical verifier exclusively from binary preferences, eliminating the need for costly rubric annotations while achieving up to 6.5 point gains on RM-Bench.
Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
This paper introduces the Auto-Rubric as Reward (ARR) framework, which externalizes implicit preference knowledge into explicit rubrics for multimodal alignment. It proposes Rubric Policy Optimization (RPO) to stabilize policy gradients, achieving better performance in text-to-image and image editing tasks.
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
DeltaRubric is a research paper introducing a two-step multimodal preference evaluation approach using a single MLLM to improve reward modeling reliability through joint planning and verification.