Thanks AK for featuring our work! More details: http://weiyana.github.io/PerceptionRubrics…

PerceptionRubrics

Source: https://weiyana.github.io/PerceptionRubrics/

PerceptionRubrics Calibrating Multimodal Evaluation to Human Perception

ICML 2026

Hongbo Peng*2,Yanlin Lai*3,Liang Zhao2,Kangheng Lin2,En Yu2,Keyu Lv2,Han Zhou2, Yin Tang4,Haodong Li2,Mitt Huang2,Hangyu Guo2,Jianjian Sun2,Zheng Ge2,Xiangyu Zhang2,Daxin Jiang2,Vishal M. Patel†1

**▶Johns Hopkins University▶StepFun▶Tsinghua University▶**Independent *Core Contribution†Corresponding Author

Overview

Current perception benchmarks suffer from anevaluation paradox: leaderboards are increasingly saturated, yet models remain perceptually brittle in real-world use. We introducePerceptionRubrics, a rubric-based evaluation framework that shifts assessment from holistic semantic matching torigorous atomic auditing. It pairs1,038information-dense images withover 10,000instance-specific rubrics, derived fromGolden Captionsconstructed through a novelCircular Peer-Reviewconsensus pipeline and distilled into a dual-stream system ofMust-Right(essential facts) andEasy-Wrong(fine-grained details) rubrics.

Crucially, PerceptionRubrics implements aGated Scoringmechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields three insights:

(1)**The Reliability Gap.**Models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains such as GUIs. (2)**Open–Closed Stratification.Contrary to reasoning trends, a persistent~8%**perception deficit separates the open-source frontier from proprietary leaders. (3)**Human-Aligned Rigor.**Our gated metrics substantially out-align conventional benchmarks with human preference, validating that strict perceptual fidelity is a prerequisite for reliable generation.

Fig 1:**Motivation.**Existing benchmarks can favor responses with key omissions, whereas humans prefer responses that capture more perceptually important details. PerceptionRubrics more clearly distinguishes model capabilities than DetailCaps and DOCCI.

Benchmark Construction

To bypass the visual-grounding gap of direct image-to-rubric generation, we adopt acaption-centric pipeline. We first transcribe each image into a comprehensiveGolden Caption, then distill rubrics from it.

• **Step 1 — Circular Peer-Review.**Three top-tier MLLMs act as a “jury-and-generator” ensemble: they generate independent descriptions, then iteratively compare, rank, and rewrite them against the visual evidence to synthesize a superior consensus caption.
• **Step 2 — Strict Consensus Filtering.**Human experts act as final verifiers under a discard-on-divergence protocol—only high-consensus samples are lightly verified and kept, focusing human effort on high-confidence data.
• Dual-Stream Rubrics.From each Golden Caption we extractMust-Rightrubrics (a priori essential facts) andEasy-Wrongrubrics (a posteriori pitfalls mined from a pool of model error patterns) using domain-specific adaptive prompts.

Fig 2:Construction pipeline.Golden Captions are synthesized via circular peer-review (top), then serve as anchors to generate Must-Right and Easy-Wrong rubrics via domain-specific prompting (bottom). The resulting benchmark contains1,038images and10,718rubrics (4,053Must-Right +6,665Easy-Wrong; ~10.3 per image) spanningseven domains: Natural Scene, Document & OCR, Digital UI/UX, Structured Data, STEM & Expert, Logic & Puzzle, and Creative & Cultural. Golden Captions average770words—5–6× denser than prior detailed-captioning benchmarks.

Gated Scoring

We use an LLM-as-a-Judge to verify each rubric as a boolean (True/False), then apply anon-linear, gatedaggregation that mirrors human error sensitivity:

• **Must-Right as the Gate.**If a response failsanyMust-Right criterion, the gate closes (G= 0) and the score collapses to zero—a single fatal hallucination is a binary failure, not a minor fluctuation.
• **Easy-Wrong for Differentiation.**For responses that pass the gate (G= 1), the final score is the pass rate over the Easy-Wrong rubrics, rewarding robustness against subtle, density-rich cognitive traps.

This ensures a high score reflects not coarse semantic proximity but genuine perceptual reliability, distinguishing acceptable approximations from catastrophic failures.

Main Results

We evaluate25leading MLLMs. PerceptionRubrics reveals a pronounced performance stratification obscured by traditional holistic benchmarks. The strongest model reaches only70.07%, while a widely used proprietary model (GPT-4o) scores just12.59%. Performance is highest on natural scenes and lowest in theGUIdomain, and the best open-source model still trails the proprietary state-of-the-art byover 8%—a gap that persists despite convergence on reasoning tasks.

#ModelParamsDocLogicCreativeGUINaturalSTEMStructuredOverall Table 1:Fine-grained performanceacross seven domains. Click a column header to sort; per-column best is highlighted. All values are percentages (%).

Analysis

The Reliability Gap & Failure Modes

ComparingAtomic Accuracy(mean pass rate over individual rubrics) with the stricterMust-Right Pass Rate(all constraints satisfied) exposes a systematicReliability Gap: models pass most atomic checks yet fail their strict conjunction. The gap narrows as capability increases. TheGUIdomain dominates perceptual failures across models.

Fig 3:Failure analysis.(Left) Distribution of error sources across models. (Right) Atomic Accuracy vs. the stricter Must-Right-All-Pass rate.

Consistency of Perceptual Capabilities

We find a near-perfect linear correlation (R2≈ 0.98) between Must-Right Pass Rate and Easy-Wrong accuracy: models that fail to ground essential facts inevitably struggle with subtle details and hallucination. Robust fine-grained understanding critically depends on foundational perception.

Fig 4:Correlationbetween basic perceptual reliability and hallucination resistance.

Alignment with Human Preference

Against theVision Arenahuman-preference leaderboard, PerceptionRubrics shows the strongest alignment among compared captioning benchmarks, achieving aPearson 0.916and a perfectSpearman 1.000rank correlation. In contrast, DOCCI and DetailCaps assign nearly indistinguishable (or even anti-correlated) scores to models with markedly different human ratings.

Fig 5:**Human alignment.**Benchmark score vs. Vision Arena Elo across overlapping models. PerceptionRubrics correlates most strongly with human preference.

Resistance to Length Bias

Caption length exhibits a negligible relationship with PerceptionRubrics scores, indicating the metric rewards precise, verifiable perception rather than verbosity.

Fig 6:**Length bias.**Score vs. response word count for representative models.## Evaluation Robustness & Rubric Coverage

Swapping the judge model preserves the ranking order, and evaluation stability improves monotonically as rubric coverage increases—confirming the robustness of both the rubric-generation pipeline and the resulting metric to judge choice and sampling variability.

Fig 7: (Left) Consistent rankings under different judges. (Right) Stability improves with rubric coverage.

BibTeX

If you find this work useful, please consider citing it:

@article{wei2026perceptionrubrics,
  title={PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception},
  author={Wei, Yana and Peng, Hongbo and Lai, Yanlin and Zhao, Liang and Lin, Kangheng
  and Yu, En and Lv, Keyu and Zhou, Han and Tang, Yin and Li, Haodong and Huang, Mitt
  and Guo, Hangyu and Sun, Jianjian and Ge, Zheng and Zhang, Xiangyu and Jiang, Daxin
  and Patel, Vishal M.},
  journal={arXiv preprint arXiv:2606.28322},
  year={2026}
}

Thanks for your reading and support! For questions or suggestions, feel free to contact us:[email protected],[email protected]

AK (@_akhaliq): PerceptionRubrics

Calibrating Multimodal Evaluation to Human Perception