PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment
Summary
PRM-as-a-Judge 1.5 is a toolkit that provides fine-grained metrics and reliability tools for evaluating embodied robotic models, moving beyond binary success rates to assess process progress and execution quality.
View Cached Full Text
Cached at: 08/17/26, 07:45 AM
Paper page - PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment
Source: https://huggingface.co/papers/2608.14284 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
PRM-as-a-Judge 1.5 provides fine-grained process metrics and reliability tools to evaluate embodied robotic models beyond binary success rates.
Fine-grained robotic evaluation matters for understandingembodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turnsrollout videosinto denseprogress curvesand derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterizefailure-side progress,post-drawdown recovery, and success-sideexecution quality, helping users understand embodied model capability. Based on therollout videosfrom benchmarks, we perform a comprehensive assessment of theembodied models, providing some fine-grained metric results and key findings. We further introduceRoboPulse++to evaluate the reliability ofprocess reward models(PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2608\.14284
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.14284 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.14284 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.14284 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Process Rewards with Learned Reliability
BetaPRM is a process reward model that predicts both a step-level success probability and the reliability of that prediction using a Beta belief from Monte Carlo continuations, enabling adaptive computation allocation that reduces token usage by up to 33.57% while improving accuracy.
SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification
SCI-PRM introduces a tool-aware Process Reward Model for scientific reasoning, trained on the SCIPRM70K dataset featuring 'Chain-of-Tool' trajectories that interleave reasoning with scientific tool execution. It enables effective test-time scaling and serves as a dense reward signal in reinforcement learning, outperforming proprietary models like GPT-5-Mini on tool-calling steps across scientific benchmarks.
PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
Introduces PReMISE, a framework for discovering and auditing policy-level rubrics for LLM judges along four axes: structural adequacy, reliability, preference fit, and adversarial robustness.
PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis
This paper introduces PRISM, a four-stage data synthesis framework for training multimodal LLMs to follow prioritized rubrics, and PRISM-Eval, a judge-free evaluation suite. With only 10K samples, PRISM lifts Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on PRISM-Eval while preserving general benchmark performance, and gains transfer to other open-source MLLMs.
Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
RAMP is a production-grounded evaluation framework for LLM agents that exposes significant capability degradation invisible to static benchmarks, showing task completion rates collapsing from 100% to 20% across serial workflows. The framework assesses 15 mainstream models on realistic compiler-construction workloads with complex toolchain interactions and staged recovery mechanisms.