Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

Hugging Face Daily Papers Papers
code-verification agentic-ai reinforcement-learning competitive-programming llm-evaluation mixture-of-experts

Summary

该论文提出了一个让模型仅凭题面和源码判断并诊断候选程序正确性、并以生成反例作为可验证证据的任务，构建了 USACO-JUDGE 基准，并训练了 35B MoE 代理验证器 ARBITER，显著提升了代码验证、选择与改进效果。

Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit.
Original Article
View Cached Full Text

Cached at: 10/14/26, 06:53 PM

Paper page - Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

Source: https://huggingface.co/papers/2610.18760

Abstract

Test-timescalingforlargelanguagemodels(LLMs)caninvolvegeneratingmultiplecandidatesolutionsforselectionorrevisinganearlierattempt.Toguideselectionandrevision,weneedaverifierthatcanestimatecandidatecorrectnessandprovideevidenceoffailure.Incompetitiveprogramming,however,officialjudgingsystemsprovideauthoritativeverdictsunderfinitesubmissionbudgets,withoutrevealingfailureexplanationsorfailinginputs.Toprovideinformativefeedbackwithoutconsumingofficialsubmissions,weformulateataskinwhichamodeljudgesanddiagnosesacandidateprogramusingonlytheproblemstatementandsourcecode,withgeneratedcounterexamplesservingascheckableevidence.Toevaluatethistask,webuildUSACO-JUDGEfromtheUSACO2026contests,usingacceptedandunacceptedprogramswrittenbyhumanexpertsandmultipleLLMs.WethentrainARBITER,a35Bmixture-of-experts(MoE)agenticverifier,withreinforcementlearninginagenericexecutionsandbox.Trainingraisescounterexamplesuccessonincorrectprogramsfrom40.61%to62.67%andmeanbinaryverdictaccuracyonUSACO-JUDGEfrom71.41%to87.41%,approachingfrontierverificationperformance.Beyondverification,ARBITERoutperformsbaselinesinbothcodeselectionandrefinement.InanIOI2026casestudy,ARBITER-guidedrefinementraisesGPT-OSS-120B’sexpectedscoreabovethegold-medalthresholdundertheofficialsubmissionlimit.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2610\.18760

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.18760 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.18760 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.18760 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Verifier-Grounded Agentic RL for Provably Correct Code

arXiv cs.LG

提出一种以程序验证器为奖励信号的智能体强化学习框架，通过读取验证器诊断信息（而非二元判决）并加入完整性审计，显著提升Qwen3系列模型在DafnyBench和SWE-Proof等验证基准上的正确率（最高达100%）。

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

Hugging Face Daily Papers

本文提出 ReVeR，一种点式（pointwise）rubric 奖励模型，仅通过分析指令本身生成硬性规则与分级原则，并借助工具增强的独立评判来评估响应，避免了从候选响应中推导标准所引入的偏置。实验表明 ReVeR 在偏好准确率、测试时扩展及下游策略优化方面优于成对 rubric 奖励模型。