Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

Hugging Face Daily Papers Papers
reinforcement-learning audio-reasoning rewards-design ai-research acoustic-understanding

Summary

This paper introduces AudioRubrics, a reinforcement learning framework that uses self-evolving, audio-grounded rubric rewards to supervise audio reasoning, outperforming existing baselines across three benchmarks.

Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.
Original Article
View Cached Full Text

Cached at: 08/10/26, 06:13 AM

Paper page - Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

Source: https://huggingface.co/papers/2608.02831 Published on Aug 3

·

Submitted byhttps://huggingface.co/ParadiseYu

Fangxuon Aug 10

Abstract

Audioreasoningisessentialformachineunderstandingoftheacousticworld.Reinforcementlearningwithverifiablerewardscanelicitsuchreasoning,yetexistingrewarddesignsarecomplementaryintheirlimitations:outcome-basedrewardssuperviseonlythefinalanswerandletthemodelreachitwithoutattendingtotheaudio,whereasprocess-basedrewardsscorethereasoningitselfbutrelyoncoarse,hand-crafted,andfixedcriteriathatneitheradapttoeachquestionnorstaygroundedintheacousticevidence.Moreover,questionsdifferinwhattheydemand,withsomehingingonperceptionandothersonmulti-stepreasoning,andanystaticcriterionweakensasthepolicyimproves.Supervisingthereasoningprocesswithfine-grained,audio-grounded,andadaptiverewardsisthereforecrucial,yetchallengingsincesuchrewardsareimpracticaltodesignbyhandforeverysample.Tothisend,weintroduceAudioRubrics,areinforcementlearningframeworkthatsupervisesaudioreasoningwithself-evolving,audio-groundedrubricrewards.AudioRubricssynthesizesper-samplerubricsfromtherawwaveformand,conditionedonthemodel’sownrollouts,regeneratesandreweightscriteriapergroup,supplyingacontinuouslearningsignalthatkeepstargetingthecurrentpolicy’sweaknessesasstaticcriteriasaturate.ComprehensiveevaluationsacrossthreeaudioreasoningbenchmarksrevealthatAudioRubricssubstantiallyoutperformsawiderangeofopen-sourceandtraining-basedbaselines.Furthermore,ouranalysisshowsthatthegainsscalewiththecapabilityoftherubricgeneratorandjudge,andAudioRubricsconvergestoastablereasoninglengththatavoidsbothdegeneratecollapseandunboundedgrowth.Theimprovementinaudioperceptionfurtherdemonstratestheeffectivenessofanchoringsupervisionintheacousticevidence.Ourprojectpageisavailableathttps://audiorubrics.github.io.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2608\.02831

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### umd-zhou-lab/AudioRubrics Audio-Text-to-Text• 12B• Updated5 days ago • 18

Datasets citing this paper1

#### umd-zhou-lab/AVQA-Audio-Rubrics Viewer• Updatedabout 4 hours ago • 40.4k • 89

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Hugging Face Daily Papers

This paper introduces POW3R, a policy-aware rubric reward framework for reinforcement learning with verifiable rewards (RLVR). It shows that static rubric aggregation misallocates learning signal, and POW3R achieves faster convergence and better performance across multiple settings.

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

Hugging Face Daily Papers

This paper introduces RubricEM, a reinforcement learning framework that uses rubric-guided policy decomposition and reflection-based meta-policy evolution to train deep research agents for long-form tasks. The resulting RubricEM-8B model demonstrates strong performance on long-form research benchmarks by leveraging stage-aware planning and denser semantic feedback.

V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning

Hugging Face Daily Papers

Introduces a rubric-based reinforcement learning method to improve visual faithfulness in vision-language models by decomposing responses into atomic propositions and scoring them on visual faithfulness, reasoning consistency, and instruction following.

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

Hugging Face Daily Papers

MetaRubric 通过交替进行证据感知的策略优化与响应引导的评分标准适配，解决基于评分标准的强化学习中「空洞奖励（Vacuous Credit）」问题，在 PubMedQA 上比静态评委 GRPO 提升 6.00–20.40 个百分点。