Small Language Models as Judges for Rubric-Based Reinforcement Learning
Summary
This paper demonstrates that small probe-based language models can replace larger generative models for rubric-based reinforcement learning rewards, enhancing efficiency while preserving agreement and transferability.
View Cached Full Text
Cached at: 09/03/26, 07:53 PM
Paper page - Small Language Models as Judges for Rubric-Based Reinforcement Learning
Source: https://huggingface.co/papers/2608.30005
Abstract
Small probe-based judges can replace large generative models for rubric-based reinforcement learning rewards, improving efficiency while maintaining agreement and transferability.
Rubric-based reinforcement learningextends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models:Generative verdicts, Yes/NoLogprob margins, andProbe judges. Across both datasets, theQwen3-1.7BProbe judge achieves the strongestcriterion-level agreementamong these methods, outperforming Generative and Logprob judges. Used as aGRPOreward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7times more reward-judge time. Task and domain transfer experiments further suggest thatProbe judgespreserve criterion-level reward structure across settings.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.30005
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.30005 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.30005 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.30005 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models
SKILLER is a natural-language-driven reinforcement learning framework that automatically generates executor-specific agent skills for small language models, matching closed-source performance at lower cost.
Reinforcing Recursive Language Models (18 minute read)
The article explores reinforcement learning fine-tuning of small (4B) recursive language models (RLMs) to perform evidence selection from scientific documents, showing that RL-trained 4B models match Claude Sonnet 4.6 performance at a fraction of the size and cost.
Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
This paper systematically investigates instability in reinforcement learning for small language model agents (70-500M parameters), identifying three failure modes and proposing robust techniques including a merge-and-reinitialize adapter approach and safety mechanisms; it achieves stable convergence and improved win rates.
Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
Presents B1ade, a minimalist RAG architecture with a 335M zero-training embedding model and a 1B SLM trained via GRPO on 723M tokens, showing emergent attribution behavior and competitive QA performance without large-scale pretraining.
Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
This paper systematically investigates failure modes in reinforcement learning for small language models (70-500M parameters) using PPO, identifies silent LoRA freezing, numerical overflow, and catastrophic policy collapse, and proposes a robust system with merge-and-reinitialize adapters, float32 precision, and a safety mechanism. The approach converges stably and outperforms baselines with less data.