从结构到有效性：机制引导的大语言模型评分量规生成与优化

# From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
Source: [https://arxiv.org/html/2610.13375](https://arxiv.org/html/2610.13375)
Feifei Qian1,2, Xuan Chen2, Lixin Cui3, Chaoyang Mei2,Fanlin Meng2, Shuyao Xiao1, Ming Cui2, Junxi Yin2, Lu Bai11School of Artificial Intelligence, Beijing Normal University2KE Holdings Inc\.3School of Information, Central University of Finance and Economicsfeifei\_qian@mail\.bnu\.edu\.cnchenxuan046@ke\.combailu@bnu\.edu\.cn

###### Abstract

Rubrics have been widely used to evaluate and optimize language models on open\-ended tasks\. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective\. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation\. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion\-level judgments, and sophisticated score aggregation provides limited additional gains\. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement \(ARTS\-Hybrid\), which combines adaptive tree construction with semantic\-first, response\-calibrated refinement\. Experiments across benchmarks show that ARTS\-Hybrid improves both end\-to\-end evaluation performance and intrinsic rubric quality\. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS\-Hybrid provide effective signals for model optimization\.

## 1Introduction

Large language models \(LLMs\) have made remarkable progress in recent years and are increasingly applied to open\-ended tasks\([Shi et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib1)\)\. Unlike tasks with verifiable outcomes, such as mathematical reasoning and code generation, open\-ended tasks often admit multiple valid responses and lack objective ground\-truth signals\. Consequently, reliable evaluation has become a critical bottleneck in the development of LLMs\. LLM\-as\-a\-Judge\([Gu et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib2);[Zheng et al\., 2023](https://arxiv.org/html/2610.13375#bib.bib31)\)has emerged as a scalable alternative to costly human evaluation, with many existing methods relying on a single holistic score to assess response quality\. However, this scalar evaluation collapses multiple dimensions of response quality, such as correctness, completeness, and relevance, into a coarse\-grained judgment, making it difficult to provide fine\-grained feedback for model improvement\.

Inspired by the long\-standing use of rubrics in educational assessment\([Brookhart, 2018](https://arxiv.org/html/2610.13375#bib.bib3)\), rubric\-based evaluation offers a structured alternative by decomposing overall response quality into explicit task\-specific criteria\. In the context of LLMs, rubrics are increasingly becoming an important tool for model evaluation\([Hashemi et al\., 2024](https://arxiv.org/html/2610.13375#bib.bib5)\)and optimization\([Gunjal et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib4)\)\. Early rubric construction methods primarily rely on language models to generate evaluation criteria directly from task instructions\([Viswanathan et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib6);[Lin et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib7)\), often resulting in incomplete coverage and inconsistent levels of granularity\. Subsequent contrastive methods derive more discriminative evaluation criteria by identifying differences between preferred and rejected responses\([Liu et al\., 2026b](https://arxiv.org/html/2610.13375#bib.bib8);[Tan et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib9);[Xie et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib10)\)\. More recent work has introduced structural decomposition\([Gao et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib11);[Shen et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib12)\)and iterative refinement\([Zhang et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib13);[Li et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib14)\)to further improve the coverage and reliability of the rubric\. In parallel, online approaches dynamically update rubrics based on policy model rollouts\([Sheng et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib15);[Shao et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib16)\), allowing evaluation criteria to evolve with the capabilities of the policy model\. However, whether rubrics are constructed offline or dynamically evolved with the policy model, existing methods typically rely on aggregate outcomes, such as preference accuracy or downstream rewards, to guide their optimization\. Although these metrics quantify end\-to\-end performance on a given dataset, the mechanisms underlying rubric effectiveness remain largely unexplained\. The effectiveness of a rubric is influenced by multiple interdependent factors, such as how the criteria are constructed and organized\. Consequently, the observed gains cannot be reliably attributed to specific design choices, providing limited systematic guidance on how rubrics should be structured or refined\. This highlights the need for a systematic mechanistic analysis to disentangle the effects of individual design choices\.

In this paper, we move beyond end\-to\-end comparisons and dissect rubric effectiveness along three stages, including how evaluation criteria are structurally organized, how individual criteria are operationalized, and how criterion\-level scores are aggregated\. Through systematic mechanistic analyses, we uncover three main findings\. First, the structural decomposition improves criterion coverage and supports more targeted diagnosis\. Second, the semantically sound evaluation criteria do not necessarily yield discriminative and stable judgments, motivating the use of response\-based calibration evidence\. Third, with consistent criterion content and criterion\-level judgments across settings, the alternative score aggregation strategies provide only limited additional gains\.

Guided by these findings, we introduceAdaptiveRubricTreeSearch withHybridrefinement \(ARTS\-Hybrid\), a mechanism\-guided framework that formulates rubric construction as a constrained search over hierarchical criterion structures\. Specifically, ARTS\-Hybrid constructs rubrics through structured decomposition, where each expansion follows a semantic\-first, response\-calibrated selection process to identify the child criteria\. We evaluate ARTS\-Hybrid in terms of end\-to\-end evaluation performance and intrinsic rubric quality\. In addition, we examine whether its benefits can be transferred to a standalone rubric generator and whether the generated rubrics provide effective reward signals for policy optimization\. Our contributions are summarized as follows:

- •A systematic mechanistic analysis of rubric effectiveness\.We conduct a systematic analysis of rubric design by disentangling three interdependent factors, including criterion organization, criterion operationalization, and score aggregation\.
- •A mechanism\-guided generation and refinement framework\.We translate these findings into ARTS\-Hybrid, which integrates structured decomposition with semantic\-first, response\-calibrated refinement\. Experimental results show that ARTS\-Hybrid achieves competitive performance across benchmarks while producing high\-quality rubrics\.
- •Validation beyond rubric\-based evaluation\.We further show that the benefits of ARTS\-Hybrid can be transferred to a standalone rubric generator and the generated rubrics provide effective reward signals for policy optimization\.

## 2Related Work

Automatic rubric generation and refinement\.Existing approaches to automatic rubric construction can be broadly categorized into three paradigms according to how rubrics are updated: one\-shot generation, offline iterative refinement, and online co\-evolution\. One\-shot methods generate evaluation criteria directly from each task instruction\([Viswanathan et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib6);[Lin et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib7)\), with some methods additionally conditioning on candidate responses\([Gunjal et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib4)\)\. Contrastive variants\([Liu et al\., 2026b](https://arxiv.org/html/2610.13375#bib.bib8);[Tan et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib9);[Xie et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib10)\)further compare preferred and rejected responses to identify the dimensions that distinguish stronger outputs from weaker ones, yielding more discriminative criteria\. Despite leveraging richer response evidence, one\-shot methods still construct rubrics in a single pass, with limited opportunities to correct missing or redundant criteria\. This has motivated the development of offline iterative refinement, which progressively validates and revises the generated rubrics\. Among them, RRD\([Shen et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib12)\)and RubricHub\([Li et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib14)\)adopt coarse\-to\-fine refinement strategies, decomposing broad criteria into finer\-grained, actionable evaluation items to improve coverage and applicability\. Q\-World\([Gao et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib11)\)combines hierarchical and lateral expansion to explore a broader evaluation space across different levels of abstraction, task scenarios, and evaluation perspectives\. These methods demonstrate the empirical benefits of hierarchical decomposition in rubric construction\. Concurrent work\([Yang et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib17)\)has also explored tree\-structured rubric construction, further extending hierarchical decomposition into an explicit structural representation\. Beyond offline refinement, more recent online approaches update rubrics throughout policy optimization using newly generated rollouts\([Sheng et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib15);[Shao et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib16);[Chen et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib32)\), allowing the evaluation criteria to evolve with the policy\.

Rubrics as rewards for LLM optimization\.Rubrics can serve not only as evaluation instruments but also as reward functions for open\-ended tasks\. Rubrics as Rewards \(RaR\)\([Gunjal et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib4)\)constructs instance\-specific criteria for each task and uses an LLM to judge whether a policy response satisfies each criterion\. The resulting criterion\-level scores are then aggregated into a scalar reward to guide policy optimization, extending reinforcement learning to open\-ended tasks without programmatically verifiable outcomes\. Subsequent work further improves the adaptability of rubric\-based rewards\. InfiMed\-ORBIT\([Wang et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib18)\)dynamically generates case\-conditioned criteria for medical dialogues and employs them in incremental reinforcement learning, whereas DR\-Tulu\([Shao et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib16)\)continually updates rubrics using rollouts from the current policy, allowing the reward specification to co\-evolve with model capabilities and emerging failure modes\. Meanwhile, OptimSyn\([Fan et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib19)\)optimizes rubrics from the perspective of training utility, using influence estimates of synthetic examples on the target model as feedback to align rubric optimization more directly with downstream performance\.

Overall, existing studies have substantially improved rubric generation and refinement\. However, they primarily focus on how to construct or update better rubrics, while the mechanisms underlying their effectiveness remain underexplored\. In particular, it remains unclear how structural decomposition, criterion\-level validity, and score aggregation each contribute to overall rubric effectiveness\.

## 3What Makes an Effective Rubric? A Mechanistic Analysis

To address this gap, we conduct a systematic mechanistic analysis of rubric effectiveness centered on three research questions\.RQ1:How does structural organization affect rubric quality?RQ2:What makes an individual criterion effective in practice?RQ3:Does score aggregation affect rubric evaluation? We construct a controlled analysis set of 1,500 instances through stratified sampling from HealthBench\([Arora et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib20)\)and RubricBench\([Zhou et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib21)\)\. Both datasets provide expert\-authored atomic evaluation criteria, which we treat as gold rubric leaves\. For each query, we manually construct and validate agold rubric treeby recursively grouping semantically related leaves into higher\-level evaluation dimensions\. Additional details on the datasets and sampling procedure are provided in Appendix[A\.1](https://arxiv.org/html/2610.13375#A1.SS1)\.

### 3\.1RQ1: How Does Structural Organization Affect Rubric Quality?

To answer RQ1, we examine two potential roles of structural organization in rubric\-based evaluation\. We investigate whether it helps the generator discover evaluation criteria more systematically and whether the resulting hierarchy provides useful context for judging individual criteria\.

Structural organization for criterion generator\.We first examine whether structural organization enables more systematic exploration of the evaluation space, rather than merely increasing the number of generated criteria\. We compare tree\-structured and flat rubric generation using the same generator\. The hierarchical approach recursively constructs a rubric tree through structured decomposition, whereas the flat approach directly generates a set of criteria without an explicit hierarchy\. To control for the additional generator calls introduced by recursive decomposition, we further includeFlat\-Multi, which independently runs flat generation the same number of times as the corresponding tree construction for each instance and reports the average performance\. We use GLM\-4\.7\-Flash as the generator and align the generated criteria with expert\-annotated gold rubrics to measure coverage and novelty, with metric definitions provided in Appendix[A\.2](https://arxiv.org/html/2610.13375#A1.SS2)\. We use DeepSeek\-V4\-Flash\-GA and Llama3\.1\-405B\-Instruct as two independent judges\.

Table 1:Comparison between tree\-structured and flat rubric generation\.As shown in Table[1](https://arxiv.org/html/2610.13375#S3.T1), tree\-structured rubric generation outperforms bothFlatandFlat\-Multiin criterion coverage and valid novelty across judge models\. Notably, the gains persist when the number of generator calls is matched, indicating that they cannot be explained by additional generation alone\. These results suggest that hierarchical decomposition provides a more systematic scaffold for exploring the evaluation space and recovering fine\-grained criteria that flat generation may overlook\.

##### Hierarchical context for criterion judgment\.

To examine whether hierarchical structure provides useful semantic context for criterion\-level judgments, we construct controlled counterfactual responses based on the gold rubric trees\. We first verify whether the reference response satisfies all gold criteria and revise it when necessary to obtain a canonical correct response\. Starting from this response, we apply minimal local edits to construct three types of counterfactuals with increasing structural scope: \(1\) single\-leaf, violating one gold leaf criterion; \(2\) within\-dimension multi\-leaf, jointly violating multiple leaves under the same semantic dimension; and \(3\) cross\-dimension multi\-leaf, introducing violations across multiple semantic dimensions\. Detailed editing operations are provided in Appendix[A\.3](https://arxiv.org/html/2610.13375#A1.SS3)\. We formulate this evaluation as a controlled criterion\-level violation detection task\. Given a query, a counterfactual response, and its gold rubric, the judge model determines whether each rubric leaf is satisfied\. The predefined intervention targets serve as ground\-truth labels\. We evaluate the performance using exact\-match accuracy, i\.e\., a prediction is correct only if the judge model identifies all and only the violated leaves\. We compare three ways of using rubric structure during judgment\.Flatevaluates each leaf independently without hierarchical context\.Treeuses the rubric hierarchy for top\-down routing, selecting potentially violated branches before making leaf\-level judgments\.Pathalso draws on the rubric tree but removes intermediate routing decisions, instead providing the full root\-to\-leaf semantic path when evaluating every criterion\.

Table 2:Criterion\-level violation detection under different uses of rubric hierarchy\. All results are reported as percentages, and colored values indicate absolute changes from Flat in percentage points\.As shown in Table[2](https://arxiv.org/html/2610.13375#S3.T2), the top\-downTreerouting does not outperformFlatunder either judge model\. This does not imply that hierarchical structure is inherently ineffective\. Instead, the parent\-level routing introduces a decision bottleneck, i\.e\., if the judge fails to identify a violation at an internal node, the corresponding descendant leaves are excluded from further inspection, allowing a single routing error to obscure all violations within that branch\. By contrast,Pathconsistently outperformsFlatacross all six settings, suggesting that higher\-level evaluation dimensions provide useful semantic context for interpreting the scope of individual criteria\. Overall, hierarchical structure is more effective as a semantic scaffold for criterion judgment than as a direct basis for top\-down pruning without explicit recall safeguards\.

To verify that the gains fromPatharise from hierarchical semantics rather than merely increased context length, we introduce aShuffled\-Pathcontrol\. Specifically, we replace the intermediate dimensions along each root\-to\-leaf path with semantically mismatched dimensions at the same depth, while keeping all other inputs unchanged\. Detailed results are reported in Appendix[A\.4](https://arxiv.org/html/2610.13375#A1.SS4)\. Having established the role of structural organization, we next shift from how criteria are organized to the effectiveness of the individual criteria themselves\.

### 3\.2RQ2: What Makes an Individual Criterion Effective in Practice?

We characterize an effective criterion by two operational properties:discriminative powerandjudge stability\. The former captures whether a criterion can distinguish responses of different quality, while the latter reflects whether its judgments remain consistent across judge models\. We investigate how these properties can be realized in practice through two complementary mechanisms,semantic specificationandresponse\-based calibration\. We conduct this analysis on 750 preference instances from RubricBench\([Zhou et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib21)\)\. For each instance, we assess discriminative power and judge stability using the two responses provided by the dataset and two independent judge models\. We measure discriminative power by the variance of criterion scores across two responses under a fixed judge, and judge stability by the variance in scores across judges for the same response and criterion\. In addition, we construct a calibration panel of six responses spanning strong, medium, and weak quality levels for each query\. These responses serve only as signals for candidate criterion selection and are not used in the final evaluation\. Detailed construction and scoring procedures are provided in Appendix[B\.2\.1](https://arxiv.org/html/2610.13375#A2.SS2.SSS1)\.

Table 3:Effects of semantic specification and response\-based calibration on criterion\-level properties and pairwise evaluation accuracy, evaluated using Llama\-3\.3\-70B\-Instruct and Qwen2\.5\-72B\-Instruct\([Qwen et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib29)\)as judge models\.Invaliddenotes the percentage of selected candidate revisions that fail the semantic\-validity constraint and is not applicable toDirect\.Can semantic specification induce operational effectiveness?We first examine whether specifying the desired properties during criterion generation is sufficient to realize them in practice\. Specifically, we compare two settings,DirectandSemantic\.Directgenerates criteria without additional constraints, whereasSemanticexplicitly encourages criteria that are semantically valid, discriminative, and judge\-stable\. We assess the resulting criteria in terms of discriminative power, cross\-judge variance, and pairwise evaluation accuracy\. As shown in Table[3](https://arxiv.org/html/2610.13375#S3.T3), semantic specification shifts both operational propertiesin the desired direction, but only marginally\. This reveals a gap between specifying desirable criterion properties and realizing them in actual evaluation behavior\.

Table 4:Pairwise evaluation accuracy under different score aggregation schemes\.Grounding criterion effectiveness through response\-based calibration\.The limited gains from semantic specification motivate us to introduce empirical evidence from actual model responses\. Rather than assessing criterion quality solely from its semantic description, response\-based calibration evaluates how candidate criteria behave on a panel of calibration responses, providing direct evidence of their discriminative power and judge stability\. We compare two settings,CalibrationandSemantic\+Calibration, both operating on the same fixed pool of local rubric revisions\. The former selects candidates based on their behavioral characteristics on calibration responses, whereas the latter applies the same utility only after semantic filtering\. Detailed definitions are provided in Appendix[A\.5](https://arxiv.org/html/2610.13375#A1.SS5)\. As shown in Table[3](https://arxiv.org/html/2610.13375#S3.T3),Calibrationachieves the highest discriminative power and the lowest cross\-judge variance\. However, 17\.84% of the selected criteria are semantically invalid, indicating that operational effectiveness alone does not ensure alignment with the underlying task requirements\. In contrast, applying semantic constraints before calibration eliminates invalid selections while retaining most of the gains in discrimination and stability\. Consequently,Semantic\+Calibrationachieves the highest average pairwise evaluation accuracy\. These results show that semantic constraints are necessary to prevent task\-misaligned criteria, while response\-based calibration is needed to identify criteria that are discriminative and stable in practice\.

### 3\.3RQ3: Does Score Aggregation Affect Rubric Evaluation?

RQ1 and RQ2 examine the roles of criterion organization and criterion\-level effectiveness in rubric\-based evaluation respectively\. We now turn to the final stage, where criterion\-level judgments are aggregated into an overall score\. By fixing the criterion content and varying only the aggregation strategy, we assess whether weighting and hierarchical aggregation contribute additional gains\. For theFlatsetting, the rubric generator independently assigns each leafllan importance scoreal∈\{1,…,5\}a\_\{l\}\\in\\\{1,\.\.\.,5\\\}, which is normalized across all leaves aswl=al/∑k∈ℒakw\_\{l\}=a\_\{l\}/\\sum\_\{k\\in\\mathcal\{L\}\}a\_\{k\}, whereℒ\\mathcal\{L\}denotes the set of leaf criteria\.Flat\+WUapplies the global weighting scheme introduced in RRD\([Shen et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib12)\)to leaf\-level scores before aggregation, whileTree\+WUapplies the same weighting locally within the hierarchy\. All methods share identical rubric leaves and criterion\-level judgments, differing only in how leaf scores are weighted and aggregated\.

As shown in Table[4](https://arxiv.org/html/2610.13375#S3.T4), neither hierarchical aggregation nor WU weighting yields consistent improvements\. With DeepSeek\-V4\-Flash\-GA as the judge,Flatachieves the highest accuracy, whileFlat\+WUprovides only a marginal gain of 0\.27 percentage points under Llama3\.1\-405B\-Instruct\. Overall, once criterion\-level evaluation is established, the final aggregation strategy provides limited additional gains, suggesting that rubric effectiveness depends more on how criteria are designed\.

## 4ARTS\-hybrid: Mechanism\-Guided Rubric Generation and Refinement

![Refer to caption](https://arxiv.org/html/2610.13375v1/Framework.png)Figure 1:Overview of the ARTS\-Hybrid framework\.Building on the above mechanistic findings, we introduceARTS\-Hybrid, a rubric generation and refinement framework that translates these insights into a principled construction process\. Given a query, ARTS\-Hybrid progressively constructs hierarchical evaluation criteria through adaptive tree search\. At each node expansion, semantic constraints first identify feasible candidate decompositions, while calibration responses provide empirical evidence to guide the final selection\. Once the search terminates, ARTS\-Hybrid performs global validation and refinement of the final leaf criteria, followed by normalized weight assignment to produce the final rubric\. We next describe the three key components of ARTS\-Hybrid in detail, including adaptive tree construction, constrained candidate selection, and global validation with weight assignment\.

### 4\.1Adaptive Rubric Tree Construction

Motivated by the findings from RQ1, we adopt a hierarchical tree structure for adaptive rubric construction\. Given a query, ARTS\-Hybrid first initializes a root rubric representing overall response quality\. The root serves only as a virtual structural node for organizing subsequent decomposition and does not directly contribute to the final score\. Rather than expanding the tree in a predetermined order, ARTS\-Hybrid maintains unprocessed nodes in a priority queue and performs best\-first search according to their importance and depth\. For each nodevvat depthdvd\_\{v\}, we independently estimate an importance scoreI⁡\(v\)∈\{1,…,5\}I\(v\)\\in\\\{1,\\ldots,5\\\}according to its relevance to the current task\. The expansion priority is defined as

P⁡\(v\)=I⁡\(v\)​γdv,P\(v\)=I\(v\)\\gamma^\{d\_\{v\}\},\(1\)whereγ∈\(0,1\]\\gamma\\in\(0,1\]denotes the depth discount factor\. At each iteration, ARTS\-Hybrid selects the highest\-priority unexpanded node from the queue and determines whether it is sufficiently atomic and independently assessable to serve as a leaf criterion\. Nodes that satisfy these conditions are finalized as leaves\. Otherwise, the rubric generator produces multiple finer\-grained child criteria, with the selected children added to the rubric tree and priority queue for further expansion\. This process continues until no expandable nodes remain or a predefined stopping condition is reached\.

### 4\.2Semantic\-First Response\-Calibrated Selection

Once an expandable node is selected, ARTS\-Hybrid selects the most suitable decomposition from the generated candidate set\. Following the findings from RQ2, semantic constraints and response\-based calibration play complementary roles\. Accordingly, ARTS\-Hybrid adopts a semantic\-first, response\-calibrated selection strategy\.

##### Semantic feasibility\.

For each expandable nodevv, the rubric generator first produces a pool of candidate child criteria\. These criteria are then combined into multiple candidate decompositions𝒞v\\mathcal\{C\}\_\{v\}, where eachC∈𝒞vC\\in\\mathcal\{C\}\_\{v\}represents a possible set of children for nodevv\. ARTS\-Hybrid applies a multi\-stage semantic validation procedure to determine which decompositions are feasible with respect to the queryqqand the current rubric treeTT\. The validation considers several aspects of rubric quality, including criterion clarity, task relevance, independent assessability, coverage of parent requirements, and semantic non\-redundancy among child criteria\. Only candidate decompositions that pass all semantic checks are retained in the feasible set

ℱv=\{C∈𝒞v\|SemValid⁡\(C,q,T\)=1\},\\mathcal\{F\}\_\{v\}=\\left\\\{C\\in\\mathcal\{C\}\_\{v\}\\;\\middle\|\\;\\operatorname\{SemValid\}\(C,q,T\)=1\\right\\\},\(2\)whereSemValid⁡\(C,q,T\)\\operatorname\{SemValid\}\(C,q,T\)indicates whether candidate decompositionCCsatisfies all semantic constraints with respect to the queryqqand the current rubric treeTT\. Semantically invalid candidates are filtered out at this stage and do not proceed to subsequent response\-based calibration\.

##### Response\-based calibration\.

As demonstrated in RQ2, semantic information alone is insufficient to determine how effectively a criterion operates on actual model responses\. We therefore use response\-based calibration to provide empirical evidence for selecting among semantically valid decompositions\. For each query, we construct a panel of six calibration responses spanning strong, medium, and weak quality levels and reuse it throughout the tree search\. For every semantically feasible candidate, two independent calibration judges evaluate each criterion across six calibration responses, yielding a6×26\\times 2response\-judge score matrix\. This matrix captures criterion behavior along two complementary dimensions\. The variation across responses reflects the criterion’s discriminative power, whereas variation across judges captures the consistency of its scoring behavior\. We further compare response\-score profiles across child criteria to identify behavioral redundancy\. Based on these signals, we define the following calibration objective for each feasible candidate decompositionCCas

J⁡\(C\)=αD​D​\(C\)\+αA​A​\(C\)\+αS​S​\(C\)−λred​R​\(C\),J\(C\)=\\alpha\_\{D\}D\(C\)\+\\alpha\_\{A\}A\(C\)\+\\alpha\_\{S\}S\(C\)\-\\lambda\_\{\\mathrm\{red\}\}R\(C\),\(3\)whereD⁡\(C\)D\(C\)measures response discrimination,A⁡\(C\)A\(C\)andS⁡\(C\)S\(C\)capture complementary aspects of cross\-judge consistency, andR⁡\(C\)R\(C\)penalizes redundant scoring behavior among child criteria\. Detailed definitions of these terms are provided in Appendix[B\.1](https://arxiv.org/html/2610.13375#A2.SS1)\. The final decomposition for nodevvis then selected from the semantically feasible set by

Cv∗=arg⁡maxC∈ℱv​J​\(C\)\.C\_\{v\}^\{\*\}=\\underset\{C\\in\\mathcal\{F\}\_\{v\}\}\{\\arg\\max\}\\;J\(C\)\.\(4\)In this way, semantic validation determines which decompositions are admissible, while response\-based calibration determines which admissible decomposition is most effective in practice\.

### 4\.3Global Validation and Weight Assignment

While the above local search ensures semantic validity at each expansion step, locally valid decompositions do not necessarily yield a coherent and complete rubric\. ARTS\-Hybrid therefore performs global validation once the tree search is complete, ensuring full coverage of the evaluation requirements, leaf\-level atomicity, and semantic non\-redundancy\. When global validation identifies remaining issues, ARTS\-Hybrid refines the leaf set by recovering missing evaluation requirements and restructuring existing criteria when necessary\. With the leaf set established, ARTS\-Hybrid assigns a normalized weight to each criterion\. Motivated by the findings from RQ3, we avoid propagating weights through the rubric hierarchy and instead estimate the importance of each leaf directly\. Specifically, for each leafll, the rubric generator assigns an importance scoreal∈\{1,…,5\}a\_\{l\}\\in\\\{1,\\ldots,5\\\}based on the query and the criterion itself\.

## 5Evaluation of ARTS\-Hybrid

We evaluate ARTS\-Hybrid in terms of end\-to\-end evaluation performance and intrinsic rubric quality\. In addition, we examine whether its benefits can be transferred to a standalone rubric generator and extended to policy optimization through rubric\-based rewards\.

### 5\.1Experimental Setup

Benchmarks and baselines\.We evaluate on preference benchmarks across diverse domains and difficulty levels, including JudgeBench\([Tan et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib9)\), RM\-Bench Chat\([Liu et al\., 2026a](https://arxiv.org/html/2610.13375#bib.bib22)\), the Chat and Chat\-Hard subsets of RewardBench\([Lambert et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib23)\), the Precise\-IF and Focus subsets of RewardBench 2\([Malik et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib24)\), and RubricBench\([Zhou et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib21)\)\. Each instance contains a query, a pair of candidate responses, and a preference label\. We report pairwise accuracy against human preferences as the primary metric\. We compare ARTS\-Hybrid with Direct Judge, Query\-Only, Response\-Aware, and Q\-World\([Gao et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib11)\)\. These baselines represent direct evaluation, query\-only rubric generation, response\-aware rubric generation, and structured rubric refinement, respectively\. We reproduce all baselines under the same evaluation protocol using the same generator and judge models\. Detailed implementation settings are provided in Appendix[B\.2](https://arxiv.org/html/2610.13375#A2.SS2)\.

### 5\.2Main Results

End\-to\-end evaluation performance\.Table[5](https://arxiv.org/html/2610.13375#S5.T5)reports pairwise evaluation accuracy across six benchmark settings\. ARTS\-Hybrid achieves the highest accuracy on five of the six settings and remains competitive on PreciseIF\. In particular, ARTS\-Hybrid improves over both direct judgment and one\-shot rubric generation on most benchmarks, while also outperforming Q\-World on five settings\. These results suggest that the mechanism\-guided construction process provides a robust basis for rubric\-based evaluation across diverse preference tasks\.

Table 5:Pairwise accuracy \(%\) across benchmarks\. The best result is highlighted in bold\.Intrinsic rubric quality\.We further evaluate the intrinsic quality of the generated rubrics using DeepSeek\-V4\-Flash as an independent evaluator, with results from an additional evaluator provided in Appendix[C](https://arxiv.org/html/2610.13375#A3.SS0.SSS0.Px1)\. As shown in Table[6](https://arxiv.org/html/2610.13375#S5.T6), ARTS\-Hybrid achieves the highest scores across all four quality dimensions, including coverage, concreteness, non\-redundancy, and actionability\. This result complements the end\-to\-end evaluation results by showing that the gains are accompanied by improvements in the quality of the generated criteria themselves\.

Table 6:Comparison of intrinsic rubric quality\. Higher values indicate better performance\.Rubric generator distillation\.The ARTS\-Hybrid incurs additional computation primarily during offline rubric construction\. We therefore examine whether the benefits of mechanism\-guided construction can be distilled into a standalone rubric generator, avoiding repeated execution of the full ARTS\-Hybrid pipeline\. Specifically, we fine\-tune the same Qwen3\-4B rubric generator using three sources of supervision, including direct\-prompt rubrics, expert\-provided rubrics from the original dataset, and the rubrics generated by our framework, with the untrained Qwen3\-4B generator as the baseline\. As shown in Table[7](https://arxiv.org/html/2610.13375#S5.T7), training on ARTS\-Hybrid rubrics achieves the highest accuracy on JudgeBench\. These results suggest that the benefits obtained through offline mechanism\-guided construction can be transferred to a standalone rubric generator, providing a potential way to reduce reliance on the full ARTS\-Hybrid pipeline for subsequent rubric generation\.

Table 7:The performance of Qwen3\-4B rubric generators under different SFT supervision sources\.Rubric\-as\-Reward for policy optimization\.Beyond evaluation, we examine whether ARTS\-Hybrid can provide effective reward signals for policy optimization\. We stratifiedly sample 3,000 training instances from OpenRubrics\([Liu et al\., 2026b](https://arxiv.org/html/2610.13375#bib.bib8)\)and optimize Qwen3\-4B using Dr\.GRPO\([Liu et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib25)\)\. We compare reward signals derived from the same three rubric sources described above and additionally report the untrained Qwen3\-4B model as the base model\. Table[8](https://arxiv.org/html/2610.13375#S5.T8)reports the performance under both thinking and non\-thinking settings\. The ARTS\-Hybrid achieves the best results in both settings, while directly generated and expert\-provided rubrics provide limited gains over the base model\. These results suggest that the advantages of ARTS\-Hybrid are not limited to evaluation, but can also translate into more effective reward signals for policy optimization\.

Table 8:The score performance on HealthBench under different rubric\-based rewards\.Additional experiments\.We conduct two additional experiments in Appendix[C](https://arxiv.org/html/2610.13375#A3)\. \(i\)Cross\-judge robustness\.We evaluate the generated rubrics with different judge models to assess whether their effectiveness generalizes across evaluators\. \(ii\)Ablation study\.We conduct an ablation study to examine the contributions of semantic validation and response\-based calibration\.

## 6Conclusion

In this work, we systematically investigate what makes rubric design effective from three perspectives, including criterion organization, criterion\-level operationalization, and score aggregation\. Guided by the resulting mechanistic findings, we introduce ARTS\-Hybrid, which combines adaptive tree construction with semantic\-first, response\-calibrated refinement\. Experiments show that ARTS\-Hybrid produces higher\-quality rubrics and achieves strong evaluation performance across different judge models\. Beyond evaluation, our reinforcement learning experiments further show that well\-designed rubrics can provide effective reward signals for model optimization\.

### Reproducibility Statement

To support reproducibility, we release the manually constructed counterfactual responses and gold rubric trees used in the mechanistic analysis, together with the core code for both the mechanistic analysis and ARTS\-Hybrid, in our code repository \([https://github\.com/SophiaQian02/Tree\_of\_Rubrics\_code](https://github.com/SophiaQian02/Tree_of_Rubrics_code)\)\. The construction and refinement procedures of ARTS\-Hybrid are described in the main text\. Detailed implementation settings for the mechanistic analysis and ARTS\-Hybrid are provided in the appendix\. Appendix[A\.1](https://arxiv.org/html/2610.13375#A1.SS1)describes the data sources and sampling procedure, while Appendix[A\.3](https://arxiv.org/html/2610.13375#A1.SS3)details the construction of counterfactual responses\. The response\-based calibration setup, including the construction of calibration responses and the calibration judges, is provided in Appendix[B\.2\.1](https://arxiv.org/html/2610.13375#A2.SS2.SSS1)\. Appendix[B\.2\.2](https://arxiv.org/html/2610.13375#A2.SS2.SSS2)summarizes the key hyperparameters and their search ranges\. Finally, Appendix[D](https://arxiv.org/html/2610.13375#A4)provides the core prompts used in the mechanistic analysis and ARTS\-Hybrid\.

## References

- R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel,et al\.Healthbench: evaluating large language models towards improved human health\.arXiv preprint arXiv:2505\.08775\.Cited by:[§A\.1](https://arxiv.org/html/2610.13375#A1.SS1.p1.1),[§B\.2\.4](https://arxiv.org/html/2610.13375#A2.SS2.SSS4.p1.1),[§3](https://arxiv.org/html/2610.13375#S3.p1.1)\.
- Brookhart \(2018\)S\. M\. BrookhartAppropriate criteria: key to effective rubrics\.InFrontiers in education,Vol\.3,pp\. 22\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1)\.
- Chenet al\.\(2026\)Y\. Chen, H\. Li, Q\. Ai, F\. Zhu, T\. Chua, M\. Zhang, and Y\. LiuGenRubric: self\-evolving rubric generation for scalable LLM evaluation\.CoRRabs/2608\.29856\.External Links:[Link](https://doi.org/10.48550/arXiv.2608.29856),[Document](https://dx.doi.org/10.48550/ARXIV.2608.29856)Cited by:[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Fanet al\.\(2026\)Z\. Fan, R\. Chen, T\. Hu, R\. Peng, Z\. Huang, H\. Xu, Y\. Chen, J\. Wu, J\. Zhao, and Z\. LiuOptimsyn: influence\-guided rubrics optimization for synthetic data generation\.arXiv preprint arXiv:2604\.00536\.Cited by:[§2](https://arxiv.org/html/2610.13375#S2.p2.1)\.
- Gaoet al\.\(2026\)S\. Gao, Y\. Su, P\. Sui, C\. Ginder, and M\. ZitnikQworld: question\-specific evaluation criteria for llms\.arXiv preprint arXiv:2603\.23522\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.13375#S5.SS1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix C](https://arxiv.org/html/2610.13375#A3.SS0.SSS0.Px3.p1.1)\.
- Guet al\.\(2026\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu,et al\.A survey on llm\-as\-a\-judge\.The Innovation7\(6\)\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p1.1)\.
- Gunjalet al\.\(2026\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InProceedings of ICLR,pp\. 127924–127945\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1),[§2](https://arxiv.org/html/2610.13375#S2.p2.1)\.
- Hashemiet al\.\(2024\)H\. Hashemi, J\. Eisner, C\. Rosset, B\. Van Durme, and C\. KedzieLLM\-rubric: a multidimensional, calibrated approach to automated evaluation of natural language texts\.InProceedings of ACL,pp\. 13806–13834\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMistral 7b\.External Links:2310\.06825Cited by:[§B\.2\.1](https://arxiv.org/html/2610.13375#A2.SS2.SSS1.Px1.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi,et al\.Rewardbench: evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 1755–1797\.Cited by:[§5\.1](https://arxiv.org/html/2610.13375#S5.SS1.p1.1)\.
- Liet al\.\(2026\)S\. Li, J\. Zhao, H\. Ren, Z\. Wei, Y\. Zhou, J\. Yang, S\. Liu, K\. Zhang, and C\. WeiRubrichub: a comprehensive and highly discriminative rubric dataset via automated coarse\-to\-fine generation\.InProceedings of ACL,pp\. 31320–31344\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Linet al\.\(2025\)B\. Y\. Lin, Y\. Deng, K\. Chandu, A\. Ravichander, V\. Pyatkin, N\. Dziri, R\. Le Bras, and Y\. ChoiWildbench: benchmarking llms with challenging tasks from real users in the wild\.InProceedings of ICLR,Vol\.2025,pp\. 47852–47870\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Liuet al\.\(2026a\)D\. Liu, F\. Yang, X\. Wang, S\. Yan, J\. Chai, J\. Li, Y\. Ban, Z\. Mao, W\. Lin, and G\. YinCDRRM: contrast\-driven rubric generation for reliable and interpretable reward modeling\.InProceedings of ACM SIGKDD,pp\. 3152–3163\.Cited by:[§5\.1](https://arxiv.org/html/2610.13375#S5.SS1.p1.1)\.
- Liuet al\.\(2026b\)T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. WangOpenrubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.InProceedings of ACL,pp\. 17417–17437\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1),[§5\.2](https://arxiv.org/html/2610.13375#S5.SS2.p4.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.InProceedings of COLM,Cited by:[§5\.2](https://arxiv.org/html/2610.13375#S5.SS2.p4.1)\.
- Maliket al\.\(2026\)S\. Malik, V\. Pyatkin, S\. Land, J\. Morrison, N\. Smith, H\. Hajishirzi, and N\. LambertRewardbench 2: advancing reward model evaluation\.InProceedings of ICLR,Vol\.2026,pp\. 144839–144866\.Cited by:[§5\.1](https://arxiv.org/html/2610.13375#S5.SS1.p1.1)\.
- Qwenet al\.\(2025\)Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115Cited by:[Table 3](https://arxiv.org/html/2610.13375#S3.T3)\.
- Shaoet al\.\(2025\)R\. Shao, A\. Asai, S\. Z\. Shen, H\. Ivison, V\. Kishore, J\. Zhuo, X\. Zhao, M\. Park, S\. G\. Finlayson, D\. Sontag,et al\.Dr tulu: reinforcement learning with evolving rubrics for deep research\.arXiv preprint arXiv:2511\.19399\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1),[§2](https://arxiv.org/html/2610.13375#S2.p2.1)\.
- Shenet al\.\(2026\)W\. F\. Shen, X\. Qiu, C\. Whitehouse, L\. Alazraki, S\. Goel, F\. Barbieri, T\. Willi, A\. Mathur, and I\. LeontiadisRethinking rubric generation for improving llm judge and reward modeling for open\-ended tasks\.arXiv preprint arXiv:2602\.05125\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1),[§3\.3](https://arxiv.org/html/2610.13375#S3.SS3.p1.1)\.
- Shenget al\.\(2026\)L\. Sheng, W\. Ma, R\. Hong, X\. Wang, A\. Zhang, and T\. ChuaReinforcing chain\-of\-thought reasoning with self\-evolving rubrics\.arXiv preprint arXiv:2602\.10885\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Shiet al\.\(2025\)Z\. Shi, Y\. Chen, H\. Li, W\. Sun, S\. Ni, Y\. Lyu, R\. Fan, B\. Jin, Y\. Weng, M\. Zhu,et al\.Deep research: a systematic survey\.arXiv preprint arXiv:2512\.02038\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p1.1)\.
- Spearman \(1987\)C\. SpearmanThe proof and measurement of association between two things\.The American journal of psychology100\(3/4\),pp\. 441–471\.Cited by:[§B\.1](https://arxiv.org/html/2610.13375#A2.SS1.SSS0.Px5.p1.3)\.
- Tanet al\.\(2025\)S\. Tan, S\. Zhuang, K\. Montgomery, W\. Tang, A\. Cuadron, C\. Wang, R\. Popa, and I\. StoicaJudgebench: a benchmark for evaluating llm\-based judges\.InProceedings of ICLR,Vol\.2025,pp\. 63277–63303\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.13375#S5.SS1.p1.1)\.
- Viswanathanet al\.\(2026\)V\. Viswanathan, Y\. Sun, X\. Kong, M\. Cao, G\. Neubig, and S\. WuChecklists are better than reward models for aligning language models\.InProceedings of NeurIPS,Vol\.38,pp\. 114728–114754\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Wanget al\.\(2025\)P\. Wang, P\. Liu, Q\. Zuo, Z\. Sang, C\. Xie, and H\. YangInfimed\-orbit: aligning llms on open\-ended complex tasks via rubric\-based incremental training\.arXiv preprint arXiv:2510\.15859\.Cited by:[§2](https://arxiv.org/html/2610.13375#S2.p2.1)\.
- Xieet al\.\(2025\)L\. Xie, S\. Huang, Z\. Zhang, A\. Zou, Y\. Zhai, D\. Ren, K\. Zhang, H\. Hu, B\. Liu, H\. Chen, Z\. Liu, and B\. DingAuto\-rubric: learning to extract generalizable criteria for reward modeling\.CoRRabs/2510\.17314\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1),[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§B\.2\.1](https://arxiv.org/html/2610.13375#A2.SS2.SSS1.Px1.p1.1)\.
- Yanget al\.\(2026\)H\. Yang, L\. Pan, X\. Li, Z\. Chen, Z\. Zhang, Y\. Lu, H\. Li, and H\. WangRubrics on trial: evolving rubrics from a single query via synthetic pairwise evidence\.arXiv preprint arXiv:2607\.15092\.Cited by:[§2](https://arxiv.org/html/2610.13375#S2.p1.1)\.
- Zhanget al\.\(2026\)J\. Zhang, Z\. Wang, L\. Gui, S\. Mysore Sathyendra, J\. Jeong, V\. Veitch, W\. Wang, Y\. He, B\. Liu, and L\. JinChasing the tail: effective rubric\-based reward modeling for large language model post\-training\.InProceedings of ICLR,Vol\.2026,pp\. 133430–133457\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InProceedings of the NeurIPS,NIPS ’23,Red Hook, NY, USA\.Cited by:[§1](https://arxiv.org/html/2610.13375#S1.p1.1)\.
- Zhouet al\.\(2026\)J\. Zhou, Q\. Zhang, Y\. Wang, F\. Lyu, Y\. Ming, C\. Xu, Q\. Sun, K\. Zheng, P\. Kang, X\. Liu,et al\.RubricBench: aligning model\-generated rubrics with human standards\.InProceedings of ACL,pp\. 31179–31200\.Cited by:[§A\.1](https://arxiv.org/html/2610.13375#A1.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.13375#S3.SS2.p1.1),[§3](https://arxiv.org/html/2610.13375#S3.p1.1),[§5\.1](https://arxiv.org/html/2610.13375#S5.SS1.p1.1)\.

## Appendix AMechanistic Analysis: Experimental Details

### A\.1dataset

The HealthBench dataset\([Arora et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib20)\)provides physician\-authored criteria for evaluating open\-ended medical responses, while RubricBench\([Zhou et al\., 2026](https://arxiv.org/html/2610.13375#bib.bib21)\)covers a broader range of domains, including safety, instruction following, dialogue, STEM, and coding, with expert\-annotated atomic criteria, candidate response pairs, and human preference labels\. We sample 750 instances from fixed versions of each dataset using stratified sampling\. For HealthBench, we perform proportional sampling based on theme combinations and rubric cardinality, preserving the original distributions of task themes and rubric complexity\. For RubricBench, we retain all Safety and Instruction Following instances and sample the remaining instances from the Chat, STEM, and Code categories according to their original task domains, data sources, and rubric\-count intervals\. The resulting mechanistic analysis set comprises 1,500 instances across six task categories\. We further summarize the distributions of task categories and rubric counts in Figure[2](https://arxiv.org/html/2610.13375#A1.F2)\.

Figure 2:Distribution of task categories and rubric counts in the mechanism analysis set\.
### A\.2metric

##### Coverage\.

For theii\-th instance, letGiG\_\{i\}denote the set of expert\-annotated gold leaves andRiR\_\{i\}the set of generated rubric criteria\. We define the coverage indicator as

ci​g=\{1,if the generated rubric fully covers gold leaf​g,0,otherwise\.c\_\{ig\}=\\begin\{cases\}1,&\\text\{if the generated rubric fully covers gold leaf \}g,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(5\)
The coverage of instanceiiis computed as

Coveragei=1\|Gi\|​∑g∈Gici​g\.\\operatorname\{Coverage\}\_\{i\}=\\frac\{1\}\{\|G\_\{i\}\|\}\\sum\_\{g\\in G\_\{i\}\}c\_\{ig\}\.\(6\)

##### Novelty\.

Similarly, the novelty indicator is defined as

vi​r=\{1,if​r​is valid, query\-relevant, and absent from​Gi,0,otherwise\.v\_\{ir\}=\\begin\{cases\}1,&\\text\{if \}r\\text\{ is valid, query\-relevant, and absent from \}G\_\{i\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(7\)
The valid novelty of instanceiiis computed as

Noveltyi=1\|Ri\|​∑r∈Rivi​r\.\\operatorname\{Novelty\}\_\{i\}=\\frac\{1\}\{\|R\_\{i\}\|\}\\sum\_\{r\\in R\_\{i\}\}v\_\{ir\}\.\(8\)

### A\.3Counterfactual response

In this section, we detail the construction of the counterfactual responses\. We follow a minimal\-intervention principle, modifying only the content required to violate the targeted gold leaves while leaving unrelated content unchanged\. The edits include removing required information, altering factual content, violating formatting constraints, omitting necessary safety warnings, modifying applicability conditions, and introducing behaviors specified by negative criteria\. After each edit, we re\-evaluate the response against all gold leaves and retain it only if the targeted leaves change state while all non\-targeted leaves remain unchanged\. If a sample fails this isolation check, it is further revised and discarded if the required isolation cannot be achieved\. Across the 1,500 instances, this procedure yields 1,498 single\-leaf, 1,482 within\-dimension multi\-leaf, and 1,409 cross\-dimension multi\-leaf counterfactual responses\.

### A\.4Shuffled Path

For each counterfactual type, we randomly sample 50% of the instances for which such mismatched paths can be constructed\. We repeat this procedure with two random seeds, denoted as Seed 1 and Seed 2, which correspond to 20260813 and 123456, respectively\. For a fair comparison,Correct PathandShuffled Pathare evaluated on the same random subset\. Consequently, theCorrect Pathscores vary across seeds and differ from the full\-set results in Table[2](https://arxiv.org/html/2610.13375#S3.T2)\.

Table 9:Comparison between correct and shuffled hierarchical paths\. Values in parentheses denote changes fromCorrecttoShuffled\.The experimental results are reported in Table[9](https://arxiv.org/html/2610.13375#A1.T9)\. The correct path generally outperform their shuffled counterparts across judge models, random seeds, and counterfactual types\. This suggests that the gains from path conditioning cannot be attributed to additional context length alone\. This indicates that the judge benefits specifically from the structured semantics encoded by the hierarchy\.

### A\.5Calibration\-Based Selection for RQ2

For each local candidateCC, we use two calibration judges to independently score every criterion on the same set of six calibration responses using a five\-point scale\. Based on these scores, we characterize each candidate set from four aspects, including response discrimination, cross\-judge variance, empirical redundancy, and the number of criteria\. Response discrimination captures how much the criterion scores vary across calibration responses and is measured by their average variance\. Cross\-judge variance reflects the extent to which the two judges disagree on the same criterion\-response pair\. Finally, the empirical redundancy captures similarity in scoring behavior across criteria and is defined as the proportion of criterion pairs whose response\-score profiles have a Pearson correlation above 0\.90\. We combine these signals into a calibration utility that rewards response discrimination while penalizing cross\-judge variance, empirical redundancy, and unnecessary growth in the number of criteria, as defined below\.

URQ2​\(C\)=DRQ2​\(C\)−VRQ2​\(C\)−0\.10​RRQ2​\(C\)−0\.02​max⁡\(0,\|C\|−1\)\.U\_\{\\mathrm\{RQ2\}\}\(C\)=D\_\{\\mathrm\{RQ2\}\}\(C\)\-V\_\{\\mathrm\{RQ2\}\}\(C\)\-0\.10R\_\{\\mathrm\{RQ2\}\}\(C\)\-0\.02\\max\(0,\|C\|\-1\)\.\(9\)ForSemantic\+Calibration, this utility is applied only after semantic filtering\. A candidate replaces the default semantic choice only when its utility improves by at least0\.050\.05\. The calibration responses are used exclusively for candidate selection, while the discrimination, judge variance, and pairwise accuracy reported in Table[3](https://arxiv.org/html/2610.13375#S3.T3)are computed independently on the held\-out response pairs\.

## Appendix BARTS\-Hybrid: Experimental Details

### B\.1calibration objective

For a semantically feasible decompositionCCof the current node, letc∈Cc\\in Cdenote a child criterion\. Each criterion is independently evaluated byn=2n=2calibration judges on the same set ofm=6m=6calibration responses\. We define

pc,r,j∈\[0,1\],p\_\{c,r,j\}\\in\[0,1\],\(10\)wherepc,r,jp\_\{c,r,j\}denotes the probability assigned by judgejjthat responserrsatisfies criterioncc\.

##### Candidate\-level score aggregation\.

To characterize the behavior of a candidate decompositionCCon each response, we first aggregate scores across its child criteria using a soft\-min operator as

zr,j\(C\)=\(1\|C\|∑c∈Cmax\(pc,r,j,ϵ\)−β\)−1/β,z\_\{r,j\}\(C\)=\\left\(\\frac\{1\}\{\|C\|\}\\sum\_\{c\\in C\}\\max\(p\_\{c,r,j\},\\epsilon\)^\{\-\\beta\}\\right\)^\{\-1/\\beta\},\(11\)whereϵ=10−6\\epsilon=10^\{\-6\}andβ=8\\beta=8\. This aggregation approximates the minimum score across child criteria, such that low satisfaction on any criterion reduces the overall score of the candidate decomposition\. WhenCCcontains a single criterion, we directly setzr,j​\(C\)=pc,r,jz\_\{r,j\}\(C\)=p\_\{c,r,j\}\.

Based on the aggregated scoreszr,j​\(C\)z\_\{r,j\}\(C\), we next define the response discriminationD⁡\(C\)D\(C\), judge agreementA⁡\(C\)A\(C\), score stabilityS⁡\(C\)S\(C\), and behavioral redundancyR⁡\(C\)R\(C\)that constitute the calibration objective in Equation[3](https://arxiv.org/html/2610.13375#S4.E3)\.

##### Response discrimination\.

The response discrimination measures whether a candidate decomposition produces sufficiently varied scores across the calibration responses\. We compute the variance of the aggregated scores across responses for each judge and then average over judges as

D⁡\(C\)=1n​∑j=1nVarr⁡\[zr,j​\(C\)\]\.D\(C\)=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\\operatorname\{Var\}\_\{r\}\\left\[z\_\{r,j\}\(C\)\\right\]\.\(12\)The largerD⁡\(C\)D\(C\)indicates greater variation across responses and thus stronger discriminative power\.

##### Cross\-judge agreement\.

The cross\-judge agreement measures whether different calibration judges assign similar scores to the same criterion\-response pair\. For each criterioncc, we first compute the score variance across judges for each response as

vc,r=Varj⁡\[pc,r,j\]\.v\_\{c,r\}=\\operatorname\{Var\}\_\{j\}\\left\[p\_\{c,r,j\}\\right\]\.\(13\)We then define the agreement of criterionccas

a⁡\(c\)=11\+1m​∑r=1mvc,r\.a\(c\)=\\frac\{1\}\{1\+\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}v\_\{c,r\}\}\.\(14\)The overall agreement of a candidate decomposition is obtained by averaging across its child criteria:

A⁡\(C\)=1\|C\|​∑c∈Ca⁡\(c\)\.A\(C\)=\\frac\{1\}\{\|C\|\}\\sum\_\{c\\in C\}a\(c\)\.\(15\)The largerA⁡\(C\)A\(C\)indicates greater agreement between judges on the same responses and criteria\.

##### Judge stability\.

Beyond agreement on individual responses, the judge stability measures whether different judges exhibit similar discrimination behavior across the calibration responses\. For each judgejj, we first compute the variance of the aggregated scores across responses as

uj​\(C\)=Varr⁡\[zr,j​\(C\)\]\.u\_\{j\}\(C\)=\\operatorname\{Var\}\_\{r\}\\left\[z\_\{r,j\}\(C\)\\right\]\.\(16\)We then define judge stability as

S⁡\(C\)=11\+Varj⁡\[uj​\(C\)\]\.S\(C\)=\\frac\{1\}\{1\+\\operatorname\{Var\}\_\{j\}\\left\[u\_\{j\}\(C\)\\right\]\}\.\(17\)WhileA⁡\(C\)A\(C\)captures agreement on individual responses,S⁡\(C\)S\(C\)captures the consistency of discrimination patterns across judges, providing a complementary measure of overall scoring stability\.

##### Score redundancy\.

The score redundancy measures the similarity of scoring patterns across child criteria on the calibration responses\. We first average each criterion’s scores across judges as

p¯c,r=1n​∑j=1npc,r,j\.\\bar\{p\}\_\{c,r\}=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}p\_\{c,r,j\}\.\(18\)For each criterioncc, let

𝐩¯c=\[p¯c,1,…,p¯c,m\]\\bar\{\\mathbf\{p\}\}\_\{c\}=\\left\[\\bar\{p\}\_\{c,1\},\\ldots,\\bar\{p\}\_\{c,m\}\\right\]\(19\)denote the score vector across calibration responses\. We use Spearman rank correlation\([Spearman, 1987](https://arxiv.org/html/2610.13375#bib.bib30)\)to measure the similarity between the scoring profiles of each pair of child criteria, and defineR⁡\(C\)R\(C\)as the maximum positive correlation

R⁡\(C\)=maxc,c′∈C⁡max⁡\(0,ρS​\(𝐩¯c,𝐩¯c′\)\)\.R\(C\)=\\max\_\{\\begin\{subarray\}\{c\}c,c^\{\\prime\}\\in C\\end\{subarray\}\}\\max\\left\(0,\\,\\rho\_\{\\mathrm\{S\}\}\\left\(\\bar\{\\mathbf\{p\}\}\_\{c\},\\bar\{\\mathbf\{p\}\}\_\{c^\{\\prime\}\}\\right\)\\right\)\.\(20\)When\|C\|=1\|C\|=1, we setR⁡\(C\)=0R\(C\)=0\. The largerR⁡\(C\)R\(C\)indicates stronger similarity in scoring behavior among child criteria, suggesting increased redundancy\.

### B\.2Implementation Details

We use GLM\-4\.7\-Flash as the common rubric generator and judge model across all evaluation methods\. For ARTS\-Hybrid, we follow the response construction and calibration protocol introduced in RQ2, using the same calibration panel and judge models\. The specific models used for calibration response generation and judging are listed in Table[10](https://arxiv.org/html/2610.13375#A2.T10)\.

Table 10:Calibration response generators and judges used in ARTS\-Hybrid\. Each response generator produces two responses, and each calibration judge evaluates all six responses\.RoleTarget QualityModelResponse GeneratorStrongQwen3\-30B\-A3B\-Instruct\-2507Response GeneratorMediumLlama\-3\.1\-70B\-InstructResponse GeneratorWeakMistral\-7B\-Instruct\-v0\.2Calibration Judge–Qwen2\.5\-72B\-InstructCalibration Judge–Llama\-3\.3\-70B\-Instruct#### B\.2\.1Response\-Based Calibration Setup

Figure 3:Independent validation of calibration response quality\. DeepSeek\-V4\-Flash\-GA and Gemini\-3\.8\-Flash independently evaluate responses generated at the intended strong, medium, and weak quality levels\.##### Calibration responses\.

For each query, we pre\-generate six calibration responses before rubric construction and keep them fixed throughout the tree search\.The same calibration panel is shared across all candidate criteria\.To cover diverse response quality and generation characteristics, we use three model families, with two responses generated by each model\. Specifically, we use Qwen3\-30B\-A3B\-Instruct\-2507\([Yang et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib26)\)to generate two strong responses, Llama\-3\.1\-70B\-Instruct to generate two medium\-quality responses, and Mistral\-7B\-Instruct\-v0\.2\([Jiang et al\., 2023](https://arxiv.org/html/2610.13375#bib.bib28)\)to generate two weak responses\. We further control the intended quality level through group\-specific generation instructions\. For the strong group, the instructions emphasize accuracy, completeness and adherence to explicit requirements, with a variant encouraging greater conciseness\. For the medium group, the instructions require the response to address the main task while allowing minor omissions or incomplete explanations, with a variant encouraging greater elaboration\. For the weak group, the instructions preserve overall plausibility while allowing incomplete, vague, or poorly organized content, with a variant introducing a minor factual error\. Therefore, these responses provide a common calibration basis spanning different quality levels and response characteristics\.

To verify that the calibration responses follow the intended quality levels, we use two strong independent evaluators, DeepSeek\-V4\-Flash\-GA and Gemini\-3\.8\-Flash, to assess their response quality\. As shown in Figure[3](https://arxiv.org/html/2610.13375#A2.F3), both evaluators exhibit a consistent quality gradient, with average scores decreasing from the strong to medium and weak response groups\. This pattern aligns with the intended quality ordering and supports the use of these responses as a quality\-diverse calibration panel\.

##### Calibration judges\.

We use Qwen2\.5\-72B\-Instruct and Llama\-3\.3\-70B\-Instruct as two independent calibration judges\. Each judge evaluates the candidate criteria on the same calibration responses, providing the scoring signals used for response\-based candidate selection\. The calibration responses are used exclusively during rubric construction and are excluded from the final evaluation\.

Table 11:The hyperparameter configurations used by ARTS\-Hybrid\.

#### B\.2\.2Hyperparameters

In this section, we summarize the hyperparameter configurations used in ARTS\-Hybrid, including the search settings for rubric tree construction, the thresholds and regeneration settings used during semantic and calibration\-based selection, and the weights and regularization terms in the calibration objective\. Table[11](https://arxiv.org/html/2610.13375#A2.T11)reports the corresponding values used in our experiments\.

#### B\.2\.3Rubric Generator Distillation

We fine\-tune Qwen3\-4B on the same stratified set of 3,000 OpenRubrics instances using rubrics from different supervision sources\. All variants use identical training examples and hyperparameters\. We apply LoRA with rank 16, a scaling factor of 32, and dropout of 0\.05 for one epoch, with a learning rate of2×10−52\\times 10^\{\-5\}, a global batch size of 32, and a maximum sequence length of 4,096\. Training is conducted on 8 NVIDIA H200 GPUs\. For evaluation on JudgeBench, we use GLM\-4\.7\-Flash as the common response judge and to resolve ties in rubric\-aggregated scores\.

#### B\.2\.4Rubric\-as\-Reward Training Details

We initialize all policy optimization experiments from the same Qwen3\-4B model and use a fixed stratified set of 3,000 OpenRubrics prompts\. For each rollout, a frozen Qwen2\.5\-7B\-Instruct model independently evaluates whether the response satisfies each rubric criterion\. The resulting binary criterion\-level judgments are aggregated using normalized criterion weights to produce a scalar reward\. We optimize the policy with Dr\.GRPO for three epochs, corresponding to 186 optimization steps\. Each step contains 48 prompts with eight rollouts per prompt\. We use an actor learning rate of1×10−61\\times 10^\{\-6\}, maximum prompt and response lengths of 4,096 tokens, and a low\-variance KL penalty with a coefficient of 0\.001\. Following Dr\.GRPO, we disable group\-reward standard\-deviation normalization and use sequence\-level token\-sum loss aggregation\. All reward\-source variants use identical policy initialization, training prompts, graders, and optimization settings\. The thinking and non\-thinking settings differ only in whether thinking mode is enabled\. For evaluation on HealthBench\([Arora et al\., 2025](https://arxiv.org/html/2610.13375#bib.bib20)\), we use Qwen2\.5\-7B\-Instruct as the judge model to evaluate each generated response against the physician\-authored criteria and aggregate the criterion\-level judgments into the final HealthBench score\.

## Appendix CAdditional Results

##### Cross\-Evaluator rubric quality\.

To assess robustness across evaluators, we repeat the intrinsic rubric quality evaluation using Llama3\.1\-405B\-Instruct as an additional independent evaluator\. As shown in Table[12](https://arxiv.org/html/2610.13375#A3.T12), the ARTS\-Hybrid achieves the highest overall score and performs best in concreteness, non\-redundancy, and actionability\.

Table 12:Comparison of intrinsic rubric quality\. Higher values indicate better performance\.
##### Training dynamics of rubric\-based rewards\.

To further examine how different rubric\-based rewards affect policy optimization, we track three diagnostic metrics throughout Dr\.GRPO training\. Figure[4](https://arxiv.org/html/2610.13375#A3.F4)\(a\) reports theInformative Group Rate, defined as the fraction of prompt groups whose rollout rewards are not identical and therefore yield non\-zero GRPO advantages\. ARTS\-Hybrid provides informative relative reward signals for more prompt groups than Direct\-Prompt, while OpenRubrics Expert achieves the highest rate\. Notably, ARTS\-Hybrid still yields better downstream performance than OpenRubrics Expert, suggesting that reward effectiveness depends not only on how often rewards differentiate responses, but also onwhether these differences reflect task\-relevant response quality\. Figure[4](https://arxiv.org/html/2610.13375#A3.F4)\(b\) shows the mean response length during training\. The three curves closely overlap and remain stable at approximately 2,000\-2,100 tokens, indicating that the gains from ARTS\-Hybrid are not driven by a preference for longer responses\. Finally, Figure[4](https://arxiv.org/html/2610.13375#A3.F4)\(c\) shows similar and stable KL loss across all three methods, suggesting that the advantage of ARTS\-Hybrid does not arise from more aggressive policy updates\.

Figure 4:Training dynamics under different rubric\-based reward signals, including informative group rate, response length, and KL loss\.
##### Cross\-judge robustness\.

In addition, we examine whether the gains of ARTS\-Hybrid are sensitive to the choice of judge model\. As shown in Table[13](https://arxiv.org/html/2610.13375#A3.T13), the ARTS\-Hybrid achieves the highest pairwise accuracy with both Qwen3\.5\-397B\-A17B and Llama3\.1\-405B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.13375#bib.bib27)\), with particularly clear gains under the latter\. Notably, while the performance of one\-shot rubric generation varies considerably across judges, the ARTS\-Hybrid maintains consistent improvements over direct judgment\. These results suggest that the effectiveness of the generated rubrics is not tied to a specific evaluator and can generalize across different judge models\.

Table 13:Pairwise accuracy \(%\) across judge models\. The best result is highlighted in bold\.
##### Ablation study\.

We conduct an ablation study to examine the contributions of semantic validation and response\-based calibration\. The results are reported in Table[14](https://arxiv.org/html/2610.13375#A3.T14)\. When semantic validation is removed, the performance decreases across all four benchmarks, with the largest drop on the RM\-Bench Chat dataset\. When response\-based calibration is removed, performance decreases on RM\-Bench Chat, PreciseIF, and JudgeBench datasets, while remaining comparable on Focus\. Overall, both components contribute to the performance of ARTS\-Hybrid, with semantic validation showing a more consistent effect across benchmarks\.

Table 14:Ablation study of semantic specification and response\-based calibration\. All results are pairwise evaluation accuracies \(%\)\. The best result in each column is highlighted in bold\.

## Appendix DPrompts

In this section, we provide the core prompts used in the mechanistic analysis and ARTS\-Hybrid\. We omit auxiliary prompts for formatting, retries, and other implementation\-specific operations\.

### D\.1Prompts for Mechanistic Analysis

We first provide the prompts used in the mechanistic analysis for RQ1\. These include the flat and hierarchical rubric generation prompts used in Table[1](https://arxiv.org/html/2610.13375#S3.T1), as well as the criterion judgment prompts used to evaluate counterfactual responses in Table[2](https://arxiv.org/html/2610.13375#S3.T2)\.

##### Rubric generation\.

For flat rubric generation, we use the following prompt\.

UsingONLYthequery,directlycreateaflatsetofatomicevaluationrubrics\.

QUERY

<QUERY\>

Eachitemmustbeindependentlyjudgeable,relevant,non\-overlapping,anddescribewhatagoodresponsemustsatisfy\.Donotcreateahierarchy,inspectorspeculateaboutcandidateresponses,orrevealananswer\.

Returntheresultinthefollowingformat\.

\{”rubrics”:\[

\{”criterion”:”…”,”category”:”…”\}

\]\}

Generate3to<MAX\_LEAVES\>items,stoppingwhenthequeryisadequatelycovered\.

For hierarchical rubric generation proceeds in two stages\. We first construct the initial branches of the rubric tree and then recursively examine each node to determine whether further decomposition is needed\. The prompts for these two stages are provided below\.

Createthefirstlevelofanadaptiveevaluation\-rubrictreeusingONLYthequerybelow\.

QUERY

<QUERY\>

Therootis”Overallresponsequality”Splititinto2\-8distinctandusefulsemanticbranches\.Eachbranchshoulddescribewhatagoodresponsemustsatisfywithoutrevealingtheanswer\.Donotinspectorspeculateaboutcandidateresponses\.

Returntheresultinthefollowingformat\.

\{”children”:\[

\{”name”:”…”,”criterion”:”…”\}

\]\}

Decidewhetherthecurrentrubricnodeshouldremainatomicorbefurtherdecomposed,usingonlythequeryanditssemanticpath\.

QUERY

<QUERY\>

NODEPATH

<NODE\_PATH\>

CURRENTCRITERION

<CRITERION\>

CURRENTDEPTH

<DEPTH\>

MAXIMUMDEPTH

<MAX\_DEPTH\>

Keepthenodeatomicwhenitrepresentsoneindependentlyjudgeablerequirementorfailuremode\.Splititonlywhenitcontainsmultiplenon\-overlappingrequirementsthatwouldbemorediagnosticifassessedseparately\.Avoidunarysplitsandcosmeticrestatements\.

Returneitheranatomicdecisionorasplitcontaining2\-6child

criteria\.

##### Counterfactual violation detection\.

We provide the core prompt used for counterfactual violation detection under thePathcondition\. The root\-to\-leaf path provides semantic context for judging the final leaf criterion, while intermediate nodes are not used as routing gates\.

YouaregivenONEroot\-to\-leafsemanticpathfromalargerGoldrubrictree\.Thefinaliteminthepathistheonlyleafbeingjudged\.

Usetheparentnodestounderstandtheleaf’sintendedsemanticscope,terminology,andrequiredevidence\.Parentnodesarecontextratherthanseparatelyscoredroutinggates\.

QUERY

<QUERY\>

RESPONSE

<RESPONSE\>

FULLSEMANTICPATH

<SEMANTIC\_PATH\>

LEAFCRITERION

<CRITERION\>

DESIREDSTATE

<DESIRED\_STATE\>

Determinewhethertheresponsesatisfiesthefinalleafcriterion\.

Returntheresultinthefollowingformat\.

\{”passes”:true,”evidence”:”…”\}

We next provide the core prompts used in RQ2, where we examine how semantic assessment and response\-based calibration provide complementary signals for criterion selection\.

##### Semantic and calibration\-based criterion selection\.

The Semantic Assessment Prompt determines whether a local revision is semantically better than, or at least no worse than, the original Direct Rubric\. The Calibration Scoring Prompt evaluates how candidate criteria behave on actual responses\. As described above, calibration\-based candidate selection is performed programmatically using the utility defined in Equation[9](https://arxiv.org/html/2610.13375#A1.E9), rather than through an additional model call\. We therefore provide only the Calibration Scoring Prompt below\.

Youareanindependentsemanticrubriccomparator\.CompareanimmutableDirectrubricsetwithonelocalrevisioncandidate\.

QUERY

<QUERY\>

DIRECTRUBRIC

<DIRECT\_RUBRIC\>

CANDIDATEREVISION

<CANDIDATE\>

Comparethetwoalternativesalongfourdimensions\.

1\.Coverageofmaterialrequirementsgroundedinthequery\.

2\.Atomicityofindependentlyfail\-ablerequirements\.

3\.Concreteandobservablescoreability\.

4\.Absenceoftruesemanticduplicationorsubsumption\.

Foreachdimension,decidewhetherDirectisbetter,Candidateisbetter,ortheyaretied\.Identifyanyrequirementthatislost,weakened,broadened,orduplicatedbythecandidate\.

DonotuseGoldrubrics,responses,calibrationscores,orpreferencelabels\.

Returnthecomparisonforallfourdimensions,anycandidateregressions,andabriefsummary\.

ScoreeverycalibrationresponseagainsttheONErubricbelowindependently\.

QUERY

<QUERY\>

RUBRIC

<RUBRIC\>

CALIBRATIONRESPONSES

<CALIBRATION\_RESPONSES\>

Fordesired\_state=criterion\_met,ahighscoremeansthattheresponsesatisfiesthecriterion\.Fordesired\_state=criterion\_not\_met,ahighscoremeansthattheundesirablebehaviorisabsent\.

Usethefollowingscale\.

1meansclearlyviolatingthedesiredstate\.

2meansmostlyviolatingit\.

3meansmixed,partial,orgenuinelyborderline\.

4meansmostlysatisfyingitwithaminordefect\.

5meansclearlyandfullysatisfyingit\.

Judgeonlyobservableresponsecontent\.Donotusetheresponse’sdeclaredqualitytargetasasubstituteforinspection\.

Returnonescoreforeachcalibrationresponse\.

We do not provide a separate prompt for RQ3, as the same judging protocol is used across all settings, with criterion\-level judgments held fixed while only the score aggregation scheme is varied\.

### D\.2Prompts for ARTS\-Hybrid

In this section, we provide the core prompts used in ARTS\-Hybrid\. We first present the node decomposition prompt used to generate candidate child criteria, followed by the semantic validation prompt used to identify semantically feasible candidates\. Calibration\-based selection does not require an additional prompt, as feasible candidates are selected programmatically using the calibration objective defined above\. Once the tree search is complete, ARTS\-Hybrid performs global validation over the resulting leaf set, for which we also provide the corresponding prompt below\.

USERQUERY

<QUERY\>

PARENTRUBRIC

<PARENT\_RUBRIC\>

Generate<N\_CANDIDATES\>sub\-rubricsthatjointlypreserveandcoverthemeaningoftheparentrubricforthisquery\.

Eachsub\-rubricmustbeconcrete,verifiable,independentlymeaningful,specifictothequery,andsufficientlyunambiguousfordifferentjudgestointerpretconsistently\.Avoidredundantnear\-duplicates\.

UseANDsemantics\.Everyselectedchildshouldrepresentanecessaryaspectoftheparent\.Donotassignweights\.

Returntheselectedsub\-rubrics\.

Evaluatewhetherarubriccriterionissuitableforthisexacttask\.

USERQUERY

<QUERY\>

CRITERION

<CRITERION\>

Assessthefollowingpropertiesona0\-1scale\.

\-concretemeansspecific,observable,andnotvague\.

\-verifiablemeansanswerableasYES/NOfromtheresponse\.

\-unambiguousmeanslikelytobeinterpretedconsistentlyacrossjudges\.

\-task\_specificmeanstiedtothequeryratherthangenericquality\.

\-distinctivemeanscapableofdistinguishingresponsesofdifferentquality\.

\-leaf\_qualitymeanssuitableasafinalscoringcriterion\.

\-atomicmeanscontainingexactlyoneindependentlyfail\-ablerequirement\.

Alsodeterminewhetherthecriterionissubstantive,format\-only,andanexplicitqueryrequirement\.

Acriterionissemanticallyadmissibleonlyifitisconcrete,verifiable,unambiguous,andtask\-specific\.Avalidbutnon\-atomiccriterionmayberetainedasaninternalnodebutnotasafinalleaf\.

Returntheassessmentforallpropertiestogetherwiththefinaldecisionandabriefreason\.

Checkwhetherthefinalleafrubricsetadequatelycoverstheevaluationneedsofthistask\.

USERQUERY

<QUERY\>

ROOTRUBRIC

<ROOT\_RUBRIC\>

FINALLEAFRUBRICS

<LEAF\_RUBRICS\>

Theleafsetmustsatisfythefollowingrequirements\.

\-Covertheprincipaltask\-specificrequirements\.

\-Preserveeveryexplicitqueryconstraint\.

\-Expressindependentlyfail\-ablerequirementsasseparateleaves\.

\-Containconcreteanddirectlyscoreablecriteria\.

\-Avoidsemanticduplicationandstrictsubsumption\.

\-Includesubstantivecorrectness,reasoning,ordomaincriteriawhenthetaskcontainssubstantivecontent\.

Ifvalidationfails,identifythemissingaspects,atomicityviolations,andredundantpairs\.

Returnthevalidationdecisiontogetherwiththecorrespondingdiagnostics\.

## Appendix ECase Study of ARTS\-Hybrid Rubric Construction

We present a representative example to illustrate how ARTS\-Hybrid structures and refines evaluation criteria compared with direct rubric generation\.

QueryWhat is the author’s purpose for writing this article? What words or phrases indicate that purpose? How does this text connect back to Chapter One and the Prologue ofHidden Figures?

Direct\-Generated RubricD1\.Identify the author’s primary purpose\.D2\.Provide at least two words or phrases indicating that purpose\.D3\.Explain how the text connects to both Chapter One and the Prologue\.D4\.Write the response in complete sentences\.

ARTS\-Hybrid\-Generated RubricA1\.Identify the author’s purpose for writing the article\.A2\.Cite specific words or phrases that indicate this purpose\.A3\.Explain how the article connects to Chapter One\.A4\.Support the Chapter One connection with details from the article\.A5\.Explain how the article connects to the Prologue\.A6\.Use relevant evidence to support the cross\-text connections\.

Compared with direct generation, the ARTS\-Hybrid produces finer\-grained criteria that separate independently assessable requirements\. The directly generated rubric combines the connections to Chapter One and the Prologue into a single criterion, while introducing additional constraints such as requiring at least two supporting phrases and complete sentences\.