SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

# Sketch, Expand, Sort, Summarize, EvaluateLLM-as-Judge Evaluation via Structured Decomposition
Source: [https://arxiv.org/html/2608.18303](https://arxiv.org/html/2608.18303)
###### Abstract

LLM\-as\-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity\. We proposeSESSE\(Sketch, Expand, Sort, Summarize, Evaluate\), a training\-free framework that decomposes holistic judgment into structured sub\-questions mined directly from the judge’s own error cases — requiring no oracle responses, task\-specific rubrics, or fine\-tuning\. On RewardBench \(n=1,000n\{=\}1\{,\}000\), SESSE achieves near\-parity with the chain\-of\-thought baseline and is competitive with RISE\-Judge\-32B \(92\.7%\), a fine\-tuned specialist, while remaining fully training\-free\. Per\-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token\.

## 1Introduction

LLM\-as\-judge evaluation has become a standard paradigm for scalable response quality assessment — popularized by MT\-Bench[20](https://arxiv.org/html/2608.18303#bib.bib4), Chatbot Arena[1](https://arxiv.org/html/2608.18303#bib.bib21), and AlpacaFarm[2](https://arxiv.org/html/2608.18303#bib.bib5)— yet it reduces evaluation to a single holistic A/B preference choice\. Even with chain\-of\-thought prompting, the reasoning is unstructured free text produced in a single pass — offering no mechanism to decompose quality into auditable dimensions, isolate which aspects drove the preference, or detect systematic annotation noise\.

![Refer to caption](https://arxiv.org/html/2608.18303v1/figure/sesse_high_level.png)Figure 1:SESSE replaces holistic judgment with structured decomposition — criteria emerge automatically from the judge’s own error cases, producing per\-criterion vote evidence\.Existing approaches fall into two paradigms, each with a critical limitation\. The first,*criterion\-based*methods — G\-Eval[10](https://arxiv.org/html/2608.18303#bib.bib11), FActScore[11](https://arxiv.org/html/2608.18303#bib.bib12), UniEval[21](https://arxiv.org/html/2608.18303#bib.bib19), CheckList[14](https://arxiv.org/html/2608.18303#bib.bib14)— decompose quality into predefined dimensions but require*manually specified criteria*or human\-designed rubrics\. The second,*trained specialist*methods, fits directly to labeled data: oracle\-dependent approaches such as DeCE[17](https://arxiv.org/html/2608.18303#bib.bib17)and CheckEval[8](https://arxiv.org/html/2608.18303#bib.bib8)derive criteria from gold responses, while RISE\-Judge[18](https://arxiv.org/html/2608.18303#bib.bib15)and other fine\-tuned variants — JudgeLM[22](https://arxiv.org/html/2608.18303#bib.bib20), Prometheus 2[6](https://arxiv.org/html/2608.18303#bib.bib7), Auto\-J[9](https://arxiv.org/html/2608.18303#bib.bib10)— improve calibration via SFT\+DPO at the cost of training data and parameter updates\. SESSE differs from all prior work: criteria emerge*directly from the judge’s own error distribution*on standard pairwise labels, requiring no oracle responses, task\-type annotation, or fine\-tuning\.

We proposeSESSE, a training\-free framework that replaces holistic judgment with structured decomposition\. Error cases from a first\-pass holistic run on a dev set drive automatic mining of evaluation criteria\. These criteria are clustered intokkgroups and generalized into a reusable A/B/NA \(Not Applicable\) sub\-question bank\. Independent per\-criterion voting produces an interpretable audit trail and a final preference; ordering randomization across position mitigates positional bias[15](https://arxiv.org/html/2608.18303#bib.bib16), and constraining the judge to discrete tokens A/B/NA yields superior calibration over open\-ended scoring[19](https://arxiv.org/html/2608.18303#bib.bib18);[13](https://arxiv.org/html/2608.18303#bib.bib13)\.

Decomposition addresses the reliability problem central to holistic judgment: a single output token conflates every quality dimension into one binary decision, so two judges — or one judge across two runs — can disagree without any way to localize why\. By contrast, each SESSE sub\-question is independently verifiable: an annotator, a second model, or the same judge at a later time can re\-check a single A/B/NA vote in isolation, without re\-deriving the entire holistic judgment\. This mirrors why checklist\- and rubric\-style protocols improve inter\-rater agreement over free\-form human judgments[14](https://arxiv.org/html/2608.18303#bib.bib14): breaking one hard judgment into many easy ones bounds disagreement to specific, inspectable criteria rather than an opaque holistic vote\.

Our contributions are:\(i\)SESSE, a fully automated 5\-stage pipeline that mines evaluation criteria from judge error cases, clusters them into a reusable sub\-question bank, and aggregates per\-criterion votes — without oracle responses, rubrics, or fine\-tuning; and\(ii\)empirical evidence that SESSE is competitive with fine\-tuned specialist evaluators on RewardBench while providing per\-criterion diagnostic evidence unavailable from holistic judgment\.

## 2Method

![Refer to caption](https://arxiv.org/html/2608.18303v1/figure/sesse_detail_level.png)Figure 2:SESSE pipeline: offline bank construction \(Stages 0–4, once\) and online per\-example inference \(Stage 5\)\.SESSE operates as a five\-stage pipeline \(Figure[2](https://arxiv.org/html/2608.18303#S2.F2)\)\.Stages 0–1 \(Error Collection, Sketch\)\.Stage 0 runs the judge holistically on the dev set, collecting cases where the prediction disagrees with the ground\-truth label\. Stage 1 prompts the judge to generate conditional evaluation criteria from these errors as sub\-questions:“If \[protasis: the condition under which the criterion applies\], which response better satisfies \[apodosis: the evaluation property\]: A, B, or NA?”Criteria emerge from the judge’s own failure distribution — unlike Prometheus[5](https://arxiv.org/html/2608.18303#bib.bib6)and G\-Eval[10](https://arxiv.org/html/2608.18303#bib.bib11), which require manually specified dimensions\.

Stages 2–3 \(Expand, Sort\)\.Stage 2 embeds candidate sub\-questions and clusters them intokkthematic groups\. Stage 3 ranks sub\-questions within each cluster via greedy farthest\-first traversal[4](https://arxiv.org/html/2608.18303#bib.bib3), producing a diversity\-ordered active bank of depthn≤nmaxn\\leq n\_\{\\max\}without rerunning clustering, enforcingcriterion orthogonalityacross the bank\. This ranked structure enables depth ablation at inference time \(Appendix[A](https://arxiv.org/html/2608.18303#A1)\)\.

Stage 4 \(Summarize\)\.An LLM call produces onegeneralizedprotasis per cluster, serving as a human\-readable cluster label\. Near\-duplicate clusters are merged via LLM binary equivalence judgment; sub\-questions from merged clusters are pooled and top\-nnre\-selected\.

Stage 5 \(Evaluate\)\.Tier 1 presents each abstract protasis to the judge and selects only relevant clusters for the current example, skipping inapplicable clusters to reduce per\-example cost\. Tier 2 answers each selected question with itsoriginal\(error\-derived\) protasis paired with its apodosis, preserving the concrete, instance\-specific phrasing from Stage 1\. When a criterion does not apply \(e\.g\., a code cluster on a poetry prompt\), the judge returns NA; NA votes are excluded from the A/B tally, letting the judge gracefully skip criteria that do not apply\. The final preference is the majority of all A and B ballots across selected clusters; if A and B votes are equal, the example is declared a tie and excluded from the accuracy denominator\.

## 3Experiments

### 3\.1Setup

Dataset:RewardBench[7](https://arxiv.org/html/2608.18303#bib.bib9), 2,985 pairwise preference examples across chat, chat\-hard, safety, and reasoning subsets\. Dev \(n=1,985n\{=\}1\{,\}985\) for error collection; held\-out val \(n=1,000n\{=\}1\{,\}000\) for all reported metrics\. The dev/val split is a random partition of the full 2,985\-example set: 1,000 examples held out for val, with the remainder used as dev\.

Metric:Pairwise preference accuracy on held\-out val\. Table[1](https://arxiv.org/html/2608.18303#S3.T1)reports CoT holistic accuracy split by whether SESSE reaches a decision:*non\-tie*rows \(SESSE commits to a preference\) and*tie*rows \(equal A/B votes, SESSE has no preference to offer\); Tie% is the fraction of tied examples\. SESSE accuracy is reported on non\-tie rows only\.

Models:Qwen2\-VL\-7B[16](https://arxiv.org/html/2608.18303#bib.bib1), Gemini Flash Lite, and Gemini 2\.5 Flash[3](https://arxiv.org/html/2608.18303#bib.bib2)\.

Configuration:k=25k\{=\}25clusters \(selected by preliminary silhouette analysis\),n=10n\{=\}10, temperature 0\. Optimal depthn∗n^\{\*\}per model is selected by dev accuracy\.

### 3\.2Main Results

Table 1:RewardBench val \(n=1,000n\{=\}1\{,\}000\), dev\-optimaln∗n^\{\*\}\.‡p<0\.05\{\}^\{\\ddagger\}p\{<\}0\.05McNemar \(non\-tie subset\)\.†SFT\+DPO\.Table[1](https://arxiv.org/html/2608.18303#S3.T1)presents SESSE results at the dev\-optimaln∗n^\{\*\}across all three evaluated models\. To ensure a fair comparison, all gaps between the holistic baseline and SESSE are computed on non\-tie rows only, matching the McNemar test, which likewise restricts its analysis to the pairs where the two methods disagree, testingH0H\_\{0\}: SESSE and the CoT holistic baseline are equally likely to be correct when their predictions diverge\. Gemini 2\.5 Flash shows a minor 1\.3% relative gap, which is not statistically significant \(p\>0\.05p\{\>\}0\.05\)\. Flash Lite and Qwen2\-VL\-7B show statistically significant relative gaps of 6\.8% and 5\.0%, respectively \(p<0\.05p\{<\}0\.05\)\. Notably, SESSE with Gemini 2\.5 Flash reaches 93\.3% accuracy alongside a low 3\.5% tie rate, placing it in the same performance tier as the specialist evaluator RISE\-Judge \(32B\)[18](https://arxiv.org/html/2608.18303#bib.bib15)\(92\.7%\); while RISE\-Judge requires SFT\+DPO fine\-tuning, SESSE achieves comparable performance while remaining fully training\-free and providing per\-criterion evidence\.

Depth ablation \(Appendix[A](https://arxiv.org/html/2608.18303#A1)\) shows a per\-tier pattern: the capable judge \(Gemini 2\.5 Flash\) saturates atn=2n\{=\}2sub\-questions per cluster, the mid\-tier judge \(Flash Lite\) benefits from broader evidence and peaks atn=5n\{=\}5, while the weakest judge \(Qwen2\-VL\-7B\) peaks earlier, atn=3n\{=\}3, before degrading\. Qwen2\-VL\-7B’s degradation pastn=3n\{=\}3reveals a bank quality effect: accuracy peaks atn=3n\{=\}3then degrades, because 51\.9% of instances are covered by a single dominant protasis in the self\-generated bank — beyondn=3n\{=\}3, the depth budget is consumed by redundant sub\-questions from that cluster\. A cross\-model transfer experiment — regenerating the bank with Gemini while keeping Qwen2\-VL\-7B as judge — confirms the accuracy ceiling is set by judge capability, not bank quality \(Table[B\.1](https://arxiv.org/html/2608.18303#A2.T1), Appendix[B](https://arxiv.org/html/2608.18303#A2)\)\.

SESSE tie rows are harder examples: holistic accuracy on tie rows is consistently lower than on non\-tie rows across all three models \(Table[1](https://arxiv.org/html/2608.18303#S3.T1)\)\. The gap is most pronounced for capable judges — Gemini 2\.5 Flash drops from 94\.5% to 69\.5% on tie rows, suggesting that ties signal genuine ambiguity rather than arbitrary disagreement\.

## 4Discussion

Training\-free parity with fine\-tuned judges\.SESSE’s near\-parity with CoT holistic for Gemini 2\.5 Flash \(Table[1](https://arxiv.org/html/2608.18303#S3.T1)\) represents a minor trade\-off: near\-parity in exchange for structured, auditable per\-criterion evidence at no training cost\. CoT holistic itself already outperforms RISE\-Judge[18](https://arxiv.org/html/2608.18303#bib.bib15), so SESSE’s comparable accuracy mainly shows that it preserves the base model’s advantage without requiring fine\-tuning\. For mid\-tier judges, the gap is significant but reflects a different dynamic: Flash Lite’s drop stems primarily from Tier\-1 false positives on safety subsets \(seeDomain\-specific signal recoverybelow\), not from the decomposition mechanism itself\.

Domain\-specific signal recovery\.SESSE’s impact varies substantially across RewardBench subsets\. Ondonotanswer, Gemini 2\.5 Flash gains \+20\.6% relative over holistic \(75\.9%→91\.5%75\.9\\%\\rightarrow 91\.5\\%\) as explicit safety\-refusal criteria isolate a signal that holistic judgment conflates with general preference\. Conversely,xstest\-should\-responddrops 27\.1% relative due to Tier\-1 false positives on borderline\-safe prompts, pointing to Tier\-1 precision as the key failure mode for domain\-sensitive subsets\.

When to prefer SESSE\.SESSE is preferable when interpretability matters: the per\-criterion vote trail enables annotation auditing and exposes label noise, justifying its inference overhead beyond raw accuracy\.

Tie rate as a deployment\-time capability signal\.Tie% is a label\-free proxy for judge reliability: Qwen2\-VL\-7B’s 21\.3% tie rate — versus 2\.6–3\.5% for the Gemini models — predicts its accuracy gap, since the judge cannot form a directional preference on roughly one\-in\-five examples \(which score only 66\.7% accuracy when forced holistically; Table[1](https://arxiv.org/html/2608.18303#S3.T1)\)\. A high Tie% on a held\-out probe thus signals a capability mismatch or domain shift without requiring ground\-truth labels, letting a practitioner switch judges, retarget the bank, or flag examples for human review\.

Failure modes\.When SESSE diverges from holistic judgment, analysis reveals capability\-dependent failure modes \(see Table[C\.1](https://arxiv.org/html/2608.18303#A3.T1), Appendix[C](https://arxiv.org/html/2608.18303#A3), for the full three\-bucket taxonomy\)\. We conducted further analysis on Flash Lite and Gemini 2\.5 Flash \(SESSE accuracy above 80%\) to understand failure modes\. For Flash Lite, disagreements split roughly evenly between genuine label ambiguity \(49%, high vote entropy\), decomposition failure \(26%\), and semantic equivalence \(26%\), reflecting a mid\-tier judge without a single dominant failure signature\. For the capable judge \(Gemini 2\.5 Flash\), 48% of disagreements instead reflect*semantic equivalence*between responses \(C3\): these pairs have higher response similarity than the dataset average \(0\.75 vs\. 0\.69 mean cosine similarity between Sentence\-BERT embeddings;[12](https://arxiv.org/html/2608.18303#bib.bib22)\), and the bank correctly returns high NA rates when no meaningful distinction exists\. This C3 signature is not a model failure but a dataset property — nearly half of Gemini’s degradation rows encode stylistic preference rather than substantive quality differences, making the per\-criterion vote distribution a lightweight benchmark audit tool independent of accuracy measurement\. In all cases, per\-criterion vote distributions make failure modes explicitly diagnosable, a feature absent from monolithic A/B scoring\.

## 5Conclusion

We presented SESSE, a training\-free framework that decomposes holistic LLM\-as\-judge evaluation into automatically derived A/B/NA sub\-questions mined from the judge’s own error distribution\. On RewardBench, SESSE achieves a non\-significant gap to CoT holistic for capable judges and is in the same performance range as a fine\-tuned specialist evaluator \(RISE\-Judge 32B\) while remaining training\-free\. Beyond accuracy, the per\-criterion vote distributions provide interpretable diagnostic evidence — surfacing failure modes and benchmark quality issues that holistic scoring leaves opaque — a use case that scales with judge capability rather than against it\.

## Limitations

Bank portability\.The question bank is derived from a specific judge model’s error distribution on a specific dev set\. Changing the judge model or evaluation domain requires re\-running Stages 0–4 \(a one\-time offline cost\)\. Amortization strategies — cross\-domain bank transfer, incremental error pool updates, shared banks across related judge models — are not explored here\.

Cluster count & benchmark scope\.We fixk=25k\{=\}25by silhouette analysis; ablation overkkand generalization beyond RewardBench \(open\-ended generation, multi\-turn, non\-English\) are deferred to future work\.

Inference cost\.SESSE atn=10n\{=\}10requires∼282\{\\sim\}282LLM calls per example vs\. 1 holistic; capable judges saturate atn=2n\{=\}2, reducing cost proportionally with no accuracy loss\.

Decomposition limits\.A/B/NA sub\-questions cannot capture end\-to\-end execution evaluation \(code compilation, derivation correctness\) — the primary source of SESSE degradation on reasoning and code subsets\.

## Ethical Considerations

SESSE evaluates LLM outputs using an LLM judge, inheriting any biases present in the judge model’s error distribution\. Criteria mined from judge failures may encode systematic biases — users deploying SESSE in high\-stakes annotation pipelines should audit the generated question bank for bias before use\. All experiments use publicly available models and datasets\.

## References

- Chianget al\.\(2024\)W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, B\. Zhu, H\. Zhang, M\. I\. Jordan, J\. E\. Gonzalez, and I\. StoicaChatbot Arena: an open platform for evaluating LLMs by human preference\.InProceedings of the 41st International Conference on Machine Learning,pp\. 8359–8388\.Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p1.1)\.
- Duboiset al\.\(2023\)Y\. Dubois, X\. Li, R\. Taori, T\. Zhang, I\. Gulrajani, J\. Ba, C\. Guestrin, P\. Liang, and T\. B\. HashimotoAlpacaFarm: a simulation framework for methods that learn from human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p1.1)\.
- Gemini Team \(2024\)Gemini TeamGemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.Cited by:[§3\.1](https://arxiv.org/html/2608.18303#S3.SS1.p3.1)\.
- Gonzalez \(1985\)T\. F\. GonzalezClustering to minimize the maximum intercluster distance\.Theoretical Computer Science38,pp\. 293–306\.Cited by:[§2](https://arxiv.org/html/2608.18303#S2.p2.1)\.
- Kimet al\.\(2024a\)S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. SeoPrometheus: inducing fine\-grained evaluation capability in language models\.InProceedings of the Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.18303#S2.p1.1)\.
- Kimet al\.\(2024b\)S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. SeoPrometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.
- Lambertet al\.\(2024\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. HajishirziRewardBench: evaluating reward models for language modeling\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2403.13787)Cited by:[§3\.1](https://arxiv.org/html/2608.18303#S3.SS1.p1.1)\.
- Leeet al\.\(2025\)Y\. Lee, J\. Kim, J\. Kim, H\. Cho, J\. Kang, P\. Kang, and N\. KimCheckEval: a reliable LLM\-as\-a\-judge framework for evaluating text generation using checklists\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15771–15798\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.796/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.796),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.
- Liet al\.\(2024\)J\. Li, S\. Sun, W\. Yuan, R\. Fan, H\. Zhao, and P\. LiuGenerative judge for evaluating alignment\.InProceedings of the 12th International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1),[§2](https://arxiv.org/html/2608.18303#S2.p1.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,Cited by:[§4](https://arxiv.org/html/2608.18303#S4.p5.1)\.
- Renet al\.\(2023\)J\. Ren, Y\. Zhao, T\. Vu, P\. J\. Liu, and B\. LakshminarayananSelf\-evaluation improves selective generation in large language models\.InNeurIPS 2023 Workshop on Robustness of Few\-shot and Zero\-shot Learning in Foundation Models,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p3.1)\.
- Ribeiroet al\.\(2020\)M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. SinghBeyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1),[§1](https://arxiv.org/html/2608.18303#S1.p4.1)\.
- Wanget al\.\(2024a\)P\. Wang, L\. Li, L\. Chen, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not robust multiple choice selectors\.InProceedings of the 12th International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p3.1)\.
- Wanget al\.\(2024b\)P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, Y\. Fan, K\. Dang, M\. Du, X\. Ren, R\. Men, D\. Liu, C\. Zhou, J\. Zhou, and J\. LinQwen2\-VL: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2409.12191)Cited by:[§3\.1](https://arxiv.org/html/2608.18303#S3.SS1.p3.1)\.
- Yuet al\.\(2025a\)F\. Yu, N\. Seedat, D\. Herrmannova, F\. Schilder, and J\. R\. SchwarzBeyond pointwise scores: decomposed criteria\-based evaluation of LLM responses\.arXiv preprint arXiv:2509\.16093\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2509.16093)Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.
- Yuet al\.\(2025b\)J\. Yu, S\. Sun, X\. Hu, J\. Yan, K\. Yu, and X\. LiImprove LLM\-as\-a\-judge ability as a general ability\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 14099–14115\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.712/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.712)Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.18303#S3.SS2.p1.1),[Table 1](https://arxiv.org/html/2608.18303#S3.T1.2.6.1.1.2),[§4](https://arxiv.org/html/2608.18303#S4.p1.1)\.
- Zhaoet al\.\(2021\)Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. SinghCalibrate before use: improving few\-shot performance of language models\.InProceedings of the 38th International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p3.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p1.1)\.
- Zhonget al\.\(2022\)M\. Zhong, Y\. Liu, D\. Yin, Y\. Mao, Y\. Jiao, P\. Liu, C\. Zhu, H\. Ji, and J\. HanTowards a unified multi\-dimensional evaluator for text generation\.Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.131)Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.
- Zhuet al\.\(2024\)L\. Zhu, X\. Wang, and X\. WangJudgeLM: fine\-tuned large language models are scalable judges\.InProceedings of the 38th AAAI Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2608.18303#S1.p2.1)\.

## Appendix ADepth Ablation

Table[A\.1](https://arxiv.org/html/2608.18303#A1.T1)sweepsn=1n\{=\}1–1010sub\-questions per cluster for all three self\-generated banks \(bold = best per model\)\. All models plateau quickly: accuracy stabilizes within the first few values ofnnand shows little change beyond that, suggesting a small question budget is sufficient once the most informative sub\-questions are exhausted\. Qwen2\-VL\-7B is the exception, showing a slight degradation at largernndue to limited cluster diversity in the self\-generated bank\.

Table A\.1:Accuracy \(excl\. ties\) vs\.nn\(self\-generated banks; bold =n∗n^\{\*\}\)\.
## Appendix BCross\-Model Bank Transfer

Table[B\.1](https://arxiv.org/html/2608.18303#A2.T1)compares Qwen2\-VL\-7B accuracy when evaluated with a self\-generated bank versus a bank whose questions were generated by Gemini 2\.5 Flash, holding the judge model fixed\. Both banks reach the same peak accuracy \(66\.3%\), showing that substituting a stronger model’s questions does not meaningfully change evaluation performance when the judge model is held fixed\.

Table B\.1:Qwen2\-VL\-7B with Gemini\-generated vs\. self\-generated bank \(val\)\.
## Appendix CFailure Mode Taxonomy

Table[C\.1](https://arxiv.org/html/2608.18303#A3.T1)reports the per\-configuration breakdown of degradation rows — examples where SESSE and the holistic baseline disagree\. We report this breakdown only for configurations with SESSE accuracy above 80% \(Flash Lite, Gemini 2\.5 Flash\); at lower accuracy, degradation rows are too frequent to characterize as a small set of distinct failure signatures\. Each row is attributed to one of three buckets:C1 \(Ambiguous\)reflects genuine label difficulty, where sub\-question votes are evenly split;C2 \(Decomposition failure\)indicates the judge is confidently wrong, with≥\\geq65% of non\-NA votes against the ground\-truth label;C3 \(Semantic equivalence\)indicates the bank correctly declines to pick a winner, returning high NA rates when the two responses are too similar to discriminate\.

Table C\.1:Degradation bucket distribution\. C1 = genuine ambiguity \(high vote entropy\); C2 = decomposition failure \(SESSE consistently wrong≥\\geq65% non\-NA votes against label\); C3 = semantic equivalence \(high NA rate, responses too similar to discriminate\)\. Percentages of degradation rows per config\.