Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

# Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Source: [https://arxiv.org/html/2606.06546](https://arxiv.org/html/2606.06546)
Tao Liu1,2, Ye Lu1,2, Ruohua Zhang1,2, Siyu Song1,2, Wentao Liu1,2,3, Aimin Zhou1,2,3, and Hao Hao1,\* 1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China 2School of Computer Science and Technology, East China Normal University, Shanghai 200062, China 3Shanghai Innovation Institute, Shanghai 200231, China Corresponding author:Hao Hao \(hhao@mail\.ecnu\.edu\.cn\)

###### Abstract

Evaluating large language models \(LLMs\) for education requires measuring how models teach, not only what they know\. Existing benchmarks emphasize domain\-general correctness or depend on manually designed rubrics that scale poorly to long\-tail pedagogical scenarios\. We introduceElmes\+, an end\-to\-end framework for constructing, refining, and applying fine\-grained scenario\-specific rubrics\.Elmes\+combines a declarative multi\-agent engine for teacher–student–judge interactions withSceneGen, a self\-evolving module that co\-optimizes evaluation criteria and test data from expert\-defined pedagogical dimensions\. UsingElmes\+, we build Edu\-330, covering 330 scenarios across 11 subjects, 3 grade bands, and 10 task types, with over 1,000 second\-level indicators\. Experiments on Edu\-330 and four expert\-authored gold\-standard scenarios show that educational capability is multidimensional: top\-tier LLMs differ mainly in creativity and values integration, knowledge\-strong models may fail at Socratic scaffolding, and the education\-specialized InnoSpark achieves the best human\-evaluated average score\. LLM judges preserve human\-comparable rankings with much lower scoring variance, but exhibit judge\-specific biases such as self\-preference\. Ablations show that expert\-scored few\-shot anchoring improves human–LLM alignment, while reasoning enforcement and greedy decoding are model\-dependent\.Elmes\+thus provides scalable diagnostic infrastructure for pedagogically grounded LLM evaluation\.

Elmes\+: Automated Construction of Fine\-Grained Evaluation Rubrics for Large Language Models in Long\-Tail Educational Scenarios

Tao Liu1,2, Ye Lu1,2, Ruohua Zhang1,2, Siyu Song1,2, Wentao Liu1,2,3, Aimin Zhou1,2,3, and Hao Hao1,\*1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China2School of Computer Science and Technology, East China Normal University, Shanghai 200062, China3Shanghai Innovation Institute, Shanghai 200231, ChinaCorresponding author:Hao Hao \(hhao@mail\.ecnu\.edu\.cn\)

## 1Introduction

Large language models have rapidly entered educational technology, powering intelligent tutoring, lesson planning, and personalized content generation\(Kasneciet al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib1); Yanet al\.,[2024](https://arxiv.org/html/2606.06546#bib.bib2)\)\. Yet evaluating whether an LLM is genuinely effective as a pedagogical agent remains difficult: unlike factual question answering, educational tasks are subjective, multi\-dimensional, and context\-dependent, requiring models to balance content accuracy, scaffolding, emotional sensitivity, and cultural appropriateness\(Wanget al\.,[2024a](https://arxiv.org/html/2606.06546#bib.bib17)\)\.

Existing evaluation paradigms remain insufficient\. General\-purpose benchmarks such as MMLU\-Pro\(Wanget al\.,[2024b](https://arxiv.org/html/2606.06546#bib.bib4)\), C\-Eval\(Huanget al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib5)\), and GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.06546#bib.bib6)\)emphasize knowledge or reasoning rather than pedagogical process quality; interactive frameworks such as MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib9)\)and Chatbot Arena\(Chianget al\.,[2024](https://arxiv.org/html/2606.06546#bib.bib10)\)provide coarse preference signals; and education\-specific benchmarks\(Danet al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib12); Xuet al\.,[2025](https://arxiv.org/html/2606.06546#bib.bib13)\)cover limited scenarios with manually curated rubrics\.

The core challenge is the rubric construction bottleneck: fine\-grained criteria require pedagogical expertise, while educational scenarios grow combinatorially with subject, grade, and task type\.

We address this challenge withElmes\+, which automates educational LLM evaluation from rubric generation to scoring and refinement\. Our design principle is that evaluation metrics and test data should co\-evolve: score distributions can reveal overly strict, overly lenient, or weakly discriminative rubrics, while test data can be regenerated to better exercise target pedagogical dimensions\.

Elmes\+makes the following contributions:

1. 1\.Multi\-Agent Evaluation Engine\.We design a declarative YAML\-to\-DAG engine that supports teacher–student–judge role configurations, conditional routing, trajectory\-level scoring, and expert\-in\-the\-loop scenario refinement for both single\-turn and multi\-turn educational tasks\.
2. 2\.Self\-Evolving Rubric Synthesis \(SceneGen\)\.We introduce a closed\-loop synthesis module that starts from 4 expert\-defined dimensions and co\-evolves rubrics and test data by diagnosing overly strict, overly lenient, or low\-discrimination metrics, producing over 1,298 second\-level indicators for 330 scenarios\.
3. 3\.Large\-scale Benchmark and Expert\-level Evaluation\.We constructEdu\-330, a benchmark spanning 330 core scenarios, and complement it with 4 expert\-designed pedagogical scenarios to reveal model\-specific educational profiles and trade\-offs overlooked by domain\-agnostic benchmarks\.
4. 4\.Consistency Enhancement and Human\-AI Alignment\.We analyze and improve LLM\-as\-a\-judge reliability through expert anchoring and reasoning constraints, reducing mean\-score deviation from human experts by approximately 30% while maintaining high inter\-rater reliability\.

## 2Related Work

### 2\.1General\-Purpose LLM Benchmarks

LLM evaluation has been dominated by knowledge\- and reasoning\-centric benchmarks\. MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.06546#bib.bib8)\)and MMLU\-Pro\(Wanget al\.,[2024b](https://arxiv.org/html/2606.06546#bib.bib4)\)assess broad factual knowledge, C\-Eval\(Huanget al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib5)\)targets Chinese\-language knowledge understanding, and GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.06546#bib.bib6)\)and MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.06546#bib.bib8)\)evaluate mathematical reasoning\. While effective at measuring what a model knows, these benchmarks do not assess how well it can teach, which requires evaluating pedagogical process quality rather than outcome correctness alone\.

### 2\.2Interactive and Judge\-Based Evaluation

The LLM\-as\-a\-Judge paradigm\(Zhenget al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib9)\)addresses the scalability limitations of human evaluation by using strong LLMs to assess weaker ones\. MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib9)\)evaluates multi\-turn conversational quality, while Chatbot Arena\(Chianget al\.,[2024](https://arxiv.org/html/2606.06546#bib.bib10)\)crowdsources pairwise preferences at scale\. However, their holistic or pairwise preference signals lack the dimensional granularity needed for educational diagnostics; a model may “win” overall while failing in specific pedagogical dimensions \(e\.g\., metacognition cultivation or emotional support\)\.

Several studies have documented systematic biases in LLM judges, including position bias, verbosity bias, and self\-preference\(Wanget al\.,[2026](https://arxiv.org/html/2606.06546#bib.bib3); Zhenget al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib9)\)\. Prometheus\(Kimet al\.,[2024](https://arxiv.org/html/2606.06546#bib.bib20)\)trains dedicated judge models with fine\-grained, rubric\-based evaluation\. Auto\-J\(Liet al\.,[2024](https://arxiv.org/html/2606.06546#bib.bib11)\)generates critique\-based training data for judge models\. We build on this line of work through multi\-judge ensembling and expert\-anchored calibration, while automatically generating rubrics rather than assuming them as given\.

### 2\.3Education\-Specific LLM Evaluation

Recent work has begun addressing educational evaluation specifically\. EduChat\(Danet al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib12)\)provides a pedagogically aligned chatbot but lacks systematic evaluation rubrics, while EduBench\(Xuet al\.,[2025](https://arxiv.org/html/2606.06546#bib.bib13)\)focuses on educational knowledge rather than teaching process quality\. EQGBench\(Zhouet al\.,[2025](https://arxiv.org/html/2606.06546#bib.bib14)\)targets educational question generation, MathDial\(Macinaet al\.,[2023](https://arxiv.org/html/2606.06546#bib.bib16)\)evaluates mathematical tutoring dialogues, and TutorBench\(Srinivasaet al\.,[2025](https://arxiv.org/html/2606.06546#bib.bib15)\)assesses tutoring quality with process\-level metrics but fixed, manually designed rubrics\.

These efforts still cover limited educational scenarios and often depend on manually designed rubrics, making extension difficult\.Elmes\+addresses both limitations through automated rubric generation and iterative refinement that can scale to hundreds of scenarios with minimal expert effort\.

## 3TheElmes\+Framework

Elmes\+comprises two integrated components \(Figure[1](https://arxiv.org/html/2606.06546#S3.F1)\): a multi\-agent evaluation engine \(§[3\.1](https://arxiv.org/html/2606.06546#S3.SS1)\) and a self\-evolving rubric synthesis module \(§[3\.2](https://arxiv.org/html/2606.06546#S3.SS2)\)\.

![Refer to caption](https://arxiv.org/html/2606.06546v1/x1.png)Figure 1:Overview ofElmes\+\.SceneGenevolves rubrics and test data from expert\-defined pedagogical dimensions, while the multi\-agent engine orchestrates interactions among teachers, students, and judges and improves consistency with expert anchoring and multi\-judge ensembling\.### 3\.1Multi\-Agent Evaluation Engine

#### 3\.1\.1Declarative Configuration System

A central design principle ofElmes\+is to specify evaluation scenarios declaratively rather than programmatically\. Each YAML file defines an educational scenario by binding a model to an instructional role, injecting task\-specific content, specifying the interaction flow, and attaching a rubric\-based judge, allowing experts to adapt scenarios without changing framework code\. Figure[2](https://arxiv.org/html/2606.06546#S3.F2)shows a simplified question\-generation scenario\.

models: test\_model: judge\_model: agents: test\_model: model: test\_model role: educational question generator content: "Generate a contextualized math problem for \{question\}" tasks: mode: iter content: \- question: "Design a Grade 4 problem on equilateral\-triangle perimeter \.\.\.\." \(Generated bySceneGen\) directions: \[START to test\_model, test\_model to END\] evaluation: model: judge\_model format: \[Generated bySceneGen\]

Figure 2:Simplified YAML configuration for defining an educational question\-generation scenario\.
#### 3\.1\.2DAG\-Based Interaction and Evaluation

Thedirectionsspecification is compiled into a LangGraph state graph, where each node represents an agent and edges define the conversation flow\. Edges can either forward one agent’s output directly to the next agent or invoke a router that selects the next node from the current dialogue state, with built\-in support for keyword\-based termination and multi\-condition gates\. For multi\-turn scenarios such as Socratic tutoring, teacher and student agents interact in a loop controlled by a maximum turn budgetTmaxT\_\{\\max\}and task\-specific exit conditions\. The resulting dialogue trajectoryτ=\(m1,m2,…,mn\)\\tau=\(m\_\{1\},m\_\{2\},\\ldots,m\_\{n\}\)is then evaluated by the judge agent under the scenario\-specific rubric\. Given a rubricℛ=\{\(di,wi\)\}i=1D\\mathcal\{R\}=\\\{\(d\_\{i\},w\_\{i\}\)\\\}\_\{i=1\}^\{D\}, where eachdid\_\{i\}is a pedagogical evaluation dimension generated or refined bySceneGenandwiw\_\{i\}is its weight, the overall score is computed as:

Score​\(τ,ℛ\)=∑i=1Dwi⋅si​\(τ,di\)\\text\{Score\}\(\\tau,\\mathcal\{R\}\)=\\sum\_\{i=1\}^\{D\}w\_\{i\}\\cdot s\_\{i\}\(\\tau,d\_\{i\}\)\(1\)wheresi​\(τ,di\)s\_\{i\}\(\\tau,d\_\{i\}\)denotes the judge\-assigned score for trajectoryτ\\tauon dimensiondid\_\{i\}\.

#### 3\.1\.3Interactive Human Expert Interface for Scenario Configuration

To support expert\-in\-the\-loop refinement,Elmes\+provides a Gradio\-based interface that lets educators and curriculum designers load, inspect, edit, and rerun scenario configurations without writing code\. Experts can inject domain knowledge intoSceneGen\-generated rubrics, adjust vague or misaligned criteria, and calibrate test cases through immediate evaluation feedback\. In practice, educators typically converge after 2–3 rounds of configuration updates\. The full interface is shown in Appendix[A](https://arxiv.org/html/2606.06546#A1), Figure[6](https://arxiv.org/html/2606.06546#A1.F6)\.

### 3\.2Self\-Evolving Rubric Synthesis:SceneGen

Another key innovation ofElmes\+isSceneGen, a module that automatically generates and iteratively refines evaluation rubrics for educational scenarios\. The core design is to let evaluation metrics and test data co\-evolve: deficiencies in one can be detected through analysis of the other and corrected in subsequent iterations\.

![Refer to caption](https://arxiv.org/html/2606.06546v1/x2.png)Figure 3:SceneGenworkflow\. From expert\-defined dimensions, this pipeline generates initial metrics and iteratively co\-refines synthesis prompts and evaluation criteria with anchor calibration, early stopping, and best\-version rollback\.#### 3\.2\.1Expert\-Defined Dimensional Framework

Rather than generating evaluation criteria from scratch,SceneGenbegins with a compact set of initial pedagogical dimensions\. In this work, the initial dimensions are derived from no fewer than 70 interviews with education experts and grounded in established theories, including Shulman’s pedagogical content knowledge \(PCK\), Vygotsky’s zone of proximal development, and the TPACK model\(Vygotsky,[1978](https://arxiv.org/html/2606.06546#bib.bib22); Mishra and Koehler,[2006](https://arxiv.org/html/2606.06546#bib.bib23)\)\. This process yields four starting dimensions with expert\-assigned weights:

- •Personalization\(w1=0\.3w\_\{1\}=0\.3\): Adaptation to learner profiles, cognitive levels, and individual needs\.
- •Teaching Method Professionalism\(w2=0\.4w\_\{2\}=0\.4\): Quality of pedagogical strategies, scaffolding, and domain expertise\.
- •Creativity Stimulation\(w3=0\.2w\_\{3\}=0\.2\): Encouragement of critical thinking, divergent reasoning, and intellectual curiosity\.
- •Values Integration\(w4=0\.1w\_\{4\}=0\.1\): Incorporation of positive societal values and cultural sensitivity\.

These weighted dimensions seedSceneGen’s generation of fine\-grained, scenario\-specific rubrics, with larger weights assigning a higher proportion of customized metrics to the corresponding seed dimension\. The seed dimensions are replaceable inputs, makingSceneGena general rubric synthesis framework; this paper instantiates it with the four dimensions above for Edu\-330 and all experiments\.

#### 3\.2\.2Initial Metric Generation

For each educational scenarioss,SceneGengenerates an initial set of fine\-grained metrics from the scenario description and the weighted seed dimensions:

ℳ\(0\)=MetricGen​\(s,\{\(di,wi,desci\)\}i=14\)\\mathcal\{M\}^\{\(0\)\}=\\text\{MetricGen\}\\bigl\(s,\\ \\\{\(d\_\{i\},w\_\{i\},\\text\{desc\}\_\{i\}\)\\\}\_\{i=1\}^\{4\}\\bigr\)\(2\)Here,ssspecifies the educational context \(e\.g\., “middle school mathematics: function concepts”\), and each generated metricm∈ℳ\(0\)m\\in\\mathcal\{M\}^\{\(0\)\}is represented as:

m=\(name,dim,weight,desc,pos\_ex,neg\_ex\)m=\(\\texttt\{name\},\\ \\texttt\{dim\},\\ \\texttt\{weight\},\\ \\texttt\{desc\},\\ \\texttt\{pos\\\_ex\},\\ \\texttt\{neg\\\_ex\}\)\(3\)wherenamelabels the metric,dimlinks it to a seed dimension,weightinherits the correspondingwiw\_\{i\}, anddescgives an operational scoring criterion\. The positive and negative examples act as scoring anchors, helping the judge calibrate scores against concrete quality references rather than abstract descriptions alone\.

#### 3\.2\.3QA Data Generation

Given the current synthesis promptPsyn\(k−1\)P\_\{\\text\{syn\}\}^\{\(k\-1\)\}, the data generator producesNNQA pairs with diverse learner personas and question openings\. To balance controllability and realism, each test case uses a dual\-question design: an internal questionqintq^\{\\text\{int\}\}with explicit length and formatting constraints guides answer generation, while a semantically equivalent user\-facing questionqusrq^\{\\text\{usr\}\}removes such mechanical instructions to simulate real queries\. The resulting pairs\(qusr,a\)\(q^\{\\text\{usr\}\},a\)are evaluated by the multi\-agent engine under the current metric setℳ\(k−1\)\\mathcal\{M\}^\{\(k\-1\)\}, producing an aggregate score that measures data quality and guides the update toℳ\(k\)\\mathcal\{M\}^\{\(k\)\}after this round\.

#### 3\.2\.4Dual\-Track Evolution

Based on the scores and judge rationales from the base loop,SceneGenjointly evolves the synthesis prompt and the evaluation metrics\.

##### Synthesis Prompt Evolution

The prompt optimizer receives the lowest\-scoring QA pairs, including the original question, generated answer, and judge rationale, and rewritesPsynP\_\{\\text\{syn\}\}to address recurring failure patterns\. The revised prompt adds constraints for under\-performing dimensions while preserving instructions associated with high\-scoring cases\.

##### Evaluation Metric Evolution

In parallel,SceneGenuses per\-metric averages and score distributions to identify overly strict, overly lenient, or weakly discriminative criteria\. It only revises metric descriptions and scoring examples while preserving each metric’s name, parent dimension, and weight, so scores remain comparable across iterations\.

#### 3\.2\.5Convergence Safeguards

A key risk in co\-evolution is that changes in score may reflect either better generated data or changed evaluation standards\.SceneGentherefore uses three safeguards implemented in the workflow\. First, it freezes a small set of QA pairs after the first iteration and re\-scores them with later metrics; large changes in their weighted scores indicate possible metric drift\. Second, it tracks the best synthesis prompt and evaluation prompt according to the weighted average scores¯w\\bar\{s\}\_\{w\}and rolls back to that version when a later update degrades performance\. Third, it stops early when recent weighted\-score improvements remain below a configured threshold, avoiding unnecessary iterations after convergence\.

## 4Experiments

### 4\.1Evaluation Results on Edu\-330 Benchmark

We formulate the educational scenario space as a Cartesian product𝒮=𝒟×ℒ×𝒯\\mathcal\{S\}=\\mathcal\{D\}\\times\\mathcal\{L\}\\times\\mathcal\{T\}, comprising 11 disciplines \(Chinese, Mathematics, English, Physics, Chemistry, Biology, Geography, History, Politics, Physical Education, Moral Education\), 3 grade bands \(elementary, middle, high school\), and 10 pedagogical task types \(e\.g\., concept explanation and problem\-solving\)\. ApplyingSceneGento this space gives Edu\-330; for detailed model comparison, we evaluate a representative 66\-scenario subset that preserves the original discipline and grade distributions\.

Figure[4](https://arxiv.org/html/2606.06546#S4.F4)summarizes performance across 6 LLMs, including GPT\-5, Claude\-Opus\-4\.5, Qwen\-2\.5\-72B\-Instruct, Qwen3\-235B\-A22B, DeepSeek\-R1, and Kimi\-k2\.5\. All models are evaluated with the same inputs, rubrics, and primary judge model \(Gemini\-2\.5\-Pro\) under 5\-point scoring\.

![Refer to caption](https://arxiv.org/html/2606.06546v1/figure/llm_eval_overview.png)Figure 4:Performance comparison of mainstream models on the Edu\-330 benchmark\.The Figure[4](https://arxiv.org/html/2606.06546#S4.F4)show clear performance differences across Tier\-1 models like GPT\-5 from weaker models such as Qwen\-2\.5\-72B, demonstrating Edu\-330 provides strong discriminative power\. Furthermore, radar chart reveals that the primary performance gap lies not in Professionalism, but rather in the Creativity and Values\. This indicates that current model limitations in educational scenarios stem from a deficiency in pedagogical delivery and teaching methodology, rather than a lack of factual knowledge\.

### 4\.2Evaluation Results on Expert\-in\-the\-Loop Scenarios

The marginal performance differences among top\-tier models suggest a difficulty ceiling in purely LLM\-generated datasets, which highlights the critical importance of an expert\-in\-the\-loop approach\. To establish a "gold standard" for consistency verification, we invited experts to manually author four enhanced scenarios through ourElmes\+system:Guided Tutoring,Knowledge Explanation,Interdisciplinary Lesson Planning, andScenario\-based Question Generation\. These hand\-written tasks are designed to comprehensively cover educational dimensions across diverse user roles \(teachers, students, and parents\), various interaction modalities \(single\-turn and multi\-turn\), and a wide range of content formats \(see Appendix[B](https://arxiv.org/html/2606.06546#A2)for detailed definitions\)\.

We collected responses from four representative target models \(DeepSeek\-R1, Gemini\-2\.5\-Pro, GPT\-4o, and the education\-vertical model InnoSpark\) across the four gold\-standard scenarios\. Then we invited 11 human experts to conduct a blind scoring of the outputs, without knowing the source models, as the ground\-truth baseline scores\. The expert panel achieved Cronbach’sα\\alphabetween 0\.87 and 0\.89 across scenarios, indicating very high inter\-rater reliability\.

Table 1:Human expert evaluation results across 4 scenarios and 4 dimensions \(5\-point scale\)\. GT: Guided Tutoring, SQ: Scenario\-based Question Generation, KE: Knowledge Explanation, IL: Interdisciplinary Lesson Plan\. Per: Personalization, Cre: Creativity, Val: values, Pro: Professionalism\.As shown in Table[1](https://arxiv.org/html/2606.06546#S4.T1), several conclusions emerged:

- •Difficulty and Potential: The expert\-authored scenarios significantly raised the difficulty, resulting in a general decline in model scores\. Furthermore, the education\-vertical model, InnoSpark, achieved the highest average score \(3\.98\), indicating the room for deeper exploration in specialized pedagogical applications\.
- •Knowledge vs\. Pedagogical: Scores in theProdimension are consistently higher than those inCre\. This trend echoes our earlier findings from the synthesized Edu\-330 benchmark, reaffirming the primary bottleneck for current LLMs in educational scenarios is engaging teaching methodologies and pedagogical adaptability\.

### 4\.3Consistency and Stability ofElmes\+

To further investigate the accuracy and stability ofElmes\+, we introduced a panel of representative models to serve as judges: GPT\-4\.1, Claude\-4\-Sonnet, Gemini\-2\.5\-Pro, and Kimi\-k2\-turbo\-preview\. We assess scoring stability by evaluating the same samples 10 times\.

Table 2:Comparison of Variance \(Var\) and Mean Scores\(Mean\) Across Four Educational Scenarios for Various LLM Judges and Human Experts\. The highest Mean values are highlighted in green, and the lowest are in red\.DeepSeek\-R1Gemini\-2\.5GPT\-4oInnoSparkJudgeVarMeanVarMeanVarMeanVarMeanGuided TutoringClaude\-4\-Sonnet0\.0671\.8640\.054\\cellcolorred\!251\.5800\.016\\cellcolorgreen\!254\.0060\.0153\.121Gemini\-2\.5\-Pro0\.036\\cellcolorred\!252\.1040\.024\\cellcolorgreen\!254\.0080\.011\\cellcolorgreen\!254\.0080\.1033\.961GPT\-4\.10\.0391\.9070\.011\\cellcolorred\!251\.5690\.0332\.9090\.063\\cellcolorgreen\!253\.520Kimi\-k2\-Turbo0\.006\\cellcolorred\!251\.7900\.0602\.6620\.0133\.3030\.024\\cellcolorgreen\!254\.132Human Experts0\.112\\cellcolorred\!252\.7570\.1573\.9360\.2023\.4530\.163\\cellcolorgreen\!254\.033Knowledge ExplanationClaude\-4\-Sonnet0\.0044\.1030\.001\\cellcolorgreen\!254\.4750\.001\\cellcolorred\!254\.0640\.0024\.127Gemini\-2\.5\-Pro0\.0104\.2400\.002\\cellcolorgreen\!254\.6560\.008\\cellcolorred\!254\.0530\.0134\.297GPT\-4\.10\.0024\.7560\.001\\cellcolorgreen\!254\.8840\.0024\.7140\.006\\cellcolorred\!254\.566Kimi\-k2\-Turbo0\.0033\.8930\.009\\cellcolorgreen\!254\.3480\.0043\.7710\.007\\cellcolorred\!253\.671Human Experts0\.1614\.2560\.195\\cellcolorgreen\!254\.5020\.458\\cellcolorred\!253\.3890\.2363\.934Interdisciplinary Lesson PlanningClaude\-4\-Sonnet0\.0023\.4760\.002\\cellcolorgreen\!254\.2260\.000\\cellcolorred\!253\.2290\.0014\.036Gemini\-2\.5\-Pro0\.0094\.4490\.0004\.7170\.002\\cellcolorred\!254\.0950\.001\\cellcolorgreen\!254\.739GPT\-4\.10\.0004\.3860\.0004\.8450\.002\\cellcolorred\!254\.3530\.000\\cellcolorgreen\!254\.872Kimi\-k2\-Turbo0\.0083\.9250\.0014\.3600\.003\\cellcolorred\!253\.3850\.002\\cellcolorgreen\!254\.486Human Experts0\.180\\cellcolorred\!253\.1610\.1573\.9360\.2023\.4530\.163\\cellcolorgreen\!254\.033Scenario\-based Question GenerationClaude\-4\-Sonnet0\.0024\.5220\.000\\cellcolorgreen\!254\.8260\.002\\cellcolorred\!253\.5940\.0004\.418Gemini\-2\.5\-Pro0\.0074\.5740\.000\\cellcolorgreen\!254\.8740\.005\\cellcolorred\!253\.5240\.0094\.670GPT\-4\.10\.0004\.9840\.000\\cellcolorgreen\!255\.0000\.001\\cellcolorred\!254\.7240\.0004\.938Kimi\-k2\-Turbo0\.0024\.6020\.001\\cellcolorgreen\!254\.8600\.001\\cellcolorred\!253\.6280\.0034\.658Human Experts0\.1283\.9580\.171\\cellcolorgreen\!254\.0690\.387\\cellcolorred\!252\.8150\.2953\.982

The comparative results in Table[2](https://arxiv.org/html/2606.06546#S4.T2)reveal critical patterns in stability:

##### Relative Ranking Alignment

: Overall, LLM judges demonstrate comparable capability to human experts in distinguishing high\-performing models\. Specifically, models such as Gemini and InnoSpark receive consistently higher average scores, while GPT\-4o and DeepSeek\-R1 exhibit relatively lower mean scores across evaluations\.

From a scenario\-specific perspective, this alignment is further substantiated\. For instance, GPT\-4o shows significantly depressed performance inScenario\-based Question Generation\. Conversely, Gemini’s exceptionally strong performance inKnowledge Explanation\. These findings validate the effectiveness of theElmes\+in conducting reliable evaluations of LLM educational capabilities\.

##### Superior Stability and Reproducibility

From the perspective of variance, all LLM judges demonstrate significantly higher stability than human experts\. While human scoring variance typically ranges from 0\.11 to 0\.45, LLM judges consistently maintain a variance around 0\.01, with some values approaching zero\. This indicates that the ELMES framework effectively eliminates common uncertainties and fatigue\-induced errors inherent in human subjective evaluation, providing results that are highly reproducible\.

##### Different models exhibit unique evaluative tendencies

GPT\-4\.1 demonstrates a lenient scoring pattern, frequently assigning near\-ceiling scores \(e\.g\., 4\.884 for Gemini\-2\.5 inKnowledge Explanationand 5\.000 inScenario\-based Question Generation\)\. In contrast, Claude\-4\-Sonnet applies consistently stringent standards, particularly in challenging scenarios likeGuided Tutoringwhere it assigns notably low scores\. These divergent tendencies manifest primarily as systematic additive biases rather than rank\-altering distortions, as relative model ordering remains consistent despite absolute score variations\.

![Refer to caption](https://arxiv.org/html/2606.06546v1/figure/paper_combined_no_qwen_mean_variance_no_title.png)Figure 5:Effects of Prompting and Sampling Strategies on the Mean\-Score Bias and Variance Bias of Different Judge Models on Guided Tutoring\.
##### The "self\-preference" phenomenon

It is noteworthy to address the issue of self\-preference bias, where Gemini exhibited ‘problem misinterpretation’ in the Guided Tutoring, providing explanations that deviated from the intended question\. Human experts, while noting the error by giving low scores for thematic relevance, awarded relatively high marks \(3\.936\) for other pedagogical indicators\. In contrast, most LLM judges demonstrated a "zero\-tolerance" policy toward factual errors, leading to systematically lower aggregate scores\. Paradoxically, when Gemini\-2\.5\-Pro served as its own judge, it exhibited severe self\-preference\. Despite the factual error that other models detected, it assigned itself a score of 4\.008—surpassing even the human experts’ lenient average and ranking itself higher than any other judge did\.

These findings underscore the necessity of introducing methods in automated evaluation to neutralize the baseline deviations and "blind spots" of any model\.

### 4\.4Prompt Strategy Ablation

We ablate three scoring optimization strategies on the most complex scenario \(Guided Tutoring\): few\-shot anchoring via expert\-scored examples, reasoning enforcement requiring explicit chain\-of\-thought analysis prior to scoring, and greedy decoding by setting the temperature to 0\. Figure[5](https://arxiv.org/html/2606.06546#S4.F5)presents the ablation results on prompt strategies:

- •Few\-shot anchoring is effective but costly: Providing few\-shot examples consistently improves alignment \(e\.g\., reducing mean deviation by 20% for Claude and GPT\-4\.1\)\. However, this performance gain comes at the direct cost of additional human expert labor required to craft high\-quality reference examples\.
- •Reasoning enforcement is model\-dependent: While reasoning enforcement helps lower evaluation variance, whether it improves consistency is highly model\-dependent \(e\.g\., providing a significant\+0\.992\+0\.992alignment shift for GPT\-4\.1, but remaining marginal for others\)\.
- •Zero temperature yields limited utility: Due to the low baseline variance, the variance reduction achieved by greedy decoding is barely noticeable\. More critically, it risks trapping the model in suboptimal choices, which can significantly widen the consistency gap from expert scores \(e\.g\., Gemini:−0\.962\-0\.962, GPT:−0\.615\-0\.615\)\.

## 5Conclusion

This paper presentedElmes\+, an end\-to\-end framework for scalable, pedagogically grounded evaluation of LLMs in education\. By combining a declarative teacher–student–judge engine with the self\-evolvingSceneGenmodule,Elmes\+automatically constructs and refines fine\-grained, scenario\-specific rubrics while supporting both single\-turn and multi\-turn assessment\.

Experiments on Edu\-330 and four expert\-authored gold\-standard scenarios show that educational capability is a multidimensional diagnostic problem rather than a single ranking task\. Model gaps are most visible in creativity, values integration, and guided tutoring; the education\-specialized InnoSpark achieves the strongest human\-evaluated average, while LLM judges provide stable relative rankings but still exhibit biases such as self\-preference\.Ablations further show that expert\-scored few\-shot anchoring improves human–LLM alignment, whereas reasoning enforcement and greedy decoding remain model\-dependent\. Future work will extendElmes\+beyond text\-only settings by incorporating multimodal interactions, real learner outcome data, broader expert review, and curriculum\-adaptive rubric generation\.

## Limitations

WhileElmes\+provides a scalable approach for evaluating educational LLMs, we acknowledge a few boundaries in its current scope:

- •Simulated Environments:Elmes\+evaluates pedagogical capabilities within an offline, simulated setting\. Although our synthetic student profiles are realistic, they cannot fully capture the unpredictable cognitive dynamics of actual learners\. Future work will need to align these synthetic metrics with empirical learning outcomes in real\-world deployments\.
- •Modality Constraints:The current framework focuses exclusively on text\-based interactions\. It does not yet evaluate multimodal educational scenarios, such as voice\-based conversational tutoring or handwritten homework analysis\.

## References

- W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, H\. Zhang, B\. Zhu, M\. Jordan, J\. E\. Gonzalez,et al\.\(2024\)Chatbot arena: an open platform for evaluating llms by human preference\.arXiv preprint arXiv:2403\.04132\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.06546#S2.SS2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06546#S2.SS1.p1.1)\.
- Educhat: a large\-scale language model\-based chatbot system for intelligent education\.arXiv preprint arXiv:2308\.02773\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.3](https://arxiv.org/html/2606.06546#S2.SS3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§2\.1](https://arxiv.org/html/2606.06546#S2.SS1.p1.1)\.
- Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, Y\. Fu,et al\.\(2023\)C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.Advances in neural information processing systems36,pp\. 62991–63010\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06546#S2.SS1.p1.1)\.
- E\. Kasneci, K\. Seßler, S\. Küchemann, M\. Bannert, D\. Dementieva, F\. Fischer, U\. Gasser, G\. Groh, S\. Günnemann, E\. Hüllermeier,et al\.\(2023\)ChatGPT for good? on opportunities and challenges of large language models for education\.Learning and individual differences103,pp\. 102274\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p1.1)\.
- S\. Kim, J\. Shin, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, R\. Shin, S\. Kim, J\. Thorne, M\. Seo,et al\.\(2024\)Prometheus: inducing fine\-grained evaluation capability in language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 29927–29962\.Cited by:[§2\.2](https://arxiv.org/html/2606.06546#S2.SS2.p2.1)\.
- J\. Li, S\. Sun, W\. Yuan, R\. Fan, P\. Liu,et al\.\(2024\)Generative judge for evaluating alignment\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 27547–27574\.Cited by:[§2\.2](https://arxiv.org/html/2606.06546#S2.SS2.p2.1)\.
- J\. Macina, N\. Daheim, S\. Chowdhury, T\. Sinha, M\. Kapur, I\. Gurevych, and M\. Sachan \(2023\)Mathdial: a dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5602–5621\.Cited by:[§2\.3](https://arxiv.org/html/2606.06546#S2.SS3.p1.1)\.
- P\. Mishra and M\. J\. Koehler \(2006\)Technological pedagogical content knowledge: a framework for teacher knowledge\.Teachers college record108\(6\),pp\. 1017–1054\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.06546#S3.SS2.SSS1.p1.1)\.
- R\. S\. Srinivasa, Z\. Che, C\. B\. C\. Zhang, D\. Mares, E\. Hernandez, J\. Park, D\. Lee, G\. Mangialardi, C\. Ng, E\. H\. Cardona,et al\.\(2025\)TutorBench: a benchmark to assess tutoring capabilities of large language models\.arXiv preprint arXiv:2510\.02663\.Cited by:[§2\.3](https://arxiv.org/html/2606.06546#S2.SS3.p1.1)\.
- L\. S\. Vygotsky \(1978\)Mind in society: the development of higher psychological processes\.Vol\.86,Harvard university press\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.06546#S3.SS2.SSS1.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu,et al\.\(2024a\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9440–9450\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p1.1)\.
- S\. Wang, T\. Xu, H\. Li, C\. Zhang, J\. Liang, J\. Tang, P\. S\. Yu, and Q\. Wen \(2026\)Large language models for education: a survey and outlook\.IEEE Signal Processing Magazine42\(6\),pp\. 51–63\.Cited by:[§2\.2](https://arxiv.org/html/2606.06546#S2.SS2.p2.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024b\)Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06546#S2.SS1.p1.1)\.
- B\. Xu, Y\. Bai, H\. Sun, Y\. Lin, S\. Liu, X\. Liang, Y\. Li, Z\. Dong, J\. Zhang, Y\. Deng,et al\.\(2025\)Edubench: a comprehensive benchmarking dataset for evaluating large language models in diverse educational scenarios\.arXiv preprint arXiv:2505\.16160\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.3](https://arxiv.org/html/2606.06546#S2.SS3.p1.1)\.
- L\. Yan, L\. Sha, L\. Zhao, Y\. Li, R\. Martinez\-Maldonado, G\. Chen, X\. Li, Y\. Jin, and D\. Gašević \(2024\)Practical and ethical challenges of large language models in education: a systematic scoping review\.British Journal of Educational Technology55\(1\),pp\. 90–112\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2606.06546#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.06546#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2606.06546#S2.SS2.p2.1)\.
- C\. Zhou, M\. Wang, T\. Zhang, Q\. Zhu, J\. Li, and H\. Huang \(2025\)From answers to questions: eqgbench for evaluating llms’ educational question generation\.arXiv preprint arXiv:2508\.10005\.Cited by:[§2\.3](https://arxiv.org/html/2606.06546#S2.SS3.p1.1)\.

## Appendix

## Appendix AInteractive UI

1. 1\.Visual Configuration Generator\.Experts construct evaluation scenarios through a structured form\-based interface rather than editing raw YAML files\. The panel provides accordion sections for global settings \(concurrency, retry policies\), model configuration \(teacher, student, and judge models with API endpoints\), prompt editing \(system and user prompts for each agent role\), task data entry \(via an interactive spreadsheet supporting student personas and test questions\), and evaluation rubric design \(dimension names, types, and descriptions\)\. A one\-click example loader populates all fields with a complete working configuration, allowing experts to start from a template and iteratively customize\.
2. 2\.Advanced Configuration Editor\.For experts comfortable with YAML, a syntax\-highlighted editor allows direct manipulation of configuration files\. Experts can load existing configurations from a dropdown menu, modify any parameter, and save updated versions\. This panel also serves as the bridge between the visual generator \(which outputs YAML\) and the execution engine \(which consumes YAML\), enabling expert review of the generated configuration before execution\.
3. 3\.One\-Click Execution Console\.Experts can trigger individual pipeline stages \(generate,export,eval\) or the full pipeline with a single button click\. Real\-time streaming output displays the evaluation progress, including per\-scenario scores and any errors encountered\. This eliminates the need for command\-line interaction and makes the evaluation workflow accessible to non\-technical educators\.
4. 4\.Evaluation Results Dashboard\.After evaluation completes, the results panel renders interactive HTML reports with radar charts comparing model performance across evaluation dimensions, per\-task score breakdowns with judge reasoning, and expandable case panels showing complete dialogue trajectories alongside their scores\. Experts can browse evaluation directories, navigate to specific scenarios, and visually identify where models succeed or fail\.

![Refer to caption](https://arxiv.org/html/2606.06546v1/figure/ui-1.jpg)\(a\)Code\-free configuration interface
![Refer to caption](https://arxiv.org/html/2606.06546v1/figure/ui-2.jpg)\(b\)Automatically generated visualization

Figure 6:The proposed interface for code\-free scenario configuration and automated visualization\.\(a\)Experts configure educational evaluation scenarios through an intuitive visual dashboard, defining content and parameters without programming\.\(b\)The framework automatically generates a structured schematic diagram visualizing the configured scenario\.
## Appendix BEvaluation Scenarios

We instantiateElmes\+across four representative pedagogical task types that span the spectrum from single\-turn generation to multi\-turn interactive teaching\.

### B\.1Knowledge Point Explanation

The teacher agent provides a systematic, personalized explanation of a specific knowledge point to a learner with a defined cognitive profile\. This is asingle\-turntask evaluated on 6 dimensions: role adherence, emotional support, knowledge mastery, teaching method appropriateness, content design quality, and responsiveness to learner characteristics\.

### B\.2Guided Problem\-Solving Teaching

The teacher agent engages in amulti\-turn Socratic dialoguewith a simulated student agent, guiding the student to solve a problem through scaffolded questioning rather than direct answer provision\. This scenario is evaluated on 14 dimensions organized into four categories: content fidelity \(accuracy, role consistency, topic relevance\), guided instruction \(guidance quality, directionality, personalization, metacognition cultivation\), values \(cultural integration, value orientation\), and affective support \(emotional awareness, appropriateness, inclusivity\)\.

### B\.3Interdisciplinary Lesson Plan Generation

The teacher agent produces a comprehensive lesson plan integrating concepts from multiple academic disciplines\. Thissingle\-turntask is evaluated on 15 dimensions covering thematic coherence, activity design, assessment system quality, implementation feasibility, and logical structure\.

### B\.4Contextualized Question Generation

The teacher agent generates an examination question embedded in an authentic real\-world context, along with a complete solution\. Thissingle\-turntask is evaluated on 5 dimensions: problem content quality, solution quality, context quality, pedagogical utility, and value alignment\.

### B\.5Human Expert Scoring Guidelines

To ensure the empirical rigor of our baseline, the human expert panel was provided with the exact scenario definitions and fine\-grained rubrics detailed above\. Evaluators conducted a strict double\-blind review, where the identity of the source model for each pedagogical response was completely concealed and randomized\. Experts graded each generated output on a 5\-point Likert scale \(1: Completely Fails to Meet Criteria, 5: Perfectly Meets Criteria\) across every individual sub\-dimension\. These fine\-grained annotations were subsequently aggregated and averaged to derive the four consolidated core dimensions presented in our main text evaluation: Professionalism, Personalization, Creativity, and Values\.

## Appendix CSelf\-Evolving Algorithm

Algorithm 1SceneGen: Self\-Evolving Rubric Synthesis1:Scene

s∈𝒮s\\in\\mathcal\{S\}, dimensions

\{\(di,wi\)\}\\\{\(d\_\{i\},w\_\{i\}\)\\\}, max iterations

KK, QA count

NN, patience

pp, threshold

ϵ\\epsilon
2:

Psyn\(0\)←InitialSynthesisPromptP\_\{\\text\{syn\}\}^\{\(0\)\}\\leftarrow\\texttt\{InitialSynthesisPrompt\}
3:

ℳ\(0\)←MetricGen​\(s,\{\(di,wi\)\}\)\\mathcal\{M\}^\{\(0\)\}\\leftarrow\\texttt\{MetricGen\}\(s,\\\{\(d\_\{i\},w\_\{i\}\)\\\}\)⊳\\triangleright6–10 metrics with dimension labels and weights

4:

Psyn∗,ℳ∗,s¯w∗←Psyn\(0\),ℳ\(0\),−∞P^\{\*\}\_\{\\text\{syn\}\},\\mathcal\{M\}^\{\*\},\\bar\{s\}^\{\*\}\_\{w\}\\leftarrow P\_\{\\text\{syn\}\}^\{\(0\)\},\\mathcal\{M\}^\{\(0\)\},\-\\infty⊳\\trianglerightBest\-version tracking

5:for

k=1k=1to

KKdo

6:

\{\(qjint,qjusr,aj\)\}j=1N←GenerateQA​\(s,Psyn\(k−1\),personas, lengths\)\\\{\(q\_\{j\}^\{\\text\{int\}\},q\_\{j\}^\{\\text\{usr\}\},a\_\{j\}\)\\\}\_\{j=1\}^\{N\}\\leftarrow\\texttt\{GenerateQA\}\(s,P\_\{\\text\{syn\}\}^\{\(k\-1\)\},\\text\{personas, lengths\}\)⊳\\trianglerightDual\-Q \+ length control

7:

𝐒\(k\)←∅\\mathbf\{S\}^\{\(k\)\}\\leftarrow\\emptyset
8:forparallel

j=1j=1to

NN,

m∈ℳ\(k−1\)m\\in\\mathcal\{M\}^\{\(k\-1\)\}do⊳\\trianglerightModel\-as\-Judge

9:

sj,m,rj,m←JudgeScore​\(qjusr,aj,m\)s\_\{j,m\},r\_\{j,m\}\\leftarrow\\texttt\{JudgeScore\}\(q\_\{j\}^\{\\text\{usr\}\},a\_\{j\},m\)⊳\\trianglerightScore∈\[1,5\]\\in\[1,5\]\+ reasoning

10:

𝐒\(k\)←𝐒\(k\)∪\{\(j,m,sj,m,rj,m\)\}\\mathbf\{S\}^\{\(k\)\}\\leftarrow\\mathbf\{S\}^\{\(k\)\}\\cup\\\{\(j,m,s\_\{j,m\},r\_\{j,m\}\)\\\}
11:endfor

12:

s¯m←1N​∑jsj,m\\bar\{s\}\_\{m\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{j\}s\_\{j,m\}⊳\\trianglerightPer\-metric averages

13:

s¯w\(k\)←∑mwm​s¯m∑mwm\\bar\{s\}\_\{w\}^\{\(k\)\}\\leftarrow\\frac\{\\sum\_\{m\}w\_\{m\}\\bar\{s\}\_\{m\}\}\{\\sum\_\{m\}w\_\{m\}\}⊳\\trianglerightWeighted average score

14:if

k=1k=1then

15:

𝒜←SelectAnchors​\(\{\(qj,aj\)\}\)\\mathcal\{A\}\\leftarrow\\texttt\{SelectAnchors\}\(\\\{\(q\_\{j\},a\_\{j\}\)\\\}\)⊳\\trianglerightFreeze anchor QA set

16:else

17:

𝐒𝒜\(k\)←ScoreAnchors​\(𝒜,ℳ\(k−1\)\)\\mathbf\{S\}^\{\(k\)\}\_\{\\mathcal\{A\}\}\\leftarrow\\texttt\{ScoreAnchors\}\(\\mathcal\{A\},\\mathcal\{M\}^\{\(k\-1\)\}\)⊳\\trianglerightRe\-score frozen anchors

18:if

\|s¯𝒜\(k\)−s¯𝒜\(k−1\)\|\>τ\|\\bar\{s\}^\{\(k\)\}\_\{\\mathcal\{A\}\}\-\\bar\{s\}^\{\(k\-1\)\}\_\{\\mathcal\{A\}\}\|\>\\tauthen

19:warn“metric semantic drift detected”⊳\\trianglerightAnchor calibration

20:endif

21:endif

22:if

s¯w\(k\)\>s¯w∗\\bar\{s\}\_\{w\}^\{\(k\)\}\>\\bar\{s\}^\{\*\}\_\{w\}then⊳\\trianglerightBest\-version tracking

23:

Psyn∗,ℳ∗,s¯w∗←Psyn\(k−1\),ℳ\(k−1\),s¯w\(k\)P^\{\*\}\_\{\\text\{syn\}\},\\mathcal\{M\}^\{\*\},\\bar\{s\}^\{\*\}\_\{w\}\\leftarrow P\_\{\\text\{syn\}\}^\{\(k\-1\)\},\\mathcal\{M\}^\{\(k\-1\)\},\\bar\{s\}\_\{w\}^\{\(k\)\}
24:else

25:

Psyn\(k−1\),ℳ\(k−1\)←Psyn∗,ℳ∗P\_\{\\text\{syn\}\}^\{\(k\-1\)\},\\mathcal\{M\}^\{\(k\-1\)\}\\leftarrow P^\{\*\}\_\{\\text\{syn\}\},\\mathcal\{M\}^\{\*\}⊳\\trianglerightRollback to best

26:endif

27:if

k\>pk\>pand

maxi∈\[k−p,k\)⁡\(s¯w\(i\+1\)−s¯w\(i\)\)<ϵ\\max\_\{i\\in\[k\-p,k\)\}\(\\bar\{s\}\_\{w\}^\{\(i\+1\)\}\-\\bar\{s\}\_\{w\}^\{\(i\)\}\)<\\epsilonthen

28:break⊳\\trianglerightEarly stopping

29:endif

30:

obs←AnalyzePathologies​\(s¯m,𝐒\(k\)\)\\text\{obs\}\\leftarrow\\texttt\{AnalyzePathologies\}\(\\bar\{s\}\_\{m\},\\mathbf\{S\}^\{\(k\)\}\)
31:

Psyn\(k\)←EvolveSynthesis​\(Psyn\(k−1\),𝐒\(k\),s¯w\(k\),all QAs ranked\)P\_\{\\text\{syn\}\}^\{\(k\)\}\\leftarrow\\texttt\{EvolveSynthesis\}\(P\_\{\\text\{syn\}\}^\{\(k\-1\)\},\\mathbf\{S\}^\{\(k\)\},\\bar\{s\}\_\{w\}^\{\(k\)\},\\text\{all QAs ranked\}\)⊳\\trianglerightEnhanced signal

32:

ℳ\(k\)←AlignMetrics​\(EvolveMetrics​\(ℳ\(k−1\),𝐒\(k\),obs\)\)\\mathcal\{M\}^\{\(k\)\}\\leftarrow\\texttt\{AlignMetrics\}\(\\texttt\{EvolveMetrics\}\(\\mathcal\{M\}^\{\(k\-1\)\},\\mathbf\{S\}^\{\(k\)\},\\text\{obs\}\)\)⊳\\trianglerightConstrained evolution

33:endforreturn

Psyn∗P^\{\*\}\_\{\\text\{syn\}\},

ℳ∗\\mathcal\{M\}^\{\*\},

\{\(qj,aj\)\}\\\{\(q\_\{j\},a\_\{j\}\)\\\}⊳\\trianglerightReturn best version

## Appendix DCase Study: Scenario Configuration Details

To provide a concrete illustration of our evaluation framework, Figure[7](https://arxiv.org/html/2606.06546#A4.F7)presents the YAML configuration for a scenario:Chinese \- Elementary \- Ancient Poetry Enlightenment\.

models: test\_model: judge\_model: agents: test\_model: model: test\_model prompt: \- role: system content: You are a helpful assistant\. \- role: user content: "Your task is scenario: Chinese \- Elementary \- Ancient Poetry Enlightenment and Imagination Teaching\. \{question\}" tasks: mode: iter content: \- question: "I am a teacher named Pan Tie, 33 years old, male\. My hobbies are stargazing, astrophotography, and DIY computer assembly\. Professionally, my logic is average, and I tend to rely on authority but am open to feedback\. I am currently responsible for teaching ancient poetry to third graders\. I found that students cannot understand the exaggeration technique in ’pluck the stars with my hand’ from the poem ’Lodging at a Mountain Temple’\. Their specific confusion is that they insist hands can truly touch the stars\. As an ancient poetry enlightenment expert, please first analyze the specific cognitive barriers of these students in poetry learning, and then provide a targeted solution\. Can you explain this comprehensively and deeply? It would be best to design some cross\-disciplinary teaching activities combining my astronomy hobbies, including four complete instructional stages: situational introduction, confusion deconstruction, interactive exploration, and consolidation/transfer\." \- question: "I am a parent of a second\-grade student, female, 35 years old, with a busy career, having only 20 minutes of companion time each evening\. My child has a sensitive personality, is responsive to visual learning but resists mechanical memorization\. The specific confusion is interpreting ’yu qiong qian li mu’ \(desiring to exhaust the thousand\-mile vision\) as ’wanting a thousand miles of poor eyes’, failing to understand the ancient\-modern semantic shift of the character ’qiong’ \(exhaust/poor\)\. As an ancient poetry enlightenment expert, please first analyze the specific cognitive barriers of this student, and then give a targeted solution\. I want to learn in depth how to design short, highly effective home tutoring games, including thorough discussion, examples, and analysis\." \- question: "I am MillerJose, 14 years old, an eighth\-grade student, male\. My hobbies are Chess and Robot Making\. I have low confidence, focus on academics but am hesitant in social interactions, and prefer solitary activities\. I want to help my younger brother, who is in the fourth grade, learn ’Looking at the Lushan Waterfall’\. My brother’s specific confusion is the inability to imagine the sense of spatial height in ’flying down three thousand feet’\. As an ancient poetry enlightenment expert, please first analyze his specific cognitive barriers in poetry learning, and then provide a targeted solution\. Tell me how to use my robot\-making hobby to design auxiliary teaching aids, including necessary details and examples\." \(Remaining 7 multi\-profile evaluation queries synthesized bySceneGen\) directions: \- START \-\> test\_model \- test\_model \-\> END evaluation: name: judge model: judge\_model prompt: \- role: user content: \| You are an expert in education, specializing in evaluating scenarios for Chinese\-Elementary\-Ancient Poetry Enlightenment\. Please rate the following response using a 1\-5 scale\.\.\. format: \- field: Accuracy of Poetry Knowledge \- field: Heuristic Teaching Guidance \- field: Grade\-level Language Adaptability \- field: Completeness of Instructional Stages \- field: Student Profile Alignment \- field: Format Instruction Compliance \- field: Personalized Problem Diagnosis \- field: Stimulation and Guidance of Imagination \- field: Integration of Traditional Cultural Values \(Comprehensive fine\-grained rubrics generated bySceneGen\)

Figure 7:YAML configuration samples for the elementary school ancient poetry enlightenment scenario\.