Knowledge Distillation for Automated AI Tutor Evaluation

# Knowledge Distillation for Automated AI Tutor Evaluation
Source: [https://arxiv.org/html/2607.10647](https://arxiv.org/html/2607.10647)
Tahmid Al Hannan, Diego Garcia, Alex Njoroge, Suha Al Juboori, Tarek Sakakini Folsom Lake College Folsom, CA, USA \{w2124644, w2079995, w2140050\}@apps\.losrios\.edu, \{aljubos, sakakit\}@flc\.losrios\.edu

###### Abstract

The rapid integration of Large Language Models \(LLMs\) into K\-12 and higher education has outpaced the development of reliable methods for evaluating their pedagogical quality\. As the research community starts to explore the space of automating evaluation of AI tutors, we introduce FATE \(FLC AI Tutor Evaluator\), a specialized 8B\-parameter language model designed to evaluate AI tutors\. Aligned with the four core evaluation tracks from the BEA 2025 Shared Task, our model assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability\. Because pedagogical evaluation is a specialized task with limited labeled data, we leverage knowledge distillation from a frontier LLM to generate additional supervision, yielding absolute performance gains up to 22\.63 percentage points\. Finally, we demonstrate FATE’s utility as an automated evaluator by benchmarking instructional responses generated by popular commercial models, including ChatGPT, Claude, Gemini, and DeepSeek\. On average, we have found that Gemini 2\.5 Flash perfomed best \(82\.88%\), then ChatGPT 5\.5 Instant \(80\.75%\), followed by DeepSeek V4 Flash \(80\.13%\) and Claude Sonnet 4\.6 \(74\.00%\)\.

Knowledge Distillation for Automated AI Tutor Evaluation

Tahmid Al Hannan, Diego Garcia, Alex Njoroge, Suha Al Juboori, Tarek SakakiniFolsom Lake CollegeFolsom, CA, USA\{w2124644, w2079995, w2140050\}@apps\.losrios\.edu, \{aljubos, sakakit\}@flc\.losrios\.edu

## 1Introduction

Large language models \(LLMs\) are becoming increasingly prevalent in education, with AI usage in observed K–12 classrooms increasing by 84% from 2024 to 2025McGehee \([Technical report](https://arxiv.org/html/2607.10647#bib.bib6)\)\. As AI tutors become more widely adopted, reliable methods for evaluating their pedagogical quality are increasingly important\. The BEA 2025 Shared Task introduced a benchmark for evaluating AI tutors across multiple pedagogical dimensions \(Figure[1](https://arxiv.org/html/2607.10647#S1.F1)\)Kochmaret al\.\([2025a](https://arxiv.org/html/2607.10647#bib.bib5)\)\. However, human evaluation is costly and difficult to scale, while automatic metrics often fail to capture qualities such as guidance, actionability, and instructional effectivenessMarquez\-Carpinteroet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib18)\)\. This motivates automated evaluators capable of assessing tutoring interactions at scaleTacket al\.\([2023](https://arxiv.org/html/2607.10647#bib.bib7)\)\.

![Refer to caption](https://arxiv.org/html/2607.10647v1/x1.png)Figure 1:Example dataset point from BEA 2025\. Below each AIT response is the human evaluation using four metrics: Mistake Identification \(MI\), Mistake Location \(ML\), Providing Guidance \(PG\), and Actionability\.Recent evaluation frameworks extend beyond traditional benchmarks\. Multimodal environments such as TutorBench provide comprehensive evaluations but require substantial computational resources and are often mismatched with the text\-based chatbot interfaces most widely used todaySrinivasaet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib12)\); Kajanet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib8)\)\. Text\-based tutoring has also been shown to outperform multimodal approaches in some educational settingsSyalet al\.\([2026](https://arxiv.org/html/2607.10647#bib.bib9)\), motivating lightweight, specialized evaluators\.

We present FATE \(FLC AI Tutor Evaluator\), a lightweight evaluator based on Llama 3\.1 8B that assesses AI tutors across the four pedagogical dimensions of the BEA 2025 Shared Task\. Trained on 1,450 student\-tutor conversations using knowledge distillation from Claude Opus 4\.7, FATE also enables benchmarking of commercial AI tutors including ChatGPT, Claude, Gemini, and DeepSeek\.

Our contributions are:

- •Training FATE, a lightweight specialized model for AIT evaluation\.
- •Demonstrating the effectiveness of knowledge distillation for pedagogical evaluation given limited labeled data\.
- •Deploying FATE to benchmark popular commercial AI tutors\.

## 2Prior Work

Intelligent tutoring systems \(ITSs\) have long provided adaptive, step\-level support to students, with prior work showing that ITSs can approach the effectiveness of human tutoring in some settingsVanLehn \([2011](https://arxiv.org/html/2607.10647#bib.bib19)\)\. However, LLM\-based tutors differ from classical ITSs because they generate open\-ended natural\-language responses rather than operating over fixed solution graphs or hand\-authored cognitive models\. This shift makes evaluation more difficult: a response may be fluent and correct while still failing to diagnose the student’s misconception, locate the error, provide useful guidance, or suggest actionable next steps\.

Recent work has begun to formalize this evaluation problem\. The AI Teacher Test proposed measuring whether generative models can behave pedagogically in educational dialogueTack and Piech \([2022](https://arxiv.org/html/2607.10647#bib.bib20)\)\. The BEA 2023 Shared Task benchmarked teacher\-response generation, while the BEA 2025 Shared Task reframed the problem around evaluating AI tutor responses across Mistake Identification, Mistake Location, Providing Guidance, and ActionabilityTacket al\.\([2023](https://arxiv.org/html/2607.10647#bib.bib7)\); Kochmaret al\.\([2025b](https://arxiv.org/html/2607.10647#bib.bib25)\)\. Other recent benchmarks, including MRBench, TutorGym, and TutorBench, further show that tutoring quality is multi\-dimensional and cannot be reduced to final\-answer correctness aloneMauryaet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib14)\); Weitekampet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib26)\); Srinivasaet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib12)\)\.

FATE is also related to broader work on LLM\-based evaluation\. LLM\-as\-judge methods have shown promise for scalable assessment of open\-ended model outputs, but also exhibit biases related to position, verbosity, and model preferenceZhenget al\.\([2023](https://arxiv.org/html/2607.10647#bib.bib21)\); Liuet al\.\([2023](https://arxiv.org/html/2607.10647#bib.bib22)\)\. Learned evaluators such as Prometheus and Prometheus 2 demonstrate that open models can be trained to provide rubric\-conditioned judgmentsKimet al\.\([2024a](https://arxiv.org/html/2607.10647#bib.bib23),[b](https://arxiv.org/html/2607.10647#bib.bib24)\)\. FATE follows this direction but specializes it for AI\-tutor evaluation: rather than building a general\-purpose judge, we distill supervision from a frontier model into a lightweight 8B evaluator aligned with the four BEA pedagogical dimensions\.

## 3Method

At the center of our method is fine\-tuning a pretrained lightweight LLM on the BEA 2025 Shared Task dataset incorporating heuristics from previous workFanet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib10)\)\. We then leverage knowledge distillation from larger models, especially given the limited size of the original human\-annotated dataset\.

### 3\.1Data

Our primary dataset is the annotated development set from the BEA 2025 Shared Task on evaluating AI tutors\. It consists of 300 mathematics tutoring dialogues designed to assess the pedagogical quality of AI tutor responses rather than response generation itselfKochmaret al\.\([2025a](https://arxiv.org/html/2607.10647#bib.bib5)\)\. Each dialogue contains a multi\-turn conversation ending with a student’s final utterance, typically expressing a misconception, incorrect answer, or confusion\. Multiple candidate tutor responses are provided for the same dialogue, generated by frontier LLM tutors \(e\.g\., Claude, ChatGPT, Gemini, and Llama\)\.

Each tutor response is annotated by expert evaluators across four pedagogical dimensions:Mistake Identification,Mistake Location,Providing Guidance, andActionability\. Each dimension receives one of three labels \(Yes,To Some Extent, orNo\) allowing direct comparison of different tutoring systems under identical student interactions while grounding evaluation in expert pedagogical judgments\.

Table 1:Performance metrics before and after knowledge distillation
### 3\.2Training

We evaluated Llama 3\.2 3B, Llama 3\.1 8B, and Qwen 2\.5 7B, selecting Llama 3\.1 8B based on overall performance\. Models trained on the original BEA dataset used 3 epochs, a learning rate of2×10−42\\times 10^\{\-4\}, and 4 gradient accumulation steps\. After incorporating the knowledge distillation dataset, training used 1 epoch with the same learning rate and 16 accumulation steps\. LoRA fine\-tuning with 4\-bit quantization enabled efficient training on A100 and T4 GPUs\.

Rather than training a single evaluator, we fine\-tuned four task\-specific models corresponding to Mistake Identification, Mistake Location, Providing Guidance, and Actionability\. Each model used prompts describing the dialogue history and evaluation rubric, following the prompting strategy ofFanet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib10)\)\.

To improve generalization, we adopted the dialogue\-shuffling augmentation strategy proposed byFanet al\.\([2025](https://arxiv.org/html/2607.10647#bib.bib10)\), which reorders historical dialogue turns during training\. Despite preserving conversation content, this augmentation consistently improved macro F1 and accuracy\.

### 3\.3Knowledge Distillation

FATE was initially trained using only the 300 student–tutor conversations from the BEA 2025 Shared TaskKochmaret al\.\([2025a](https://arxiv.org/html/2607.10647#bib.bib5)\)\. Although these conversations contained 2,480 AI tutor \(AIT\) and human tutor responses, the resulting model plateaued at an average macro F1 score of 74\.84%, largely due to the dataset’s limited size and diversity\. To improve generalization, we expanded the training set through knowledge distillation, generating synthetic student–tutor conversations with a frontier LLM\. Following the findings ofStantonet al\.\([2021](https://arxiv.org/html/2607.10647#bib.bib13)\), which show that excessively large distillation datasets can hinder optimization, we deliberately limited the amount of synthetic data while targeting our four pedagogical evaluation dimensions\.

We first analyzed the label distribution of the BEA dataset and observed substantial class imbalance across all evaluation categories\. For example, Mistake Identification contained 1069Yes, 592No, and 137To Some Extentlabels\. To improve balance, Claude Opus 4\.7 was prompted to generate nine tutor responses for each synthetic conversation: three high\-quality, three average, and three poor responses\. This process produced 1,350 conversations and 12,150 labeled tutor responses\. Taking Mistake Identification as an example, the distribution became substantially more balanced, increasing to 4,387Yes, 6,630No, and 1,133To Some Extentexamples\.

Table 2:Performance of Llama 3\.1 8B under different training strategies for the Mistake Identification task\.

## 4Experimental Results

We first compared training strategies\. Table[2](https://arxiv.org/html/2607.10647#S3.T2)shows that dialogue shuffling consistently improved performance, whereas downsampling reduced accuracy\. We report both*lenient*metrics, which merge the*Yes*and*To Some Extent*labels, and*strict*metrics, which evaluate all three labels separately\.

### 4\.1Impact of Knowledge Distillation

Using the selected Llama 3\.1 8B configuration, we incorporated the synthetic knowledge distillation dataset described in Section[3\.3](https://arxiv.org/html/2607.10647#S3.SS3)\. As shown in Table[1](https://arxiv.org/html/2607.10647#S3.T1), knowledge distillation substantially improved performance across all four pedagogical dimensions\. For Mistake Identification, lenient F1 increased from 84\.04% to 96\.08%, while lenient accuracy improved from 84\.98% to 96\.13%\.

To better understand the effect of knowledge distillation, we analyzed the confusion matrices for Mistake Identification before \(Figure[3](https://arxiv.org/html/2607.10647#S4.F3)\) and after \(Figure[3](https://arxiv.org/html/2607.10647#S4.F3)\) knowledge distillation\. In both settings, the model accurately classified clear*Yes*and*No*responses but struggled with the more ambiguous*To Some Extent*category\. In the baseline model, 50\.0% of*To Some Extent*examples were predicted as*Yes*\. Although knowledge distillation further improved accuracy on clear positive \(97\.7%\) and negative \(94\.5%\) examples, it also increased this bias, with 65\.1% of ambiguous examples classified as*Yes*\. This behavior explains the larger gains under lenient scoring than strict scoring and suggests that compact evaluators remain challenged by nuanced, partial\-credit pedagogical judgments\.

![Refer to caption](https://arxiv.org/html/2607.10647v1/matrix_original.png)

Figure 2:Llama 3\.1 8B without knowledge distillation confusion matrix for Mistake Identification\.![Refer to caption](https://arxiv.org/html/2607.10647v1/matrix_distill.png)

Figure 3:Llama 3\.1 8B with knowledge distillation confusion matrix for Mistake Identification\.
### 4\.2External Tutors

To benchmark commercial AI tutors, we extracted conversation histories from the BEA 2025 dataset and generated tutor responses using Claude Sonnet 4\.6, ChatGPT 5\.5 Instant, Gemini 2\.5 Flash, and DeepSeek V4 Flash\. FATE then evaluated these responses \(Table[3](https://arxiv.org/html/2607.10647#S4.T3)\)\.

Gemini achieved the highest overall average accuracy, leading in Mistake Identification, Mistake Location, and Actionability, while DeepSeek performed best on Providing Guidance\. Across all models, Actionability remained the weakest category, indicating that providing concrete next\-step guidance continues to be a challenge for current AI tutors\.

Table 3:FATE’s evaluation of commercially available AI Tutors\.

## 5Conclusion

We presented FATE, a lightweight LLM for automated AI tutor evaluation\. Knowledge distillation substantially improved performance on this specialized task with limited labeled data, increasing lenient F1 by up to 22\.63 %\. Applying FATE to commercial AI tutors further revealed a consistent weakness in actionability\.

Future work will package FATE as a practical evaluation tool for educational institutions to support AI tutor adoption and assessment\.

## LLM Usage Acknowledgement

We acknowledge that large language models were used to assist in the preparation of this paper, specifically for improving or polishing the language of certain sentences\. All substantive ideas, analyses, and conclusions remain the responsibility of the authors\.

## References

- BJTU at bea 2025 shared task: task\-aware prompt tuning and data augmentation for evaluating ai math tutors\.InProceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications,Cited by:[§3\.2](https://arxiv.org/html/2607.10647#S3.SS2.p2.1),[§3\.2](https://arxiv.org/html/2607.10647#S3.SS2.p3.1),[§3](https://arxiv.org/html/2607.10647#S3.p1.1)\.
- K\. Kajan, W\. Shi, and D\. Wanatowski \(2025\)STEM undergraduates’ perceptions of ai chatbots: a cross\-sectional descriptive survey\.InAI in Education,Vol\.1,pp\. 4\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p2.1)\.
- S\. Kim, J\. Shin, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, R\. Shin, S\. Kim, J\. Thorne, M\. Seo,et al\.\(2024a\)Prometheus: inducing fine\-grained evaluation capability in language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 29927–29962\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p3.1)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024b\)Prometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 4334–4353\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p3.1)\.
- E\. Kochmar, K\. K\. Maurya, K\. Petukhova, K\. Srivatsa, A\. Tack, and J\. Vasselli \(2025a\)Findings of the bea 2025 shared task on pedagogical ability assessment of ai\-powered tutors\.arXiv preprint arXiv:2507\.10579\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.10647#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2607.10647#S3.SS3.p1.1)\.
- E\. Kochmar, K\. K\. Maurya, K\. Petukhova, K\. Srivatsa, A\. Tack, and J\. Vasselli \(2025b\)Findings of the bea 2025 shared task on pedagogical ability assessment of ai\-powered tutors\.arXiv preprint arXiv:2507\.10579\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p2.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p3.1)\.
- L\. Marquez\-Carpintero, A\. Lopez\-Sellers, and M\. Cazorla \(2025\)Simulation of teaching behaviours in intelligent tutoring systems: a review using large language models\.Artificial Intelligence Review\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p1.1)\.
- K\. K\. Maurya, K\. A\. Srivatsa, K\. Petukhova, and E\. Kochmar \(2025\)Unifying ai tutor evaluation: an evaluation taxonomy for pedagogical ability assessment of llm\-powered ai tutors\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 1234–1251\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p2.1)\.
- N\. McGehee \(Technical report\)Artificial intelligence and student usage in online learning: a longitudinal analysis of usage patterns, achievement, and perceptions in k\-12 virtual education\.Technical reportTech\. Rep\.\(Michigan Virtual, 2025\) accessed: 2026\-02\-24\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p1.1)\.
- R\. S\. Srinivasa, Z\. Che, C\. B\. C\. Zhang, D\. Mares, E\. Hernandez, J\. Park, D\. Lee, G\. Mangialardi, C\. Ng, E\. H\. Cardona,et al\.\(2025\)Tutorbench: a benchmark to assess tutoring capabilities of large language models\.arXiv preprint arXiv:2510\.02663\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p2.1),[§2](https://arxiv.org/html/2607.10647#S2.p2.1)\.
- S\. Stanton, P\. Izmailov, P\. Kirichenko, A\. A\. Alemi, and A\. G\. Wilson \(2021\)Does knowledge distillation really work?\.Advances in neural information processing systems34,pp\. 6906–6919\.Cited by:[§3\.3](https://arxiv.org/html/2607.10647#S3.SS3.p1.1)\.
- A\. Syal, L\. S\. X\. Prince, E\. Gultepe, N\. B\. Brown, and S\. Sridhar \(2026\)A dialogue\-based framework for correcting multimodal errors in ai\-assisted stem education\.arXiv preprint arXiv:2605\.04131\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p2.1)\.
- A\. Tack, E\. Kochmar, Z\. Yuan, S\. Bibauw, and C\. Piech \(2023\)The bea 2023 shared task on generating ai teacher responses in educational dialogues\.InProceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2023\),pp\. 785–795\.Cited by:[§1](https://arxiv.org/html/2607.10647#S1.p1.1),[§2](https://arxiv.org/html/2607.10647#S2.p2.1)\.
- A\. Tack and C\. Piech \(2022\)The ai teacher test: measuring the pedagogical ability of blender and gpt\-3 in educational dialogues\.arXiv preprint arXiv:2205\.07540\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p2.1)\.
- K\. VanLehn \(2011\)The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems\.Educational psychologist46\(4\),pp\. 197–221\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p1.1)\.
- D\. Weitekamp, M\. N\. Siddiqui, and C\. J\. MacLellan \(2025\)Tutorgym: a testbed for evaluating ai agents as tutors and students\.InInternational Conference on Artificial Intelligence in Education,pp\. 361–376\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2607.10647#S2.p3.1)\.