Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index

# Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index
Source: [https://arxiv.org/html/2608.05411](https://arxiv.org/html/2608.05411)
###### Abstract

Large language models \(LLMs\) are increasingly used as AI tutors, but a correct answer is not always a pedagogically appropriate one\. In classroom learning, effective help depends not only on correctness, but also on whether a response matches the learner’s current foundation, the course sequence, and the timing of concept introduction\. Existing evaluations focus mainly on answer quality, leaving this instructional fit under\-measured\. We present the Pedagogical Suitability Index \(PSI\), a composite metric of six theory\-informed sub\-scores that evaluates how well LLM\-generated tutoring responses align with learner readiness and curricular progression, and we further use PSI as a structured feedback signal for response improvement\. We evaluate four LLM tutors \(ChatGPT, Gemini, Gemma 4, and Qwen 3\) across 240 scenario\-based evaluations using paired standard and defective prompts, then apply a PSI\-guided regeneration protocol to 62 weak\-performing cases\. Baseline differences across the four tested models were modest overall \(PSI range: 0\.557 to 0\.638\), and open\-weight and closed models did not exhibit a clear separation in pedagogical fit\. Under the tested prompt perturbations, overall PSI remained largely stable \(Δ=−0\.002\\Delta=\-0\.002\), though sub\-score trade\-offs emerged\. More importantly, PSI\-guided feedback substantially improved weak\-performing cases: 51 of 62 cases improved \(82\.3%\)\. Focused manual evaluation of the 62 PSI\-selected weak cases provides initial evidence that the identified weaknesses are instructionally meaningful and that many PSI\-guided regenerations correspond to human\-judged improvement\. These results suggest that learner\- and curriculum\-aware alignment may matter more for effective tutoring than model category alone, and that such alignment is both measurable and improvable\.

## IIntroduction

A student asks an AI tutor why a program failed\. The tutor responds with a fluent explanation and perhaps even a working fix\. Yet the response may still be pedagogically mistimed: it may assume concepts the student has not learned, skip prerequisite reasoning that instruction is intended to build, or provide an answer that resolves the immediate problem without strengthening long\-term understanding\. In such cases, correctness alone is an incomplete indicator of tutoring quality\. For educational use, the central question is not only whether an answer is right, but whether the help is appropriate for this learner at this point in the instructional sequence\.

This challenge arises because LLMs typically lack access to crucial instructional context\. In many real settings, a model does not know which concepts the learner has already mastered, where the class currently is in the course progression, or how recently related ideas were introduced\. As a result, an LLM may produce a generally correct response that is poorly matched to the learner’s foundation or to the intended pacing of instruction\. This motivates the need for evaluation methods that go beyond factual accuracy and instead measurepedagogical fit: whether a response respects prerequisite dependencies, provides suitable scaffolding, aligns with the target cognitive level, and accounts for the timing of concept exposure\. Prior work on evaluating AI\-generated educational content has focused primarily on factual accuracy or domain\-specific quality measures\[[28](https://arxiv.org/html/2608.05411#bib.bib3)\], without fully capturing these dimensions of learner\- and curriculum\-aware support\. The Zone of Proximal Development\[[25](https://arxiv.org/html/2608.05411#bib.bib1)\]provides theoretical grounding: effective instruction should target the gap between independent capability and guided achievement\.

To address this gap, we introduce the Pedagogical Suitability Index \(PSI\), a theory\-informed metric for evaluating whether LLM\-generated tutoring responses are appropriate for a learner’s current foundation and instructional context\. We instantiate this metric on a benchmark of 30 tutoring scenarios, validated by student raters for realism and course fit, and use it to compare four commercial and open\-weight LLM tutors under both standard and defective prompting conditions\. We then show that PSI can serve not only as an evaluation metric, but also as a structured feedback signal for targeted regeneration, and we further conduct focused manual evaluation of the PSI\-selected weak cases to examine whether the identified weaknesses and subsequent improvements are instructionally meaningful\.

Our findings suggest that differences in baseline pedagogical fit across current models are present but modest, and do not support a simple open\-versus\-closed distinction\. The more consequential finding is that pedagogical fit can be measured and improved when responses are evaluated against learner readiness, prerequisite structure, and instructional timing\. PSI\-guided feedback improved 82\.3% of weak cases\. All Supplementary materials available at https://doi\.org/10\.5281/zenodo\.19603300

## IIRelated Work

Prior evaluation work on AI\-generated educational responses has focused primarily on answer correctness and text quality\[[28](https://arxiv.org/html/2608.05411#bib.bib3),[27](https://arxiv.org/html/2608.05411#bib.bib12)\]\. AI for personalized learning is well studied\[[5](https://arxiv.org/html/2608.05411#bib.bib2)\], and knowledge tracing models\[[16](https://arxiv.org/html/2608.05411#bib.bib17),[24](https://arxiv.org/html/2608.05411#bib.bib18)\]can track student mastery over time\. In parallel, research on prompt robustness\[[30](https://arxiv.org/html/2608.05411#bib.bib21),[7](https://arxiv.org/html/2608.05411#bib.bib20)\]and self\-refinement\[[17](https://arxiv.org/html/2608.05411#bib.bib23),[6](https://arxiv.org/html/2608.05411#bib.bib24),[4](https://arxiv.org/html/2608.05411#bib.bib25)\]has shown that LLM behavior can shift under noisy inputs and can improve when structured feedback is provided\. Student prompts frequently exhibit deficiencies including missing context and incorrect terminology\[[9](https://arxiv.org/html/2608.05411#bib.bib22)\]\. These lines of work are valuable, but they do not directly address whether a response is pedagogically appropriate for a learner’s current instructional context\.

From a learning perspective, pedagogical appropriateness depends on several dimensions that have been studied separately\. Work on prerequisite relations\[[20](https://arxiv.org/html/2608.05411#bib.bib7),[15](https://arxiv.org/html/2608.05411#bib.bib8),[2](https://arxiv.org/html/2608.05411#bib.bib6)\]and knowledge graphs\[[12](https://arxiv.org/html/2608.05411#bib.bib19),[3](https://arxiv.org/html/2608.05411#bib.bib10)\]highlights the importance of concept ordering\. Research on scaffolding\[[11](https://arxiv.org/html/2608.05411#bib.bib11)\]and text complexity\[[14](https://arxiv.org/html/2608.05411#bib.bib14)\]emphasizes the structure and accessibility of instructional support\. Bloom’s revised taxonomy\[[1](https://arxiv.org/html/2608.05411#bib.bib15)\]focuses on cognitive level, with recent NLP\-based automation\[[26](https://arxiv.org/html/2608.05411#bib.bib16)\]\. The Ebbinghaus forgetting curve\[[18](https://arxiv.org/html/2608.05411#bib.bib13)\]points to the importance of timing and retention\. Together, these literatures suggest that effective tutoring should be evaluated along multiple learner\- and curriculum\-aware dimensions, not correctness alone\. Related work on similarity measurement\[[29](https://arxiv.org/html/2608.05411#bib.bib4)\], activity sequence evaluation\[[10](https://arxiv.org/html/2608.05411#bib.bib5)\], and component interaction analysis\[[21](https://arxiv.org/html/2608.05411#bib.bib30)\]further informs our sub\-score design\.

Our work builds on these strands\[[22](https://arxiv.org/html/2608.05411#bib.bib9)\]but differs in two ways\. First, it operationalizes these dimensions within a single composite metric for pedagogical fit\. Second, it uses that metric not only for diagnosis, but also as a structured signal for targeted response improvement\.

## IIIThe Pedagogical Suitability Index

PSI is a weighted composite of six normalized sub\-scores:

PSI=∑i=16wi​Si,Si∈\[0,1\],∑wi=1\\mathrm\{PSI\}=\\sum\_\{i=1\}^\{6\}w\_\{i\}\\,S\_\{i\},\\quad S\_\{i\}\\in\[0,1\],\\quad\\sum w\_\{i\}=1\(1\)Because existing work does not provide a clear empirical basis for weighting these six pedagogical dimensions relative to one another, we adopt equal weights \(wi=1/6w\_\{i\}=1/6\) in this study\.

Knowledge Distance\(SK​DS\_\{KD\}\) measures how well the response targets the learner’s zone of proximal development:SK​D=1−\|sim​\(R,Kstudent\)−δopt\|S\_\{KD\}=1\-\|\\text\{sim\}\(R,K\_\{\\text\{student\}\}\)\-\\delta\_\{\\text\{opt\}\}\|, wheresim​\(R,Kstudent\)\\text\{sim\}\(R,K\_\{\\text\{student\}\}\)denotes concept overlap between the response and the student’s knowledge state, andδopt=0\.65\\delta\_\{\\text\{opt\}\}=0\.65is the target overlap level, informed by prior work on optimal instructional distance in ZPD\-based frameworks\. Sensitivity of PSI to this parameter has not been tested in the current study\.

Temporal Violation\(ST​VS\_\{TV\}\) penalizes responses that introduce concepts before their prerequisites\[[10](https://arxiv.org/html/2608.05411#bib.bib5)\]:ST​V=1−violations/checksS\_\{TV\}=1\-\\text\{violations\}/\\text\{checks\}, whereviolationscounts prerequisite\-order inversions andchecksdenotes the number of prerequisite\-order comparisons\.

Scaffolding Density\(SS​DS\_\{SD\}\) measures the proportion of instructional support elements\[[11](https://arxiv.org/html/2608.05411#bib.bib11)\]:SS​D=\(Nscaffold/Ntotal\)⋅αadaptS\_\{SD\}=\(N\_\{\\text\{scaffold\}\}/N\_\{\\text\{total\}\}\)\\cdot\\alpha\_\{\\text\{adapt\}\}, whereNscaffoldN\_\{\\text\{scaffold\}\}is the number of scaffold indicators detected,NtotalN\_\{\\text\{total\}\}is the total number of propositions, andαadapt∈\[0,1\]\\alpha\_\{\\text\{adapt\}\}\\in\[0,1\]rewards learner\-specific adaptation\.

Ebbinghaus Forgetting\(SE​FS\_\{EF\}\) incorporates retention timing for referenced concepts\[[18](https://arxiv.org/html/2608.05411#bib.bib13)\]:SE​F=∑iλi​e−ti/SiS\_\{EF\}=\\sum\_\{i\}\\lambda\_\{i\}\\,e^\{\-t\_\{i\}/S\_\{i\}\}, wheretit\_\{i\}is time since last exposure,SiS\_\{i\}is memory strength, andλi\\lambda\_\{i\}is a concept importance weight \(∑λi=1\\sum\\lambda\_\{i\}=1\)\. In this implementation,SE​FS\_\{EF\}depends on scenario metadata rather than response content, yielding identical values across models\. We discuss this limitation in Section VI\.

Cognitive Load\(SC​LS\_\{CL\}\) penalizes avoidable processing demands\[[14](https://arxiv.org/html/2608.05411#bib.bib14)\]:SC​L=1−Lextraneous/LtotalS\_\{CL\}=1\-L\_\{\\text\{extraneous\}\}/L\_\{\\text\{total\}\}, whereLextraneousL\_\{\\text\{extraneous\}\}denotes extraneous content \(filler, hedging, tangential material\) andLtotalL\_\{\\text\{total\}\}denotes total response length\.

Bloom’s Alignment\(SR​AS\_\{RA\}\) measures cognitive\-level match\[[1](https://arxiv.org/html/2608.05411#bib.bib15)\]:SR​A=1−\|bR−bT\|/5S\_\{RA\}=1\-\|b\_\{R\}\-b\_\{T\}\|/5, wherebRb\_\{R\}andbTb\_\{T\}are the detected and target Bloom levels, respectively\.

The current implementation uses proxy measures \(regex patterns, keyword matching\) rather than deep semantic analysis\. We characterize PSI as a theory\-informed operationalized benchmark rather than a validated psychometric instrument \(see Section VI\)\.

## IVExperimental Setup

### IV\-ACourse Context and Scenario Design

We designed our evaluation around CSCI 150 \(Introduction to Computer Science\) at the University of Montana, a 15\-week Python\-based course for students with no prior programming experience\. The curriculum progresses from hardware concepts and variables \(Weeks 1–3\) through operators, conditionals, and loops \(Weeks 4–9\) to functions, data structures, NLP, and image processing \(Weeks 10–15\)\. This progression from concrete, syntax\-focused tasks to abstract, multi\-concept integration provides a natural testbed for evaluating AI tutor adaptation across a full semester\.

We created 30 structured scenarios \(2 per week\) spanning Bloom levels 1–4 \(Remember:nn=1, Understand:nn=7, Apply:nn=16, Analyze:nn=6\), each with explicit learner profiles, prerequisite maps, and a realistic student question reflecting common misconceptions from actual CSCI 150 tutoring interactions\. To validate the scenarios, two undergraduate students who had completed CSCI 150 independently rated each scenario on two 5\-point scales:Realism\(whether the question reflects something a real student would ask\) andCourse Fit\(whether the topic and difficulty match the corresponding week\)\. The mean ratings were 3\.67 \(SD = 0\.69\) for Realism and 4\.10 \(SD = 0\.60\) for Course Fit, with inter\-rater agreement within±\\pm1 point at 83\.3% for both criteria\. Only one scenario \(S14, Mid\-term Review – Debugging\) received a mean Realism score below 2\.5, suggesting that the scenario set broadly reflects realistic tutoring situations as perceived by students who took the course\.

To evaluate prompt robustness, each scenario includes a paireddefectiveprompt simulating common student communication patterns\. We designed eight defect categories based on observed student behavior: Too Short \(nn=7\), Keyword Only \(nn=7\), Missing Context \(nn=12\), Unclear Goal \(nn=12\), Code Dump \(nn=5\), Vague Error \(nn=8\), Wrong Terminology \(nn=6\), and Multi\-Issue \(nn=3\)\. These categories capture common ways in which student prompts become instructionally incomplete or noisy, ranging from missing contextual details to ambiguous goals and incorrect technical language\. Each scenario may exhibit one or more defect types\. This paired design yields30×2×4=24030\\times 2\\times 4=240total Round 1 evaluations\.

### IV\-BConcept Taxonomy

We constructed a prerequisite graph of 85 Python concepts organized hierarchically, from foundational concepts \(e\.g\.,variables,print\_function\) to advanced topics \(e\.g\.,dependency\_parsing,tensorflow\_intro\)\. Each concept has explicitly defined prerequisites; for instance,list\_comprehensionrequires bothlistsandfor\_loop\. This graph encodes the pedagogical structure of the course and enables automated detection of temporal violations in AI responses\.

### IV\-CAI Tutor Sources

We evaluated four LLMs spanning commercial and open\-weight models: ChatGPT \(OpenAI, GPT\-4\.5\)\[[19](https://arxiv.org/html/2608.05411#bib.bib26)\], Gemini \(Google, Gemini 3\)\[[13](https://arxiv.org/html/2608.05411#bib.bib27)\], Gemma 4 \(Google, 27B, open\-weight\)\[[8](https://arxiv.org/html/2608.05411#bib.bib29)\], and Qwen 3 \(Alibaba, 235B\-A22B, open\-weight\)\[[23](https://arxiv.org/html/2608.05411#bib.bib28)\]\. Including both commercial and open\-weight models allows us to examine whether model category is a meaningful predictor of pedagogical fit\. Each system received identical prompts containing student context, prior knowledge, current learning objectives, and the student’s question\. Responses were collected via standard web interfaces, with one prompt submitted per model at a time to ensure independent responses\.

### IV\-DPSI Implementation

The PSI evaluation pipeline operationalizes each sub\-score as follows:

Knowledge Distance\(SK​DS\_\{KD\}\) extracts concepts from each response using keyword matching against the 85\-concept taxonomy, computes Jaccard\-like overlap with the student’s known concepts, and measures deviation fromδopt=0\.65\\delta\_\{\\text\{opt\}\}=0\.65\.

Temporal Violation\(ST​VS\_\{TV\}\) checks the first\-appearance ordering of concepts against the prerequisite graph\. For each pair\(ci,cj\)\(c\_\{i\},c\_\{j\}\)wherecic\_\{i\}is prerequisite ofcjc\_\{j\}, we verify thatcic\_\{i\}appears beforecjc\_\{j\}in the response\.

Scaffolding Density\(SS​DS\_\{SD\}\) uses 25 regex\-based indicators in five categories: worked examples \(code blocks, output demonstrations\), step\-by\-step guidance, hints and prompts, comprehension checks, and analogies\. An adaptation multiplier rewards learner\-specific references\.

Cognitive Load\(SC​LS\_\{CL\}\) detects extraneous content using patterns for filler phrases \(e\.g\., “as an AI”\), hedging language, and tangential content\.

Bloom’s Alignment\(SR​AS\_\{RA\}\) classifies response cognitive level via weighted keyword frequency across six Bloom levels, with bonus weight for executable code examples\.

### IV\-EPSI\-Guided Regeneration Protocol

We selected 62 weak\-performing Round 1 cases for regeneration based primarily on the lowest PSI scores under the defective\-prompt condition\. We focused on this condition because it represents the more instructionally noisy setting, making it a stronger testbed for targeted pedagogical improvement\. The number selected from each model varied because the PSI score distributions differed across models: ChatGPT \(nn=4\), Gemini \(nn=23\), Gemma 4 \(nn=22\), and Qwen 3 \(nn=13\)\. ChatGPT had the fewest weak cases because its baseline PSI distribution was higher overall, leaving fewer cases below the selection threshold\. This PSI\-based selection targeted the cases with the weakest measured pedagogical fit for Round 2 regeneration\. The unequal per\-model sample sizes limit strong cross\-model conclusions from the regeneration data\.

Each regeneration prompt included five components: \(a\) the original student context and question, \(b\) the model’s Round 1 response, \(c\) the overall PSI score and all six sub\-scores, \(d\) a structured diagnosis of specific weaknesses with identified issues and severity, and \(e\) explicit improvement requirements formatted as a checklist\. For example, a case with lowSS​DS\_\{SD\}\(scaffolding density of 0\.05\) would receive feedback specifying: “Add at least one worked code example with expected output, a step\-by\-step breakdown, and a comprehension check question\.” The regenerated responses were then re\-evaluated using the same PSI pipeline\.

### IV\-FManual Validation

We conducted focused manual evaluation on the 62 PSI\-selected weak cases used in Round 2 regeneration\. For each case, a course instructor with prior experience teaching CSCI 150 examined both the original Round 1 response and the regenerated Round 2 response against the course materials \(29 slide decks, 31 Colab notebooks\)\. This evaluation assessed two criteria:Meaningful Improvement\(whether the regenerated response demonstrates better scaffolding structure than the original\) andWeaknesses Resolved\(whether the specific PSI\-identified issues have been adequately addressed\)\. Because this manual evaluation was limited to the PSI\-selected weak cases and was performed by a single expert rater, the results should be interpreted as preliminary\.

## VResults

### V\-ACross\-Model Comparison \(Round 1\)

TABLE I:Mean PSI Sub\-Scores \(Standard Prompts,nn=30 per model\)As shown in Table[I](https://arxiv.org/html/2608.05411#S5.T1), PSI scores ranged from 0\.557 \(Gemini\) to 0\.638 \(ChatGPT\), with Qwen 3 \(0\.595\) and Gemma 4 \(0\.565\) in between\. The overall spread of 0\.081 points is modest, and model type \(commercial versus open\-weight\) did not cleanly predict PSI rank: the open\-weight Qwen 3 outperformed the commercial Gemini\. The primary differentiator across models isSS​DS\_\{SD\}: ChatGPT \(0\.492\) provides over five times the scaffolding of Gemini \(0\.087\), suggesting that scaffolding behavior varies substantially even when overall pedagogical fit is broadly comparable\. Two sub\-scores show limited discriminative power in the current implementation:SE​FS\_\{EF\}is identical across models \(0\.334\) because it depends on scenario metadata, andSC​LS\_\{CL\}is near\-saturated \(≥\\geq0\.996\)\.

Fig\.[1](https://arxiv.org/html/2608.05411#S5.F1)shows that PSI decreases with target Bloom level across all models, from a cross\-model mean of 0\.60 at Remember to 0\.56 at Analyze\. This trend is consistent with the pedagogical fit perspective: as prerequisite chains lengthen and target cognitive levels rise, maintaining learner\-appropriate responses becomes harder for all models\.

![Refer to caption](https://arxiv.org/html/2608.05411v1/x1.png)Figure 1:Mean PSI by target Bloom level across all four models\. All models show a declining trend as cognitive complexity increases\.
### V\-BPrompt Robustness

Table[II](https://arxiv.org/html/2608.05411#S5.T2)compares PSI under standard versus defective prompts\.

TABLE II:PSI Under Standard vs\. Defective Prompts \(nn=30 per cell\)Across the tested scenarios, overall PSI changed only marginally \(Δ=−0\.002\\Delta=\-0\.002\), although sub\-score trade\-offs emerged \(Table[III](https://arxiv.org/html/2608.05411#S5.T3)\)\.

TABLE III:Sub\-Score Impact of Defective Prompts \(All Models,nn=120\)The sub\-score trade\-offs are consistent with the pedagogical fit perspective:ST​VS\_\{TV\}improved \(\+0\.059\+0\.059\), suggesting models compensate for missing context by imposing stronger prerequisite ordering, whileSK​DS\_\{KD\}\(−0\.041\-0\.041\) andSS​DS\_\{SD\}\(−0\.043\-0\.043\) both decreased, reflecting difficulty calibrating learner\-specific knowledge distance and scaffolding without explicit context\.

### V\-CPSI\-Guided Regeneration \(Round 2\)

Table[IV](https://arxiv.org/html/2608.05411#S5.T4)summarizes regeneration results\.

TABLE IV:Round 2 Regeneration ResultsPSI\-guided feedback produced improvement in 51 of 62 cases \(82\.3%\), with a mean gain of\+0\.049\+0\.049PSI points\. All four models improved on average, though unequal sample sizes \(4 to 23 per model\) limit strong cross\-model conclusions\.

Table[V](https://arxiv.org/html/2608.05411#S5.T5)shows the sub\-score changes\.

TABLE V:Sub\-Score Changes in Regeneration \(Mean of 62 Cases\)The dominant driver isSS​DS\_\{SD\}, which increased from 0\.073 to 0\.345 \(\+0\.272\+0\.272\), the largest single sub\-score change in the study\. This suggests that scaffolding is the dimension most responsive to structured feedback: when the regeneration prompt explicitly requests worked examples, step\-by\-step breakdowns, and comprehension checks, models incorporate them\.SK​DS\_\{KD\}also improved \(\+0\.039\+0\.039\), suggesting better knowledge calibration in regenerated responses\. However,ST​VS\_\{TV\}declined \(−0\.034\-0\.034\) andSR​AS\_\{RA\}decreased \(−0\.055\-0\.055\), indicating that adding scaffolding content can sometimes disrupt prerequisite ordering or shift the response’s cognitive level\.SE​FS\_\{EF\}remained unchanged, consistent with its metadata\-dependent nature\. These patterns suggest that single\-round feedback is most effective for concrete, instruction\-like dimensions \(scaffolding\), with diminishing impact on dimensions requiring deeper pedagogical judgment\. This trade\-off also suggests that the observed improvements reflect targeted responses to specific feedback instructions rather than uniform optimization of the composite score, and that multi\-round feedback balancing multiple dimensions may be needed to avoid such trade\-offs\.

### V\-DManual Validation

Table[VI](https://arxiv.org/html/2608.05411#S5.T6)presents the focused manual evaluation results for the 62 PSI\-selected weak cases\.

TABLE VI:Manual Evaluation of 62 PSI\-Selected Weak CasesThe “weaknesses resolved” rate \(82\.3%\) matched the automated PSI improvement count \(51/62\), providing initial evidence that PSI\-indicated improvement often corresponds to human\-judged pedagogical gains\. The “meaningful improvement” rate was lower \(74\.2%\), indicating that some PSI gains were not clearly visible instructionally\.

## VIDiscussion

### VI\-APSI as a Measure of Pedagogical Fit

Our results suggest that PSI captures meaningful differences in how well LLMs align their responses with learner readiness and curricular context\. As a diagnostic tool, it identifies scaffolding density \(SS​DS\_\{SD\}\) as the primary differentiator, with a 5\.7×\\timesvariation between the highest \(ChatGPT, 0\.492\) and lowest \(Gemini, 0\.087\) performers\. As a feedback signal, structured PSI diagnostics improve 82\.3% of weak cases, withSS​DS\_\{SD\}showing the largest gain \(\+0\.272\+0\.272\)\. This dual use, measuring pedagogical fit and then translating that measurement into targeted improvement, is the central contribution\.

Methodological note\.PSI should be interpreted as an operationalized, theory\-informed benchmark rather than a definitive psychometric instrument\. Two sub\-scores \(SE​FS\_\{EF\},SC​LS\_\{CL\}\) show limited discriminative power in the current implementation:SE​FS\_\{EF\}is identical across models because it depends on scenario metadata, andSC​LS\_\{CL\}is near\-saturated \(≥\\geq0\.996\)\. In the current implementation, most cross\-model differentiation comes from the remaining four sub\-scores\. PSI’s value lies in enabling consistent, curriculum\-aware comparison across models and generating structured feedback for improvement, while future work should refine component validity, weighting schemes, and inter\-rater reliability\.

### VI\-BComparison with Prior Work

PSI contributes to an emerging line of work on composite evaluation of AI\-generated tutoring content, simultaneously considering prerequisite ordering, scaffolding quality, retention timing, cognitive load, and Bloom alignment\. Prior frameworks have focused on isolated dimensions: factual accuracy\[[28](https://arxiv.org/html/2608.05411#bib.bib3)\], text readability\[[14](https://arxiv.org/html/2608.05411#bib.bib14)\], or knowledge tracing\[[16](https://arxiv.org/html/2608.05411#bib.bib17)\]\. PSI unifies these under a single interpretable score while maintaining diagnostic capability through sub\-score analysis\. The regeneration protocol extends recent work on self\-refinement\[[17](https://arxiv.org/html/2608.05411#bib.bib23)\]by showing that domain\-specific pedagogical feedback can produce targeted, measurable improvements\.

### VI\-CPrompt Robustness

Under the tested prompt perturbations, overall PSI remained largely stable, although sub\-score trade\-offs emerged, particularly between prerequisite ordering \(ST​VS\_\{TV\}:\+0\.059\+0\.059\) and knowledge calibration \(SK​DS\_\{KD\}:−0\.041\-0\.041\)\. Models appear to compensate for missing context by imposing default pedagogical structure, maintaining instructional ordering at the cost of learner\-specific calibration\. These findings are limited to the 30 tested scenarios and eight defect categories; broader generalization requires further study\.

### VI\-DImplications for AI System Design

Three implications follow from these results\. First, scaffolding is the most actionable improvement target: it varies substantially across models \(0\.087 to 0\.492\) and responds strongly to structured feedback \(SS​DS\_\{SD\}:\+0\.272\+0\.272\)\. Second, retention\-aware support likely requires external learner\-history data, sinceSE​FS\_\{EF\}remained unchanged across both models and rounds\. Third, the fact that all four models improved on average under PSI\-guided feedback suggests that curriculum\-aware alignment may matter more for tutoring quality than model category alone\.

### VI\-ELimitations

Four categories of limitation should be noted\.Measurement: PSI relies on proxy measures \(regex, keyword matching\) rather than deep semantic analysis, and two sub\-scores \(SE​FS\_\{EF\},SC​LS\_\{CL\}\) show limited discriminative power in the current implementation\.Scope: the study is based on one course \(CSCI 150\), one concept taxonomy, and 30 scenarios; generalization to other courses and disciplines remains to be established\.Experimental design: results reflect a single snapshot of each model under specific interface and parameter settings, the regeneration protocol uses a single feedback round with unequal per\-model sample sizes \(4 to 23\), and we report descriptive comparisons without inferential statistics\.Validation: focused manual evaluation covered only the 62 PSI\-selected weak cases and was performed by a single expert rater without inter\-rater reliability measurement\.

## VIIConclusion

We presented PSI, a six\-component metric for evaluating and improving the pedagogical fit of LLM\-based AI tutors\. Across 240 automated evaluations and 62 regeneration cases, pedagogical fit differed across models, but not in a way that supports a simple ranking narrative or a clear open\-versus\-closed divide\. Instead, the main finding is that effective AI tutoring depends less on model category alone and more on whether responses are aligned with learner readiness, prerequisite structure, cognitive level, and instructional timing\. Within this study, baseline PSI ranged from 0\.557 to 0\.638, overall performance remained largely stable under defective prompts, and PSI\-guided feedback improved 82\.3% of weak cases, with scaffolding density showing the largest gain \(\+0\.272\+0\.272\)\. Focused manual evaluation of the PSI\-selected weak cases provides initial evidence that PSI\-based improvement corresponds to human\-judged pedagogical gains\.

PSI should be understood as a theory\-informed operationalized benchmark with known limitations, including equal weighting, proxy\-based sub\-score computation, and two components \(SE​FS\_\{EF\},SC​LS\_\{CL\}\) with limited discriminative power in the current implementation\. Its value lies in enabling consistent cross\-model comparison on pedagogical dimensions and, more importantly, in translating diagnostic measurement into structured feedback that demonstrably improves learner\-aligned output quality\. These results suggest that pedagogical fit is both measurable and improvable through curriculum\-aware evaluation\. Future work should test PSI in live tutoring settings with authentic student\-AI interactions and examine whether the same patterns hold across additional courses and measured learning outcomes\. These extensions would help establish whether the pedagogical\-fit perspective generalizes beyond the present benchmark\.

## References

- \[1\]N\. E\. Adams\(2015\)Bloom’s taxonomy of cognitive learning objectives\.Journal of the Medical Library Association: JMLA103\(3\),pp\. 152\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1),[§III](https://arxiv.org/html/2608.05411#S3.p7.4)\.
- \[2\]R\. Alatrash, M\. A\. Chatti, N\. Wibowo, and Q\. U\. Ain\(2025\)Inferring prerequisite knowledge concepts in educational knowledge graphs: a multi\-criteria approach\.InInternational Joint Conference on Knowledge Graphs,pp\. 288–303\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[3\]C\. Alzetta, A\. Miaschi, G\. Adorni, F\. Dell’Orletta, F\. Koceva, S\. Passalacqua, and I\. Torre\(2019\)Prerequisite or not prerequisite? that’s the problem\! an nlp\-based approach for concept prerequisite learning\.InProceedings of the Sixth Italian Conference on Computational Linguistics \(CliC\-it 2019\),pp\. 14–21\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[4\]Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[5\]R\. Bodily and K\. Verbert\(2017\)Trends and issues in student\-facing learning analytics reporting systems research\.InProceedings of the seventh international learning analytics & knowledge conference,pp\. 309–318\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[6\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2023\)Teaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[7\]M\. Dahiya, R\. Gill, N\. Niu, H\. Gudaparthi, and Z\. Peng\(2024\)Leveraging chatgpt to predict requirements testability with differential in\-context learning\.In2024 IEEE International Conference on Information Reuse and Integration for Data Science \(IRI\),pp\. 170–175\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[8\]DeepMind\(2026\)Gemma 4\.Note:Google\[Online\]\. Available:https://deepmind\.google/models/gemma/gemma\-4/Cited by:[§IV\-C](https://arxiv.org/html/2608.05411#S4.SS3.p1.1)\.
- \[9\]P\. Denny, J\. Leinonen, J\. Prather, A\. Luxton\-Reilly, T\. Amarouche, B\. A\. Becker, and B\. N\. Reeves\(2024\)Prompt problems: a new programming exercise for the generative ai era\.InProceedings of the 55th ACM Technical Symposium on Computer Science Education V\. 1,pp\. 296–302\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[10\]S\. Doroudi, K\. Holstein, V\. Aleven, and E\. Brunskill\(2016\)Sequence matters but how exactly? a method for evaluating activity sequences from data\.\.Grantee Submission\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1),[§III](https://arxiv.org/html/2608.05411#S3.p3.4)\.
- \[11\]P\. Fernandes, M\. Treviso, D\. Pruthi, A\. Martins, and G\. Neubig\(2022\)Learning to scaffold: optimizing model explanations for teaching\.Advances in Neural Information Processing Systems35,pp\. 36108–36122\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1),[§III](https://arxiv.org/html/2608.05411#S3.p4.5)\.
- \[12\]F\. Gasparetti, C\. De Medio, C\. Limongelli, F\. Sciarrone, and M\. Temperini\(2018\)Prerequisites between learning objects: automatic extraction based on a machine learning approach\.Telematics and Informatics35\(3\),pp\. 595–610\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[13\]Google DeepMind\(2025\)Gemini 3 model\.Note:Google\[Online\]\. Available:https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-Pro\-Model\-Card\.pdfCited by:[§IV\-C](https://arxiv.org/html/2608.05411#S4.SS3.p1.1)\.
- \[14\]C\. Li, X\. Wang, and L\. Qian\(2025\)Exploring syntactic complexity and text readability in an elt textbook series for chinese english majors\.SAGE Open15\(1\),pp\. 21582440251323619\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1),[§III](https://arxiv.org/html/2608.05411#S3.p6.4),[§VI\-B](https://arxiv.org/html/2608.05411#S6.SS2.p1.1)\.
- \[15\]C\. Liang, Z\. Wu, W\. Huang, and C\. L\. Giles\(2015\)Measuring prerequisite relations among concepts\.InProceedings of the 2015 conference on empirical methods in natural language processing,pp\. 1668–1674\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[16\]Z\. Liu, Q\. Liu, J\. Chen, S\. Huang, J\. Tang, and W\. Luo\(2022\)PyKT: a python library to benchmark deep learning based knowledge tracing models\.Advances in Neural Information Processing Systems35,pp\. 18542–18555\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1),[§VI\-B](https://arxiv.org/html/2608.05411#S6.SS2.p1.1)\.
- \[17\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback, 2023\.URL https://arxiv\. org/abs/2303\.176512303\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1),[§VI\-B](https://arxiv.org/html/2608.05411#S6.SS2.p1.1)\.
- \[18\]J\. M\. Murre and J\. Dros\(2015\)Replication and analysis of ebbinghaus’ forgetting curve\.PloS one10\(7\),pp\. e0120644\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1),[§III](https://arxiv.org/html/2608.05411#S3.p5.7)\.
- \[19\]OpenAI\(2025\)GPT\-4\.5\.Note:OpenAI\[Online\]\. Available:https://openai\.com/index/gpt\-4\-5\-system\-card/Cited by:[§IV\-C](https://arxiv.org/html/2608.05411#S4.SS3.p1.1)\.
- \[20\]S\. Pal, V\. Arora, and P\. Goyal\(2020\)Finding prerequisite relations between concepts using textbook\.arXiv preprint arXiv:2011\.10337\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[21\]Z\. Peng, M\. Dahiya, T\. Khalil, N\. Niu, T\. Bhowmik, and Y\. Yang\(2023\)Resource\-centric goal model slicing for detecting feature interactions\.In2023 IEEE 24th International Conference on Information Reuse and Integration for Data Science \(IRI\),pp\. 58–63\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[22\]Z\. Peng, P\. Rathod, N\. Niu, T\. Bhowmik, H\. Liu, L\. Shi, and Z\. Jin\(2021\)Environment\-driven abstraction identification for requirements\-based testing\.In2021 IEEE 29th International Requirements Engineering Conference \(RE\),pp\. 245–256\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p3.1)\.
- \[23\]Qwen Team\(2025\)Qwen3: think deeper, act faster\.Note:Alibaba Cloud\[Online\]\. Available:https://qwenlm\.github\.io/blog/qwen3/Cited by:[§IV\-C](https://arxiv.org/html/2608.05411#S4.SS3.p1.1)\.
- \[24\]S\. Shen, Q\. Liu, Z\. Huang, Y\. Zheng, M\. Yin, M\. Wang, and E\. Chen\(2024\)A survey of knowledge tracing: models, variants, and applications\.IEEE Transactions on Learning Technologies17,pp\. 1858–1879\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[25\]L\. S\. Vygotsky\(1978\)Mind in society: the development of higher psychological processes\.Vol\.86,Harvard university press\.Cited by:[§I](https://arxiv.org/html/2608.05411#S1.p2.1)\.
- \[26\]A\. Waheed, M\. Goyal, N\. Mittal, D\. Gupta, A\. Khanna, and M\. Sharma\(2021\)Bloomnet: a robust transformer based model for bloom’s learning outcome classification\.InProceedings of the 4th International Conference on Natural Language and Speech Processing \(ICNLSP 2021\),pp\. 209–218\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[27\]V\. Wen, Z\. Peng, and Y\. Chen\(2025\)The ai imitation game: a cognitive comparison of mimicry in large language models\.In2025 IEEE International Conference on Information Reuse and Integration and Data Science \(IRI\),pp\. 79–84\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.
- \[28\]L\. Yan, L\. Sha, L\. Zhao, Y\. Li, R\. Martinez\-Maldonado, G\. Chen, X\. Li, Y\. Jin, and D\. Gašević\(2024\)Practical and ethical challenges of large language models in education: a systematic scoping review\.British Journal of Educational Technology55\(1\),pp\. 90–112\.Cited by:[§I](https://arxiv.org/html/2608.05411#S1.p2.1),[§II](https://arxiv.org/html/2608.05411#S2.p1.1),[§VI\-B](https://arxiv.org/html/2608.05411#S6.SS2.p1.1)\.
- \[29\]K\. Zhou, K\. Ethayarajh, D\. Card, and D\. Jurafsky\(2022\)Problems with cosine as a measure of embedding similarity for high frequency words\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 401–423\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p2.1)\.
- \[30\]K\. Zhu, J\. Wang, J\. Zhou, Z\. Wang, H\. Chen, Y\. Wang, L\. Yang, W\. Ye, Y\. Zhang, N\. Gong,et al\.\(2023\)Promptrobust: towards evaluating the robustness of large language models on adversarial prompts\.InProceedings of the 1st ACM workshop on large AI systems and models with privacy and safety analysis,pp\. 57–68\.Cited by:[§II](https://arxiv.org/html/2608.05411#S2.p1.1)\.