A Unified Response Framework for Questions and Tutoring Dialogues

# A Unified Response Framework for Questions and Tutoring Dialogues
Source: [https://arxiv.org/html/2610.19532](https://arxiv.org/html/2610.19532)
###### Abstract

Existing student models in educational applications largely focus on student responses to questions\. However, as tutoring becomes more accessible at large scale, it is important to analyze dialogues between tutors and students, which may reveal fine\-grained evidence on student knowledge\. In this work, we propose a unified framework that models both student responses to questions and tutor\-posed subtasks during dialogues on a shared item response theory \(IRT\) scale\. We first identify mathematical subtasks in tutoring dialogues, rewrite them into self\-contained questions, and identify the correctness of student responses to these subtasks\. We then develop an IRT framework that estimates question/subtask difficulty from text, tracks students’ evolving abilities across question\-answering and dialogue interactions, and a novel method to leverages tutor moves between consecutive subtasks to regularize their difficulty values\. We evaluate our framework on two real\-world math education datasets containing both human\- and AI\-tutored dialogues\. Results show that our framework outperforms strong baselines on both question difficulty prediction and response correctness prediction\.

## 1Introduction

Estimating student knowledge and item characteristics from students’ responses to mathematical questions/items is central to online learning platforms, which supports personalized practice, targeted tutoring, and reliable assessment\. There are two major types of student response modeling approaches: Item response theory \(IRT\) models place static student ability and question difficulty on a common scale[Rasch \(1960\)](https://arxiv.org/html/2610.19532#bib.bib9), while knowledge tracing \(KT\) models track how a student’s knowledge evolves over time across a sequence of questions[Corbett and Anderson \(1994\)](https://arxiv.org/html/2610.19532#bib.bib15)\. One key limitation of these models is that they only account for students’ response records, which may leave out fine\-grained information on their knowledge\. For example, for multiple\-choice questions, only knowing whether a student selected the correct option or which incorrect option they selected does not allow us to pinpoint their misunderstanding\.

Figure 1:Student responses to tutor\-posed subtasks in tutoring dialogues reveal fine\-grained insights into the student’s changing knowledge, while tutor moves help infer the relative difficulty of subtasks\.Powered by generative artificial intelligence \(AI\), online learning platforms increasingly combine standalone question\-answering practice with on\-demand tutoring[Eedi \(2024\)](https://arxiv.org/html/2610.19532#bib.bib16);[Khan Academy \(2023\)](https://arxiv.org/html/2610.19532#bib.bib17);[Carnegie Learning \(2024\)](https://arxiv.org/html/2610.19532#bib.bib18)\. Tutoring dialogues have significant potential in providing us fine\-grained insights into student knowledge: a tutoring dialogue is not merely an explanation attached to a question; it is a sequence of smaller assessment and learning events\. A tutor may ask the student to recall a fact, perform an intermediate calculation, or justify a step; after observing the response, the tutor may scaffold or push the student toward more rigorous reasoning[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib14)\. The student’s knowledge can therefore be more accurately estimated and may change throughout the dialogue\. Figure[1](https://arxiv.org/html/2610.19532#S1.F1)illustrates this process\. The original response only indicates that the student answered the fraction\-division question incorrectly, whereas the dialogue reveals an initial conceptual error, success on a simpler scaffolded task, correct intermediate calculations, and a remaining gap in justification\. These*tutor\-posed*tasks and student responses may provide substantially more insights into the student’s latent knowledge than the original response alone\.

#### Related Work

However, existing approaches do not provide a rigorous way to analyze original question responses and tutor\-posed tasks together\. Prior work has explored question difficulty prediction from question content[Huang et al\. \(2017\)](https://arxiv.org/html/2610.19532#bib.bib22);[Benedetto et al\. \(2021\)](https://arxiv.org/html/2610.19532#bib.bib26)and simulated student responses[Scarlatos et al\. \(2025b\)](https://arxiv.org/html/2610.19532#bib.bib2), as well as KT in tutoring dialogues[Scarlatos et al\. \(2025a\)](https://arxiv.org/html/2610.19532#bib.bib11)\. More recently,[Huang et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib35)incorporate tutor\-turn difficulty and IRT into conversational KT\. However, two challenges remain for jointly modeling original questions and dialogue subtasks\. First, tutoring dialogues follow personalized trajectories, resulting in subtasks that are often not shared across students and thus difficult to calibrate using traditional IRT\. Second, many tutor\-posed tasks implicitly refer to quantities, expressions, or reasoning steps introduced earlier in the problem or dialogue, making them difficult to represent as self\-contained, standalone questions\. Together, these challenges leave open the question of how to place original questions and tutor\-posed tasks on the same scale and use them to track student knowledge\. We provide a detailed discussion of related work in Appendix[A](https://arxiv.org/html/2610.19532#A1)\.

#### Contributions

In this work, we proposeURF\-QD\(Unified Response Framework for Questions and tutoring Dialogues\), a framework for jointly modeling responses to original questions and mathematical subtasks arising in corresponding tutoring dialogues\. Our contributions are threefold\. First, we identify tutor turns that pose mathematical tasks and rewrite these context\-dependent tasks as concise, self\-contained questions, enabling their difficulties and student responses to be modeled jointly with those of original questions\. Second, we introduce a dynamic IRT model that tracks the student’s evolving knowledge state from responses to both original questions and dialogue subtasks; importantly, we introduce a novel tutor move\-implied regularization term to supervise a relative ordering between consecutive subtasks\. Third, we evaluate our framework on two real\-world math education datasets containing both question\-answering records and tutoring dialogues\. Extensive evaluation shows that our framework consistently improves question difficulty prediction and response correctness prediction\. We use human evaluation and qualitative analyses to validate and illustrate components of our framework, and present a preliminary example of how our framework can be used for large language model \(LLM\) tutor benchmarking in the Appendix\. Our implementation will be made publicly available to enable future research on student assessment from dialogues\.

![Refer to caption](https://arxiv.org/html/2610.19532v1/urf-qd_main_fig.png)Figure 2:Overview of URF\-QD\. \(A\) Original questions and rewritten dialogue subtasks are modeled on a shared IRT scale using text\-based difficulty estimates and evolving student ability\. \(B\) Relative\-order regularization encourages scaffolded tasks to be easier and rigor\-promoting tasks harder than preceding tasks\. \(C\) Response correctness uses BCE with the regularization loss, while question difficulty uses GT\-difficulty MSE, dialogue\-subtask BCE, and the regularization loss\. GT denotes IRT\-estimated “ground\-truth” question difficulties and student abilities obtained from observed student responses to original questions\.

## 2Problem Formulation

We consider the setting of students working in digital learning platforms, where they practice on math questions and interact with tutors when they want additional support\. Our goal is to estimate a student’s ability from how they respond to bothquestionsandtutor\-posed subtasksthat arise naturally during tutoring dialogues\.

For each studentjj, we denote the binary\-valued correctness of their response to an original questionqiq\_\{i\}asyi,j∈\{0,1\}y\_\{i,j\}\\in\\\{0,1\\\}, where11corresponds to a correct response\. When receiving tutoring support on questionqiq\_\{i\}, the student and the tutor engage in a dialoguedi,j=\(s0,t1,s1,…,tM,sM\)d\_\{i,j\}=\(s\_\{0\},t\_\{1\},s\_\{1\},\.\.\.,t\_\{M\},s\_\{M\}\), wheresms\_\{m\}andtmt\_\{m\}denote student and tutor utterances respectively andMMis the number of total tutor\-student turn pairs\. The initial student turns0s\_\{0\}is present when the student initiates the conversation\.

We focus on tutor turns that pose a mathematical subtask, i\.e\., when the tutor requires the student to produce a mathematically meaningful response, such as an intermediate calculation, factual recall, method selection, interpretation, or explanation\. We treat student responses to these tutor\-posed tasks as additional response observations\. We detail how these tasks are identified and transformed into standalone questions in Section[3\.1](https://arxiv.org/html/2610.19532#S3.SS1.SSS0.Px2)\.

For each studentjj, we order all scored interactions chronologically and denote the resulting sequence asHj=\(\(xj,1,yj,1\),…,\(xj,T,yj,T\)\)H\_\{j\}=\(\(x\_\{j,1\},y\_\{j,1\}\),\.\.\.,\(x\_\{j,T\},y\_\{j,T\}\)\), wherexj,kx\_\{j,k\}denotes the mathematical task associated with interactionkk, andyj,k∈\{0,1\}y\_\{j,k\}\\in\\\{0,1\\\}denotes the corresponding response correctness\. An interaction may correspond either to an original question or to a subtask arising within a tutoring dialogue\.

Given the interaction historyHj,<kH\_\{j,<k\}, our primary modeling objective is to estimatep⁡\(yj,k=1\|xj,k,Hj,<k\)p\(y\_\{j,k\}=1\|x\_\{j,k\},H\_\{j,<k\}\), while placing student ability and the difficulties of original questions and dialogue tasks on a common IRT scale\. In this way, question\-answering and tutoring\-dialogue interactions provide complementary observations for modeling student performance\.

## 3Methodology

We now detail our framework for jointly modeling student performance in question\-answering activities and tutoring dialogues\. Our key idea is to place original questions and tutor\-posed dialogue subtasks on the same IRT parameter scale\. As shown in Figure[2](https://arxiv.org/html/2610.19532#S1.F2), our framework has three main components\. First, we identify mathematical subtasks posed by tutors during dialogues and transform these context\-dependent tasks into the format of standalone questions\. Second, we use a dynamic IRT model to jointly model responses to original questions and transformed dialogue subtasks on a shared scale\. Third, we leverage tutor pedagogical move labels to impose difficulty ordering constraints between consecutive tutor\-posed tasks, to provide additional supervision when estimating their difficulties\. We note that, for simplicity, we restrict ourselves to 1PL IRT models[Hambleton and Swaminathan \(2013\)](https://arxiv.org/html/2610.19532#bib.bib19), which is often referred to as the Rasch model[Rasch \(1960\)](https://arxiv.org/html/2610.19532#bib.bib9), where each question is only characterized by a single scalar\-valued difficulty parameter\. Our framework can be extended to other IRT models, although difficulty estimation is known to be more accurate than scale/discrimination estimation for 2PL IRT models[Chen and Chen \(2026\)](https://arxiv.org/html/2610.19532#bib.bib28)\.

### 3\.1Subtask Transformation

Tutoring dialogues contain many intermediate mathematical tasks that provide additional evidence of student ability\. However, tutor utterances are often context\-dependent and differ substantially in form from standalone assessment questions\. We therefore first identify tutor turns that constitute mathematical subtasks and then rewrite each identified subtask into a standalone question, allowing original questions and subtasks to be represented in a common form for subsequent modeling\.

#### Subtask Identification

For each dialoguedi,jd\_\{i,j\}, we consider every tutor turn that is immediately followed by a student turn as a candidate subtask\. We prompt GPT\-5\.4 with the complete dialogue to identify the valid mathematical subtasks: for each candidate, it determines whether the tutor turn poses an explicit mathematical task and whether the subsequent student turn attempts to answer it\. We retain only candidates that satisfy both conditions\. Following[Scarlatos et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib1), for each retained tutor subtask, we annotate whether the student responsesms\_\{m\}correctly answers the tutor\-posed task\. For each scored response, we denote this correctness label asyi,j,m∈\{0,1\}y\_\{i,j,m\}\\in\\\{0,1\\\}\. Turns without an applicable mathematical response are left unscored and are not used as response observations\.

#### Standalone Question Transformation

For every identified tutor subtasktmt\_\{m\}, we prompt GPT\-5\.4 to rewrite the tutor’s request as a concise, standalone assessment questionui,j,mu\_\{i,j,m\}\. Given the original problem and dialogue with the target turn marked, the model resolves contextual references and supplies only the information needed to make the question self\-contained while preserving its mathematical demand\. We provide further prompt details in Appendix[G](https://arxiv.org/html/2610.19532#A7)\.

For each scored subtask, we map the transformed questionui,j,mu\_\{i,j,m\}and its response labelyi,j,my\_\{i,j,m\}to the corresponding interactionkkin the student’s chronological historyHj:xj,k=ui,j,m,yj,k=yi,j,m\.H\_\{j\}:x\_\{j,k\}=u\_\{i,j,m\},y\_\{j,k\}=y\_\{i,j,m\}\.This mapping represents dialogue subtasks in the same form as original questions, enabling both interaction types to be modeled jointly\.

### 3\.2URF\-QD: Joint Modeling of Questions and Dialogue Subtasks

Given the chronological interaction historyHjH\_\{j\}, which contains both original question responses and transformed dialogue subtask interactions, we model student performance using a dynamic IRT framework\. We process interactions sequentially and maintain a latent student state𝐡j,k\\mathbf\{h\}\_\{j,k\}that summarizes the student’s interactions up to stepkk\. At interactionkk, we first encode the current taskxj,kx\_\{j,k\}and estimate its difficulty\. We then project the preceding state𝐡j,k−1\\mathbf\{h\}\_\{j,k\-1\}to a scalar ability and use a 1PL IRT model to predict the student’s response\. After observingyj,ky\_\{j,k\}, we update the student state using the task representation and response correctness\.

#### Task Encoder and Difficulty

We encode each taskxj,kx\_\{j,k\}using an LLM\. Let𝐙j,k=\(𝐳j,k,1,…,𝐳j,k,Lj,k\)\\mathbf\{Z\}\_\{j,k\}=\(\\mathbf\{z\}\_\{j,k,1\},\\ldots,\\mathbf\{z\}\_\{j,k,L\_\{j,k\}\}\)denote the last\-layer hidden state representations of the question sequencesLj,kL\_\{j,k\}\. We obtain a fixed\-dimensional task representation by mean pooling:𝐞j,k=1Lj,k​∑ℓ=1Lj,k𝐳j,k,ℓ\.\\mathbf\{e\}\_\{j,k\}=\\frac\{1\}\{L\_\{j,k\}\}\\sum\_\{\\ell=1\}^\{L\_\{j,k\}\}\\mathbf\{z\}\_\{j,k,\\ell\}\.We map this representation to a scalar difficulty using a multilayer perceptron \(MLP\)gdiffg\_\{\\mathrm\{diff\}\}:bj,k=gdiff​\(𝐞j,k\)\.b\_\{j,k\}=g\_\{\\mathrm\{diff\}\}\(\\mathbf\{e\}\_\{j,k\}\)\.The task encoder and difficulty prediction head are shared across original questions and dialogue subtasks, placing both on the same latent difficulty scale\. This text\-based parameterization is particularly important for dialogue subtasks, since different students may encounter different tutor\-posed tasks even when working on the same original question\. Rather than assigning an independent difficulty parameter to every observed task, the model estimates difficulty from its textual content, enabling the learned difficulty function to generalize across questions, students, and dialogues\.

#### Dynamic Student State and Ability

We represent studentjj’s interaction history with a latent state𝐡j,k\\mathbf\{h\}\_\{j,k\}that summarizes all observed task\-response pairs through stepkk\. After observing the student’s responseyj,ky\_\{j,k\}to taskxj,kx\_\{j,k\}, we incorporate this new evidence into the state\. We first represent response correctness using a learned embedding:𝐫j,k=Ey​\(yj,k\),\\mathbf\{r\}\_\{j,k\}=E\_\{y\}\(y\_\{j,k\}\),whereEyE\_\{y\}contains separate learnable embeddings for correct and incorrect responses to both original questions and dialogue subtasks\. We then concatenate resulting correctness embedding with task’s textual embedding\. One can use any deep learning\-based KT method for this dynamic update; unless otherwise noted, we defer to the deep KT \(DKT\) method[Piech et al\. \(2015\)](https://arxiv.org/html/2610.19532#bib.bib10)\.

#### IRT Response Model

Before the student attempts taskxj,kx\_\{j,k\}, we estimate their current ability by projecting the preceding state to a scalar:aj,k=ga​\(𝐡j,k−1\),a\_\{j,k\}=g\_\{a\}\(\\mathbf\{h\}\_\{j,k\-1\}\),wheregag\_\{a\}is a MLP\. This pre\-interaction ability is used in the IRT response model below and does not depend on the current responseyj,ky\_\{j,k\}\. After that response is observed, the updated state𝐡j,k\\mathbf\{h\}\_\{j,k\}is used to estimate ability at interactionk\+1k\+1\. Unlike a static IRT ability parameter,aj,ka\_\{j,k\}is dynamic throughout the student’s interaction sequence and updated when new responses are observed\. Therefore, dialogue subtask responses serve not only as correctness\-prediction targets, but also as additional observations that refine future ability estimates\. Given the student’s current abilityaj,ka\_\{j,k\}and the difficulty of the current task,bj,kb\_\{j,k\}, we model response correctness using a 1PL logistic IRT model:

p⁡\(yj,k=1∣xj,k,Hj,<k\)=σ⁡\(aj,k−bj,k\),p\(y\_\{j,k\}=1\\mid x\_\{j,k\},H\_\{j,<k\}\)=\\sigma\(a\_\{j,k\}\-b\_\{j,k\}\),whereσ⁡\(⋅\)\\sigma\(\\cdot\)denotes the sigmoid function\. Because original questions and dialogue subtasks share the same task encoder, difficulty head, ability projection, and IRT response function, their difficulties and the student’s dynamic ability are represented on a common latent scale\.

### 3\.3Difficulty Order Regularization

Correctness observations supervise the relationship between student ability and task difficulty, but tutoring dialogues provide an additional source of information: the tutor’s pedagogical move can imply a relative difficulty ordering between consecutive subtasks\. We exploit two cases,scaffolding after struggleandpushing for rigor[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib14), and use them to regularize task difficulties\.

#### Pedagogical Move Annotation

For each pair of consecutive tutor subtasks separated by a single student response, we prompt GPT\-5\.4 to determine whether the transition implies a tutor move that implies any relationship between the subtask difficulties\. The model is provided with the original problem, the raw tutoring dialogue with subtasks marked, eligible tutor\-turn pairs, and the annotated correctness label of the intervening student response\. We withhold the student’s response to the second subtask, so that the relation is determined from the tutor’s pedagogical action rather than from whether the student subsequently succeeds or fails\.

We denote the set of turns that correspond to scaffolding after struggle moves asΩ1\\Omega\_\{1\}\. In practice, when students respond incorrectly to tutor\-posed subtasks or indicate that they do not know the answer, tutors often scaffold by breaking down the original subtask into smaller, simpler, more concrete subtasks; such moves imply thatbj,k<bj,k−1b\_\{j,k\}<b\_\{j,k\-1\}\. Similarly, we denote the set of pushing for rigor turns asΩ2\\Omega\_\{2\}\. A pair is assigned toΩ2\\Omega\_\{2\}when the previous student response is correct and the subsequent tutor subtask increases the required depth of reasoning, for example by eliciting a justification, explanation, derivation, interpretation, proof, or other conceptual reasoning; such moves implybj,k\>bj,k−1b\_\{j,k\}\>b\_\{j,k\-1\}\. We assign no constraint when the intervening response cannot be scored for correctness, when the two subtasks are unrelated, or when the second tutor turn merely repeats the task, requests a generic retry, continues a routine multi\-step solution, or otherwise does not substantively change the cognitive demand\.

#### Regularization

For each\(j,k\)∈Ω1\(j,k\)\\in\\Omega\_\{1\}, the second task should be easier than the preceding task\. We define the corresponding hinge loss as

ℒscaf=1\|Ω1\|​∑\(j,k\)∈Ω1max⁡\{bj,k−bj,k−1,0\}\.\\mathcal\{L\}\_\{\\text\{scaf\}\}=\\frac\{1\}\{\|\\Omega\_\{1\}\|\}\\sum\_\{\(j,k\)\\in\\Omega\_\{1\}\}\\max\\\{b\_\{j,k\}\-b\_\{j,k\-1\},0\\\}\.The loss is positive when the scaffolded task is more difficult than the previous task, violating the tutor move, and zero otherwise\. We define the loss for pushing for rigor moves,ℒrigor\\mathcal\{L\}\_\{\\text\{rigor\}\}, in a similar way, withbj,kb\_\{j,k\}andbj,k−1b\_\{j,k\-1\}reversed\. We simply sum these two regularization terms together to obtainℒorder\\mathcal\{L\}\_\{\\mathrm\{order\}\}and add it to the objectives detailed below, using them as soft penalties to encourage the difficulty encoder to learn dialogue subtask difficulties that are consistent with tutoring moves\.

### 3\.4Prediction Targets, Training Objectives

We study two complementary prediction targets using separate training objectives\.*Question difficulty prediction*evaluates whether the shared difficulty encoder can accurately estimate the difficulty of previously unseen questions\.*Response correctness prediction*evaluates whether original\-question and dialogue\-subtask responses can jointly track a student’s evolving ability and accurately predict their future performance\.

#### Question Difficulty Prediction

Here, we examine whether dialogue subtasks provide additional calibration to help us estimate the difficulty of original questions\. We first fit an IRT model on observed student responses and use it to obtain a “ground\-truth” difficultyb~i\\tilde\{b\}\_\{i\}for each original questionqiq\_\{i\}\. We then train the shared difficulty predictor to recover these difficulties, while using dialogue interactions as additional supervision\. For an original questionqiq\_\{i\}, letbib\_\{i\}denote the difficulty value predicted by the encoder given the question content as input\. We supervise these predictions using mean squared error over the original questions available for training, asℒdiff​\-​MSE=1\|𝒬train\|​∑i∈𝒬train\(bi−b~i\)2\.\\mathcal\{L\}\_\{\\mathrm\{diff\\text\{\-\}MSE\}\}=\\frac\{1\}\{\|\\mathcal\{Q\}\_\{\\mathrm\{train\}\}\|\}\\sum\_\{i\\in\\mathcal\{Q\}\_\{\\mathrm\{train\}\}\}\\left\(b\_\{i\}\-\\tilde\{b\}\_\{i\}\\right\)^\{2\}\.

Since dialogue subtasks do not have corresponding ground\-truth difficulty labels, we additionally train on the observed student response correctness to extract information on subtask difficulty indirectly\. Leta~j\\tilde\{a\}\_\{j\}denote the fixed reference ability of studentjjfrom the fitted IRT, and letbi,j,mb\_\{i,j,m\}be the predicted difficulty of transformed dialogue subtaskui,j,mu\_\{i,j,m\}, we minimizeℒdiff​\-​BCE\\mathcal\{L\}\_\{\\mathrm\{diff\\text\{\-\}BCE\}\}, the standard binary cross\-entropy \(BCE\) loss between actual subtask response correctness labelsyi,j,my\_\{i,j,m\}and predicted correct response probabilityσ⁡\(a~j−bi,j,m\)\\sigma\(\\tilde\{a\}\_\{j\}\-b\_\{i,j,m\}\)\. We combine these loss terms asℒdiff=ℒdiff​\-​MSE\+ℒdiff​\-​BCE\+λ​ℒorder,\\mathcal\{L\}\_\{\\mathrm\{diff\}\}=\\mathcal\{L\}\_\{\\mathrm\{diff\\text\{\-\}MSE\}\}\+\\mathcal\{L\}\_\{\\mathrm\{diff\\text\{\-\}BCE\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{order\}\},whereλ\\lambdais the relative\-order regularization weight\.

#### Response Correctness Prediction

Here, given the interaction historyHj,<kH\_\{j,<k\}and the current taskxj,kx\_\{j,k\}, we predict whether studentjjwill respond correctly toxj,kx\_\{j,k\}using dynamic student abilityaj,ka\_\{j,k\}and predicted task difficultybj,kb\_\{j,k\}\. We define the BCE lossℒBCE\\mathcal\{L\}\_\{\\mathrm\{BCE\}\}analogously toℒdiff−BCE\\mathcal\{L\}\_\{\\mathrm\{diff\-BCE\}\}and combine it with the difficulty ordering regularization term to obtain the objective ofℒresp=ℒBCE\+λ​ℒorder\\mathcal\{L\}\_\{\\mathrm\{resp\}\}=\\mathcal\{L\}\_\{\\mathrm\{BCE\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{order\}\}\. In addition to contributing to the training objective, dialogue subtask responses also update the student’s knowledge state, thereby influencing future predictions\. This formulation enables us to investigate whether training on student responses to subtasks in dialogues provide additional, fine\-grained information on their knowledge\.

## 4Experimental Evaluation

We now detail our experimental setup to validate the effectiveness of our framework\.

### 4\.1Datasets

We conduct experiments on two math tutoring datasets collected by two major online learning platforms,EediandKhan Academy\. For simplicity, we refer to these datasets by their respective platform names\. Both datasets contain student interactions with math questions, where students can initiate chat\-based tutoring conversations on demand at any point during the question\-answering process\. The Eedi dataset contains conversations between real students and human tutors, whereas the Khan Academy dataset contains conversations between real students and an AI tutor model\. After processing, the Eedi dataset contains588588students,3,6893,689questions,1,7751,775dialogues, and152,280152,280question\-answering records, while the Khan Academy dataset contains136136students,3,4233,423questions,1,8001,800dialogues, and56,52956,529question\-answering records\. We provide additional dataset details in Appendix[B](https://arxiv.org/html/2610.19532#A2)\.

### 4\.2Metrics

For question difficulty prediction, following prior work[Huang et al\. \(2017\)](https://arxiv.org/html/2610.19532#bib.bib22);[Scarlatos et al\. \(2025b\)](https://arxiv.org/html/2610.19532#bib.bib2), we use the widely adopted Pearson Correlation Coefficient \(PCC\) to measure the linear correlation between predicted and ground\-truth question difficulties, Spearman Rank Correlation Coefficient \(SCC\) to measure the correlation between predicted and ground\-truth difficulty rankings among questions, Mean Squared Error \(MSE\) to measure squared prediction error, and Mean Absolute Error \(MAE\) to measure absolute prediction error\. For response correctness prediction, we use Area Under the ROC Curve \(AUC\), accuracy, and F1 score metrics\.

### 4\.3Baselines

We compare URF\-QD against kNN Mean, Random Forest, ICL kNN, and Diff SFT \(ModernBERT and Qwen3\.5\-4B\) for question difficulty prediction\. For response correctness prediction, we compare against DKT, DKT \(Joint\), AKT, simpleKT, and URF\-QD \(Orig\. Question\) to assess the contribution of dialogue subtask responses within and beyond our framework\. We use*Joint*to denote training on both original\-question and dialogue\-subtask responses and*Reg\.*to denote relative difficulty order regularization\. We provide detailed baseline descriptions in Appendix[C\.1](https://arxiv.org/html/2610.19532#A3.SS1)\.

### 4\.4Experimental Setup

Table 1:Performance comparison of methods on question difficulty prediction across datasets\. Best performance isboldedand second best isunderlined\.∗denotes statistically significant improvement over all baselines \(p<0\.05p<0\.05\)\.We report the average performance on 5\-fold cross validation\. We split datasets across problems \(students\) for question difficulty prediction \(response correctness prediction\) evaluation, to evaluate generalization to unseen problems \(students\)\. We use Qwen3\.5\-4B for URF\-QD and evaluate ICL kNN using GPT\-5\.4\. We provide full experimental details in Appendix[C\.2](https://arxiv.org/html/2610.19532#A3.SS2)\.

## 5Results, Analysis, and Discussion

We now detail our quantitative results on question difficulty prediction and student response correctness prediction, and qualitatively analyze the impact of regularization on both tasks\.

### 5\.1Quantitative Results

#### Our framework excels at question difficulty prediction\.

Table[1](https://arxiv.org/html/2610.19532#S4.T1)reports the average performance \(and standard deviation\) on both datasets for question difficulty prediction\. We see that URF\-QD \(Joint \+ Reg\.\) outperforms all baselines across all four metrics\. This consistent result shows that dialogue subtasks provide useful supervision for predicting difficulty of unseen original questions, even though the subtasks themselves have no ground\-truth difficulty labels\. On Eedi, compared with the strongest baseline, URF\-QD reduces MSE by7\.7%7\.7\\%and improves PCC and SCC by6\.5%6\.5\\%and8\.0%8\.0\\%, respectively\. On Khan Academy, URF\-QD similarly reduces MSE and MAE, and improves PCC and SCC\. The improvements over all baselines are statistically significant for MSE, PCC, and SCC on Eedi and for PCC on Khan Academy \(p<0\.05p<0\.05under paired t\-test\)\.

These results suggest that analyzing tutor\-posed subtasks do provide us with some information on question difficulty, which can be transferred to original questions via the shared text\-based difficulty encoder, once put on the same IRT scale\. This finding supports our central hypothesis that tutoring dialogues are not merely auxiliary conversational data; intermediate mathematical subtasks are important formative assessments that can directly improve question difficulty modeling\.

Table 2:Performance comparison of methods on response correctness prediction across datasets\. Best performance isboldedand second best isunderlined\.∗denotes statistically significant improvement over all baselines \(p<0\.05p<0\.05\)\.
#### Dialogue subtask responses improve response correctness prediction\.

Table[2](https://arxiv.org/html/2610.19532#S5.T2)reports the average performance \(and standard deviation\) on both datasets on response correctness prediction\. URF\-QD \(Joint \+ Reg\.\) again achieves the best performance on every metric for both Eedi and Khan Academy, showing that original question and dialogue subtask responses provide complementary evidence about students’ evolving knowledge\. On Eedi, compared with URF\-QD \(Orig\. Question\), incorporating dialogue subtasks and relative\-order regularization improves AUC and accuracy by6\.3%6\.3\\%and4\.5%4\.5\\%, respectively\. The gains in AUC and accuracy are statistically significant over all baselines \(p<0\.05p<0\.05under paired t\-test\)\. On Khan Academy, the full model also consistently improves all three metrics over its original\-question\-only counterpart\. DKT \(Joint\) similarly outperforms DKT across all metrics on both datasets, further confirming that dialogue responses provide complementary evidence for estimating dynamic student ability and predicting future performance\.

#### Ablation Study\.

We conduct an ablation study on Eedi to assess the contribution of each framework component\. Results show that incorporating dialogue subtasks improves both question difficulty and response correctness prediction, while relative\-order regularization provides additional gains for difficulty prediction\. We provide detailed ablation results and analyses in Appendix[D\.1](https://arxiv.org/html/2610.19532#A4.SS1)\.

### 5\.2Qualitative Results

#### Dialogue subtasks provide fine\-grained evidence of question difficulty\.

We now use a qualitative case study to illustrate how dialogue subtasks provide additional evidence for estimating the difficulty of an unseen original question\. Table[3](https://arxiv.org/html/2610.19532#S5.T3)shows a mixed\-number division problem that requires several mathematical operations, including converting mixed numbers, dividing fractions, and expressing the result in the requested form\. The question\-only Diff SFT baseline predicts a difficulty of 0\.483, underestimating the ground\-truth difficulty of 0\.625 by 0\.142\. The related dialogue subtasks capture mathematical demands that are required to solve the original problem, with their difficulties estimated jointly using response evidence from other questions and dialogues\. The relative\-order regularization further constrains the preceding subtask in a scaffolding pair to be harder than the scaffolded follow\-up, resulting in predicted difficulties of 0\.487 for Subtask 1 and 0\.256 for Subtask 2\. Because the original question requires the same mixed\-number division demand together with additional operations, it should be more difficult than Subtask 1\. The Diff SFT estimate of 0\.483 is therefore too low, whereas URF\-QD predicts a difficulty of 0\.573, reducing the absolute error to 0\.052\. Overall, this example shows how dialogue\-subtask supervision and relative\-order regularization help calibrate the difficulty of complex questions by providing fine\-grained evidence about their constituent mathematical demands\.

Table 3:Qualitative example of dialogue\-subtask supervision supporting difficulty prediction for an unseen original question\. Compared with the strongest baseline, URF\-QD assigns a difficulty closer to ground truth\.

## 6Human Evaluation

We conduct two human evaluations on the Eedi dataset with three PhD student researchers with math tutoring experience to assess the reliability of tutor move annotations and the quality of subtask rewriting\. GPT\-5\.4 achieves a macro\-F1 of 0\.754 for tutor move annotation, while85\.5%85\.5\\%of rewritten subtasks satisfy both dialogue\-independent completeness and mathematical\-demand preservation\. We provide additional details in Appendix[E](https://arxiv.org/html/2610.19532#A5)\.

## 7Conclusions and Future Work

In this paper, we presented URF\-QD, a framework for jointly modeling original questions and tutor\-posed subtasks on a shared IRT scale\. The key is to transform dialogue subtasks into standalone questions and leverage pedagogical relations to regularize their difficulty estimates\. Through experiments on two real\-world math learning datasets, we show that URF\-QD improves both question difficulty prediction and response correctness prediction\. In Appendix[F](https://arxiv.org/html/2610.19532#A6), we showcase how our framework opens up the possibility of many downstream analyses, benchmarking an LLM tutor’s ability to pose tasks at appropriate difficulty levels that maximize student learning outcomes\.

There are several avenues for future work\. First, the shared dynamic scale could support more analysis into the strategy and behavior of LLM tutors\. Future work should also use the extracted insights to guide adaptive subtask posing for both human and LLM tutors\. Second, future work should extend URF\-QD to incorporate visual elements, such as images and diagrams, that commonly appear in mathematics questions\. Finally, future work should examine whether the framework generalizes to other educational domains and settings\.

## Limitations

There are several limitations to our work\. First, although we conduct experiments on two real\-world math learning datasets, the Khan Academy dataset is not publicly available, and only a subset of the Eedi dataset is publicly accessible\. To the best of our knowledge, there are no other readily available datasets containing both student question\-answering records and corresponding tutoring dialogues at the granularity required by our framework, making broader evaluation challenging\. Second, since neither dataset provides ground\-truth question difficulty labels, we use difficulty estimates obtained by fitting an IRT model to observed student responses as reference labels\. These estimates depend on the assumptions and quality of the fitted IRT model and may not perfectly reflect the true difficulty of each question\. Finally, we restrict our framework to the 1PL IRT model and binary response correctness labels, which may not fully capture the complexity of student knowledge and mathematical responses\. Extending our framework to more expressive IRT models and finer\-grained response evaluations is an important direction for future work\.

## Ethical Considerations

There are several potential societal benefits to our work\. By jointly modeling student responses to original questions and tutoring dialogues, our framework can provide more fine\-grained estimates of student knowledge and question difficulty, potentially supporting more accurate educational assessment and personalized learning\. In particular, leveraging intermediate mathematical subtasks in tutoring dialogues may help educators better understand students’ learning needs and provide more targeted tutoring support\. There are also several potential risks\. Inaccurate estimates of student ability or question difficulty could lead to inappropriate learning recommendations or unfair assessments\. Additionally, our framework relies on LLM\-based annotations and transformations of tutoring dialogues, which may introduce errors or biases into downstream predictions\. Such biases could disproportionately affect students from underrepresented populations or with different linguistic backgrounds\. Moreover, demographic information is unavailable in our datasets, limiting our ability to evaluate potential disparities across student populations\. We emphasize that our framework is intended to support, rather than replace, educators’ judgments, and further evaluation of fairness and downstream learning outcomes is necessary before deployment in real\-world educational settings\.

## References

- AlKhuzaeyet al\.\(2024\)S\. AlKhuzaey, F\. Grasso, T\. R\. Payne, and V\. TammaText\-based question difficulty prediction: a systematic review of automatic approaches\.International Journal of Artificial Intelligence in Education34\(3\),pp\. 862–914\.External Links:ISSN 1560\-4292,[Document](https://dx.doi.org/https%3A//doi.org/10.1007/s40593-023-00362-1),[Link](https://www.sciencedirect.com/science/article/pii/S156042922600315X)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Benedettoet al\.\(2021\)L\. Benedetto, G\. Aradelli, P\. Cremonesi, A\. Cappelli, A\. Giussani, and R\. TurrinOn the application of transformers for estimating the difficulty of multiple\-choice questions from text\.InProceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications,J\. Burstein, A\. Horbach, E\. Kochmar, R\. Laarmann\-Quante, C\. Leacock, N\. Madnani, I\. Pilán, H\. Yannakoudakis, and T\. Zesch \(Eds\.\),Online,pp\. 147–157\.External Links:[Link](https://aclanthology.org/2021.bea-1.16/)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.SS0.SSS0.Px1.p1.1)\.
- Benedetto \(2023\)L\. BenedettoA quantitative study of nlp approaches to question difficulty estimation\.External Links:2305\.10236,[Link](https://arxiv.org/abs/2305.10236)Cited by:[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.External Links:2005\.14165,[Link](https://arxiv.org/abs/2005.14165)Cited by:[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Carnegie Learning \(2024\)Carnegie LearningLiveHint overview\.Note:Online:[https://support\.carnegielearning\.com/help\-center/math/livehint/article/livehint\-overview/](https://support.carnegielearning.com/help-center/math/livehint/article/livehint-overview/)Cited by:[§1](https://arxiv.org/html/2610.19532#S1.p2.1)\.
- Chaiklinet al\.\(2003\)S\. Chaiklinet al\.The zone of proximal development in vygotsky’s analysis of learning and instruction\.Vygotsky’s educational theory in cultural context1\(2\),pp\. 39–64\.Cited by:[Appendix F](https://arxiv.org/html/2610.19532#A6.p1.1)\.
- Chen and Chen \(2026\)S\. Chen and J\. ChenFrom text to parameters: predicting item parameters from embedding regularization with reliability and design ceilings\.External Links:2607\.07141,[Link](https://arxiv.org/abs/2607.07141)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§3](https://arxiv.org/html/2610.19532#S3.p1.1)\.
- Corbett and Anderson \(1994\)A\. T\. Corbett and J\. R\. AndersonKnowledge tracing: modeling the acquisition of procedural knowledge\.User modeling and user\-adapted interaction4\(4\),pp\. 253–278\.Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p2.1),[§1](https://arxiv.org/html/2610.19532#S1.p1.1)\.
- Eedi \(2024\)EediHelp all your students succeed in math\.Note:Online:[https://eedi\.com/us](https://eedi.com/us)Cited by:[§1](https://arxiv.org/html/2610.19532#S1.p2.1)\.
- Ghoshet al\.\(2020\)A\. Ghosh, N\. Heffernan, and A\. S\. LanContext\-aware attentive knowledge tracing\.External Links:2007\.12324,[Link](https://arxiv.org/abs/2007.12324)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p2.1),[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px2.p1.1)\.
- Gombertet al\.\(2024\)S\. Gombert, L\. Menzel, D\. Di Mitri, and H\. DrachslerPredicting item difficulty and item response time with scalar\-mixed transformer encoder models and rational network regression heads\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),E\. Kochmar, M\. Bexte, J\. Burstein, A\. Horbach, R\. Laarmann\-Quante, A\. Tack, V\. Yaneva, and Z\. Yuan \(Eds\.\),Mexico City, Mexico,pp\. 483–492\.External Links:[Link](https://aclanthology.org/2024.bea-1.40/)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Hambleton and Swaminathan \(2013\)R\. K\. Hambleton and H\. SwaminathanItem response theory: principles and applications\.Springer Science & Business Media\.Cited by:[§3](https://arxiv.org/html/2610.19532#S3.p1.1)\.
- Huanget al\.\(2026\)S\. Huang, A\. Scarlatos, J\. Lee, and A\. LanInterpretable difficulty\-aware knowledge tracing in tutor\-student dialogues\.External Links:2605\.01097,[Link](https://arxiv.org/abs/2605.01097)Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2017\)Z\. Huang, Q\. Liu, E\. Chen, H\. Zhao, M\. Gao, S\. Wei, Y\. Su, and G\. HuQuestion difficulty prediction for reading problems in standard tests\.InProceedings of the Thirty\-First AAAI Conference on Artificial Intelligence,AAAI’17,pp\. 1352–1359\.Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.19532#S4.SS2.p1.1)\.
- Kapooret al\.\(2026\)R\. Kapoor, S\. T\. Truong, N\. Haber, M\. A\. Ruiz\-Primo, and B\. W\. DominguePrediction of item difficulty for reading comprehension items by creation of annotated item repository\.External Links:2502\.20663,[Link](https://arxiv.org/abs/2502.20663)Cited by:[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Khan Academy \(2023\)Khan AcademySupercharge your teaching experience with khanmigo\.Note:Online:[https://www\.khanmigo\.ai/](https://www.khanmigo.ai/)Cited by:[§1](https://arxiv.org/html/2610.19532#S1.p2.1)\.
- Kochmaret al\.\(2025\)E\. Kochmar, K\. K\. Maurya, K\. Petukhova, K\. A\. Srivatsa, A\. Tack, and J\. VasselliFindings of the BEA 2025 shared task on pedagogical ability assessment of AI\-powered tutors\.InProceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2025\),E\. Kochmar, B\. Alhafni, M\. Bexte, J\. Burstein, A\. Horbach, R\. Laarmann\-Quante, A\. Tack, V\. Yaneva, and Z\. Yuan \(Eds\.\),Vienna, Austria,pp\. 1011–1033\.External Links:[Link](https://aclanthology.org/2025.bea-1.77/),[Document](https://dx.doi.org/10.18653/v1/2025.bea-1.77),ISBN 979\-8\-89176\-270\-1Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p1.1)\.
- Liuet al\.\(2023\)Z\. Liu, Q\. Liu, J\. Chen, S\. Huang, and W\. LuoSimpleKT: a simple but tough\-to\-beat baseline for knowledge tracing\.External Links:2302\.06881,[Link](https://arxiv.org/abs/2302.06881)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p2.1),[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px2.p1.1)\.
- Macinaet al\.\(2023\)J\. Macina, N\. Daheim, S\. Chowdhury, T\. Sinha, M\. Kapur, I\. Gurevych, and M\. SachanMathDial: a dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5602–5621\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.372/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.372)Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p1.1)\.
- Macinaet al\.\(2025\)J\. Macina, N\. Daheim, I\. Hakimi, M\. Kapur, I\. Gurevych, and M\. SachanMathtutorbench: a benchmark for measuring open\-ended pedagogical capabilities of llm tutors\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 204–221\.Cited by:[Appendix F](https://arxiv.org/html/2610.19532#A6.p1.1)\.
- OpenAI \(2026\)OpenAIIntroducing GPT\-5\.4\.External Links:[Link](https://openai.com/index/introducing-gpt-5-4/)Cited by:[§C\.2](https://arxiv.org/html/2610.19532#A3.SS2.p1.1)\.
- Petukhova and Kochmar \(2025\)K\. Petukhova and E\. KochmarIntent matters: enhancing AI tutoring with fine\-grained pedagogical intent annotation\.InProceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2025\),E\. Kochmar, B\. Alhafni, M\. Bexte, J\. Burstein, A\. Horbach, R\. Laarmann\-Quante, A\. Tack, V\. Yaneva, and Z\. Yuan \(Eds\.\),Vienna, Austria,pp\. 860–872\.External Links:[Link](https://aclanthology.org/2025.bea-1.63/),[Document](https://dx.doi.org/10.18653/v1/2025.bea-1.63),ISBN 979\-8\-89176\-270\-1Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p1.1)\.
- Piechet al\.\(2015\)C\. Piech, J\. Spencer, J\. Huang, S\. Ganguli, M\. Sahami, L\. Guibas, and J\. Sohl\-DicksteinDeep knowledge tracing\.External Links:1506\.05908,[Link](https://arxiv.org/abs/1506.05908)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p2.1),[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.19532#S3.SS2.SSS0.Px2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§C\.2](https://arxiv.org/html/2610.19532#A3.SS2.p1.1)\.
- Rasch \(1960\)G\. RaschProbabilistic models for some intelligence and attainment tests\.Studies in mathematical psychology,Danmarks Paedagogiske Institut\.External Links:ISBN 9780598554512,LCCN 62038085,[Link](https://books.google.com/books?id=XnlevwEACAAJ)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.p1.1),[§3](https://arxiv.org/html/2610.19532#S3.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-bert: sentence embeddings using siamese bert\-networks\.External Links:1908\.10084,[Link](https://arxiv.org/abs/1908.10084)Cited by:[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Scarlatoset al\.\(2025a\)A\. Scarlatos, R\. S\. Baker, and A\. LanExploring knowledge tracing in tutor\-student dialogues using llms\.InProceedings of the 15th International Learning Analytics and Knowledge Conference,LAK 2025,pp\. 249–259\.External Links:[Link](http://dx.doi.org/10.1145/3706468.3706501),[Document](https://dx.doi.org/10.1145/3706468.3706501)Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.SS0.SSS0.Px1.p1.1)\.
- Scarlatoset al\.\(2025b\)A\. Scarlatos, N\. Fernandez, C\. Ormerod, S\. Lottridge, and A\. LanSMART: simulated students aligned with item response theory for question difficulty prediction\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25071–25094\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1274/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1274),ISBN 979\-8\-89176\-332\-6Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.19532#S4.SS2.p1.1)\.
- Scarlatoset al\.\(2026\)A\. Scarlatos, J\. Lee, S\. Woodhead, and A\. LanSimulated students in tutoring dialogues: substance or illusion?\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 42349–42385\.External Links:[Link](https://aclanthology.org/2026.acl-long.1960/),ISBN 979\-8\-89176\-390\-6Cited by:[§C\.2](https://arxiv.org/html/2610.19532#A3.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.19532#S3.SS1.SSS0.Px1.p1.1)\.
- Waineret al\.\(2000\)H\. Wainer, N\. J\. Dorans, R\. Flaugher, B\. F\. Green, and R\. J\. MislevyComputerized adaptive testing: a primer\.Routledge\.Cited by:[Appendix F](https://arxiv.org/html/2610.19532#A6.SS0.SSS0.Px3.p1.1)\.
- Warneret al\.\(2024\)B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. PoliSmarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.External Links:2412\.13663,[Link](https://arxiv.org/abs/2412.13663)Cited by:[§C\.1](https://arxiv.org/html/2610.19532#A3.SS1.SSS0.Px1.p1.1)\.
- Yeung \(2019\)C\. YeungDeep\-irt: make deep learning based knowledge tracing explainable using item response theory\.External Links:1904\.11738,[Link](https://arxiv.org/abs/1904.11738)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p2.1)\.
- Zelikmanet al\.\(2023\)E\. Zelikman, W\. Ma, J\. Tran, D\. Yang, J\. Yeatman, and N\. HaberGenerating and evaluating tests for k\-12 students with language model simulations: a case study on sentence reading efficiency\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 2190–2205\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.135/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.135)Cited by:[§A\.1](https://arxiv.org/html/2610.19532#A1.SS1.p1.1)\.
- Zentet al\.\(2025\)M\. Zent, D\. Smith, and S\. WoodheadPIIvot: a lightweight NLP anonymization framework for question\-anchored tutoring dialogues\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 27479–27488\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1397/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1397),ISBN 979\-8\-89176\-332\-6Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p1.1)\.
- Zhanget al\.\(2026\)A\. Zhang, A\. Ross, K\. Patel, J\. Bernardo, R\. Bowie, A\. T\. Ribeiro, D\. Halper, H\. Valayaputtur, J\. Andreas, S\. Loeb, L\. Lucy, K\. Lo, and R\. KnightTutorMoments‑preview: when help is unhelpful — evaluating ai tutors for productive struggle\.External Links:[Link](https://tutormoments.allen.ai/)Cited by:[§A\.2](https://arxiv.org/html/2610.19532#A1.SS2.p2.1),[Appendix F](https://arxiv.org/html/2610.19532#A6.p1.1),[§1](https://arxiv.org/html/2610.19532#S1.p2.1),[§3\.3](https://arxiv.org/html/2610.19532#S3.SS3.p1.1)\.

## Appendix ARelated Work

### A\.1Question Difficulty Prediction and Student Response Modeling

IRT estimates student ability and item difficulty from observed responses[Rasch \(1960\)](https://arxiv.org/html/2610.19532#bib.bib9), but requires response data to calibrate unseen questions\. Text\-based difficulty prediction addresses this cold\-start setting using linguistic features and pretrained language models[Huang et al\. \(2017\)](https://arxiv.org/html/2610.19532#bib.bib22);[Benedetto et al\. \(2021\)](https://arxiv.org/html/2610.19532#bib.bib26);[Gombert et al\. \(2024\)](https://arxiv.org/html/2610.19532#bib.bib27);[AlKhuzaey et al\. \(2024\)](https://arxiv.org/html/2610.19532#bib.bib3)\. Recent work also predicts IRT parameters from item embeddings[Chen and Chen \(2026\)](https://arxiv.org/html/2610.19532#bib.bib28)\. Another direction simulates student responses with language models[Zelikman et al\. \(2023\)](https://arxiv.org/html/2610.19532#bib.bib29), including SMART, which aligns simulated students with IRT abilities and estimates difficulty from scored synthetic responses[Scarlatos et al\. \(2025b\)](https://arxiv.org/html/2610.19532#bib.bib2)\. Unlike these item\-centric methods, our framework leverages real student responses to tutor\-posed mathematical subtasks as additional supervision for difficulty prediction, allowing dialogue evidence to inform difficulty estimates for unseen original questions\.

KT models students’ evolving knowledge from interaction histories, ranging from Bayesian KT[Corbett and Anderson \(1994\)](https://arxiv.org/html/2610.19532#bib.bib15)and DKT[Piech et al\. \(2015\)](https://arxiv.org/html/2610.19532#bib.bib10)to attention\-based models such as AKT and simpleKT[Ghosh et al\. \(2020\)](https://arxiv.org/html/2610.19532#bib.bib12);[Liu et al\. \(2023\)](https://arxiv.org/html/2610.19532#bib.bib13)\. Deep\-IRT further combines neural KT with interpretable IRT parameters[Yeung \(2019\)](https://arxiv.org/html/2610.19532#bib.bib30)\. These methods primarily model predefined exercise\-response sequences, whereas our framework incorporates dialogue\-derived subtasks as additional response observations and tracks students’ evolving abilities across both original questions and tutoring dialogues on a shared IRT scale\.

### A\.2Student Modeling in Tutoring Dialogues

Tutoring dialogues reveal student understanding through intermediate mathematical tasks and adaptive feedback\. Datasets such as MathDial[Macina et al\. \(2023\)](https://arxiv.org/html/2610.19532#bib.bib31)and QATD\-2k[Zent et al\. \(2025\)](https://arxiv.org/html/2610.19532#bib.bib32)support research on tutoring strategies and pedagogical feedback\([Petukhova and Kochmar, 2025](https://arxiv.org/html/2610.19532#bib.bib33);[Kochmar et al\., 2025](https://arxiv.org/html/2610.19532#bib.bib34)\)\. More directly,[Scarlatos et al\. \(2025a\)](https://arxiv.org/html/2610.19532#bib.bib11)explore KT in tutoring dialogues, and[Huang et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib35)incorporate tutor\-turn difficulty and IRT into conversational KT\. However, these approaches focus on predicting dialogue responses rather than jointly modeling original\-question and subtask difficulties to predict the difficulty of unseen questions\.

Tutors also adjust task demands by scaffolding after struggle or pushing for rigor after success\([Zhang et al\., 2026](https://arxiv.org/html/2610.19532#bib.bib14)\)\. We use these pedagogical transitions to constrain the relative difficulty of consecutive subtasks\. Together with standalone subtask rewriting, this enables tutoring dialogues to provide supervision for both question difficulty and student ability estimation\.

## Appendix BAdditional Dataset Details

We provide additional details on the two datasets\. Both datasets are fully anonymized, so demographic information about individual students and tutors is not available\. The Eedi dataset contains multiple\-choice math question records and tutoring dialogues between real students and human tutors\. After processing, the dataset contains4,1754,175subtask instances across1,7751,775dialogues\. Each question is associated with a set of “subjects”, which we use to define the knowledge components \(KCs\)\. We assign each dialogue the same KCs as its associated question\. Among the subtask instances,467467correspond to scaffold after student struggle and4848correspond to push for rigor\. The Khan Academy dataset contains4,9964,996subtask instances across1,8001,800dialogues\. Each question is associated with a list of “skills”, which we similarly use to define the KCs for the corresponding dialogue\. The dataset contains549549cases of scaffold after student struggle and2121cases of push for rigor\. The questions from both datasets span a broad range of mathematical topics, including Algebra, Geometry, and Number\.

## Appendix CAdditional Experimental Details

We provide additional details on the baseline methods and experimental implementation for reproducibility\.

### C\.1Baselines

#### Difficulty Prediction Baselines

Following prior work[AlKhuzaey et al\. \(2024\)](https://arxiv.org/html/2610.19532#bib.bib3);[Benedetto \(2023\)](https://arxiv.org/html/2610.19532#bib.bib4);[Kapoor et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib5), we adopt a strong fine\-tuning baseline,Diff SFT, which fine\-tunes a recent and capable encoder\-based language model, specifically ModernBERT\-base[Warner et al\. \(2024\)](https://arxiv.org/html/2610.19532#bib.bib6)\. We use a regression setup, minimizing MSE, to directly predict question difficulties given the question’s text and metadata\. We additionally trainDiff SFT \(Qwen3\.5\-4B\)using the same question encoder as our framework\. We then report aRandom Forestbaseline, which is common for difficulty prediction[AlKhuzaey et al\. \(2024\)](https://arxiv.org/html/2610.19532#bib.bib3);[Gombert et al\. \(2024\)](https://arxiv.org/html/2610.19532#bib.bib27), where we encode the item’s question text and meta data using SBERT model[Reimers and Gurevych \(2019\)](https://arxiv.org/html/2610.19532#bib.bib7), and use the resulting embeddings as input features to fit a random forest regressor to item difficulty\. We also use a simple yet strong baseline,kNN Mean, that retrieves thekkquestions in the train set with the highest SBERT embedding cosine similarity, and takes their mean difficulty as the predicted test item difficulty\. We search overk∈\[1,5\]k\\in\[1,5\]and select the best performing value on the validation set, selectingk=4k=4\. Finally, we adapt an LLM prompting baseline leveraging in\-context learning \(ICL\)[Brown et al\. \(2020\)](https://arxiv.org/html/2610.19532#bib.bib8),ICL kNN, where we prompt GPT\-5\.4 with k in\-context examples of questions retrieved from the train set using k\-Nearest Neighbors \(kNN\)\. We use the same SBERT model as above, and perform a similar hyperparameter search, selectingk=4k=4\. We provide the prompt details in Appendix[G](https://arxiv.org/html/2610.19532#A7)\. All these baselines use only original questions and their reference difficulties, without dialogue\-subtask response supervision or difficulty ordering regularization\.

#### Response Prediction Baselines

For student response correctness prediction, we compare against three KT baselines:DKT[Piech et al\. \(2015\)](https://arxiv.org/html/2610.19532#bib.bib10), a simple recurrent neural network\-based KT method, andDKT \(Joint\), which is trained on the combined chronological sequence of original question and dialogue subtask responses;AKT[Ghosh et al\. \(2020\)](https://arxiv.org/html/2610.19532#bib.bib12), a full attention\-based KT method; andsimpleKT[Liu et al\. \(2023\)](https://arxiv.org/html/2610.19532#bib.bib13)a simple but robust KT method\. Unless marked as Joint, these baselines are trained only on original question responses\. We evaluate all methods on the same held\-out original\-question response events\. Dialogue\-subtask responses are incorporated as additional training observations and as history for the joint models\.

We additionally reportURF\-QD \(Orig\. Question\), an original question\-only variant of our response prediction model\. It retains the same text\-based difficulty estimator, sequential student state model, and IRT response function as our full framework, but excludes all dialogue subtask interactions and the difficulty ordering regularizer\. This baseline isolates the contribution of dialogue evidence within the proposed architecture\.

### C\.2Implementation Details

We use Qwen3\.5\-4B[Qwen Team \(2026\)](https://arxiv.org/html/2610.19532#bib.bib20)as the frozen LLM encoder for our framework and multi\-qa\-mpnet\-base\-dot\-v1 SBERT model for baselines and ablation studies\. We use GPT\-5\.4[OpenAI \(2026\)](https://arxiv.org/html/2610.19532#bib.bib21)for the prompting\-based baseline \(ICL kNN\), mathematical subtask identification, subtask correctness labeling, standalone\-question transformation, and pedagogical move annotation\. We adopt the LLM\-based correctness annotation protocol of[Scarlatos et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib1), which demonstrated substantial agreement with expert human annotations \(Cohen’sκ\\kappa= 0\.722\), comparable to human inter\-annotator agreement \(κ\\kappa= 0\.719\), providing empirical support for its reliability\.

For question difficulty prediction, we obtain25602560\-dimensional question embeddings by mean pooling over the final\-layer hidden states of the frozen LLM encoder\. We use a three\-layer MLP and GELU activations to predict scalar question difficulty\. We train the difficulty predictor using AdamW with a fixed learning rate of5⋅10−35\\cdot 10^\{\-3\}, full\-batch optimization, and gradient norm clipping at10\.010\.0for120120epochs\. For the ICL \(kNN\) baseline, we ask the model to first reason about the question’s difficulty in a chain\-of\-thought manner before generating a numeric difficulty prediction\.

To obtain reference student abilities and question difficulties, we fit a 1PL IRT model using observed student responses to original questions\. We fit the reference IRT model on all eligible original\-question responses, without a train/validation/test split, including responses associated with questions subsequently assigned to validation and test sets\. We implement IRT fitting using SciPy with the L\-BFGS\-B optimization algorithm, setting the maximum number of iterations to500500\. We apply L2 regularization to both student ability and question difficulty parameters, with a regularization strength of1\.01\.0for each\.

For response correctness prediction, we train an LSTM with a hidden\-state dimension of200200to model the evolving student state\. We represent response correctness using a learned3232\-dimensional embedding, which is concatenated with the25602560\-dimensional question embedding to form a25922592\-dimensional LSTM input\. Before predicting the current response, we project the preceding LSTM hidden state to a scalar student ability using a two layer MLP and a GELU activation\. We train the model using AdamW with a fixed learning rate of1⋅10−31\\cdot 10^\{\-3\}, gradient norm clipping at10\.010\.0, and a batch size of6464for1010epochs\. For both prediction tasks, we set the difficulty\-order regularization weight toλ=1\\lambda=1\.

We compute AUC using scikit\-learn and Pearson and Spearman correlations using NumPy\. To the best of our knowledge, the software and models used in our implementation either have open\-source licenses or do not list a license\. In addition, our use of these resources, including the OpenAI API, is consistent with their intended terms of use\. We will ensure that the license and terms reflect the sources on which our implementation builds when we release code\. We use ChatGPT to assist with writing and editing the manuscript\.

## Appendix DAdditional Quantitative Evaluation

Table 4:Ablation study on question difficulty prediction \(Eedi\)\. URF\-QD\(Full\) is equivalent to URF\-QD\(Joint \+ Reg\.\)\. Questions Only excludes subtasks, which is equivalent to Diff SFT \(Qwen3\.5\-4B\); w/o Reg\. keeps original question and dialogue subtasks but removes relative\-order regularization; w/o Rewrite uses original tutor\-posed subtasks instead of standalone rewrites; SBERT \(MPNet\) replaces Qwen3\.5\-4B encoder with a lightweight MPNet\-based sentence encoder\.Table 5:Ablation study on response correctness prediction \(Eedi\)\. URF\-QD\(Full\) is equivalent to URF\-QD\(Joint \+ Reg\.\)\. Questions Only excludes subtasks, which is equivalent to URF\-QD \(Orig\. Question\); w/o Reg\. removes relative\-order regularization; w/o Rewrite uses original tutor\-posed subtasks; SBERT \(MPNet\) replaces Qwen3\.5\-4B encoder with a lightweight MPNet\-based sentence encoder; and simpleKT Backbone replaces the default DKT\-based student\-state model\.### D\.1Additional Ablation Study

We analyze the contribution of each component of our framework on Eedi, which we choose as the representative dataset because its dialogues involve real human tutors and students, yielding more relative\-order constraints and higher\-quality tutoring interactions\.

#### Question Difficulty Prediction\.

Table[4](https://arxiv.org/html/2610.19532#A4.T4)reports the ablation results for question difficulty prediction\. The original\-question\-only variant performs worse than the full model, showing that the decomposed subtasks provide additional information for estimating question difficulty\. Removing the relative\-order regularization also leads to consistent performance drops across all four metrics, suggesting that explicitly modeling relative difficulty helps calibrate question difficulty across the two task types when placing them on a shared scale\. Using the original subtasks without rewriting generally outperforms the Diff SFT baseline, improving MSE, PCC, and SCC, although MAE is slightly worse; the full model further improves over this variant across all four metrics, indicating that rewriting produces more self\-contained subtask descriptions that are more informative for difficulty prediction\. Finally, replacing the Qwen3\.5\-4B question encoder with a lightweight MPNet\-based SBERT encoder leads to the largest performance drop, with MSE increasing from 0\.350 to 0\.398 and PCC decreasing from 0\.458 to 0\.341, suggesting that representations from the larger language model are more effective for question difficulty prediction\.

#### Response Correctness Prediction\.

Table[5](https://arxiv.org/html/2610.19532#A4.T5)reports the ablation results for response correctness prediction\. Removing the subtasks and using only the original question decreases AUC from 0\.637 to 0\.599 and accuracy from 0\.598 to 0\.572, supporting that decomposed subtasks provide useful information beyond the original question\. Unlike for question difficulty prediction, removing the relative\-order regularization does not hurt response prediction\. One possible explanation is that the comparatively sparse ordering constraints are diluted by the large number of interaction records, suggesting that this regularization is more beneficial for difficulty estimation\. Removing the rewriting step leads to performance drops in AUC and accuracy, while replacing the Qwen3\.5\-4B encoder with a lightweight SBERT encoder also generally reduces performance, highlighting the value of self\-contained subtask representations and richer semantic encodings\. Finally, replacing DKT with simpleKT yields comparable results, indicating that URF\-QD is not strongly dependent on a particular knowledge tracing model\.

\(a\)Question difficulty prediction\.\(b\)Response correctness prediction\.
Figure 3:Impact of the relative\-order regularization strengthλ\\lambdaon the two prediction targets\.
#### Encoder Fine\-tuning\.

We also compare the default setup, which uses Qwen3\.5\-4B as a frozen encoder, with jointly fine\-tuning the encoder and downstream prediction modules\. Fine\-tuning reduces performance on both tasks\. For question difficulty prediction, MSE and MAE increase to0\.4080\.408and0\.4860\.486, while PCC and SCC decrease to0\.3850\.385and0\.3590\.359, respectively\. For response correctness prediction, AUC, accuracy, and F1 decrease to0\.6260\.626,0\.5890\.589, and0\.5520\.552, respectively\. Joint fine\-tuning also substantially increases training time\. In this setting, keeping Qwen3\.5\-4B frozen is both more effective and more efficient\.

### D\.2Impact of Relative\-Order Regularization

We further analyze the effect of the relative\-order regularization strengthλ\\lambda\. For question difficulty prediction,λ=0\\lambda=0is equivalent to removing the relative\-order regularization\. As shown in Figure[3](https://arxiv.org/html/2610.19532#A4.F3)[3\(a\)](https://arxiv.org/html/2610.19532#A4.F3.sf1),λ=0\.1\\lambda=0\.1performs similarly toλ=0\\lambda=0, suggesting that a weak regularization signal has limited effect\. Performance improves asλ\\lambdaincreases to0\.50\.5and11, whileλ=1\\lambda=1andλ=3\\lambda=3achieve comparable performance across the four metrics\. In contrast, increasingλ\\lambdato1010leads to a clear performance drop\. A possible reason is that overly strong regularization causes the training objective to focus excessively on relative difficulty ordering rather than accurately fitting the absolute difficulty labels\.

For response correctness prediction, Figure[3](https://arxiv.org/html/2610.19532#A4.F3)[3\(b\)](https://arxiv.org/html/2610.19532#A4.F3.sf2)shows that performance remains relatively stable acrossλ∈\{0,0\.1,0\.5,1,3\}\\lambda\\in\\\{0,0\.1,0\.5,1,3\\\}, suggesting that response prediction is not sensitive to regularization weight within a moderate range\. In contrast, settingλ=10\\lambda=10substantially hurts all three metrics, likely because overly strong regularization constrains the model at the expense of optimizing the response\-prediction objective\.

To further examine how the regularization affects learned question difficulties, we evaluate the satisfaction rate of the pedagogically implied difficulty orderings\. Specifically, we calculate the proportion of subtask pairs satisfyingbj,k<bj,k−1b\_\{j,k\}<b\_\{j,k\-1\}for scaffolding after struggle \(Ω1\\Omega\_\{1\}\) andbj,k\>bj,k−1b\_\{j,k\}\>b\_\{j,k\-1\}for pushing for rigor \(Ω2\\Omega\_\{2\}\)\.

As shown in Table[6](https://arxiv.org/html/2610.19532#A5.T6), the relative\-order regularization substantially improves the consistency between predicted difficulties and pedagogical ordering constraints\. Without regularization \(λ=0\\lambda=0\), the overall satisfaction rate is only 0\.699\. Increasingλ\\lambdato 1 raises this rate to 0\.899, with improvements for both scaffolding after struggle \(0\.697 to 0\.898\) and pushing for rigor \(0\.756 to 0\.911\)\. This result indicates that the regularization effectively encourages the difficulty predictor to capture the relative mathematical demands reflected in tutors’ pedagogical moves\.

While increasingλ\\lambdafurther improves order satisfaction, reaching 0\.914 atλ=10\\lambda=10, it leads to worse difficulty prediction performance, suggesting that excessive regularization may compromise absolute difficulty estimation\. Overall, these results demonstrate that our proposed relative\-order regularization effectively aligns learned task difficulties with the pedagogically implied difficulty orderings in tutoring dialogues, providing better\-calibrated difficulty estimates that benefit downstream predictions when applied at a moderate strength\.

## Appendix EHuman Evaluation Details

We conduct two human evaluations on the Eedi dataset to assess our tutor move annotations and subtask rewriting quality\. Three PhD student researchers with mathematics tutoring experience are recruited to conduct the human evaluations\. All participants voluntarily agreed to participate and received no financial compensation\. Participation was free of coercion\. They are informed of the evaluation procedures and voluntarily agreed to participate before conducting the annotations; additional demographic and geographic information is not collected\.

Table 6:Difficulty order satisfaction rates under different relative\-order regularization strengthsλ\\lambda\.Ω1\\Omega\_\{1\}andΩ2\\Omega\_\{2\}denote scaffolding after struggle and pushing for rigor, respectively\.First, annotators are asked to label consecutive subtask pairs in 25 randomly sampled dialogues \(204 turns\) as scaffolding after struggle, pushing for rigor, or other tutor moves\. Inter\-annotator agreement, in Fleiss’sκ\\kappa, isκ=0\.651\\kappa=0\.651for scaffolding\-after\-struggle labels and0\.5910\.591for push\-for\-rigor labels, showing moderate\-to\-good agreement\. Against human annotations, GPT\-5\.4 achieves a macro\-averaged positive\-class F1 of0\.7540\.754across the two target pedagogical moves, with F1 scores of0\.8330\.833for scaffolding after struggle and0\.6740\.674for pushing for rigor\. In Eedi, pushing for rigor is considerably less frequent than scaffolding after struggle \(48 vs\. 467 labeled pairs\)\. This class imbalance may partly contribute to the lower annotation agreement and F1 score for pushing for rigor\.

Second, annotators are asked to evaluate 110 rewritten subtasks fordialogue\-independent completeness, i\.e\., the rewritten question being self\-contained and solvable without dialogue context andmathematical\-demand preservation, i\.e\., the rewritten question retaining the original subtask’s mathematical requirements\. Annotators judge92\.7%92\.7\\%and90\.0%90\.0\\%of subtasks as satisfying the respective criteria, with85\.5%85\.5\\%receiving positive judgments on both criteria\. Together, these evaluations support the reliability of LLM\-based tutor move annotation and subtask rewriting\.

## Appendix FLLM Tutor Benchmarking

We now provide an example of how URF\-QD opens door for many learning analytics tasks\. In this preliminary example, we investigate how the zone of proximal development[Chaiklin and others \(2003\)](https://arxiv.org/html/2610.19532#bib.bib24)takes effect in tutoring dialogues: by studying the relationship between tutor\-posed task difficulty and student learning outcomes, we can study what types of tasks during scaffolding are the most beneficial\. Using these findings, we can create a benchmark to evaluate the pedagogical capabilities of LLMs, in terms of posing scaffolding tasks at the right difficulty level to facilitate student learning\. Such a benchmark would complement existing benchmarks such as MathTutorBench[Macina et al\. \(2025\)](https://arxiv.org/html/2610.19532#bib.bib25)and TutorMoments[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.19532#bib.bib14), which do not use principled psychometric models and only rely on LLM\-as\-a\-judge\.

#### Task difficulty and next question correctness

We revisit the Eedi dialogues using the previously fitted, five\-fold student\-disjoint dynamic 1PL model with a LSTM and frozen text encoder\. For each dialogue, for all tutor\-posed subtasks, we compute the average gap in the estimated student abilityθ\\thetaand the turn difficultybb\. The learning outcome measure we study is the correctness of the student’s response on the first original question response after the dialogue’s recorded start, which shares at least one KC label with the dialogue and does not have an associated dialogue\. This filtering yields 553 dialogues from 327 students and 514 next question responses\. We perform a simple linear fit, shown in Figure[4](https://arxiv.org/html/2610.19532#A6.F4), which reveals a positive relationship between next question response outcomes andθ−b\\theta\-b, with slope0\.6130\.613and a p\-value of0\.0010\.001\. Therefore, empirical data suggests that larger gaps, corresponding to easier subtasks relative to the student’s ability, are associated with higher subsequent response correctness\.

#### Human versus LLM tutor tasks

With this finding, we now perform a comparison between human tutors and LLM tutors to examine how they tend to pose subtasks during the dialogue\. For each subtask, we prompt GPT\-5\.4 with the original problem and preceding dialogue to generate an appropriate next tutoring task and a standalone rewrite\. We then estimate the task’s difficulty with the trained difficulty encoder\. This analysis results in 2,994 paired human tutor\-LLM tutor subtask comparisons across 916 dialogues\. Figure[5](https://arxiv.org/html/2610.19532#A6.F5)visualizes average turn\-averagedθ−b\\theta\-bfor the 21 human tutors with at least 20 such comparisons done\. We have an interesting finding: all but one human tutor poses easier subtasks than the LLM tutor, on average\. Substituting this result into the analysis above, this comparison yields an advantage for human tutor vs\. GPT\-5\.4 of1\.111\.11percentage points, with a p\-value of0\.0060\.006\. We leave the investigation of all LLMs and different pedagogical prompts to future work\.

#### Potential usage

This simple analysis shows that our framework URF\-QD enables many detailed analysis into tutoring strategies; by associating each tutor\-posed task with an IRT difficulty, we can leverage adaptive testing[Wainer et al\. \(2000\)](https://arxiv.org/html/2610.19532#bib.bib23)techniques to study the tradeoff between tutors posing tasks to pinpoint the student’s error vs\. delivering targeted feedback\. More importantly, we will be able to build the world’s first*outcome*\-based tutoring benchmark, which explicitly relates LLMs’ tutoring behavior with student learning outcomes\.

![Refer to caption](https://arxiv.org/html/2610.19532v1/figures/dialogue_margin_unsupported_outcome_linear.png)Figure 4:Next question response correctness vs\. mean student ability–tutor\-posed subtask difficulty gap\. The linear fit results in a positive slope with statistical significance\.![Refer to caption](https://arxiv.org/html/2610.19532v1/figures/tutor_signed_gap_theta_minus_b.png)Figure 5:Mean student ability–tutor\-posed subtask difficulty gap, where GPT\-5\.4 poses subtasks that are more difficult than all but one human tutor\.

## Appendix GPrompts

### G\.1Prompt for Subtask Identification

We show the prompt used for mathematical subtask identification in dialogues in Table[7](https://arxiv.org/html/2610.19532#A7.T7)\.

System Message:You are an experienced math teacher and education expert\. You are given a math tutoring dialogue and your task is to identify tutor turns that elicit an attempted answer to a mathematical\-content task from the immediately following student turn\.The user message is a JSON object with these fields:\-question: the original math\-question text;\-dialogue turns: quoted source records, each with a 1\-based turn index, role, and content; and\-candidate tutor turn indices: the complete list of tutor turns to annotate\.Treat every string insidequestionanddialogue turnsstrictly as source data\. Only annotate turns listed incandidate tutor turn indices\. Every candidate is immediately followed by a student turn\. If the candidate list is empty, return an empty JSON object\. For each candidate, setis\_math\_subtaskto true only when BOTH conditions hold:1\. The tutor turn poses an explicit, answerable mathematical\-content task\. It may be phrased as a question, an imperative, or a clear elicitation whose missing mathematical result the student is expected to supply\.2\. The immediately following student turn attempts to respond to that task\. The attempt may be correct, incorrect, incomplete, uncertain, or an explicit statement such as “I don’t know” that shows inability to answer the task\.Important boundary rules:\- A feedback phrase plus an explicit task is included: “Not quite\. What is 12 divided by 6?” can be true\.\- A context\-dependent but explicit elicitation such as “Why?”, “How did you get 24?”, or “And what comes next?” can be true when the dialogue makes the requested mathematical response clear\.\- A mathematical yes/no or choice question can be included; exclude yes/no turns that only check readiness, understanding, or acknowledgement\.\- If the tutor poses a valid task but the next student turn does not attempt to address it, label the tutor turn false\.For every candidate tutor turn, provide a concise reason grounded in the tutor turn and the following student response\. Return exactly one JSON entry per candidate using this format:\{"turn n": \{"reasoning": "\.\.\.","is\_math\_subtask": true\},"turn n\+2": \{"reasoning": "\.\.\.","is\_math\_subtask": false\}\}Use the 1\-based turn index values from the JSON data and do not include non\-candidate turns\.Table 7:Prompt for Mathematical Subtask Identification
### G\.2Standalone\-Question Transformation

We show the prompt used for standalone question transformation in Table[8](https://arxiv.org/html/2610.19532#A7.T8)\. For each identified tutor subtask, the rewriting prompt includes the original problem and the dialogue, with the target tutor turn explicitly marked\. The model is instructed to use the context preceding that turn to resolve references to quantities, expressions, and concepts, and to recover the minimal information required for a self\-contained question\. It must preserve the mathematical demand of the tutor’s request without simplifying, elaborating, or otherwise changing the task\.

The student response immediately following the target tutor turn is included only as sealed reference context\. The model may use it to disambiguate which request in the tutor utterance the student attempted to answer, but it is explicitly prohibited from using the student’s answer, reasoning, wording, or response\-specific numerical information in the rewritten question\. This safeguard prevents the observed response from leaking into the task representation and its estimated difficulty\.

System Message:You are a math assessment item writer\. Rewrite one tutor\-posed mathematical subtask as a concise, standalone question in the same neutral style as the original multiple\-choice question\.Preserve the target subtask’s actual cognitive demand\. Do not solve it, change its difficulty, add a hint, add scaffolding, disclose an answer, or change an explanation/justification request into a yes/no question\. Preserve whether the task asks for a calculation, a solution step, recall of a fact or method, a choice of method or operation, an explanation or justification, identification or interpretation, or a report of an approach or specific point of confusion\.Assume the rewritten question will be shown without the original problem statement or source dialogue\. Use the original question and dialogue only to resolve references such as “it”, “that”, “the second term”, “this angle”, or an omitted expression, and to recover givens that the target tutor turn assumes\. Include the minimum context needed to make the rewritten task understandable on its own\. Do not import extra hints, worked reasoning, corrections, or answers from other turns\. If a diagram is essential, describe only the information needed to pose the same task without reading off or revealing its answer\.The immediately following student response is sealed reference material\. It is supplied only to disambiguate which tutor request the student attempted\. Never copy or paraphrase its answer, numbers, expressions, claims, choice, reasoning, mistake, uncertainty, or wording into the standalone question\. Do not use it to repair, complete, or solve the task\. Information may appear in the output only when it is independently established by the original question, the target tutor turn, or context preceding the response\.Output requirements:\- Output only the rewritten assessment item, with no preamble, commentary, reasoning, quotation marks, JSON, or Markdown fence\.\- The first characters must be exactlyQuestion:\.\- Do not mention a tutor, student, dialogue, conversation, turn, prior/previous statement, correctness, observed answer, following response, or rewriting\.\- Do not say “as discussed”, “as stated above”, “based on the previous step”, or use another backward reference\. Replace it with the necessary mathematical content\.\- UseAnswer A:,Answer B:, and so on only when the target tutor turn itself asks the student to choose among presented alternatives\.\- If the target asks a mathematical yes/no question without presenting answer options, keep it as a direct question and do not invent Yes/No options\.\- When options are genuinely required, place each option on its own line, begin withAnswer A:, continue alphabetically without gaps, preserve the actual alternatives, and never mark the correct one\.\- Preserve mathematical notation where useful\.The final text must be a question that could be shown directly to a student without access to the source dialogue or the original problem statement\.Table 8:Prompt for Standalone Mathematical Subtask Transformation
### G\.3Pedagogical Move Annotation

We show the prompts used to annotate the pedagogical moves*scaffold after struggle*and*push for rigor*in Table[9](https://arxiv.org/html/2610.19532#A7.T9)and Table[10](https://arxiv.org/html/2610.19532#A7.T10), respectively\.

### G\.4Prompt for Model

We show the prompt used for ICL\(kNN\) baseline with chain\-of\-thought reasoning in Table[11](https://arxiv.org/html/2610.19532#A7.T11)\.

System Message:You are a math teacher, and your task is to annotate relative task difficulty in mathematics tutoring dialogues\.The supplied candidate pairs have already passed three hard gates:1\. Both tutor turns are accepted mathematical subtasks\.2\. The later tutor turn occurs immediately after the earlier task’s student response, with no intervening raw dialogue turn\.3\. The student response to the earlier task is incorrect\.For every candidate pair \(earlier tutor turn, later tutor turn\), decide whether the later tutor task is genuinely easier scaffolding of the failed earlier task\. Markis\_case=trueonly when the later task reduces the mathematical demand by breaking down, narrowing, simplifying, concretizing, or adding helpful structure to the earlier failed task\. The later task must be meaningfully easier than the earlier task\.Markfalsefor a retry or paraphrase, a correction or feedback statement without an easier demand, an unrelated or new task, or ordinary progression to another step\. Do not use the correctness of the later student response to decide\. Do not add or remove candidates\. Return one decision for every supplied pair and no others, using the exact JSON format requested in the user prompt\.User prompt:Determine if the tutor is performing scaffolding after student struggle in two consecutive tutor turns in the dialogue: a subquestion is followed immediately in next tutor turn by a tutor task after the student’s incorrect response\. For a true pair, the follow\-up turn is treated as easier than the failed turn so the fixed order is difficulty\(later\) < difficulty\(earlier\)\.Question: Which of the following numbers is a factor of6060and a multiple of 6?Answer A: 3Answer B: 12Answer C: 20Answer D: 120Full Raw Dialogue:Raw turn 1 \(tutor\): HiRaw turn 2 \(student\): HeyRaw turn 3 \(tutor\) \[ACCEPTED SUBTASK\]: So remember we’re looking for a number in the 6 times table here\. That also goes into 60Raw turn 4 \(student\): So i think the awnser would be 3 if im correct \(Student Correctness: Incorrect\)Raw turn 5 \(tutor\) \[ACCEPTED SUBTASK\]: Ah that’s a factor of 60, but is it in the 6 times table?Raw turn 6 \(student\): ohh sorry silly me it must be 12 thenRaw turn 7 \(tutor\): Well corrected\. Yep it’s 12\. Good to go?Raw turn 8 \(student\): thanks\. yesSTRUCTURALLY ELIGIBLE CANDIDATES \[\(3, 5\)\]Return exactly this JSON shape and no other text:\{\\\{"pairs": \["pair": \[earlier\_tutor\_turn,later\_tutor\_turn\], "reasoning": "one sentence explanation", "is\_case": true or false\]\}\\\}Table 9:Prompt for Scaffold After Struggle Annotation in DialogueSystem Message:You are a math teacher, and your task is to annotate relative task difficulty in mathematics tutoring dialogues\.The supplied candidate pairs have already passed three hard gates:1\. Both tutor turns are accepted mathematical subtasks\.2\. the later tutor turn occurs immediately after the earlier task’s student response \(there is no intervening raw dialogue turn\); and3\. The student response to the earlier task is correct\.For every candidate pair \(earlier tutor turn, later tutor turn\), decide whether the later tutor task pushes for rigor by asking student for deeper mathematical reasoning, justification, explanation, verification, or greater precision beyond the student’s preceding response\. Markis\_case=truewhen the later task requires a reason, conceptual explanation, proof, defense, justification of why the answer or method is correct, or deeper calculation\.Mark false when the later task merely asks for the next solution step, a final answer, recall, identification, interpretation, selection of a method, confirmation, or a report of steps without a why/justification demand\. A prompt such as "How did you get that?" is true only when context makes it a request for explanatory reasoning, not merely a procedural report\. Despite the case name, do not require the later student response to be incorrect; its correctness is an outcome, not an inclusion gate\.User prompt:Determine if the tutor is pushing for rigor in two consequtive tutor turns in the dialogue: an subquestion is followed immediately in next tutor turn by a tutor task after the student’s correct response\. For a true pair, the follow\-up turn is asking student for deeper mathematical reasoning, justification, or explanation of previous turn so the fixed order is difficulty\(later\) \> difficulty\(earlier\)\.Question: Here is a function machine\. What should replace the star?The examples map 3 to 12 and 7 to 28\.Answer A:×4\\times 4Answer B:÷4\\div 4Answer C:\+9\+9Answer D:\+19\+19Full Raw Dialogue:Raw turn 1 \(tutor\): Hi\!Raw turn 2 \(student\): hiRaw turn 3 \(tutor\)\[ACCEPTED SUBTASK\]: So, how could we get from 3 to 12? WHat do we need to do?Raw turn 4 \(student\):×4\\times 4\(Student Correctness: Correct\)Raw turn 5 \(tutor\)\[ACCEPTED SUBTASK\]: Great\! And then check that it also works to get from 7 to 28Raw turn 6 \(student\): so would the answer be ARaw turn 7 \(tutor\): Yes\! Well done\! Bye for nowSTRUCTURALLY ELIGIBLE CANDIDATES \[\(3, 5\)\]Return exactly this JSON shape and no other text:\{\\\{"pairs": \["pair": \[earlier\_tutor\_turn,later\_tutor\_turn\], "reasoning": "one sentence explanation", "is\_case": true or false\]\}\\\}Table 10:Prompt for Push For Rigor Annotation in DialogueSystem Message:You are an experienced mathematics teacher\. Your job is to predict the difficulty of a mathematics multiple choice question\. Please follow these instructions carefully:\- You will be given a question, including its answer choices\. Use all of this information to predict the question’s difficulty\.\- You will also be given several example questions and their ground\-truth difficulties\. Use these to calibrate your prediction for the current question\.\- Difficulties are continuous real numbers and can be positive or negative\. A higher value means the question is more difficult\.\- First think step\-by\-step about the question’s difficulty based on its content and the examples\.\- Write your output with exactly this template: "Thinking: <brief step\-by\-step reasoning\>Difficulty: <real\-valued difficulty prediction\>"\. On the Difficulty line, output only one finite real number after the label, formatted with exactly two digits after the decimal point\.User prompt:\#\#\#\# Example 1 \#\#\#\#Question:<nearest\-neighbor question 1\>——Difficulty: <IRT difficulty 1\>…\\dots\#\#\#\# Example 4 \#\#\#\#Question:<nearest\-neighbor question 4\>——Difficulty: <IRT difficulty 4\>\#\#\#\# Current Question \#\#\#\#Question:<question\>——Table 11:Prompt for ICL \(kNN\) difficulty prediction