大语言模型辅导互动对学生表现的增量预测价值

# Incremental Predictive Value of LLM Tutoring Interactions for Student Performance
Source: [https://arxiv.org/html/2610.33488](https://arxiv.org/html/2610.33488)
CCS:Applied computing EducationCCS:Computing methodologies Machine learningZeyu DongNote:Corresponding authors\.email:[dongzeyu123@outlook\.com](mailto:[email protected])Affiliation:Crestwood Preparatory College,Toronto,Ontario,CanadaMikael Kwame Kuadaemail:[mikekuada12@gmail\.com](mailto:[email protected])Affiliation:Hammond High Magnet School,Hammond,Louisiana,United States,Asa Holley Buchakjian\-Tweedyemail:[asaholleybt@gmail\.com](mailto:[email protected])Affiliation:Spaulding High School,Rochester,New Hampshire,United StatesandIbrahem Aljabeaemail:[ialjab2@lsu\.edu](mailto:[email protected])Affiliation:Louisiana State University,Baton Rouge,Louisiana,United States

© none

###### Abstract\.

We evaluate whether semester\-level interaction features from a large language model tutor improve assignment\-score prediction beyond constant baselines\. StudyChat provides 16,851 student–assistant exchanges from two offerings of an undergraduate artificial intelligence course, with outcomes matched to 175 of 203 student\-semester records\. Six fitted estimators and training\-fold mean and median baselines share repeated nested cross\-validation\. After averaging each student’s held\-out predictions across ten repeats, Gradient Boosting achieves mean absolute error of 3\.023 grade points, versus 3\.138 for the median baseline\. The paired difference is−0\.114\-0\.114, with a conditional 95% bootstrap interval of\[−0\.245,0\.013\]\[\-0\.245,0\.013\]\. Representation changes and a semester\-specific median yield similarly small gains\. In the lowest observed\-score quartile, however, boosting’s error is 7\.30 points, versus 8\.07 for the median and 6\.67 for the mean; Ridge has the lowest fitted\-model error at 6\.37 despite its worse overall error of 3\.331\. Scores concentrate near the ceiling: 160 students \(91\.4%\) score above 90, and the lowest quartile contributes 62\.1% of boosting’s total absolute error\. This retrospective comparison shows how baseline choice and error distribution affect predictive value, informing evaluation of student\-support tools without establishing a learning effect\.

###### Keywords:

educational data mining, learning analytics, large language models, baseline evaluation, responsible AI

## 1\.Introduction

Conversational tutoring logs record how students seek help, offering a possible basis for feedback and instructor support\. Before such traces inform educational decisions, their predictive value must be separated from the performance of a simple baseline\. When scores concentrate near the ceiling, a constant prediction can achieve small average error while poorly representing lower\-scoring students\. Evaluating only overall accuracy can therefore obscure the errors most relevant to a support decision\.

We examine this distinction in StudyChat\([McNichols et al\., 2026](https://arxiv.org/html/2610.33488#bib.bib1)\), a research release with 16,851 exchanges from two offerings of one undergraduate artificial intelligence course\. Semester\-level features are available for 203 student\-semester records, of which 175 have assignment outcomes\. Among these matched records, 160 \(91\.4%\) score above 90\. The supervised sample therefore consists of 175 outcome\-bearing records, not 16,851 independent learning outcomes\.

StudyChat’s original assessment\-specific regressions add usage and dialogue\-act counts to prior assessment scores\([McNichols et al\., 2026](https://arxiv.org/html/2610.33488#bib.bib1)\)\. Changes in explained variance in that setting do not establish held\-out prediction of a semester aggregate or describe how prediction errors are distributed\. We address that evaluation gap through two questions: do semester\-level interaction summaries reduce held\-out assignment\-score error relative to training\-fold constants, and do baseline and estimator comparisons change across observed score levels? This is a complementary analysis of the same corpus, not a test of added value beyond prior academic performance\.

The main finding is a contrast between small aggregate gains and uneven error\. Gradient Boosting’s student\-averaged mean absolute error \(MAE\) is 3\.023 points, versus 3\.138 for a median baseline, with a conditional 95% interval for the difference crossing zero\. In the lowest observed\-score quartile, boosting outperforms the median but not the mean baseline; Ridge has the smallest fitted\-model error there despite its worse overall rank\. The baseline and evaluation population both affect the apparent predictive value\.

The analysis provides three connected contributions:

- •A repeated nested comparison of six fitted estimators and two constant baselines quantifies the incremental value of 14 interaction features\.
- •All\-model outcome\-stratum contrasts identify where aggregate rankings change and separate learner performance from baseline choice\.
- •Representation, cohort\-transfer, and held\-out attribution checks examine the scope of the predictive result and its interpretation\.

For learning analytics, these results establish evaluation requirements before predictions guide support; no support intervention is tested\. Full\-semester features make this a retrospective study\. Prospective evaluation requires verified assessment\-relative cutoffs\.

![The pipeline separates data preparation, feature construction, retrospective prediction, exploratory clustering, and held-out model diagnostics.](https://arxiv.org/html/2610.33488v1/fig0_pipeline.png)Figure 1\.Study workflow\. Student\-semester aggregation precedes repeated nested prediction; imputation and applicable scaling are learned within training partitions\. Baseline contrasts assess predictive value\. Clustering and attributions are separate exploratory diagnostics, not validated learning styles or intervention rules\. Small inset plots are schematic, not study results\.The pipeline separates data preparation, feature construction, retrospective prediction, exploratory clustering, and held\-out model diagnostics\.Figure[1](https://arxiv.org/html/2610.33488#acmlabel1)distinguishes the supervised evaluation from the exploratory descriptions of interaction patterns\.

## 2\.Related Work

#### AI assistance and educational outcomes\.

Research on AI in education spans tutoring, feedback, assessment, and learner support\. Reviews describe technical adoption alongside uneven educator involvement in evaluation\([Zawacki\-Richter et al\., 2019](https://arxiv.org/html/2610.33488#bib.bib5)\)\. Kasneci et al\. discuss opportunities and risks of language\-model assistance, including the need for pedagogical oversight\([Kasneci et al\., 2023](https://arxiv.org/html/2610.33488#bib.bib6)\)\. Bastani et al\. compare tutor designs with and without learning safeguards in high\-school mathematics\([Bastani et al\., 2025](https://arxiv.org/html/2610.33488#bib.bib7)\)\. Lau and Guo interview programming instructors about their planned responses to AI coding tools\([Lau and Guo, 2023](https://arxiv.org/html/2610.33488#bib.bib8)\)\. These studies distinguish educational practice and learning effects from the prediction of observed assignment scores addressed here\.

#### Learning analytics and the evaluation gap\.

Learning analytics relates traces to educational processes and outcomes\([Siemens, 2013](https://arxiv.org/html/2610.33488#bib.bib2);[Ferguson, 2012](https://arxiv.org/html/2610.33488#bib.bib3)\), while educational data mining uses submissions, assessments, and activity records for prediction\([Romero and Ventura, 2020](https://arxiv.org/html/2610.33488#bib.bib4)\)\. StudyChat supplies dialogues, generated dialogue\-act labels, and normalized outcomes\([McNichols et al\., 2026](https://arxiv.org/html/2610.33488#bib.bib1)\)\. Its assessment\-specific regressions add utterance or dialogue\-act counts to a prior\-score baseline and examine explained variance and coefficients\. We instead compare held\-out MAE for a semester\-level aggregate on shared nested splits and examine errors across observed outcomes\. The different target, information set, and error estimand distinguish the two analyses; using the same corpus does not provide independent\-cohort replication\.

#### Prediction and explanation\.

Nested cross\-validation separates parameter search from outer evaluation\([Varma and Simon, 2006](https://arxiv.org/html/2610.33488#bib.bib11);[Cawley and Talbot, 2010](https://arxiv.org/html/2610.33488#bib.bib10)\)\. Repeated folds remain dependent, so we distinguish descriptive fold averages from fixed\-fit student\-bootstrap contrasts\([Efron, 1979](https://arxiv.org/html/2610.33488#bib.bib19)\)\. SHAP decomposes a fitted prediction into additive contributions\([Lundberg and Lee, 2017](https://arxiv.org/html/2610.33488#bib.bib9)\); numerical additivity alone does not establish accuracy or educational validity\. These distinctions matter when explanations might otherwise be read as recommendations about how a student should learn\.

## 3\.Data and Evaluation

### 3\.1\.Cohort, Target, and Information Available

StudyChat covers COMPSCI 383 at the University of Massachusetts Amherst in Fall 2024 and Spring 2025\([McNichols et al\., 2026](https://arxiv.org/html/2610.33488#bib.bib1)\)\. Joining on student and semester matches 65 of 84 Fall records and 110 of 119 Spring records to nonmissing outcomes\. Of 181 grade records, six lack matching behavioral records; 28 behavioral records lack outcomes\. Missing outcomes are not imputed\.

The target isyi=100​llmiy\_\{i\}=100\\,\\texttt\{llm\}\_\{i\}, the released points\-weighted aggregate of assignments a1–a7, rather than an exam or overall course grade\. Assignment maxima are\(30,100,55,61,65,113,101\)\(30,100,55,61,65,113,101\)in Fall and\(31,100,55,90,105,113,135\)\(31,100,55,90,105,113,135\)in Spring, with nominal totals 525 and 629\. The inspected release metadata does not document missing/unsubmitted\-item treatment\. We inherit the nonmissing aggregate rather than reconstructing it\. Matched scores have mean 94\.77, median 96\.18, and sample standard deviation 6\.12; six are below 80\. The pooled evaluation gives each record equal weight, with 65/175 from Fall and 110/175 from Spring\.

Preparation parses millisecond timestamps, aggregates student\-semester records, and checks unique keys and joins; raw\-exchange deduplication is not performed\. The primary representation contains 14 operational predictors \(Table[1](https://arxiv.org/html/2610.33488#S3.T1)\), excluding conversation persistence \(P/C−1P/C\-1\) from the original 15 features, wherePPcounts exchanges andCCcounts chats\. Identifiers and semester are not learned\-model predictors\. Contextual\-question and writing\-request ratios use GPT\-4\.1\-generated dialogue\-act annotations, stored asclarification\_ratioandsolution\_request\_ratio\. These are recorded language categories, not validated measures of motivation, metacognition, or learning styles\.

Table 1\.Primary features\.PP: exchanges;CC: chats;WW: prompt tokens;UU: distinct prompt tokens\. Tokens match lowercase\[a\-z0\-9’\]\+\. Percentages use the 0–100 scale\.The example pattern matches “examples”, “e\.g\.”, “for instance”, “show me”, “demonstrate”, “demo”, or “sample”\. Long legacy chat spans and one large tokens\-per\-segment value are retained and examined through sensitivity analyses\. All features summarize the full observed semester\. Outcomes do not enter feature construction, but activity may occur during or after target assignments\. Official deadlines and grade\-availability times are unavailable for a verified forecast cutoff\.

### 3\.2\.Models and Nested Evaluation

Mean and median dummy regressors are compared with Ridge, Elastic Net, Random Forest\([Breiman, 2001](https://arxiv.org/html/2610.33488#bib.bib14)\), Extra Trees\([Geurts et al\., 2006](https://arxiv.org/html/2610.33488#bib.bib24)\), Gradient Boosting\([Friedman, 2001](https://arxiv.org/html/2610.33488#bib.bib13)\), and a Keras multilayer perceptron \(MLP\)\. Non\-neural models use scikit\-learn\([Pedregosa et al\., 2011](https://arxiv.org/html/2610.33488#bib.bib12)\)\. All eight estimators share five outer folds repeated ten times \(seed 42\)\. Each outer\-training partition has four shuffled inner folds, selecting parameters by validation MAE with seed42\+1000​r\+f42\+1000r\+ffor zero\-based repeatrrand foldff\. Search uses 40 randomized candidates where applicable, all 25 Ridge values, and eight neural candidates\. Each dummy predicts the mean or median outcome of its outer\-training partition; held\-out outcomes do not determine those predictions\.

Each inner\-training fit and outer refit learns median imputation from its training partition\. Ridge, Elastic Net, and MLP also applylog⁡\(1\+x\)\\log\(1\+x\)to prompt/conversation counts, depth, prompt/response lengths, span, questions per chat, tokens per segment, and active days, then fit standardization\. Trees retain unlogged features\.

Table 2\.Available search specification\. Random Forest and Extra Trees use 500 trees\. Resolved budgets and selected parameters are saved; the full historical sampled\-candidate trace and dirty source are not archived\. Boosting’s search definition is available in later versioned source and agrees with selected parameters, not a certified original source snapshot\.In Table[2](https://arxiv.org/html/2610.33488#S3.T2), numeric feature\-sampling choices are fractions of the 14 predictors;sqrtselects their square\-root count\.

The historical MLP uses a PyTorch backend, ReLU hidden layers, linear output, dropout after the first hidden layer, Huber loss \(δ=3\\delta=3\), Adam, batch size 16, and at most 500 epochs\. Early stopping uses an internal 20% split, patience 35, minimum improvement10−410^\{\-4\}, and restored best weights\. The learning rate halves after 12 stagnant epochs, down to10−510^\{\-5\}\. Search uses one seed; outer refits average seeds 42, 43, and 44\. Preprocessing is fitted before the internal early\-stopping split, so those validation rows participate in fitted transformations; outer held\-out rows do not\.

### 3\.3\.Estimands and Uncertainty

Forn=175n=175matched records, lety^i​r\\hat\{y\}\_\{ir\}be recordii’s held\-out prediction in repeatrr\. We report the error after averaging itsR=10R=10predictions:

\(1\)y^¯i=R−1​∑ry^i​r,MAEstudent=n−1​∑i\|yi−y^¯i\|\.\\bar\{\\hat\{y\}\}\_\{i\}=R^\{\-1\}\\sum\_\{r\}\\hat\{y\}\_\{ir\},\\qquad\\mathrm\{MAE\}\_\{\\mathrm\{student\}\}=n^\{\-1\}\\sum\_\{i\}\|y\_\{i\}\-\\bar\{\\hat\{y\}\}\_\{i\}\|\.This repeated\-cross\-validation aggregate differs from a single deployment fit\. We also report mean fold MAE, root mean squared error \(RMSE\), andR2R^\{2\}\. For training\-fold constant baselinebb, the paired contrast is

\(2\)Δb=n−1​∑i\(\|yi−y^¯i,model\|−\|yi−y^¯i,b\|\)\.\\Delta\_\{b\}=n^\{\-1\}\\sum\_\{i\}\\bigl\(\|y\_\{i\}\-\\bar\{\\hat\{y\}\}\_\{i,\\mathrm\{model\}\}\|\-\|y\_\{i\}\-\\bar\{\\hat\{y\}\}\_\{i,b\}\|\\bigr\)\.Negative differences favor the learner\. The median is the lower\-error constant under both primary MAE summaries\. Paired 95% percentile intervals use 5,000 student resamples; marginal MAE intervals use 2,000\. Predictions stay fixed: intervals do not include retraining, tuning, comparator/model selection, or multiplicity adjustments\. The 0\.5\-point reduction reference is an operational reporting convention, not a preregistered or validated educational threshold\.

Post\-hoc outcome quartiles usepandas\.qcut, right\-closed with the minimum included and ties kept together\. Rounded cutpoints are 94\.286, 96\.184, and 97\.905; counts are 45, 44, 44, and 42\. Within\-stratum intervals use 5,000 paired resamples \(seed 42\), conditional on the same averaged predictions and unadjusted across models/strata\.

## 4\.Results

### 4\.1\.Incremental Prediction

Table[3](https://arxiv.org/html/2610.33488#S4.T3)reports the complete primary comparison\. MLP has the smallest mean fold MAE \(3\.061\), followed by boosting \(3\.062\) and Random Forest \(3\.071\)\. Their student\-averaged MAEs are 3\.027, 3\.023, and 3\.038, versus 3\.138 for the median\. Ridge, Elastic Net, and Extra Trees have larger point errors; all six fitted models have negative mean foldR2R^\{2\}\.

Table 3\.All eight estimators on 50 shared folds and 175 matched records\. Student MAE scores ten\-repeat\-averaged predictions\. Paired differences use the median dummy and conditional 95% percentile intervals\. All six learner intervals include zero\. All displayed metrics are rounded once from their underlying values to three decimals\.Boosting’s point reduction in student MAE is 0\.1144 points\. Its signed model\-minus\-median difference has conditional 95% interval\[−0\.2451,0\.0125\]\[\-0\.2451,0\.0125\]\. MLP and Random Forest have point reductions of 0\.1107 and 0\.0998 points, respectively; their intervals are\[−0\.2546,0\.0290\]\[\-0\.2546,0\.0290\]and\[−0\.2434,0\.0428\]\[\-0\.2434,0\.0428\]\. All three intervals include zero, and the point gains remain below 0\.5 points\.

The recorded diagnostic\-model rule first retains learners that beat the best dummy in fold MAE, then favors a lower\-complexity family within 0\.5 points of the lowest error\. Boosting and Random Forest share the tree\-family rank; boosting’s lower error breaks that tie, while MLP ranks as more complex\. This outer\-result\-dependent choice makes subsequent selected\-model diagnostics exploratory\. The predictive claim rests on all models\.

### 4\.2\.Baseline Dependence and Error Distribution

Boosting’s MAEs across Q1–Q4 are 7\.30, 0\.91, 1\.04, and 2\.74\. The 45 Q1 students contribute 62\.1% of its total absolute error\. In Q1, boosting’s difference from the median is−0\.767\-0\.767\[−1\.091,−0\.457\]\[\-1\.091,\-0\.457\], but the difference from the mean is\+0\.629\+0\.629\[0\.294,0\.944\]\[0\.294,0\.944\]\. The mean’s lower prediction level is advantageous there despite its worse overall MAE of 3\.440\.

Table 4\.Student\-averaged MAE by observed\-score quartile for all estimators\. Counts are 45, 44, 44, and 42; their weighted errors reproduce overall MAE\.Ridge has the smallest fitted\-model Q1 error, 6\.37, but its mean\-relative difference−0\.301\-0\.301\[−0\.694,0\.066\]\[\-0\.694,0\.066\]includes zero\. All six learners’ Q1 intervals lie below zero against the median; none lies wholly below zero against the mean \(Figure[2](https://arxiv.org/html/2610.33488#acmlabel2)\)\. Apparent improvement against one constant therefore does not alone establish added lower\-tail information\. Q1 is the lowest quarter of a high\-scoring cohort, not a prospective risk group\.

Figure 2\.Paired Q1 contrasts for all six learners on the same 45 records\. Circles compare with the median; squares compare with the mean\. Lines are conditional 95% percentile intervals from 5,000 paired within\-stratum draws\. Negative differences favor the learner\. Strata are defined after observing outcomes; intervals are not multiplicity\-adjusted\.All model\-minus\-median intervals are negative\. Against the mean, three intervals cross zero and three are positive\.For quartileqqwithnqn\_\{q\}students, letMAEq\\mathrm\{MAE\}\_\{q\}denote its mean absolute error andcqc\_\{q\}its share of total absolute error\. Then

\(3\)MAEstudent\\displaystyle\\mathrm\{MAE\}\_\{\\mathrm\{student\}\}=∑q=14nqn​MAEq,\\displaystyle=\\sum\_\{q=1\}^\{4\}\\frac\{n\_\{q\}\}\{n\}\\mathrm\{MAE\}\_\{q\},cq\\displaystyle c\_\{q\}=nq​MAEqn​MAEstudent\.\\displaystyle=\\frac\{n\_\{q\}\\mathrm\{MAE\}\_\{q\}\}\{n\\,\\mathrm\{MAE\}\_\{\\mathrm\{student\}\}\}\.Thus, Q1 contains 25\.7% of the matched records but contributes 62\.1% of boosting’s absolute error\. Improving a model’s error in one stratum can be offset by larger errors elsewhere; overall MAE alone does not show that trade\-off\.

### 4\.3\.Representation and Cohort Sensitivity

Boosting student MAE is 3\.024 with all 15 features and 3\.021 with robust duration/segmentation definitions\. The latter use a 30\-minute inactivity threshold, a 60\-minute within\-session gap cap, and code stripping in sentence segmentation while retaining the token numerator\. Removing the three request\-type ratios gives 3\.070; removing prompt/conversation counts and active days gives 3\.079; removing follow\-up ratio gives 3\.023\. All five use 50 folds, and all intervals against their corresponding median dummies include zero\. A training\-fold median for each semester gives student MAE 3\.134 and fold MAE 3\.138\. Boosting’s student\-MAE difference is−0\.111\-0\.111\[−0\.243,0\.021\]\[\-0\.243,0\.021\]\(5,000 paired draws, seed 42\)\.

Transfer fits preprocessing, four\-fold inner tuning, and the final estimator on the source semester only\. It retains candidate budgets, direction seeds 50042/50043, and one neural search seed with three final members\. Fall\-to\-Spring trains on 65 and tests on 110 records; reverse transfer exchanges those counts \(Table[5](https://arxiv.org/html/2610.33488#S4.T5)\)\. Boosting’s gains over the better constant are 0\.113 and 0\.031 points\. Paired uncertainty is unavailable because transfer prediction rows were not archived\.

Table 5\.Transfer MAE within the same course\. The better constant is the mean for Fall\-to\-Spring and median for Spring\-to\-Fall\. These are point estimates\.Alternate\-target runs retain the 175 students, 14 legacy predictors, 50 outer folds, and four\-fold inner MAE selection with 40 candidates where applicable\. For the exam aggregate, the median MAE is 6\.946 versus mean 7\.098; Random Forest, Ridge, and boosting give 6\.909, 6\.939, and 6\.958\. Their paired median\-relative intervals all include zero:\[−0\.371,0\.304\]\[\-0\.371,0\.304\],\[−0\.323,0\.309\]\[\-0\.323,0\.309\], and\[−0\.251,0\.288\]\[\-0\.251,0\.288\]\. For the third exam, median MAE is 8\.287 versus mean 8\.612; boosting, Random Forest, and Ridge give 8\.387, 8\.467, and 8\.625, with intervals\[−0\.024,0\.225\]\[\-0\.024,0\.225\],\[−0\.025,0\.387\]\[\-0\.025,0\.387\], and\[−0\.063,0\.733\]\[\-0\.063,0\.733\]\. These are full\-semester retrospective checks, not prospective exam forecasts\.

### 4\.4\.Associations and Explanation Fidelity

Four of 14 Pearson tests pass Benjamini–Hochbergq<0\.05q<0\.05\([Benjamini and Hochberg, 1995](https://arxiv.org/html/2610.33488#bib.bib23)\): contextual\-question ratio \(r=−0\.329r=\-0\.329\), active days \(0\.2480\.248\), vocabulary diversity \(−0\.199\-0\.199\), and conversation count \(0\.1970\.197\)\. Adjustment covers that screen only\. Removing six scores below 80 changes the contextual\-question correlation to−0\.173\-0\.173; its Spearman coefficient is−0\.162\-0\.162\. Thus, a nominally prominent association is sensitive to the lower\-score tail\. It agrees in direction with StudyChat’s Assignment\-5 result on the same corpus\([McNichols et al\., 2026](https://arxiv.org/html/2610.33488#bib.bib1)\)\. Dialogue depth and follow\-up ratio also have absolute Spearman correlation 0\.993\.

Table 6\.The four associations retained by the 14\-test Pearson screen on 175 records\. Adjustedqqvalues apply only to that screen\. Pearson intervals use 2,000 student resamples; Spearman coefficients provide a complementary rank\-based summary\. Vocabulary diversity’s interval includes zero despite its adjusted parametric test, illustrating the different assumptions\.Table[6](https://arxiv.org/html/2610.33488#S4.T6)reports both coefficient types and uncertainty\.

Tree\-path\-dependent TreeSHAP explains transformed held\-out boosting rows asfm​\(𝐱i\)=ϕ0​m\+∑j=114ϕi​j​mf\_\{m\}\(\\mathbf\{x\}\_\{i\}\)=\\phi\_\{0m\}\+\\sum\_\{j=1\}^\{14\}\\phi\_\{ijm\}\. Averaging each student’s ten baseline/contribution vectors preserves the decomposition of the averaged prediction\. Refit predictions match saved values within1\.42×10−141\.42\\times 10^\{\-14\}points; the maximum additivity residual is5\.68×10−145\.68\\times 10^\{\-14\}\. These are numerical checks on evaluated inputs, not certification of the historical source or educational validity\. Mean absolute contributions lead with contextual questions \(0\.332 points\), conversation count \(0\.258\), and questions per chat \(0\.150\)\. Yet median absolute outcome error is 1\.69 and the maximum is 40\.93\. Precise explanation of a fitted prediction does not imply an accurate prediction, and contribution magnitudes do not identify causal learning mechanisms\.

Figure 3\.Held\-out attribution magnitudes for 14 predictors\. Each bar is the mean absolute ten\-repeat\-averaged contribution over 175 records; conditional 95% percentile intervals use 2,000 student resamples\. The ordering describes the selected boosting function under tree\-path\-dependent TreeSHAP and correlated inputs\.Contextual questions, conversation count, and questions per chat have the largest absolute SHAP contributions\.Figure[3](https://arxiv.org/html/2610.33488#acmlabel3)shows all predictors rather than only the largest contributions\.

### 4\.5\.Exploratory Interaction Profiles

The companion clustering analysis uses eight standardized dialogue\-act percentages for all 203 records, separately from supervised predictors\. K\-means\([Lloyd, 1982](https://arxiv.org/html/2610.33488#bib.bib15)\)and Ward clustering describe request mixtures\. Fork=2,…,6k=2,\\ldots,6, silhouette\([Rousseeuw, 1987](https://arxiv.org/html/2610.33488#bib.bib16)\)selects six; extending to twelve selects eight, while Davies–Bouldin\([Davies and Bouldin, 1979](https://arxiv.org/html/2610.33488#bib.bib17)\)selects ten\. All evaluated partitions above six contain singletons\.

Subset sensitivity samples 203 indices with replacement 200 times, discards repetitions, and refits on distinct rows with fixed scaling\. All original rows are assigned to the nearest refitted centroid; adjusted Rand index \(ARI\)\([Hubert and Arabie, 1985](https://arxiv.org/html/2610.33488#bib.bib18)\)compares assignments with the full\-cohort reference\. Reference/subset fits use 20 initializations, versus 100 in selection\. This is not a multiplicity\-preserving bootstrap\. At the operational median\-ARI cutoff 0\.60,k=2,9,10k=2,9,10pass\. The finer two contain singletons;k=2k=2has a wide central 90% range,\[0\.235,1\.000\]\[0\.235,1\.000\]\.

The proportions are compositional\([Aitchison, 1982](https://arxiv.org/html/2610.33488#bib.bib21);[Pal and Heumann, 2022](https://arxiv.org/html/2610.33488#bib.bib22)\)\. A separate gap analysis\([Tibshirani et al\., 2001](https://arxiv.org/html/2610.33488#bib.bib20)\)selects two by the one\-standard\-error rule, but its uniform\-reference draws do not preserve the sum constraint\. For the original six groups, Welch \(p=0\.158p=0\.158\) and Kruskal–Wallis \(p=0\.138p=0\.138\) leave location evidence inconclusive; median\-centered Levene \(p=0\.002783p=0\.002783\) indicates unequal dispersion\. These results do not establish stable learning styles or validate adaptive instruction by cluster\.

## 5\.Educational Implications and Scope

### 5\.1\.From Prediction to Student Support

The study supplies an empirical check before interaction traces are used to guide student support\. The observed small gains provide little evidence for allocating feedback or instructor attention from this representation alone\. Meanwhile, Q1 error concentration shows why population\-average error is insufficient for evaluating a tool intended to assist lower\-performing students\. A support system should be assessed for its intended decision and population, not adopted solely because its average score error is small\.

For responsible use, educators should define the intended decision, the information available when it is made, and the costs of missed support and unnecessary alerts before fitting a model\. Those choices determine which baselines and evaluation populations are relevant\. Our result concerns prediction beyond constants; prior academic performance remains an important untested comparator\. It also does not imply that asking contextual questions harms learning\. Such questions may express confusion, task requirements, productive help seeking, or annotation error\.

Interaction measurement is distinct from learning\-process measurement\. Counts and ratios discard dialogue order, task semantics, response correctness, and verified timing\. The study does not directly measure motivation, emotion, or self\-regulation\. An educator\-reviewed application could investigate these dimensions, but the present features should not be renamed as psychological constructs\. Likewise, outcome\-defined error heterogeneity is not a demographic fairness audit\. Absence of protected\-group evaluation precludes a claim of bias\-free predictions\.

### 5\.2\.Selection, Ethics, and Reproducibility

Outcome matching is 77\.4% in Fall and 92\.4% in Spring\. A saved exploratory comparison finds longer prompts among matched records: standardized mean difference 0\.463, with 95% percentile interval\[0\.231,0\.672\]\[0\.231,0\.672\]\. It uses 1,000 separate\-group draws and standardizer\(sM2\+sU2\)/2\\sqrt\{\(s\_\{M\}^\{2\}\+s\_\{U\}^\{2\}\)/2\}from matched \(MM\) and unmatched \(UU\) group sample variances\. A shared PCG64 generator seeded at 0 preserves row order, draws matched before unmatched samples, and processes prompt count, conversation count, depth, then prompt length without resetting\. The matched cohort differs from the behavioral sample; unavailable outcomes leave the direction of selection bias unresolved\.

Both offerings are from one course\. Fixed\-fit uncertainty excludes retraining/search variation, and intervals crossing zero are not an equivalence test\. Generated act labels lack a supplied human construct\-validation study\. Prior knowledge and outside study or AI use are unmeasured\.

This secondary analysis uses the opt\-in, de\-identified research release\. Source authors report consent\([McNichols et al\., 2026](https://arxiv.org/html/2610.33488#bib.bib1)\); no participants were newly recruited, contacted, or assigned an intervention\. We do not claim a new institutional approval\. Reporting uses aggregates rather than participant text, identifiers, or row\-level outcomes\.

Saved predictions and resolved configuration support numerical replay of the complete eight\-estimator run\. The original dirty training source was not archived, and the recorded commit lacks boosting\. A completion manifest reports Keras 3\.15\.1, whereas a separate environment snapshot records it as unavailable\. Later versioned code documents search design, not exact historical implementation\. These limitations are distinct from numerical agreement with the saved outputs\. Replay checks all 400 model\-fold target/semester orders; raw fold values are averaged before rounding\.

A separate source\-frozen check retained the same cohort, splits, and search budgets\. In 90 minutes, both constants, Ridge, Elastic Net, and Random Forest completed 50 folds each and matched point errors and paired intervals to four decimals\. Extra Trees was interrupted; boosting and cohort MLP were not reached\. This partial same\-cohort check does not replace the historical comparison or provide independent validation\.

### 5\.3\.Future Work

Future evaluation should first fix prediction times before assignment deadlines and restrict features to information available by each cutoff\. Comparing interaction summaries with prior assessment and course\-context predictors on held\-out courses would test added value beyond existing academic records\. Richer representations could retain dialogue sequence, task content, and response correctness, accompanied by human validation of dialogue\-act labels and evaluation of missing outcomes and retraining variation\.

Educators and learners should then help define actionable support and acceptable error costs\. A controlled study of instructor\-reviewed support could assess whether earlier feedback improves subsequent unaided performance, retention, or self\-regulated learning\. Prespecified measures of missed support, unnecessary alerts, and instructor workload would distinguish predictive improvements from educational benefit\. None of these intervention outcomes is established by the present retrospective evaluation\.

## 6\.Conclusion

Semester\-level StudyChat features yield small overall assignment\-score gains over training\-fold medians, while baseline and estimator rankings vary across observed outcomes\. Boosting’s overall gain is 0\.114 points with a conditional interval crossing zero; its Q1 advantage over the median reverses against the mean\. All\-model reporting and explicit error populations are therefore necessary to interpret incremental predictive value in this cohort\. For learning analytics, the contribution is a baseline\-referenced assessment of this representation and the students who carry its error\. It informs the design and evaluation of support tools without demonstrating a learning effect\. Prospective cross\-course prediction and an educator\-informed intervention study are the next tests of practical educational value\.

## References

- Aitchison \(1982\)J\. AitchisonThe statistical analysis of compositional data\.Journal of the Royal Statistical Society: Series B \(Methodological\)44\(2\),pp\. 139–160\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1982.tb01195.x)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p3.1)\.
- Bastaniet al\.\(2025\)H\. Bastani, O\. Bastani, A\. Sungu, H\. Ge, Ö\. Kabakcı, and R\. MarimanGenerative AI without guardrails can harm learning: evidence from high school mathematics\.Proceedings of the National Academy of Sciences122\(26\),pp\. e2422633122\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2422633122)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px1.p1.1)\.
- Benjamini and Hochberg \(1995\)Y\. Benjamini and Y\. HochbergControlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B \(Methodological\)57\(1\),pp\. 289–300\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1995.tb02031.x)Cited by:[§4\.4](https://arxiv.org/html/2610.33488#S4.SS4.p1.1)\.
- Breiman \(2001\)L\. BreimanRandom forests\.Machine Learning45,pp\. 5–32\.External Links:[Document](https://dx.doi.org/10.1023/A%3A1010933404324)Cited by:[§3\.2](https://arxiv.org/html/2610.33488#S3.SS2.p1.1)\.
- Cawley and Talbot \(2010\)G\. C\. Cawley and N\. L\. C\. TalbotOn over\-fitting in model selection and subsequent selection bias in performance evaluation\.Journal of Machine Learning Research11,pp\. 2079–2107\.Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px3.p1.1)\.
- Davies and Bouldin \(1979\)D\. L\. Davies and D\. W\. BouldinA cluster separation measure\.IEEE Transactions on Pattern Analysis and Machine IntelligencePAMI\-1\(2\),pp\. 224–227\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.1979.4766909)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p1.1)\.
- Efron \(1979\)B\. EfronBootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176344552)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px3.p1.1)\.
- Ferguson \(2012\)R\. FergusonLearning analytics: drivers, developments and challenges\.International Journal of Technology Enhanced Learning4\(5/6\),pp\. 304–317\.External Links:[Document](https://dx.doi.org/10.1504/IJTEL.2012.051816)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px2.p1.1)\.
- Friedman \(2001\)J\. H\. FriedmanGreedy function approximation: a gradient boosting machine\.The Annals of Statistics29\(5\),pp\. 1189–1232\.External Links:[Document](https://dx.doi.org/10.1214/aos/1013203451)Cited by:[§3\.2](https://arxiv.org/html/2610.33488#S3.SS2.p1.1)\.
- Geurtset al\.\(2006\)P\. Geurts, D\. Ernst, and L\. WehenkelExtremely randomized trees\.Machine Learning63\(1\),pp\. 3–42\.External Links:[Document](https://dx.doi.org/10.1007/s10994-006-6226-1)Cited by:[§3\.2](https://arxiv.org/html/2610.33488#S3.SS2.p1.1)\.
- Hubert and Arabie \(1985\)L\. Hubert and P\. ArabieComparing partitions\.Journal of Classification2,pp\. 193–218\.External Links:[Document](https://dx.doi.org/10.1007/BF01908075)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p2.1)\.
- Kasneciet al\.\(2023\)E\. Kasneci, K\. Sessler, S\. Küchemann, M\. Bannert, D\. Dementieva, F\. Fischer, U\. Gasser, G\. Groh, S\. Günnemann, E\. Hüllermeier, S\. Krusche, G\. Kutyniok, T\. Michaeli, C\. Nerdel, J\. Pfeffer, O\. Poquet, M\. Sailer, A\. Schmidt, T\. Seidel, M\. Stadler, J\. Weller, J\. Kuhn, and G\. KasneciChatGPT for good? on opportunities and challenges of large language models for education\.Learning and Individual Differences103,pp\. 102274\.External Links:[Document](https://dx.doi.org/10.1016/j.lindif.2023.102274)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px1.p1.1)\.
- Lau and Guo \(2023\)S\. Lau and P\. J\. GuoFrom “ban it till we understand it” to “resistance is futile”: how university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot\.InProceedings of the 2023 ACM Conference on International Computing Education Research V\.1,pp\. 106–121\.External Links:[Document](https://dx.doi.org/10.1145/3568813.3600138)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px1.p1.1)\.
- Lloyd \(1982\)S\. P\. LloydLeast squares quantization in PCM\.IEEE Transactions on Information Theory28\(2\),pp\. 129–137\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1982.1056489)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p1.1)\.
- Lundberg and Lee \(2017\)S\. M\. Lundberg and S\. LeeA unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems 30,pp\. 4765–4774\.Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px3.p1.1)\.
- McNicholset al\.\(2026\)H\. McNichols, F\. Ikram, and A\. LanThe StudyChat dataset: analyzing student dialogues with ChatGPT in an artificial intelligence course\.InLAK26: 16th International Learning Analytics and Knowledge Conference,pp\. 53–63\.External Links:[Document](https://dx.doi.org/10.1145/3785022.3785029)Cited by:[§1](https://arxiv.org/html/2610.33488#S1.p2.1),[§1](https://arxiv.org/html/2610.33488#S1.p3.1),[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.33488#S3.SS1.p1.1),[§4\.4](https://arxiv.org/html/2610.33488#S4.SS4.p1.1),[§5\.2](https://arxiv.org/html/2610.33488#S5.SS2.p3.1)\.
- Pal and Heumann \(2022\)S\. Pal and C\. HeumannClustering compositional data using Dirichlet mixture model\.PLOS ONE17\(5\),pp\. e0268438\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0268438)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p3.1)\.
- Pedregosaet al\.\(2011\)F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and É\. DuchesnayScikit\-learn: machine learning in python\.Journal of Machine Learning Research12,pp\. 2825–2830\.Cited by:[§3\.2](https://arxiv.org/html/2610.33488#S3.SS2.p1.1)\.
- Romero and Ventura \(2020\)C\. Romero and S\. VenturaEducational data mining and learning analytics: an updated survey\.WIREs Data Mining and Knowledge Discovery10\(3\),pp\. e1355\.External Links:[Document](https://dx.doi.org/10.1002/widm.1355)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px2.p1.1)\.
- Rousseeuw \(1987\)P\. J\. RousseeuwSilhouettes: a graphical aid to the interpretation and validation of cluster analysis\.Journal of Computational and Applied Mathematics20,pp\. 53–65\.External Links:[Document](https://dx.doi.org/10.1016/0377-0427%2887%2990125-7)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p1.1)\.
- Siemens \(2013\)G\. SiemensLearning analytics: the emergence of a discipline\.American Behavioral Scientist57\(10\),pp\. 1380–1400\.External Links:[Document](https://dx.doi.org/10.1177/0002764213498851)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px2.p1.1)\.
- Tibshiraniet al\.\(2001\)R\. Tibshirani, G\. Walther, and T\. HastieEstimating the number of clusters in a data set via the gap statistic\.Journal of the Royal Statistical Society: Series B \(Statistical Methodology\)63\(2\),pp\. 411–423\.External Links:[Document](https://dx.doi.org/10.1111/1467-9868.00293)Cited by:[§4\.5](https://arxiv.org/html/2610.33488#S4.SS5.p3.1)\.
- Varma and Simon \(2006\)S\. Varma and R\. SimonBias in error estimation when using cross\-validation for model selection\.BMC Bioinformatics7,pp\. 91\.External Links:[Document](https://dx.doi.org/10.1186/1471-2105-7-91)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px3.p1.1)\.
- Zawacki\-Richteret al\.\(2019\)O\. Zawacki\-Richter, V\. I\. Marín, M\. Bond, and F\. GouverneurSystematic review of research on artificial intelligence applications in higher education—where are the educators?\.International Journal of Educational Technology in Higher Education16\(1\),pp\. 39\.External Links:[Document](https://dx.doi.org/10.1186/s41239-019-0171-0)Cited by:[§2](https://arxiv.org/html/2610.33488#S2.SS0.SSS0.Px1.p1.1)\.