From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

arXiv cs.LG Papers
forecasting effort progress online-learning intelligent-tutoring-systems supervised-prediction analytics

Summary

This paper introduces engagement forecasting for intelligent tutoring systems, predicting weekly minutes practiced and new skills mastered using interaction logs from 425 middle-school students. Feature-based models reduce error by 22-33% over heuristic baselines, offering explainable patterns for tutor-learner goal setting.

arXiv:2605.12788v1 Announce Type: new Abstract: Sustained effort is essential for realizing the benefits of intelligent tutoring systems (ITS), yet many learners disengage or underuse available practice time. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week. Using interaction log data from 425 middle-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks. We show that these feature-based models reduce mean absolute error (MAE) by 22-33% relative to heuristic baselines, including fixed-percentile rules adapted from prior work in other behavioral domains. We find that percentile heuristics systematically overpredict, whereas feature-based models better track student practice trajectories across weeks. To support explainability, we analyze feature importance and ablations, revealing target-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner-state and content difficulty signals. Finally, in a semi-structured user interview case study with eight college tutors, we examine how tutors reasoned about system-generated predictive features when setting goals with students. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor-learner goal setting and timely instructional decisions.
Original Article
View Cached Full Text

Cached at: 05/14/26, 06:18 AM 

# From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning
Source: [https://arxiv.org/abs/2605.12788](https://arxiv.org/abs/2605.12788)
[View PDF](https://arxiv.org/pdf/2605.12788)

> Abstract:Sustained effort is essential for realizing the benefits of intelligent tutoring systems \(ITS\), yet many learners disengage or underuse available practice time\. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week\. Using interaction log data from 425 middle\-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks\. We show that these feature\-based models reduce mean absolute error \(MAE\) by 22\-33% relative to heuristic baselines, including fixed\-percentile rules adapted from prior work in other behavioral domains\. We find that percentile heuristics systematically overpredict, whereas feature\-based models better track student practice trajectories across weeks\. To support explainability, we analyze feature importance and ablations, revealing target\-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner\-state and content difficulty signals\. Finally, in a semi\-structured user interview case study with eight college tutors, we examine how tutors reasoned about system\-generated predictive features when setting goals with students\. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis\. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS\. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor\-learner goal setting and timely instructional decisions\.

## Submission history

From: Eric S\. Qiu \[[view email](https://arxiv.org/show-email/9c5a9495/2605.12788)\] **\[v1\]**Tue, 12 May 2026 22:04:06 UTC \(629 KB\)

Similar Articles

Forecasting Future Behavior as a Learning Task

arXiv cs.AI

This paper proposes Behavior Forecasters, a learned approach that predicts an LRM's future behavior (e.g., answer consistency and input sensitivity) from its reasoning trajectory, outperforming GPT-5.4 and Claude Opus 4.6 at lower cost.

Forecasting Future Behavior as a Learning Task

Hugging Face Daily Papers

This paper proposes training Behavior Forecasters to predict large reasoning model outputs from single trajectories, outperforming large language models like GPT-5.4 and Claude Opus-4.6 at lower computational cost, bypassing traditional explainability methods.