TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education

# A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education
Source: [https://arxiv.org/html/2608.26184](https://arxiv.org/html/2608.26184)
## TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI\-Assisted Programming EducationConference:The 39th Annual ACM Symposium on User Interface Software and Technology; November 02–05, 2026; Detroit, MI, USAThe 39th Annual ACM Symposium on User Interface Software and Technology \(UIST ’26\), November 02–05, 2026, Detroit, MI, USADOI:[10\.1145/3830398\.3830712](https://doi.org/10.1145/3830398.3830712)ISBN:979\-8\-4007\-2856\-3/2026/11

David BarronOrcID:[0009\-0005\-7252\-7762](https://orcid.org/0009-0005-7252-7762)Affiliation:Virginia Tech,Blacksburg,Virginia,USAemail:[dbarron410@vt\.edu](mailto:[email protected])Xiaohang TangOrcID:[0000\-0002\-2691\-9280](https://orcid.org/0000-0002-2691-9280)Affiliation:Virginia Tech,Blacksburg,Virginia,USAemail:[xiaohangtang@vt\.edu](mailto:[email protected]),Rezky DwisantikaOrcID:[0009\-0003\-3192\-090X](https://orcid.org/0009-0003-3192-090X)Affiliation:Sepuluh Nopember Institute of Technology,Surabaya,Jawa Timur,Indonesiaemail:[rezkysantika21@gmail\.com](mailto:[email protected]),Minsun KimOrcID:[0009\-0002\-4593\-2672](https://orcid.org/0009-0002-4593-2672)Affiliation:Virginia Tech,Blacksburg,Virginia,USAemail:[minsunkim@vt\.edu](mailto:[email protected]),David H\. Smith IVOrcID:[0000\-0002\-6572\-4347](https://orcid.org/0000-0002-6572-4347)Affiliation:Virginia Tech,Blacksburg,Virginia,USAemail:[dhsmith4@vt\.edu](mailto:[email protected]),Jiaming CuiOrcID:[0000\-0002\-2685\-2776](https://orcid.org/0000-0002-2685-2776)Affiliation:Virginia Tech,Blacksburg,Virginia,USAemail:[jiamingcui@vt\.edu](mailto:[email protected])andYan ChenOrcID:[0000\-0002\-1646\-6935](https://orcid.org/0000-0002-1646-6935)Affiliation:Virginia Tech,Blacksburg,Virginia,USAemail:[ych@vt\.edu](mailto:[email protected])

2026; © cc

###### Abstract\.

AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners’ needs\. We present TutorTrace, a dataset and behavioral abstraction pipeline that makes learners’ behavioral context visible and computable in real time from low\-level IDE telemetry\. Across four deployments in two introductory Python courses \(N=480\), TutorTrace captures approximately 180K telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics\. From this foundation, we derive a taxonomy of learner activity before the first AI query, between consecutive queries, and across the full session, enabling systems to respond not just to what learners say, but to what they have done leading up to the help\-seeking moment\. In a preliminary classroom evaluation, behavior\-aware prompts were associated with a decrease in intervals between queries with no independent work from 50\.0% to 20\.7%\. As an additional demonstration of downstream utility, we evaluate TutorTrace on two held\-out prediction tasks: whether a learner will query within the next 60 seconds \(AUROC=\.726\{=\}\.726\) and whether an upcoming query reflects guided or dependent help\-seeking \(AUROC=\.717\{=\}\.717\)\. Together, these findings show how behavioral context can enable adaptive AI tutoring at scale\.

###### Keywords:

behavioral analysis, intelligent learning environment, educational technology, student behavior classification, LLM\-assisted programming, IDE telemetry, human\-AI interaction

††cc\-license:by## 1\.Introduction

Table 1\.Comparison of TutorTrace with existing CS education and AI\-assisted programming datasets: the 2nd CSEDM Data Challenge\([CSEDM Data Challenge Organizers 2021](https://arxiv.org/html/2608.26184#bib.bib10);[Edwards and Murali 2017](https://arxiv.org/html/2608.26184#bib.bib11)\), ProgSnap2\([Price et al\. 2020](https://arxiv.org/html/2608.26184#bib.bib29)\), CodeAid\([Kazemitabaar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib21)\), and CodeWatcher\([Basha et al\. 2025](https://arxiv.org/html/2608.26184#bib.bib4)\)\.Large Language Models have improved personalized computing education by providing learners with near\-instant, in\-depth responses built on a knowledge base no single tutor can compete with\([Tang et al\. 2025](https://arxiv.org/html/2608.26184#bib.bib35)\)\. Yet, one\-on\-one human tutoring remains the gold standard for improving learner outcomes\([Bloom 1984](https://arxiv.org/html/2608.26184#bib.bib6)\)\. One key reason is that human tutors respond not only to what learners explicitly say, but also to what their observable behaviors suggest they implicitly need\([Wood et al\. 1976](https://arxiv.org/html/2608.26184#bib.bib37);[Chi et al\. 2001](https://arxiv.org/html/2608.26184#bib.bib8)\)\. Consider what this could look like at scale\.

> Imagine if every student in a 400\-person programming course had a tutor sitting beside them, observing what they do and where they struggle\. One who knows when to push them harder and when to slow down and offer support\. One who could tell the difference between a student who is struggling and one who hasn’t put in the effort\. Achieving this vision requires systems that can infer learners’ evolving behavioral context in real time from fine\-grained interaction data, a capability and data resource not yet available to our community\.

The human tutor’s comparative advantage lies in the behavioral context that precedes the learner’s help\-seeking moment\. Current AI tutoring systems are typically limited to the question itself, missing the struggle or lack thereof that preceded it\. To bridge this gap, systems must be able to programmatically detect the behavioral patterns that human tutors observe intuitively\.

This requires datasets that expose fine\-grained student programming behavior at scale\. Most existing datasets capture only subsets of the learner’s programming process, such as static code submissions or AI interaction logs\. Few provide the continuous, fine\-grained interaction telemetry required to enable full\-fidelity replays of learner programming sessions \(Table[1](https://arxiv.org/html/2608.26184#S1.T1)\)\. Systems that do capture interaction at this granularity\([Mozannar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib28);[Zhang et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib39)\)were not built to release classroom\-scale datasets\.

Prior work shows that the behavioral context surrounding a help\-seeking moment is associated with learning outcomes\([Ma et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib25)\), and specific interaction patterns distinguish effective from ineffective AI use\([Ma et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib25);[Shen and Tamkin 2026](https://arxiv.org/html/2608.26184#bib.bib32)\)\. Students who invest independent effort before receiving instruction learn more from it\([Kapur 2014](https://arxiv.org/html/2608.26184#bib.bib20);[Sinha and Kapur 2021](https://arxiv.org/html/2608.26184#bib.bib33)\), and a student’s behavioral context directly informs the tutor’s pedagogical strategy for intervening\([Harvey et al\. 2025](https://arxiv.org/html/2608.26184#bib.bib14)\)\. Yet current AI tutoring systems typically respond to the information learners explicitly provide rather than the behavioral process that preceded it\. As a result, two learners with similar code, errors, and questions may receive similar support despite arriving at that moment through fundamentally different processes\.

We present TutorTrace, a dataset and automated behavioral abstraction infrastructure designed to make this context visible and computable\. The dataset comprises fine\-grained IDE telemetry from 480 students across four deployments in two introductory Python courses\. An automated pipeline transforms this telemetry into continuously computed observable metrics and behavioral sequences\. Unlike existing datasets that capture only what learners submit or ask, TutorTrace captures the story of how they got there, making each learner’s evolving behavioral context programmatically accessible in real time\.

To demonstrate the utility of TutorTrace for future research and system design, we examine how its representations of behavioral context can support adaptive AI tutoring\. First, we derive a three\-window behavioral taxonomy that organizes learner activity before the first AI query, between consecutive queries, and through recurring re\-querying patterns across the session\. Next, we conduct a preliminary comparison examining whether behavior\-aware prompting is associated with changes in learners’ observable activity between queries\. Finally, we demonstrate additional predictive utility through two tasks: predicting whether a learner will initiate an AI interaction within the next 60 seconds and whether an upcoming query will reflect guided or dependent help\-seeking\. Together, these capabilities enable systems to detect, respond to, and anticipate learner behavior surrounding AI interactions\. We make the following contributions:

- •TutorTrace Dataset:A publicly available dataset111[https://github\.com/divadbaroon/TutorTrace\_dataset\_and\_benchmark/tree/uist](https://github.com/divadbaroon/TutorTrace_dataset_and_benchmark/tree/uist)from 480 learners across four deployments in two introductory Python courses, comprising approximately 180K fine\-grained IDE telemetry events mapped to 13,633 automatically classified behavioral segments, along with 27 continuously computed observable metrics surrounding 1,386 AI interactions\.
- •TutorTrace Classifier:An automated behavioral classifier that transforms raw IDE telemetry into labeled behavioral sequences in real time\. The classifier is grounded in expert\-developed rules and validated against expert labels and student self\-reports, with overall pairwise agreement ranging from 78% to 87%\.
- •TutorTrace Taxonomy and Downstream Utility:We demonstrate how the representations provided by TutorTrace can support adaptive AI tutoring through: - –A three\-window behavioral taxonomy comprising ten profiles that characterize activity before the first AI query, between consecutive queries, and through recurring re\-querying patterns across the session\. - –A preliminary classroom comparison in which behavior\-aware prompting was associated with a decrease in Passive inter\-query windows from 50\.0% to 20\.7% and greater observable activity between queries\. - –Two held\-out prediction tasks demonstrating that observable metrics support prediction of query imminence within 60 seconds \(AUROC=\.726\{=\}\.726\) and guided versus dependent help\-seeking \(AUROC=\.717\{=\}\.717\)\.

## 2\.Related Work

### 2\.1\.Prior Taxonomies in CS Education

Taxonomies typically organize distinct, non\-overlapping categories across one or more dimensions while also revealing relationships among the taxa\([Irvine 2021](https://arxiv.org/html/2608.26184#bib.bib17)\)\. Past studies in CS education have employed educational taxonomies to design learning objectives and assess learning effectiveness\([Smith et al\. 2025](https://arxiv.org/html/2608.26184#bib.bib34)\), though researchers have argued that general\-purpose frameworks such as Bloom’s taxonomy and SOLO do not fully capture the distinctive characteristics of computer science learning, particularly in programming\-related contexts\([Fuller et al\. 2007](https://arxiv.org/html/2608.26184#bib.bib13)\)\.

After the rise of LLMs, behavioral analysis of programming interaction has become more important and complex\. Copilot, an LLM\-based code generation tool, can produce correct solutions to many introductory programming problems, raising questions about how its presence reshapes student work\([Wermelinger 2023](https://arxiv.org/html/2608.26184#bib.bib36)\)\. In CS education specifically, LLM\-based programming assistants support students during coding tasks while also reshaping their interaction patterns, problem\-solving processes, and reliance on AI assistance\([Kazemitabaar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib21);[Liffiton et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib23);[Hou et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib15)\)\. As a result, programming sessions generate large amounts of human\-AI interaction traces not easily described by traditional educational taxonomies alone\([Mozannar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib28);[Barke et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib3)\)\. To address this gap, Mozannar et al\.\([Mozannar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib28)\)proposed a taxonomy representing AI\-assisted coding sessions as timelines of transitions across programmer activities, while Barke et al\.\([Barke et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib3)\)showed that Copilot use also reflects broader interaction modes, such as acceleration and exploration\.

Despite these developments, taxonomies specifically designed to characterize student behavior in LLM\-assisted programming contexts remain rare\. In this work, we introduce a taxonomy of learner behavioral profiles for AI\-assisted programming education, aiming to make student–AI interaction patterns more visible and interpretable in educational settings\.

### 2\.2\.Help\-Seeking Quality and AI Dependency

With the advancement of AI, students increasingly rely on them to complete assignments rather than as learning aids, raising concerns about over\-reliance and reduced independent problem\-solving\([Jošt et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib18);[Xue et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib38);[Liu et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib24)\)\. Recent work in CS education has developed tools that leverage LLMs for scaffolded hint generation while incorporating guardrails to prevent solution delegation\([Kazemitabaar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib21);[Hou et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib15);[Liffiton et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib23)\)\. However, even when system\-level constraints limit the model’s output, students bypass them when given the option: Kapoor et al\.\([Kapoor et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib19)\)deployed an AI TA with an optional “See Solution” control that disabled the guardrails, and 50% of 885 students used it on at least one problem, with 14% using it on all three\.

To mitigate these behaviors, several approaches shape or qualify student queries before they reach the underlying LLM\. CodeHelp\([Liffiton et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib23)\)replaces a single free\-form box with structured fields for language, code, error message, and issue description, and runs an LLM\-based sufficiency check that returns a clarification request when a query lacks critical information\. CodeAid\([Kazemitabaar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib21)\)similarly offers feature\-specific input templates that scaffold how students frame a request\. However, these approaches intervene based on the content of the query itself, not the behavioral context that preceded it\. A student who spent five minutes debugging independently before asking and a student who re\-queried immediately after the last response may submit identical questions, yet require fundamentally different pedagogical responses\. TutorTrace addresses this gap by making the behavioral context surrounding each query visible and computable, enabling interventions grounded not in what the student says, but in what they did before saying it\.

![Screenshot of the student IDE showing a task description and test cases on the left, a code editor and terminal in the center, and an AI chat window on the right.](https://arxiv.org/html/2608.26184v1/figures/Vizpi.png)Figure 1\.The student interface with task description, code editor, terminal, test cases, and AI chat window\.Screenshot of the student IDE showing a task description and test cases on the left, a code editor and terminal in the center, and an AI chat window on the right\.

## 3\.System

TutorTrace is a task\-based IDE platform with LLM support, similar in structure to LeetCode222[https://leetcode\.com/](https://leetcode.com/)and HackerRank333[https://www\.hackerrank\.com/](https://www.hackerrank.com/)\.These platforms share a fixed layout containing a task description, test cases, code editor, terminal, and AI chat window, constraining student activity to a well\-defined interaction space, making fine\-grained behavioral observation within this environment both feasible and reproducible\. As students complete programming tasks, TutorTrace passively captures and translates low\-level telemetry into behavioral sequences and observable metrics in real time, providing programmatic access to the behavioral context that human tutors observe intuitively but that existing AI tutoring systems do not see\.

### 3\.1\.Telemetry Capture

TutorTrace captures fine\-grained user interactions within the interface, recording 37 event types across six source regions \(Table[2](https://arxiv.org/html/2608.26184#S3.T2)\)\. Events are captured continuously on the client side and batched to the back end every five seconds with no impact to the student’s workflow\. Each recorded event includes a millisecond timestamp, source region, and payload\. A student’s full stream of telemetry events enables a full\-fidelity replay of their session in the Behavioral Labeling Interface \(Appendix[A](https://arxiv.org/html/2608.26184#A1), Figure[4](https://arxiv.org/html/2608.26184#acmlabel5)\) and serves as the foundational input to our behavioral abstraction pipeline \(Figure[2](https://arxiv.org/html/2608.26184#acmlabel3)\)\.

Table 2\.Telemetry events captured by TutorTrace, grouped by source region\.RegionExample EventsExample PayloadCode EditorTYPE, DELETE, PASTE,Characters, cursor\(11 types\)COPY, CUT, UNDO,position, selectionREDO, SELECT, INDENTrangeTerminalRUN, OUTPUT, ERROR,stdout, error type\(6 types\)RESULT, SELECT, COPYand message, testpass/fail countsChatQUERY, RESPONSE,Message content,\(9 types\)TYPE, DELETE, PASTE,length, latency,SELECT, COPYsender IDTask/TestsSELECT, COPYSelected text,\(4 types\)source regionGlobalMOUSE\_CLICK,Coordinates, active\(5 types\)TAB\_STATE,region, tabWINDOW\_RESIZE,visibility, panelPANEL\_RESIZE,dimensionsMOUSE\_MOVESessionSTART, ENDWindow dimensions,\(2 types\)durationA complete schema of all 37 event types with example payloads is provided in Appendix[C](https://arxiv.org/html/2608.26184#A3), Table[10](https://arxiv.org/html/2608.26184#A3.T10)

### 3\.2\.LLM Integration

Our system is integrated with GPT\-4o444[https://openai\.com/index/hello\-gpt\-4o/](https://openai.com/index/hello-gpt-4o/), accessible via the chat window panel\. The prompt is grounded in prior literature on pedagogical prompting strategies for AI tutoring and is structured around three escalating levels of scaffolding\([Wood et al\. 1976](https://arxiv.org/html/2608.26184#bib.bib37);[Kapur 2014](https://arxiv.org/html/2608.26184#bib.bib20);[Roll et al\. 2011](https://arxiv.org/html/2608.26184#bib.bib30)\)\. It was iteratively refined through pilot testing until the LLM consistently adhered to the following principles \(Appendix[D](https://arxiv.org/html/2608.26184#A4)\):

1. \(1\)Socratic Questioning:Guide the student toward the answer through targeted questions rather than direct explanation\. Avoid revealing solutions\([Kazemitabaar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib21)\)\.
2. \(2\)Conceptual Hint:Provide a high\-level conceptual nudge identifying the relevant concept or approach without specifying implementation\([Liffiton et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib23)\)\.
3. \(3\)Concrete Scaffolding:Provide direct, specific guidance using pseudocode and blanks when the student has demonstrated sustained effort without progress\([Kapur 2014](https://arxiv.org/html/2608.26184#bib.bib20)\)\.

## 4\.Behavioral Classification

While prior work often derives behavioral labels from heuristic rules alone, we ground ours in the judgment of four domain\-expert annotators, each with teaching and research experience in CS education\. To label learner sessions, annotators used the Behavioral Labeling Interface \(Appendix[A](https://arxiv.org/html/2608.26184#A1)\), which presents a replay of the learner’s session, a session timeline, and a behavioral codebook for annotating observed behavior directly onto the timeline\. Annotators independently coded replays of learner sessions and met weekly to reconcile disagreements in their classifications\.

### 4\.1\.Codebook Development

We derived our initial behavioral categories from prior literature on programming behavior and help\-seeking\([Mozannar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib28);[Blikstein 2011](https://arxiv.org/html/2608.26184#bib.bib5)\)\. We iteratively refined the codebook through 10 pilot labeling sessions drawn from a preliminary deployment in an intermediate Python course at our institution\. During this phase, four domain\-expert annotators independently segmented and labeled 10 pilot sessions \(each 5–13 minutes of learner activity, mean 9 minutes\) using the Behavioral Labeling Interface \(Appendix[A](https://arxiv.org/html/2608.26184#A1)\)\. The annotators met weekly to reconcile disagreements over segment boundaries and behavioral classifications, updating the codebook as needed \(Appendix[B](https://arxiv.org/html/2608.26184#A2), Table[9](https://arxiv.org/html/2608.26184#A2.T9)\)\. The finalized codebook, presented in Table[3](https://arxiv.org/html/2608.26184#S4.T3), served as the basis for the segmentation and classification rules used by our automated behavioral classification pipeline\.

Table 3\.Behavioral codebook used by domain\-expert annotators to label learner sessions\. The table lists each behavior, its subtypes and definition, and the corresponding automated segmentation and classification rule derived through annotator consensus except where noted\.†\\daggerTesting was not manually annotated\. Because test execution is directly observable, eachTERMINAL\_RUNevent was automatically inserted as a run marker on the Behavioral Labeling Interface timeline\.

### 4\.2\.Behavioral Segmentation

The most challenging aspect of manual annotation was determining precise segment boundaries\. Across the pilot sessions, annotators generally agreed on which behavior was occurring but differed in the exact timing of its start and end\. The mean discrepancy in boundary placement across annotators was 2\.3 seconds\. During weekly reconciliation meetings, we found this stemmed from the cognitive load of continuously watching session replays rather than any conceptual disagreement on what constituted a behavioral transition\. Annotators reported that identifying the behavior itself was straightforward, whereas pinpointing the exact time of transition required sustained attention that naturally degraded over longer sessions\.

To address this, the first author reviewed all independently annotated boundaries and synthesized them into proposed consensus segment boundaries\. To verify these, all annotators then rewatched each session while the first author verbally announced each boundary as it occurred\. This reconciliation process yielded consensus on all segment boundaries\. We used the resulting consensus boundaries to derive the automated segmentation rules described in Appendix[F](https://arxiv.org/html/2608.26184#A6), Algorithm[1](https://arxiv.org/html/2608.26184#alg1)\.

Table 4\.Pairwise agreement among learner self\-reports, expert annotations, and the automated classifier across six user\-study sessions\. Cohen’sκ\\kappa\([Cohen 1960](https://arxiv.org/html/2608.26184#bib.bib9)\)and raw percentage agreement are reported for each behavior and overall\.
### 4\.3\.Validation

We validated the pipeline in two stages\. First, we established inter\-rater reliability among human annotators across the 10 pilot sessions to confirm the codebook was sufficiently stable before scaling\. Second, we conducted a user study with 13 participants who completed two Python programming tasks and then self\-labeled their behavior using the Behavioral Labeling Interface\. Our domain\-expert annotators independently labeled six of these sessions, allowing us to triangulate agreement across three sources: learner self\-reports, expert annotations, and the automated classifier\. Testing was excluded from the agreement analysis because test executions were directly observed fromTERMINAL\_RUNevents rather than independently annotated\. Across the three pairwise comparisons, overall Cohen’sκ\\kapparanged from 0\.73 to 0\.83, and overall raw agreement ranged from 78% to 87% \(Table[4](https://arxiv.org/html/2608.26184#S4.T4)\)\.

We also found several recurring edge cases that raw IDE telemetry could not differentiate alone\. For instance, after receiving an error, a learner might ignore it and continue implementing a separate feature, makingImplementingdifficult to distinguish fromDebugging\. In the same vein,Thinking about Codemay be difficult to distinguish fromThinking about Error\. For these situations, semantic understanding is needed to differentiate the behavior\. We discuss further in Section[9](https://arxiv.org/html/2608.26184#S9)\.

Table 5\.The 35 candidate observable metrics grouped by observable area, drawn from prior IDE\-based learning analytics literature\([Blikstein 2011](https://arxiv.org/html/2608.26184#bib.bib5);[Mozannar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib28)\)\. Metrics marked†\\daggerwere excluded during pruning\.Observable AreaMetricsCode activitycode\_editscode\_edit\_ratechars\_insertedchars\_deletedcode\_deletesnet\_code\_growthdelete\_type\_ratiocode\_pastes†Terminal activityterminal\_runsterminal\_errorsmax\_consecutive\_errorsmean\_time\_between\_runs\_sError recoveryerror\_self\_fixerror\_reading\_time\_serror\_to\_edit\_sfailed\_test\_to\_edit\_sfailed\_test\_self\_fixTime distributiontime\_in\_editor\_stime\_in\_terminal\_stime\_in\_chat\_stime\_in\_task\_stime\_in\_tests\_slongest\_idle\_stab\_hidden\_time\_s†Chat behaviorthinking\_time\_sseeking\_help\_time\_sduration\_sresponse\_reading\_time\_schat\_to\_code\_latency\_sInterface eventstab\_switches†copy\_events†paste\_events†undo\_events†redo\_events†select\_events†
†Excluded during pruning: sparse metrics \(\>80%\>80\\%zeros\) and metrics capturing activity outside the instrumented workspace\.

## 5\.Observable Metrics

While behavioral sequences capture the temporal progression of learner behavior, they do not provide insights into the learners’ aggregate activity within these temporal windows, such as the frequency of code edits, terminal runs, and character deletions\. To capture this aggregate activity, we compute observable metrics that convert raw IDE telemetry into intuitive measures of the amount, frequency, and distribution of learner activity within a specific temporal window\. Together, behavioral sequences and observable metrics provide complementary perspectives on the temporal evolution and aggregate characteristics of learner behavior within the programming environment\.

We compute 35 candidate observable metrics drawn from prior IDE\-based learning analytics literature \(Table[5](https://arxiv.org/html/2608.26184#S4.T5)\)\. These metrics capture complementary dimensions of code activity, terminal activity, error recovery, time distribution, chat behavior, and interface events\. After excluding sparse metrics \(\>80%\>80\\%zeros\) and metrics capturing activity outside the instrumented workspace, 27 observable metrics remain\. Together, the retained metrics provide general\-purpose aggregate representations of learner activity\.

![Refer to caption](https://arxiv.org/html/2608.26184v1/figures/System4.png)Figure 2\.TutorTrace behavioral abstraction pipeline\. Raw telemetry events \(A\) are aggregated into window\-specific observable metrics \(B\) and auto\-segmented into behavioral sequences \(C\)\. Observable metrics support assignment to the behavioral profiles derived in thePipeline diagram\. Telemetry from the student IDE streams into raw event tokens, which are aggregated into per\-window observable metrics and auto\-segmented into behavioral sequences\. Metrics feed profile assignment in a unified Behavioral Profiles card listing three, four, and three profiles for Windows 1, 2, and 3, while the sequence feeds the prompt directly\. An AI tutor prompt box shows the profile, session pattern, recent sequence, and intervention appended after the system line\.TutorTrace Taxonomy \(D\), while behavioral sequences preserve recent ordered activity\. These representations can be incorporated into an AI tutor prompt to support behavior\-aware adaptation\.
## 6\.Deployments and Dataset

We deployed TutorTrace across four deployments in two introductory Python courses at our institution \(Table[6](https://arxiv.org/html/2608.26184#S6.T6)\)\. Deployments 1 and 2 were conducted with Instructor A using a task focused on list indexing, slicing, and in\-place manipulation with a 15\-minute time limit\. Deployments 3 and 4 were conducted with Instructor B using a task focused on nested lists and iteration\. Full task descriptions and test cases are provided in Appendix[I](https://arxiv.org/html/2608.26184#A9)\. Due to scheduling constraints in the second course with Instructor B, students in Deployments 3 and 4 completed the task under a 10\-minute time limit rather than the 15 minutes used in Deployments 1 and 2, which may have contributed to the lower completion rates in those sessions\.

The predictive models reported in Section[7\.4\.5](https://arxiv.org/html/2608.26184#S7.SS4.SSS5)were trained on Deployment 1 \(Instructor A, morning,n=190n=190\) and evaluated on held\-out Deployment 2 \(Instructor A, afternoon,n=113n=113\), providing a cross\-cohort evaluation under the same instructor and task but at a different time of day\. Deployments 3 and 4 supported a preliminary classroom evaluation examining whether behaviorally aware prompts could support changes in learner behavior at scale \(Section[7\.3](https://arxiv.org/html/2608.26184#S7.SS3)\)\. Deployment 3 \(Instructor B, morning,n=70n=70\) served as the baseline condition, while Deployment 4 \(Instructor B, afternoon,n=107n=107\) served as the intervention condition, in which the AI tutor’s prompts were informed by each learner’s behavioral profile\.

Table 6\.Deployment context across four sessions\.Together, these deployments yield the TutorTrace foundation dataset, comprising approximately 180K raw telemetry events, 27 continuously computed observable metrics, and 13,633 automatically classified behavioral sequences \(Layers A–C in Figure[2](https://arxiv.org/html/2608.26184#acmlabel3)\)\.

## 7\.The TutorTrace Taxonomy

While our real\-time behavioral abstraction pipeline enables systems to be behaviorally aware, making this awareness actionable requires identifying the recurring behavioral signatures that are both distinct and associated with differences in task outcomes\. We’re interested in both distinct behavioral signatures and task outcomes because learners who are performing poorly with high effort, and those that are performing well with low effort, each require fundamentally different pedagogical responses\. We therefore derive the TutorTrace Taxonomy, which organizes learner activity surrounding AI help\-seeking into recurring behavioral profiles, their respective outcomes, and expert\-informed interventions that are provided to the AI tutor’s prompt\.

![Table of the three-window taxonomy: each row shows a profile with an example behavioral sequence strip, its description, intervention, share of observations, descriptive statistics, and completion rate.](https://arxiv.org/html/2608.26184v1/taxonomy.png)Figure 3\.The TutorTrace three\-window behavioral taxonomy\. Each window captures a distinct moment in the help\-seeking cycle, characterized by different behavioral signals and requiring different intervention strategies\.Table of the three\-window taxonomy: each row shows a profile with an example behavioral sequence strip, its description, intervention, share of observations, descriptive statistics, and completion rate\.### 7\.1\.Taxonomy Construction

We organize the analysis across three temporal windows, each capturing a distinct stage of the help\-seeking cycle: learner activity before the first AI query \(Window 1\), activity between consecutive queries \(Window 2\), and recurring inter\-query patterns across the session \(Window 3\)\. We truncate each session at its first all\-pass test result, which marks the end of task\-directed activity and prevents post\-completion behavior from influencing the profiles\. For learners who do not complete the task, we retain activity through the observed session end\. We also exclude individual windows containing more than 30 seconds of tab\-hidden time to limit the influence of unobserved off\-platform activity

Windows 1 and 2 follow an outcome\-guided clustering procedure intended to identify behaviorally distinct profiles that also differ meaningfully in task outcomes\. We first separate observations with no observable programming effort, defined as zero code edits and zero terminal runs, into rule\-defined profiles\. These observations represent a qualitatively distinct state rather than simply a low\-activity version of active behavior, and including them in K\-means could cause the zero\-inflated observations to dominate the resulting clusters\.

For the remaining active observations, we construct a window\-specific candidate metric pool so that clustering uses only measures meaningful within that stage of the help\-seeking cycle\. We exclude metrics that are inapplicable to the window or zero\-valued in more than 80% of observations because they provide little discriminative information and may produce unstable or artifact\-driven clusters\.

We exhaustively evaluate every three\-metric combination from the resulting candidate pool\. For each combination, we standardize the metrics and apply K\-means\([McQueen 1967](https://arxiv.org/html/2608.26184#bib.bib27)\)withK∈\{2,3\}K\\in\\\{2,3\\\}using 10 initializations\. We discard solutions containing fewer than 15 observations in any cluster\. For each remaining solution, we calculate its silhouette score\([Rousseeuw 1987](https://arxiv.org/html/2608.26184#bib.bib31)\)and the spread between its highest and lowest task\-completion rates\. Candidate solutions are ranked according to:

score=silhouette×completion\-rate spread\.\\text\{score\}=\\text\{silhouette\}\\times\\text\{completion\-rate spread\}\.We perform clustering using behavioral metrics only\. Completion rates are used to select among candidate solutions that produce behaviorally distinct groups\. Because completion informs the selection of the final solution, the resulting differences in completion rates across clusters are descriptive rather than confirmatory\. After selecting a solution, we assign profile names based on the behavioral patterns represented by each cluster centroid\.

#### 7\.1\.1\.Window 1: Before the First Query

In Window 1 each learner contributes one set of aggregate metrics capturing their activity before their first query\. Learners with zero code edits and zero terminal runs during this period are assigned to the rule\-defined Cold Start profile \(n=31n\{=\}31\)\. The remaining 121 active learners are included in the exhaustive search\.

The highest\-ranked solution usestime\_in\_editor\_s,time\_in\_terminal\_s, andtime\_in\_chat\_satK=2K\{=\}2\(s=0\.420s\{=\}0\.420; completion\-rate spread=25\.3\{=\}25\.3percentage points\)\. Based on their behavioral centroids, the resulting clusters are labeled Oriented \(n=100n\{=\}100\) and Struggling \(n=21n\{=\}21\)\. Together with Cold Start, these profiles distinguish initial help\-seeking that follows no observable programming attempt, a moderate active attempt, or an extended period of unsuccessful effort\.

#### 7\.1\.2\.Window 2: Between Consecutive Queries

In Window 2, each interval between consecutive AI queries contributes one set of aggregate metrics capturing the learner’s activity after one AI response and before their next query\. Learners may therefore contribute multiple Window 2 observations and may exhibit different profiles at different points in a session\.

Inter\-query windows with zero code edits and zero terminal runs following the preceding AI response are assigned to the rule\-defined Passive profile \(n=318n\{=\}318\)\. We combine eligible post\-response and subsequent\-effort metrics into one candidate pool and apply the shared exhaustive search to the remaining 368 active windows\. Each interval inherits the task\-completion outcome of its corresponding session\.

The highest\-ranked solution usestime\_in\_editor\_s,thinking\_time\_s, anderror\_self\_fixatK=3K\{=\}3\(s=0\.547s\{=\}0\.547; completion\-rate spread=33\.8\{=\}33\.8percentage points\)\. Based on their centroids, the resulting clusters are labeled Iterating \(n=290n\{=\}290\), Debugging \(n=56n\{=\}56\), and Spinning \(n=22n\{=\}22\)\. Together with Passive, these profiles distinguish repeated queries made without observable programming activity from queries following brief iteration, active error recovery, or prolonged activity with limited execution\.

#### 7\.1\.3\.Window 3: Session\-Wide Re\-querying Patterns

Whereas Window 2 characterizes individual inter\-query moments, Window 3 summarizes whether particular forms of inter\-query behavior recur across the session\. We aggregate each learner’s valid Window 2 intervals and retain learners with at least two such intervals, ensuring that the representation captures a recurring pattern rather than a single observation\. This yields 112 eligible learners\.

We assign each inter\-query interval to one of three mutually exclusive behavioral categories: Passive, containing no code edits or terminal runs; Tested, containing at least one terminal run; or Active Untested, containing code edits but no terminal run\. We then represent each learner using four session\-level measures: the proportion of Passive intervals, the proportion of Tested intervals, the proportion of Active Untested intervals, and the longest consecutive Passive streak normalized by the learner’s number of valid intervals\.

We standardize these four measures and apply K\-means forK∈\{2,3,4,5,6\}K\\in\\\{2,3,4,5,6\\\}using 100 initializations\. As in the preceding windows, we exclude solutions containing fewer than 15 learners in any cluster\. Unlike Windows 1 and 2, task completion is not used for either clustering or model selection; we select the eligible solution with the highest silhouette score\.

The highest silhouette score occurs atK=3K\{=\}3\(s=0\.399s\{=\}0\.399\), producing the Passive Re\-querying \(n=47n\{=\}47\), Active Testing \(n=31n\{=\}31\), and Untested Editing \(n=34n\{=\}34\) profiles\. The assignments are identical across 20 random seeds \(ARI\([Hubert and Arabie 1985](https://arxiv.org/html/2608.26184#bib.bib16)\)=1\.000\{=\}1\.000\)\.

### 7\.2\.Characteristics of Behavioral Profiles

The three windows capture complementary aspects of AI help\-seeking: activity before a learner’s first query, activity after an AI response and before the learner’s next query, and recurring re\-querying patterns across the session\. Because task completion informed the exploratory selection of the Window 1 and Window 2 clustering solutions, their completion rates are reported as descriptive characteristics rather than as independent evidence of profile validity\. Task completion did not inform the construction or selection of the Window 3 profiles; its completion rates are likewise reported descriptively\.

##### Window 1: Before the First Query\.

Cold Start learners comprised 20% of the Window 1 population \(n=31n\{=\}31\)\. These learners made no code edits or terminal runs before their first query, although their pre\-query periods averaged 72 seconds\. The corresponding sessions had a 97% completion rate\. Oriented learners \(n=100n\{=\}100, 66%\) exhibited moderate pre\-query activity and had a 92% completion rate\. In contrast, Struggling learners \(n=21n\{=\}21, 14%\) spent substantially longer before querying, averaging 165 seconds, 4\.2 terminal runs, and 2\.5 errors\. Their corresponding sessions had a 67% completion rate\.

These profiles complicate the assumption that greater pre\-query effort necessarily signals better progress\. In which, Cold Start learners completed the task at high rates despite making no observable programming attempt, whereas Struggling learners invested the most time and encountered the most errors but completed at the lowest rate\. Rather than implying that low effort is beneficial, this contrast suggests different pedagogical needs: Cold Start learners may require prompting to engage independently, while Struggling learners may have exhausted their current strategies and require more direct scaffolding\. An adaptive tutor should therefore respond not only to the amount of prior activity, but also to what that activity indicates about the learner’s progress\.

##### Window 2: Between Queries\.

Passive behavior accounted for 46% of valid inter\-query windows \(n=318n\{=\}318\)\. In these windows, learners issued another query without editing or executing their code after receiving the preceding AI response\. The corresponding sessions had an 81% completion rate\.

Among active windows, Iterating was the most common profile \(n=290n\{=\}290, 42%\)\. These windows involved relatively brief thinking periods and modest execution activity, and their corresponding sessions had an 84% completion rate\. Debugging windows \(n=56n\{=\}56, 8%\) involved more frequent execution and error recovery, averaging 3\.7 terminal runs and 2\.8 errors\. Their corresponding sessions had a 79% completion rate\. Spinning windows \(n=22n\{=\}22, 3%\) involved substantially longer thinking periods, averaging 149 seconds, but comparatively limited execution activity\. Their corresponding sessions had the lowest completion rate among the Window 2 profiles at 50%\.

Because learners may contribute multiple inter\-query windows, these percentages characterize help\-seeking moments rather than fixed learner types\. Together, the profiles distinguish repeated queries made without observable programming activity from queries following iteration, active debugging, or prolonged activity with limited execution\. These contexts could help an adaptive tutor determine whether to prompt independent work, support error recovery, or help a learner move beyond an unproductive pattern\.

##### Window 3: Session\-Wide Re\-querying Patterns\.

Passive Re\-querying learners comprised 42% of the Window 3 population \(n=47n\{=\}47\)\. On average, 70% of their valid inter\-query intervals contained no code edits or terminal runs, 22% included code execution, and 8% included editing without execution\. The corresponding sessions had a 79% completion rate\.

Active Testing learners comprised 28% of the population \(n=31n\{=\}31\)\. Their sessions were characterized by regular code execution: on average, 77% of their inter\-query intervals included at least one terminal run, while 20% were Passive and 4% involved editing without execution\. The corresponding sessions had an 81% completion rate\.

Untested Editing learners comprised the remaining 30% \(n=34n\{=\}34\)\. On average, 43% of their inter\-query intervals included code edits without a subsequent terminal run, compared with 35% Passive intervals and 23% Tested intervals\. The corresponding sessions had an 88% completion rate\.

Whereas Window 2 characterizes behavior within individual inter\-query intervals, Window 3 summarizes which forms of inter\-query activity recur across a learner’s session\. Together, the three windows provide complementary context about when learners request AI assistance, what they do between requests, and whether those behaviors persist across the session\.

### 7\.3\.Preliminary Evaluation

TutorTrace enables systems to detect and understand behavioral context in real time\. The natural next question is whether acting on this information is associated with changes in learners’ observable behavior\. We therefore deployed the baseline system in a morning session \(Deployment 3,n=48n\{=\}48eligible AI users\) and a treatment version in an afternoon session \(Deployment 4,n=85n\{=\}85eligible AI users\) for the same course and task\.

In the treatment condition, the system appended a behavior\-aware intervention block selected from the learner’s recent activity to the AI tutor’s prompt at each interaction \(Appendix[G](https://arxiv.org/html/2608.26184#A7)\)\. We subsequently applied the revised Window 2 taxonomy to both deployments to examine whether their profile distributions and observable activity differed\. This preliminary evaluation does not measure learning gains or establish a causal effect\.

Passive inter\-query windows accounted for 50\.0% of valid windows in the baseline deployment and 20\.7% in the intervention deployment \(−\-29\.3 pp\)\. Iterating windows accounted for 39\.1% and 56\.0%, respectively \(\+\+16\.9 pp\)\. The overall Window 2 profile distribution differed between deployments \(χ2​\(3\)=27\.55\\chi^\{2\}\(3\)\{=\}27\.55,p<\.0001p\{<\}\.0001\)\. Because learners could contribute multiple windows, we also compared each learner’s proportion of Passive windows\. This learner\-level comparison was likewise significant \(U=1533U\{=\}1533,p<\.0001p\{<\}\.0001\)\.

The intervention deployment also exhibited more code edits per valid inter\-query window \(23\.7 vs\. 11\.7\), more terminal runs \(2\.1 vs\. 1\.0\), and longer periods of activity between queries \(74\.9 s vs\. 47\.0 s\)\. Completion among eligible AI users was 33\.3% in the baseline deployment and 43\.5% in the intervention deployment\. This difference was not statistically significant \(p=\.273p\{=\}\.273\) and remains confounded by the between\-session design\.

These exploratory findings indicate that behavior\-aware prompting was associated with less Passive re\-querying and greater observable activity between queries\. However, the nonrandomized, between\-session design does not establish that the intervention caused these differences\.

Table 7\.Held\-out AUROC for query imminence \(query within 60 seconds\) and help\-seeking type \(guided vs\. dependent within 15 seconds\) across nested feature representations\. Models were trained on Deployment 1 and evaluated on Deployment 2 \(n=113n\{=\}113\)\.Δ\\Deltais relative to raw telemetry; bold indicates the best result\.Table 8\.Guided and dependent help\-seeking by behavioral profile\. Percentages are calculated within profiles\.
### 7\.4\.Demonstration of Downstream Utility: Prediction Tasks

The preliminary evaluation demonstrates how behavioral profiles can support reactive adaptation during an AI interaction\. We next examine whether the telemetry and behavioral abstractions provided by TutorTrace support two additional predictive decisions:query imminence, whether a learner will submit an AI query within the next 60 seconds, andhelp\-seeking type, whether an upcoming query reflects guided or dependent help\-seeking\. Guided queries articulate a specific need, question, or concern, whereas dependent queries offload identifying the problem or determining the next step to the AI\.

Together, predicting when a learner is likely to query and whether that query will be dependent could allow systems to intervene preemptively and encourage behaviors associated with more guided help\-seeking\. For example, 96\.7% of Cold Start queries were dependent, compared with 30\.9% and 28\.6% for Oriented and Struggling learners, respectively \(Table[8](https://arxiv.org/html/2608.26184#S7.T8)\)\.

#### 7\.4\.1\.Prediction Tasks

Forquery\-imminence prediction, the system maintains a 30\-second observation window that advances in 5\-second steps\. Each window is labeled according to whether the learner submits an AI query within the following 60 seconds\.

Forhelp\-seeking\-type prediction, each query is paired with the learner’s activity during the 15 seconds before submission\. The model predicts whether the query is guided, articulating a specific goal, concept, problem, or misunderstanding, or dependent, offloading identification of the problem or next step through a vague request, completion directive, or code without a specific question\. This distinction parallels adaptive and unproductive help\-seeking in prior tutoring research\([Aleven et al\. 2016](https://arxiv.org/html/2608.26184#bib.bib2);[Marwan et al\. 2020](https://arxiv.org/html/2608.26184#bib.bib26)\)\.

We use GPT\-4o to label all queries offline as guided or dependent\. For each query, the model receives the query text, the learner’s code at submission, and the preceding chat history\. The full labeling prompt is provided in Appendix[H](https://arxiv.org/html/2608.26184#A8)\. Two human raters independently classified 97 queries and achieved substantial agreement \(κ=\.897\\kappa\{=\}\.897\)\([Landis and Koch 1977](https://arxiv.org/html/2608.26184#bib.bib22)\)\. Agreement between GPT\-4o and the two raters wasκ=\.709\\kappa\{=\}\.709andκ=\.690\\kappa\{=\}\.690, respectively\.

For the prediction tasks, semantic labeling inputs were excluded\. Both tasks omit query\-composition events \(CHAT\_TYPE,CHAT\_DELETE,CHAT\_PASTE, andCHAT\_QUERY\), query text, source\-code content, chat history, and the subsequent AI response\. This prevents query\-imminence models from detecting query composition, while help\-seeking\-type windows end immediately before submission\. The models therefore rely only on preceding behavioral telemetry to predict when help\-seeking will occur and what form it will take\.

#### 7\.4\.2\.Feature Representations

For both tasks, we compare three nested feature representations to evaluate whether the higher\-level abstractions produced by TutorTrace provide predictive value beyond raw telemetry\. Features are computed strictly within each task’s observation window: 30 seconds for query imminence and 15 seconds for help\-seeking type\.

- •Raw telemetry: counts of each telemetry event type within the task\-specific window\.
- •\+Observable metrics: adds the observable metrics of Section[5](https://arxiv.org/html/2608.26184#S5), computed within the same window\.
- •\+Behavioral sequences: adds features derived from the auto\-classified behavioral sequences: time in each behavior, the current and preceding behavior, and counts of transitions between behaviors\.

#### 7\.4\.3\.Experimental Protocol

For each task and feature representation, we train a Random Forest classifier using Deployment 1 \(Instructor A, morning,n=190n\{=\}190\) and evaluate it on held\-out Deployment 2 \(Instructor A, afternoon,n=113n\{=\}113; Table[6](https://arxiv.org/html/2608.26184#S6.T6)\)\. Deployment 2 instances are excluded from model fitting and hyperparameter selection\.

We report area under the receiver operating characteristic curve \(AUROC\)\([Fawcett 2006](https://arxiv.org/html/2608.26184#bib.bib12)\)as the primary measure of predictive discrimination\. AUROC measures how consistently a model ranks positive instances above negative instances across classification thresholds\.

#### 7\.4\.4\.Profiles and Help\-Seeking Type

Table[8](https://arxiv.org/html/2608.26184#S7.T8)reports guided and dependent help\-seeking across behavioral profiles\. In Window 1, 96\.7% of labeled first queries from Cold Start learners were dependent\. In contrast, first queries from Oriented and Struggling learners were more often guided \(69\.1% and 71\.4%, respectively\)\. Learners who queried without first editing or executing code therefore exhibited a markedly different help\-seeking distribution from those who engaged in observable programming activity before their first query\.

Differences were less pronounced in Window 2\. Queries following all four inter\-query profiles were more often dependent than guided, with rates ranging from 63\.6% for Spinning to 74\.2% for Passive\. Because learners may contribute multiple Window 2 observations, these percentages describe profile–query pairs rather than independent learner groups\. We therefore interpret them as descriptive associations between the activity preceding a query and the form of help\-seeking that follows\.

#### 7\.4\.5\.Prediction Results

Table[7](https://arxiv.org/html/2608.26184#S7.T7)reports held\-out AUROC across the three nested feature representations\. For query\-imminence prediction, raw telemetry achieved an AUROC of 0\.689, while the full representation including behavioral\-sequence features achieved 0\.719\. Observable metrics alone performed best at 0\.726, an absolute improvement of 0\.037 over raw telemetry\.

A similar pattern emerged for help\-seeking\-type prediction\. Raw telemetry achieved an AUROC of 0\.690, while the full representation including behavioral\-sequence features achieved 0\.705\. Observable metrics alone performed best at 0\.717, an absolute improvement of 0\.027 over raw telemetry\.

Across both tasks, observable metrics provided the largest gain over raw event counts\. Behavioral\-sequence features remained above raw telemetry but did not improve on observable metrics in this Random Forest\([Breiman 2001](https://arxiv.org/html/2608.26184#bib.bib7)\)evaluation\. Together, these results suggest that window\-level summaries of learner activity capture useful signal about both the timing and form of help\-seeking; Appendix[J](https://arxiv.org/html/2608.26184#A10)reports window\-size sensitivity, complete feature contributions, and profile distributions\.

## 8\.Discussion

Our findings motivate a central design question: when an AI tutor can see that a learner has been consistently passive, anticipate that a query may be approaching, and estimate that the request will reflect dependent help\-seeking, how should it respond? TutorTrace provides a foundation for educational system designers to build around this question\. By making learner behavior observable and computable in real time, TutorTrace enables a new class of support that guides learners toward the behavioral states where productive AI use can emerge as a natural byproduct rather than as an explicit goal\.

Prior approaches to AI overreliance have emphasized AI literacy\([Ma et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib25)\), restrictions on system use, and guardrails on model responses\([Kazemitabaar et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib21);[Liffiton et al\. 2023](https://arxiv.org/html/2608.26184#bib.bib23);[Hou et al\. 2024](https://arxiv.org/html/2608.26184#bib.bib15);[Kapoor et al\. 2026](https://arxiv.org/html/2608.26184#bib.bib19)\)\. Our findings suggest a complementary direction: supporting the learner’s behavioral process\. In which, rather than asking only how students should be taught to use AI, we ask how can adaptive systems support the behaviors that make productive help\-seeking a likely outcome\.

### 8\.1\.Design Implications

Interpret a query through the behavior that preceded it\.In Window 1, 96\.7% of labeled first queries from Cold Start learners were dependent, compared with 30\.9% for Oriented and 28\.6% for Struggling learners\. Although these associations do not establish causality, they demonstrate that the query alone provides an incomplete account of the learner’s needs\. Tutors should incorporate evidence of prior editing, execution, thinking, and error recovery when deciding how to respond\.

Scaffold the learners behaviors\.Learners who repeatedly query without editing or executing code may benefit from an intervention that prompts a concrete action before further assistance, whereas learners who edit without testing may benefit from being prompted to run and inspect their code\. In our preliminary comparison, behavior\-aware prompting was associated with a reduction in Passive windows from 50\.0% to 20\.7% and increased observable activity between queries\. Because this comparison was nonrandomized, it provides initial evidence rather than a causal estimate\.

Match support to the learner’s demonstrated effort\.The taxonomy suggests that uniform scaffolding is unlikely to serve every learner equally\. A Cold Start learner may require orientation toward an initial step, a Passive learner may need encouragement to act independently, and a learner engaged in extended unsuccessful effort may require more direct support\. The goal is therefore not simply to restrict assistance, but to provide the level and form of support appropriate to the learner’s current behavioral context\.

### 8\.2\.Broader Implications

Toward behaviorally aware learning systems\.This work enables a new class of behaviorally aware systems that respond not only to what learners say, but also to how they have worked leading up to a help\-seeking moment\. By incorporating this process\-level context, such systems can create conditions for learners to succeed by preserving productive struggle, supporting learner agency, and providing scaffolding appropriate to their recent behavior\.

## 9\.Limitations and Future Work

Our data were collected during short introductory programming tasks at a single institution, limiting generalizability across courses, tasks, populations, and learning environments\. The preliminary evaluation compared a morning baseline deployment with an afternoon intervention deployment without random assignment\. Cohort composition, time of day, or other unmeasured factors may therefore contribute to the observed behavioral differences\. Future work should evaluate the taxonomy and interventions across institutions using randomized studies\.

The Window 1 and Window 2 profiles were selected through an exploratory procedure partly informed by task\-completion differences\. Their completion rates should therefore be interpreted as descriptive characteristics rather than independent validation\. Guided and dependent help\-seeking labels were generated by GPT\-4o and validated against two human raters on a subset of queries\. Despite substantial agreement, these labels remain approximations rather than direct measurements of learners’ cognitive engagement\.

Our system also uses a single tutor model, GPT\-4o\. Behavioral patterns and intervention effects may differ across models with different capabilities, response styles, or scaffolding strategies\. Future work should test whether the profiles and prediction tasks remain stable across models\.

Finally, the rule\-based auto\-segmentation pipeline lacks semantic understanding of learner code and intent\. For example, it may classify edits as Debugging while an unresolved error remains even when the learner is implementing an unrelated feature\. Future work should combine behavioral telemetry with lightweight semantic code analysis and test whether reduced Passive re\-querying improves knowledge retention, transfer, and long\-term help\-seeking behavior\.

## 10\.Conclusion

We presented TutorTrace, a dataset and real\-time behavioral abstraction pipeline that makes the context surrounding learner–AI interactions computable\. Across 480 learners in four introductory Python deployments, TutorTrace captures approximately 180K telemetry events, 13,633 auto\-classified behavioral segments, and 27 observable metrics validated against expert labels and student self\-reports\.

We derived a three\-window taxonomy characterizing activity before the first AI query, between consecutive queries, and across the session\. In a preliminary between\-deployment evaluation, behavior\-aware prompting was associated with a reduction in Passive inter\-query windows from 50\.0% to 20\.7%\. Observable metrics also improved held\-out AUROC from 0\.689 to 0\.726 for query imminence and from 0\.690 to 0\.717 for help\-seeking type\.

We release the TutorTrace dataset, taxonomy, classifier, and code through the repository linked in the Introduction to support AI tutors that respond not only to what learners say, but also to what they do\.

## References

- \(1\)
- Aleven et al\.\(2016\)Vincent Aleven, Ido Roll, Bruce M McLaren, and Kenneth R Koedinger\. 2016\.Help helps, but only so much: Research on help seeking with intelligent tutoring systems\.*International Journal of Artificial Intelligence in Education*26, 1 \(2016\), 205–223\.
- Barke et al\.\(2023\)Shraddha Barke, Michael B James, and Nadia Polikarpova\. 2023\.Grounded copilot: How programmers interact with code\-generating models\.*Proceedings of the ACM on Programming Languages*7, OOPSLA1 \(2023\), 85–111\.
- Basha et al\.\(2025\)Manaal Basha, Aimeê M Ribeiro, Jeena Javahar, Cleidson RB De Souza, and Gema Rodríguez\-Pérez\. 2025\.Codewatcher: Ide telemetry data extraction tool for understanding coding interactions with llms\. In*2025 IEEE International Conference on Software Maintenance and Evolution \(ICSME\)*\. IEEE, 935–939\.
- Blikstein \(2011\)Paulo Blikstein\. 2011\.Using learning analytics to assess students’ behavior in open\-ended programming tasks\. In*Proceedings of the 1st International Conference on Learning Analytics and Knowledge*\. 110–116\.
- Bloom \(1984\)Benjamin S Bloom\. 1984\.The 2 sigma problem: The search for methods of group instruction as effective as one\-to\-one tutoring\.*Educational researcher*13, 6 \(1984\), 4–16\.
- Breiman \(2001\)Leo Breiman\. 2001\.Random forests\.*Machine learning*45, 1 \(2001\), 5–32\.
- Chi et al\.\(2001\)Michelene TH Chi, Stephanie A Siler, Heisawn Jeong, Takashi Yamauchi, and Robert G Hausmann\. 2001\.Learning from human tutoring\.*Cognitive Science*25, 4 \(2001\), 471–533\.
- Cohen \(1960\)Jacob Cohen\. 1960\.A coefficient of agreement for nominal scales\.*Educational and Psychological Measurement*20, 1 \(1960\), 37–46\.
- CSEDM Data Challenge Organizers \(2021\)CSEDM Data Challenge Organizers\. 2021\.The 2nd CSEDM Data Challenge\.[https://sites\.google\.com/asu\.edu/csedm\-ws\-lak\-2019/home?authuser=0](https://sites.google.com/asu.edu/csedm-ws-lak-2019/home?authuser=0)CodeWorkout CS1 dataset, Spring and Fall 2019 semesters, in ProgSnap2 format\.
- Edwards and Murali \(2017\)Stephen H\. Edwards and Krishnan Panamalai Murali\. 2017\.CodeWorkout: Short Programming Exercises with Built\-in Data Collection\. In*Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education*\(Bologna, Italy\)*\(ITiCSE ’17\)*\. Association for Computing Machinery, New York, NY, USA, 188–193\.[doi:10\.1145/3059009\.3059055](https://doi.org/10.1145/3059009.3059055)
- Fawcett \(2006\)Tom Fawcett\. 2006\.An introduction to ROC analysis\.*Pattern recognition letters*27, 8 \(2006\), 861–874\.
- Fuller et al\.\(2007\)Ursula Fuller, Colin G Johnson, Tuukka Ahoniemi, Diana Cukierman, Isidoro Hernán\-Losada, Jana Jackova, Essi Lahtinen, Tracy L Lewis, Donna McGee Thompson, Charles Riedesel, et al\.2007\.Developing a computer science\-specific learning taxonomy\.*ACm SIGCSE Bulletin*39, 4 \(2007\), 152–170\.
- Harvey et al\.\(2025\)Emma Harvey, Allison Koenecke, and Rene F Kizilcec\. 2025\." Don’t Forget the Teachers": Towards an Educator\-Centered Understanding of Harms from Large Language Models in Education\. In*Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems*\. 1–19\.
- Hou et al\.\(2024\)Xinying Hou, Zihan Wu, Xu Wang, and Barbara J Ericson\. 2024\.Codetailor: Llm\-powered personalized parsons puzzles for engaging support while learning programming\. In*Proceedings of the Eleventh ACM Conference on Learning@ Scale*\. 51–62\.
- Hubert and Arabie \(1985\)Lawrence Hubert and Phipps Arabie\. 1985\.Comparing partitions\.*Journal of Classification*2, 1 \(1985\), 193–218\.[doi:10\.1007/BF01908075](https://doi.org/10.1007/BF01908075)
- Irvine \(2021\)Jeff Irvine\. 2021\.Taxonomies in education: Overview, comparison, and future directions\.*Journal of Education and Development*5, 2 \(2021\), 1\.
- Jošt et al\.\(2024\)Gregor Jošt, Viktor Taneski, and Sašo Karakatič\. 2024\.The impact of large language models on programming education and student learning outcomes\.*Applied Sciences*14, 10 \(2024\), 4115\.
- Kapoor et al\.\(2026\)Amanpreet Kapoor, Paul Denny, Leo Porter, Stephen MacNeil, and Marc Diaz\. 2026\.Exploring Student Behaviors and Motivations when using AI Teaching Assistants with Optional Guardrails\. In*Proceedings of the 28th Australasian Computing Education Conference*\. 22–31\.
- Kapur \(2014\)Manu Kapur\. 2014\.Productive failure in learning math\.*Cognitive Science*38, 5 \(2014\), 1008–1022\.
- Kazemitabaar et al\.\(2024\)Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig, and Tovi Grossman\. 2024\.Codeaid: Evaluating a classroom deployment of an llm\-based programming assistant that balances student and educator needs\. In*Proceedings of the 2024 chi conference on human factors in computing systems*\. 1–20\.
- Landis and Koch \(1977\)J Richard Landis and Gary G Koch\. 1977\.The measurement of observer agreement for categorical data\.*biometrics*\(1977\), 159–174\.
- Liffiton et al\.\(2023\)Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny\. 2023\.Codehelp: Using large language models with guardrails for scalable support in programming classes\. In*Proceedings of the 23rd Koli calling international conference on computing education research*\. 1–11\.
- Liu et al\.\(2026\)Chang Liu, Qinyi Zhou, Xinjie Shen, Xingyu Bruce Liu, Tongshuang Wu, and Xiang ‘Anthony’ Chen\. 2026\.Behavioral Indicators of Overreliance During Interaction with Conversational Language Models\. In*Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems*\(Barcelona, Spain\)*\(CHI ’26\)*\. Association for Computing Machinery, New York, NY, USA, Article 790, 23 pages\.[doi:10\.1145/3772318\.3790332](https://doi.org/10.1145/3772318.3790332)
- Ma et al\.\(2026\)Qianou Ma, Kenneth R Koedinger, and Tongshuang Wu\. 2026\.Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science\. In*Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems**\(CHI ’26\)*\. Association for Computing Machinery, New York, NY, USA, Article 139, 22 pages\.[doi:10\.1145/3772318\.3791283](https://doi.org/10.1145/3772318.3791283)
- Marwan et al\.\(2020\)Samiha Marwan, Anay Dombe, and Thomas W\. Price\. 2020\.Unproductive Help\-seeking in Programming: What it is and How to Address it\. In*Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education*\(Trondheim, Norway\)*\(ITiCSE ’20\)*\. Association for Computing Machinery, New York, NY, USA, 54–60\.[doi:10\.1145/3341525\.3387394](https://doi.org/10.1145/3341525.3387394)
- McQueen \(1967\)James B McQueen\. 1967\.Some methods of classification and analysis of multivariate observations\. In*Proc\. of 5th Berkeley Symposium on Math\. Stat\. and Prob\.*281–297\.
- Mozannar et al\.\(2024\)Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz\. 2024\.Reading between the lines: Modeling user behavior and costs in AI\-assisted programming\. In*Proceedings of the 2024 CHI conference on human factors in computing systems*\. 1–16\.
- Price et al\.\(2020\)Thomas W Price, David Hovemeyer, Kelly Rivers, Ge Gao, Austin Cory Bart, Ayaan M Kazerouni, Brett A Becker, Andrew Petersen, Luke Gusukuma, Stephen H Edwards, et al\.2020\.Progsnap2: A flexible format for programming process data\. In*Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education*\. 356–362\.
- Roll et al\.\(2011\)Ido Roll, Vincent Aleven, Bruce M McLaren, and Kenneth R Koedinger\. 2011\.Improving students’ help\-seeking skills using metacognitive feedback in an intelligent tutoring system\.*Learning and Instruction*21, 2 \(2011\), 267–280\.
- Rousseeuw \(1987\)Peter J Rousseeuw\. 1987\.Silhouettes: A graphical aid to the interpretation and validation of cluster analysis\.*J\. Comput\. Appl\. Math\.*20 \(1987\), 53–65\.
- Shen and Tamkin \(2026\)Judy Hanwen Shen and Alex Tamkin\. 2026\.How AI Impacts Skill Formation\.arXiv:2601\.20245 \[cs\.CY\][https://arxiv\.org/abs/2601\.20245](https://arxiv.org/abs/2601.20245)
- Sinha and Kapur \(2021\)Tanmay Sinha and Manu Kapur\. 2021\.When problem solving followed by instruction works: Evidence for productive failure\.*Review of Educational Research*91, 5 \(2021\), 761–798\.
- Smith et al\.\(2025\)Julie M\. Smith, Jacob Koressel, Sofía De Jesús, Joe Kmoch, and Bryan Twarek\. 2025\.Comparing Learning Taxonomies With Computer Science K\-12 Standards\. In*Annual Meeting of the American Educational Research Association*\. American Educational Research Association\.[doi:10\.3102/2190401](https://doi.org/10.3102/2190401)
- Tang et al\.\(2025\)Xiaohang Tang, Sam Wong, Marcus Huynh, Zicheng He, Yalong Yang, and Yan Chen\. 2025\.SPHERE: Supporting Personalized Feedback at Scale in Programming Classrooms with Structured Review of Generative AI Outputs\. In*Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems**\(CHI EA ’25\)*\. Association for Computing Machinery, New York, NY, USA, Article 467, 17 pages\.[doi:10\.1145/3706599\.3720203](https://doi.org/10.1145/3706599.3720203)
- Wermelinger \(2023\)Michel Wermelinger\. 2023\.Using github copilot to solve simple programming problems\. In*Proceedings of the 54th ACM Technical Symposium on Computer Science Education V\. 1*\. 172–178\.
- Wood et al\.\(1976\)David Wood, Jerome S Bruner, and Gail Ross\. 1976\.The role of tutoring in problem solving\.*Journal of Child Psychology and Psychiatry*17, 2 \(1976\), 89–100\.
- Xue et al\.\(2024\)Yuankai Xue, Hanlin Chen, Gina R Bai, Robert Tairas, and Yu Huang\. 2024\.Does ChatGPT help with introductory programming? An experiment of students using ChatGPT in CS1\. In*Proceedings of the 46th International conference on software engineering: software engineering education and training*\. 331–341\.
- Zhang et al\.\(2026\)Ashley Ge Zhang, Yan\-Ru Jhou, Yinuo Yang, Shamita Rao, Maryam Arab, Yan Chen, and Steve Oney\. 2026\.Editrail: Understanding AI Usage by Visualizing Student\-AI Interaction in Code\.*arXiv preprint arXiv:2601\.20085*\(2026\)\.

## Appendix ABehavioral Labeling Interface

![Screenshot of the labeling tool with a session replay, an annotatable timeline, and the behavioral codebook.](https://arxiv.org/html/2608.26184v1/figures/BehavioralLabelingInterface.png)Figure 4\.The Behavioral Labeling Interface used by expert labelers to annotate student sessions\. The interface presents a session replay, a timeline for placing segment boundaries, and the behavioral codebook for classification\.Screenshot of the labeling tool with a session replay, an annotatable timeline, and the behavioral codebook\.
## Appendix BCodebook Change History

The behavioral codebook was iteratively refined across 10 pilot labeling sessions through weekly reconciliation meetings\. Table[9](https://arxiv.org/html/2608.26184#A2.T9)documents each revision with the corresponding date and specific change\.

Table 9\.Codebook revision history across the pilot labeling phase, derived from platform configuration changes\.
## Appendix CTelemetry Event Schema

Table 10\.Full telemetry event schema with example payloads from live deployments\. Each event includes a millisecond timestamp, source region, and event\-specific payload\. Long event names and payloads are abbreviated for space\.Event TypeDescriptionExample PayloadCode Editor \(11 types\)CODE\_TYPETyped characters\{code:"l", changes:\[\{to:0, from:0, text:"l"\}\], raw\_action:"input\.type"\}CODE\_DELETEBackspace/Delete\{code:"\.\.\.grade\_boo", changes:\[\{to:101, from:100, text:""\}\], raw\_action:"delete\.backward"\}CODE\_DELETE\_SEL\.Deleted selection\{code:"", changes:\[\{to:145, from:0, text:""\}\], raw\_action:"delete\.selection"\}CODE\_PASTEPasted content\{code:"\.\.\.append\(\[78,84,91\]\)", changes:\[\{to:75, from:75, text:"\[78,84,91\]"\}\], raw\_action:"input\.paste"\}CODE\_COPYCopied to clipboard\{selected\_text:"grade\_book = \[\[88,92,75\],\.\.\.\]"\}CODE\_CUTCut to clipboard\{code:"", changes:\[\{to:186, from:0, text:""\}\], raw\_action:"delete\.cut"\}CODE\_UNDOCtrl\+Z\{code:"\.\.\.append\(\)", changes:\[\{to:88, from:76, text:""\}\], raw\_action:"undo"\}CODE\_REDOCtrl\+Y\{changes:\[\.\.\.\], raw\_action:"redo"\}CODE\_SELECTHighlighted text\{selected\_text:"print\(grade\_book\)"\}CODE\_INDENTTab key\{code:"\.\.\.for grade in grade\_book:\\n ", changes:\[\{to:96, from:96, text:" "\}\], raw\_action:"input\.indent"\}CODE\_UNKNOWNUnrecognized edit\{code:"\.\.\.li", changes:\[\{to:57, from:57, text:"i"\}\], raw\_action:"input\.type\.compose"\}Terminal \(6 types\)TERMINAL\_RUNClicked Run Tests\{code\_length:180\}TERMINAL\_OUTPUTstdout printed\{output:"95"\}TEST\_CASE\_RESULTTest results\{passed\_count:2, total\_tests:3, passrate:66\.7, detailed\_results:\[\{passed:true, test\_name:"Test Case 1"\}, \.\.\.\]\}TERMINAL\_ERRORRuntime error\{type:"UnknownError", message:"Traceback\.\.\.IndexError: list assignment index out of range"\}TERMINAL\_SELECTHighlighted output\{selected\_text:"IndexError: list assignment index out of range"\}TERMINAL\_COPYCopied output\{selected\_text:"IndexError: list assignment index out of range"\}Chat \(9 types\)CHAT\_TYPETyped in chat\{text:"h"\}CHAT\_DELETEDeleted in chat\{key\_used:"Backspace", text\_before:"\.\.\.command fr", text\_after:"\.\.\.command f", cursor\_position:34\}CHAT\_PASTEPasted into chat\{pasted\_text:"IndexError: list assignment index out of range"\}CHAT\_QUERYSent message to AI\{text:"hi", length:2\}CHAT\_RESPONSEAI response\{content:"Great start\! Let’s tackle this step by step\.\.\.", length:322, latency\_ms:2495\}CHAT\_SELECT\_INPUTHighlighted input\{selected\_text:"what would code be"\}CHAT\_SELECT\_HIST\.Highlighted history\{selected\_text:"what should I do?"\}CHAT\_COPY\_INPUTCopied from input\{selected\_text:"\.\.\."\}CHAT\_COPY\_HIST\.Copied from history\{selected\_text:"what should I do?"\}Task / Tests \(4 types\)TASK\_SELECTHighlighted task\{selected\_text:"\[78, 84, 91"\}TASK\_COPYCopied from task\{selected\_text:"\[78, 84, 91\]"\}TEST\_SELECTHighlighted test\{selected\_text:"New"\}TEST\_COPYCopied from test\{selected\_text:"Print all scores"\}Global \(5 types\)MOUSE\_CLICKClicked in workspace\{x:0\.588, y:0\.183, region:"CODE\_EDITOR", tag:"DIV", className:"cm\-content"\}MOUSE\_MOVEMouse position \(50ms\)\{x:0\.384, y:0\.272, region:"CODE\_EDITOR"\}TAB\_STATEBrowser tab switch\{visible:false\}WINDOW\_RESIZEBrowser resized\{width:1588, height:901\}PANEL\_RESIZEPanel resized\{panel:"left", newSize:456\}Session \(2 types\)SESSION\_STARTEntered workspace\{window\_width:1588, window\_height:901\}SESSION\_ENDLeft workspace\{duration\_ms:542000\}
## Appendix DLLM System Prompt

The system prompt provided to GPT\-4o for every student interaction is shown in Figure[5](https://arxiv.org/html/2608.26184#acmlabel6)\. The prompt implements the three\-level scaffolding framework described in Section[3\.2](https://arxiv.org/html/2608.26184#S3.SS2)\. The behavioral context blocks referenced in its final paragraph are reproduced in Appendix[G](https://arxiv.org/html/2608.26184#A7)\. These blocks could override the default escalation order by directing the model to begin at a higher scaffolding level, but did not override the prohibition against providing runnable code\.

Role:You are a supportive programming tutor embedded in an educational coding environment\. Your role is to help students develop problem\-solving skills through guided inquiry, not to solve problems for them\.

Task Context:You will be provided with the task description, the student’s current code, their test results, and the conversation history\. Use the conversation history to track escalation level and whether the student has made progress\.

Core Approach — Three\-Level Scaffolding:

Level 1 — Socratic Questioning \(default\):Ask one targeted question that directs the student’s attention to the relevant part of their code or problem\. Do not provide hints at this level\.

Level 2 — Conceptual Hint:Name the relevant concept or method without showing syntax, then ask a follow\-up question\.

Level 3 — Concrete Scaffolding:Provide a pseudocode skeleton with blanks for the current step only\. Never show the structure of the full solution\. All specific values, indices, and strings must be replaced with\_\_\_\.

Absolute Rules:

1. \(1\)Never write runnable code in any programming language\. When showing patterns, use only pseudocode with blank placeholders \(\_\_\_\)\.
2. \(2\)Never scaffold more than one step at a time\.
3. \(3\)Never skip escalation levels\. Escalate exactly one level at a time\.
4. \(4\)When the student moves to a new step or concept, always reset to Level 1\.

Escalation Rules:Start at Level 1 for each new concept\. After each response, assess whether the student made progress\. If the current level did not result in progress, escalate one level\. Only one escalation per response\. If the student has no code and expresses uncertainty, begin at Level 2 with a conceptual hint about the first step only\. When the student completes a step and moves to a new concept, reset to Level 1\.

Response Guidelines:Keep responses concise \(2–4 sentences plus one question\)\. Ask only one question per response\. Reference specific lines, variables, or test failures from the student’s actual code\.

Behavioral Context Override:If a behavioral context block is present, it describes what the student was doing before they asked\. Use it to adjust scaffolding approach and escalation pace\. The behavioral context never overrides the rule against writing runnable code; it only changes how scaffolding is delivered \(e\.g\., starting at a higher level, trying a different angle, acknowledging effort\)\.

Figure 5\.The LLM tutoring system prompt provided to GPT\-4o for every student interaction, implementing three escalating levels of scaffolding with a behavioral context override\.The complete tutoring system prompt, laid out in two columns\. It defines the tutor role, the task context provided at each interaction, and three escalating scaffolding levels: Socratic questioning, conceptual hint, and concrete scaffolding with pseudocode blanks\. It then lists absolute rules forbidding runnable code and multi\-step scaffolds, escalation rules for moving one level at a time and resetting on new concepts, response guidelines limiting length and questions, and a behavioral context override explaining how appended behavior blocks adjust scaffolding without overriding the code prohibition\.
## Appendix EFoundation Dataset Structure

Table[11](https://arxiv.org/html/2608.26184#A5.T11)summarizes the abstraction layers of the released foundation dataset and its overall scale\.

Table 11\.Foundation dataset structure\. Layers A–C comprise the TutorTrace foundation; Layer D is derived in the TutorTrace Taxonomy\.LayerRepresentationScaleA: Raw TelemetryTimestamped IDE events∼\{\\sim\}180KB: Observable MetricsEffort intensity per window27 featuresC: Behavioral SequencesAuto\-classified segments13,633D: Behavioral ProfilesClustered profiles10 profilesStudent sessions480AI interactions1,386Queries840
## Appendix FAuto\-Segmentation Algorithm

Algorithm[1](https://arxiv.org/html/2608.26184#alg1)presents the seven\-step auto\-segmentation pipeline that translates raw telemetry events into labeled behavioral sequences\. Each step corresponds to a rule derived from expert\-labeler consensus during codebook development \(Section[4\.1](https://arxiv.org/html/2608.26184#S4.SS1)\)\.

Algorithm 1Auto\-Behavioral Classification Pipeline1:Events

E=\[e1,…,en\]E=\[e\_\{1\},\\ldots,e\_\{n\}\]with timestamps relative to session start; session duration

DD
2:Ordered list of behavioral segments

SS, each with behavior label and thinking subtype

3:Step 1: Build Major Segments

4:Classify each event by category:Code,Terminal,ChatInput,ChatResponse

5:Group consecutive same\-category events into segments

6:iftwo consecutiveCodeevents are separated by

≥6\\geq 6s, split into two segments

7:Assign initial behavior:Code

→\\toImplementing,Terminal

→\\toTesting,ChatInput

→\\toSeeking Help,ChatResponse

→\\toThinking

8:Step 2: Fill Gaps with Thinking

9:foreach gap between consecutive segmentsdo

10:ifgap

≥3\\geq 3sthen

11:Insert Thinking segment spanning the gap

12:else

13:Extend the preceding segment to close the gap

14:endif

15:endfor

16:Apply same logic to gaps before the first segment and after the last segment

17:Step 3: Merge Short Testing Segments

18:foreach Testing segment with duration

<1\.5<1\.5sdo

19:Absorb into the adjacent segment \(prefer next; fall back to previous\)

20:endfor

21:Step 4: Absorb Pre\-Query Pauses

22:foreach Thinking segment of duration

≤5\\leq 5sdo

23:ifpreceded by Implementing or Debuggingandfollowed by Seeking Helpthen

24:Store pause duration as metadata on the Seeking Help segment

25:Extend the preceding segment to cover the pause

26:Remove the Thinking segment

27:endif

28:endfor

29:Step 5: Apply Error State

30:Maintain flag

hasUnresolvedError←false\\textit\{hasUnresolvedError\}\\leftarrow\\textsc\{false\}
31:foreach segment in chronological orderdo

32:ifsegment containsTERMINAL\_ERRORor a failedTEST\_CASE\_RESULTthen

33:

hasUnresolvedError←true\\textit\{hasUnresolvedError\}\\leftarrow\\textsc\{true\}
34:endif

35:ifsegment contains aTEST\_CASE\_RESULTwhere all tests passthen

36:

hasUnresolvedError←false\\textit\{hasUnresolvedError\}\\leftarrow\\textsc\{false\}
37:endif

38:ifsegment is ImplementingandhasUnresolvedErrorthen

39:Relabel segment as Debugging

40:endif

41:endfor

42:Step 6: Classify Thinking Subtypes

43:foreach Thinking segment in chronological orderdo

44:ifno code, terminal, or chat activity has occurred yetthen

45:Label asThinking: Task⊳\\trianglerightReading the task description

46:elseifpreceding segment was Seeking Helpthen

47:Label asThinking: Response⊳\\trianglerightReading AI reply

48:elseifmost recent terminal run produced an unresolved errorthen

49:Label asThinking: Error⊳\\trianglerightReading an error message

50:else

51:Label asThinking: Code⊳\\trianglerightReviewing own code

52:endif

53:endfor

54:Step 7: Post\-Process

55:Fix any remaining unlabeled segments \(terminal events

→\\toTesting; otherwise

→\\toThinking\)

56:Merge consecutive segments with the same behavior label

57:Re\-index all segment IDs

58:return

SS

## Appendix GBehavior\-Aware Intervention Prompts

In the treatment condition \(Section[7\.3](https://arxiv.org/html/2608.26184#S7.SS3)\), the system appended one of five behavior\-aware context blocks to the student’s query\. Cold Start was used for a learner’s first query when no code edit or terminal run had occurred\. For subsequent queries, the system selected among Passive, Iterating, Debugging, and Spinning using online activity rules derived during system development\. These deployment\-time rules used the same profile names as the later taxonomy but represented precursor heuristic states rather than assignments from the revised clustering procedure\. The prompts below reproduce the exact intervention text used during deployment\. The revised taxonomy was subsequently applied to both deployments to compare their Window 2 profile distributions\.

Cold Start

This is the student’s first time asking for help\. They have not written or run any code yet\. Give them something concrete to start with — a pseudocode scaffold with blanks to fill in\. End your response with a single sentence starting with “Try:” that gives them that pseudocode starting point so they know what to write next\.

Passive

This student received help but has not written or run any code since\. Try a different angle from your last response — do not repeat the same explanation\. Do NOT escalate your scaffolding level — stay at the same level but approach it differently\. If the student asks “how to fix” or requests the answer directly, acknowledge, redirect, and re\-ask with pseudocode\. If the student has just completed a step and is asking what to do next, start at Level 1 for the new step\. If the question is purely conceptual, answer it naturally\. Otherwise, end your response with a single sentence starting with “Try:” that gives them one concrete next action using pseudocode with blanks if code is involved\.

Iterating

This student wrote some code but has not tested it\. Do not give more explanation\. If their current code would produce meaningful output \(e\.g\. contains a print statement, an expression, or enough logic to show a result\), end your response with a single sentence starting with “Try:” telling them to run it\. If running it would produce no output, end with a “Try:” that asks them to add a specific print statement first so they can see what their code is doing\.

Debugging

This student has been coding, testing, and hitting errors\. They are engaged and struggling\. Recognize their effort\. Name the specific error and the misconception behind it so they can read errors independently\. Escalate to Level 2 or 3 immediately\. End your response with a single sentence starting with “Try:” giving them the targeted fix as a pseudocode pattern with blanks\.

Spinning

This student has run their code many times and keeps hitting errors\. They do not need encouragement — they need to be unblocked\. Name the concept or method they need, explain briefly why their current approach fails, and escalate to Level 3 immediately\. End your response with a single sentence starting with “Try:” that gives them the precise pseudocode pattern they need, with blanks for the specific values\.

## Appendix HHelp\-Seeking Type Labeling Prompt

The system prompt shown in Figure[6](https://arxiv.org/html/2608.26184#acmlabel7)was provided to GPT\-4o to classify each student query as guided or dependent\. The model received the student’s query, current code, and preceding chat history\. These semantic inputs were used only to generate the offline help\-seeking labels and were excluded from the prediction models described in Section[7\.4](https://arxiv.org/html/2608.26184#S7.SS4)\.

You are an educational assistant analyzing student help requests in introductory programming courses\.

Given a student query, their current code, and chat history, classify the query as eitherGUIDEDorDEPENDENT\.

GUIDED —The student demonstrates independent thinking\. They have identified what they need help with and are actively steering their learning\. This includes:

- •Asking a specific question about a concept \(“What is a nested loop?”\)
- •Identifying a specific problem or confusion \(“I’m not sure what to put in the print statement”\)
- •Describing what they tried and what went wrong \(“I tried using a for loop but it only prints the first item”\)
- •Asking how to approach a specific step \(“How do I iterate through each student’s grades?”\)
- •Requesting clarification on a specific point from a prior AI response \(“What do you mean by iterating over the inner list?”\)
- •Answering the AI’s question with specific information \(“The error is IndexError on line 5”\)

The key indicator: the student has done some cognitive work to formulate what they need\. The query communicates a specific need, question, or confusion, even if brief\.

DEPENDENT —The student is offloading cognitive work to the AI with minimal independent effort\. This includes:

- •Pasting code with no question or description of the problem \(implicit “fix this for me”\)
- •Vague requests with no specifics \(“help,” “it doesn’t work,” “idk”\)
- •Pure acknowledgments that delegate next steps \(“ok do that,” “yeah,” “sure,” “go ahead”\)
- •Empty or near\-empty messages \(“?,” stray characters\)
- •Requests that ask the AI to do the work \(“can you just write it for me,” “give me the code”\)
- •Repeating the assignment prompt or pasting instructions without any attempt or question
- •Answering the AI’s question with no effort \(“idk,” “I don’t know,” “you tell me”\)

The key indicator: the student has not done cognitive work to identify what they need\. They are asking the AI to do the thinking for them\.

Boundary cases — apply these rules:

- •“I don’t know how to do X”→\\rightarrowGUIDED\(they identified what they do not know\)
- •“I don’t know” or “I’m stuck” alone, with no specifics→\\rightarrowDEPENDENT
- •“Is this right?” with code→\\rightarrowDEPENDENT
- •“Is this right? I’m not sure if my loop handles the last element”→\\rightarrowGUIDED
- •Code pasted with “Why does this print the whole list instead of individual scores?”→\\rightarrowGUIDED
- •Code pasted with no question→\\rightarrowDEPENDENT
- •“What about edge cases?” in the context of an ongoing conversation→\\rightarrowGUIDED
- •“Ok,” “Thanks,” or “Got it” alone→\\rightarrowDEPENDENT
- •“Ok, but how does that work with nested lists?”→\\rightarrowGUIDED

Confidence:

- •high: clearly guided or clearly dependent
- •medium: leans one way but has some ambiguity
- •low: genuinely on the boundary

Return only a JSON object:

\{ "queryEngagement": "guided" or "dependent", "rationale": "1 sentence explaining why", "confidence": "high/medium/low" \}

Figure 6\.The help\-seeking type labeling prompt provided to GPT\-4o to classify each student query as guided or dependent from the query text, the student’s code at submission time, and the preceding chat history\.The complete help\-seeking classification prompt, laid out in two columns\. It instructs the model to label each student query as guided or dependent, defines guided queries as demonstrating independent thinking with bulleted examples, defines dependent queries as offloading cognitive work with bulleted examples, lists explicit boundary\-case rules mapping ambiguous phrasings to labels, defines high, medium, and low confidence, and specifies a JSON output object with the label, a one\-sentence rationale, and a confidence level\.
## Appendix ITask Descriptions

Full task descriptions and test cases are reproduced below as presented to students\.

Students were given a pre\-populated list of song names and asked to complete four list manipulation steps within 15 minutes\.

Deployments 1–2: Playlist \(Python, 15 min\)

You are given a list of song names calledplaylist\. Complete the following steps:

1. \(1\)The song at index 4 was added by mistake\. Replace it with “Purple Rain”\.
2. \(2\)Remove the last 3 songs from the playlist using slicing\.
3. \(3\)Reverse the playlist in place\.
4. \(4\)Print every other song in the playlist using slicing with a step value\.

Test cases:

1. 1\."Purple Rain" in playlist→\\toTrue
2. 2\.len\(playlist\)→\\to5
3. 3\.playlist\[0\]→\\to"Purple Rain"
4. 4\.playlist\[::2\]→\\to\["Purple Rain", "Imagine", "Bohemian Rhapsody"\]

Students were given a nested list of test scores and asked to complete three list manipulation steps within 10 minutes\.

Deployments 3–4: Grade Book \(Python, 10 min\)

You are given a list of lists calledgrade\_book, where each inner list contains a student’s test scores\. Complete the following steps:

1. \(1\)A new student joined the class with scores\[78, 84, 91\]\. Add their scores tograde\_book\.
2. \(2\)The first student’s third score was entered incorrectly\. Update it to80\.
3. \(3\)Using a nested for loop, print every individual score ingrade\_book, one per line\.

Test cases:

1. 1\.len\(grade\_book\)→\\to4
2. 2\.grade\_book\[3\]→\\to\[78, 84, 91\]
3. 3\.grade\_book\[0\]\[2\]→\\to80
4. 4\.Nested loop prints all 12 scores, one per line\.

## Appendix JPrediction Breakdowns: Observation Windows, Feature Contributions, and Profiles

Table 12\.Held\-out AUROC across observation\-window sizes for both prediction tasks and all three feature layers\. The 30\-second query\-imminence row and 15\-second help\-seeking\-type row correspond to Table[7](https://arxiv.org/html/2608.26184#S7.T7); bold marks the configuration reported in the subsequent tables\.Table 13\.Complete feature\-contribution list \(Random\-Forest importances, all features with importance\>0\>0\) for query imminence at its reported configuration \(30 s window, observable layer\)\.Table 14\.Complete feature\-contribution list \(Random\-Forest importances, all features with importance\>0\>0\) for help\-seeking type at its reported configuration \(15 s window, observable layer\)\.Table 15\.Distribution of behavioral contexts among held\-out query\-imminence observation windows at the reported window \(30 s\), with each context’s positive rate\. Profiles are evaluation strata from the deployment\-1–2 taxonomy models\.Table 16\.Distribution of behavioral profiles among held\-out help\-seeking\-type queries at the reported window \(15 s\), with each profile’s guided rate\.