Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration

# Principal Trait Analysis: Towards Deriving “Skills” in Human-AI Collaboration
Source: [https://arxiv.org/html/2608.11460](https://arxiv.org/html/2608.11460)
Kai DuAffiliation:OpenRefinery\.aiEmail:[kai@openrefinery\.ai](mailto:)Andrew LanAffiliation:University of Massachusetts AmherstEmail:[andrewlan@cs\.umass\.edu](mailto:)

###### Abstract

Large Language Model \(LLM\)\-powered coding agents are increasingly used in professional settings to write software more rapidly, via human\-artificial intelligence \(AI\) collaboration\. In this new era of work, it is important to understand what kinds of human behavior traits contribute to task success\. Moreover, we need to uncover keyskillsrequired for modern professionals, which can inform educators on how to foster the development of these skills among students\. Existing guidelines for human\-AI collaboration are built from either top\-down theory or empirical, context\-specific observations of human\-AI interactions\. However, since LLM capabilities are rapidly improving, theory may not be able to explain emerging interaction patterns, while empirical guidelines may become obsolete very quickly\. In this work, we explore an automated, data\-driven approach to uncover patterns, which we termtraits, of effective human\-AI interaction that are aligned with collaborative coding taskoutcomes\. We propose Principal Trait Analysis \(PTA\), a Principal Component Analysis\-inspired algorithm for deriving common traits from patterns in LLM conversations\. Our algorithm uses a series of LLM prompting stages to analyze large corpora of human\-AI collaborative session traces, deriving common traits across the dataset and scoring each human collaborator on how well their usage style aligns with these traits\. While unsupervised and data\-driven, the approach also allows domain expertise to be injected during trait discovery\. We select the most distinguishing traits to be those that exhibit highest variance across collaborators\. We evaluate PTA on two human\-AI collaborative coding datasets, including an educational setting \(students working with an AI tutor\) and a professional setting \(developers working with an AI coding agent\)\. We find that PTA\-derived traits are significant in explaining collaborator behavior across both settings and that they can help predict task outcomes\. However, whether some can qualify as ‘‘skills’’ remains to be seen, due to inconclusive results on generalizability and how user traits evolve over time\.111Early stage, preliminary work\. A principled version of this paper is forthcoming\.

## 1Introduction

LLM coding assistants are increasingly used in software development and integrated into computer science education, yet it is still unclear what distinguishes effective from ineffective use in either setting\. The same model, given to two software developers, can produce vastly different code depending on how each developer interacts with the model, and the productivity gains from coding assistants vary substantially across developers\[[15](https://arxiv.org/html/2608.11460#bib.bib15)\]\. Likewise, two students can have vastly different usage patterns which result in different learning experiences and subsequent outcomes\[[16](https://arxiv.org/html/2608.11460#bib.bib16),[17](https://arxiv.org/html/2608.11460#bib.bib17)\]\. Uncovering these usage patterns and relating them to outcomes would give i\) developers guidelines on how to collaborate with coding agents and ii\) instructors guidelines on how to teach “AI use skills” to computer science students\.

However, uncovering these patterns and establishing universal guidelines is difficult, since the space of relevant behavioral traits is large and rapidly changing as LLM capabilities improve\. As a result, prescriptive guidelines derived by hand from a snapshot of interactions quickly become stale and may not generalize across settings\. To keep pace with the rapid changes in this field, more automatic and data\-driven analyses should be explored, so that new traits can be re\-derived as human\-AI collaboration evolves into new settings and scenarios\.

Prior approaches to establishing such guidelines have largely been deductive\. Top\-down work imports frameworks from learning theory, human\-AI collaboration, or software engineering practice and applies them as fixed rubrics\[[39](https://arxiv.org/html/2608.11460#bib.bib39)\]\. Such rubrics are interpretable and theoretically grounded, but they are fixed at design time and can be brittle when tools and settings change, especially as AI becomes more capable at a rapid pace; measuring the competencies or skills they define is itself an open problem\[[13](https://arxiv.org/html/2608.11460#bib.bib13)\]\. Bottom\-up work instead analyzes interaction logs directly, through qualitative coding workflows\[[44](https://arxiv.org/html/2608.11460#bib.bib44)\]or dialogue\-act taxonomies applied to tutoring dialogue\[[41](https://arxiv.org/html/2608.11460#bib.bib41),[43](https://arxiv.org/html/2608.11460#bib.bib43),[42](https://arxiv.org/html/2608.11460#bib.bib42)\]and to student\-AI conversations\[[1](https://arxiv.org/html/2608.11460#bib.bib1)\], with more recent efforts mining such logs with LLMs in an open\-ended fashion\[[45](https://arxiv.org/html/2608.11460#bib.bib45)\]\. These analyses stay close to observed behavior, but their annotation schemes are labor\-intensive to build and maintain, and the resulting behavioral categories are only sometimes tied to measurable outcomes\. Therefore, they cannot be reliably characterized as skills either: in educational research literature on knowledge components, a skill is expected to improve with practice along a learning curve and generalize across tasks\[[18](https://arxiv.org/html/2608.11460#bib.bib18)\]\. In this line of work, candidate skill decompositions are validated by how well they fit learning curves\[[19](https://arxiv.org/html/2608.11460#bib.bib19)\]\.

![Refer to caption](https://arxiv.org/html/2608.11460v1/figures/pta_v1_narrow.png)Figure 1:PTA visualized stage\-by\-stage\. Our approach first derives a large set of observations from a conversation corpus which are iteratively refined into a smaller set of candidate traits\. Each session of the corpus is scored by the traits which are then used to derive a smaller set of distinct, principal traits\.### 1\.1Contributions

In this work, we propose a data\-driven algorithm to derive informativetraitsin human\-AI collaborative coding sessions and use this approach to analyze two real\-world conversation datasets\. Our approach,Principal Trait Analysis \(PTA\), is a principal component analysis \(PCA\)\-inspired pipeline that extracts behavior observations from raw conversations, derives interpretable trait dimensions by clustering, scores each human collaborator on every trait, and selects the most distinguishing traits\. We extract these traits through a series of unsupervised stages, via LLM\-based prompting and text\-embedding\-based clustering steps, to uncover traits that explain common collaborator behavior patterns\. This process intuitively mirrors how PCA derives principal component vectors that explain the most amount of variance in datasets\. We then perform an outcome\-aligned evaluation that measures whether the derived traits explain and predict downstream outcomes\. We conduct experiments on two datasets: a student\-AI tutor dialogue corpus \(StudyChat\) and a professional coder\-AI coding agent interaction session trace corpus \(SWE\-Chat\), showing both shared and setting\-specific structure in effective LLM use\. We find that our approach yield traits that are often significant in explaining outcomes\. However, we caution that these traits may not fit the definition of skills, due to limited generalizability across collaborative settings\.

## 2Related Work

### 2\.1Topic Modeling and Clustering with LLMs

LLMs have been widely applied to derive interpretable structure from text corpora, and have been evaluated directly against classical topic models such as LDA\[[22](https://arxiv.org/html/2608.11460#bib.bib22)\]\. Prompt\-based topic modeling pipelines generate and refine natural\-language topic labels\[[25](https://arxiv.org/html/2608.11460#bib.bib25),[36](https://arxiv.org/html/2608.11460#bib.bib36)\], and goal\-conditioned variants discover structure relevant to a stated analysis purpose\[[26](https://arxiv.org/html/2608.11460#bib.bib26),[29](https://arxiv.org/html/2608.11460#bib.bib29)\]\. Closer to our derivation stages, other work uses LLM feedback to guide the clustering itself, both in resolving ambiguous assignments and in choosing an appropriate granularity\[[23](https://arxiv.org/html/2608.11460#bib.bib23)\], or generates labels inductively from the data before hierarchically clustering them into higher\-level themes\[[24](https://arxiv.org/html/2608.11460#bib.bib24)\]\. Related work also proposes methods of visualizing these derived structures in dashboards for monitoring and manual analysis of text corpora\[[31](https://arxiv.org/html/2608.11460#bib.bib31),[28](https://arxiv.org/html/2608.11460#bib.bib28)\]\. Large scale chat datasets, in particular, have been a growing area of research interest, as LLM\-collaborative workflows are being used across a wider variety of tasks and large\-scale datasets of LLM conversations are made available\[[2](https://arxiv.org/html/2608.11460#bib.bib2),[12](https://arxiv.org/html/2608.11460#bib.bib12)\]\. Therefore similar analysis pipelines are now being applied to mine insights from chat datasets\[[30](https://arxiv.org/html/2608.11460#bib.bib30),[31](https://arxiv.org/html/2608.11460#bib.bib31)\]\. PTA’s trait\-derivation stages build on this machinery, using LLMs to derive observations from chat datasets, conditioned on the collaborative context and optionally a theory lens\.

### 2\.2Outcome Prediction with LLM Judged Features

A growing body of work uses LLM\-scored features as inputs to downstream predictive tasks, building on prior work that used embedding\-based approaches to predict outcomes in critical domains such as counseling outcomes\[[40](https://arxiv.org/html/2608.11460#bib.bib40)\]\. Examples include queries answered by an LLM over clinical notes to predict diagnoses\[[33](https://arxiv.org/html/2608.11460#bib.bib33),[36](https://arxiv.org/html/2608.11460#bib.bib36)\], patient reports to predict intervention outcome\[[27](https://arxiv.org/html/2608.11460#bib.bib27)\], business reviews to predict ratings\[[35](https://arxiv.org/html/2608.11460#bib.bib35)\], and binary questionnaire items to predict fMRI response\[[34](https://arxiv.org/html/2608.11460#bib.bib34)\]\. Specific to conversation corpora, prior work has used LLM\-derived features to score and predict user satisfaction\[[32](https://arxiv.org/html/2608.11460#bib.bib32)\]\. However, while LLM\-based feature extraction and scoring is convenient, it is unclear which features are more informative than others without directly regressing on task outcome\. In our work, we explore a methodology that extracts traits as downstream predictive features, scores conversations based on these traits, and determines the most informative traits prior to outcome regression\. This methodology enables an unsupervised extraction of principal traits through a process which is independent of outcome variable in derivation, but is still informative in explaining variance in downstream outcomes\.

### 2\.3Tutoring Dialogues and LLMs in Education

The relationship between conversation behavior and learning outcomes has a long history in education research\. Prior work correlated manually coded dialogue acts with learning gains in tutoring dialogues\[[41](https://arxiv.org/html/2608.11460#bib.bib41),[43](https://arxiv.org/html/2608.11460#bib.bib43)\], modeled help\-seeking behavior in intelligent tutoring systems\[[21](https://arxiv.org/html/2608.11460#bib.bib21)\], and discovered latent dialogue modes over annotated act sequences\[[42](https://arxiv.org/html/2608.11460#bib.bib42)\]\. More recently, LLMs have been explored for capabilities in education tasks such as automated scoring and content generation\. In this line of work, related studies have scored dialogues on predefined psychological constructs such as persistence\[[39](https://arxiv.org/html/2608.11460#bib.bib39)\]and mined student–AI chat logs for knowledge gaps\[[45](https://arxiv.org/html/2608.11460#bib.bib45)\]\.

In a parallel thread, educators are now interested in teaching effective usage of AI tools, resulting in a growing number of “AI literacy” frameworks which establish effective principles for AI usage with learning and professional objectives\[[8](https://arxiv.org/html/2608.11460#bib.bib8)\]\. Many of these frameworks are grounded in pre\-existing learning and tutoring frameworks such as self\-regulated learning\[[9](https://arxiv.org/html/2608.11460#bib.bib9)\]and the Interactive, Constructive, Active, and Passive \(ICAP\) framework for cognitive engagement\[[10](https://arxiv.org/html/2608.11460#bib.bib10)\]\. However, since communication styles between AI and human tutors are distinct, new frameworks are being developed to characterize these styles and related competencies\[[11](https://arxiv.org/html/2608.11460#bib.bib11)\]\. Assessing these competencies is itself an open problem: recent work finds substantial misalignment between self\-reported and objective\-based measures of AI literacy\[[13](https://arxiv.org/html/2608.11460#bib.bib13)\]\. However, much of this work relies on manual coding and inspection of conversation data which does not scale readily, especially given the rapid development of LLM systems\.

## 3Methodology

PTA, a Principal Component Analysis \(PCA\)\-inspired approach, produces a representative set of textual descriptions we call*principal traits*from a corpus of human collaborator\-AI collaborative conversation sessions\. Each trait details a behavior pattern present in the corpus, analogous to principal component vectors derived from PCA\. Moreover, PTA also produces a "trait score" score for each session that enables session\-level and collaborator\-level behavioral analysis\. In Figure[1](https://arxiv.org/html/2608.11460#S1.F1), we visualize the 4 stages of PTA: i\) behavior\-observation extraction, ii\) observation clustering, iii\) trait scoring, and iv\) principal\-trait extraction\. In this section, we detail our methodology and relate each stage of PTA to PCA\.

PTA’s first two stages, extraction and clustering, have some resemblance of computing principal component vectors\. Since collaborator\-AI interaction data is textual, we cannot initialize with random vectors in, e\.g\., power iterations, to find the leading principal component of a matrix\[[46](https://arxiv.org/html/2608.11460#bib.bib46)\]\. Instead, we begin by creating a large pool of observations proposed via LLM prompting, which we narrow down into a more representative pool via clustering\. The third stage, trait scoring, is analogous to derivation of the per\-sample loadings for each principal component in PCA\. We use LLM\-as\-a\-judge to rate how much each collaborator\-AI interaction session exhibits the qualities described in each trait\. The final stage, trait selection, resembles selecting the principal components which explain the most variance in the data\. Since orthogonality between textual data is undefined, we greedily select traits that i\) explains the most variance in scores across collaborators while ii\) have low correlation with already selected traits in scoring and are also textually diverse\.

#### Notation\.

We first define notation shared across stages\. We consider a corpus ofMMsessions𝒮=\{s1,…,sM\}\\mathcal\{S\}=\\\{s\_\{1\},\\ldots,s\_\{M\}\\\}, each authored by a collaboratoru⁡\(si\)∈𝒰u\(s\_\{i\}\)\\in\\mathcal\{U\}, wheresis\_\{i\}denotes the textual interaction trace of a collaborator\-AI session\. From𝒮\\mathcal\{S\}, PTA produces a set ofKKprincipal traitsC∗=\{c1∗​…,cK∗\}C^\{\*\}=\\\{c^\{\*\}\_\{1\}\\ldots,c^\{\*\}\_\{K\}\\\}and a trait\-score matrix𝐙∈\{1,…,5\}M×K\\mathbf\{Z\}\\in\\\{1,\\ldots,5\\\}^\{M\\times K\}, wherezi​jz\_\{ij\}represents to what extent sessionsis\_\{i\}exhibits principal traitcj∗c^\{\*\}\_\{j\}\. In addition, we use a text\-embedding modelE⁡\(⋅\)∈ℝdE\(\\cdot\)\\in\\mathbb\{R\}^\{d\}to embed text in an LLML⁡\(⋅\)L\(\\cdot\)’s embedding space to calculate textual similarity, for clustering and ranking purposes\.

![Refer to caption](https://arxiv.org/html/2608.11460v1/figures/derivation_studychat.png)Figure 2:PTA clustering stages visualized\. Panels show one shared t\-SNE projection of all observation text embeddings from the StudyChat generic pool\. Raw observations \(a\) are clustered intok=100k=100micro\-clusters \(b\),∙\\bulletindicates cluster centroids\. The micro\-clusters are agglomeratively merged to remove duplicates and form candidate traits \(c\),∘\\circindicates candidate centroids and lines indicate which micro\-clusters are merged\. Finally the selection algorithm choosesk=10k\{=\}10principal traits \(d\),⋆\\starindicates selected traits\.
#### Stage 1: Behavior observations\.

In this stage, our goal is to propose a set of surface\-level behavior*observations*for collaborator\-AI interaction sessions\. We prompt an LLM to read the session and emit a small set of brief behavior observations \(up to 5\) corresponding to the collaborator’s behavior in this session, i\.e\.,L⁡\(si\)→\{o1,…​oK​0\}L\(s\_\{i\}\)\\rightarrow\\\{o\_\{1\},\\dots o\_\{K0\}\\\},K​0≤5K0\\leq 5\. We experiment with two prompting approaches,genericobservations without any specific guidelines andlens\-basedobservations, where the LLM is provided a textual description of a relevant theory towards understanding behavior in a particular context, e\.g\., self\-regulation theory in student\-AI tutoring sessions\. Full prompts for observation extraction according to all lenses appear in Appendix[C](https://arxiv.org/html/2608.11460#A3)\.

#### Stage 2: Trait derivation by clustering\.

In this stage, we form a set of candidate traits𝒞=\{c1​…​cK\}\\mathcal\{C\}=\\\{c\_\{1\}\\dots c\_\{K\}\\\}, by aggregating all observations across all sessions\. These candidate traits are analogous to the principal components formed in PCA and are textual representations of commonly observed behaviors that contain a title and description; Table[2](https://arxiv.org/html/2608.11460#S5.T2)shows examples\. To form these candidate traits, we perform top\-down clustering, bottom\-up agglomerative clustering, and a text de\-duplication step to form descriptive patterns of common behavior\. First, we perform top\-down clustering on each lens observation set by embedding every observation with a sentence encoderE⁡\(oi\)=𝐞iE\(o\_\{i\}\)=\\mathbf\{e\}\_\{i\}and clustering the embeddings viakk\-means to formK1K\_\{1\}micro\-clusters\[[48](https://arxiv.org/html/2608.11460#bib.bib48)\]\. We then use an LLM to name each cluster by passingK2K\_\{2\}observations whose embeddings are closest to the cluster centroid: we prompt the LLM to name the cluster and provide a short description of summarizing the behavior, i\.e\.,L⁡\(𝐨i′\)→ωiL\(\\mathbf\{o\}^\{\\prime\}\_\{i\}\)\\rightarrow\\omega\_\{i\}, whereωi\\omega\_\{i\}is a textual description of a cluster based off cluster centroid nearest\-neighbors𝐨i′\\mathbf\{o\}^\{\\prime\}\_\{i\}\. Second, we perform bottom\-up agglomerative clustering on the micro\-clusterdescriptionembeddingsE⁡\(ωi\)=𝝎iE\(\\omega\_\{i\}\)=\\boldsymbol\{\\omega\}\_\{i\}to combine similar clusters, until we reach a threshold ofK3K\_\{3\}candidate traits\. Finally, we merge any near\-duplicate candidates, whose definition embeddings exceed a cosine similarity thresholdτ\\tau\. This clustering and reduction process is repeated for all lens and all candidates, which we then combined into a set ofK4K\_\{4\}candidate traits𝒞\\mathcal\{C\}\. Figure[2](https://arxiv.org/html/2608.11460#S3.F2)\(b\) and \(c\) shows the resulting reduction: thousands of raw observations reduced to hundreds of micro\-clusters and then further into dozens of named candidate traits\. By combining these top\-down and bottom\-up approaches, we create a set of diverse candidate traits that reflect common collaborator behavior across sessions\.

#### Stage 3: Trait scoring\.

In this stage, we from a trait\-score matrix𝐙∈\{1​…​5\}N×K4\\mathbf\{Z\}\\in\\\{1\\dots 5\\\}^\{N\\times K\_\{4\}\}from the candidate traits𝒞\\mathcal\{C\}, wherezi​jz\_\{ij\}represents the extent to which a collaborator exhibits behavioral traitjjin sessionii\. To compute these scores, we use an LLM\-as\-a\-judge approach\[[49](https://arxiv.org/html/2608.11460#bib.bib49)\]for every \(session, trait\) pair, to rate how the collaborator exhibits the trait on a 5\-point Likert scale,L⁡\(si,cj\)→zi​j∈\{1,…,5\}L\(s\_\{i\},c\_\{j\}\)\\rightarrow z\_\{ij\}\\in\\\{1,\\ldots,5\\\}\. We rate each trait in a separate LLM call to prevent judge bias towards a particular polarity within a session\. We also average𝐙\\mathbf\{Z\}over sessionsper\-collaboratorand form another score matrix𝐙′∈ℝ\|𝒰\|×K\\mathbf\{Z^\{\\prime\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{U\}\|\\times K\}, wherezi​j′z^\{\\prime\}\_\{ij\}is the mean score score on traitjjacross all sessions involving collaboratoruiu\_\{i\}\. We later use this score matrix for per\-user behavioral analysis\. Intuitively, this scoring process is analogous to computing the loadings for each principal component in PCA, where each LLM call is a textual analog of performing a projection of a data sample onto the direction of a principal component\. We detail the scoring prompts used in this stage in Appendix[C\.3](https://arxiv.org/html/2608.11460#A3.SS3)\.

Algorithm 1Principal\-trait selection \(Stage 4\)1:score matrix

𝐙\\mathbf\{Z\}\(

M×KM\\times K\), trait embeddings

ω1,…,ωk4\\mathbf\{\\omega\}\_\{1\},\\ldots,\\mathbf\{\\omega\}\_\{k\_\{4\}\}, size

KK, similarity threshold

ε\\varepsilon
2:for

j=1,…,k4j=1,\\ldots,k\_\{4\}do

3:

ρ⁡\[j\]←z⁡\(itemTotal⁡\(𝐙,j\)\)\+z⁡\(communality⁡\(𝐙,j\)\)\\rho\[j\]\\leftarrow z\(\\mathrm\{itemTotal\}\(\\mathbf\{Z\},j\)\)\+z\(\\mathrm\{communality\}\(\\mathbf\{Z\},j\)\)⊳\\trianglerightrelevance, Eq\.[3](https://arxiv.org/html/2608.11460#S3.Ex1)

4:endfor

5:

C∗=←\[arg⁡maxj⁡ρ⁡\[j\]\]C^\{\*\}=\\leftarrow\[\\,\\arg\\max\_\{j\}\\rho\[j\]\\,\]⊳\\trianglerightseed with the most relevant trait

6:while

len⁡\(C∗\)<K\\mathrm\{len\}\(C^\{\*\}\)<Kdo

7:foreachcandidate

j∉C∗j\\notin C^\{\*\}do

8:

ρ\[j\]←ρ\[j\]−mean\(\|corr\(𝐙\[:,j\],𝐙\[:,n\]\)\|forn∈C∗\)\\rho\[j\]\\leftarrow\\rho\[j\]\-\\mathrm\{mean\}\\big\(\\,\|\\mathrm\{corr\}\(\\mathbf\{Z\}\[:,j\],\\,\\mathbf\{Z\}\[:,n\]\)\|\\text\{ for \}n\\in C^\{\*\}\\,\\big\)⊳\\trianglerighttrait score\-space redundancy

9:endfor

10:

A←\[j:ρ\[j\]≥max\(ρ\)−ε\]A\\leftarrow\\big\[\\,j:\\rho\[j\]\\geq\\max\(\\rho\)\-\\varepsilon\\,\\big\]
11:

j⋆←arg⁡maxj∈A​minn∈C∗⁡cosDist⁡\(ωj,ωn\)j^\{\\star\}\\leftarrow\\arg\\max\_\{j\\in A\}\\;\\min\_\{n\\in C^\{\*\}\}\\,\\mathrm\{cosDist\}\(\\mathbf\{\\omega\}\_\{j\},\\mathbf\{\\omega\}\_\{n\}\)⊳\\trianglerightmost text embedding\-space diverse candidate

12:

C∗\.append⁡\(j⋆\)C^\{\*\}\.\\mathrm\{append\}\(j^\{\\star\}\)
13:endwhile

14:return

C∗C^\{\*\}

#### Stage 4: Principal trait selection\.

In this stage, we form a subset ofKKprincipal traitsC∗=\{c1∗​…,cK∗\}C^\{\*\}=\\\{c^\{\*\}\_\{1\}\\ldots,c^\{\*\}\_\{K\}\\\}from the candidate traitsCCusing the score matrix𝐙\\mathbf\{Z\}\. To perform this subset, we use a greedy selection algorithm that captures how sessions vary by trading off two criteria:relevanceandredundancy\. We detail the full procedure in Algorithm[1](https://arxiv.org/html/2608.11460#alg1)\. We first score therelevanceof each candidate trait to measure how strongly it loads on the shared behavioral factor, which we compute by combining two complementary measures:itemTotalandcommunality:

ρ⁡\[j\]←z⁡\(itemTotal⁡\(𝐙,j\)\)\+z⁡\(communality⁡\(𝐙,j\)\)\\rho\[j\]\\leftarrow z\(\\mathrm\{itemTotal\}\(\\mathbf\{Z\},j\)\)\+z\(\\mathrm\{communality\}\(\\mathbf\{Z\},j\)\)Here,ρ⁡\[j\]\\rho\[j\]is the relevance score of traitjj;OPENitemTotal⁡\(𝐙,j\)\)=corr⁡\(𝐙⋅j,1K−1​∑m≠j𝐙⋅m\)\\mathrm\{itemTotal\}\(\\mathbf\{Z\},j\)\)=\\mathrm\{corr\}\\Big\(\\mathbf\{Z\}\_\{\\cdot j\},\\ \\tfrac\{1\}\{K\-1\}\\textstyle\\sum\_\{m\\neq j\}\\mathbf\{Z\}\_\{\\cdot m\}\\Big\)is the item\-total correlation, a statistic from classical test theory, which measures how strongly the trait correlates with the other candidate traits\[[50](https://arxiv.org/html/2608.11460#bib.bib50)\];communality⁡\(𝐙,j\)=hj2=∑f=1mλj​f2\\mathrm\{communality\}\(\\mathbf\{Z\},j\)=h\_\{j\}^\{2\}=\\sum\_\{f=1\}^\{m\}\\lambda\_\{jf\}^\{2\}is the communality of each trait as the the fraction of traitjj’s variance explained by the shared factors of a factor\-analysis model\[[51](https://arxiv.org/html/2608.11460#bib.bib51)\]; andz⁡\(⋅\)z\(\\cdot\)standardizes each measure to zero mean and unit variance so both relevance measures can be combined additively\. Specifically, we compute communality by fitting a factor\-analysis model to the standardized score matrix by maximum likelihood estimation\[[52](https://arxiv.org/html/2608.11460#bib.bib52)\]and computehj2h\_\{j\}^\{2\}from the shared factor loadings\. This process is analogous to computing the eigenvalues of each principal component, where we score each trait based off how this trait scores vary with other traits based off average trait and latent factor covariance\.

After computing relevance, we then conduct a greedy selection procedure that is adjusted byredundancyof subsequent selected traits compared to already selected ones\. We adjust the selection both within the trait score space and text embedding space to encourage diversity in both spaces\. This adjustment is necessary: in PCA, one can simply remove the impact of the leading principal component when finding the next principal component by subtracting it out\. However, since orthogonality is undefined for textual traits, we instead subtract the average magnitude of the correlation between each candidate and the existing, already\-selected traits in the score space:

ρ\[j\]←ρ\[j\]−mean\(\|corr\(𝐙\[:,j\],𝐙\[:,n\]\)\|forn∈C∗\)\.\\mathrm\{\\rho\}\[j\]\\leftarrow\\rho\[j\]\-\\mathrm\{mean\}\\big\(\\,\|\\mathrm\{corr\}\(\\mathbf\{Z\}\[:,j\],\\,\\mathbf\{Z\}\[:,n\]\)\|\\text\{ for \}n\\in C^\{\*\}\\,\\big\)\.Here,ρ⁡\[j\]\\mathrm\{\\rho\}\[j\]is the score\-space adjusted relevance score of traitjj\. This step penalizes principal trait candidates that have scores that are highly correlated with previous ones traits\. From this adjusted set of principal trait candidates, we then select the trait with least textual description similarity to existing ones\. Specifically, we take traits within a thresholdε\\varepsilonof the nearest candidate in text embedding space and select one that is most dissimilar from already selected traits\. Specifically, we select the next trait by computing:

j⋆←arg⁡maxj∈A​minn∈C∗⁡cosDist⁡\(ωj,ωn\)\.j^\{\\star\}\\leftarrow\\arg\\max\_\{j\\in A\}\\;\\min\_\{n\\in C^\{\*\}\}\\,\\mathrm\{cosDist\}\(\\mathbf\{\\omega\}\_\{j\},\\mathbf\{\\omega\}\_\{n\}\)\.Here,j⋆j^\{\\star\}is the index of the next trait,AAis a gated subset of traits withinε\\varepsilonof the nearest candidate, andωj=E⁡\(cj\)\\mathbf\{\\omega\}\_\{j\}=E\(c\_\{j\}\)the text\-embedding of the descriptions of each candidate trait\. This step penalizes principal trait candidates that have similar text representations to the existing candidates\. Overall, this selection algorithm is analogous to PCA’s criterion: it selects final principal traits from candidates that maximize explained score variance across sessions and minimize trait similarity in the text\-embedding and score spaces\.

## 4Experimental Setup

In this section, we detail our experiments to evaluate the effectiveness of PTA in deriving traits in two different human\-AI collaboration settings\.

### 4\.1Datasets

The first dataset we analyze isStudyChat\[[1](https://arxiv.org/html/2608.11460#bib.bib1)\], a collection of conversations between students and an LLM tutor in a university programming\-based artificial intelligence course\. We analyze 1,540 tutoring sessions authored by 171 students across two semesters\. Students complete seven programming assignments and three examinations over a semester\. We investigate student’s examination performance as the outcome variable\. The two semesters involve different students, assignments, and exams, so we analyze outcomes on each semester separately but derive principal traits from the entire dataset\.

The second dataset we analyze isSWE\-Chat\[[2](https://arxiv.org/html/2608.11460#bib.bib2)\], a collection of conversations between professional developers and AI\-powered coding agents working on real software repositories\. We analyze 2,774 sessions that come from developers with at least two sessions\. We investigate the per\-session success signal as the outcome variable\. In SWE\-Chat, each session addresses a distinct software task rather than a shared assignment across all students in StudyChat, which we account for in our evaluation metrics\.

### 4\.2Metrics

We measure whether the derived principal traits explain collaborator outcomes beyond a standard prior of average outcome on previous scored tasks\[[53](https://arxiv.org/html/2608.11460#bib.bib53)\]\. We define our outcome target variable and features to account for the different settings across the two datasets below\.

#### Shared Regression Setup

We measure whether the derived principal traits explain significant variance beyond a univariate regression on the average prior collaborator outcome\. Specifically, we first run a single feature regression, where the feature is the running average prior outcome for a collaborator, and the target is the subsequent outcome \(exam score for StudyChat, session score for SWE\-Chat\)\. We then run a second regression where the features are the prior combined withK=10K=10principal traits, measure theR2R^\{2\}improvement, and report statistical significance according to an F\-Test which controls for the feature count differences between the two regressions\.

#### StudyChat: Explanatory Exam Regression

In StudyChat, students complete a shared set of assignments and examinations, so we use the exam scores as our outcome target, and compute features by averaging traits observed in all sessions recorded before each exam\. Specifically, we treat each of the two later examinations,e2e\_\{2\}ande3e\_\{3\}, as a separate target, and we take the mean of the student’s earlier examinations as the prior\. Each of the171171students therefore contributes two samples, one per exam, for342342target outcomes in total\. Since the effective sample size is too small to support a held\-out partition, we therefore report explanatory improvementR2R^\{2\}from in\-sample fits\.

#### SWE\-Chat: Predictive Difficulty\-Aligned Regression

In SWE\-Chat, each session addresses a distinct task, ranging from quick webpage fixes to advanced distributed backend deployments, so we normalize outcomes by task difficulty and compute features fromper\-sessiontrait scores\. Specifically, we fit an item response theory type model,Pr⁡\(success\)=σ⁡\(𝐰a⊤​𝐮−𝐰d⊤​𝐭\)\\Pr\(\\text\{success\}\)=\\sigma\\big\(\\mathbf\{w\}\_\{a\}^\{\\top\}\\mathbf\{u\}\-\\mathbf\{w\}\_\{d\}^\{\\top\}\\mathbf\{t\}\\big\), in which collaborator behavior traits𝐮\\mathbf\{u\}are related to ability and task\-property traits𝐭\\mathbf\{t\}are related to task difficulty\[[54](https://arxiv.org/html/2608.11460#bib.bib54)\]\. Because each session carries its own outcome, this setup yields one regression target per session, and the sample is large enough that we report predictiveR2R^\{2\}on held\-out sessions\. Specifically, we perform cross validation to create 4 sets of train\-test splits of our data, divided across collaborators\. We use the train split to train the IRT outcome model, and report the average held\-outR2R^\{2\}improvement across all folds\.

### 4\.3Experimental Details

#### Extraction Lenses\.

We extract observations under a set of lenses, each of which frames the extraction prompt around a body of domain\-specific theory\. For StudyChat, we use four lenses drawn from education research: self\-regulated learning\[[9](https://arxiv.org/html/2608.11460#bib.bib9)\]combined with the ICAP engagement framework\[[10](https://arxiv.org/html/2608.11460#bib.bib10)\], the 4D AI\-fluency competencies\[[11](https://arxiv.org/html/2608.11460#bib.bib11)\], question sophistication\[[20](https://arxiv.org/html/2608.11460#bib.bib20)\], and academic help\-seeking\[[21](https://arxiv.org/html/2608.11460#bib.bib21)\]\. We adapt the third lens from the technical dimension of the AI Literacy Heptagon, treating the precision and depth of a student’s questions as a conversational proxy for that dimension\[[20](https://arxiv.org/html/2608.11460#bib.bib20)\]\. For SWE\-Chat, we use four lenses drawn from software\-engineering practice: specification quality, verification and oversight, agency and control, and craftsmanship and process discipline\. We frame the first, second, and fourth around knowledge areas of the Software Engineering Body of Knowledge\[[4](https://arxiv.org/html/2608.11460#bib.bib4)\], covering requirements, testing, and construction and quality respectively\. We frame the third around a taxonomy of human\-AI collaboration in software engineering\[[5](https://arxiv.org/html/2608.11460#bib.bib5)\]and a study of how professional developers supervise coding agents rather than delegate to them wholesale\[[6](https://arxiv.org/html/2608.11460#bib.bib6)\]\. We provide the full prompts for every lens in Appendix[C](https://arxiv.org/html/2608.11460#A3)\.

We report two trait derivations per dataset: a*generic*pool derived from the observations without lenses, and a*lens\-ensemble*pool that merges the candidate traits derived from every lens, including the generic pool\. The generic pools contain4343traits on StudyChat and2020on SWE\-Chat\. The lens\-ensemble pools contain159159and5959traits, respectively\.

Table 1:Significantly improved explainability over baseline feature, both datasets\.We observe that PTA traits help explain behavioral variance in student behavior \(StudyChat\) and are predictive indicators of session success in software development \(SWE\-Chat\)\.
#### PTA Hyperparameters\.

We detail the hyperparmeters used in our experiments\. We setK=10K=10,K1=100K\_\{1\}=100,K2=10K\_\{2\}=10,τ=0\.85\\tau=0\.85, andε=0\.25\\varepsilon=0\.25for both datasets\. We setK3K\_\{3\}the aggregation target to4545and1212for StudyChat and SWE\-Chat respectively to account for the differences in the sizes of the datasets\.

#### Language Models\.

For observation extraction and trait scoring we use the GPT5\.5 to produce observations, cluster names, and de\-duplication\[[56](https://arxiv.org/html/2608.11460#bib.bib56)\], and deploy a self\-hostedQwen3\-4Bmodel\[[3](https://arxiv.org/html/2608.11460#bib.bib3)\]for scoring\. For text\-embeddings we use ModernBERT\[[7](https://arxiv.org/html/2608.11460#bib.bib7)\]to form mini\-clusters from the observations and MPNet\[[47](https://arxiv.org/html/2608.11460#bib.bib47)\]to embed trait definitions during de\-duplication and selection\.

#### Baselines\.

We compare PTA derived traits against two feature sets that are defined for StudyChat\.Dialogue actsuses the counts of the 8 broad dialogue\-act categories per turn, provided in the dataset which we derived from manual coding and observations of the sessions\.Bloom’s taxonomyuses the 6 levels of the revised Bloom’s taxonomy\[[14](https://arxiv.org/html/2608.11460#bib.bib14)\]as an alternative set of traits\. We prompt our scoring model to measure the degree to which each of the levels of the schema was present in the dataset through the same scoring prompt structure as the candidate traits and use the scores as regression features\. See Appendix[B](https://arxiv.org/html/2608.11460#A2)for further details\. To the best of our knowledge, we could not find an established taxonomy of predictive dialogue traits similar to what the learning sciences provide for software engineering task outcomes\. We therefore evaluate SWE\-Chat against the prior\-outcome regression alone\.

## 5Results and Analysis

In this section, we analyze the results of our experiments towards understanding the validity of the derived principal traits and assess if they can be interpreted as skills\.

Table 2:Significant traits in both datasets\.Traits individually significant \(p<0\.05p<0\.05\) in the pooled incremental models of Table[1](https://arxiv.org/html/2608.11460#S4.T1), with descriptions of what scoring high on each trait means\.β\\betaindicates the learned coefficient in the regression where outcome is the target dependent variable\. Further trait information is shown in appendix Table[3](https://arxiv.org/html/2608.11460#A1.T3)\.TraitDescription \(high pole\)β\\beta\(pp\)StudyChat—*lens\-ensemble pool*conceptual\_understanding\_orientationThe student uses the AI tutor to build, verify, and deepen understanding of concepts, mechanisms, reasoning, code behavior, or results\+\+0\.29 \(\.050\)active\_feedback\_engagementThe student actively processes, evaluates, and iteratively refines AI\-provided help or feedback to improve their own work−\-0\.39 \(\.030\)specificity\_of\_uncertaintyA student clearly identifies what they do and do not understand when seeking help−\-0\.45 \(\.015\)StudyChat—*generic pool*question\_context\_elaborationA student provides context, goals, prior attempts, and details when asking the AI tutor for help\+\+0\.65 \(\.002\)goal\_directed\_steeringThe student actively monitors and redirects the tutor’s responses to align with their intended task, constraints, evidence, or preferred format−\-0\.24 \(\.024\)task\_context\_specificityThe student provides concrete, task\-specific materials or details—such as code, data, schemas, drafts, or outputs—for the tutor to work from−\-0\.40 \(\.008\)SWE\-Chat—*lens\-ensemble pool*workflow\_control\_delegationThe user directs the AI coding agent’s working process, ranging from terse high\-level autonomous delegation to highly prescriptive, command\-driven workflow orchestration\+\+0\.025 \(<<\.001\)change\_discipline\_and\_maintainabilityThe user guides an AI coding agent toward scoped, incremental, reusable, convention\-aligned, and well\-documented changes\+\+0\.015 \(\.029\)workflow\_structure\_incrementalityThe user directs the AI coding agent through explicit upfront planning, staged or phase\-gated workflows, small incremental changes, and defined verification criteria\+\+0\.013 \(\.042\)delegation\_specificityThe user specifies constraints, approach, implementation details, and acceptance criteria before handing work to the AI coding agent−\-0\.017 \(\.003\)evidence\_driven\_quality\_oversightThe user guides and evaluates the AI coding agent through concrete evidence, verification, actionable defect reports, and regression\-focused validation−\-0\.027 \(<<\.001\)SWE\-Chat—*generic pool*workflow\_control\_delegationThe user directs the AI coding agent’s working process, ranging from terse high\-level autonomous delegation to highly prescriptive, command\-driven workflow orchestration\+\+0\.022 \(<<\.001\)contextual\_workflow\_guidanceThe user explicitly supplies or directs the agent to gather task context, inspect repository state, use specific paths or commands, and follow verification or documentation expectations\+\+0\.011 \(\.025\)diagnostic\_evidence\_provisionThe user reports bugs or failures with concrete, actionable evidence such as reproduction steps, expected\-versus\-actual behavior, logs, stack traces, commands, screenshots, CI artifacts, or failing inputs−\-0\.021 \(<<\.001\)### 5\.1PTA Traits Validity

In Table[1](https://arxiv.org/html/2608.11460#S4.T1), we report the results of our evaluation of how well PTA\-derived traits explain variance in outcomes on both datasets\. On the StudyChat dataset, we observe that the addition of the 10 PTA traits improves explainability significantly in the Fall 2024 semester when using traits derived from the generic trait pool\. However, we see that this result does not hold in the Spring 2025 semester\. This observation suggests that traits derived from PTA have limited generalizability, even across similar course settings in two different semester, which is evidence that traits should not be interpreted as skills\. However, we do observe that the theory informed lens traits are borderline significant in the Spring semester\. This observation suggests that insights from learning theory can be used to derive traits that are more explanatory, and that further theoretical advancements of pedological theory of student\-AI dialogue may shrink the gap towards an interpretation of derived traits as skills\. We further observe that the added explanatory power of the principal traits is more statistically powered than coded dialogue act or Bloom’s taxonomy baselines\. This observation suggests that baseline coding schemes are either too specific to explain overall behavior in the case of dialogue acts and too general in the case of Bloom’s\. Instead, our traits form a more appropriate representation of student behavior\. Overall, these results suggest that while PTA\-derived traits are are powerful in explaining student behavior, they fall short in serving as generalizable skill definitions which can inform pedagogical practice\.

On the SWE\-chat dataset, we observe that PTA derived traits provide stronger behavior signals that are predictive of session outcome\. Given the more granular task of session outcome prediction, the SWE\-Chat dataset has an order of magnitude more samples than StudyChat, an we observe consistently significant predictive improvements over the baseline prior\. This observation indicates that PTA traits are more predictive on larger datasets that have a larger pool of samples and indicates that our approach improves with scale\. However, we notice the overallR2R^\{2\}magnitude in SWE\-Chat is significantly lower, which indicates that prior session performance is not as strong an outcome indicator in SWE\-Chat as in StudyChat\. Overall, these results indicate that PTA traits are significant predictors and that this approach is flexible across distinct conversation datasets and outcomes, but lack of clear baselines to compare our approach to makes clear interpretation difficult\.

### 5\.2Significant Traits

In Table[2](https://arxiv.org/html/2608.11460#S5.T2), we report the individually significant traits derived from the regression analyses in the previous section\. Since our traits are text\-based, we see that these traits are not only explanatory, but are also interpretable\. We observe that all traits imply intuitively outcome\-positive behaviors, even though the learned regression coefficientβ\\betaindicates some are negative in the regression model fit\. While the positive tone indicates that all traits correspond to aspects which should be positively correlated with outcomes, derived coefficients are negative for some traits\. One possible explanation is that some of the negatively correlated traits correspond tightly to a small subset of collaborators who used an LLM in a certain way, but scored negatively on the outcome due to external confounders\.

In interpreting the descriptions for the StudyChat\-derived traits, we observe that behaviors that encourage conceptual understanding are positively correlated with exam outcomes, whereas goal\-directed and task\-delegating behaviors are negatively correlated with outcomes\. Specifically, we see thatconceptual\_understanding\_orientationandquestion\_context\_elaborationare traits positively correlated with exam outcomes and indicate deep conceptual engagement\. However,task\_context\_specificityis negatively correlated with outcomes, which appears to contradict the results on other traits\. One possible explanation is that certain types of help\-seeking behaviors are negatively correlated with outcomes, and these questioning behaviors may be confounded by student ability\. Notably, these unsupervised findings corroborate with human\-made observations on the StudyChats dataset that conceptual questioning dialogue acts were positively correlated with outcomes\[[1](https://arxiv.org/html/2608.11460#bib.bib1)\]\. The remaining significant traits are all negatively correlated with outcomes and may suggest a pattern of task\-delegation behaviors, where the student asks the LLM to complete a portion of the assignment\. This result suggests that these traits emerge when students are not engaging with learning outcomes, which is aligned with prior work analyzing this dataset\[[55](https://arxiv.org/html/2608.11460#bib.bib55)\]\.

In interpreting the descriptions for the SWE\-chat\-derived traits, we observe that behaviors where developers provide a clear role for the LLM are positively correlated with outcomes, whereas evidence\-based workflows are negatively correlated with outcomes\. Traits where the user gives specific, scoped tasks to the agent result in positive outcomes such asworkflow\_control\_ delegation, found significant in both pools, andworkflow\_structure\_incrementality\. However, when the developer provides loosely\-defined roles and instead gives constraint\-based guidelines or relies on the coding agent’s interpretation of evidence, the collaboration often results in negative outcomes, such asdelegation\_specificityandevidence\_driven\_quality\_oversight\.

All together, interpretation of these traits suggest reasonable patterns, but sometimes contradictory results\. One possible explanation for these contradictions is that the semantic clustering in the candidate trait derivation process is overly aggressive: some observations which contain similar words but unique behavioral traits, are combined erroneously\. This process results in traits that correspond to a large set of underlying skills, increasing explanatory power at the cost of interpretability\. Overall, while these traits may provide reasonable directions for further analysis into behavioral patterns, they do not, by themselves, appear to be grounded enough to be interpreted as a definitive skill taxonomy in human\-AI collaboration\.

![Refer to caption](https://arxiv.org/html/2608.11460v1/figures/temporal_gpt55_2x2_v3.png)Figure 3:Trait scores over time\.Each panel overlays that pool’s significant traits \(Table[2](https://arxiv.org/html/2608.11460#S5.T2)\) as raw mean scores on the 1–5 rating scale\. Cool colors \(blue, green, teal\) and warm \(red, orange\) are used for positive and negative signed traits respectively\. We observe relatively flat behaviors, with some positive indicators for StudyChat lens\-ensemble \(top\-right\)\.
### 5\.3Temporal Analysis

Another key to the definition of skills is their learning curves\. In Figure[3](https://arxiv.org/html/2608.11460#S5.F3), we plot average human collaborators’ scores on the significant PTA traits over time, to analyze whether these traits reflect typical learning curve dynamics\. For the StudyChat traits, we show usage as it changes over the course of the semester and for the SWE\-Chat traits, we show usage across subsequent sessions\. We observe that for the StudyChat traits, there appears to be a generally positive progression of trait behavior, particularly for theconceptual\_understanding\_orientationtrait that tends to increase over time\. This trend suggests that this trait does appear to be learned over time as students progress in the semester\. However, an alternative explanation is that assignments later in the course, which are generally more complex, may lend towards more of this usage pattern, independent of student skill acquisition\. Other traits appear mostly fixed or stable across time\. For SWE\-Chat, we observe a stable or slightly declining trend over time for all traits\. This stability suggests that behavior in subsequent sessions for software developers does not deviate much from prior sessions: there do not seem to be any clear indications of behavior change or developers learning skills\. This observation is perhaps due to the short time span in which the SWE\-chats corpus was collected, where human\-AI collaboration skill development would likely not occur\. Another possible explanation is that for professional developers, their skills are perhaps mostly stable, especially in a productivity\-oriented setting, where they may not prioritize in finding better ways to interact with the coding agent\. As a result, we may only be investigating a short, locally\-flat section of their learning curve, where most learning activities happened outside of the dataset’s scope\.

## 6Conclusions

In this paper, we introduced Principal Trait Analysis, a data\-driven, principal component analysis\-inspired approach that derives interpretable, traits of LLM usage directly from human\-AI interaction session trace datasets\. We evaluated our approach on human\-AI collborative conversations in a software development setting, demonstrating our approach in both a learning and professional context\. Our approach recovers traits that explain outcomes beyond prior achievement heuristics and seem readily interpretable\. However, our analysis reveals that the traits are inconsistent with the qualities required to interpret these traits as human\-AI collaboration skills\.

Our work outlines avenues for future research into the derived traits and overall methodology\. A possible next step for future work would be to test this methodology across a larger educational dataset, beyond classroom\-scale, and not isolated to computer science\. Moreover, additional validation of the derived traits via human evaluation could expose more details of the quality and deeper implications behind these derived traits\. These investigations may reveal gaps in our current methodology\. Finally, this work focuses on datasets where the LLM is not given any pedagogical intervention or alignment, but a natural question would be to see whether traits derived from collaborator behavior with a pedagogically aligned LLM are noticeably different\.

## References

- \[1\]Hunter McNichols, Fareya Ikram, and Andrew Lan\.The StudyChat dataset: Analyzing student dialogues with ChatGPT in an artificial intelligence course\.InProceedings of the 16th International Learning Analytics and Knowledge Conference \(LAK ’26\), pages 53–63, 2026\.
- \[2\]SALT\-NLP\.SWE\-chat: Coding agent interactions from real users in the wild\.arXiv preprint arXiv:2604\.20779, 2026\.
- \[3\]Qwen Team\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388, 2025\.
- \[4\]Hironori Washizaki, editor\.Guide to the Software Engineering Body of Knowledge \(SWEBOK Guide\), Version 4\.0\.IEEE Computer Society, 2024\.
- \[5\]Christoph Treude and Marco A\. Gerosa\.How developers interact with AI: A taxonomy of human\-AI collaboration in software engineering\.arXiv preprint arXiv:2501\.08774, 2025\.
- \[6\]Ruanqianqian Huang, Avery Reyna, Sorin Lerner, Haijun Xia, and Brian Hempel\.Professional software developers don’t vibe, they control: AI agent use for coding in 2025\.arXiv preprint arXiv:2512\.14012, 2025\.
- \[7\]Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, et al\.Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.arXiv preprint arXiv:2412\.13663, 2024\.
- \[8\]Duri Long and Brian Magerko\.What is AI literacy? Competencies and design considerations\.InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2020\. doi:10\.1145/3313831\.3376727\.
- \[9\]Barry J\. Zimmerman\.Becoming a self\-regulated learner: An overview\.Theory Into Practice, 41\(2\):64–70, 2002\. doi:10\.1207/s15430421tip4102\_2\.
- \[10\]Michelene T\. H\. Chi and Ruth Wylie\.The ICAP framework: Linking cognitive engagement to active learning outcomes\.Educational Psychologist, 49\(4\):219–243, 2014\. doi:10\.1080/00461520\.2014\.965823\.
- \[11\]Rick Dakan and Joseph Feller\.Framework for AI fluency \(practical summary document\), version 1\.1\.Ringling\.edu/ai/, 2025\.[https://ringling\.libguides\.com/ai/framework](https://ringling.libguides.com/ai/framework)\. Retrieved 31 July 2026\.
- \[12\]Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng\.WildChat: 1M ChatGPT interaction logs in the wild\.InICLR, 2024\.[https://arxiv\.org/abs/2405\.01470](https://arxiv.org/abs/2405.01470)\.
- \[13\]Shan Zhang, Ruiwei Xiao, Anthony F\. Botelho, Guanze Liao, Thomas K\. F\. Chiu, John Stamper, and Kenneth R\. Koedinger\.How to assess AI literacy: Misalignment between self\-reported and objective\-based measures\.InLAK, 2026\.[https://arxiv\.org/abs/2601\.06101](https://arxiv.org/abs/2601.06101)\.
- \[14\]David R\. Krathwohl\.A revision of Bloom’s taxonomy: An overview\.Theory Into Practice, 41\(4\):212–218, 2002\. doi:10\.1207/s15430421tip4104\_2\.
- \[15\]Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer\.The impact of AI on developer productivity: Evidence from GitHub Copilot\.arXiv preprint arXiv:2302\.06590, 2023\.
- \[16\]Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman\.How novices use LLM\-based code generators to solve CS1 coding tasks in a self\-paced learning environment\.InKoli Calling International Conference on Computing Education Research, 2023\. doi:10\.1145/3631802\.3631806\.
- \[17\]James Prather, Brent N\. Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S\. Randrianasolo, Brett A\. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs\.The widening gap: The benefits and harms of generative AI for novice programmers\.InACM Conference on International Computing Education Research \(ICER\), 2024\. doi:10\.1145/3632620\.3671116\.
- \[18\]Kenneth R\. Koedinger, Albert T\. Corbett, and Charles Perfetti\.The knowledge\-learning\-instruction framework: Bridging the science\-practice chasm to enhance robust student learning\.Cognitive Science, 36\(5\):757–798, 2012\. doi:10\.1111/j\.1551\-6709\.2012\.01245\.x\.
- \[19\]Hao Cen, Kenneth Koedinger, and Brian Junker\.Learning factors analysis — a general method for cognitive model evaluation and improvement\.InInternational Conference on Intelligent Tutoring Systems \(ITS\), pages 164–175, 2006\. doi:10\.1007/11774303\_17\.
- \[20\]Veronika Hackl, Alexandra Mueller, and Maximilian Sailer\.The AI literacy heptagon: A structured approach to AI literacy in higher education\.Computers and Education: Artificial Intelligence, 10:100540, 2026\.
- \[21\]Vincent Aleven, Ido Roll, Bruce M\. McLaren, and Kenneth R\. Koedinger\.Help helps, but only so much: Research on help seeking with intelligent tutoring systems\.International Journal of Artificial Intelligence in Education, 26\(1\):205–223, 2016\.
- \[22\]Yida Mu, Chun Dong, Kalina Bontcheva, and Xingyi Song\.Large language models offer an alternative to the traditional approach of topic modelling\.InLREC\-COLING, 2024\.[https://aclanthology\.org/2024\.lrec\-main\.887/](https://aclanthology.org/2024.lrec-main.887/)\.
- \[23\]Yuwei Zhang, Zihan Wang, and Jingbo Shang\.ClusterLLM: Large language models as a guide for text clustering\.InEMNLP, 2023\.[https://aclanthology\.org/2023\.emnlp\-main\.858/](https://aclanthology.org/2023.emnlp-main.858/)\.
- \[24\]Mian Zhong, Pristina Wang, and Anjalie Field\.HICode: Hierarchical inductive coding with LLMs\.InEMNLP, 2025\.[https://aclanthology\.org/2025\.emnlp\-main\.1580/](https://aclanthology.org/2025.emnlp-main.1580/)\.
- \[25\]Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer\.TopicGPT: A prompt\-based topic modeling framework\.InNAACL, 2024\.
- \[26\]Zihan Wang, Jingbo Shang, and Ruiqi Zhong\.Goal\-driven explainable clustering via language descriptions\.InEMNLP, 2023\.
- \[27\]Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt\.Goal driven discovery of distributional differences via language descriptions\.InNeurIPS, 2023\.
- \[28\]Michelle S\. Lam, Janice Teoh, James A\. Landay, Jeffrey Heer, and Michael S\. Bernstein\.Concept induction: Analyzing unstructured text with high\-level concepts using LLooM\.InCHI, 2024\.
- \[29\]Zhouhang Xie, Tushar Khot, Bhavana Dalvi Mishra, Harshit Surana, Julian McAuley, Peter Clark, and Bodhisattwa Prasad Majumder\.Latent factor models meet instructions: Goal\-conditioned latent factor discovery without task supervision\.InNAACL, 2025\.
- \[30\]Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, et al\.TnT\-LLM: Text mining at scale with large language models\.InKDD, 2024\.
- \[31\]Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, et al\.Clio: Privacy\-preserving insights into real\-world AI use\.arXiv preprint arXiv:2412\.13678, 2024\.
- \[32\]Ying\-Chun Lin, Jennifer Neville, Jack W\. Stokes, et al\.Interpretable user satisfaction estimation for conversational systems with large language models\.arXiv preprint arXiv:2403\.12388, 2024\.
- \[33\]Denis Jered McInerney, Geoffrey Young, Jan\-Willem van de Meent, and Byron C\. Wallace\.CHiLL: Zero\-shot custom interpretable feature extraction from clinical notes with large language models\.InFindings of EMNLP, 2023\.
- \[34\]Vinamra Benara, Chandan Singh, John X\. Morris, et al\.Crafting interpretable embeddings by asking LLMs questions\.InNeurIPS, 2024\.
- \[35\]Josh Magnus Ludan, Qing Lyu, Yue Yang, Liam Dugan, Mark Yatskar, and Chris Callison\-Burch\.Interpretable\-by\-design text understanding with iteratively generated concept bottleneck\.arXiv preprint arXiv:2310\.19660, 2023\.
- \[36\]Jean Feng, Avni Kothari, Luke Zier, Chandan Singh, and Yan Shuo Tan\.Bayesian concept bottleneck models with LLM priors\.InNeurIPS, 2025\.
- \[37\]Václav Balek, Lukáš Sýkora, Vilém Sklenák, and Tomáš Kliegr\.LLM\-based feature generation from text for interpretable machine learning\.Machine Learning, 114:241, 2025\.
- \[38\]Gen Li, Li Chen, Cheng Tang, et al\.Single\-agent vs\. multi\-agent LLM strategies for automated student reflection assessment\.InPAKDD, 2025\.
- \[39\]Teresa M\. Ober, Ying Zhang, Diego Zapata\-Rivera, et al\.Using LLMs to identify indicators of persistence from students’ dialogues with a pedagogical agent\.Journal of Educational Data Mining, 18\(1\), 2026\.
- \[40\]Tim Althoff, Kevin Clark, and Jure Leskovec\.Large\-scale analysis of counseling conversations: An application of natural language processing to mental health\.Transactions of the Association for Computational Linguistics, 4:463–476, 2016\.
- \[41\]Diane Litman and Kate Forbes\-Riley\.Correlations between dialogue acts and learning in spoken tutoring dialogues\.Natural Language Engineering, 12\(2\):161–176, 2006\.
- \[42\]Kristy Elizabeth Boyer, Robert Phillips, Amy Ingram, Eun Young Ha, Michael Wallis, Mladen Vouk, and James Lester\.Investigating the relationship between dialogue structure and tutoring effectiveness: A hidden Markov modeling approach\.International Journal of Artificial Intelligence in Education, 21\(1–2\):65–81, 2011\.
- \[43\]Arthur C\. Graesser\.Conversations with AutoTutor help students learn\.International Journal of Artificial Intelligence in Education, 26:124–132, 2016\.
- \[44\]Louisa Parks and Wim Peters\.Natural language processing in mixed\-methods text analysis: A workflow approach\.International Journal of Social Research Methodology, 26\(4\), 2023\.
- \[45\]Yuheng Fu, Boyang Wu, and Joseph Jay Williams\.Mining the gold: Student–AI chat logs as rich sources for automated knowledge gap detection\.arXiv preprint arXiv:2512\.22404, 2025\.
- \[46\]Mises, RV and Pollaczek\-Geiringer, HildaPraktische verfahren der gleichungsauflösung\.ZAMM\-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 9\(1\):58–77, 1929\.
- \[47\]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie\-Yan Liu\.MPNet: Masked and permuted pre\-training for language understanding\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2020\.
- \[48\]Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam\.Thekk\-means algorithm: A comprehensive survey and performance evaluation\.Electronics, 9\(8\):1295, 2020\.
- \[49\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS Datasets and Benchmarks Track\), 2023\.
- \[50\]Sten Henrysson\.Correction of item\-total correlations in item analysis\.Psychometrika, 28\(2\):211–218, 1963\.
- \[51\]Harry H\. Harman\.Modern Factor Analysis\.University of Chicago Press, Chicago, 3rd edition, 1976\.
- \[52\]Richard J\. Rossi\.Mathematical Statistics: An Introduction to Likelihood Based Inference\.John Wiley & Sons, New York, 2018\.
- \[53\]Kelly Hallberg, Thomas D\. Cook, Peter M\. Steiner, and M\. H\. Clark\.Pretest measures of the study outcome and the elimination of selection bias: Evidence from three within study comparisons\.Prevention Science, 19\(3\):274–283, 2018\.
- \[54\]Paul De Boeck and Mark Wilson, editors\.Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach\.Springer, New York, 2004\.
- \[55\]Daiana Rinja, Eduardo Araujo Oliveira, Sonsoles López\-Pernas, Mohammed Saqr, Marcus Specht, and Kamila Misiejuk\.Unpacking vibe coding: Help\-seeking processes in student–AI interactions while programming\.InProceedings of the 27th International Conference on Artificial Intelligence in Education \(AIED\)\. Springer, 2026\.
- \[56\]OpenAI\.GPT\-5\.5 system card\.Technical report, OpenAI, April 2026\.[https://deploymentsafety\.openai\.com/gpt\-5\-5/gpt\-5\-5\.pdf](https://deploymentsafety.openai.com/gpt-5-5/gpt-5-5.pdf)\.

## Appendix ASignificant Traits with Definitions

Table 3:Significant traits with definitions\.The traits of Table[2](https://arxiv.org/html/2608.11460#S5.T2), with each trait’s definition as written during trait derivation — the high pole first, and the low pole in italics where the definition states one — and the derivation pass \(*origin*\) each trait came from\. Origin: Gen==unsteered \(generic\) pass, Help==help\-seeking lens, Clarity==communication\-clarity lens, Craft==craftsmanship and process lens, Verif==verification and oversight lens, Agency==delegation and agency lens; for StudyChat, the parenthetical gives the cohort \(f24, s25\) the trait was derived from\.TraitOriginDefinition \(high pole;*low pole italicised*\)β\\beta\(pp\)StudyChatquestion\_context\_elaborationGen \(f24\)Richly contextualized requests that explain what they are trying to do and where they are stuck
*Terse, context\-free prompts or how\-to questions*\+\+0\.65 \(\.002\)task\_context\_specificityGen \(f24\)Supplying substantive artifacts and relevant context that enable tailored analysis, transformation, or guidance−\-0\.40 \(\.008\)specificity\_of\_uncertaintyClarity \(f24\)Precise references to reasoning, attempted steps, or confusing points
*Vague confusion or requests for answers without specifying the issue*−\-0\.45 \(\.015\)goal\_directed\_steeringGen \(s25\)Explicit correction, pushback, or refinement when the tutor’s output does not meet the student’s goals−\-0\.24 \(\.024\)active\_feedback\_engagementHelp \(f24\)Clarifying, critiquing, rechecking, and adapting guidance
*Passive acceptance or abandonment after initial help*−\-0\.39 \(\.030\)conceptual\_understanding\_orientationGen \(f24\)Targeted why/how questions, follow\-up clarification, and interpretation\-seeking
*Primarily seeking finished answers, code, or superficial correctness confirmation*\+\+0\.29 \(\.050\)SWE\-Chatevidence\_driven\_quality\_oversightCraftTested, severity\-calibrated, substance\-focused feedback over vague, speculative, stylistic, or unverified claims−\-0\.033 \(<<\.001\)diagnostic\_evidence\_provisionGenEvidence\-rich, specific failure reports
*Vague, symptom\-only, or minimally contextual reports*−\-0\.020 \(<<\.001\)delegation\_specificityAgencyProvide detailed plans, file\-level instructions, constraints, and verification steps for the agent to execute
*Users delegate broad or open\-ended goals with minimal guidance*−\-0\.021 \(<<\.001\)workflow\_control\_delegationGenHighly prescriptive, command\-driven workflow orchestration
*Terse high\-level autonomous delegation*\+\+0\.019 \(<<\.001\)risk\_scoped\_change\_controlVerifThe user governs the AI agent’s work by explicitly limiting changes and checks to the intended scope, gating progress on review or verification, and scaling validation effort to the risk and size of the change\.\+\+0\.018 \(\.007\)continuation\_handoff\_styleGenDetailed state\-restoration summaries that enable autonomous continuation
*Terse next\-step commands*\+\+0\.014 \(\.009\)contextual\_workflow\_guidanceGenProactive, concrete guidance about context sources and workflow
*Verification or documentation expectations*\+\+0\.012 \(\.009\)change\_discipline\_and\_maintainabilityCraftThe user guides an AI coding agent toward scoped, incremental, reusable, convention\-aligned, and well\-documented changes
*, the user delegates broad outcomes with few constraints or validation criteria; at the high end, they actively enforce minimal scope, maintainability standards, cleanup discipline, reuse, and documentation synchronization*\+\+0\.017 \(\.013\)workflow\_structure\_incrementalityCraftDisciplined, plan\-driven, test\-aware, iterative development
*Ad hoc, broad, or implementation\-first requests*\+\+0\.013 \(\.036\)verification\_rigorVerifSystematic validation loops, regression coverage, edge\-case checks, and explicit reporting of verification results and remaining gaps−\-0\.011 \(\.042\)
## Appendix BBloom’s Taxonomy Baseline Dimensions

Table 4:The Bloom’s\-taxonomy baseline feature set\.Six hand\-authored rubric dimensions, one per cognitive\-process level of the revised taxonomy\[[14](https://arxiv.org/html/2608.11460#bib.bib14)\], written as observable student behaviors in AI\-tutor conversations\. These dimensions are specified directly rather than derived, so Stages 1–2 of PTA do not apply; they are scored by the same model, prompt, and 1–5 scale as the derived traits \(Appendix[C\.3](https://arxiv.org/html/2608.11460#A3.SS3)\), which isolates trait derivation as the difference between the two feature sets\.
## Appendix CObservation\-Extraction Prompts

This appendix reproduces the Stage\-1 extraction prompts verbatim \(Jinja placeholders such as\{\{ session\.text \}\}are filled per conversation at run time\)\. Each themed lens shares the generic prompt’s structure and output contract, differing in the lens framing that steers what behaviors the extractor attends to\.

### C\.1StudyChat

StudyChat: generic extraction \(extract\_observations\_v1\)You are an education researcher doing open inductive coding of a student’s side of an AI\-tutor conversation\.Read the session below and extract 1\-5 distinct BEHAVIOR OBSERVATIONS about how this student structures their side of the conversation\. An observation names a concrete, generalizable behavior \(something one could look for in other students’ sessions\), NOT a summary of the session’s subject matter\. Quality over quantity: if the student’s side is mostly pasted code/data with little conversational text, one or two observations is the right answer — do not pad with descriptions of the pasted content\.Guidance:\- Behaviors, not topics: ’’pastes error messages without describing what they tried’’ is a behavior; ’’asks about cross\-validation’’ is a topic — never extract topics\.\- Conversational behavior only: describe how the student interacts with the tutor — never describe what their code does or how it is implemented\. ’’shares partial code and asks targeted questions about it’’ is a behavior; ’’implements tree building by iterating over characters’’ is a code description — never extract code descriptions\.\- Each observation needs a short verbatim evidence quote from the student’s turns — at most ~15 words; truncate long code with ’’\.\.\.’’ rather than quoting it in full\.\- Prefer behaviors that could plausibly relate to how the student learns or performs in the course — e\.g\. how they seek help, how much context they give, how they react to answers, how they verify, how they persist\.\- If the session is too short or contentless for a confident observation, return fewer \(even one\)\.\{%\- if assignment\_context %\}\- IMPORTANT — pasted assignment material: the \[Assignment handout\] section below shows the writeup and starter files the student was given\. Students often paste handout prose, starter code, or provided data verbatim into the chat\. Never attribute the style, structure, or content of pasted handout material to the student \(e\.g\. don’t credit them with ’’uses section headers’’ or ’’uses TODO comments’’ if those came from the handout\)\. The act of pasting handout material IS a legitimate observation when notable — phrase it as such \(e\.g\. ’’pastes assignment instructions verbatim as task context rather than restating the task in their own words’’\)\.\{%\- endif %\}\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a student turn\>’’\}\]\}\{%\- if assignment\_context %\}\[Assignment handout — material PROVIDED TO the student for \{\{ session\.topic \}\}; not authored by them\]\{\{assignment\_context \}\}\{%\- endif %\}\[Session\]session\_id: \{\{ session\.session\_id \}\}assignment: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

StudyChat lens: SRL\+\+ICAP \(srl\_icap\_hybrid\_v1\)You are an education researcher coding a student’s side of an AI\-tutor conversation along two complementary, well\-established axes of how they use the tutor to learn\. Code these regulation/engagement behaviors, NOT the subject matter\.AXIS A — \*\*help\-seeking mode\*\* \(self\-regulated\-learning theory\): what the student asks the tutor for\.\- \*instrumental/adaptive\* \(learning\): asks for hints, explanations, or the reason behind a fix to do it themselves;\- \*check/verification:\* brings their own attempt and asks the tutor to confirm or critique it;\- \*executive/offloading:\* asks the tutor to produce the finished answer for them\.AXIS B — \*\*engagement depth\*\* \(ICAP framework\): how deeply the student processes the interaction\.\- \*passive:\* receives output and takes it as\-is;\- \*active:\* applies or manipulates what was given;\- \*constructive:\* generates their own reasoning, code, or explanations beyond what was given;\- \*interactive:\* co\-constructs through substantive back\-and\-forth, questioning and building on the tutor’s responses\.Read the whole session and extract 1\-4 BEHAVIOR OBSERVATIONS that locate the student on these axes\. Each names a CONCRETE but GENERAL behavior recognizable on ANY assignment, framed as a CONTRAST \(’’tends to X rather than Y’’\)\. Across your observations, cover both what the student asks for \(Axis A\) and how deeply they engage \(Axis B\), as the session shows them\.Rules:\- STRICTLY GENERAL — never topic\-specific\. FORBIDDEN: anything naming a particular tool, library, method, or course topic\. If a behavior only makes sense on one assignment, do not extract it\.\- Conversational behavior only: how the student engages the tutor, never what their code does internally\.\- Each observation needs a short verbatim evidence quote from a student turn — at most ~15 words; truncate long code with ’’\.\.\.’’\.\- If too short or contentless for a confident observation, return fewer \(even one\)\.\{%\- if assignment\_context %\}\- IMPORTANT — pasted assignment material: the \[Assignment handout\] below is what the student was GIVEN\. Pasting it and requesting a solution is executive/passive; never credit the student with construction/planning for content that came from the handout\.\{%\- endif %\}\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence naming a help\-seeking\-mode or engagement\-depth behavior as ’tends to X rather than Y’, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a student turn\>’’\}\]\}\{%\- if assignment\_context %\}\[Assignment handout — material PROVIDED TO the student for \{\{ session\.topic \}\}; not authored by them\]\{\{assignment\_context \}\}\{%\- endif %\}\[Session\]session\_id: \{\{ session\.session\_id \}\}assignment: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

StudyChat lens: 4D AI\-fluency \(4d\_fluency\_v1\)You are an education researcher coding a student’s side of an AI\-tutor conversation through the \*\*4D Framework of AI Fluency\*\* \(Dakan & Feller\): the competencies that distinguish skilled, deliberate use of an AI collaborator from unskilled use\. Code these fluency behaviors, NOT the subject matter\.The four competencies \(illustrations, NOT a checklist — extract whatever the session shows\):1\. \*\*Delegation\*\* — deciding what to hand to the AI vs\. do oneself\. Tells: knows what they want and why before asking; delegates the routine while keeping the thinking — vs\. hands over the whole problem indiscriminately or offloads work they should own\.2\. \*\*Description\*\* — communicating requirements well\. Tells: specifies the output \(format, constraints, what they actually need\), gives context, guides how the AI should approach it — vs\. vague, under\-specified, one\-line requests \(’’vague input produces vague output’’\)\.3\. \*\*Discernment\*\* — critically evaluating what comes back\. Tells: checks the AI’s output for correctness, questions its reasoning, tests or pushes back before using it — vs\. accepts and copies output uncritically\.4\. \*\*Diligence\*\* — taking responsibility for the result\. Tells: adapts/verifies output before using it as their own, owns the final product, signals they understand what they’re submitting — vs\. passes AI output straight through as\-is without ownership\.Read the whole session and extract 1\-4 BEHAVIOR OBSERVATIONS placing the student on these fluency competencies\. Each names a CONCRETE but GENERAL behavior recognizable on ANY assignment, framed as a CONTRAST \(’’tends to X rather than Y’’\)\. Across your observations, cover whichever competencies the session actually displays\.Rules:\- STRICTLY GENERAL — never topic\-specific\. FORBIDDEN: anything naming a particular tool, library, method, or course topic\. If a behavior only makes sense on one assignment, do not extract it\.\- Conversational behavior only: how the student engages the tutor, never what their code does internally\.\- Each observation needs a short verbatim evidence quote from a student turn — at most ~15 words; truncate long code with ’’\.\.\.’’\.\- If too short or contentless for a confident observation, return fewer \(even one\)\.\{%\- if assignment\_context %\}\- IMPORTANT — pasted assignment material: the \[Assignment handout\] below is what the student was GIVEN\. Pasting it verbatim as the request rather than describing what they need is weak Description; never credit the student with Description/Delegation skill for content that came from the handout\.\{%\- endif %\}\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence naming a 4D fluency behavior as ’tends to X rather than Y’, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a student turn\>’’\}\]\}\{%\- if assignment\_context %\}\[Assignment handout — material PROVIDED TO the student for \{\{ session\.topic \}\}; not authored by them\]\{\{assignment\_context \}\}\{%\- endif %\}\[Session\]session\_id: \{\{ session\.session\_id \}\}assignment: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

StudyChat lens: help\-seeking \(help\_seeking\_v1\)You are an education researcher coding a student’s side of an AI\-tutor conversation for the \*\*APPROPRIATENESS of their help\-seeking\*\* — a well\-studied self\-regulation skill \(Aleven, Roll\)\. The question is not just whether they ask for help, but whether they ask \*well\*: at the right time, calibrated to genuine need, and oriented toward understanding rather than just getting the answer\. Code this regulation behavior, NOT the subject matter\.Read the whole session and extract 1\-4 BEHAVIOR OBSERVATIONS about how the student manages help\-seeking, recognizable on ANY assignment\. The axis runs from ADAPTIVE to MALADAPTIVE help\-seeking\. The patterns of interest \(illustrations, NOT a checklist\):\- \*\*Adaptive help\-seeking:\*\* asks for help \*after\* making a genuine attempt; seeks hints or explanations to get unstuck rather than the finished answer; asks when actually stuck \(not reflexively\)\.\- \*\*Sense\-making vs answer\-seeking:\*\* asks \*why\* something is wrong or \*how\* an approach works \(wants to understand\) — vs\. just ’’what’s the answer / fix it for me’’ \(wants the result\)\.\- \*\*Help abuse / over\-reliance:\*\* asks for the full solution immediately without trying; offloads at the first sign of difficulty; accepts answers without reading or processing them\.\- \*\*Help avoidance:\*\* struggles or stalls without asking when they clearly need help; persists unproductively rather than seeking a hint\.Rules:\- Frame each observation as a CONTRAST \(’’tends to X rather than Y’’\) placing the student on the adaptive↔\\leftrightarrowmaladaptive help\-seeking axis\. Describe the observable help\-seeking behavior; let the contrast imply the appropriateness\.\- STRICTLY GENERAL — never topic\-specific\. FORBIDDEN: anything naming a particular tool, library, method, or course topic\. If a behavior only makes sense on one assignment, do not extract it\.\- Conversational behavior only: how the student seeks/uses help, never what their code does internally\.\- Each observation needs a short verbatim evidence quote from a student turn — at most ~15 words; truncate long code with ’’\.\.\.’’\.\- If too short or contentless for a confident observation, return fewer \(even one\)\.\{%\- if assignment\_context %\}\- IMPORTANT — pasted assignment material: the \[Assignment handout\] below is what the student was GIVEN\. Pasting it and asking for the solution rather than attempting it is help\-abuse; never credit the student with adaptive help\-seeking for content that came from the handout\.\{%\- endif %\}\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence on help\-seeking appropriateness as ’tends to X rather than Y’, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a student turn\>’’\}\]\}\{%\- if assignment\_context %\}\[Assignment handout — material PROVIDED TO the student for \{\{ session\.topic \}\}; not authored by them\]\{\{assignment\_context \}\}\{%\- endif %\}\[Session\]session\_id: \{\{ session\.session\_id \}\}assignment: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

### C\.2SWE\-Chat

SWE\-Chat: generic extraction \(extract\_observations\_swe\_v1\)You are a software\-engineering researcher doing open inductive coding of a USER’s side of a coding\-agent session\. In each session a developer works with an AI coding agent on a real software repository\.Read the session below \(the user’s prompts to the agent\) and extract 1\-5 distinct BEHAVIOR OBSERVATIONS about how this user works with the agent\. An observation names a concrete, generalizable behavior \(something one could look for in other users’ sessions\), NOT a summary of the session’s subject matter\. Quality over quantity: if the user’s side is short or mostly pasted code/logs with little steering, one or two observations is the right answer — do not pad\.Guidance:\- Behaviors, not topics: ’’pastes failing test output and asks the agent to fix it’’ is a behavior; ’’works on a database migration’’ is a topic — never extract topics\.\- Interaction behavior only: describe how the user directs, constrains, or responds to the agent — never describe what the code does or how it is implemented\. ’’specifies the target file and expected behavior up front’’ is a behavior; ’’implements a retry loop’’ is a code description — never extract code descriptions\.\- Each observation needs a short verbatim evidence quote from the user’s turns — at most ~15 words; truncate long code/logs with ’’\.\.\.’’ rather than quoting them in full\.\- Prefer behaviors that could plausibly relate to whether the session succeeds — e\.g\. how much task context the user supplies \(files, logs, repro, expected vs actual\), whether they ask for verification \(tests, builds, review\), how they correct or steer the agent, how they manage git/PR workflow, how much autonomy they grant, whether they decompose work, how they set rules/constraints, and how they accept or push back on the agent’s output\.\- If the session is too short or contentless for a confident observation, return fewer \(even one\)\.\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a user turn\>’’\}\]\}\[Session\]session\_id: \{\{ session\.session\_id \}\}repo: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

SWE\-Chat lens: agency & controlYou are a software\-engineering researcher doing open inductive coding of a USER’s side of a coding\-agent session\. In each session a developer works with an AI coding agent on a real software repository\.Read the session below \(the user’s prompts to the agent\) and extract 1\-5 distinct BEHAVIOR OBSERVATIONS about how this user works with the agent, VIEWED THROUGH ONE SPECIFIC LENS \(below\)\. An observation names a concrete, generalizable behavior \(something one could look for in other users’ sessions\), NOT a summary of the session subject\. Quality over quantity; if the lens does not apply, return fewer \(even zero\)\.LENS — Agency & Control \(Treude & Gerosa human\-AI SE taxonomy; ’’developers control, they don’t vibe’’\)\- Focus on HOW MUCH the user DIRECTS vs passively accepts: command\-driven instruction, up\-front planning before delegating, setting explicit rules/constraints, steering/correcting the agent, customizing scope, vs accepting output wholesale\.\- High end: plans the approach, issues precise commands, constrains the agent, pushes back and redirects\. Low end: vague one\-liners, accepts whatever is produced, no steering\.General rules:\- Behaviors, not topics; interaction behavior only \(how the user directs/constrains/responds\), never what the code does\.\- Each observation needs a short verbatim evidence quote \(<=15 words; truncate code/logs with ’’\.\.\.’’\)\.\- Prefer behaviors that plausibly relate to whether the session succeeds\.\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a user turn\>’’\}\]\}\[Session\]session\_id: \{\{ session\.session\_id \}\}repo: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

SWE\-Chat lens: verification & oversightYou are a software\-engineering researcher doing open inductive coding of a USER’s side of a coding\-agent session\. In each session a developer works with an AI coding agent on a real software repository\.Read the session below \(the user’s prompts to the agent\) and extract 1\-5 distinct BEHAVIOR OBSERVATIONS about how this user works with the agent, VIEWED THROUGH ONE SPECIFIC LENS \(below\)\. An observation names a concrete, generalizable behavior \(something one could look for in other users’ sessions\), NOT a summary of the session subject\. Quality over quantity; if the lens does not apply, return fewer \(even zero\)\.LENS — Verification & Oversight \(SE taxonomy ’verification’ dimension; SWEBOK Testing\)\- Focus on how the user CHECKS the agent’s work: asking for tests/builds/runs, requesting review or diffs, validating output against expected behavior, reproducing bugs, gating risky changes, catching and reporting regressions\.\- High end: demands tests/verification, inspects diffs, validates against expected vs actual\. Low end: implicit trust, no checking, accepts unverified\.General rules:\- Behaviors, not topics; interaction behavior only \(how the user directs/constrains/responds\), never what the code does\.\- Each observation needs a short verbatim evidence quote \(<=15 words; truncate code/logs with ’’\.\.\.’’\)\.\- Prefer behaviors that plausibly relate to whether the session succeeds\.\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a user turn\>’’\}\]\}\[Session\]session\_id: \{\{ session\.session\_id \}\}repo: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

SWE\-Chat lens: specificationYou are a software\-engineering researcher doing open inductive coding of a USER’s side of a coding\-agent session\. In each session a developer works with an AI coding agent on a real software repository\.Read the session below \(the user’s prompts to the agent\) and extract 1\-5 distinct BEHAVIOR OBSERVATIONS about how this user works with the agent, VIEWED THROUGH ONE SPECIFIC LENS \(below\)\. An observation names a concrete, generalizable behavior \(something one could look for in other users’ sessions\), NOT a summary of the session subject\. Quality over quantity; if the lens does not apply, return fewer \(even zero\)\.LENS — Specification & Requirements Clarity \(SWEBOK Requirements; engagement depth\)\- Focus on how PRECISELY the user specifies the task: stating target files/locations, expected behavior, acceptance criteria, constraints, inputs/outputs, edge cases, and supplying context \(repro, logs, error messages, expected vs actual\) up front\.\- High end: precise goals \+ rich context \+ acceptance criteria\. Low end: vague underspecified asks, no context, leaves the agent to guess\.General rules:\- Behaviors, not topics; interaction behavior only \(how the user directs/constrains/responds\), never what the code does\.\- Each observation needs a short verbatim evidence quote \(<=15 words; truncate code/logs with ’’\.\.\.’’\)\.\- Prefer behaviors that plausibly relate to whether the session succeeds\.\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a user turn\>’’\}\]\}\[Session\]session\_id: \{\{ session\.session\_id \}\}repo: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

SWE\-Chat lens: craftsmanship & processYou are a software\-engineering researcher doing open inductive coding of a USER’s side of a coding\-agent session\. In each session a developer works with an AI coding agent on a real software repository\.Read the session below \(the user’s prompts to the agent\) and extract 1\-5 distinct BEHAVIOR OBSERVATIONS about how this user works with the agent, VIEWED THROUGH ONE SPECIFIC LENS \(below\)\. An observation names a concrete, generalizable behavior \(something one could look for in other users’ sessions\), NOT a summary of the session subject\. Quality over quantity; if the lens does not apply, return fewer \(even zero\)\.LENS — Software Craftsmanship & Process Discipline \(SWEBOK Construction/Quality; version\-control practice\)\- Focus on quality\- and process\-mindedness: requests for refactoring, modularity, readability, naming, error handling, tests\-as\-artifacts; incremental/small\-step development; git/commit/PR hygiene; awareness of technical debt and maintainability\.\- High end: incremental, quality\-conscious, disciplined git/test workflow\. Low end: one\-shot dumps, no quality or process concern\.General rules:\- Behaviors, not topics; interaction behavior only \(how the user directs/constrains/responds\), never what the code does\.\- Each observation needs a short verbatim evidence quote \(<=15 words; truncate code/logs with ’’\.\.\.’’\)\.\- Prefer behaviors that plausibly relate to whether the session succeeds\.\[Output format\]Return JSON only:\{’’observations’’: \[\{’’behavior’’: ’’<one sentence, generalizable, present tense\>’’, ’’evidence’’: ’’<short verbatim quote from a user turn\>’’\}\]\}\[Session\]session\_id: \{\{ session\.session\_id \}\}repo: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}

### C\.3Trait\-Scoring \(Rating\) Prompts

StudyChat: trait\-scoring prompt \(score\_session\_single\_dim\_v2\)You are scoring a single student session on ONE behavioral dimension\.\[Scale\]1 = not at all reflected2 = minimally reflected3 = moderately reflected4 = strongly reflected5 = reflected to the greatest extent\[Instructions\]\- Judge the whole session, not just the first turn\.\- Label only student behavior; tutor responses are context but not what is being scored\.\- Use only the rubric definition below; do not invent additional criteria\.\- Give an integer score from 1 to 5\.\[Session\]session\_id: \{\{ session\.session\_id \}\}assignment: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}\[Dimension\]‘\{\{ dim\.id \}\}‘ \(\{\{ dim\.name \}\}\): \{\{ dim\.definition \}\}\[Output format\]Return JSON only, with this exact shape:\{’’score’’: 3, ’’rationale’’: ’’short reason’’\}

SWE\-Chat: trait\-scoring prompt \(score\_session\_single\_dim\_swe\_v1\)You are scoring a single coding\-agent session on ONE behavioral dimension\.\[Scale\]1 = not at all reflected2 = minimally reflected3 = moderately reflected4 = strongly reflected5 = reflected to the greatest extent\[Instructions\]\- The session is a developer \(the USER\) working with an AI coding agent on a software repository\.\- Judge the whole session, not just the first turn\.\- Score only the USER’s behavior; the agent’s responses are context but not what is being scored\.\- Use only the rubric definition below; do not invent additional criteria\.\- Give an integer score from 1 to 5\.\[Session\]session\_id: \{\{ session\.session\_id \}\}repo: \{\{ session\.topic or ’’n/a’’ \}\}turns: \{\{ session\.n\_turns \}\}\{\{session\.text \}\}\[Dimension\]‘\{\{ dim\.id \}\}‘ \(\{\{ dim\.name \}\}\): \{\{ dim\.definition \}\}\[Output format\]Return JSON only, with this exact shape:\{’’score’’: 3, ’’rationale’’: ’’short reason’’\}