超越作答评分：面向数学简答题自动评分的人工智能感知认知监督框架

# Beyond Response Scoring: An AI-aware Cognitive Supervision Framework for Automated Mathematical Short Answer Grading
Source: [https://arxiv.org/html/2610.41215](https://arxiv.org/html/2610.41215)
###### Abstract

Transformer\-based language models have achieved remarkable success in automated short answer grading \(ASAG\)\. However, current methods in ASAG struggle to comprehend, reason about and evaluate complex mathematical answers involving symbols and formulas\. These score\-only methods fail to generalize across diverse responses and skill states associated with identical scores\. Moreover, they ignore the potential upward bias introduced by generative Artificial Intelligence \(AI\) usage in assessing students’ performance and knowledge mastery\. To address these limitations, we proposeAICG\(AI\-awareCognitive\-supervisedGrading\), a two\-branch multi\-task framework\. The first branch combines RoBERTa\-based regression, ordinal, and classification heads for individual\-answer grading\. The second branch uses an expert\-designed Q\-matrix to associate responses with assessed skills and jointly predicts G\-DINA\-derived skill mastery and learning gain\. Through the shared encoder, these skill\-level targets provide cognitive supervision for answer grading beyond score labels\. AICG further uses a Gen\-AI detector, trained through supervised contrastive pretraining followed by linear classification, to estimate assistance probabilities and convert them into reliability weights, attenuating unreliable cognitive evidence during learning assessment\. We conduct experiments on 49,402 instructor\-graded responses spanning 19 academic semesters\. Relative to RoBERTa regression, AICG reduces mean absolute error \(MAE\) by 50\.3% and improves quadratic weighted kappa \(QWK\) from 0\.798 to 0\.927\. Ablations identify Q\-matrix\-guided cognitive supervision as the primary source of improvement, while AI\-aware weighting further reduces overall MAE by 4\.2% over the unweighted cognitive model and consistently improves MAE and QWK across the four most recent post\-Gen\-AI semesters\.

Department of Computer Science

North Carolina State University

Raleigh, NC, USA

## Introduction

Automated grading of mathematical short answerssits at the intersection of*natural language understanding*,*symbolic reasoning*, and*cognitive assessment*: responses interleave prose and derivations, while partial credit depends on demonstrated reasoning rather than only the final answer\. With generative AI, the same fluent, high\-scoring response may reflect genuine mastery or a borrowed solution, increasing the importance of accurate and reliable grading\. Existing work approaches this problem from two complementary directions:*individual\-answer grading*, which maps each response to a score, and*Q\-matrix\-based skill modeling*, which grounds assessment in an explicit cognitive structure\. Both have made substantial progress, but neither addresses their complementary limitations or the reliability threat introduced by generative AI\.

Individual\-answer grading\.In automated short\-answer grading \(ASAG\), pretrained language models and LLMs encode response semantics with substantially less manual feature engineering\([Burrows et al\. 2015](https://arxiv.org/html/2610.41215#bib.bib1);[Latif and Zhai 2024](https://arxiv.org/html/2610.41215#bib.bib5);[Ferreira Mello et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib10);[Chu et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib11)\)\. However, mathematical responses interleave natural\-language explanations, symbolic formulas, and incomplete derivations, while partial credit may depend on multiple reasoning principles\. Current language models remain unreliable in domain\-specific symbolic and compositional reasoning\([Latif and Zhai 2024](https://arxiv.org/html/2610.41215#bib.bib5);[Jiang and Bosch 2024](https://arxiv.org/html/2610.41215#bib.bib12);[Ferreira Mello et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib10)\)and may misinterpret valid intermediate or mathematically equivalent work\. More fundamentally, supervision is compressed into a scalar label: identical scores can represent different reasoning and skill states, limiting generalization to unseen solution patterns\. Figure[1](https://arxiv.org/html/2610.41215#Sx1.F1)illustrates this difficulty: the student correctly applies conditional independence twice but stops before Bayes’ rule; the score of0\.600\.60records the earned credit but not the underlying reasoning pattern\.

Q\-matrix\-based skill modeling\.A complementary perspective explicitly represents the cognitive structure of assessment\. A Q\-matrix associates items with skills, while cognitive\-diagnosis models such as G\-DINA infer mastery from response patterns\([de la Torre 2011](https://arxiv.org/html/2610.41215#bib.bib3)\)\. Student and question representations complement answer features\([Zhang et al\. 2020](https://arxiv.org/html/2610.41215#bib.bib8)\), and neural cognitive\-diagnosis models demonstrate the value of explicit skill structure\([Wang et al\. 2020](https://arxiv.org/html/2610.41215#bib.bib4);[Liu et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib14);[Qian et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib15);[Gu et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib16)\)\. Figure[1](https://arxiv.org/html/2610.41215#Sx1.F1)illustrates this value: the missing Bayes step indicates weak Bayesian reasoning, while the Q\-matrix links this evidence across items assessing the same skill\. However, these models rely on binary item\-response records and leave the semantic evidence in free\-text solutions unused\. Generative AI further complicates assessment because an assisted answer may earn its assigned grade while overstating independent mastery\. Although recent detectors identify machine\-generated text\([Hans et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib17);[Bao et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib18)\), hard filtering remains vulnerable to rewriting and distribution shift and may disproportionately affect non\-native writers\([Sadasivan et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib9);[Liang et al\. 2023](https://arxiv.org/html/2610.41215#bib.bib7)\), making binary removal unsuitable\.

Question\.Givenp⁡\(O\)=0\.5p\(O\)=0\.5, supposeAAandBBare conditionally independent givenOOandO¯\\bar\{O\}\. Letp⁡\(A∣O\)=p⁡\(A¯∣O¯\)=0\.8p\(A\\mid O\)=p\(\\bar\{A\}\\mid\\bar\{O\}\)=0\.8,p⁡\(B∣O\)=0\.75p\(B\\mid O\)=0\.75, andp⁡\(B∣O¯\)=0\.5p\(B\\mid\\bar\{O\}\)=0\.5\. Determinep⁡\(O∣A∩B\)p\(O\\mid A\\cap B\)\.Reference AnswerPrinciple/WeightDefinition of Conditional Independent Events \(0\.3\)p⁡\(A∩B∣O\)=p⁡\(A∣O\)∗P⁡\(B∣O\)=0\.8∗0\.75=0\.6\.p\(A\\cap B\\mid O\)=p\(A\\mid O\)\*P\(B\\mid O\)=0\.8\*0\.75=0\.6\.Definition of Conditional Independent Events \(0\.3\)p⁡\(A∩B∣∼O\)=p⁡\(A∣∼O\)∗P⁡\(B∣∼O\)=\(1−0\.8\)∗0\.5=0\.10\.p\(A\\cap B\\mid\{\\sim\}O\)=p\(A\\mid\{\\sim\}O\)\*P\(B\\mid\{\\sim\}O\)=\(1\-0\.8\)\*0\.5=0\.10\.Bayes’s Rule \(0\.4\)p⁡\(O∣A∩B\)=p⁡\(A∩B∣O\)∗p⁡\(O\)p⁡\(A∩B∣O\)∗p⁡\(O\)\+p⁡\(A∩B∣∼O\)∗p\(∼O\)\\displaystyle p\(O\\mid A\\cap B\)=\\frac\{p\(A\\cap B\\mid O\)\*p\(O\)\}\{p\(A\\cap B\\mid O\)\*p\(O\)\+p\(A\\cap B\\mid\{\\sim\}O\)\*p\(\{\\sim\}O\)\}=0\.6∗0\.5/\(0\.6∗0\.5\+0\.1∗0\.5\)=0\.857143\.=0\.6\*0\.5/\(0\.6\*0\.5\+0\.1\*0\.5\)=0\.857143\.Student AnswerTotal score: 0\.60Definition of Conditional Independent Events: 1*conditionally independent events:*p⁡\(A∩B∣O\)=p⁡\(A∣O\)∗p⁡\(B∣O\)p\(A\\cap B\\mid O\)=p\(A\\mid O\)\*p\(B\\mid O\)\-\-\-\>​p​\(A∩B∣O\)=0\.8∗0\.75=0\.6\\texttt\{\-\-\-\>\}\\;p\(A\\cap B\\mid O\)=0\.8\*0\.75=0\.6Definition of Conditional Independent Events: 1p⁡\(A∩B∣∼O\)=p⁡\(A∣∼O\)∗p⁡\(B∣∼O\)p\(A\\cap B\\mid\{\\sim\}O\)=p\(A\\mid\{\\sim\}O\)\*p\(B\\mid\{\\sim\}O\)\-\-\-\>​0\.2∗0\.5=0\.1\\texttt\{\-\-\-\>\}\\;0\.2\*0\.5=0\.1Bayes’s Rule: 0*I don’t know how to do the rest*Weighted score:0\.3​\(1\)\+0\.3​\(1\)\+0\.4​\(0\)=0\.600\.3\(1\)\+0\.3\(1\)\+0\.4\(0\)=0\.60\.

Figure 1:Alignment between weighted reference principles and partial\-credit evidence\. Blue, green, and red boxes denote reference steps, demonstrated principles, and the missing application of Bayes’s rule, respectively\.We proposeAICG\(AI\-awareCognitive\-supervisedGrading\), a two\-branch multi\-task framework\. The first branch applies regression, ordinal, and classification heads to a shared RoBERTa representation\([Liu et al\. 2019](https://arxiv.org/html/2610.41215#bib.bib2)\), capturing complementary continuous, ordered, and categorical grading signals\. The second branch uses the Q\-matrix to aggregate skill\-relevant responses and predicts G\-DINA\-derived pretest mastery, posttest mastery, and learning gain\. Joint optimization transfers this cognitive supervision to the shared response encoder without feeding predicted grades into the cognitive branch\. AICG further trains a Gen\-AI detector through supervised contrastive pretraining\([Khosla et al\. 2020](https://arxiv.org/html/2610.41215#bib.bib19)\)followed by linear classification\. The estimated assistance probabilities are converted into soft reliability weights applied only to cognitive evidence, preserving the grading treatment of each response\. Our contributions are threefold:

- •AICG is the first Transformer\-based ASAG framework to jointly optimize answer grading with Q\-matrix\-aligned mastery and learning\-gain supervision\.
- •We introduce AI\-aware reliability weighting that attenuates potentially assisted cognitive evidence without discarding responses or altering their grades\.
- •AICG outperforms five strong baselines in a ten\-year expanding\-window evaluation while maintaining robust cross\-semester generalization\.

## Methodology

Figure[2](https://arxiv.org/html/2610.41215#Sx2.F2)presents AICG, which comprises two major components\. The first is the*Gen\-AI Detector*, which assesses AI involvement in responses ranging from partially assisted to fully generated and assigns each response a continuous AI\-assistance probability\. The second is a multi\-task grading component built on a separate RoBERTa encoder shared by two branches\. The*Individual\-Response Grading*branch predicts continuous, ordinal, and categorical grades from each response’s semantic representation\. The*Q\-Matrix\-Based Cognitive Supervision*branch combines Q\-matrix\-guided skill selection with AI\-aware evidence weighting to predict pretest mastery, posttest mastery, and learning gain, thereby supervising grading with cognitive information\. The Gen\-AI Detector is used only to adjust the reliability of cognitive evidence\. The two branches retain independent heads while jointly updating the shared RoBERTa encoder\.

![Refer to caption](https://arxiv.org/html/2610.41215v1/Figures/AICG.png)Figure 2:Overview of AICG\. The Gen\-AI Detector supplies AI\-assistance probabilities only to the Q\-Matrix\-Based Cognitive Supervision branch\. The shared RoBERTa encoder supports parallel cognitive\-supervision and individual\-response\-grading branches\. Solid and dashed arrows denote prediction and supervision flows, respectively\.### Problem Formulation

Let𝒮\\mathcal\{S\}denote the student set, withssindexing a student; let𝒯=\{pre,post\}\\mathcal\{T\}=\\\{\\mathrm\{pre\},\\mathrm\{post\}\\\}denote the assessment phases, witht∈𝒯t\\in\\mathcal\{T\}indexing a phase; and let𝒦=\{1,…,K\}\\mathcal\{K\}=\\\{1,\\ldots,K\\\}denote the cognitive\-skill set, withkkindexing a skill andK=10K=10\. For studentssin phasett,ℛst\\mathcal\{R\}\_\{s\}^\{t\}denotes the observed\-response set, withrrindexing a response\.

For responserrfrom studentss,xsrx\_\{s\}^\{r\}denotes the textual input andysr∈\[0,1\]y\_\{s\}^\{r\}\\in\[0,1\]the normalized instructor\-assigned score\. The inputxsrx\_\{s\}^\{r\}concatenates the question identifier, student answer, and principle metadata using separator tokens\. The Q\-matrix vector𝐪sr∈\{0,1\}K\\mathbf\{q\}\_\{s\}^\{r\}\\in\\\{0,1\\\}^\{K\}associates the response with assessed skills:q⁡\(s,r,k\)∈\{0,1\}q\(s,r,k\)\\in\\\{0,1\\\}indicates whether the corresponding item assesses skillkk\. The assessment phase and Q\-matrix vector remain structured inputs rather than being appended to the response text\.

At the response level,y^sr∈\[0,1\]\\hat\{y\}\_\{s\}^\{r\}\\in\[0,1\]denotes the predicted continuous score\. The ground\-truth and predicted ordered categories are denoted bydsr∈\{0,1,2,3,4\}d\_\{s\}^\{r\}\\in\\\{0,1,2,3,4\\\}andd^sr∈\{0,1,2,3,4\}\\hat\{d\}\_\{s\}^\{r\}\\in\\\{0,1,2,3,4\\\}, respectively\. The ground\-truth category is derived fromysry\_\{s\}^\{r\}using boundaries0\.1250\.125,0\.3750\.375,0\.6250\.625, and0\.8750\.875\.

At the student–skill level,c~​\(s,k,t\)∈\[0,1\]\\widetilde\{c\}\(s,k,t\)\\in\[0,1\]denotes the reliability\-adjusted G\-DINA mastery target, andc^​\(s,k,t\)∈\[0,1\]\\hat\{c\}\(s,k,t\)\\in\[0,1\]its prediction\. The reliability\-adjusted learning\-gain targetg~sk∈\[−1,1\]\\widetilde\{g\}\_\{s\}^\{k\}\\in\[\-1,1\]is the difference between the corresponding posttest and pretest mastery targets, whileg^sk∈\[−1,1\]\\hat\{g\}\_\{s\}^\{k\}\\in\[\-1,1\]denotes its prediction\. AICG is supervised byc~​\(s,k,t\)\\widetilde\{c\}\(s,k,t\)andg~sk\\widetilde\{g\}\_\{s\}^\{k\}, which are derived from AI\-aware, Q\-matrix\-aligned response evidence\.

### Gen\-AI Detector

An AI\-assisted response may provide unreliable evidence of independent mastery\. We therefore train the Gen\-AI Detector separately to estimate continuous AI\-assistance probabilities\.

#### AI\-Assisted Response Construction\.

Complete LLM rewriting can introduce stylistic artifacts that encourage shortcut learning\. We instead mask 30%–50% of each authentic response and use an offline DeepSeek model to reconstruct the masked tokens, preserving 50%–70% of the student text\. Original responses are labeled human\-written; partially completed and fully generated correct responses form the AI\-assisted class\. Each source response and its variants remain in the same partition to prevent leakage\. Classes are matched by item, score category, and length, and generator transfer is evaluated through leave\-one\-generator\-out splits\. Further details are provided in Appendix\.

#### Supervised Contrastive Pretraining\.

For responserr,𝐯r\\mathbf\{v\}\_\{r\}represents the textual input to the Gen\-AI Detector, andar∈\{0,1\}a\_\{r\}\\in\\\{0,1\\\}represents its source label, withar=0a\_\{r\}=0for human\-written text andar=1a\_\{r\}=1for partially assisted or fully generated text\. Training uses balanced mini\-batches with equal numbers of responses from both classes\.

The Gen\-AI detector employs a RoBERTa encoderRoBERTadet\\operatorname\{RoBERTa\}\_\{\\mathrm\{det\}\}, where\[CLS\]\\mathrm\{\[CLS\]\}denotes the final hidden state of the sequence\-initial classification token\. The learnable projection matrix and bias are represented by𝐖proj\\mathbf\{W\}\_\{\\mathrm\{proj\}\}and𝐛proj\\mathbf\{b\}\_\{\\mathrm\{proj\}\}, respectively\. The projected representation𝐮r\\mathbf\{u\}\_\{r\}is calculated as:

𝐮r=𝐖proj​RoBERTadet​\(𝐯r\)\[CLS\]\+𝐛proj\.\\mathbf\{u\}\_\{r\}=\\mathbf\{W\}\_\{\\mathrm\{proj\}\}\\operatorname\{RoBERTa\}\_\{\\mathrm\{det\}\}\\left\(\\mathbf\{v\}\_\{r\}\\right\)\_\{\\mathrm\{\[CLS\]\}\}\+\\mathbf\{b\}\_\{\\mathrm\{proj\}\}\.\(1\)
The normalized contrastive embedding is denoted by𝐞r\\mathbf\{e\}\_\{r\}, and∥⋅∥2\\\|\\cdot\\\|\_\{2\}denotes the Euclidean norm\. It is computed as:

𝐞r=𝐮r‖𝐮r‖2\.\\mathbf\{e\}\_\{r\}=\\frac\{\\mathbf\{u\}\_\{r\}\}\{\\\|\\mathbf\{u\}\_\{r\}\\\|\_\{2\}\}\.\(2\)
Letℬ\\mathcal\{B\}denote the response indices in a balanced mini\-batch\. For anchorr∈ℬr\\in\\mathcal\{B\},𝒫⁡\(r\)\\mathcal\{P\}\(r\)contains the non\-anchor responses sharing labelara\_\{r\};r\+r^\{\+\}indexes a response in𝒫⁡\(r\)\\mathcal\{P\}\(r\), andr′r^\{\\prime\}indexes a non\-anchor response inℬ\\mathcal\{B\}\. With temperatureτ=0\.05\\tau=0\.05, the supervised contrastive objectiveℒcon\\mathcal\{L\}\_\{\\mathrm\{con\}\}encourages within\-class compactness and between\-class separation\. It is formulated as:

ℒcon=−1\|ℬ\|∑r∈ℬ1\|𝒫⁡\(r\)\|∑r\+∈𝒫⁡\(r\)logexp⁡\(𝐞r⊤​𝐞r\+/τ\)∑r′∈ℬ∖\{r\}exp⁡\(𝐞r⊤​𝐞r′/τ\)\.\\mathcal\{L\}\_\{\\mathrm\{con\}\}=\-\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{r\\in\\mathcal\{B\}\}\\frac\{1\}\{\|\\mathcal\{P\}\(r\)\|\}\\sum\_\{r^\{\+\}\\in\\mathcal\{P\}\(r\)\}\\log\\frac\{\\exp\\\!\\left\(\\mathbf\{e\}\_\{r\}^\{\\top\}\\mathbf\{e\}\_\{r^\{\+\}\}/\\tau\\right\)\}\{\\displaystyle\\sum\_\{r^\{\\prime\}\\in\\mathcal\{B\}\\setminus\\\{r\\\}\}\\exp\\\!\\left\(\\mathbf\{e\}\_\{r\}^\{\\top\}\\mathbf\{e\}\_\{r^\{\\prime\}\}/\\tau\\right\)\}\.\\vskip\-2\.0pt\(3\)

#### Probability Estimation\.

After contrastive pretraining, the Gen\-AI Detector encoder and projection layer are frozen\. A two\-class linear classifier, parameterized by𝐖cls\\mathbf\{W\}\_\{\\mathrm\{cls\}\}and𝐛cls\\mathbf\{b\}\_\{\\mathrm\{cls\}\}, is trained on the frozen representations𝐮r\\mathbf\{u\}\_\{r\}using cross\-entropy loss\.

Probability calibration uses the group\-disjoint validation set𝒱cal\\mathcal\{V\}\_\{\\mathrm\{cal\}\}, whose size is\|𝒱cal\|\|\\mathcal\{V\}\_\{\\mathrm\{cal\}\}\|\. A candidate temperature is represented byT\>0T\>0, andT∗T^\{\*\}is the value selected by minimizing the average negative log\-likelihood\. The subscriptara\_\{r\}selects the probability assigned to the ground\-truth source label of responserr\. The optimal temperature is calculated as:

T∗\\displaystyle T^\{\*\}=arg⁡minT\>0−1\|𝒱cal\|∑r∈𝒱cal\\displaystyle=\\underset\{T\>0\}\{\\arg\\min\}\\;\-\\frac\{1\}\{\|\\mathcal\{V\}\_\{\\mathrm\{cal\}\}\|\}\\sum\_\{r\\in\\mathcal\{V\}\_\{\\mathrm\{cal\}\}\}\(4\)log⁡\[Softmax⁡\(𝐖cls​𝐮r\+𝐛clsT\)ar\]\.\\displaystyle\\log\\\!\\Bigg\[\\operatorname\{Softmax\}\\left\(\\frac\{\\mathbf\{W\}\_\{\\mathrm\{cls\}\}\\mathbf\{u\}\_\{r\}\+\\mathbf\{b\}\_\{\\mathrm\{cls\}\}\}\{T\}\\right\)\_\{a\_\{r\}\}\\Bigg\]\.
The calibrated AI\-assistance probability for responserris denoted byprAI∈\[0,1\]p\_\{r\}^\{\\mathrm\{AI\}\}\\in\[0,1\]\. Becausear=1a\_\{r\}=1represents partially assisted or fully generated text, this probability is computed as the class\-1 output:

prAI=Softmax⁡\(𝐖cls​𝐮r\+𝐛clsT∗\)1\.p\_\{r\}^\{\\mathrm\{AI\}\}=\\operatorname\{Softmax\}\\left\(\\frac\{\\mathbf\{W\}\_\{\\mathrm\{cls\}\}\\mathbf\{u\}\_\{r\}\+\\mathbf\{b\}\_\{\\mathrm\{cls\}\}\}\{T^\{\*\}\}\\right\)\_\{1\}\.\(5\)
For the grading dataset,pAI​\(s,r\)p^\{\\mathrm\{AI\}\}\(s,r\)denotes the calibrated AI\-assistance probability of responserrfrom studentss\. Calibration is evaluated using expected calibration error and the Brier score\. Grading\-test responses are excluded from Gen\-AI Detector training and temperature estimation\.

### Individual\-Response Grading

As shown in Figure[2](https://arxiv.org/html/2610.41215#Sx2.F2), AICG employs a RoBERTa encoder shared with the cognitive branch but separate from the Gen\-AI Detector\. For inputxsrx\_\{s\}^\{r\}, the encoder produces𝐡sr∈ℝD\\mathbf\{h\}\_\{s\}^\{r\}\\in\\mathbb\{R\}^\{D\}, the dropout\-regularized final hidden state of the sequence\-initial\[CLS\]\\mathrm\{\[CLS\]\}token, whereDDis the hidden dimension\. Regression, ordinal, and categorical heads operate independently on𝐡sr\\mathbf\{h\}\_\{s\}^\{r\}\. The regression head uses a sigmoid functionσ⁡\(⋅\)\\sigma\(\\cdot\)to produce the continuous scorey^sr\\hat\{y\}\_\{s\}^\{r\}\. The ordinal head produces𝐨sr∈ℝ4\\mathbf\{o\}\_\{s\}^\{r\}\\in\\mathbb\{R\}^\{4\}, whereo⁡\(s,r,j\)o\(s,r,j\)denotes the logit for thresholdj∈\{0,…,3\}j\\in\\\{0,\\ldots,3\\\}\. The categorical head produces𝝃sr∈ℝ5\\boldsymbol\{\\xi\}\_\{s\}^\{r\}\\in\\mathbb\{R\}^\{5\}for the five grade categories\.

For categoryd∈\{0,…,4\}d\\in\\\{0,\\ldots,4\\\},αd\\alpha\_\{d\}denotes its training weight\. We set\(α0,…,α4\)=\(1\.0,2\.0,2\.8,2\.0,0\.7\)\(\\alpha\_\{0\},\\ldots,\\alpha\_\{4\}\)=\(1\.0,2\.0,2\.8,2\.0,0\.7\)and assignηsr=αdsr\\eta\_\{s\}^\{r\}=\\alpha\_\{d\_\{s\}^\{r\}\}to responserrfrom studentss\. The mini\-batchℬgrade\\mathcal\{B\}\_\{\\mathrm\{grade\}\}contains student–response pairs, and its total weight isΩgrade=∑\(s,r\)∈ℬgradeηsr\\Omega\_\{\\mathrm\{grade\}\}=\\sum\_\{\(s,r\)\\in\\mathcal\{B\}\_\{\\mathrm\{grade\}\}\}\\eta\_\{s\}^\{r\}\.

The regression lossℒreg\\mathcal\{L\}\_\{\\mathrm\{reg\}\}usesSmoothL1β\\operatorname\{SmoothL1\}\_\{\\beta\}, which is quadratic belowβ=0\.10\\beta=0\.10and linear otherwise\. Its category\-weighted average is calculated as:

ℒreg=∑\(s,r\)∈ℬgradeηsr​SmoothL1β⁡\(y^sr,ysr\)Ωgrade\.\\mathcal\{L\}\_\{\\mathrm\{reg\}\}=\\frac\{\\displaystyle\\sum\_\{\(s,r\)\\in\\mathcal\{B\}\_\{\\mathrm\{grade\}\}\}\\eta\_\{s\}^\{r\}\\operatorname\{SmoothL1\}\_\{\\beta\}\\left\(\\hat\{y\}\_\{s\}^\{r\},y\_\{s\}^\{r\}\\right\)\}\{\\Omega\_\{\\mathrm\{grade\}\}\}\.\(6\)
For thresholdjj, the ordinal target isκ\(s,r,j\)=𝟙\[dsr\>j\]\\kappa\(s,r,j\)=\\mathbb\{1\}\[d\_\{s\}^\{r\}\>j\], where𝟙​\[⋅\]\\mathbb\{1\}\[\\cdot\]denotes the indicator function\. The positive and negative training counts areNj\+N\_\{j\}^\{\+\}andNj−N\_\{j\}^\{\-\}, respectively, and the positive\-class weight isνj=clip⁡\(Nj−/max⁡\(Nj\+,1\),0\.15,5\.0\)\\nu\_\{j\}=\\operatorname\{clip\}\(N\_\{j\}^\{\-\}/\\max\(N\_\{j\}^\{\+\},1\),0\.15,5\.0\)\. Using binary cross\-entropy with logitsBCEνj\\operatorname\{BCE\}\_\{\\nu\_\{j\}\}, the ordinal lossℒord\\mathcal\{L\}\_\{\\mathrm\{ord\}\}is calculated as:

ℒord=∑\(s,r\)∈ℬgradeηsr4​∑j=03BCEνj⁡\(o⁡\(s,r,j\),κ⁡\(s,r,j\)\)Ωgrade\.\\mathcal\{L\}\_\{\\mathrm\{ord\}\}=\\frac\{\\displaystyle\\sum\_\{\(s,r\)\\in\\mathcal\{B\}\_\{\\mathrm\{grade\}\}\}\\frac\{\\eta\_\{s\}^\{r\}\}\{4\}\\sum\_\{j=0\}^\{3\}\\operatorname\{BCE\}\_\{\\nu\_\{j\}\}\\left\(o\(s,r,j\),\\kappa\(s,r,j\)\\right\)\}\{\\Omega\_\{\\mathrm\{grade\}\}\}\.\(7\)
The categorical probabilities are𝝅sr=Softmax⁡\(𝝃sr\)\\boldsymbol\{\\pi\}\_\{s\}^\{r\}=\\operatorname\{Softmax\}\(\\boldsymbol\{\\xi\}\_\{s\}^\{r\}\), whereπ⁡\(s,r,d\)\\pi\(s,r,d\)denotes the probability of categorydd\. With focal parameterγ=2\\gamma=2, the category\-weighted classification lossℒcls\\mathcal\{L\}\_\{\\mathrm\{cls\}\}is defined as:

ℒcls=−∑\(s,r\)∈ℬgradeηsr​\(1−π⁡\(s,r,dsr\)\)γ​log⁡π⁡\(s,r,dsr\)Ωgrade\.\\mathcal\{L\}\_\{\\mathrm\{cls\}\}=\-\\frac\{\\displaystyle\\sum\_\{\(s,r\)\\in\\mathcal\{B\}\_\{\\mathrm\{grade\}\}\}\\eta\_\{s\}^\{r\}\\left\(1\-\\pi\(s,r,d\_\{s\}^\{r\}\)\\right\)^\{\\gamma\}\\log\\pi\(s,r,d\_\{s\}^\{r\}\)\}\{\\Omega\_\{\\mathrm\{grade\}\}\}\.\(8\)
The three losses form the answer\-level scoring objective\. The categorical head provides auxiliary supervision, whereas the final discrete graded^sr\\hat\{d\}\_\{s\}^\{r\}is decoded from the ordinal head and is calculated as:

d^sr=∑j=03\[σ\(o\(s,r,j\)\)≥0\.5\]\.\\hat\{d\}\_\{s\}^\{r\}=\\sum\_\{j=0\}^\{3\}\\mathbb\{1\}\\\!\\left\[\\sigma\\\!\\left\(o\(s,r,j\)\\right\)\\geq 0\.5\\right\]\.\(9\)

### Q\-Matrix\-Based Cognitive Supervision

#### Q\-Matrix\-Guided Skill Selection\.

The cognitive branch connects response embeddings to student\-level skill targets without using predicted grades\. The Q\-matrix\-selected responses for studentss, skillkk, and phasettform the setℐ⁡\(s,k,t\)\\mathcal\{I\}\(s,k,t\), which is defined asℐ⁡\(s,k,t\)=\{r∈ℛst:q⁡\(s,r,k\)=1\}\\mathcal\{I\}\(s,k,t\)=\\\{r\\in\\mathcal\{R\}\_\{s\}^\{t\}:q\(s,r,k\)=1\\\}\. The corresponding observation mask is defined asμ\(s,k,t\)=𝟙\[\|ℐ\(s,k,t\)\|\>0\]\\mu\(s,k,t\)=\\mathbb\{1\}\[\|\\mathcal\{I\}\(s,k,t\)\|\>0\]\. Only responses associated with skillkkcontribute to its representation\. Whenμ⁡\(s,k,t\)=0\\mu\(s,k,t\)=0, the entry is excluded from pooling, cognitive losses, evaluation, and reporting; any stored zero serves only as a batching placeholder\.

#### AI\-Aware Weighting\.

Q\-matrix relevance does not ensure that a response reflects independent student knowledge\. The cognitive\-evidence reliability of responserrfrom studentssis represented bywsr∈\[δ,1\]w\_\{s\}^\{r\}\\in\[\\delta,1\], whereδ=0\.30\\delta=0\.30is the minimum reliability\. Using the previously defined AI\-assistance probabilitypAI​\(s,r\)p^\{\\mathrm\{AI\}\}\(s,r\), the reliability is calculated as:

wsr=max⁡\(δ,1−pAI​\(s,r\)\)\.w\_\{s\}^\{r\}=\\max\\\!\\left\(\\delta,1\-p^\{\\mathrm\{AI\}\}\(s,r\)\\right\)\.\(10\)
This monotonic weighting attenuates responses with high estimated AI involvement without imposing a hard detection threshold\. The lower bound preserves limited evidence from responses that may still contain student\-authored reasoning\.

#### Reliability\-Weighted Cognitive Targets\.

AICG incorporates response reliability into G\-DINA inference\([de la Torre 2011](https://arxiv.org/html/2610.41215#bib.bib3)\)\. A latent mastery profile is represented by𝜻∈\{0,1\}K\\boldsymbol\{\\zeta\}\\in\\\{0,1\\\}^\{K\}, whereζk\\zeta\_\{k\}is itskkth component and indicates mastery of skillkk\. The G\-DINA item parameters are denoted by𝚯\\boldsymbol\{\\Theta\}\. The functionφ⁡\(s,r,𝜻,𝚯\)\\varphi\(s,r,\\boldsymbol\{\\zeta\};\\boldsymbol\{\\Theta\}\)represents the probability of a correct response to the item associated with responserrunder profile𝜻\\boldsymbol\{\\zeta\}\. Its skill interactions are constrained by the previously defined Q\-matrix vector𝐪sr\\mathbf\{q\}\_\{s\}^\{r\}\.

The normalized scoreysry\_\{s\}^\{r\}is treated as fractional item\-response evidence\. The reliability\-weighted G\-DINA log\-likelihood for studentssin phasettunder profile𝜻\\boldsymbol\{\\zeta\}is represented by𝒥w​\(s,t,𝜻\)\\mathcal\{J\}\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}\)\. It is computed by weighting each fractional Bernoulli contribution as:

𝒥w​\(s,t,𝜻\)=∑r∈ℛstwsr​\[ysr​log⁡φ⁡\(s,r,𝜻,𝚯\)\+\(1−ysr\)​log⁡\[1−φ⁡\(s,r,𝜻,𝚯\)\]\]\.\\mathcal\{J\}\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}\)=\\sum\_\{r\\in\\mathcal\{R\}\_\{s\}^\{t\}\}w\_\{s\}^\{r\}\\left\[\\begin\{aligned\} &y\_\{s\}^\{r\}\\log\\varphi\(s,r,\\boldsymbol\{\\zeta\};\\boldsymbol\{\\Theta\}\)\\\\\[\-1\.0pt\] &\+\(1\-y\_\{s\}^\{r\}\)\\log\\\!\\Big\[1\-\\varphi\(s,r,\\boldsymbol\{\\zeta\};\\boldsymbol\{\\Theta\}\)\\Big\]\\end\{aligned\}\\right\]\.\(11\)
The mastery\-profile prior is represented byPr0⁡\(𝜻\)\\Pr\_\{0\}\(\\boldsymbol\{\\zeta\}\), while𝜻′\\boldsymbol\{\\zeta\}^\{\\prime\}indexes alternative profiles in the posterior normalizer\. The previously defined reliability\-adjusted mastery targetc~​\(s,k,t\)\\widetilde\{c\}\(s,k,t\)is computed as the posterior expectation ofζk\\zeta\_\{k\}:

c~​\(s,k,t\)=∑𝜻∈\{0,1\}Kζk​Pr0⁡\(𝜻\)​exp⁡\(𝒥w​\(s,t,𝜻\)\)∑𝜻′∈\{0,1\}KPr0⁡\(𝜻′\)​exp⁡\(𝒥w​\(s,t,𝜻′\)\)\.\\widetilde\{c\}\(s,k,t\)=\\sum\_\{\\boldsymbol\{\\zeta\}\\in\\\{0,1\\\}^\{K\}\}\\zeta\_\{k\}\\frac\{\\Pr\_\{0\}\(\\boldsymbol\{\\zeta\}\)\\exp\\\!\\left\(\\mathcal\{J\}\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}\)\\right\)\}\{\\displaystyle\\sum\_\{\\boldsymbol\{\\zeta\}^\{\\prime\}\\in\\\{0,1\\\}^\{K\}\}\\Pr\_\{0\}\(\\boldsymbol\{\\zeta\}^\{\\prime\}\)\\exp\\\!\\left\(\\mathcal\{J\}\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}^\{\\prime\}\)\\right\)\}\.\(12\)
Accordingly, the reliability\-adjusted learning\-gain target is calculated asg~sk=c~​\(s,k,post\)−c~​\(s,k,pre\)\\widetilde\{g\}\_\{s\}^\{k\}=\\widetilde\{c\}\(s,k,\\mathrm\{post\}\)\-\\widetilde\{c\}\(s,k,\\mathrm\{pre\}\)\. G\-DINA parameters and targets are estimated separately within each historical development window; evaluation semesters are excluded from target construction\.

#### Reliability\-Weighted Skill Evidence\.

The same reliability values weight the response embeddings used by the cognitive branch\. For an observed student–skill–phase tuple,𝐡~​\(s,k,t\)∈ℝD\\widetilde\{\\mathbf\{h\}\}\(s,k,t\)\\in\\mathbb\{R\}^\{D\}represents the AI\-aware skill evidence\. It is calculated by averaging the reliability\-scaled embeddings selected by the Q\-matrix as:

𝐡~​\(s,k,t\)=1\|ℐ⁡\(s,k,t\)\|​∑r∈ℐ⁡\(s,k,t\)wsr​𝐡sr\.\\widetilde\{\\mathbf\{h\}\}\(s,k,t\)=\\frac\{1\}\{\|\\mathcal\{I\}\(s,k,t\)\|\}\\sum\_\{r\\in\\mathcal\{I\}\(s,k,t\)\}w\_\{s\}^\{r\}\\mathbf\{h\}\_\{s\}^\{r\}\.\(13\)
The denominator is the number of selected responses rather than their total reliability\. Consequently, the representation magnitude decreases when all available evidence is unreliable\.

#### Pretest, Posttest, and Learning\-Gain Modeling\.

Each skillkkhas a learnable embeddingϵk∈ℝD\\boldsymbol\{\\epsilon\}\_\{k\}\\in\\mathbb\{R\}^\{D\}\. The phase\-specific functionsfpref\_\{\\mathrm\{pre\}\}andfpostf\_\{\\mathrm\{post\}\}are shared across skills and implemented as linear–tanh\\tanh–dropout–linear–sigmoid networks\. The notationftf\_\{t\}selects the corresponding function for phasett\. The previously defined mastery predictionc^​\(s,k,t\)\\hat\{c\}\(s,k,t\)combines the AI\-aware skill representation with its skill embedding and is calculated as:

c^​\(s,k,t\)=ft​\(𝐡~​\(s,k,t\)\+ϵk\)\.\\hat\{c\}\(s,k,t\)=f\_\{t\}\\\!\\left\(\\widetilde\{\\mathbf\{h\}\}\(s,k,t\)\+\\boldsymbol\{\\epsilon\}\_\{k\}\\right\)\.\(14\)
The previously defined gain prediction is calculated asg^sk=c^​\(s,k,post\)−c^​\(s,k,pre\)\\hat\{g\}\_\{s\}^\{k\}=\\hat\{c\}\(s,k,\\mathrm\{post\}\)\-\\hat\{c\}\(s,k,\\mathrm\{pre\}\)\. Gain is observed only when skillkkappears in both phases\. The corresponding mask is defined asμgain​\(s,k\)=μ⁡\(s,k,pre\)​μ​\(s,k,post\)\\mu^\{\\mathrm\{gain\}\}\(s,k\)=\\mu\(s,k,\\mathrm\{pre\}\)\\mu\(s,k,\\mathrm\{post\}\)\.

For phasett, the observed student–skill pairs formℰt=\{\(s,k\):μ⁡\(s,k,t\)=1\}\\mathcal\{E\}\_\{t\}=\\\{\(s,k\):\\mu\(s,k,t\)=1\\\}\. The mastery lossℒt\\mathcal\{L\}\_\{t\}is the mean squared error over these observed pairs and is calculated as:

ℒt=1\|ℰt\|​∑\(s,k\)∈ℰt\(c^​\(s,k,t\)−c~​\(s,k,t\)\)2,t∈\{pre,post\}\.\\mathcal\{L\}\_\{t\}=\\frac\{1\}\{\|\\mathcal\{E\}\_\{t\}\|\}\\sum\_\{\(s,k\)\\in\\mathcal\{E\}\_\{t\}\}\\left\(\\hat\{c\}\(s,k,t\)\-\\widetilde\{c\}\(s,k,t\)\\right\)^\{2\},\\qquad t\\in\\\{\\mathrm\{pre\},\\mathrm\{post\}\\\}\.\(15\)
Accordingly,ℒpre\\mathcal\{L\}\_\{\\mathrm\{pre\}\}andℒpost\\mathcal\{L\}\_\{\\mathrm\{post\}\}denote Equation \([15](https://arxiv.org/html/2610.41215#Sx2.E15)\) evaluated for the pretest and posttest phases, respectively\.

The gain\-observation set is defined asℰgain=\{\(s,k\):μgain​\(s,k\)=1\}\\mathcal\{E\}\_\{\\mathrm\{gain\}\}=\\\{\(s,k\):\\mu^\{\\mathrm\{gain\}\}\(s,k\)=1\\\}\. The gain lossℒgain\\mathcal\{L\}\_\{\\mathrm\{gain\}\}is the mean squared error over skills observed in both phases and is formulated as:

ℒgain=1\|ℰgain\|​∑\(s,k\)∈ℰgain\(g^sk−g~sk\)2\.\\mathcal\{L\}\_\{\\mathrm\{gain\}\}=\\frac\{1\}\{\|\\mathcal\{E\}\_\{\\mathrm\{gain\}\}\|\}\\sum\_\{\(s,k\)\\in\\mathcal\{E\}\_\{\\mathrm\{gain\}\}\}\\left\(\\hat\{g\}\_\{s\}^\{k\}\-\\widetilde\{g\}\_\{s\}^\{k\}\\right\)^\{2\}\.\(16\)
The Full Cognitive variant replacesc~​\(s,k,t\)\\widetilde\{c\}\(s,k,t\)andg~sk\\widetilde\{g\}\_\{s\}^\{k\}with their unweighted counterpartsc⁡\(s,k,t\)c\(s,k,t\)andgskg\_\{s\}^\{k\}\.

### Multi\-Task Learning

The coefficientsλreg\\lambda\_\{\\mathrm\{reg\}\},λord\\lambda\_\{\\mathrm\{ord\}\}, andλcls\\lambda\_\{\\mathrm\{cls\}\}weight the grading losses, whileλpre\\lambda\_\{\\mathrm\{pre\}\},λpost\\lambda\_\{\\mathrm\{post\}\}, andλgain\\lambda\_\{\\mathrm\{gain\}\}weight the cognitive losses\. The complete AICG objective is defined as:

ℒAICG=λreg​ℒreg\+λord​ℒord\+λcls​ℒcls\+λpre​ℒpre\+λpost​ℒpost\+λgain​ℒgain\.\\begin\{split\}\\mathcal\{L\}\_\{\\mathrm\{AICG\}\}=\{\}&\\lambda\_\{\\mathrm\{reg\}\}\\mathcal\{L\}\_\{\\mathrm\{reg\}\}\+\\lambda\_\{\\mathrm\{ord\}\}\\mathcal\{L\}\_\{\\mathrm\{ord\}\}\+\\lambda\_\{\\mathrm\{cls\}\}\\mathcal\{L\}\_\{\\mathrm\{cls\}\}\\\\ &\+\\lambda\_\{\\mathrm\{pre\}\}\\mathcal\{L\}\_\{\\mathrm\{pre\}\}\+\\lambda\_\{\\mathrm\{post\}\}\\mathcal\{L\}\_\{\\mathrm\{post\}\}\+\\lambda\_\{\\mathrm\{gain\}\}\\mathcal\{L\}\_\{\\mathrm\{gain\}\}\.\\end\{split\}\(17\)
We set\(λreg,λord,λcls\)=\(0\.50,0\.35,0\.15\)\(\\lambda\_\{\\mathrm\{reg\}\},\\lambda\_\{\\mathrm\{ord\}\},\\lambda\_\{\\mathrm\{cls\}\}\)=\(0\.50,0\.35,0\.15\)and\(λpre,λpost,λgain\)=\(0\.10,0\.10,0\.10\)\(\\lambda\_\{\\mathrm\{pre\}\},\\lambda\_\{\\mathrm\{post\}\},\\lambda\_\{\\mathrm\{gain\}\}\)=\(0\.10,0\.10,0\.10\)\. The grading objectives remain primary, while the cognitive objectives provide auxiliary supervision to the shared encoder\. All coefficients are fixed across semesters and random seeds\.

## Experiments

### Data and Evaluation Protocol

AICG is evaluated on 49,402 instructor\-graded mathematical responses from 3,861 student–semester records across 19 observed semesters \(F14–F24\); F17 has no observations\. Each assessment contains eight pretest and eight aligned posttest items\. An expert Q\-matrix maps these items to ten skills, and G\-DINA estimates pretest mastery, posttest mastery, and learning gain for auxiliary supervision\. We distinguish thepre\-Gen\-AI period\(F14–S22\) from thepost\-Gen\-AI period\(F22–F24\)\.

#### Gen\-AI Detection\.

The Gen\-AI Detector is developed using S18–S22 data\. We sample 14,044 responses from 26,039 authentic answers and construct one AI\-assisted counterpart per response, yielding 28,088 balanced examples\. Source\-response groups remain intact across training \(24,375\), validation \(2,709\), and held\-out testing \(1,004\)\. Cross\-semester generalization is evaluated on 11,458 labeled human and synthetically assisted responses from F14–S17\. The trained Gen\-AI Detector then assigns continuous AI\-assistance probabilities to F22–F24 responses\. At a threshold of \.5, it identifies 2,896 of 13,874 responses \(20\.87%\) as potentially assisted, with semester\-level rates ranging from 13\.40% to 23\.85%\.

#### Grading\.

Each post\-Gen\-AI semester is evaluated using an expanding window containing only earlier semesters\. Historical students are split 90/10 for training and validation, and the evaluation semester is excluded from optimization and model selection\.

### Baselines, Metrics, and Implementation

#### Baselines\.

We compare five ASAG baselines:BERT, a supervised Transformer grader\([Zhu et al\. 2022](https://arxiv.org/html/2610.41215#bib.bib20)\);RoBERTa, a score\-only model using AICG’s backbone\([Liu et al\. 2019](https://arxiv.org/html/2610.41215#bib.bib2)\);ASQ\-DBN, which combines answer, student, and question representations\([Zhang et al\. 2020](https://arxiv.org/html/2610.41215#bib.bib8)\);GPT\-4o Scoring, a fixed\-prompt grader without parameter updates\([OpenAI 2024](https://arxiv.org/html/2610.41215#bib.bib21);[Ferreira Mello et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib10)\); andGradeOpt, a rubric\-aware multi\-agent grader\([Chu et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib11)\)\. All baselines use the same expanding\-window splits\. GradeOpt optimizes its guidelines using historical development data only, and no baseline receives Q\-matrix targets, cognitive labels, or AI\-aware weights\.

#### Evaluation Metrics\.

For grading, we report mean absolute error \(MAE\), accuracy within absolute tolerances of \.05, \.10, and \.20, exact accuracy, Macro\-F1, and quadratic weighted kappa \(QWK\)\. Scores are bounded to\[0,1\]\[0,1\]and mapped to five ordered categories using cut points \.125, \.375, \.625, and \.875\. For Gen\-AI detection, we report accuracy, precision, recall, and F1\. Results are reported as mean and sample standard deviation over five random seeds for trainable models and five independent runs for prompt\-based graders\. Dataset splits, configurations, prompts, and implementation details are provided in Appendix\.

## Results

### Gen\-AI Detection

Pre\-Gen\-AI period \(F14\-S22\):Table[1](https://arxiv.org/html/2610.41215#Sx4.T1)shows that our Gen\-AI detector’s performance is robust on both the within\-semester and the downstream cross\-semester testing data where we have synthetic data with ground truth\.

Table 1:Gen\-AI detector performance\. F17 is excluded because no data were collected in that semester\.Post\-Gen\-AI period \(F22\-F24\):Applying the Gen\-AI Detector identifies 2,896 of 13,874 responses, 20\.87%, from F22–F24 as potentially AI\-assisted\. More specifically, it varies from 13\.40% to 23\.85% across different semesters\. These calibrated assistance probabilities serve as reliability signals for AI\-aware cognitive\-target construction and Q\-matrix\-guided skill\-evidence aggregation\.

### Overall Performance

Table[2](https://arxiv.org/html/2610.41215#Sx4.T2)shows that AICG achieves the best result on every evaluation metric against all five baselines\. Compared with GradeOpt, the strongest external baseline, AICG reduces MAE by 44\.9%, improves QWK from \.842 to \.927, and increases tolerance\-based and exact accuracy by 8\.7%–15\.7% percentage points\. Relative to RoBERTa, the strongest fully trained answer\-only baseline, AICG reduces MAE by 50\.3%, improves QWK from \.798 to \.927, and increases tolerance\-based and exact accuracy by 11\.6%–19\.6%\.

These results demonstrate the effectiveness of integrating individual\-answer scoring with cognitive supervision\. Although ordinal and categorical objectives improve the RoBERTa baseline, the largest gains arise from Q\-matrix\-guided mastery and learning\-gain supervision, which exposes skill\-level distinctions compressed by scalar scores and helps the shared encoder represent diverse reasoning patterns\. AI\-aware reliability weighting further improves the strongest unweighted cognitive model by attenuating potentially assisted cognitive evidence while preserving its contribution to answer grading\.

Table 2:Overall grading performance across F22–F24\. Values are mean \(standard deviation\) over five repetitions\. Best completed results are shown in bold\.
### Ablation Study

Table[2](https://arxiv.org/html/2610.41215#Sx4.T2)progressively introduces AICG’s components\. RoBERTa uses only answer\-level regression and serves as the base configuration\.

RoBERTa\+Oadds ordinal supervision, reducing MAE from \.0873 to \.0842 and improving QWK from \.798 to \.813 by explicitly modeling score\-category order\.Full Scoringfurther adds categorical supervision, reducing MAE to \.0823 and improving QWK to \.831 by learning category\-specific partial\-credit boundaries\.Full Scoring \+ Gainintroduces learning\-gain supervision without mastery states, reducing MAE to \.0715 and improving QWK to \.871 by capturing longitudinal changes unavailable from individual answer scores\.Full Scoring \+ Stateinstead adds pretest and posttest mastery supervision, reducing MAE to \.0527 and improving QWK to \.906\. Its larger improvement indicates that mastery states provide more direct and stable targets than gain, which is derived from their difference\.Full Scoring \+ Full Cognitivecombines State and Gain using unweighted targets and skill pooling, reducing MAE to \.0453 and improving QWK to \.924\. Mastery identifies the student’s cognitive state at each phase, while gain constrains its change across assessments\. Finally,AICGapplies AI\-aware reliability weights to both G\-DINA\-derived targets and Q\-matrix\-selected response embeddings, achieving \.0434 MAE and \.927 QWK by attenuating cognitive evidence that may not reflect independent mastery\.

#### Cross\-Semester Robustness\.

Table[3](https://arxiv.org/html/2610.41215#Sx4.T3)reports the component ablations at the beginning and end of the expanding\-window evaluation\. Their ordering remains stable across F22 and F24: scoring\-only models perform worst, State outperforms Gain, and Full Cognitive provides the strongest unweighted model\. This consistency shows that cognitive supervision remains effective as the historical training window and semester\-specific response distributions change\.

AICG achieves the lowest MAE and highest exact accuracy in F22, although Full Cognitive retains small advantages on several agreement metrics\. By F24, AICG performs best on every reported evaluation metric\. This progression reflects the role of AI\-aware reliability weighting: its effect is limited when AI\-assisted responses are uncommon but becomes more pronounced as they introduce more unreliable cognitive evidence\.

Table 3:Ablation results at the beginning and end of the expanding\-window evaluation\. Values are means over five seeds\.
#### Effect of AI\-Aware Weighting\.

Table 4:Semester\-level effect of AI\-aware weighting\. FullCog denotes Full Scoring \+ Full Cognitive\.Table[4](https://arxiv.org/html/2610.41215#Sx4.T4)isolates AI\-aware weighting by comparing AICG with FullCog, which treats all responses as equally reliable\. AICG reduces MAE in every semester, from 0\.8% in F22 to 3\.6–6\.6% in later semesters, and improves QWK in four of five semesters\. The smaller F22 gain and more consistent later gains suggest that reliability weighting becomes more beneficial as AI\-assisted responses increase\.

AI\-assisted responses may earn valid grades while providing unreliable evidence of independent mastery\. AICG addresses this mismatch by down\-weighting likely AI\-assisted responses during skill\-specific pooling without modifying their grading targets, thereby reducing noise in cognitive supervision rather than penalizing AI use\. Although the current aggregate gain is moderate because these responses remain a minority, this protection becomes increasingly important as generative AI adoption grows\.

## Related Work

Automated Short Answer Grading\.ASAG has evolved from rule\-based and feature\-engineering approaches\([Burrows et al\. 2015](https://arxiv.org/html/2610.41215#bib.bib1)\)to pretrained Transformers that encode response semantics with minimal task\-specific design\([Latif and Zhai 2024](https://arxiv.org/html/2610.41215#bib.bib5)\)\. We include BERT\([Zhu et al\. 2022](https://arxiv.org/html/2610.41215#bib.bib20)\)as a classical supervised ASAG encoder and RoBERTa\([Liu et al\. 2019](https://arxiv.org/html/2610.41215#bib.bib2)\)as a score\-only, same\-backbone baseline\. Recent work evaluates LLMs as zero\-shot or few\-shot graders\([Ferreira Mello et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib10);[Chu et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib11);[Chamieh et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib6);[Jiang and Bosch 2024](https://arxiv.org/html/2610.41215#bib.bib12);[Cong et al\. 2026](https://arxiv.org/html/2610.41215#bib.bib13)\)\. We include GPT\-4o Scoring\([OpenAI 2024](https://arxiv.org/html/2610.41215#bib.bib21);[Ferreira Mello et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib10)\)as a prompted grader and GradeOpt\([Chu et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib11)\)as a rubric\-optimization baseline\. However, these methods remain unreliable in mathematics, where scores depend on symbolic correctness, intermediate derivations, and specific reasoning principles\([Chamieh et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib6)\)\. Unlike rubric\-matching methods, AICG grounds grading in a Q\-matrix and derives cognitive targets from psychometric models\.

Q\-Matrix\-Based Skill Modeling\.Cognitive diagnosis models infer latent skill mastery from response patterns\. G\-DINA\([de la Torre 2011](https://arxiv.org/html/2610.41215#bib.bib3)\)associates items with multiple skills through a Q\-matrix and models their mastery interactions\. Neural extensions, including NeuralCD\([Wang et al\. 2020](https://arxiv.org/html/2610.41215#bib.bib4)\), ORCDF\([Qian et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib15)\), and inductive variants\([Liu et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib14)\), improve representation capacity but operate on item\-response records rather than free\-text answers\.[Zhang et al\.](https://arxiv.org/html/2610.41215#bib.bib8)[2020](https://arxiv.org/html/2610.41215#bib.bib8)combine student and question embeddings with answer text, showing that these representations complement response encoding\. We reproduce their answer–student–question model as the ASQ\-DBN baseline\. AICG instead uses G\-DINA\-derived mastery and learning\-gain targets to supervise a free\-text response encoder\.

AI\-Generated Text Detection\.Machine\-generated\-text detection includes zero\-shot methods such as Binoculars and Fast\-DetectGPT\([Hans et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib17);[Bao et al\. 2024](https://arxiv.org/html/2610.41215#bib.bib18)\), but performance can degrade under paraphrasing and distribution shift\([Sadasivan et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib9)\)\. Detectors may also disproportionately flag non\-native English writing\([Liang et al\. 2023](https://arxiv.org/html/2610.41215#bib.bib7)\)\. Existing methods typically produce binary labels for misconduct detection\. AICG instead uses a supervised contrastive Gen\-AI Detector to produce continuous probabilities and convert them into soft reliability weights\. This approach preserves each response’s grading contribution while attenuating potentially assisted cognitive evidence\.

## Conclusion

We presented AICG, a cognitive reliability\-aware multi\-task framework for automated mathematical short\-answer grading\. By jointly optimizing answer\-level scoring with Q\-matrix\-guided G\-DINA mastery and learning\-gain supervision, AICG learns representations that reflect student cognition rather than scalar labels\. An auxiliary Gen\-AI Detector, trained through supervised contrastive pretraining, converts assistance probabilities into soft weights on cognitive evidence, limiting inflated mastery estimates while preserving grading\. Across 49,402 instructor\-graded responses over ten academic years, AICG reduces MAE by 50\.3% and improves QWK from \.798 to \.927 relative to RoBERTa regression\. Ablations identify cognitive supervision as the primary source of improvement, with AI\-aware weighting providing additional post\-Gen\-AI gains\. Future work will learn Q\-matrices from response text, improve detector robustness across LLMs, evaluate cross\-course and multi\-institution generalization, and use mastery and learning\-gain profiles for personalized feedback\.

## References

- Baoet al\.\(2024\)G\. Bao, Y\. Zhao, Z\. Teng, L\. Yang, and Y\. ZhangFast\-DetectGPT: efficient zero\-shot detection of machine\-generated text via conditional probability curvature\.InProceedings of the Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bpcgcr8E8Z)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p3.1)\.
- Burrowset al\.\(2015\)S\. Burrows, I\. Gurevych, and B\. SteinThe eras and trends of automatic short answer grading\.International Journal of Artificial Intelligence in Education25\(1\),pp\. 60–117\.External Links:[Document](https://dx.doi.org/10.1007/s40593-014-0026-8)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p2.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Chamiehet al\.\(2024\)I\. Chamieh, T\. Zesch, and K\. GiebermannLLMs in short answer scoring: limitations and promise of zero\-shot and few\-shot approaches\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),pp\. 309–315\.External Links:[Link](https://aclanthology.org/2024.bea-1.25/)Cited by:[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Chuet al\.\(2025\)Y\. Chu, H\. Li, K\. Yang, H\. Shomer, Y\. Copur\-Gencturk, L\. Kaldaras, K\. Haudek, J\. Krajcik, N\. Shin, H\. Liu, and J\. TangA LLM\-powered automatic grading framework with human\-level guidelines optimization\.InProceedings of the 18th International Conference on Educational Data Mining,pp\. 31–41\.External Links:[Link](https://educationaldatamining.org/EDM2025/proceedings/)Cited by:[Appendix A](https://arxiv.org/html/2610.41215#A1.SSx7.SSS0.Px6.p1.1),[Introduction](https://arxiv.org/html/2610.41215#Sx1.p2.1),[Baselines\.](https://arxiv.org/html/2610.41215#Sx3.SSx2.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2610.41215#Sx4.T2.1.6.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Conget al\.\(2026\)L\. Cong, L\. Hammerla, S\. Hahn, S\. Gombert, H\. Drachsler, and U\. KroehneAutomatic short answer grading with LLMs: from memorization to reasoning\.InProceedings of the 16th International Conference on Learning Analytics and Knowledge,pp\. 75–84\.External Links:[Document](https://dx.doi.org/10.1145/3785022.3785031)Cited by:[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- de la Torre \(2011\)J\. de la TorreThe generalized dina model framework\.Psychometrika76\(2\),pp\. 179–199\.External Links:[Document](https://dx.doi.org/10.1007/S11336-011-9207-7)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Reliability\-Weighted Cognitive Targets\.](https://arxiv.org/html/2610.41215#Sx2.SSx4.SSS0.Px3.p1.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p2.1)\.
- Ferreira Melloet al\.\(2025\)R\. Ferreira Mello, C\. Pereira Junior, L\. Rodrigues, F\. D\. Pereira, L\. Cabral, N\. Costa, G\. Ramalho, and D\. GaševićAutomatic short answer grading in the LLM era: does GPT\-4 with prompt engineering beat traditional models?\.InProceedings of the 15th International Conference on Learning Analytics and Knowledge,pp\. 93–103\.External Links:[Document](https://dx.doi.org/10.1145/3706468.3706481)Cited by:[Appendix A](https://arxiv.org/html/2610.41215#A1.SSx7.SSS0.Px5.p1.1),[Introduction](https://arxiv.org/html/2610.41215#Sx1.p2.1),[Baselines\.](https://arxiv.org/html/2610.41215#Sx3.SSx2.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2610.41215#Sx4.T2.1.3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Guet al\.\(2025\)H\. Gu, G\. Luo, X\. Dong, S\. Li, and D\. ZhouRevisiting cognition in neural cognitive diagnosis\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 402–412\.External Links:[Document](https://dx.doi.org/10.1145/3690624.3709319)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1)\.
- Hanset al\.\(2024\)A\. Hans, A\. Schwarzschild, V\. Cherepanova, H\. Kazemi, A\. Saha, M\. Goldblum, J\. Geiping, and T\. GoldsteinSpotting LLMs with binoculars: zero\-shot detection of machine\-generated text\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 17519–17537\.External Links:[Link](https://proceedings.mlr.press/v235/hans24a.html)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p3.1)\.
- Jiang and Bosch \(2024\)L\. Jiang and N\. BoschShort answer scoring with GPT\-4\.InProceedings of the 11th ACM Conference on Learning at Scale,pp\. 438–442\.External Links:[Document](https://dx.doi.org/10.1145/3657604.3664685)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p2.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Khoslaet al\.\(2020\)P\. Khosla, P\. Teterwak, C\. Wang, A\. Sarna, Y\. Tian, P\. Isola, A\. Maschinot, C\. Liu, and D\. KrishnanSupervised contrastive learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 18661–18673\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/%0Ad89a66c7c80a29b1bdbab0f2a1a94af8-Abstract.html)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p4.1)\.
- Latif and Zhai \(2024\)E\. Latif and X\. ZhaiFine\-tuning ChatGPT for automatic scoring\.Computers and Education: Artificial Intelligence6,pp\. 100210\.External Links:[Document](https://dx.doi.org/10.1016/j.caeai.2024.100210)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p2.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Lianget al\.\(2023\)W\. Liang, M\. Yuksekgonul, Y\. Mao, E\. Wu, and J\. ZouGPT detectors are biased against non\-native english writers\.Patterns4\(7\),pp\. 100779\.External Links:[Document](https://dx.doi.org/10.1016/j.patter.2023.100779)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p3.1)\.
- Liuet al\.\(2024\)S\. Liu, J\. Shen, H\. Qian, and A\. ZhouInductive cognitive diagnosis for fast student learning in web\-based intelligent education systems\.InProceedings of the ACM Web Conference 2024,pp\. 4260–4271\.External Links:[Document](https://dx.doi.org/10.1145/3589334.3645589)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p2.1)\.
- Liuet al\.\(2019\)Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. StoyanovRoBERTa: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1907.11692)Cited by:[Appendix A](https://arxiv.org/html/2610.41215#A1.SSx7.SSS0.Px2.p1.1),[Introduction](https://arxiv.org/html/2610.41215#Sx1.p4.1),[Baselines\.](https://arxiv.org/html/2610.41215#Sx3.SSx2.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2610.41215#Sx4.T2.1.2.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- OpenAI \(2024\)OpenAIHello GPT\-4o\.Note:OpenAIExternal Links:[Link](https://openai.com/index/hello-gpt-4o/)Cited by:[Appendix A](https://arxiv.org/html/2610.41215#A1.SSx7.SSS0.Px5.p1.1),[Baselines\.](https://arxiv.org/html/2610.41215#Sx3.SSx2.SSS0.Px1.p1.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.
- Qianet al\.\(2024\)H\. Qian, S\. Liu, M\. Li, B\. Li, Z\. Liu, and A\. ZhouORCDF: an oversmoothing\-resistant cognitive diagnosis framework for student learning in online education systems\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 2455–2466\.External Links:[Document](https://dx.doi.org/10.1145/3637528.3671988)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p2.1)\.
- Sadasivanet al\.\(2025\)V\. S\. Sadasivan, A\. Kumar, S\. Balasubramanian, W\. Wang, and S\. FeiziCan AI\-generated text be reliably detected? stress testing AI text detectors under various attacks\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=OOgsAZdFOt)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p3.1)\.
- Wanget al\.\(2020\)F\. Wang, Q\. Liu, E\. Chen, Z\. Huang, Y\. Chen, and Y\. YinNeural cognitive diagnosis for intelligent education systems\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 6153–6161\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i04.6103)Cited by:[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p2.1)\.
- Zhanget al\.\(2020\)Y\. Zhang, C\. Lin, and M\. ChiGoing deeper: automatic short\-answer grading by combining student and question models\.User Modeling and User\-Adapted Interaction30\(1\),pp\. 51–80\.External Links:[Document](https://dx.doi.org/10.1007/s11257-019-09251-6)Cited by:[Appendix A](https://arxiv.org/html/2610.41215#A1.SSx7.SSS0.Px4.p1.1),[Introduction](https://arxiv.org/html/2610.41215#Sx1.p3.1),[Baselines\.](https://arxiv.org/html/2610.41215#Sx3.SSx2.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2610.41215#Sx4.T2.1.5.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p2.1)\.
- Zhuet al\.\(2022\)X\. Zhu, H\. Wu, and L\. ZhangAutomatic short\-answer grading via BERT\-based deep neural networks\.IEEE Transactions on Learning Technologies15\(3\),pp\. 364–375\.External Links:[Document](https://dx.doi.org/10.1109/TLT.2022.3175537),[Link](https://doi.org/10.1109/TLT.2022.3175537)Cited by:[Appendix A](https://arxiv.org/html/2610.41215#A1.SSx7.SSS0.Px3.p1.1),[Baselines\.](https://arxiv.org/html/2610.41215#Sx3.SSx2.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2610.41215#Sx4.T2.1.4.1),[Related Work](https://arxiv.org/html/2610.41215#Sx5.p1.1)\.

## Appendix AAdditional Data & Methodology & Experimental Details

### Dataset Composition – Data

Table[5](https://arxiv.org/html/2610.41215#A1.T5)summarizes the composition of 49,402 responses from 3,861 student–semester records\. The pre\-Gen\-AI period \(F14–S22\) is aggregated, whereas the post\-Gen\-AI semesters \(F22–F24\) are reported separately with their pretest and posttest response counts and estimated AI\-assistance rates\.

Table 5:Composition of the longitudinal ASAG dataset\.SemesterStudentsPre AnswersPost AnswersTotal AnswersAI AnswersAI PercentagePre\-Gen\-AI2,53019,07616,45235,52800\.00%F222772,0201,0843,10441613\.40%S231701,3477352,08248223\.15%F233762,9461,5314,4771,05523\.56%S242188908981,78836520\.41%F242901,2061,2172,42357823\.85%Post\-Gen\-AI1,3318,4095,46513,8742,89620\.87%Overall3,86127,48521,91749,4022,896–
### Expanding\-Window Evaluation Splits – Data

Table[6](https://arxiv.org/html/2610.41215#A1.T6)summarizes the expanding\-window splits\. For each test semester, AICG is trained and validated using only earlier semesters, while the target semester is reserved exclusively for evaluation\.

Table 6:Expanding\-window evaluation splits\.
### Q\-Matrix and G\-DINA Target Construction – Data

#### Q\-matrix\.

The assessment contains eight pretest items and eight aligned isomorphic or closest\-match posttest items\. The expert\-designed Q\-matrix maps these items to ten probability principles: Addition Theorem for Three Events \(A3\), Total Probability Theorem \(TPT\), Definition of Conditional Probability \(CP\), Addition Theorem for Two Events \(A2\), De Morgan’s Law \(DMT\), Complement Theorem \(CT\), Bayes’ Rule \(BR\), Definition of Independent Events \(IE\), Definition of Mutually Exclusive Events \(MET\), and Definition of Conditionally Independent Events \(CIE\)\. Table[7](https://arxiv.org/html/2610.41215#A1.T7)reports the complete item–principle mapping\.

Table 7:Expert Q\-matrix for the pretest and aligned posttest items\.

### Gen\-AI Detector Data Construction – Methodology

#### AI\-Assisted Response Construction\.

Complete LLM rewriting often creates conspicuous differences in length, fluency, and style, encouraging shortcut learning\. We instead construct partially assisted responses that preserve most of the original student text\.

For an authentic human responserr, letnrn\_\{r\}denote its number of tokens and𝝌r\\boldsymbol\{\\chi\}\_\{r\}its token sequence, whereχrℓ\\chi^\{\\ell\}\_\{r\}denotes the original token at positionℓ∈\{1,…,nr\}\\ell\\in\\\{1,\\ldots,n\_\{r\}\\\}\. Let𝐦r\\mathbf\{m\}\_\{r\}denote a binary masking vector, wheremrℓ=1m^\{\\ell\}\_\{r\}=1indicates a masked position andmrℓ=0m^\{\\ell\}\_\{r\}=0an original token\. The masking ratioρr\\rho\_\{r\}is defined as:

ρr=1nr​∑ℓ=1nrmrℓ\.\\rho\_\{r\}=\\frac\{1\}\{n\_\{r\}\}\\sum\_\{\\ell=1\}^\{n\_\{r\}\}m^\{\\ell\}\_\{r\}\.\(18\)
The mask is sampled such thatρr∈\[0\.30,0\.50\]\\rho\_\{r\}\\in\[0\.30,0\.50\]\. Let𝝌~r\\widetilde\{\\boldsymbol\{\\chi\}\}\_\{r\}denote the resulting masked sequence\. An offline DeepSeek model𝒢\\mathcal\{G\}reconstructs only its masked positions, whereυrℓ\\upsilon^\{\\ell\}\_\{r\}denotes the token generated for positionℓ\\ell\.

Let𝐙r\\mathbf\{Z\}\_\{r\}denote the resulting partially AI\-assisted response, wherezrℓz^\{\\ell\}\_\{r\}denotes its token at positionℓ\\ell\. A generated token is used at a masked position, whereas the original token is retained otherwise\. The resulting token is defined as:

zrℓ=\{υrℓ,if​mrℓ=1,χrℓ,if​mrℓ=0\.z^\{\\ell\}\_\{r\}=\\begin\{cases\}\\upsilon^\{\\ell\}\_\{r\},&\\text\{if \}m^\{\\ell\}\_\{r\}=1,\\\\ \\chi^\{\\ell\}\_\{r\},&\\text\{if \}m^\{\\ell\}\_\{r\}=0\.\\end\{cases\}\(19\)
This procedure preserves 50–70% of the original response, retaining student\-specific linguistic and mathematical characteristics while reducing obvious stylistic distinctions\. Original responses are labeled human\-written, whereas partially completed and fully generated correct responses form the AI\-assisted class\.

#### Detector Controls\.

Source human responses and their synthetic counterparts share a group identifier and cannot cross data partitions\. Detector inputs are matched by item, score stratum, and length bin to reduce correctness\- and length\-based shortcuts\. The calibration temperature is fitted on the validation set and then fixed\. Generator transfer is evaluated by withholding one generator family from detector development\.

### AI\-Aware Mastery\-Target Weighting – Methodology\.

The standard G\-DINA model produces an unweighted mastery targetc⁡\(s,k,t\)c\(s,k,t\)for studentss, principlekk, and phasett\. For AICG, each responserris assigned the reliability weight

wsr=max⁡\(δ,1−pAI​\(s,r\)\),δ=0\.30,w\_\{s\}^\{r\}=\\max\\\!\\left\(\\delta,1\-p^\{\\mathrm\{AI\}\}\(s,r\)\\right\),\\qquad\\delta=0\.30,wherepAI​\(s,r\)p^\{\\mathrm\{AI\}\}\(s,r\)is the calibrated AI\-assistance probability\. Letℓs,r​\(𝜻,Θ\)\\ell\_\{s,r\}\(\\boldsymbol\{\\zeta\};\\Theta\)denote the G\-DINA log\-likelihood contribution of responserrunder mastery profile𝜻\\boldsymbol\{\\zeta\}and item parametersΘ\\Theta\. The reliability\-weighted log\-likelihood is

Jw​\(s,t,𝜻\)=∑r∈Rstwsr​ℓs,r​\(𝜻,Θ\)\.J\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}\)=\\sum\_\{r\\in R\_\{s\}^\{t\}\}w\_\{s\}^\{r\}\\,\\ell\_\{s,r\}\(\\boldsymbol\{\\zeta\};\\Theta\)\.The corresponding weighted posterior is

Pw​\(𝜻∣Rst\)=P0​\(𝜻\)​exp⁡\[Jw​\(s,t,𝜻\)\]∑𝜻′P0​\(𝜻′\)​exp⁡\[Jw​\(s,t,𝜻′\)\],P\_\{w\}\(\\boldsymbol\{\\zeta\}\\mid R\_\{s\}^\{t\}\)=\\frac\{P\_\{0\}\(\\boldsymbol\{\\zeta\}\)\\exp\\\!\\left\[J\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}\)\\right\]\}\{\\sum\_\{\\boldsymbol\{\\zeta\}^\{\\prime\}\}P\_\{0\}\(\\boldsymbol\{\\zeta\}^\{\\prime\}\)\\exp\\\!\\left\[J\_\{w\}\(s,t;\\boldsymbol\{\\zeta\}^\{\\prime\}\)\\right\]\},and the AI\-aware mastery target is its posterior marginal:

c~​\(s,k,t\)=∑𝜻ζk​Pw​\(𝜻∣Rst\)\.\\widetilde\{c\}\(s,k,t\)=\\sum\_\{\\boldsymbol\{\\zeta\}\}\\zeta\_\{k\}P\_\{w\}\(\\boldsymbol\{\\zeta\}\\mid R\_\{s\}^\{t\}\)\.Finally, the reliability\-adjusted learning\-gain target is

g~s,k=c~​\(s,k,post\)−c~​\(s,k,pre\)\.\\widetilde\{g\}\_\{s,k\}=\\widetilde\{c\}\(s,k,\\mathrm\{post\}\)\-\\widetilde\{c\}\(s,k,\\mathrm\{pre\}\)\.Full Cognitive uses the corresponding unweighted targets obtained by setting all response weights to one\. AI assistance therefore modifies only the reliability of cognitive evidence; it does not alter the instructor\-assigned response\-level grading labels\.

### AICG Hyperparameter Settings – Experiments

Table[8](https://arxiv.org/html/2610.41215#A1.T8)reports implementation settings not specified in the main paper\. The Gen\-AI Detector and multi\-task grading components both useroberta\-base\. Hyperparameters are fixed across all evaluation semesters and random seeds\.

Table 8:Implementation settings for each component of AICG\.ComponentSettingFinal ValueGen\-AI Detector: contrastive pretrainingBackboneroberta\-baseMaximum sequence length512 tokensProjection dimension768Balanced mini\-batch size64 \(32 per class\)OptimizerAdamWLearning rate2×10−52\\times 10^\{\-5\}Training epochs10Gradient\-norm threshold1\.0Gen\-AI Detector: linear classificationFrozen componentsEncoder and projection layerClassifier768\-to\-2 linear layerMaximum sequence length256 tokensMini\-batch size64OptimizerAdamLearning rate2×10−42\\times 10^\{\-4\}Training epochs5Multi\-task GradingBackboneroberta\-baseMaximum sequence length512 tokensMini\-batch sizeOne student recordOptimizerAdamWLearning rate2×10−52\\times 10^\{\-5\}Weight decay0\.010\.01Training epochs8Dropout probability0\.100\.10Learning\-rate scheduleLinear decayWarmup proportion0\.100\.10Gradient\-norm threshold1\.0AICG evaluationRandom seeds42, 52, 62, 72, and 82Mixed\-precision training is used during supervised contrastive pretraining\. The linear classifier is trained with standard cross\-entropy while the pretrained encoder and projection layer remain frozen\. Multi\-task Grading is trained end to end with a linear learning\-rate schedule following a warmup period covering the first 10% of optimization steps\.

#### G\-DINA Configuration and Estimation\.

Pretest and posttest mastery profiles are estimated separately using a fractional\-response extension of the G\-DINA model\. Each normalized instructor\-assigned scoreys​r∈\[0,1\]y\_\{sr\}\\in\[0,1\]is used directly as fractional item\-response evidence rather than being converted into binary outcomes or ordered response categories\. For a mastery profileζ\\zeta, the contribution of responserris

ys​r​log⁡ϕ⁡\(s,r,ζ,Θ\)\+\(1−ys​r\)​log⁡\[1−ϕ⁡\(s,r,ζ,Θ\)\],y\_\{sr\}\\log\\phi\(s,r,\\zeta;\\Theta\)\+\(1\-y\_\{sr\}\)\\log\\\!\\left\[1\-\\phi\(s,r,\\zeta;\\Theta\)\\right\],whereϕ⁡\(s,r,ζ,Θ\)\\phi\(s,r,\\zeta;\\Theta\)is the G\-DINA success probability\.

Pretest and posttest parameters are estimated separately through marginal maximum likelihood using the expectation–maximization algorithm under an independent attribute distribution\. We use three random initializations, constrain success probabilities to\[10−4,0\.9999\]\[10^\{\-4\},0\.9999\], and retain the solution with the highest observed\-data likelihood\. Convergence requires the maximum absolute changes in both item success probabilities and attribute mixing proportions to fall below10−410^\{\-4\}between consecutive iterations, with a maximum of 1,000 EM iterations\.

For AICG, each fractional\-response contribution is further scaled by the reliability weightws​r=max⁡\(0\.30,1−ps​rAI\)w\_\{sr\}=\\max\(0\.30,1\-p^\{\\mathrm\{AI\}\}\_\{sr\}\), as defined in Equation \(11\)\. The resulting posterior mastery probabilities are computed using Equation \(12\)\. Full Cognitive uses the same estimation procedure withws​r=1w\_\{sr\}=1for every response\. All G\-DINA parameters and mastery targets are estimated separately within each expanding window using historical data only; the target evaluation semester is excluded from both parameter estimation and target construction\.

### Baseline Configurations – Experiments

#### Common Evaluation Protocol\.

All baselines use the same expanding\-window splits, preprocessing pipeline, grade boundaries, and evaluation metrics as AICG\. Transformer baselines receive the same textual input as AICG, consisting of the question identifier, student response, and principle metadata\. ASQ\-DBN follows its original answer, student, and question feature construction, while prompt\-based baselines receive the same available information in textual form\. No baseline uses Q\-matrix targets, cognitive labels, AI\-assistance probabilities, or reliability weights\. Trainable baselines are evaluated with random seeds4242,5252,6262,7272, and8282, whereas prompt\-based baselines use five independent runs\. Model selection uses only the historical validation partition associated with each evaluation semester\.

#### RoBERTa\.

RoBERTa is the score\-only counterpart of AICG and usesroberta\-basewith a sigmoid regression head\([Liu et al\. 2019](https://arxiv.org/html/2610.41215#bib.bib2)\)\. To isolate the contribution of cognitive and multi\-objective supervision, it uses the same tokenization, maximum sequence length of 512, student\-record batching, and regression objective as AICG\. Specifically, the model is trained with category\-weighted Smooth L1 loss usingβ=0\.10\\beta=0\.10and category weights\(1\.0,2\.0,2\.8,2\.0,0\.7\)\(1\.0,2\.0,2\.8,2\.0,0\.7\)\. Optimization uses AdamW with a learning rate of2×10−52\\times 10^\{\-5\}, weight decay of0\.010\.01, dropout of0\.100\.10, a linear warmup over the first 10% of optimization steps, with gradient clipping at 1\.0\. Training lasts for eight epochs\. Continuous predictions are bounded to\[0,1\]\[0,1\], and discrete grades are obtained using the common boundaries0\.1250\.125,0\.3750\.375,0\.6250\.625, and0\.8750\.875\.

#### BERT\.

BERT replaces the RoBERTa encoder withbert\-base\-uncasedwhile retaining the same input, regression head, loss, optimization settings, training epochs, and validation procedure\([Zhu et al\. 2022](https://arxiv.org/html/2610.41215#bib.bib20)\)\. This controlled adaptation isolates the influence of the pretrained encoder without introducing additional architectural or supervision differences\. Its continuous and discrete predictions are evaluated using the same procedures as those of RoBERTa\.

#### ASQ\-DBN\.

ASQ\-DBN follows the answer–student–question modeling principle of[Zhang et al\.](https://arxiv.org/html/2610.41215#bib.bib8)[2020](https://arxiv.org/html/2610.41215#bib.bib8)\. The answer representation is constructed using TF–IDF word unigrams and bigrams with a maximum vocabulary of 10,000 terms, fitted only on the historical training data\. Student features comprise the mean, standard deviation, and number of scores observed strictly before the evaluation semester; unseen students receive the historical population statistics and an additional missing\-history indicator\. Question features include a one\-hot question identifier and the historical mean, standard deviation, and number of responses for that question\. No score from the evaluation semester is used to construct student or question features\.

The concatenated answer, student, and question features are processed by a deep belief network with hidden dimensions 512 and 128\. Each layer is initialized through 20 epochs of contrastive\-divergence pretraining, after which the complete network is fine\-tuned using Adam with a learning rate of10−310^\{\-3\}, batch size 64, dropout of0\.200\.20, and weight decay of10−510^\{\-5\}\. Fine\-tuning runs for at most 50 epochs with early stopping after five epochs without validation improvement\. A five\-class softmax layer produces grade probabilities\. The discrete prediction is the highest\-probability category, while its continuous counterpart is calculated as the expected normalized category score,∑d=04\(d/4\)​πd\\sum\_\{d=0\}^\{4\}\(d/4\)\\pi\_\{d\}\.

#### GPT\-4o Scoring\.

GPT\-4o Scoring uses the fixed model snapshotgpt\-4o\-2024\-08\-06without parameter updates\([OpenAI 2024](https://arxiv.org/html/2610.41215#bib.bib21);[Ferreira Mello et al\. 2025](https://arxiv.org/html/2610.41215#bib.bib10)\)\. Each request contains the question identifier, principle descriptions and weights, student response, and definitions of the five ordered grade categories\. The model is instructed to return only one category in0,1,2,3,4\{0,1,2,3,4\}\. We use temperature00, top\-p=1p=1, and a maximum output length of 16 tokens\. The first valid standalone category is parsed from the output\. An invalid output triggers one deterministic repair request using the same model and prompt context\.

The predicted category is used directly for exact accuracy, Macro\-F1, and QWK\. For continuous evaluation, categoryddis mapped tod/4d/4\. Five independent API runs are conducted using the same model snapshot and fixed prompt\. Neither in\-context examples nor responses from the evaluation semester are used\.

#### GradeOpt\.

GradeOpt follows the Grader–Reflector–Refiner framework of[Chu et al\.](https://arxiv.org/html/2610.41215#bib.bib11)[2025](https://arxiv.org/html/2610.41215#bib.bib11)\. A separate grading guideline is optimized for each question using only the historical training and validation data\. The initial guideline contains the available principle metadata, principle weights, and five\-category grade definitions\. GPT\-4o is used for the Grader, Reflector, and Refiner\. The Grader uses temperature00, while the Reflector and Refiner use temperature0\.50\.5\.

Following the original configuration, guideline optimization uses an outer batch size of 64, an inner batch size of 8, five outer iterations, three inner iterations, and a UCB beam size of 4\. Each optimization batch contains equal proportions of high\-error and randomly sampled historical examples\. Cohen’s kappa on the historical validation set serves as the guideline\-selection criterion\. Inner or outer optimization terminates when no improvement is observed for two consecutive iterations\.

To preserve temporal isolation, the optimized guideline is frozen before evaluation and GradeOpt’s test\-time adaptation is disabled\. Thus, no response or label from the evaluation semester is provided to the Grader, Reflector, or Refiner during guideline optimization\. The final grader outputs one category in0,1,2,3,4\{0,1,2,3,4\}, which is mapped tod/4d/4for continuous evaluation\. Five independent optimization and evaluation runs are reported\.