Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers

# Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers
Source: [https://arxiv.org/html/2607.16085](https://arxiv.org/html/2607.16085)
###### Abstract

Increasingly, speech and language processing tasks take either audio or text directly rather than extracting features from these as the input to the classifier or regressor\. Often these systems make use of complex, for example transformer\-based, processes that have the ability to derive highly non\-linear mappings between the input and the output\. Unfortunately these systems can also learn “shortcuts” where the classifier is overly reliant on particular aspects of the input to yield the output\. For the task of language proficiency assessment, this over\-reliance can enable learners to increase their score by exploiting the shortcut rather than improving their ability\. This paper introduces a novel training criterion that is able to reduce the classifier’s reliance on shortcuts, thus for example limiting this option for malpractice in language assessment\. This process is illustrated on two forms of assessment system, one based on the audio the other on the speech recognition text\. The results show that, for both systems, there is higher correlations with features that could be exploited for malpractice than expected from the human reference, indicating an over\-reliance on these features\. By introducing the modified training criterion, this correlation can be reduced to be closer to the reference correlation\.

## IIntroduction

There is a growing adoption of end\-to\-end neural systems in speech and language processing, where raw audio or text is taken directly as input, rather than first extracting human understandable features\. Systems based on models such as BERT\[[13](https://arxiv.org/html/2607.16085#bib.bib26)\], ModernBERT\[[37](https://arxiv.org/html/2607.16085#bib.bib27)\]and wav2vec 2\.0\.0\[[2](https://arxiv.org/html/2607.16085#bib.bib25)\]make use of complex, transformer\-based processes capable of deriving highly non\-linear mappings between input and output\. Whilst they have brought substantial performance gains across a wide range of tasks, they are also more vulnerable to the risk ofshortcut learning\[[19](https://arxiv.org/html/2607.16085#bib.bib40)\]\. Here the model makes uses of aspects \(features\) of the data not only based on how reliably that feature relates to training\-set labels, but also on how easily the feature can be extracted from inputs, leading to over\-reliance on surface aspects to produce the output\.

For the task of language proficiency assessment, this over\-reliance has direct practical consequences: in a study designed to “trick” an automated essay scorer, the most effective submission simply repeated the same paragraphs 37 times, and got the highest possible score from the system while receiving the lowest possible score from the human reader\[[28](https://arxiv.org/html/2607.16085#bib.bib48)\]\. For spoken language, the analogue is straightforward: a learner who simply keeps talking, even repetitively, could exploit the same length\-correlated shortcut\. The concern that an auto\-marker can be exploited by test\-takers to inflate scores without improving their underlying ability is well\-established in Automated Essay Scoring \(AES\)\. It has long been shown that automated graders can over\-rely on essay length\[[11](https://arxiv.org/html/2607.16085#bib.bib35),[1](https://arxiv.org/html/2607.16085#bib.bib36),[16](https://arxiv.org/html/2607.16085#bib.bib37)\], and that a model exploiting length alone can achieve state\-of\-the\-art performance on benchmark datasets\[[21](https://arxiv.org/html/2607.16085#bib.bib3)\]\. This has given rise to studies aiming at improving robustness around grader validity, including rationale alignment via counterfactuals\[[36](https://arxiv.org/html/2607.16085#bib.bib7)\], detection of adversarially designed inputs\[[14](https://arxiv.org/html/2607.16085#bib.bib2)\], and human\-in\-the\-loop supplementation of automated predictions\[[9](https://arxiv.org/html/2607.16085#bib.bib23)\]\. In Spoken Language Assessment \(SLA\), temporal fluency features such as speech rate, pause distribution and word count have long been established as predictors of oral proficiency\[[35](https://arxiv.org/html/2607.16085#bib.bib44),[3](https://arxiv.org/html/2607.16085#bib.bib43),[18](https://arxiv.org/html/2607.16085#bib.bib45)\], and critically examined for their construct validity\[[18](https://arxiv.org/html/2607.16085#bib.bib45)\]but are at risk of being capitalised on\.

Since these features both correlate with human judgements and can be exploited if the auto\-marker puts too much reliance on them, the goal is to mitigate the latter whilst balancing the contribution of a feature to the predicted score\. Existing mitigation strategies, such as feature reweighting and length normalisation\[[11](https://arxiv.org/html/2607.16085#bib.bib35),[1](https://arxiv.org/html/2607.16085#bib.bib36)\], gradient\-based penalties\[[32](https://arxiv.org/html/2607.16085#bib.bib8),[33](https://arxiv.org/html/2607.16085#bib.bib10)\], and attribution\-based defences\[[34](https://arxiv.org/html/2607.16085#bib.bib5),[25](https://arxiv.org/html/2607.16085#bib.bib4),[36](https://arxiv.org/html/2607.16085#bib.bib7)\], all require the shortcut feature to be an explicit, differentiable input to the model\. Fine\-tuned encoder\-based graders, which take raw audio or speech transcripts directly, do not satisfy this condition: e\.g\. response length is absorbed implicitly into learned representations with no direct gradient path from the predicted score to this proxy feature\. We propose a training objective that penalises over\-dependence on any externally computable proxy feature by adding a rank correlation penalty term to the primary task loss\. Both losses are rank\-based, ensuring comparability across features with differing dynamic ranges\. The penalty operates purely at the output level without modifying the encoder or requiring explicit feature access, and is applicable to both text\-based and audio\-based graders\.

We demonstrate the method on the Speak & Improve 2025 Corpus\[[24](https://arxiv.org/html/2607.16085#bib.bib14),[23](https://arxiv.org/html/2607.16085#bib.bib15)\]across four task types, using a ModernBERT text grader\[[31](https://arxiv.org/html/2607.16085#bib.bib29),[29](https://arxiv.org/html/2607.16085#bib.bib13)\]and a wav2vec 2\.0 audio grader\[[6](https://arxiv.org/html/2607.16085#bib.bib28)\], with word count and Voice Activity Detection \(VAD\) time as the respective shortcut proxies\. Zero\-shot Qwen2\.5\-72B predictions\[[26](https://arxiv.org/html/2607.16085#bib.bib33)\]are included as an external reference baseline\. Our results demonstrate that shortcut reliance can be reduced in a controlled manner while preserving competitive scoring performance\. Using human raters’ own feature correlations as a reference point, we identify two complementary operating regimes: a*human\-alignment*mode that matches human reliance on proxy features, and a more aggressive*malpractice\-suppression*mode that further reduces shortcut dependence at an accepted cost in predictive accuracy\. Importantly, the penalty primarily suppresses targeted proxy correlations while leaving unrelated features largely unaffected, indicating that the intervention is selective rather than causing a general degradation in model behaviour\.

The main contributions of this work are:

- •A rank correlation penalty that controls implicit shortcut reliance in end\-to\-end models, operating at the output level without requiring explicit feature inputs\.
- •Empirical evidence of shortcut reliance across both modalities, and a principled cross\-modal comparison of its suppression under the proposed penalty\.
- •Two interpretable operating modes, i\.e\.human\-alignmentandmalpractice\-suppression, that can be selected by assessment designer\.

## IIAutomated Spoken Language Assessment

Automated SLA systems assign a proficiency scorey^\(i\)\\hat\{y\}^\{\(i\)\}to a learner utterance, using the raw audio signalx1:T\(i\)x\_\{1:T\}^\{\(i\)\}and/or an automatic speech recognition \(ASR\) transcriptw1:L\(i\)w\_\{1:L\}^\{\(i\)\}, as input, whereTTdenotes the number of audio frames andLLthe number of recognised word tokens\. The goal is to estimate the holistic judgement of a human examiner of this utterancey\(i\)y^\{\(i\)\}\. A standard training criterion for these systems, examples of which are shown in Figure[1](https://arxiv.org/html/2607.16085#S2.F1), is to minimise the Mean Squared Error \(MSE\) betweeny\(i\)\{y^\{\(i\)\}\}andy^\(i\)\{\\hat\{y\}\}^\{\(i\)\}\[[23](https://arxiv.org/html/2607.16085#bib.bib15),[6](https://arxiv.org/html/2607.16085#bib.bib28)\]

ℒ=1n​∑i=1n\(y\(i\)−y^\(i\)\)2\\displaystyle\{\\cal L\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\left\(y^\{\(i\)\}\-\{\\hat\{y\}\}^\{\(i\)\}\\right\)^\{2\}\(1\)wherennis the number of training examples\.

![Refer to caption](https://arxiv.org/html/2607.16085v1/plots/stefano-slate2023.png)Figure 1:Three typical spoken language assessment system architectures\[[4](https://arxiv.org/html/2607.16085#bib.bib30)\]: a\) hand\-crafted feature\-based grader, b\) text BERT\-based grader, and c\) audio wav2vec 2\.0\-based grader\.### II\-AHand\-crafted feature\-based auto\-markers

Early automated graders use a set ofkkinterpretable featuresz^1:k\(i\)\\hat\{z\}\_\{1:k\}^\{\(i\)\}computed directly from the audio and transcript\. A relatively simple model is then used to predict the score:

y^\(i\)=fθ​\(z^1:k\(i\)\),z^1:k\(i\)←\{x1:T\(i\),w1:L\(i\)\}\\hat\{y\}^\{\(i\)\}=f\_\{\\theta\}\\\!\\left\(\\hat\{z\}\_\{1:k\}^\{\(i\)\}\\right\),\\hat\{z\}\_\{1:k\}^\{\(i\)\}\\leftarrow\\bigl\\\{x\_\{1:T\}^\{\(i\)\},\\,w\_\{1:L\}^\{\(i\)\}\\bigr\\\}\(2\)SpeechRater\[[38](https://arxiv.org/html/2607.16085#bib.bib19),[40](https://arxiv.org/html/2607.16085#bib.bib17),[20](https://arxiv.org/html/2607.16085#bib.bib24),[10](https://arxiv.org/html/2607.16085#bib.bib42)\]and the Linguaskill Speaking auto\-marker\[[35](https://arxiv.org/html/2607.16085#bib.bib44),[39](https://arxiv.org/html/2607.16085#bib.bib20),[17](https://arxiv.org/html/2607.16085#bib.bib22)\]are examples of this form model, typical features include speech rate, pronunciation scores, measures of vocabulary and grammatical complexity\. Later iterations of SpeechRater explicitly targeted feature independence, minimising colinearity between extracted features as a design goal alongside fairness and construct relevance\[[10](https://arxiv.org/html/2607.16085#bib.bib42)\], a consideration we revisit when selecting shortcut proxies in Section[V](https://arxiv.org/html/2607.16085#S5)\. It is important to note that eachz^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}is a hand\-engineered proxy for a latent linguistic constructzj\(i\)z\_\{j\}^\{\(i\)\}that the examiner implicitly takes into account\. For example, the number of word tokens in an ASR transcript,z^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}, approximates the true spoken word countzj\(i\)z\_\{j\}^\{\(i\)\}\. While this approach is transparent and interpretable, with features directly relatable to the speaking assessment criteria, these features are unlikely to cover all attributes of the candidate\. For example higher\-order aspects of proficiency such as discourse coherence and communicative achievement are very difficult to model in this way\[[22](https://arxiv.org/html/2607.16085#bib.bib18)\]\.

### II\-BFine\-tuned encoder\-based auto\-markers

The availability of large pre\-trained acoustic and language models has enabled end\-to\-end fine\-tuning directly from the audio and transcripts with no intermediate feature extraction\. In audio\-based auto\-markers, self\-supervised models such as wav2vec 2\.0\[[2](https://arxiv.org/html/2607.16085#bib.bib25)\]are fine\-tuned on the raw waveform\[[6](https://arxiv.org/html/2607.16085#bib.bib28),[4](https://arxiv.org/html/2607.16085#bib.bib30)\], yielding:

y^\(i\)=fθ​\(x1:T\(i\)\)\\hat\{y\}^\{\(i\)\}=f\_\{\\theta\}\\\!\\left\(x\_\{1:T\}^\{\(i\)\}\\right\)\(3\)
In text\-based auto\-markers, pre\-trained foundation models such as BERT\[[13](https://arxiv.org/html/2607.16085#bib.bib26)\]and Longformer\[[7](https://arxiv.org/html/2607.16085#bib.bib32)\]are fine\-tuned on the ASR transcripts\[[31](https://arxiv.org/html/2607.16085#bib.bib29),[27](https://arxiv.org/html/2607.16085#bib.bib31),[23](https://arxiv.org/html/2607.16085#bib.bib15)\], giving:

y^\(i\)=fθ​\(w1:L\(i\)\)\\hat\{y\}^\{\(i\)\}=f\_\{\\theta\}\\\!\\left\(w\_\{1:L\}^\{\(i\)\}\\right\)\(4\)Both types of models learn a latent representation that implicitly encodes proficiency\-relevant information, rather than computing it directly and explicitly as in feature engineering\.

### II\-CLarge language models as auto\-markers

More recently, large language models \(LLMs\) have been evaluated as zero\-shot spoken assessment systems operating on ASR transcripts\[[5](https://arxiv.org/html/2607.16085#bib.bib21)\]\. Speech LLMs that accept raw speech with text instructions have also been examined\. However the zero\-shot performance of current models is not as good as the fine\-tuned variants\[[26](https://arxiv.org/html/2607.16085#bib.bib33)\]\. Additionally, their computational cost constrains their deployment for large\-scale assessment\. The present work therefore focuses on fine\-tuned audio and text encoders, with zero\-shot LLM predictions serving as a non\-fine\-tuned baseline\.

## IIIShortcut Reliance in Auto\-markers

### III\-AMeasuring shortcut reliance

Correlation between system predictions and interpretable features has long been used as a diagnostic for over\-reliance in automated assessment\. In Automated Essay Scoring \(AES\), the early version of e\-rater yielded predictions that showed minimal differences to using essay length alone\[[11](https://arxiv.org/html/2607.16085#bib.bib35)\]\. E\-rater V\.2 explicitly reduced this correlation with length as a deliberate validity criterion\[[1](https://arxiv.org/html/2607.16085#bib.bib36)\]\. Pearson and Spearman rank correlations between essay length and model scores have since been computed as standard diagnostic quantities, and it has been shown that a model exploiting length alone can achieve state\-of\-the\-art performance on benchmark datasets\[[21](https://arxiv.org/html/2607.16085#bib.bib3)\]\.

Similar concerns arise in Spoken Language Assessment \(SLA\)\. Temporal fluency features, including speech rate, pause distribution, VAD duration, and word count, correlate with proficiency and have been used as features in handcrafted scoring engines\[[35](https://arxiv.org/html/2607.16085#bib.bib44),[10](https://arxiv.org/html/2607.16085#bib.bib42),[3](https://arxiv.org/html/2607.16085#bib.bib43)\]\. The validity of such measures in automated speaking assessment was examined in\[[18](https://arxiv.org/html/2607.16085#bib.bib45)\], observing that automatic speech evaluation systems often rely on a limited range of temporal and lexical features, and finding that articulation rate and pause frequency together account for a substantial share of variation in functional adequacy scores\. These approaches all sit within the explicit feature extraction framework previously described\.

Our work addresses a distinct problem, where features are implicitly encoded by the auto\-marker from raw audio and ASR transcripts\. Let𝐲^=\{y^\(1\),…,y^\(n\)\}\\hat\{\\mathbf\{y\}\}=\\bigl\\\{\\hat\{y\}^\{\(1\)\},\\ldots,\\hat\{y\}^\{\(n\)\}\\bigr\\\}and𝐲=\{y\(1\),…,y\(n\)\}\\mathbf\{y\}=\\bigl\\\{y^\{\(1\)\},\\ldots,y^\{\(n\)\}\\bigr\\\}denote model predictions and ground\-truth human scores overnnutterances respectively\. For a given proxy, interpretable feature,𝐳^j=\{z^j\(1\),…,z^j\(n\)\}\\hat\{\\mathbf\{z\}\}\_\{j\}=\\bigl\\\{\\hat\{z\}\_\{j\}^\{\(1\)\},\\ldots,\\hat\{z\}\_\{j\}^\{\(n\)\}\\bigr\\\}, a shortcut is present when the model’s predictions are more strongly correlated with the proxy than the human labels are:

ρ​\(𝐲^,𝐳^j\)≫ρ​\(𝐲,𝐳^j\),\\rho\\\!\\left\(\\hat\{\\mathbf\{y\}\},\\,\\hat\{\\mathbf\{z\}\}\_\{j\}\\right\)\\gg\\rho\\\!\\left\(\\mathbf\{y\},\\,\\hat\{\\mathbf\{z\}\}\_\{j\}\\right\),\(5\)Hereρ​\(⋅,⋅\)\\rho\(\\cdot,\\cdot\)denotes Spearman rank correlation \(SRC\) which is defined as

ρ​\(𝐲^,𝐲\)=\(1−6​∑i=1n\(r​\(y^\(i\)\)−r​\(y\(i\)\)\)2n​\(n2−1\)\),\\rho\(\\hat\{\\mathbf\{y\}\},\\,\\mathbf\{y\}\)=\\\!\\left\(1\-\\frac\{6\\sum\_\{i=1\}^\{n\}\\bigl\(r\(\\hat\{y\}^\{\(i\)\}\)\-r\(y^\{\(i\)\}\)\\bigr\)^\{2\}\}\{n\(n^\{2\}\-1\)\}\\right\),\(6\)wherer​\(y\(i\)\)∈\{1,2,…,n\}r\(y^\{\(i\)\}\)\\in\\\{1,2,\\ldots,n\\\}denotes the rank of scorey\(i\)y^\{\(i\)\}within the batch of sizenn\. This formulation is model agnostic, it applicable to feature\-based graders, wherez^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}is an explicit model input, as well as encoder\-based graders, wherez^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}is not explicitly represented but can be computed from the raw input independently of the model\. It is also sign\-agnostic, covering positive proxies, which relate to aspects of speaking where an increase correlates with an increased score \(e\.g\. utterance length\), and negative proxies, where the speaker wants to decrease their occurrence to get a higher score \(e\.g\. disfluency count\), alike\.

### III\-BControlling explicit shortcut reliance

When the shortcut featurez^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}is an explicit model input, mitigation strategies can exploit direct access to this feature\. In prior work, feature reweighting and the introduction of content, discourse, and lexical features reduced length dependence in early systems\[[11](https://arxiv.org/html/2607.16085#bib.bib35),[1](https://arxiv.org/html/2607.16085#bib.bib36)\]\. Content\-similarity objectives were proposed to bypass length entirely by design\[[21](https://arxiv.org/html/2607.16085#bib.bib3)\], and linguistically\-informed counterfactuals have been used to diagnose and penalise reliance on identified input dimensions\[[36](https://arxiv.org/html/2607.16085#bib.bib7)\]\.

Another class of approaches uses post\-hoc interpretability tools to diagnose shortcut reliance\. Integrated gradients \(IG\)\[[34](https://arxiv.org/html/2607.16085#bib.bib5)\]attribute an importance score to each input dimension by integrating the model’s gradient along the path from a neutral baseline to the actual input, and have been applied to reveal that AES models exploits shallow surface cues, such as specific trigger words, in the input to drive model predictions\[[25](https://arxiv.org/html/2607.16085#bib.bib4)\]\. In\[[25](https://arxiv.org/html/2607.16085#bib.bib4)\]IG remain diagnostic tools rather than training\-time penalties\. The Right for the Right Reasons \(RRR\) framework\[[32](https://arxiv.org/html/2607.16085#bib.bib8)\]extends this explainability into the training process by penalising the gradient of the prediction with respect to shortcut features in addition to the general task\-specific lossℒtask\\mathcal\{L\}\_\{\\mathrm\{task\}\}, in a simplified form:

ℒ=ℒtask\+λn​∑i=1n\(∂y^\(i\)∂z^j\(i\)\)2\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{task\}\}\+\\frac\{\\lambda\}\{n\}\\sum\_\{i=1\}^\{n\}\\left\(\\frac\{\\partial\\hat\{y\}^\{\(i\)\}\}\{\\partial\\hat\{z\}\_\{j\}^\{\(i\)\}\}\\right\)^\{\\\!2\}\.\(7\)Right for Better Reasons\[[33](https://arxiv.org/html/2607.16085#bib.bib10)\]further extends this framework by replacing input gradients with influence functions as the constraint signal, demonstrating more effective correction of shortcut behaviour\.

A related line of work corrects shortcut behaviour directly at the representation level: COrrect and MItigate \(COMI\)\[[41](https://arxiv.org/html/2607.16085#bib.bib38)\]introduces a shortcut margin loss that adaptively suppresses the feature weights of identified shortcuts, using attribution\-based methods such as Local Interpretable Model\-agnostic Explanations \(LIME\) to locate shortcut tokens in text inputs\. Focus\-and\-Ignore Low\-Rank Adaptation \(FiLoRA\)\[[12](https://arxiv.org/html/2607.16085#bib.bib39)\]decomposes model adaptation into feature\-group\-aligned LoRA modules and applies instruction\-conditioned gating, allowing a natural language instruction to selectively amplify or suppress identified core or spurious feature groups in multimodal models\.

However, these approaches share a common requirement: the shortcut feature must be an explicit, accessible input to the model, so that it can be removed, reweighted, or targeted by a gradient\-based penalty\. Fine\-tuned encoder\-based auto\-markers do not satisfy these conditions\. A ModernBERT grader operating on ASR transcripts, or a wav2vec 2\.0 grader operating on raw waveforms, does not receive word count or speech duration as an input\. Instead, length information is absorbed implicitly into learned representations via attention over token sequences and position encodings\. That is, for encoder\-based graders there is no direct gradient path fromy^\(i\)\\hat\{y\}^\{\(i\)\}toz^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}; Equation[7](https://arxiv.org/html/2607.16085#S3.E7)is therefore inapplicable\.

While the feature\-level solution cannot transfer to encoder\-based graders, the correlation\-based diagnostic in Equation[5](https://arxiv.org/html/2607.16085#S3.E5)can:z^j\(i\)\\hat\{z\}\_\{j\}^\{\(i\)\}, such as word count, speaking rate, or silence duration, can be computed from the raw input independently of the model and compared againsty^\(i\)\\hat\{y\}^\{\(i\)\}at the output level\. The following section introduces a training\-time penalty that exploits this diagnostic without requiring explicit feature access\. It is applicable to both feature\-based and encoder\-based graders within a unified framework\.

## IVControlling Implicit Shortcut Reliance

This section proposes a method to control the reliance on implicit shortcuts\. As previously noted, the aim is to break the over\-reliance of the auto\-marker on one or more interpretable feature whilst maintaining the same relationship that the feature\(s\) has to human judgement\. The proposed method is applicable to any fine\-tuned encoder\-based speech or text grader and can penalise reliance on any computable feature\. As these features must by definition be human interpretable, it is also useful for safeguarding against known test malpractice strategies\. Any feature that can be communicated as actionable advice \(e\.g\. “speak for longer” or “throw in more adjectives”\) is by definition interpretable and computable so can be targeted by this approach\.

Instead of the standard MSE loss as mentioned in Section[II](https://arxiv.org/html/2607.16085#S2), we adopted a rank\-based loss because the dynamic ranges of holistic scores and individual proxy features differ substantially: for example, in the Speak & Improve 2025 Corpus\[[24](https://arxiv.org/html/2607.16085#bib.bib14),[23](https://arxiv.org/html/2607.16085#bib.bib15)\]holistic scores lie in the range\[1,6\]\[1,6\]whilst word counts can exceed100100\. Following\[[15](https://arxiv.org/html/2607.16085#bib.bib9)\], it is possible to use the SRC as the task training criterion\. However since the rank operatorr​\(⋅\)r\(\\cdot\)is discrete and non\-differentiable, it cannot be directly optimised via gradient descent\. Bothr​\(y^\(i\)\)r\(\\hat\{y\}^\{\(i\)\}\)andr​\(y\(i\)\)r\(y^\{\(i\)\}\)are replaced by the differentiable approximation of\[[8](https://arxiv.org/html/2607.16085#bib.bib11)\], yielding a fully differentiable objective\. In the case of tied values, averaged ranks are assigned: for example, if two samples share rank 2, both receive rank 2\.5, which is the standard convention for Spearman correlation, and is handled automatically by this open\-source implementation111https://github\.com/google\-research/fast\-soft\-sort\.

We propose to add one or more regularisation terms to Equation[6](https://arxiv.org/html/2607.16085#S3.E6)\. Let𝒥⊆\{1,…,k\}\\mathcal\{J\}\\subseteq\\\{1,\\ldots,k\\\}denote the set of proxy indices selected for regularisation\. For eachj∈𝒥j\\in\\mathcal\{J\}, let𝐳^j=\{z^j\(1\),…,z^j\(n\)\}\\hat\{\\mathbf\{z\}\}\_\{j\}=\\bigl\\\{\\hat\{z\}\_\{j\}^\{\(1\)\},\\ldots,\\hat\{z\}\_\{j\}^\{\(n\)\}\\bigr\\\}denote the values of proxyjjacrossnnutterances\. The full training objective to minimise is:

ℒ=−ρ​\(𝐲^,𝐲\)\+∑j∈𝒥λj​ρ​\(𝐲^,𝐳^j\),\\mathcal\{L\}=\-\\rho\\\!\\left\(\\hat\{\\mathbf\{y\}\},\\,\\mathbf\{y\}\\right\)\+\\sum\_\{j\\in\\mathcal\{J\}\}\\,\\lambda\_\{j\}\\,\\rho\\\!\\left\(\\hat\{\\mathbf\{y\}\},\\,\\hat\{\\mathbf\{z\}\}\_\{j\}\\right\),\(8\)where𝐲\\mathbf\{y\}and𝐲^\{\\hat\{\\mathbf\{y\}\}\}are the reference and prediction scores respectively andλj≥0\\lambda\_\{j\}\\geq 0controls the strength of penalisation for proxyjj\. Thus minimisingℒ\\mathcal\{L\}simultaneously maximises prediction–score correlation and minimises prediction–proxy correlation\. The choice ofλj\\lambda\_\{j\}determines the level to which the correlation with the interpretable feature, proxy,𝐳^j\\hat\{\\mathbf\{z\}\}\_\{j\}is supressed\. It can be used to reduce the correlation to the level seen in the reference scores,ρ​\(𝐲,𝐳^j\)\\rho\(\\mathbf\{y\},\{\\hat\{\\mathbf\{z\}\}\_\{j\}\}\), or set to higher values to penalise candidates who attempt to use this form of shortcut\.

## VExperimental set\-up

### V\-AS&I data

Experiments are conducted on the Speak & Improve \(S&I\) corpus\[[24](https://arxiv.org/html/2607.16085#bib.bib14),[23](https://arxiv.org/html/2607.16085#bib.bib15)\], a large\-scale dataset of spoken English responses from L2 learners collected through the Speak & Improve online practice platform222https://speakandimprove\.com\. Responses span four open speaking task types: Part 1 \(interview: short responses to personal questions\), Part 3 \(long turn: giving an opinion on a topic\), Part 4 \(long turn: describing a diagram\), and Part 5 \(communication activity: responding to questions on a topic\)\. Each part is assigned a holistic proficiency scorey\(i\)∈\[1,6\]y^\{\(i\)\}\\in\[1,6\]corresponding approximately to CEFR levels A1 to C1\+\. The corpus is divided into training \(6,642 submissions, 39,490 utterances\), development \(438 submissions, 5,616 utterances\), and evaluation \(300 submissions, 3,209 utterances\) sets\[[23](https://arxiv.org/html/2607.16085#bib.bib15)\]\.

Table[I](https://arxiv.org/html/2607.16085#S5.T1)reports Spearman rank correlations between four candidate features and the holistic reference score, alongside the feature–feature correlations that motivate our choice of shortcut proxy for each grader\. The candidate text\-based features selected are the a\) number of words \(\#words\) and b\) number of unique words \(\#uniq\), both computed from the ASR transcript\. For audio\-based features, a\) average ASR confidence \(ASR conf\) per word and b\) the candidate’s speaking time in seconds as measured by a voice activity detection system \(VAD time\), were selected for this paper\.

TABLE I:Spearmanρ\\rhobetween text\- and audio\-based features, and holistic reference score for the S&I evaluation set\.FeaturesRef score\#words\#uniq\.ASR confVAD time\#words0\.6591\.0000\.9530\.4460\.655\#uniq\.0\.680–1\.0000\.4860\.624ASR conf0\.657––1\.0000\.234VAD time0\.450–––1\.000For the text\-based grader, we adopt word countz^wc\(i\)\\hat\{z\}\_\{\\mathrm\{wc\}\}^\{\(i\)\}, derived directly from theWhisper\-small\.en333https://huggingface\.co/openai/whisper\-small\.entranscriptw1:L\(i\)w\_\{1:L\}^\{\(i\)\}, as the shortcut proxy𝐳^j\\hat\{\\mathbf\{z\}\}\_\{j\}, since it is the more directly actionable quantity \(a learner can be advised to “speak for longer”, whereas “use more unique words” is a less concrete strategy\)\. Unique word count is used as an internal consistency check: we expect the penalty applied to word count to suppress reliance on unique word count too given their high correlation\. Average ASR confidence per word is used as a selectivity check: an effective penalty should leave its correlation with model predictions largely intact, providing evidence that the intervention is targeted rather than a general degradation\.

For the audio\-based grader, we examine VAD timez^VAD\(i\)\\hat\{z\}\_\{\\mathrm\{VAD\}\}^\{\(i\)\}, computed directly from the waveformx1:T\(i\)x\_\{1:T\}^\{\(i\)\}using the Praat VAD algorithm, as the shortcut proxy, since it reflects the fact that longer spoken responses tend to contain more speech content\. Word count and average ASR confidence per word are also used consistency checks to see the impact on other attributes\.

### V\-BModels

All graders are trained at the response level: one example per response, sharing the part\-level holistic label as the target\. For the two fine\-tuned graders, the shortcut regularisation strengthλj\\lambda\_\{j\}is swept over\[0,0\.30\]\[0,0\.30\], and the differentiable rank approximation of\[[8](https://arxiv.org/html/2607.16085#bib.bib11)\]\(Section[IV](https://arxiv.org/html/2607.16085#S4)\) uses a default regularisation strength of0\.10\.1throughout\.

#### V\-B1ModernBERT and Qwen2\.5\-72B

Both text\-based graders use the ASR transcriptw1:L\(i\)w\_\{1:L\}^\{\(i\)\}fromWhisper\-small\.en\. For the ModernBERT\-based grader, the architecture follows the BERT\-based grader of\[[23](https://arxiv.org/html/2607.16085#bib.bib15)\]apart from the encoder beingModernBERT\-base444https://huggingface\.co/answerdotai/ModernBERT\-base: token\-level embeddings from the encoder are pooled into an utterance\-level representation using four parallel self\-attention heads, and passed through a two\-layer feedforward network with ReLU activations to a scalar outputy^\(i\)=fθ​\(w1:L\(i\)\)\\hat\{y\}^\{\(i\)\}=f\_\{\\theta\}\(w\_\{1:L\}^\{\(i\)\}\)\. For eachλWC\\lambda\_\{\\mathrm\{WC\}\}, ten models are trained with different random seeds and predictions are averaged across the ensemble\. The training objective follows Equation[8](https://arxiv.org/html/2607.16085#S4.E8), with word count𝐳^wc\\hat\{\\mathbf\{z\}\}\_\{\\mathrm\{wc\}\}as the shortcut proxy \(\|𝒥\|=1\|\\mathcal\{J\}\|=1\)\. Zero\-shot predictions from Qwen2\.5\-72B\[[30](https://arxiv.org/html/2607.16085#bib.bib47)\], following the natural language\-based assessment approach\[[5](https://arxiv.org/html/2607.16085#bib.bib21)\], as a system that is not fine\-tuned\.

#### V\-B2wav2vec 2\.0

The audio\-based grader uses wav2vec 2\.0\-base\[[2](https://arxiv.org/html/2607.16085#bib.bib25)\]as the pre\-trained encoder, with the standard mean\-pooling output layer replaced by four attention heads that pool frame\-level representations into a single utterance vector, consistent with\[[27](https://arxiv.org/html/2607.16085#bib.bib31)\]\. The pooled representation is passed through a two\-layer feedforward head \(hidden size4×7684\\times 768, ReLU with dropout before and after\) to a scalar output,y^\(i\)=fθ​\(x1:T\(i\)\)\\hat\{y\}^\{\(i\)\}=f\_\{\\theta\}\(x\_\{1:T\}^\{\(i\)\}\)\. The model is fine\-tuned with AdamW\. Parts 3 and 4 are trained with batch size 16, while Parts 1 and 5 are trained with batch size 64 to increase within\-batch rank variablility \(since they contain multiple short responses sharing one part\-level label\)\. All graders are trained for 2 epochs and no seed ensembling is used, owing to the higher cost of training on raw waveforms\. The training objective follows Equation[8](https://arxiv.org/html/2607.16085#S4.E8), with VAD time𝐳^VAD\\hat\{\\mathbf\{z\}\}\_\{\\mathrm\{VAD\}\}as the shortcut proxy \(\|𝒥\|=1\|\\mathcal\{J\}\|=1\)\.

### V\-CScore Calibration and Combination

Part\-level predictions from models trained with differentλ\\lambdavalues are combined into a single submission\-level score for computing Spearman rank correlation across the full test set\. As the SRC loss function does not impose any constraints on the absolute values of any predictions, only the ranking, some form of score calibration is required\. A standard approach is to apply linear calibration of part predictions to the reference score scale\[[23](https://arxiv.org/html/2607.16085#bib.bib15)\]\. This was considered but rejected here: at highλ\\lambda, the penalty can driveρ​\(𝐲^,𝐲\)\\rho\(\\hat\{\\mathbf\{y\}\},\\mathbf\{y\}\)sufficiently low that the calibration slopeaainy~cal\(p,i\)=a​y^\(p,i\)\+b\\tilde\{y\}\_\{\\mathrm\{cal\}\}^\{\(p,i\)\}=a\\,\\hat\{y\}^\{\(p,i\)\}\+bbecomes negative\. Sinceρ​\(𝐲~cal,𝐳^j\)=a⋅ρ​\(𝐲^,𝐳^j\)\\rho\(\\tilde\{\\mathbf\{y\}\}\_\{\\mathrm\{cal\}\},\\hat\{\\mathbf\{z\}\}\_\{j\}\)=a\\cdot\\rho\(\\hat\{\\mathbf\{y\}\},\\hat\{\\mathbf\{z\}\}\_\{j\}\), a negative slope flips the sign of the proxy correlation, causing a spurious bounce\-back in the measuredλ\\lambdasweep\. Henceforth, Z\-score normalisation is adopted, since it operates purely on the prediction distribution without reference to𝐲\\mathbf\{y\}and therefore cannot induce a sign flip\.

For each partp∈P=\{P1, P3, P4, P5\}p\\in P=\\\{\\text\{P1, P3, P4, P5\}\\\}and each value ofλj\\lambda\_\{j\}, the part\-level predictions𝐲^\(p\)=\{y^\(p,i\)\}i=1np\\hat\{\\mathbf\{y\}\}^\{\(p\)\}=\\bigl\\\{\\hat\{y\}^\{\(p,i\)\}\\bigr\\\}\_\{i=1\}^\{n\_\{p\}\}are normalised as:

y~\(p,i\)=y^\(p,i\)−μ\(p\)σ\(p\),\\tilde\{y\}^\{\(p,i\)\}=\\frac\{\\hat\{y\}^\{\(p,i\)\}\-\\mu^\{\(p\)\}\}\{\\sigma^\{\(p\)\}\},\(9\)whereμ\(p\)\\mu^\{\(p\)\}andσ\(p\)\\sigma^\{\(p\)\}are the mean and standard deviation of predictions for partpp, andnpn\_\{p\}is the number of utterances in partpp\. The submission\-level score for utteranceiiis then the average of normalised part predictions:

y~\(i\)=1\|P\|​∑p∈Py~\(p,i\),\\tilde\{y\}^\{\(i\)\}=\\frac\{1\}\{\|P\|\}\\sum\_\{p\\in P\}\\tilde\{y\}^\{\(p,i\)\},\(10\)where each part contributes equally regardless of prediction scale\. Per\-part Spearman rank correlations are computed directly from𝐲^\(p\)\\hat\{\\mathbf\{y\}\}^\{\(p\)\}before normalisation\. Z\-score normalisation is applied only for submission\-level combination\.

## VIExperiment results

### VI\-AText as input: ModernBERT

Table[II](https://arxiv.org/html/2607.16085#S6.T2)reports Spearman correlations between ModernBERT predictions and the reference score and three text\-derived features, alongside the corresponding human reference correlations\. Using SRC with no additional regularisation rather than MSE loss has negligible effect on both the baseline performance and level of correlation with features\. Both ModernBERT\-based models exhibit substantially higher correlations with word count and unique word count than the human reference exhibits with the same features\. This illustrates an over\-reliance, shortcut, on these features based on Equation[5](https://arxiv.org/html/2607.16085#S3.E5)\. Conversely correlations with average ASR confidence are consistent with the human reference\. The zero\-shot Qwen2\.5\-72B feature correlations are more consistent with the reference correlations\. It does not have the very high correlations of the ModernBERT system with the number of words and unique words\. This is felt to be because the zero\-shot model has had no fine\-tuning to the task, thus it has no opportunity to learn task\-specific shortcuts,

TABLE II:Baseline Text models: score and feature correlations in the S&I evaluation set\.ModelLossRef\#words\#uniq\.ASR confReference–1\.0000\.6590\.6800\.657ModernBERTMSE0\.7580\.8720\.9110\.653SRC0\.7610\.8740\.9110\.655Qwen2\.5\-72B–0\.7550\.7330\.7920\.719To reduce the influence of the number of words555Word\-count rather than the number of unique words was used as this is less sensitive to the length of the response\.on the ModernBERT score a regularisation term targeting this feature was introduced wih a controllable weightλWC\\lambda\_\{\\mathrm\{WC\}\}\. Figure[2](https://arxiv.org/html/2607.16085#S6.F2)shows how feature and score correlations evolve asλWC\\lambda\_\{\\mathrm\{WC\}\}is swept over\[0,0\.30\]\[0,0\.30\]\. AsλWC\\lambda\_\{\\mathrm\{WC\}\}increases, correlation with word count decreases steadily, crossing the human reference level atλWC≈0\.13\\lambda\_\{\\mathrm\{WC\}\}\\approx 0\.13, which defines thehuman\-alignmentoperating mode, where shortcut reliance on word count matches the level exhibited by human raters \(ρ​\(𝐲^,𝐳^WC\)≈ρ​\(𝐲,𝐳^WC\)\\rho\(\\hat\{\\mathbf\{y\}\},\\hat\{\\mathbf\{z\}\}\_\{\\mathrm\{WC\}\}\)\\approx\\rho\(\\mathbf\{y\},\\hat\{\\mathbf\{z\}\}\_\{\\mathrm\{WC\}\}\)\)\. Accuracy,ρ​\(𝐲^,𝐲\)\\rho\(\\hat\{\\mathbf\{y\}\},\\mathbf\{y\}\), remains at a competitive level\. Beyond thisλWC\\lambda\_\{\\mathrm\{WC\}\}point, both accuracy and word\-count correlation continue to decrease more sharply, defining themalpractice\-suppressionmode and reflecting the cost of shortcut suppression\. The unique word count shows a similar pattern to word count, confirming that the penalty has a consistent effect on highly correlated shortcut features\. Features that are not shortcut proxies, here represented by average ASR confidence, are minimally affected by the penalty until broader degradation sets in at highλWC\\lambda\_\{\\mathrm\{WC\}\}, providing evidence that the intervention is targeted rather than a general suppression of model reliance\.

![Refer to caption](https://arxiv.org/html/2607.16085v1/x1.png)Figure 2:Spearmanρ\\rhobetween ModernBERT\-based models’ predictions and reference features acrossλ\\lambdavalues\. Dashed horizontal lines indicate the Spearmanρ\\rhobetween reference scores and each feature\. \(S&I evaluation set\)
### VI\-BAudio as input: wav2vec 2\.0

Table[III](https://arxiv.org/html/2607.16085#S6.T3)shows that wav2vec 2\.0 exhibits higher correlation with VAD time and word count than the human reference, confirming over\-reliance consistent with Equation[5](https://arxiv.org/html/2607.16085#S3.E5)\. As with the text\-based grader, average ASR confidence shows no over\-reliance at baseline\.

TABLE III:Audio models: score and feature correlations in the S&I evaluation set\.ModelRef scoreVAD time\#wordsASR confReference1\.0000\.4500\.6590\.657wav2vec 2\.00\.6120\.7050\.7800\.514Again a regularisation term targeting in this case the VAD time with weightλVAD\\lambda\_\{\\mathrm\{VAD\}\}was introduced\. Figure[3](https://arxiv.org/html/2607.16085#S6.F3)shows how these correlations evolve asλVAD\\lambda\_\{\\mathrm\{VAD\}\}is swept over\[0,0\.30\]\[0,0\.30\]\. Correlation with VAD time decreases steadily asλVAD\\lambda\_\{\\mathrm\{VAD\}\}increases, and crosses the human reference level atλVAD≈0\.17\\lambda\_\{\\mathrm\{VAD\}\}\\approx 0\.17\(ρ​\(𝐲^,𝐳^VAD\)≈ρ​\(𝐲,𝐳^VAD\)\\rho\(\\hat\{\\mathbf\{y\}\},\\hat\{\\mathbf\{z\}\}\_\{\\mathrm\{VAD\}\}\)\\approx\\rho\(\\mathbf\{y\},\\hat\{\\mathbf\{z\}\}\_\{\\mathrm\{VAD\}\}\)\), marking thehuman\-alignmentoperating mode at a moderate cost to accuracy,ρ​\(𝐲^,𝐲\)\\rho\(\\hat\{\\mathbf\{y\}\},\\mathbf\{y\}\)\. Past this point, both continue to decline more sharply, entering themalpractice\-suppressionmode\. A similar trend is seen for word count, consistent with its relatively high correlation with VAD time \(Table[I](https://arxiv.org/html/2607.16085#S5.T1)\)\. Average ASR confidence, representing a non\-shortcut feature, remains largely stable until highλVAD\\lambda\_\{\\mathrm\{VAD\}\}, again pointing to a targeted rather than general suppression of model reliance, which is consistent with what is observed in the text\-based grader \(Section[VI\-A](https://arxiv.org/html/2607.16085#S6.SS1)\)\.

![Refer to caption](https://arxiv.org/html/2607.16085v1/plots/wav2vec2_znorm/overall_src_evl.png)Figure 3:Spearmanρ\\rhobetween wav2vec\-based model predictions and reference features acrossλ\\lambdavalues\. Dashed horizontal lines indicate the Spearmanρ\\rhobetween reference scores and each feature\. \(S&I evaluation set\)

## VIIConclusions and Future Work

This paper introduced a rank correlation penalty that discourages fine\-tuned encoder\-based graders from over\-relying on externally computable proxy features, operating at the output level without requiring access to or modification of the encoder’s internal representations\. Unlike prior explicit feature\-level mitigation strategies, which presuppose a direct gradient path from the predicted score to the shortcut feature, the proposed penalty applies to both text\-based and audio\-based graders within a unified framework\. Experiments on the Speak & Improve corpus confirmed shortcut reliance in both a ModernBERT text\-based grader and a wav2vec 2\.0 audio\-based grader, and demonstrated that the penalty reduces this over\-reliance in a controllable manner, yielding two natural operating modes: ahuman\-alignmentmode, where shortcut reliance is reduced to match the human reference level with minimal cost to accuracy,; and amalpractice\-suppressionmode, where reliance is reduced further at an accepted cost to accuracy\. Across both modalities, features that are highly correlated with the target shortcut proxy are affected by the penalty in a consistent way; features that are not shortcut proxies are minimally affected by the penalty until broader degradation sets in, providing evidence that the intervention is targeted rather than a general suppression of model reliance\.

Several directions remain open\. The zero\-shot Qwen2\.5\-72B baseline exhibits some verbosity bias, suggesting shortcut reliance is not entirely absent from large pre\-trained models; extending the approach to fine\-tuned LLM\-based graders is a natural extension\. The general formulation in Equation[8](https://arxiv.org/html/2607.16085#S4.E8)supports simultaneous penalisation over a set𝒥\\mathcal\{J\}of proxies, enabling multiple exploitable features to be targeted at once\. Finally, all experiments used holistic monologic assessment; extending the approach to dialogic assessment is another natural next step\.

## Acknowledgment

This paper reports on research supported by Cambridge University Press & Assessment, a department of The Chancellor, Masters, and Scholars of the University of Cambridge\. The authors would like to thank the ALTA Spoken Language Processing Technology Project Team for general discussions and contributions to the evaluation infrastructure\.

No use was made of Generative AI in the writing of this paper\.

## References

- \[1\]Y\. Attali\(2007\)Construct validity of e\-rater in scoring TOEFL essays\.Technical reportTechnical ReportRR\-07\-21,Educational Testing Service\.External Links:[Document](https://dx.doi.org/10.1002/j.2333-8504.2007.tb02063.x),[Link](https://www.ets.org/research/policy_research_reports/publications/report/2007/hsmn.html)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p1.1)\.
- \[2\]A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli\(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 12449–12460\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p1.1),[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p1.1),[§V\-B2](https://arxiv.org/html/2607.16085#S5.SS2.SSS2.p1.4)\.
- \[3\]P\. Bamdev, M\. S\. Grover, Y\. K\. Singla, P\. Vafaee, M\. Hama, and R\. R\. Shah\(2023\)Automated speech scoring system under the lens: evaluating and interpreting the linguistic cues for language proficiency\.International Journal of Artificial Intelligence in Education33\(1\),pp\. 119–154\.External Links:[Link](https://link.springer.com/content/pdf/10.1007/s40593-022-00291-5.pdf)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p2.1)\.
- \[4\]S\. Bannò, K\. Knill, M\. Matassoni, V\. Raina, and M\. J\.F\. Gales\(2023\)Assessment of L2 oral proficiency using self\-supervised speech representation learning\.InProc\. 9th Workshop on Speech and Language Technology in Education \(SLaTE\),pp\. 126–130\.External Links:[Document](https://dx.doi.org/10.21437/SLaTE.2023-24),[Link](https://www.isca-archive.org/slate_2023/banno23_slate.html)Cited by:[Figure 1](https://arxiv.org/html/2607.16085#S2.F1),[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p1.1)\.
- \[5\]S\. Bannò, R\. Ma, M\. Qian, S\. Tang, K\. Knill, and M\. J\.F\. Gales\(2025\)Natural Language\-based Assessment of L2 Oral Proficiency using LLMs\.arXiv preprint arXiv:2507\.10200\.External Links:[Link](https://arxiv.org/abs/2507.10200)Cited by:[§II\-C](https://arxiv.org/html/2607.16085#S2.SS3.p1.1),[§V\-B1](https://arxiv.org/html/2607.16085#S5.SS2.SSS1.p1.5)\.
- \[6\]S\. Bannò and M\. Matassoni\(2023\)Proficiency assessment of L2 spoken English using wav2vec 2\.0\.In2022 IEEE Spoken Language Technology Workshop \(SLT\),pp\. 1088–1095\.External Links:[Link](https://ieeexplore.ieee.org/document/10023019)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p4.1),[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p1.1),[§II](https://arxiv.org/html/2607.16085#S2.p1.8)\.
- \[7\]I\. Beltagy, M\. E\. Peters, and A\. Cohan\(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.External Links:[Link](https://arxiv.org/abs/2004.05150)Cited by:[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p2.1)\.
- \[8\]M\. Blondel, O\. Teboul, Q\. Berthet, and J\. Djolonga\(2020\)Fast Differentiable Sorting and Ranking\.InProc\. the 37th International Conference on Machine Learning,Vol\.119,pp\. 950–959\.External Links:[Link](https://proceedings.mlr.press/v119/blondel20a.html)Cited by:[§IV](https://arxiv.org/html/2607.16085#S4.p2.5),[§V\-B](https://arxiv.org/html/2607.16085#S5.SS2.p1.3)\.
- \[9\]A\. Chakravarty, M\. Brenchley, T\. Breakspear, I\. Lewin, and Y\. Huang\(2025\)Enhancing marker scoring accuracy through ordinal confidence modelling in educational assessments\.InProc\. the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 1498–1507\.External Links:[Link](https://aclanthology.org/2025.acl-industry.106/)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1)\.
- \[10\]L\. Chen, K\. Zechner, S\. Yoon, K\. Evanini, X\. Wang, A\. Loukina, J\. Tao, L\. Davis, C\. M\. Lee, M\. Ma, R\. Mundkowsky, C\. Lu, C\. W\. Leong, and B\. Gyawali\(2018\)Automated scoring of nonnative speech using the SpeechRaterSMv\. 5\.0 engine\.ETS Research Report Series2018\(1\),pp\. 1–31\.External Links:[Document](https://dx.doi.org/10.1002/ets2.12198),[Link](https://doi.org/10.1002/ets2.12198)Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p2.1)\.
- \[11\]M\. Chodorow and J\. Burstein\(2004\)Beyond essay length: evaluating e\-rater®’s performance on TOEFL® essays\.Technical reportTechnical ReportRR\-04\-04,Educational Testing Service\.External Links:[Link](https://www.ets.org/Media/Research/pdf/RR-04-04.pdf)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p1.1)\.
- \[12\]H\. Chung, C\. Han, Y\. Choi, S\. Ji, J\. Kim, E\. Holden, and K\. Han\(2026\)FiLoRA: focus\-and\-ignore lora for controllable feature reliance\.arXiv preprint arXiv:2602\.02060\.External Links:[Link](https://arxiv.org/abs/2602.02060)Cited by:[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p3.1)\.
- \[13\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProc\. NAACL\-HLT,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p1.1),[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p2.1)\.
- \[14\]Y\. Farag, H\. Yannakoudakis, and T\. Briscoe\(2018\-06\)Neural automated essay scoring and coherence modeling for adversarially crafted input\.InProc\. the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 263–271\.External Links:[Link](https://aclanthology.org/N18-1024/),[Document](https://dx.doi.org/10.18653/v1/N18-1024)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1)\.
- \[15\]Y\. Fathullah, P\. Radmard, A\. Liusie, and M\. J\.F\. Gales\(2024\-03\)Who Needs Decoders? Efficient Estimation of Sequence\-Level Attributes with Proxies\.InProc\. the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1478–1496\.External Links:[Link](https://aclanthology.org/2024.eacl-long.89/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.89)Cited by:[§IV](https://arxiv.org/html/2607.16085#S4.p2.5)\.
- \[16\]J\. Fleckenstein, J\. Meyer, T\. Jansen, S\. Keller, and O\. Köller\(2020\)Is a long essay always a good essay? The effect of text length on writing assessment\.Frontiers in Psychology11,pp\. 562462\.External Links:[Document](https://dx.doi.org/10.3389/fpsyg.2020.562462),[Link](https://doi.org/10.3389/fpsyg.2020.562462)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1)\.
- \[17\]S\. Gao, M\. J\.F\. Gales, and J\. Xu\(2024\)Detecting aberrant responses in automated L2 spoken English assessment\.InExploring Artificial Intelligence in Applied Linguistics,C\. A\. Chapelle, G\. H\. Beckett, and J\. Ranalli \(Eds\.\),pp\. 96–117\.External Links:[Document](https://dx.doi.org/10.31274/isudp.2024.154.07),[Link](https://www.iastatedigitalpress.com/plugins/books/154/chapter/1231)Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6)\.
- \[18\]Z\. L\. Handley and H\. Wang\(2024\)What do the measures of utterance fluency employed in automatic speech evaluation \(ase\) tell us about oral proficiency?\.Language Assessment Quarterly21\(1\),pp\. 3–32\.External Links:[Link](https://www.tandfonline.com/doi/pdf/10.1080/15434303.2023.2283839)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p2.1)\.
- \[19\]K\. Hermann, H\. Mobahi, T\. Fel, and M\. Mozer\(2024\)On the foundations of shortcut learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 43832–43868\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/bf215fa7fe70a38c5e967e59c44a99d0-Abstract-Conference.html)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p1.1)\.
- \[20\]D\. Higgins, X\. Xi, K\. Zechner, and D\. Williamson\(2011\)A three\-stage approach to the automated scoring of spontaneous spoken responses\.Computer Speech and Language25\(2\),pp\. 282–306\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S0885230810000458)Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6)\.
- \[21\]S\. Jeon and M\. Strube\(2021\-11\)Countering the influence of essay length in neural essay scoring\.InProc\. the Second Workshop on Simple and Efficient Natural Language Processing,N\. S\. Moosavi, I\. Gurevych, A\. Fan, T\. Wolf, Y\. Hou, A\. Marasović, and S\. Ravi \(Eds\.\),Virtual,pp\. 32–38\.External Links:[Link](https://aclanthology.org/2021.sustainlp-1.4/),[Document](https://dx.doi.org/10.18653/v1/2021.sustainlp-1.4)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p1.1)\.
- \[22\]K\. Knill and M\. J\.F\. Gales\(2024\)Building an auto\-marker for assessing spontaneous L2 English speech\.InLanguage Test Validation in a Digital Age,G\. Yu and J\. Xu \(Eds\.\),Studies in Language Testing, Vol\.52,pp\. 309–334\.Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6)\.
- \[23\]K\. Knill, D\. Nicholls, M\. J\.F\. Gales, M\. Qian, and P\. Stroinski\(2025\)Introducing the speak & improve corpus 2025: an l2 english speech corpus for language assessment and feedback\.In10th Workshop on Speech and Language Technology in Education \(SLaTE\),pp\. 167–171\.External Links:[Link](https://www.isca-archive.org/slate_2025/knill25_slate.html)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p4.1),[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p2.1),[§II](https://arxiv.org/html/2607.16085#S2.p1.8),[§IV](https://arxiv.org/html/2607.16085#S4.p2.5),[§V\-A](https://arxiv.org/html/2607.16085#S5.SS1.p1.1),[§V\-B1](https://arxiv.org/html/2607.16085#S5.SS2.SSS1.p1.5),[§V\-C](https://arxiv.org/html/2607.16085#S5.SS3.p1.8)\.
- \[24\]K\. Knill, D\. Nicholls, M\. J\.F\. Gales, M\. Qian, and P\. Stroinski\(2025\)The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback\.External Links:[Link](https://doi.org/10.17863/CAM.114333)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p4.1),[§IV](https://arxiv.org/html/2607.16085#S4.p2.5),[§V\-A](https://arxiv.org/html/2607.16085#S5.SS1.p1.1)\.
- \[25\]Y\. Kumar, S\. Parekh, S\. Singh, J\. J\. Li, R\. R\. Shah, and C\. Chen\(2023\-04\)Automatic essay scoring systems are both overstable and oversensitive: explaining why and proposing defenses\.Dialogue & Discourse14,pp\. 1–33\.External Links:[Link](https://aclanthology.org/2023.dnd-14.7/),[Document](https://dx.doi.org/10.5210/dad.2023.101)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p2.1)\.
- \[26\]R\. Ma, M\. Qian, S\. Tang, S\. Bannò, K\. Knill, and M\. J\.F\. Gales\(2025\)Assessment of L2 oral proficiency using speech large language models\.InProc\. Interspeech 2025,pp\. 5078–5082\.External Links:[Link](https://www.isca-archive.org/interspeech_2025/ma25b_interspeech.pdf)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.16085#S2.SS3.p1.1)\.
- \[27\]S\. W\. McKnight, A\. Civelekoglu, M\. J\.F\. Gales, S\. Bannò, A\. Liusie, and K\. Knill\(2023\)Automatic assessment of conversational speaking tests\.InProc\. SLaTE 2023,pp\. 99–103\.External Links:[Link](https://www.isca-archive.org/slate_2023/mcknight23_slate.html)Cited by:[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p2.1),[§V\-B2](https://arxiv.org/html/2607.16085#S5.SS2.SSS2.p1.4)\.
- \[28\]D\. E\. Powers, J\. C\. Burstein, M\. Chodorow, M\. E\. Fowles, and K\. Kukich\(2002\)Stumping e\-rater: challenging the validity of automated essay scoring\.Computers in Human Behavior18\(2\),pp\. 103–134\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S0747563201000528)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1)\.
- \[29\]M\. Qian, K\. M\. Knill, S\. Bannò, S\. Tang, P\. Karanasou, M\. J\.F\. Gales, and D\. Nicholls\(2025\)Speak & Improve Challenge 2025\.In10th Workshop on Speech and Language Technology in Education \(SLaTE\),pp\. 41–45\.External Links:[Document](https://dx.doi.org/10.21437/SLaTE.2025-9),ISSN 2311\-4975Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p4.1)\.
- \[30\]Qwen Team\(2025\)Qwen2\.5 technical report\.Technical reportAlibaba Group\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§V\-B1](https://arxiv.org/html/2607.16085#S5.SS2.SSS1.p1.5)\.
- \[31\]V\. Raina, M\. J\.F\. Gales, and K\. M\. Knill\(2020\)Universal Adversarial Attacks on Spoken Language Assessment Systems\.InInterspeech 2020,pp\. 3855–3859\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2020-1890),ISSN 2958\-1796Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p4.1),[§II\-B](https://arxiv.org/html/2607.16085#S2.SS2.p2.1)\.
- \[32\]A\. S\. Ross, M\. C\. Hughes, and F\. Doshi\-Velez\(2017\)Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations\.InProc\. the Twenty\-Sixth International Joint Conference on Artificial Intelligence,pp\. 2662–2670\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2017/371),[Link](https://doi.org/10.24963/ijcai.2017/371)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p2.1)\.
- \[33\]X\. Shao, A\. Skryagin, W\. Stammer, P\. Schramowski, and K\. Kersting\(2021\)Right for Better Reasons: Training Differentiable Models by Constraining their Influence Functions\.\.InProc\. the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 9533–9540\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i11.17148),[Link](https://doi.org/10.1609/aaai.v35i11.17148)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p2.2)\.
- \[34\]M\. Sundararajan, A\. Taly, and Q\. Yan\(2017\)Axiomatic attribution for deep networks\.InInternational conference on machine learning,pp\. 3319–3328\.External Links:[Link](https://proceedings.mlr.press/v70/sundararajan17a.html)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p2.1)\.
- \[35\]Y\. Wang, M\. J\.F\. Gales, K\. Knill, K\. Kyriakopoulos, A\. Malinin, R\. C\. van Dalen, and M\. Rashid\(2018\)Towards automatic assessment of spontaneous spoken english\.Speech Communication104,pp\. 47–56\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S0167639317304545)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6),[§III\-A](https://arxiv.org/html/2607.16085#S3.SS1.p2.1)\.
- \[36\]Y\. Wang, R\. Hu, and Z\. Zhao\(2024\-11\)Beyond agreement: diagnosing the rationale alignment of automated essay scoring methods based on linguistically\-informed counterfactuals\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 8906–8925\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.520/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.520)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p2.1),[§I](https://arxiv.org/html/2607.16085#S1.p3.1),[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p1.1)\.
- \[37\]B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, G\. T\. Adams, J\. Howard, and I\. Poli\(2025\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.InProc\. the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2526–2547\.External Links:[Link](https://aclanthology.org/2025.acl-long.127/)Cited by:[§I](https://arxiv.org/html/2607.16085#S1.p1.1)\.
- \[38\]X\. Xi, D\. Higgins, K\. Zechner, and D\. M\. Williamson\(2008\)Automated scoring of spontaneous speech using SpeechRater v1\.0\.Technical reportTechnical ReportETS RR\-08\-62,Educational Testing Service,Princeton, NJ\.Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6)\.
- \[39\]J\. Xu, E\. Jones, V\. Laxton, and E\. Galaczi\(2021\)Assessing L2 English speaking using automated scoring technology: examining automarker reliability\.Assessment in Education: Principles, Policy & Practice28\(4\),pp\. 411–436\.External Links:[Document](https://dx.doi.org/10.1080/0969594X.2021.1979467)Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6)\.
- \[40\]K\. Zechner, D\. Higgins, X\. Xi, and D\. M\. Williamson\(2009\)Automatic scoring of non\-native spontaneous speech in tests of spoken english\.Speech communication51\(10\),pp\. 883–895\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S0167639309000703)Cited by:[§II\-A](https://arxiv.org/html/2607.16085#S2.SS1.p1.6)\.
- \[41\]L\. Zhao, Q\. Liu, L\. Yue, W\. Chen, L\. Chen, R\. Sun, and C\. Song\(2024\)COMI: correct and mitigate shortcut learning behavior in deep neural networks\.InProc\. the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’24,New York, NY, USA,pp\. 218–228\.External Links:ISBN 9798400704314,[Link](https://doi.org/10.1145/3626772.3657729),[Document](https://dx.doi.org/10.1145/3626772.3657729)Cited by:[§III\-B](https://arxiv.org/html/2607.16085#S3.SS2.p3.1)\.