Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning

# Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning
Source: [https://arxiv.org/html/2605.16991](https://arxiv.org/html/2605.16991)
[![[Uncaptioned image]](https://arxiv.org/html/2605.16991v1/x1.png)Jan Netík](https://orcid.org/0000-0002-3888-3203)Faculty of Education, Charles University, Prague, Czech RepublicInstitute of Computer Science of the Czech Academy of Sciences, Prague, Czech Republic[![[Uncaptioned image]](https://arxiv.org/html/2605.16991v1/x2.png)Patrícia Martinková](https://orcid.org/0000-0003-4754-8543)Faculty of Education, Charles University, Prague, Czech RepublicInstitute of Computer Science of the Czech Academy of Sciences, Prague, Czech Republic

###### Abstract

Response\-freeitem\-difficulty modellingpromises to reduce reliance on response\-based calibration but is intrinsically difficult on reading\-comprehension multiple\-choice items, where difficulty depends on inferential demands across wording components\. Whereas most existing approaches extract item\-text features and pass them to a separate statistical or machine\-learning model, we fine\-tune transformer encoders end\-to\-end on the item wording, eliminating the manual feature engineering and preprocessing that discards information\. Moreover, two extensions to this joint\-encoding approach are proposed: a component\-wise variant that encodes wording components separately through a shared encoder, and a multi\-task variant that retains joint encoding and adds an auxiliarymultiple\-choice question\-answeringobjective on the shared encoder\. Each method is evaluated under a Monte Carlo subsampling design at three training\-set sizes on a held\-out test set\. We find that joint encoding is a viable end\-to\-end alternative to feature\-engineering pipelines; while the component\-wise variant shows no detectable benefit, consistent with self\-attention already harvesting the cross\-component signal, the multi\-task variant delivers significant paired improvements in the smallest\-sample regime\. Transformer fine\-tuning, especially if regularised by a suitable auxiliary task, recovers a substantial share of the wording\-derivable signal at training\-set sizes typical of applied measurement\. The framework provides a customisable interface for psychometrically motivated extensions\.

Keywords:item difficulty modelling; transformers; fine\-tuning; multiple\-choice items; reading comprehension; Rasch model

## 1Introduction

Accurate estimation of item difficulty is a cornerstone of psychometric modelling, particularly in high\-stakes educational assessment\. Reliable difficulty estimates are essential for minimising measurement error at targeted ability levels, aligning the difficulty of multiple test forms during construction, and providing test\-takers with optimally selected items in computerized adaptive testing\. In many practical settings, however, conventional estimation from a sufficient number of response patterns is unavailable: newly written items may not yet have been administered, pre\-testing can expose secure items, the target population may be too small or difficult to recruit, and automatically generated item pools may need provisional difficulty estimates before response data are available\. Expert judgments can partly fill this gap, but they are not sufficiently reliable as a sole source of information\[[1](https://arxiv.org/html/2605.16991#bib.bib66),[23](https://arxiv.org/html/2605.16991#bib.bib307)\]\. This creates a need for methods that predict item difficulty directly from the item stimulus material, that is, from the wording available before administration\.

We refer to this pre\-administration setting as*response\-free*item\-difficulty modelling\(IDM\): item difficulty is inferred from the item’s wording without requiring response patterns\[[25](https://arxiv.org/html/2605.16991#bib.bib328)\]\. Three lines of work approach the task\[[18](https://arxiv.org/html/2605.16991#bib.bib380),[1](https://arxiv.org/html/2605.16991#bib.bib66), see\]\. The oldest manually defines and derives so\-calleditem text features\[[23](https://arxiv.org/html/2605.16991#bib.bib307), readability indices, syntactic depth, lexical complexity, or, as in\]and feeds them to a statistical or machine\-learning model, typically after demanding preprocessing\[[23](https://arxiv.org/html/2605.16991#bib.bib307),[8](https://arxiv.org/html/2605.16991#bib.bib137), stopword removal, lemmatisation;\]that discards a substantial morphosyntactic layer of the wording\. The pipeline is highly interpretable but replaces the wording’s raw content with a hand\-engineered feature representation, requiring substantial preprocessing and theory\-based feature\-design effort\. The second line, rooted in explanatoryitem response theory\(IRT\), integrates fixed item features directly into theIRTlikelihood: in the linear logistic test model\[[7](https://arxiv.org/html/2605.16991#bib.bib55), LLTM;\]the item text features are hand\-coded as in the first approach and their “weights” are estimated, while also leveraging the response\-patterns simultaneously\. Some works extended the same framework using*frozen*contextual representations from a pre\-trained transformer\[[26](https://arxiv.org/html/2605.16991#bib.bib333), specifically, the encoder\-only version called “Bidirectional Encoder Representations from Transformers”,BERT; for the original architecture see\]as feature vectors forcloze\-style111Theclozetask, introduced by\[[24](https://arxiv.org/html/2605.16991#bib.bib314)\]is, in effect, identical to themasked language modelling\(MLM\) objective on whichBidirectional Encoder Representations from Transformers\(BERT\)\-family encoders are pre\-trained\[[5](https://arxiv.org/html/2605.16991#bib.bib125)\]: a token is hidden in context and the system must recover it, exactly as a test\-taker would\.items\[[15](https://arxiv.org/html/2605.16991#bib.bib219),[30](https://arxiv.org/html/2605.16991#bib.bib57)\]; thelinear logistic test model\(LLTM\) specification is retained, but the hand\-coded features are replaced by transformer outputs\. A third line, which we pursue here, takes the natural next step: rather than freezing the encoder, we fine\-tune it end\-to\-end against the difficulty target, so the representations are adapted to theIDMtask rather than borrowed from a general\-purpose pre\-training objective\. The fine\-tuning recipe is well established in the broadernatural language processing\(NLP\) community\[[12](https://arxiv.org/html/2605.16991#bib.bib383),[5](https://arxiv.org/html/2605.16991#bib.bib125)\], and has been applied tomultiple\-choice question\(MCQ\)IDMin previous work\[[16](https://arxiv.org/html/2605.16991#bib.bib248)\]\.\[[2](https://arxiv.org/html/2605.16991#bib.bib89)\]reported an independent adaptation in a different setting \(short factoid\-style items: computer\-science trivia and K\-12 mathematics problems, with limited passage context\)\. The line has not been substantially extended for reading\-comprehensionMCQsince\.

Response\-freeIDMfor reading\-comprehensionMCQis intrinsically hard\. Difficulty depends on inferential demands across passage, question, and options jointly rather than on isolated lexical features, an empirical regularity that classical psychometric cognitive\-component models of reading\-comprehension items have characterised in detail\[[6](https://arxiv.org/html/2605.16991#bib.bib38),[11](https://arxiv.org/html/2605.16991#bib.bib39),[17](https://arxiv.org/html/2605.16991#bib.bib40)\]\. Transformer fine\-tuning without further modification thus leaves substantial headroom, unless one has access to large, high\-quality datasets with diverse items \(that are both content\-diverse and combinatorially “rich”, i\.e\., featuring multiple questions and options per passage\), along with precise difficulty estimates\[[16](https://arxiv.org/html/2605.16991#bib.bib248)\]\. The recent shared task on automated prediction of item difficulty and item response time\[[31](https://arxiv.org/html/2605.16991#bib.bib361)\]for medicalMCQillustrates the regime: despite substantial model complexity across submissions, the best entries surpassed the so\-called “dummy” regressor \(a model that predicts the mean difficulty of the training set for every item\) only marginally\. The same small\-sample pressure motivates recent work that integrates response\-free predictions with response\-based calibration as informative priors to reduce pilot\-sample requirements\[[25](https://arxiv.org/html/2605.16991#bib.bib328)\]; the response\-free predictions themselves remain a load\-bearing input to that pipeline\.

To the best of our knowledge, no published study has examined if and how different fine\-tuning architectures can help in theIDMtask on reading\-comprehensionMCQs\. Two questions remain open: \(i\) whether items are best encoded jointly, as a flat passage\-question\-options sequence, or component\-wise; and \(ii\) whether the difficulty\-regression target alone is enough, or whether adding an auxiliary task that more closely follows the process underlying test\-taker behaviour \(the very mechanism that predominantly generates the difficulty\) changes the picture, particularly in the small\-sample regime where the labelled difficulty signal is sparse\. To our knowledge, this study is also the first to examine the effect of the training set size using a large, experimental dataset\.

To address these questions, we evaluate three transformer\-based methods under a nested training\-size design\. The aim is not to maximise predictive performance, which would call for domain\-specific pre\-training and extensive hyper\-parameter tuning, but to analyse whether transformer\-based models are a viable approach for reading\-comprehension multiple\-choice items, and where further structural and supervisory inductive biases pay off when training data are scarce\. A joint\-encoding approach processes the concatenated item wording as a single input, collapsing the preprocessing and modelling stages of the feature\-engineering tradition into one\. A component\-wise variant encodes the passage, question, and option set separately through a shared encoder before aggregation, making explicit at the architectural level the aforementioned functional decomposition of aMCQ\. A multi\-task variant retains joint encoding and adds an auxiliarymultiple\-choice question\-answering\(MCQA\) objective sharing the encoder with the regression task; the auxiliary task forces option\-level discrimination into the shared representation\. Both extensions are motivated by the same concern: In the small\-sample regime typical of applied psychometric work, transformer fine\-tuning has limited information from which to recover task\-relevant representations, and additional inductive biases \(structural at the input level for component\-wise encoding, supervisory at the loss level for multi\-task learning\) can act as forms ofregularisationthat the regression objective on a scarce target alone cannot supply\[[9](https://arxiv.org/html/2605.16991#bib.bib382), or may not fully exploit,\]\.

The remainder of the article proceeds as follows\. Section[2](https://arxiv.org/html/2605.16991#S2)describes the data and pseudo\-labelling procedure, defines the transformer\-based approach with joint\-encoding and the two encoding alternatives, and specifies the training and evaluation protocol\. Section[3](https://arxiv.org/html/2605.16991#S3)reports the empirical comparison across three training\-set sizes\. Section[4](https://arxiv.org/html/2605.16991#S4)interprets the findings and discusses the limitations\. Section[5](https://arxiv.org/html/2605.16991#S5)concludes the paper\.

## 2Materials and methods

### 2\.1Datasets

#### Source corpus and pseudo\-labelling\.

We work with the RACE\+\+ collection\[\[, reading\-comprehension multiple\-choice items sourced from English\-as\-a\-foreign\-language exams in China;\]\]liang2019, balanced across the three source school types \(middle, high, college\) and across passages so that no school type or single passage is over\-represented \(full balancing recipe in supplementary materials\), yielding a corpus of29,61829\{,\}618items\. We pseudo\-label every item by few\-shot prompting of a large language model\[\[, Gemini 2\.5 Flash;\]\]comanici2025 using PALRACE\[[33](https://arxiv.org/html/2605.16991#bib.bib372)\], a human\-administered subset of RACE\+\+ with Rasch difficulty estimated from response patterns, as the source of in\-context anchors\. On a held\-out pilot drawn from PALRACE itself, thelarge language model\(LLM\)’s predictions correlate atr=0\.46r=0\.46with response\-based Rasch difficulty; full prompt construction, anchor sampling, and pilot\-evaluation details are in supplementary materials\. PALRACE is used as the calibration anchor for the pseudo\-labelling pipeline only, not as a downstream evaluation target \(because of its size\); we return to the implications of pseudo\-labelled targets in Section[4](https://arxiv.org/html/2605.16991#S4)\.

#### Train / validation / test partition\.

The pseudo\-labelled corpus is partitioned at the*passage*level into a training set𝒟train\\mathcal\{D\}\_\{\\text\{train\}\}of23,13623\{,\}136items, a fixed validation set𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}ofnval=2,023n\_\{\\text\{val\}\}=2\{,\}023items, and a held\-out test set𝒟test\\mathcal\{D\}\_\{\\text\{test\}\}of4,4594\{,\}459items; passage\-level disjointness eliminates leakage through shared context\. The partition is stratified on the cross of school type and the modal Rasch quintile of each passage’s items\.

#### Training sample sizes\.

From the training set𝒟train\\mathcal\{D\}\_\{\\text\{train\}\}we draw nested sub\-samples that vary training sample size by Monte Carlo subsampling\. We consider three sizes,n≈800n\\approx 800,n≈4,300n\\approx 4\{,\}300, andn≈23,000n\\approx 23\{,\}000, spaced roughly logarithmically; the largest equals𝒟train\\mathcal\{D\}\_\{\\text\{train\}\}exactly\. For each sizennand each of ten sample seedss∈\{42,…,51\}s\\in\\\{42,\\ldots,51\\\}we draw a sub\-sample𝒯n,s⊂𝒟train\\mathcal\{T\}\_\{n,s\}\\subset\\mathcal\{D\}\_\{\\text\{train\}\}as a passage\-level prefix of a per\-seed shuffle, stratified on the same factors as the corpus partition\. The same shuffle is reused across the three sizes, so that for each fixed seed, the sub\-samples are nested,

𝒯n,s⊂𝒯n′,sfor​n<n′,\\mathcal\{T\}\_\{n,s\}\\,\\subset\\,\\mathcal\{T\}\_\{n^\{\\prime\},s\}\\quad\\text\{for \}n<n^\{\\prime\},\(1\)while across\-seed draws are independent\.𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}and𝒟test\\mathcal\{D\}\_\{\\text\{test\}\}are fixed across all cells, isolating the effect of training\-set size from sampling noise\.

### 2\.2General modelling framework

The encoder\-only family of transformers\[[5](https://arxiv.org/html/2605.16991#bib.bib125),BERTand its successors;\]consists of stacks of self\-attention layers\[[26](https://arxiv.org/html/2605.16991#bib.bib333)\]pre\-trained on a general\-purpose corpus byMLM\[[13](https://arxiv.org/html/2605.16991#bib.bib169), for the linguistic regularities encoded by such pre\-training, see\]\. The*encoder\-only*qualifier distinguishes this family from the autoregressive*decoder\-only*models used for free\-form text generation \(such as theLLMemployed in Section[2\.1](https://arxiv.org/html/2605.16991#S2.SS1)for pseudo\-labelling\): both rely on the same self\-attention mechanism but differ in pre\-training objective and use case, with encoder\-only models optimised to output representations suitable for downstream prediction tasks rather than to produce text\. For fine\-tuning to a particular task, theMLMportion of the pre\-trained model can be removed to form a “body” and any small task\-specific “head” appended\. In our case, the head is denotedgψg\_\{\\psi\}and maps a pooled representation of the text input to the item difficulty; each of the three methods in this paper instantiatesgψg\_\{\\psi\}as a single linear layer, and the methods differ in the input given togψg\_\{\\psi\}\(Sections[2\.3](https://arxiv.org/html/2605.16991#S2.SS3)–[2\.5](https://arxiv.org/html/2605.16991#S2.SS5)\) and in whether an auxiliary head and auxiliary objective are present alongside the regression \(Section[2\.5](https://arxiv.org/html/2605.16991#S2.SS5)\)\.

The general setup is as follows: A pre\-trained encoder\-only transformer, denotedEncϕ⁡\(⋅\)\\operatorname\{Enc\}\_\{\\phi\}\(\\cdot\), maps an input token sequence𝐱i\\mathbf\{x\}\_\{i\}\(not necessarily equal in length to the number of words, see below\) to its last\-layer hidden states,

𝐡i=Encϕ⁡\(𝐱i\)∈ℝL×d,\\mathbf\{h\}\_\{i\}\\,=\\,\\operatorname\{Enc\}\_\{\\phi\}\(\\mathbf\{x\}\_\{i\}\)\\,\\in\\,\\mathbb\{R\}^\{L\\times d\},whereLLis the input length,ddis the encoder hidden size \(768 for theModernBERT\-basemodel that we use here;\[[28](https://arxiv.org/html/2605.16991#bib.bib344)\]\), andϕ\\phicollects the encoder’s trainable parameters \(149 million for ModernBERT\)\. The tokenizer \(i\.e\., the algorithm that splits the input into sub\-word tokens\) prepends a special\[CLS\]token introduced inBERT\[[5](https://arxiv.org/html/2605.16991#bib.bib125)\]as the first token of every input, which is pre\-trained to summarise the sequence via a pooled representation; we write𝐡\[CLS\]∈ℝd\\mathbf\{h\}\_\{\[\\text\{CLS\}\]\}\\in\\mathbb\{R\}^\{d\}for its last\-layer hidden state and use it as the pooled sequence representation that the headgψg\_\{\\psi\}receives,

𝐫i=\(𝐡i\)\[CLS\]∈ℝd\.\\mathbf\{r\}\_\{i\}\\,=\\,\(\\mathbf\{h\}\_\{i\}\)\_\{\[\\text\{CLS\}\]\}\\,\\in\\,\\mathbb\{R\}^\{d\}\.
The encoder parametersϕ\\phiand the head parametersψ\\psiare trainedjointlyby gradient descent using the AdamW optimiser\[[14](https://arxiv.org/html/2605.16991#bib.bib31)\], as implemented in the training loop of thetransformersPython framework\[[29](https://arxiv.org/html/2605.16991#bib.bib381)\], by minimising a task\-specific lossℒ\\mathcal\{L\}defined separately for each of the three methods \(Sections[2\.3](https://arxiv.org/html/2605.16991#S2.SS3)–[2\.5](https://arxiv.org/html/2605.16991#S2.SS5)\)\. Optimization hyper\-parameters are held fixed across all methods and training sizes; we defer their values to Section[2\.6](https://arxiv.org/html/2605.16991#S2.SS6)so that all training\-time choices are gathered in one place\.

Throughout this paper,bib\_\{i\}denotes the difficulty of itemiiin the Rasch model\[[19](https://arxiv.org/html/2605.16991#bib.bib272)\]\. In our case,bib\_\{i\}is anLLMpseudo\-label based on difficulty estimates from human response patterns to RACE\+\+ items included in the PALRACE dataset \(see Section[2\.1](https://arxiv.org/html/2605.16991#S2.SS1)for further details\)\. The model’s text\-based prediction is denotedb^i\\hat\{b\}\_\{i\}\.

### 2\.3Joint\-encoding

The joint\-encoding approach follows the general transfer learning phenomenon and fine\-tuning procedure elaborated in theNLPfield by\[[12](https://arxiv.org/html/2605.16991#bib.bib383)\], and serves as the reference point against which the two methodological extensions are compared, along with the “dummy” regressor baseline\. The complete item wording is \(i\) concatenated and \(ii\) tokenized into a single input sequence𝐱i\\mathbf\{x\}\_\{i\}consisting of the passage, question, and all options of itemii, with the special\[SEP\]tokens introduced inBERT\[[5](https://arxiv.org/html/2605.16991#bib.bib125)\]marking the boundaries between individual components and the\[CLS\]token prepended at the beginning of the sequence\. The encoder produces the pooled representation𝐫i=\(Encϕ⁡\(𝐱i\)\)\[CLS\]\\mathbf\{r\}\_\{i\}=\(\\operatorname\{Enc\}\_\{\\phi\}\(\\mathbf\{x\}\_\{i\}\)\)\_\{\[\\text\{CLS\}\]\}defined in Section[2\.2](https://arxiv.org/html/2605.16991#S2.SS2), and a linear layer \(the head equivalent to multiple linear regression\) maps it directly to the predicted difficulty,

b^i=gψ​\(𝐫i\)=𝐰⊤​𝐫i\+c,\\hat\{b\}\_\{i\}\\,=\\,g\_\{\\psi\}\(\\mathbf\{r\}\_\{i\}\)\\,=\\,\\mathbf\{w\}^\{\\top\}\\mathbf\{r\}\_\{i\}\+c,\(2\)where𝐰∈ℝd\\mathbf\{w\}\\in\\mathbb\{R\}^\{d\}is the head’s weight vector andc∈ℝc\\in\\mathbb\{R\}its scalar bias, both learned from data alongside the encoder parametersϕ\\phi\(which are not initialisedde novobut retained from the intensive pre\-training\)\. In this form, the pooled encoder output𝐫i\\mathbf\{r\}\_\{i\}can be understood as a learned feature vector of the item and𝐰\\mathbf\{w\}as the corresponding feature weights, both optimised from data rather than specified by the researcher\. Equation \([2](https://arxiv.org/html/2605.16991#S2.E2)\) has the same linear form as theLLTMreferenced in the Introduction, but with one key difference – the feature vector𝐫i\\mathbf\{r\}\_\{i\}is not fixed; the encoder parametersϕ\\phiare optimised jointly with𝐰\\mathbf\{w\}, so that both the features and their weights are learned from data\.

The joint\-encoding model is trained by minimising themean\-squared error\(MSE\) between the predicted and target difficulties,

ℒreg=1N​∑i=1N\(b^i−bi\)2\.\\mathcal\{L\}\_\{\\text\{reg\}\}\\,=\\,\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\bigl\(\\hat\{b\}\_\{i\}\-b\_\{i\}\\bigr\)^\{2\}\.\(3\)The sameMSEloss is used by the component\-wise variant in Section[2\.4](https://arxiv.org/html/2605.16991#S2.SS4); the multi\-task variant in Section[2\.5](https://arxiv.org/html/2605.16991#S2.SS5)retains it and adds a second loss term for the auxiliary task\.

The following Sections[2\.4](https://arxiv.org/html/2605.16991#S2.SS4)and[2\.5](https://arxiv.org/html/2605.16991#S2.SS5)describe two extensions of this approach: component\-wise encoding, which modifies the input toEncϕ\\operatorname\{Enc\}\_\{\\phi\}so that item components are processed separately before aggregation, andmulti\-task learning\(MTL\), which adds an auxiliary objective alongside theMSEfor the regression task\.

passage \[SEP\]question \[SEP\]optionsEncϕ\\operatorname\{Enc\}\_\{\\phi\}𝐫i\\mathbf\{r\}\_\{i\}regressiongψg\_\{\\psi\}b^i\\hat\{b\}\_\{i\}

\(a\)Joint encoding
questionpassageoptionssharedEncϕ\\operatorname\{Enc\}\_\{\\phi\}𝐫i\(1\)\\mathbf\{r\}\_\{i\}^\{\(1\)\}𝐫i\(2\)\\mathbf\{r\}\_\{i\}^\{\(2\)\}𝐫i\(3\)\\mathbf\{r\}\_\{i\}^\{\(3\)\}concatregressiongψg\_\{\\psi\}b^i\\hat\{b\}\_\{i\}

\(b\)Component\-wise encoding
passage \[SEP\]question \[SEP\]options\[passage;question;option\-​m\]\\bigl\[\\text\{passage\}\\,;\\,\\text\{question\}\\,;\\,\\text\{option\-\}m\\bigr\]sharedEncϕ\\operatorname\{Enc\}\_\{\\phi\}\+𝐳t\\mathbf\{z\}\_\{t\}at\[CLS\]𝐫i\\mathbf\{r\}\_\{i\}𝐫i\(optm\)\\mathbf\{r\}\_\{i\}^\{\(\\text\{opt\}\_\{m\}\)\}reg\.gψg\_\{\\psi\}clf\. withfξf\_\{\\xi\}b^i\\hat\{b\}\_\{i\}pi,mp\_\{i,m\}tasks alternated per batch\(random scheduler\)

\(c\)Multi\-task learning

Figure 1:Three response\-free item\-difficulty models compared in this work\. \(a\) The*joint\-encoding*approach encodes the concatenated wording and predictsb^i\\hat\{b\}\_\{i\}from the\[CLS\]token hidden state\. \(b\) The*component\-wise*variant encodes passage, question, and options separately through the shared encoderEncϕ\\operatorname\{Enc\}\_\{\\phi\}and concatenates the three\[CLS\]vectors before regression\. \(c\) The*multi\-task*model retains joint encoding and adds aMCQAclassification branch that shares the encoder with the regression branch; the regression head additionally drives the predictionb^i\\hat\{b\}\_\{i\}\. A learnable task\-conditioning vector𝐳t\\mathbf\{z\}\_\{t\}is added at the\[CLS\]position of the embedding\-layer output \(see Figure[2](https://arxiv.org/html/2605.16991#S2.F2)\)\. The two branches alternate per batch under a random task scheduler; at each step, only the branch chosen for that batch is active, so the regression branch producesb^i\\hat\{b\}\_\{i\}on regression steps and the MCQA branch predicts the probability that optionmmfor itemiiis correct:pi,m=exp⁡\(zi,m\)/∑k=1Mexp⁡\(zi,k\)p\_\{i,m\}=\\exp\(z\_\{i,m\}\)/\\sum\_\{k=1\}^\{M\}\\exp\(z\_\{i,k\}\)on MCQA steps\.
### 2\.4Component\-wise encoding

AnMCQhas a natural functional decomposition into passage, question, and option set\[[6](https://arxiv.org/html/2605.16991#bib.bib38)\], and a principled way to make this structure explicit at the architectural level is to encode each component separately through a shared encoder and aggregate the resulting representations before regression\. In the joint\-encoding approach, standard self\-attention treats all input tokens symmetrically, so no architectural signal distinguishes which tokens belong to which functional component, and the model must learn this segmentation implicitly from formatting cues such as option markers\[[20](https://arxiv.org/html/2605.16991#bib.bib49), for how self\-attention heads acquire syntactic and structural regularities during pre\-training, see\]\.

A second motivation is that the difficulty\-relevant properties of distractors are fundamentally relational: a distractor is attractive not because of its surface form in isolation, but because of its semantic proximity to the correct answer or its plausibility given the question\. A single\-vector pooling of a flat input asks the encoder to preserve such relational features alongside all other item characteristics through one bottleneck, a demand that grows with item length\. By encoding components separately and combining them later, the component\-wise variant retains per\-component representations and gives the regression head access to a richer, more structured input\.

Instead of processing the entire item as a flat sequence, the component\-wise variant encodes each functional component of the item separately\. Let𝐱i\(j\)\\mathbf\{x\}\_\{i\}^\{\(j\)\}denote the token sequence of thejj\-th component of itemii, withj=1,…,Cj=1,\\ldots,Cindexing the passage, the question, and the options\. We adoptC=3C=3and join the four answer options into a single labelled string for the third component \(withA\),B\),C\), andD\)prepended to each option – as in the case of the joint\-encoding approach above\)\.

Each component is passed through the shared encoder, yielding last\-layer hidden states𝐡i\(j\)=Encϕ⁡\(𝐱i\(j\)\)\\mathbf\{h\}\_\{i\}^\{\(j\)\}=\\operatorname\{Enc\}\_\{\\phi\}\(\\mathbf\{x\}\_\{i\}^\{\(j\)\}\), and is represented by its\[CLS\]hidden state, in direct analogy with the joint\-encoding construction of𝐫i\\mathbf\{r\}\_\{i\}:

𝐫i\(j\)=\(𝐡i\(j\)\)\[CLS\]∈ℝd,j=1,…,C\.\\mathbf\{r\}\_\{i\}^\{\(j\)\}\\,=\\,\\bigl\(\\mathbf\{h\}\_\{i\}^\{\(j\)\}\\bigr\)\_\{\[\\text\{CLS\}\]\}\\,\\in\\,\\mathbb\{R\}^\{d\},\\qquad j=1,\\ldots,C\.\(4\)The encoder parametersϕ\\phiare shared across components: the same weights process the passage, the question, and the options\. The three component representations are concatenated along the feature axis and fed to a linear regression head:

b^i=gψ​\(\[𝐫i\(1\);…;𝐫i\(C\)\]\),gψ:ℝC⋅d→ℝ,\\hat\{b\}\_\{i\}\\,=\\,g\_\{\\psi\}\\\!\\Bigl\(\\bigl\[\\mathbf\{r\}\_\{i\}^\{\(1\)\};\\;\\ldots\\;;\\;\\mathbf\{r\}\_\{i\}^\{\(C\)\}\\bigr\]\\Bigr\),\\qquad g\_\{\\psi\}\\colon\\mathbb\{R\}^\{C\\cdot d\}\\to\\mathbb\{R\},\(5\)where\[⋅;⋅\]\[\\,\\cdot\\,;\\,\\cdot\\,\]denotes concatenation\. The training objective is unchanged from the joint\-encoding version,ℒreg\\mathcal\{L\}\_\{\\text\{reg\}\}in Equation \([3](https://arxiv.org/html/2605.16991#S2.E3)\)\.

In preliminary work, we also evaluated mean pooling \(averaging the encoder’s hidden states across all tokens of a component\) and mean aggregation \(averaging the three component vectors instead of concatenating them\)\. Neither alternative improved the presented variant substantially on the samples used in this study; we therefore report only the variant defined above\.

### 2\.5Multi\-task learning

TheMTLextension addresses a complementary gap\. Even with component\-wise encoding, the model is trained solely to minimise prediction error on a scalar difficulty target, which provides no signal about*why*an item is easy or hard\. Adding an auxiliary question\-answering objective forces the encoder to develop representations that enable option\-level discrimination, that is, representations from which the correct answer can be distinguished from distractors\. To form this representation, the model must naturally “attend” to the wording in a way that reflects the passage, question, and each option that is necessary to choose the key\. In the terminology of\[[21](https://arxiv.org/html/2605.16991#bib.bib50)\]\(building on the original formulation of\[[3](https://arxiv.org/html/2605.16991#bib.bib111)\]\), the auxiliary task provides aninductive biasthat causes the model to prefer representations useful for more than one objective, acting as a form ofregularisationon theshared encoderthat is most consequential when the primary regression signal is sparse or offers surface\-level shortcuts \(such as attending only to the passage and ignoring the rest of the wording\), that is, in the small\-sample regime\. Here, the bias is also substantively and psychologically motivated: to solve the item, a test\-taker must evaluate each option against the question and passage, and theMTLobjective encourages the model to perform an analogous computation\. In this sense, the auxiliary task is not only a regulariser – it is an alignment mechanism that steers the learned features toward theprocess that generates difficultyin the first place\.

For theMCQAtask, a custom input stemming from the same item is presentedMMtimes, once per answer optionmm\(form=1,…,Mm=1,\\ldots,M; withM=4M=4in our case\)\. For each option, the shared context\[\[CLS\];passage;question;\[SEP\]\]\[\\text\{\[CLS\]\};\\text\{passage\};\\text\{question\};\\text\{\[SEP\]\}\]is concatenated withoptionm\\text\{option\}\_\{m\}to form the input token sequence𝐱i\(optm\)\\mathbf\{x\}\_\{i\}^\{\(\\text\{opt\}\_\{m\}\)\}, and its pooled representation𝐫i\(optm\)=\(Encϕ⁡\(𝐱i\(optm\)\)\)\[CLS\]\\mathbf\{r\}\_\{i\}^\{\(\\text\{opt\}\_\{m\}\)\}=\\bigl\(\\operatorname\{Enc\}\_\{\\phi\}\(\\mathbf\{x\}\_\{i\}^\{\(\\text\{opt\}\_\{m\}\)\}\)\\bigr\)\_\{\[\\text\{CLS\}\]\}is the\[CLS\]hidden state of the encoder’s output for that input\. Each pooled representation is passed through a classification headfξf\_\{\\xi\}that produces a scalar “logit”\. The functionfξf\_\{\\xi\}has the same form asgψg\_\{\\psi\}of Equation \([2](https://arxiv.org/html/2605.16991#S2.E2)\), with its own parameters collected underξ=\{𝐮∈ℝd,e∈ℝ\}\\xi=\\\{\\mathbf\{u\}\\in\\mathbb\{R\}^\{d\},\\,e\\in\\mathbb\{R\}\\\}, shared across allMMoptions \(input sequences\):

zi,m=fξ​\(𝐫i\(optm\)\)=𝐮⊤​𝐫i\(optm\)\+efor​m=1,…,M\.z\_\{i,m\}\\,=\\,f\_\{\\xi\}\\\!\\bigl\(\\mathbf\{r\}\_\{i\}^\{\(\\text\{opt\}\_\{m\}\)\}\\bigr\)\\,=\\,\\mathbf\{u\}^\{\\top\}\\mathbf\{r\}\_\{i\}^\{\(\\text\{opt\}\_\{m\}\)\}\+e\\qquad\\text\{for \}m=1,\\ldots,M\\,\.\(6\)What distinguishes the two heads is the task they serve and the loss they are paired with:gψg\_\{\\psi\}outputs a scalar prediction trained under theMSEloss \(Equation[3](https://arxiv.org/html/2605.16991#S2.E3)\);fξf\_\{\\xi\}outputs per\-option “logits”zi,mz\_\{i,m\}and its parametersξ\\xiare trained using the cross\-entropy loss defined below \(Equation[7](https://arxiv.org/html/2605.16991#S2.E7)\)\. Keepingfξf\_\{\\xi\}as a single linear layer devotes the discriminative capacity required to distinguish correct from incorrect options to the encoder representations themselves, which is the reason we employ this auxiliary task in the first place – to constrain the representations for the mainIDMgoal\.

TheMCQAloss is calculated using standard cross\-entropy, where the model’s predicted raw logits over theMMoptions are passed through a softmax function \(in machine\-learning terminology, equivalent to multinomial logistic regression\), and the negative log\-likelihood is computed with respect to the integer indexmi∗m\_\{i\}^\{\*\}of the correct option:

ℒMCQA=−1N​∑i=1Nlog⁡exp⁡\(zi,mi∗\)∑m=1Mexp⁡\(zi,m\)\.\\mathcal\{L\}\_\{\\text\{MCQA\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{\\exp\(z\_\{i,m^\{\*\}\_\{i\}\}\)\}\{\\sum\_\{m=1\}^\{M\}\\exp\(z\_\{i,m\}\)\}\\,\.\(7\)
Trainingalternatesbetween the two tasks: the data loader interleaves equal numbers of batches per task, so that over a full training epoch each task contributes the same number of optimiser steps; the shared encoder parametersϕ\\phiare updated at every step\. The regression head parametersψ\\psireceive “gradients” only fromℒreg\\mathcal\{L\}\_\{\\text\{reg\}\}, and theMCQAhead parametersξ\\xionly fromℒMCQA\\mathcal\{L\}\_\{\\text\{MCQA\}\}, while the shared encoderϕ\\phiis updated by whichever task is active in a given step\. We do not form a joint loss per batch\.

A practical concern inMTLis so\-called*negative transfer*\[[27](https://arxiv.org/html/2605.16991#bib.bib376)\], where the gradients of one task interfere destructively with those of the other and degrade the shared representation rather than enrich it\. Two design choices in our setup seek to mitigate this concern\. First, we weight theMCQAloss to keep it from dominating the regression loss on the shared encoder: the regressionMSEand the cross\-entropy may not be balanced in terms of scale, and a plain sum can let the auxiliary task dictate the updates\. We multiply theMCQAloss by a constantλ=0\.05\\lambda=0\.05\[[10](https://arxiv.org/html/2605.16991#bib.bib378)\], selected by grid search over\{0\.005,0\.05,0\.5,1\}\\\{0\.005,0\.05,0\.5,1\\\}on a separaten≈800n\\approx 800sample from the training set that is not used in any sub\-sample in this study\. Because our implementation employs a stochastic task scheduler rather than computing a joint loss per batch, the gradient𝐠s\\mathbf\{g\}\_\{s\}backpropagated to the shared encoder at stepssis either∇ℒreg\\nabla\\mathcal\{L\}\_\{\\text\{reg\}\}orλ​∇ℒMCQA\\lambda\\nabla\\mathcal\{L\}\_\{\\text\{MCQA\}\}\. As tasks are sampled with equal probability, the expected gradient update is:

𝔼​\[𝐠s\]=12​\(∇ℒreg\+λ​∇ℒMCQA\)\\mathbb\{E\}\[\\mathbf\{g\}\_\{s\}\]=\\frac\{1\}\{2\}\\bigl\(\\nabla\\mathcal\{L\}\_\{\\text\{reg\}\}\+\\lambda\\nabla\\mathcal\{L\}\_\{\\text\{MCQA\}\}\\bigr\)
Second, to give the shared encoder an explicit signal about which task is being processed at each step, we introduce a learnable task\-conditioning matrix𝐙∈ℝT×d\\mathbf\{Z\}\\in\\mathbb\{R\}^\{T\\times d\}, following the additive input\-level conditioning scheme used by\[[22](https://arxiv.org/html/2605.16991#bib.bib379)\], in this case only withT=2T=2rows \(one per task\)\. The encoder hidden size isdd\. During the forward pass for the active taskt∈\{regression,MCQA\}t\\in\\\{\\text\{regression\},\\text\{MCQA\}\\\}, the row𝐳t∈ℝd\\mathbf\{z\}\_\{t\}\\in\\mathbb\{R\}^\{d\}is added to the embedding\-layer output at the\[CLS\]position before the first encoder layer reads the representation,

𝐞~i,\[CLS\]=𝐞i,\[CLS\]\+𝐳t,\\tilde\{\\mathbf\{e\}\}\_\{i,\\,\[\\text\{CLS\}\]\}\\,=\\,\\mathbf\{e\}\_\{i,\\,\[\\text\{CLS\}\]\}\+\\mathbf\{z\}\_\{t\},\(8\)where𝐞i,\[CLS\]\\mathbf\{e\}\_\{i,\\,\[\\text\{CLS\}\]\}is the pre\-conditioning embedding of the\[CLS\]token\.𝐙\\mathbf\{Z\}is initialised to zero, so training begins without any conditioning; the rows acquire non\-zero values only to the extent that the shared encoder benefits from a task\-specific adjustment\. Figure[2](https://arxiv.org/html/2605.16991#S2.F2)illustrates the placement of𝐳t\\mathbf\{z\}\_\{t\}\.

embedding layer\+𝐳t\+\\,\\mathbf\{z\}\_\{t\}at\[CLS\]1stencoder layer𝐙∈ℝT×d\\mathbf\{Z\}\\in\\mathbb\{R\}^\{T\\times d\}select rowttFigure 2:Task\-conditioning vector𝐳t\\mathbf\{z\}\_\{t\}in the multi\-task model\. The matrix𝐙∈ℝT×d\\mathbf\{Z\}\\in\\mathbb\{R\}^\{T\\times d\}stores one learnable vector per task \(T=2T=2\)\. During a forward pass, the active task selects row𝐳t\\mathbf\{z\}\_\{t\}, which is added only at the\[CLS\]position of the embedding\-layer output before the first encoder layer\. Both rows of𝐙\\mathbf\{Z\}are initialised to zero and trained jointly with the rest of the model\.
### 2\.6Training procedure and evaluation

A single fine\-tuning regime is applied to every \(training sample size×\\timesmethod×\\timessub\-sample\) cell of the experimental design, so that within\-size differences between methods reflect the architectural choice rather than hyper\-parameter selection \(although we recognise that a specific hyper\-parameter setup may further improve the performance of each method\)\. We optimise with AdamW\[[14](https://arxiv.org/html/2605.16991#bib.bib31)\]under a fixed schedule and mini\-batch configuration \(exact values in supplementary materials\), stop training early based on the validationroot mean square error\(RMSE\), and restore the best\-on\-validation checkpoint for the final test evaluation\. The held\-out test split𝒟test\\mathcal\{D\}\_\{\\text\{test\}\}is consulted only once per cell, after training has finished\. Final metrics areRMSE\(primary\), Spearmanρ\\rho, andR2R^\{2\}; atn≈800n\\approx 800andn≈4,300n\\approx 4\{,\}300we report mean and standard deviation across the ten sample seeds \(Section[2\.1](https://arxiv.org/html/2605.16991#S2.SS1)\) to expose the dependence of each metric on the training sub\-sample drawn, and a single value atn≈23,000n\\approx 23\{,\}000\(as we cannot subsample the whole training set in any way\)\. A*constant*“dummy” regressor that returns the training\-set mean ofbib\_\{i\}for every test item is evaluated at every cell as a model\-free reference, following the convention of\[[31](https://arxiv.org/html/2605.16991#bib.bib361)\]\. The dataset is publicly available atREDACTED FOR PEER REVIEW\. The replication code, detailed dataset preparation description, hyper\-parameters, and other technical specifics are provided on the Open Science Framework repository atREDACTED FOR PEER REVIEW\.

## 3Results

We present the results in two tables and three figures\. Table[1](https://arxiv.org/html/2605.16991#S3.T1)reports absolute test\-set performance per training\-set size and method\. Table[2](https://arxiv.org/html/2605.16991#S3.T2)reports paired two\-sided Wilcoxon signed\-rank tests of within\-subsample differences against joint\-encoding, across the twelve cells defined by the two comparator methods, two training sizes, and three metrics\.

Three results stand out\. First, atn≈800n\\approx 800MTLimproves on joint\-encoding across all three metrics, with 95% confidence intervals on the Hodges–Lehmann median shift excluding zero and 9 of 10 sub\-samples favouringMTLper each metric\. Second, the improvement is no longer detectable atn≈4,300n\\approx 4\{,\}300\(all CIs straddle zero, 5–6 of 10 sub\-samples favourMTL\), consistent with the design hypothesis that the auxiliary objective regularises most when the labelled difficulty signal is sparse\. Third, the component\-wise variant performs*worse*than joint\-encoding onRMSEandR2R^\{2\}at both training sizes \(CIs exclude zero\), while leaving the Spearman correlation between predicted and true difficulty statistically indistinguishable from joint\-encoding\.

Table 1:Test\-set performance per training\-set size and methodNote\.Values are mean±\\pmSD across ten random sample seeds atn≈800n\\approx 800andn≈4,300n\\approx 4\{,\}300; a single run on the full training set atn≈23,000n\\approx 23\{,\}000\. The dummy reg\. baseline \(Section[2\.6](https://arxiv.org/html/2605.16991#S2.SS6)\) returns the training\-set mean ofbib\_\{i\}for every test item; its Spearmanρ\\rhois undefined because its predictions have zero variance\. The best entry per metric within each training\-set size is set in bold\.

Table 2:Paired Wilcoxon signed\-rank tests of within\-sub\-sample differences against joint\-encodingNote\.Δ^HL\\hat\{\\Delta\}\_\{\\text\{HL\}\}is the Hodges–Lehmann estimate of the within\-sub\-sample paired difference between the method and joint\-encoding, expressed in the metric’s own units\.VVis the signed\-rank test statistic\.ppis the exact two\-sidedpp\-value\. The favourable column reports the number of sub\-samples in which the method outperforms joint\-encoding on the row’s metric\. ForRMSE, lower values are better, so favourable meansΔ^HL<0\\hat\{\\Delta\}\_\{\\text\{HL\}\}<0; forR2R^\{2\}and Spearmanρ\\rho, higher is better and favourable meansΔ^HL\>0\\hat\{\\Delta\}\_\{\\text\{HL\}\}\>0\.

Figures[3](https://arxiv.org/html/2605.16991#S3.F3)–[5](https://arxiv.org/html/2605.16991#S3.F5)visualize the paired differences between a given method and the joint\-encoding approach on one of the three reported metrics\. To make the gain in absolute performance across training sizes visually apparent, panels are vertically shifted by a constant per panel: for RMSE andR2R^\{2\}, panels are aligned so that the dummy regressor mean sits at the same vertical position across sizes, reflecting the fact that the dummy is insensitive to training set size and thus provides a stable anchor; for Spearmanρ\\rho, where the dummy correlation is undefined, panels are instead aligned at the mean absolute performance of the joint\-encoding approach across sizes\.

![Refer to caption](https://arxiv.org/html/2605.16991v1/x3.png)Figure 3:Paired differences inRMSEbetween each comparator method \(component\-wise,MTL, and the dummy regressor\) and the joint\-encoding approach, computed within the same training sub\-sample\. Panels are aligned so that the dummy regressor mean sits at the same vertical position across sizes\.![Refer to caption](https://arxiv.org/html/2605.16991v1/x4.png)Figure 4:Paired differences inR2R^\{2\}between each comparator method \(component\-wise,MTL, and the dummy regressor\) and the joint\-encoding approach, computed within the same training sub\-sample\. Panels are aligned so that the dummy regressor mean sits at the same vertical position across sizes\.![Refer to caption](https://arxiv.org/html/2605.16991v1/x5.png)Figure 5:Paired differences in Spearmanρ\\rhobetween each comparator method \(component\-wise andMTL\) and the joint\-encoding approach, computed within the same training sub\-sample\. The dummy regressor is undefined for correlation since its variance is zero\. Panels are aligned at the mean absolute performance of the joint\-encoding approach across sizes\.
## 4Discussion

This study compared three transformer\-based models for response\-free item\-difficulty prediction within a nested training\-size design on a corpus of reading\-comprehensionMCQs with pseudo\-labels anchored to response\-pattern\-based Rasch difficulty estimates\. Three findings emerge\.

First, transformer\-based models are a viable end\-to\-end model forMCQs: any trained method, including the joint\-encoding approach, outperforms the constant baseline \(training\-set mean ofbb\) by a margin substantially larger than the differences among the trained methods themselves\. At a training scale comparable to that reported in\[[31](https://arxiv.org/html/2605.16991#bib.bib361)\], i\.e\., 667 items, our methods reduceRMSEover the dummy regressor by 14–16% atn≈800n\\approx 800, against approximately 4% for the best shared\-task team \(0\.2990\.299versus0\.3110\.311on the test set\)\. The two datasets differ in stimulus domain, item structure, and difficulty scale, so a direct comparison of absolute residuals is not warranted, but a substantially larger relative reduction over the same kind of model\-free reference is consistent with our methods capturing more wording\-derivable variance\. End\-to\-end transformer fine\-tuning, therefore, efficiently collapses the preprocessing and modelling stages of the feature\-engineering tradition\.

Second, component\-wise encoding shows no detectable benefit over joint encoding under the present design\. Atn≈800n\\approx 800the component\-wise variant produces paired differences that are significantly worse onRMSEandR2R^\{2\}, and statistically indistinguishable from joint\-encoding on Spearmanρ\\rho; the pattern persists atn≈4,300n\\approx 4\{,\}300\. We read the findings as evidence that the encoder’s self\-attention is already harvesting the cross\-component signal that the component\-wise variant aims to enforce manually: imposing the decomposition at the input level neither adds new information nor reorganises the existing information in a way the simple regression head can exploit \(the difficulty under this setting must be a linear combination of pooled representations of each component\), while the joint variant leaves room for the self\-attention mechanism to benefit from the passage, question, and options being presented together early on, as opposed to the late interaction employed in our head design\. We remain moderately optimistic about variants we did not exhaust \(per\-option separate encoding, learned weighted aggregation\), but report the present result as it stands\.

Third, at the smallest training size considered \(n≈800n\\approx 800\), the multi\-task variant produces small but consistent paired improvements over the joint\-encoding approach across all three reported metrics \(RMSE,R2R^\{2\}, Spearmanρ\\rho\); this effect diminishes as the training set grows\. The improvements are small in absolute terms, but what the results consistently show is that the auxiliaryMCQAobjective provides a training signal that the regression target alone cannot, and that this signal matters precisely in the small\-sample regime where end\-to\-end fine\-tuning has the least information to recover from the wording alone or would otherwise learn surface\-level shortcuts\[[9](https://arxiv.org/html/2605.16991#bib.bib382)\]\.

It is important to note that a ceiling on purely text\-based prediction necessarily exists, since some difficulty\-relevant factors – test\-taking strategies, item\-position effects, or visual layout – leave no trace in the wording\. Moreover, the prediction target in our study itself carries two connected layers of noise that bound the achievable accuracy from above\. On the PALRACE anchor set,bib\_\{i\}is a Rasch estimate from a finite response sample and is reported with a non\-zero standard error; on the broader RACE\+\+ training corpus,bib\_\{i\}is the output of a pseudo\-labelling procedure calibrated against PALRACE Rasch \(r≈0\.46r\\approx 0\.46on the calibration anchors, Section[2\.1](https://arxiv.org/html/2605.16991#S2.SS1)\), so an additional component ofLLMerror is stacked on top of the response\-based standard error\. Any text\-based model ofbib\_\{i\}is therefore predicting an*estimate*, and both noise sources contribute to the predictive ceiling\. Rather than a limitation of the approach, these ceilings define the complementary roles that response\-free and response\-based estimation can play in a calibration workflow: response\-free predictions can provide priors or first\-pass estimates that a smaller response\-based follow\-up sample can refine\[[25](https://arxiv.org/html/2605.16991#bib.bib328)\]\.

While respecting the aforementioned limitations, we recognise multiple future directions that are promising, pushing the presented results even further\. A natural next step for theMCQAbranch, moving beyond the binary correctness target used here, is to replace the auxiliary option classification objective with response\-based per\-option probabilities, which would let the model receive an explicit signal about distractor competition rather than infer it indirectly from correct\-versus\-incorrect labels\. Another possibility would lie in adding another auxiliary head whose objective could be to label distinct wording components \(i\.e\. token\-classification task\)\. Next, there remain a number of unexplored techniques beyond design choices implemented in this work \(that may have hindered our results\) – two ready alternatives are dynamic loss\-balancing schemes\[[4](https://arxiv.org/html/2605.16991#bib.bib384)\]and gradient\-level conflict resolution, such as gradient surgery\[[32](https://arxiv.org/html/2605.16991#bib.bib385)\]\.

## 5Conclusion

This article compared three transformer fine\-tuning architectures for response\-freeIDMon reading\-comprehensionMCQs under a nested training\-size design\. Joint encoding of the concatenated item wording is a viable end\-to\-end alternative to two\-stage feature\-engineering pipelines; component\-wise encoding adds no detectable benefit over joint encoding, consistent with the encoder recovering theMCQcomponent structure through its own self\-attention; and the multi\-task variant, which adds an auxiliaryMCQAobjective on a shared encoder, delivers significant paired improvements at the smallest training size considered \(n≈800n\\approx 800\)\.

Taken together, these results show that transformer fine\-tuning enriched with structural or supervisory inductive biases recovers a substantial share of the wording\-derivable difficulty signal at training\-set sizes that are typically available in applied calibration contexts\. Predictions of this form can serve on their own when response data cannot be collected, and they may substantially shrink the response\-based sample required to refine the estimates further when such a follow\-up is feasible; combining the two regimes remains the most robust practice when both are available\[[25](https://arxiv.org/html/2605.16991#bib.bib328), as remarked, e\.g\., in\]\. A concrete next step is to replace the binary correctness target in the auxiliary head with response\-based per\-option probabilities, so that distractor competition, which the present formulation can address only indirectly, can shape the encoder directly\.

## References

- \[1\]\(2023\)Text\-based question difficulty prediction: A systematic review of automatic approaches\.34,pp\. 862–914\.External Links:ISSN 1560\-4306,[Document](https://dx.doi.org/10.1007/s40593-023-00362-1),[Link](https://doi.org/10.1007/s40593-023-00362-1)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p1.1),[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[2\]L\. Benedetto, G\. Aradelli, P\. Cremonesi, A\. Cappelli, A\. Giussani, and R\. Turrin\(2021\-04\)On the application of Transformers for estimating the difficulty of Multiple\-Choice Questions from text\.InProceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications,J\. Burstein, A\. Horbach, E\. Kochmar, R\. Laarmann\-Quante, C\. Leacock, N\. Madnani, I\. Pilán, H\. Yannakoudakis, and T\. Zesch \(Eds\.\),pp\. 147–157\.External Links:[Link](https://aclanthology.org/2021.bea-1.16)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[3\]R\. Caruana\(1997\-07\-01\)Multitask Learning\.28\(1\),pp\. 41–75\.External Links:ISSN 1573\-0565,[Document](https://dx.doi.org/10.1023/A%3A1007379606734),[Link](https://doi.org/10.1023/A:1007379606734)Cited by:[§2\.5](https://arxiv.org/html/2605.16991#S2.SS5.p1.1)\.
- \[4\]R\. Cipolla, Y\. Gal, and A\. Kendall\(2018\-06\)Multi\-task learning using uncertainty to weigh losses for scene geometry and semantics\.In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 7482–7491\.External Links:[Link](http://dx.doi.org/10.1109/CVPR.2018.00781),[Document](https://dx.doi.org/10.1109/cvpr.2018.00781)Cited by:[§4](https://arxiv.org/html/2605.16991#S4.p6.1)\.
- \[5\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding\(Website\)External Links:1810\.04805,[Document](https://dx.doi.org/10.48550/arXiv.1810.04805),[Link](http://arxiv.org/abs/1810.04805)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p1.3),[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p2.7),[§2\.3](https://arxiv.org/html/2605.16991#S2.SS3.p1.3),[footnote 1](https://arxiv.org/html/2605.16991#footnote1)\.
- \[6\]S\. E\. Embretson and C\. D\. Wetzel\(1987\)Component latent trait models for paragraph comprehension tests\.Applied Psychological Measurement11\(2\),pp\. 175–193\.External Links:[Document](https://dx.doi.org/10.1177/014662168701100207)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p3.1),[§2\.4](https://arxiv.org/html/2605.16991#S2.SS4.p1.1)\.
- \[7\]G\. H\. Fischer\(1973\)The linear logistic test model as an instrument in educational research\.Acta Psychologica37\(6\),pp\. 359–374\.External Links:[Document](https://dx.doi.org/10.1016/0001-6918%2873%2990003-6)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[8\]R\. Freedle and I\. Kostin\(1993\-07\)The prediction of TOEFL reading item difficulty: implications for construct validity\.10\(2\),pp\. 133–170\.External Links:ISSN 0265\-5322, 1477\-0946,[Document](https://dx.doi.org/10.1177/026553229301000203),[Link](https://journals.sagepub.com/doi/10.1177/026553229301000203)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[9\]R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. Wichmann\(2020\-11\-10\)Shortcut learning in deep neural networks\.2\(11\),pp\. 665–673\.External Links:ISSN 2522\-5839,[Document](https://dx.doi.org/10.1038/s42256-020-00257-z),[Link](https://www.nature.com/articles/s42256-020-00257-z)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p5.1),[§4](https://arxiv.org/html/2605.16991#S4.p4.3)\.
- \[10\]T\. Gong, T\. Lee, C\. Stephenson, V\. Renduchintala, S\. Padhy, A\. Ndirango, G\. Keskin, and O\. H\. Elibol\(2019\)A comparison of loss weighting strategies for multi\-task learning in deep neural networks\.IEEE Access7,pp\. 141627–141632\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2019.2943604)Cited by:[§2\.5](https://arxiv.org/html/2605.16991#S2.SS5.p5.7)\.
- \[11\]J\. S\. Gorin and S\. E\. Embretson\(2006\)Item difficulty modeling of paragraph comprehension items\.Applied Psychological Measurement30\(5\),pp\. 394–411\.External Links:[Document](https://dx.doi.org/10.1177/0146621606288554)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p3.1)\.
- \[12\]J\. Howard and S\. Ruder\(2018\)Universal language model fine\-tuning for text classification\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.1801.06146),[Link](https://arxiv.org/abs/1801.06146)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1),[§2\.3](https://arxiv.org/html/2605.16991#S2.SS3.p1.3)\.
- \[13\]G\. Jawahar, B\. Sagot, and D\. Seddah\(2019\-06\)What Does BERT Learn about the Structure of Language?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),pp\. 3651–3657\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1356),[Link](https://aclanthology.org/P19-1356/)Cited by:[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p1.3)\.
- \[14\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations \(ICLR\),External Links:1711\.05101,[Link](https://arxiv.org/abs/1711.05101)Cited by:[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p3.3),[§2\.6](https://arxiv.org/html/2605.16991#S2.SS6.p1.9)\.
- \[15\]A\. D\. McCarthy, K\. P\. Yancey, G\. T\. LaFlair, J\. Egbert, M\. Liao, and B\. Settles\(2021\-11\)Jump\-Starting Item Parameters for Adaptive Language Tests\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),pp\. 883–899\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.67),[Link](https://aclanthology.org/2021.emnlp-main.67/)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[16\]J\. Netík, F\. Martinek, and P\. Martinková\(2024\-07\-16/2024\-07\-19\)Fine\-tuning language models to predict item difficulty from wording\.Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1),[§1](https://arxiv.org/html/2605.16991#S1.p3.1)\.
- \[17\]Y\. Ozuru, M\. Rowe, T\. O’Reilly, and D\. S\. McNamara\(2008\)Where’s the difficulty in standardized reading tests: the passage or the question?\.Behavior Research Methods40\(4\),pp\. 1001–1015\.External Links:[Document](https://dx.doi.org/10.3758/BRM.40.4.1001)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p3.1)\.
- \[18\]S\. Peters, N\. Zhang, H\. Jiao, M\. Li, T\. Zhou, and R\. Lissitz\(2025\)Text\-based approaches to item difficulty modeling in large\-scale assessments: a systematic review\.External Links:2509\.23486,[Document](https://dx.doi.org/10.48550/arXiv.2509.23486),[Link](https://arxiv.org/abs/2509.23486)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[19\]G\. Rasch\(1960\)Probabilistic Models for Some Intelligence and Attainment Tests\.Danish Institute for Educational Research\.\.External Links:ISBN 978\-0\-598\-55451\-2Cited by:[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p4.4)\.
- \[20\]A\. Rogers, O\. Kovaleva, and A\. Rumshisky\(2020\)A primer in BERTology: what we know about how BERT works\.Transactions of the Association for Computational Linguistics8,pp\. 842–866\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00349)Cited by:[§2\.4](https://arxiv.org/html/2605.16991#S2.SS4.p1.1)\.
- \[21\]S\. Ruder\(2017\)An overview of multi\-task learning in deep neural networks\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1706.05098)Cited by:[§2\.5](https://arxiv.org/html/2605.16991#S2.SS5.p1.1)\.
- \[22\]D\. Sileo\(2023\)tasksource: a dataset harmonization framework for streamlined NLP multi\-task learning and evaluation\.External Links:2301\.05948,[Document](https://dx.doi.org/10.48550/arXiv.2301.05948),[Link](https://arxiv.org/abs/2301.05948)Cited by:[§2\.5](https://arxiv.org/html/2605.16991#S2.SS5.p7.5)\.
- \[23\]L\. Štěpánek, J\. Dlouhá, and P\. Martinková\(2023\-09\-28\)Item Difficulty Prediction Using Item Text Features: Comparison of Predictive Performance across Machine\-Learning Algorithms\.11\(19\),pp\. 4104\.External Links:ISSN 2227\-7390,[Document](https://dx.doi.org/10.3390/math11194104),[Link](https://www.mdpi.com/2227-7390/11/19/4104)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p1.1),[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[24\]W\. L\. Taylor\(1953\)"Cloze procedure": a new tool for measuring readability\.30,pp\. 415–433\.External Links:ISSN 0196\-3031Cited by:[footnote 1](https://arxiv.org/html/2605.16991#footnote1)\.
- \[25\]E\. Ulitzsch, D\. Belov, O\. Lüdtke, and A\. Robitzsch\(2025\)Using Item Parameter Predictions for Reducing Calibration Sample Requirements—A Case Study Based on a High\-Stakes Admission Test\.External Links:ISSN 1745\-3984,[Document](https://dx.doi.org/10.1111/jedm.12426),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1111/jedm.12426)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1),[§1](https://arxiv.org/html/2605.16991#S1.p3.1),[§4](https://arxiv.org/html/2605.16991#S4.p5.4),[§5](https://arxiv.org/html/2605.16991#S5.p2.1)\.
- \[26\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\(2017\)Attention Is All You Need\(Website\)External Links:1706\.03762,[Document](https://dx.doi.org/10.48550/arXiv.1706.03762),[Link](http://arxiv.org/abs/1706.03762)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p1.3)\.
- \[27\]Z\. Wang, Z\. Dai, B\. Póczos, and J\. Carbonell\(2019\)Characterizing and avoiding negative transfer\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 11293–11302\.External Links:[Link](https://arxiv.org/abs/1811.09751)Cited by:[§2\.5](https://arxiv.org/html/2605.16991#S2.SS5.p5.7)\.
- \[28\]B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. Poli\(2024\-12\-19\)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference\(Website\)External Links:2412\.13663,[Document](https://dx.doi.org/10.48550/arXiv.2412.13663),[Link](http://arxiv.org/abs/2412.13663)Cited by:[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p2.7)\.
- \[29\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush\(2020\-10\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Link](https://www.aclweb.org/anthology/2020.emnlp-demos.6)Cited by:[§2\.2](https://arxiv.org/html/2605.16991#S2.SS2.p3.3)\.
- \[30\]K\. P\. Yancey, A\. Runge, G\. LaFlair, and P\. Mulcaire\(2024\)BERT\-IRT: accelerating item piloting with BERT embeddings and explainable IRT models\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),pp\. 428–438\.Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p2.1)\.
- \[31\]V\. Yaneva, K\. North, P\. Baldwin, L\. A\. Ha, S\. Rezayi, Y\. Zhou, S\. Ray Choudhury, P\. Harik, and B\. Clauser\(2024\-06\)Findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple\-Choice Questions\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),E\. Kochmar, M\. Bexte, J\. Burstein, A\. Horbach, R\. Laarmann\-Quante, A\. Tack, V\. Yaneva, and Z\. Yuan \(Eds\.\),pp\. 470–482\.External Links:[Link](https://aclanthology.org/2024.bea-1.39/)Cited by:[§1](https://arxiv.org/html/2605.16991#S1.p3.1),[§2\.6](https://arxiv.org/html/2605.16991#S2.SS6.p1.9),[§4](https://arxiv.org/html/2605.16991#S4.p2.4)\.
- \[32\]T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. Finn\(2020\)Gradient surgery for multi\-task learning\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2001.06782),[Link](https://arxiv.org/abs/2001.06782)Cited by:[§4](https://arxiv.org/html/2605.16991#S4.p6.1)\.
- \[33\]J\. Zou, Y\. Zhang, P\. Jin, C\. Luo, X\. Pan, and N\. Ding\(2022\)PALRACE: Reading comprehension dataset with human data and labeled rationales\(Website\)External Links:arXiv,[Link](http://arxiv.org/abs/2106.12373)Cited by:[§2\.1](https://arxiv.org/html/2605.16991#S2.SS1.SSS0.Px1.p1.2)\.