Multimodal Item Parameter Estimation using Simulated Response Probabilitie

# Multimodal Item Parameter Estimation using Simulated Response Probabilities
Source: [https://arxiv.org/html/2608.10154](https://arxiv.org/html/2608.10154)
Christopher Ormerod College Board cormerod@collegeboard\.org &YoungKoung Kim College Board ykim@collegeboard\.org

###### Abstract

We present results from reconstructing multiple\-choice model \(MCM\) and three\-parameter logistic \(3PL\) model curves using a fine\-tuned multimodal large language model \(LLM\) based on Qwen3\.5\. The model is prompted and fine\-tuned to replicate choice probabilities across a large training corpus of multiple\-choice items containing both image and text stimuli, conditioned on a labeled set of student ability levels\. By learning to reproduce the systematic error patterns of students across a discrete range of abilities, the LLM implicitly captures the underlying response probabilities encoded in the 3PL and MCM curves\. This allows us to accurately approximate item difficulty on a held\-out test set directly from the model’s predicted option probabilities\.

Multimodal Item Parameter Estimation using Simulated Response Probabilities

Christopher OrmerodCollege Boardcormerod@collegeboard\.orgYoungKoung KimCollege Boardykim@collegeboard\.org

## 1Introduction

Item difficulty parameters are essential for evaluating whether an assessment accurately measures student knowledge, differentiating between students, and identifying specific learning gaps\. The field\-testing required to obtain these parameters is not only expensive and labor\-intensive, but also poses an inherent security risk\. For these reasons, researchers have explored machine learning techniques to approximate these parametersAlKhuzaey et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib1)\)\. Given the capabilities of Large Language Models \(LLMs\), a growing body of literature has emerged on their use in predicting item difficulty parameters\. Two characteristically distinct approaches have been proposed: treating the question stimulus and its features as input to a fine\-tuned LLM in a regression frameworkLi et al\. \([2025](https://arxiv.org/html/2608.10154#bib.bib9)\), and using LLMs as simulated respondents to test stimuliMaeda \([2025](https://arxiv.org/html/2608.10154#bib.bib13)\)\.

Generative LLMs have become increasingly powerful due to advances in training regimesWang et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib26)\)and the ability to scale the transformer architectureVaswani et al\. \([2017](https://arxiv.org/html/2608.10154#bib.bib25)\)to the point at which emergent capabilities ariseWei et al\. \([2022](https://arxiv.org/html/2608.10154#bib.bib27)\)\. The dominant paradigm for generative models has shifted from open\-ended text generation to instruction\-following via promptingChung et al\. \([2022](https://arxiv.org/html/2608.10154#bib.bib3)\)\. The flexibility of prompting allows researchers to instruct LLMs to answer questions in targeted ways, enabling the simulation of students with specified combinations of skillsLu and Wang \([2024](https://arxiv.org/html/2608.10154#bib.bib12)\), knowledgeHe\-Yueya et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib7)\), and ability levelsLiu et al\. \([2025](https://arxiv.org/html/2608.10154#bib.bib11)\)\. The present study extends work in which models are fine\-tuned in a parameter\-efficient manner to simulate students of varying abilityScarlatos et al\. \([2025](https://arxiv.org/html/2608.10154#bib.bib21)\), subsequently applied to multiple\-choice questionsOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\)\.

Prior work has examined the reconstruction of item characteristic curvesOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\)using the Nominal Response Model \(NRM\) and the Two\-Parameter Logistic \(2PL\) model to estimate item parametersThissen and Steinberg \([1986](https://arxiv.org/html/2608.10154#bib.bib23)\)\. Both models share the limiting assumption that the probability of a correct response approaches zero as ability decreases\. The Three\-Parameter Logistic \(3PL\) modelThissen and Steinberg \([1986](https://arxiv.org/html/2608.10154#bib.bib23)\)and the Multiple\-Choice Model \(MCM\)Thissen and Steinberg \([1984](https://arxiv.org/html/2608.10154#bib.bib22)\); Samejima \([1969](https://arxiv.org/html/2608.10154#bib.bib20)\)address this by incorporating non\-zero lower asymptotes, thereby accounting for the possibility of guessing\. First, the present work introduces several improvements to the framework ofOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\)and applies it to a large corpus of mathematics items\. Second, we demonstrate this framework for items with both text and images by fine\-tuning multimodal models with hybrid transformer\-based architectures\. While this may appear rather cumbersome, we are able to reproduce the item difficulty parameter with remarkably high accuracy compared to baseline regression\-based methods\.

We organize this paper as follows: Our method section, §[2](https://arxiv.org/html/2608.10154#S2), covers the nature of the response and item data used in this project, the response modeling applied to determine item parameters, the way in which we discretize ability levels, and the way we use fine\-tuned generative LLMs, the baseline approaches, and the metrics used for evaluation\. In §[3](https://arxiv.org/html/2608.10154#S3), we present our results, before concluding with a discussion in §[4](https://arxiv.org/html/2608.10154#S4)\.

## 2Method

To explain our method, we have raw data in the form of responses, and various models based on that data: 3PL modelsThissen and Steinberg \([1986](https://arxiv.org/html/2608.10154#bib.bib23)\), MCMsThissen and Steinberg \([1984](https://arxiv.org/html/2608.10154#bib.bib22)\), the discrete MCMsOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\), and LLMsTouvron et al\. \([2023](https://arxiv.org/html/2608.10154#bib.bib24)\)\. These models have been represented in Figure[1](https://arxiv.org/html/2608.10154#S2.F1), while the arrows, labelediitov​i​ivii, denote the various ways to define parameters of each model either in terms of the raw responses or other models\.

Responses3PLMCMLLMiii​iiii​i​iiii,i​vivvvv​ivi,v​i​iviiFigure 1:Diagram of the relationships between the data and models used in this study\.The following subsections are devoted to defining the data used in this study, the models, and the method of deriving the parameters of each model\. With all these defined, our goal is to reconstruct the parameters of the 3PL models and MCMs defined byiiandi​iii, respectively\.

### 2\.1Data

The raw data in this study consists of two distinct objects: items and responses to those items\.

#### 2\.1\.1Items

Each item consists of a stimulus and a set of four distinct options\. Since our understanding of Mathematics can be inherently visual, we allow both the stimulus and the options to have an associated image component\. In order to facilitate this combination of images as input, we transform the images into one image in the manner presented in Figure[2](https://arxiv.org/html/2608.10154#S2.F2)\.

stim

abcd

abcdStim

Figure 2:The way in which stimulus and option images were combined into a single image\. The left image depicts stimulus\-only images, the middle depicts option\-only images, and the right depicts when both stimulus and options are present\.This dataset is produced using item models, which specify the structure, content, and variable features needed to produce new, functional test itemsGierl and Haladyna \([2013](https://arxiv.org/html/2608.10154#bib.bib5)\)\. There were a total of 970 item models and a total of 4,848 items\. In order to avoid conflation, we split the items into training, development, and test sets by item model\. This gave a natural split at the item level represented in Table[1](https://arxiv.org/html/2608.10154#S2.T1)\.

### 2\.2Response Modeling

There are 13\.88 million responses\. The ability level of each respondent was independently derived from a larger calibrated dataset of items\. These responses provide us with an average of approximately 2860 responses per item with a minimum of 860 responses\. The average response ability level was 0\.31 and the standard deviation was approximately 1\.

Table 1:A representation of how the items were split into test, train, and development\.Given that each of these responses can be associated with a particular value of student ability, we can model the probability that a studentjj, with ability levelθj\\theta\_\{j\}, provides a correct answer to itemiiby

Pi​j​\(X=1\)=ci\+\(1−ci\)​σ​\(ai​\(θj−bi\)\)P\_\{ij\}\(X=1\)=c\_\{i\}\+\(1\-c\_\{i\}\)\\sigma\(a\_\{i\}\(\\theta\_\{j\}\-b\_\{i\}\)\)\(1\)whereσ\\sigmais the usual sigmoid function\. There are at least 10 correct or incorrect responses to any particular item, facilitating stable estimates of the item parameters,\{ai,bi,ci\}\\\{a\_\{i\},b\_\{i\},c\_\{i\}\\\}by minimizing the negative log likelihood function using the Limited\-memory Broyden–Fletcher–Goldfarb–Shanno with Bound constraints \(L\-BFGS\-B\)\.

To satisfy the conditions thatai\>0a\_\{i\}\>0and0<ci<10<c\_\{i\}<1, we letai=eαia\_\{i\}=e^\{\\alpha\_\{i\}\}andci=σ​\(γi\)c\_\{i\}=\\sigma\(\\gamma\_\{i\}\)\. Then our first arrow,ii, is provided by the optimization

ai,bi,ci=argmax\{∑j\[yjlogPi​j\\displaystyle a\_\{i\},b\_\{i\},c\_\{i\}=\\mathrm\{argmax\}\\left\\\{\\sum\_\{j\}\[y\_\{j\}\\log P\_\{ij\}\\right\.\+\(1−yj\)log\(1−Pi​j\)\]−λ\(αi2\+bi2\)\}\\displaystyle\\left\.\+\(1\-y\_\{j\}\)\\log\(1\-P\_\{ij\}\)\]\-\\lambda\(\\alpha\_\{i\}^\{2\}\+b\_\{i\}^\{2\}\)\\right\.\\Bigg\\\}\(ii\)where the first two terms are the usual negative log likelihood function and the third term is a smallL​2L2regularization term withλ=10−4\\lambda=10^\{\-4\}\. We used the Limited\-memory Broyden–Fletcher–Goldfarb–Shanno \(LBFGS\) method\.

The model described by \([1](https://arxiv.org/html/2608.10154#S2.E1)\) is known as the three\-parameter logistic model \(3PL\)Thissen and Steinberg \([1986](https://arxiv.org/html/2608.10154#bib.bib23)\)\. We callaia\_\{i\}the discrimination parameter,bib\_\{i\}the difficulty parameter, andcic\_\{i\}is called the guessing parameter\. It has the asymptotic property that asθj\\theta\_\{j\}is large and negative, the probability of obtaining the correct response is given by random chance; hence, it is typically assumed to be close to the reciprocal of the number of options\. Asci→0c\_\{i\}\\to 0, we obtain the two\-parameter logistic modelThissen and Steinberg \([1986](https://arxiv.org/html/2608.10154#bib.bib23)\)\.

The 3PL model is useful in defining a difficulty parameter; however, it does not allow us to provide any details regarding the strength of distractor items\. Given optionsk=1,…,Kk=1,\\ldots,K, the probability of a person of ability levelθj\\theta\_\{j\}providing answerkkto itemiiis modeled by

Pi​j​k=Pi​j​\(X=k\)=exp⁡\(ηi​j​k\)\+di​k1\+∑kexp⁡\(ηi​j​k\),P\_\{ijk\}=P\_\{ij\}\(X=k\)=\\dfrac\{\\exp\(\\eta\_\{ijk\}\)\+d\_\{ik\}\}\{1\+\\sum\_\{k\}\\exp\(\\eta\_\{ijk\}\)\},\(2\)whereηi​j​k=ai​k​θj\+bi​k\\eta\_\{ijk\}=a\_\{ik\}\\theta\_\{j\}\+b\_\{ik\}\. Given that∑kPi​j​\(X=k\)=1\\sum\_\{k\}P\_\{ij\}\(X=k\)=1anddi​k\>0d\_\{ik\}\>0, this provides us with∑kdi​k=1\\sum\_\{k\}d\_\{ik\}=1for any item\. This model is a reformulation of the multiple\-choice model from Thissen and SteinbergThissen and Steinberg \([1984](https://arxiv.org/html/2608.10154#bib.bib22)\)\. Alternatively, this generalizes the version of Samejima in whichdi​k=K−1d\_\{ik\}=K^\{\-1\}Samejima \([1969](https://arxiv.org/html/2608.10154#bib.bib20)\)\. In a similar manner to the 3PL model, we may derive the values of\{ai​k,bi​k,di​k\}\\\{a\_\{ik\},b\_\{ik\},d\_\{ik\}\\\}using the optimization

\{ai​k,bi​k,di​k\}=argmax\{∑j=1logPi​j​k\\displaystyle\\\{a\_\{ik\},b\_\{ik\},d\_\{ik\}\\\}=\\mathrm\{argmax\}\\left\\\{\\sum\_\{j=1\}\\log P\_\{ijk\}\\right\.−λ\(ai​k2\+bi​k2\)\}\\displaystyle\\left\.\-\\lambda\(a\_\{ik\}^\{2\}\+b\_\{ik\}^\{2\}\)\\right\\\}\(i​iii\)withλ=10−4\\lambda=10^\{\-4\}using LBFGS\. This provides us with the arrowi​iii\. In combination withii, this provides the target values for our modeling\.

### 2\.3Discrete Ability Modeling

The main idea fromOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\)is that we are binning the values ofθj\\theta\_\{j\}intoLLcategories\. The cutoff points for each category are denotedγi\\gamma\_\{i\}so thatIl=\(γl−1,γl\]I\_\{l\}=\(\\gamma\_\{l\-1\},\\gamma\_\{l\}\]with

−∞=γ0<γ1<…<γL=∞\.\-\\infty=\\gamma\_\{0\}<\\gamma\_\{1\}<\\ldots<\\gamma\_\{L\}=\\infty\.This provides us with intervalsI1,…,ILI\_\{1\},\\ldots,I\_\{L\}such that eachθj∈Ilj\\theta\_\{j\}\\in I\_\{l\_\{j\}\}\. The greater theLL, the more fine\-grained the division of abilities\. Each interval is associated with a loosely descriptive label\. The labels and intervals are presented in Table[2](https://arxiv.org/html/2608.10154#S2.T2)\.

Table 2:A list of the descriptive labels used to calibrate the language model\.Following the work ofOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\), for itemii, we associate each interval with a probability value for each option, which gives us a function

fi​\(Ij\)=\(ρi​j​1,…,ρi​j​K\)\.f\_\{i\}\(I\_\{j\}\)=\(\\rho\_\{ij1\},\\ldots,\\rho\_\{ijK\}\)\.\(3\)This means we obtain, as data, anL×KL\\times Kmatrix of probability values\.

Our underlying prior is that the population is normally distributed such that\{θj\}∼𝒩​\(μ,ς2\)\\\{\\theta\_\{j\}\\\}\\sim\\mathcal\{N\}\(\\mu,\\varsigma^\{2\}\)\. We find it convenient to state functions associated with this distribution explicitly\. The probability and cumulative distribution functions \(PDF & CDF\),f​\(θ\)f\(\\theta\)andF​\(θ\)F\(\\theta\)are given by

f​\(θ\)=1ς​ϕ​\(θ−μς\),F​\(θ\)=Φ​\(θ−μς\),\\displaystyle f\(\\theta\)=\\frac\{1\}\{\\varsigma\}\\phi\\left\(\\frac\{\\theta\-\\mu\}\{\\varsigma\}\\right\),\\,\\,F\(\\theta\)=\\Phi\\left\(\\frac\{\\theta\-\\mu\}\{\\varsigma\}\\right\),where

ϕ​\(z\)=12​π​e−z22,Φ​\(z\)=∫−∞zϕ​\(t\)​dt,\\displaystyle\\phi\(z\)=\\dfrac\{1\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{z^\{2\}\}\{2\}\},\\,\\,\\Phi\(z\)=\\int\_\{\-\\infty\}^\{z\}\\phi\(t\)\\mathrm\{d\}t,It makes sense to define the conditional probability thatθ∈Ij\\theta\\in I\_\{j\}as

wj=F​\(γj\)−F​\(γj−1\)w\_\{j\}=F\(\\gamma\_\{j\}\)\-F\(\\gamma\_\{j\-1\}\)
InOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\), each interval was associated with the expected value under this prior, withρi​j​k\\rho\_\{ijk\}defined as the evaluation of \([2](https://arxiv.org/html/2608.10154#S2.E2)\) at this value\. It actually makes more sense to associate the interval with the expected value of the function itself\. Provided thatPi​j​kP\_\{ijk\}is defined by \([2](https://arxiv.org/html/2608.10154#S2.E2)\), then the definition ofρi​j​k\\rho\_\{ijk\}may be determined by

ρi​j​k=1wj​∫γj−1γjf​\(θ\)​Pi​θ​k​dθ,\\displaystyle\\rho\_\{ijk\}=\\frac\{1\}\{w\_\{j\}\}\\int\_\{\\gamma\_\{j\-1\}\}^\{\\gamma\_\{j\}\}f\(\\theta\)P\_\{i\\theta k\}\\mathrm\{d\}\\theta,\(i​i​iiii\)wherePi​θ​kP\_\{i\\theta k\}is the function ofθ\\thetarather than the particular value atθj\\theta\_\{j\}\. This provides us withi​i​iiii\.

Furthermore, given that these are continuous values rather than discrete values, we can approximate the reconstruction of the MCM by optimizing

\(ai​k,\\displaystyle\(a\_\{ik\},bi​k,di​k\)=argmin\(∑jwj\(ρi​j​k−\\displaystyle b\_\{ik\},d\_\{ik\}\)=\\mathrm\{argmin\}\\Bigg\(\\sum\_\{j\}w\_\{j\}\\Bigg\(\\rho\_\{ijk\}\-1wj∫γj−1γjf\(θ\)Pi​θ​kdθ\)2\)\.\\displaystyle\\frac\{1\}\{w\_\{j\}\}\\int\_\{\\gamma\_\{j\-1\}\}^\{\\gamma\_\{j\}\}f\(\\theta\)P\_\{i\\theta k\}\\mathrm\{d\}\\theta\\Bigg\)^\{2\}\\Bigg\)\.\(i​viv\)Alternatively, we can specialize the value ofkkto be the correct option,k~\\tilde\{k\}, in which case we obtain the probability of being correct, which provides a viable path to reconstructing the parameters of \([1](https://arxiv.org/html/2608.10154#S2.E1)\)\. For completeness, we write this as

\(ai,bi\\displaystyle\(a\_\{i\},b\_\{i\},ci\)=argmin\(∑jwj\(ρi​j​k~\\displaystyle,c\_\{i\}\)=\\mathrm\{argmin\}\\Bigg\(\\sum\_\{j\}w\_\{j\}\\Bigg\(\\rho\_\{ij\\tilde\{k\}\}−1wj∫γj−1γjf\(θ\)Pi​θdθ\)2\)\\displaystyle\-\\frac\{1\}\{w\_\{j\}\}\\int\_\{\\gamma\_\{j\-1\}\}^\{\\gamma\_\{j\}\}f\(\\theta\)P\_\{i\\theta\}\\mathrm\{d\}\\theta\\Bigg\)^\{2\}\\Bigg\)\(vv\)wherePi​θP\_\{i\\theta\}is the model \([1](https://arxiv.org/html/2608.10154#S2.E1)\) where the dependency onθj\\theta\_\{j\}is replaced by the general function ofθ\\theta\. This provides us with all the arrows that do not depend on the language model\.

It is worth noting that these approximations, at each arrow, are not exact\. For example, the arrowiiprovides a mapping from the set of empirically observed responses to the IRT parameters; however, there is an underlying assumption that \([1](https://arxiv.org/html/2608.10154#S2.E1)\) is a good fit for the empirical data\. We assume that there are some regressive tendencies in which the parameters are linearly related to the final parameters\. We use the development set to appropriately and linearly model this relationship, which does not change the overall Pearson correlations, but does affect the mean squared error\.

### 2\.4Language Modeling

The Qwen model series continues to provide the research community with a suite of excellent fine\-tunable models for research purposes\. The latest series of models, Qwen3\.5, provides us with the opportunity to demonstrate the abilities of multimodal fine\-tuning\. The models we conservatively chose are the 4\-billion\- and 9\-billion\-parameter variants of Qwen3\.5\.

#### 2\.4\.1Parameter\-efficient fine\-tuning

These models are small enough to fit on a local machine with sufficient resources\. In order to train these models, we employ parameter\-efficient methodsXu et al\. \([2023](https://arxiv.org/html/2608.10154#bib.bib28)\)such as Low\-Rank Adaptation \(LoRA\)Hu et al\. \([2021](https://arxiv.org/html/2608.10154#bib.bib8)\)or LoRA with QuantizationDettmers et al\. \([2023](https://arxiv.org/html/2608.10154#bib.bib4)\)to effectively tune the model, requiring some knowledge of the underlying model structure\.

The Qwen3\.5 series not only integrates multimodal inputs but is also a hybrid structure similar to the Jamba seriesLieber et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib10)\)\. Each layer replaces multiheaded attention in the transformer architectureVaswani et al\. \([2017](https://arxiv.org/html/2608.10154#bib.bib25)\)with two distinct types of layers: Gated DeltaNet layersYang et al\. \([2025](https://arxiv.org/html/2608.10154#bib.bib29)\)and Gated Attention layersQiu et al\. \([2025](https://arxiv.org/html/2608.10154#bib.bib18)\)\. The Gated DeltaNet layers have a structure similar to that of the Mamba2 layers in that the information is stored in hidden states and preserved through a selective gating mechanism\. This means that the Gated DeltaNet layers have linear complexity while the Gated Attention layers have quadratic complexity, making this closer to a Jamba model than previous Qwen models\.

LinearLinearLinearProj\.Proj\.LinearConv\.Conv\.Conv\.L2L2Gated Delta RuleZero Centered RMSNormLinear⊗σ′\{\}\_\{\\sigma^\{\\prime\}\}σ′\{\}\_\{\\sigma^\{\\prime\}\}σ′\{\}\_\{\\sigma^\{\\prime\}\}qqkkqqGated DeltaNetFigure 3:The structure of the Gated DeltaNet layers\. Here,σ′\\sigma^\{\\prime\}is the SiLU function\.Figure[4](https://arxiv.org/html/2608.10154#S2.F4)presents the internal structure of the Gated DeltaNet and Attention layers\.

LinearLinearLinearLinearScaled dot product attentionLinear⊗σqqkkvvGated AttentionFigure 4:The structure of the Gated Attention layers\. Here,σ\\sigmais the sigmoid function\.The application of LoRA takes a selection of linear layers, of the form

whereMMis anm×nm\\times nmatrix, and replaces those linear layers with

L~​\(x\)=\(M\+B​A\)​x\+b\\tilde\{L\}\(x\)=\(M\+BA\)x\+bwhereAAis anr×nr\\times nmatrix andBBis anm×rm\\times rmatrix withr<<min⁡\(m,n\)r<<\\min\(m,n\)\. The main idea is that we freeze all other layers and keepAAandBBtrainableHu et al\. \([2021](https://arxiv.org/html/2608.10154#bib.bib8)\)\. This process has been very successful in automated scoringOrmerod and Kwako \([2024](https://arxiv.org/html/2608.10154#bib.bib14)\), standards alignmentHan et al\. \([2026](https://arxiv.org/html/2608.10154#bib.bib6)\), and item parameter estimationOrmerod \([2026](https://arxiv.org/html/2608.10154#bib.bib15)\)\.

Given instabilities in the training process encountered in the literature, we conservatively applied LoRA to the attention components, knowing this would essentially affect only a quarter of the hidden layers that would be typically used to fine\-tune models\. We would need a better understanding of how to fine\-tune the Gated DeltaNet layers before considering the other layers\.

#### 2\.4\.2Prompting

The creation of a dataset that encodes information regarding student options requires formatting the data as a sequence of chats\. Each chat consists of three components: a system text indicating the behavior of the model, a user text indicating a task to be completed, and an assistant text indicating how the task should be completed\. These texts are functions of the ability level \(`ability\_level`\), the correct answer \(`correct`\), the item text \(`stim\_text`\), and the option text \(`option\_text`\), which contains an enumeration of the options in text form\. These are presented below:

```
ΨΨSystem: You are a student who must answer
ΨΨeither A, B, C, D in a multiple choice
ΨΨquestion.
ΨΨ
ΨΨ**Ability Level**: {ability_level}.
ΨΨ
ΨΨ**Correct Answer**: {correct}
Ψ
```

```
ΨΨUser: Answer the following **Question**
ΨΨas if you were a {ability_level} student.
ΨΨ
ΨΨ**Question**
ΨΨ{stim_text}
ΨΨ
ΨΨ**Options**
ΨΨ{option_text}
Ψ
```

```
ΨΨAssistant: **Answer**:
Ψ
```

We leave the final text of the student option blank, truncating the text so that it does not satisfy the traditional requirements of a full chat\. The idea is that we are obtaining a vector of next\-token probabilities, which we will denotevvwhere the dimension ofvvis the number of tokens in the model vocabulary\. For convenience, we will assume the softmax has been applied so that each component ofvv, denotedvιv\_\{\\iota\}, is interpreted as a token probability\.

There are indices,ι1,…,ιK\\iota\_\{1\},\\ldots,\\iota\_\{K\}, associated with each option\. This allows us to construct the Mean Squared Error \(MSE\) loss function, defined by

ℒ=∑k1K​\(vιk−ρi​j​k\)2,\\mathcal\{L\}=\\sum\_\{k\}\\frac\{1\}\{K\}\\left\(v\_\{\\iota\_\{k\}\}\-\\rho\_\{ijk\}\\right\)^\{2\},\(4\)whereiidenotes the item index andjjdenotes the ability level index\. Using the standard Adam optimizer with a weight decay mechanism and a learning rate of10−510^\{\-5\}over 10 epochs provides us with the training regime used to define the fine\-tuned language model, essentially providing the arrowv​ivi\. Finally, inference using the prompt above provides the arrowv​i​ivii\.

#### 2\.4\.3Baseline models

Our baseline approach is to use the question stimulus and options as input and then treat the IRT parameters as targets for a regression problem\. This means that we take an encoder\-only transformer\-based language model, such as MetaMathYu et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib30)\)and MathBERTPeng et al\. \([2021](https://arxiv.org/html/2608.10154#bib.bib16)\), replace the head with a linear layer with one target, and use the MSE loss function, similar to above, with the outputs of the language model and the target IRT parameters as the arguments\. This approach is the one that is most common in the literatureBulut et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib2)\); Peters et al\. \([2025](https://arxiv.org/html/2608.10154#bib.bib17)\)\.

Table[3](https://arxiv.org/html/2608.10154#S2.T3)summarizes the models used in this study, together with their approximate parameter counts and references\.

Table 3:Models used in this study, with approximate parameter counts and references\. MathBERTPeng et al\. \([2021](https://arxiv.org/html/2608.10154#bib.bib16)\)is built on BERT\-base; MetaMath is a LLaMA\-2 fine\-tune available at 7B, 13B, and 70B\.

### 2\.5Metrics

In reference to the literature on item parameter prediction, there are two dominant metrics used to compare the performance of difficulty prediction models: Pearson correlation and root mean squared errorAlKhuzaey et al\. \([2024](https://arxiv.org/html/2608.10154#bib.bib1)\)\.

We would like to highlight another approach that fits nicely in the context of our study: a measure of the agreement under a discretization of difficulty\. In communicating difficulty to educators, item difficulty parameters are often difficult to interpret\. For this reason, what is often conveyed is a descriptive label that is similar to the descriptive labels we used to describe ability levels\. The labels, as functions of thebbparameter, are presented in Table[4](https://arxiv.org/html/2608.10154#S2.T4)\. Under these descriptive labels, it is useful to obtain a measure of the agreement between the predicted labels and the calibrated labels\. For this, we use the quadratic weighted kappa \(QWK\)\.

Table 4:A list of the descriptive labels used to communicate difficulty\.

## 3Results

### 3\.1Difficulty Prediction

Difficulty is the parameter of greatest practical interest, since it is the quantity most often reported to educators and used when assembling test forms\. Table[5](https://arxiv.org/html/2608.10154#S3.T5)reports the Pearson correlation and root mean squared error \(RMSE\) between the predicted and calibrated parameters\.

Table 5:The associated correlation values and RMSE values for each approach\.The fine\-tuned Qwen3\.5\-9B model attains a Pearson correlation of0\.850\.85on the difficulty parameterbb, exceeding both regression baselines—MathBERT \(0\.680\.68\) and MetaMath \(0\.750\.75\)—by a clear margin\. This corresponds to a relative improvement of roughly15%15\\%over the stronger baseline, obtained with an RMSE of0\.550\.55after the linear development\-set correction described in Section[2](https://arxiv.org/html/2608.10154#S2)\. The result indicates that the simulated\-respondent framing recovers difficulty\-relevant signal that the direct stimulus\-to\-parameter regression of the baselines does not capture\.

When difficulty is discretized into the five\-band scheme of Table[4](https://arxiv.org/html/2608.10154#S2.T4), the model reaches a QWK of0\.8350\.835against the calibrated labels, whereas MetaMath has a QWK of0\.6920\.692and MathBERT has a QWK of0\.6250\.625\.

## 4Discussion

Item difficulty modeling remains one of the most difficult tasks in computational psychometrics, and our results reinforce both the promise and the limits of the simulated\-respondent approach\. With a modest amount of parameter\-efficient fine\-tuning, a 9\-billion\-parameter multimodal model reconstructs the difficulty parameter at a fidelity \(Pearson0\.850\.85, QWK0\.8350\.835\) that comfortably exceeds strong text\-based regression baselines, and it does so for items that mix textual and visual stimuli within a single model and a single input representation \(Figure[2](https://arxiv.org/html/2608.10154#S2.F2)\)\. This suggests that the central idea—training the model to replicate the systematic pattern of option choices across a discretized range of abilities and then reading item parameters off the resulting curves—transfers cleanly to the multimodal setting without changes to the underlying architecture\.

The most informative aspect of our results is the uneven recovery across parameters\. Recovering the guessing parameterccat a correlation of0\.480\.48, where the regression baselines are very low by comparison, is arguably the clearest evidence that the approach captures genuine response behavior rather than surface features of the stimulus: a model that merely reads the question cannot easily infer how often low\-ability students will be drawn to a particular distractor, whereas a model trained to imitate those students can\. Conversely, the weak recovery of the discrimination parameteraaindicates that slope information is partially lost, most plausibly through the discretization of ability into intervals and through the regression\-based correction applied to the raw estimates\. Finer\-grained ability binning, at additional computational cost, is a natural avenue for improving slope recovery\.

Several design choices bound these results and point to concrete extensions\. First, to avoid the training instabilities reported in the literature, we applied LoRA only to the Gated Attention components of the hybrid Qwen3\.5 architecture, leaving the Gated DeltaNet layers untouched; this adapts roughly a quarter of the layers that a conventional transformer would expose to fine\-tuning\. A better understanding of how to adapt the DeltaNet layers stably would likely improve recovery, particularly of the parameters that are weakest under the present regime\. Second, every arrow in our pipeline introduces approximation error—the mapping from empirical responses to IRT parameters assumes that the 3PL and MCM are good fits, and the discrete reconstruction replaces continuous curves with binned expectations—so the reported correlations are best read as lower bounds on what the framing can achieve under tighter modeling assumptions\.

## Acknowledgments

## References

- AlKhuzaey et al\. \(2024\)Samah AlKhuzaey, Floriana Grasso, Terry R\. Payne, and Valentina Tamma\. 2024\.[Text\-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches](https://doi.org/10.1007/s40593-023-00362-1)\.*International Journal of Artificial Intelligence in Education*, 34\(3\):862–914\.
- Bulut et al\. \(2024\)Okan Bulut, Guher Gorgun, and Bin Tan\. 2024\.[Item Difficulty and Response Time Prediction with Large Language Models: An Empirical Analysis of USMLE Items](https://aclanthology.org/2024.bea-1.44/)\.In*Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\)*, pages 522–527, Mexico City, Mexico\. Association for Computational Linguistics\.
- Chung et al\. \(2022\)Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro\-Ros, Marie Pellat, Kevin Robinson, and 16 others\. 2022\.[Scaling Instruction\-Finetuned Language Models](http://arxiv.org/abs/2210.11416)\.*arXiv preprint*\.
- Dettmers et al\. \(2023\)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\. 2023\.[QLoRA: Efficient Finetuning of Quantized LLMs](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html)\.*Advances in Neural Information Processing Systems*, 36:10088–10115\.
- Gierl and Haladyna \(2013\)Mark J\. Gierl and Thomas M\. Haladyna, editors\. 2013\.[*Automatic Item Generation: Theory and Practice*](https://www.routledge.com/Automatic-Item-Generation-Theory-and-Practice/Gierl-Haladyna/p/book/9780415897518)\.Routledge\.
- Han et al\. \(2026\)Suhwa Han, Christopher Ormerod, and Frank Rijmen\. 2026\.[Fine\-tuning of Open\-source Large Language Models to Align Items to Standards](https://osf.io/cdg5v_v1)\.Preprint, OSF/EdArXiv\.
- He\-Yueya et al\. \(2024\)Joy He\-Yueya, Wanjing Anya Ma, Kanishk Gandhi, Benjamin W\. Domingue, Emma Brunskill, and Noah D\. Goodman\. 2024\.[Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models](https://doi.org/10.48550/arXiv.2407.15645)\.*arXiv preprint*\.ArXiv:2407\.15645 \[cs\.CL\]\.
- Hu et al\. \(2021\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2021\.[LoRA: Low\-Rank Adaptation of Large Language Models](https://doi.org/10.48550/arXiv.2106.09685)\.*arXiv preprint*\.
- Li et al\. \(2025\)Ming Li, Hong Jiao, Tianyi Zhou, Nan Zhang, Sydney Peters, and Robert W Lissitz\. 2025\.[Item Difficulty Modeling Using Fine\-Tuned Small and Large Language Models](https://aclanthology.org/2025.aimecon-sessions.5/)\.In*Proceedings of the Artificial Intelligence in Measurement and Education Conference \(AIME\-Con\): Coordinated Session Papers*, pages 48–55, Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States\. National Council on Measurement in Education \(NCME\)\.
- Lieber et al\. \(2024\)Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev\-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, and 3 others\. 2024\.[Jamba: A Hybrid Transformer\-Mamba Language Model](https://doi.org/10.48550/arXiv.2403.19887)\.*arXiv preprint*\.ArXiv:2403\.19887 \[cs\.CL\]\.
- Liu et al\. \(2025\)Yunting Liu, Shreya Bhandari, and Zachary A\. Pardos\. 2025\.[Leveraging LLM respondents for item evaluation: A psychometric analysis](https://doi.org/10.1111/bjet.13570)\.*British Journal of Educational Technology*, 56\(3\):1028–1052\.\_eprint: https://bera\-journals\.onlinelibrary\.wiley\.com/doi/pdf/10\.1111/bjet\.13570\.
- Lu and Wang \(2024\)Xinyi Lu and Xu Wang\. 2024\.[Generative Students: Using LLM\-Simulated Student Profiles to Support Question Item Evaluation](https://doi.org/10.1145/3657604.3662031)\.In*Proceedings of the Eleventh ACM Conference on Learning @ Scale*, pages 16–27\.ArXiv:2405\.11591 \[cs\.HC\]\.
- Maeda \(2025\)Hotaka Maeda\. 2025\.[Field\-Testing Multiple\-Choice Questions With AI Examinees: English Grammar Items](https://doi.org/10.1177/00131644241281053)\.*Educational and Psychological Measurement*, 85\(2\):221–244\.
- Ormerod and Kwako \(2024\)Chris Ormerod and Alexander Kwako\. 2024\.[Automated Text Scoring in the Age of Generative AI for the GPU\-poor](https://doi.org/10.59863/OKUU1904)\.*Chinese/English Journal of Educational Measurement and Evaluation*, 5\(3\)\.
- Ormerod \(2026\)Christopher Ormerod\. 2026\.[Reconstructing Item Characteristic Curves using Fine\-Tuned Large Language Models](https://arxiv.org/abs/2601.02580v1)\.
- Peng et al\. \(2021\)Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang\. 2021\.[MathBERT: A Pre\-Trained Model for Mathematical Formula Understanding](https://doi.org/10.48550/arXiv.2105.00377)\.*arXiv preprint*\.
- Peters et al\. \(2025\)Sydney Peters, Nan Zhang, Hong Jiao, Ming Li, Tianyi Zhou, and Robert Lissitz\. 2025\.[Text\-Based Approaches to Item Difficulty Modeling in Large\-Scale Assessments: A Systematic Review](https://arxiv.org/abs/2509.23486v1)\.
- Qiu et al\. \(2025\)Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin\. 2025\.[Gated Attention for Large Language Models: Non\-linearity, Sparsity, and Attention\-Sink\-Free](https://doi.org/10.48550/arXiv.2505.06708)\.*arXiv preprint*\.ArXiv:2505\.06708 \[cs\.CL\]\.
- Qwen Team \(2026\)Qwen Team\. 2026\.[Qwen3\.5: Towards Native Multimodal Agents](https://qwen.ai/blog?id=qwen3.5)\.
- Samejima \(1969\)Fumiko Samejima\. 1969\.[Estimation of Latent Ability Using a Response Pattern of Graded Scores](https://doi.org/10.1007/BF03372160)\.*Psychometrika*, 34\(S1\):1–97\.
- Scarlatos et al\. \(2025\)Alexander Scarlatos, Nigel Fernandez, Christopher Ormerod, Susan Lottridge, and Andrew Lan\. 2025\.[SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://doi.org/10.18653/v1/2025.emnlp-main.1274)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 25071–25094, Suzhou, China\. Association for Computational Linguistics\.
- Thissen and Steinberg \(1984\)David Thissen and Lynne Steinberg\. 1984\.[A response model for multiple choice items](https://doi.org/10.1007/BF02302588)\.*Psychometrika*, 49\(4\):501–519\.
- Thissen and Steinberg \(1986\)David Thissen and Lynne Steinberg\. 1986\.[A taxonomy of item response models](https://doi.org/10.1007/BF02295596)\.*Psychometrika*, 51\(4\):567–577\.
- Touvron et al\. \(2023\)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others\. 2023\.[Llama 2: Open Foundation and Fine\-Tuned Chat Models](https://doi.org/10.48550/arXiv.2307.09288)\.*arXiv preprint*\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\. 2017\.[Attention is All you Need](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)\.In*Advances in Neural Information Processing Systems*, volume 30\. Curran Associates, Inc\.
- Wang et al\. \(2024\)Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, and Eduard Hovy\. 2024\.[Reinforcement Learning Enhanced LLMs: A Survey](https://arxiv.org/abs/2412.10400v3)\.
- Wei et al\. \(2022\)Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H\. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus\. 2022\.[Emergent Abilities of Large Language Models](https://arxiv.org/abs/2206.07682v2)\.
- Xu et al\. \(2023\)Lingling Xu, Haoran Xie, Si\-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang\. 2023\.[Parameter\-Efficient Fine\-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://doi.org/10.48550/arXiv.2312.12148)\.*arXiv preprint*\.
- Yang et al\. \(2025\)Songlin Yang, Jan Kautz, and Ali Hatamizadeh\. 2025\.[Gated Delta Networks: Improving Mamba2 with Delta Rule](https://doi.org/10.48550/arXiv.2412.06464)\.*arXiv preprint*\.ArXiv:2412\.06464 \[cs\.CL\]\.
- Yu et al\. \(2024\)Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T\. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu\. 2024\.[MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://doi.org/10.48550/arXiv.2309.12284)\.*arXiv preprint*\.ArXiv:2309\.12284 \[cs\.CL\]\.