Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques

# Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques
Source: [https://arxiv.org/html/2608.06609](https://arxiv.org/html/2608.06609)
Hotaka Maeda[![[Uncaptioned image]](https://arxiv.org/html/2608.06609v1/orcid.png)](https://orcid.org/0009-0000-9498-786X)1Yikai Lu[![[Uncaptioned image]](https://arxiv.org/html/2608.06609v1/orcid.png)](https://orcid.org/0000-0003-4410-2589)2 1Smarter Balanced, University of California\-Santa Cruz 2Department of Educational Psychology, University of Minnesota\-Twin CitiesCorrespondence concerning this article should be addressed to Hotaka Maeda, Smarter Balanced, University of California\-Santa Cruz, 1156 High St, Santa Cruz, CA 95064\. E\-mail:[hotaka\.maeda@smarterbalanced\.org](https://arxiv.org/html/2608.06609v1/mailto:[email protected])

###### Abstract

Automated item evaluation\(AIE\) refers to the use of computational methods to assess item quality without requiring manual expert review or field testing of the items under evaluation\. We aimed to build a near\-comprehensive AIE model by predicting item acceptance and rejection from item text using historical rejection data from a large\-scale standardized testing program\. The dataset contained 52,759 English language arts \(ELA\) and mathematics items with 34% permanently rejected from future operational use\. Rejection reasons included poor psychometric properties, content issues, bias and sensitivity concerns, and non\-content issues\. We fine\-tuned a DeBERTaV3\-large classifier on raw item text, a second DeBERTa classifier on Qwen3\-generated item critiques, and a fusion model combining representations from both\. The fusion model achieved the strongest overall performance \(Accuracy=\.75=\.75, F1=\.64=\.64, AUC=\.80=\.80, Sensitivity=\.64=\.64, Specificity=\.81=\.81\)\. Prediction for math \(F1=\.73=\.73, AUC=\.86=\.86\) was considerably more accurate than ELA \(F1=\.51=\.51, AUC=\.72=\.72\)\. Lowering the decision threshold from \.5 to \.25 raised average sensitivity for ELA and math to \.88 and \.91, while reducing specificity to \.31 and \.56, respectively, which may be preferable in automated item generation contexts where generating items is cheaper than evaluating them\. Incorporating item critiques alongside raw item text improved performance across most rejection reasons\. The model assigned higher rejection probabilities to more difficult items\. However, the fusion model struggled to identify items flagged for bias, sensitivity, fairness, or accessibility, especially for ELA\. These findings suggest that text\-based AIE is feasible in some areas and may offer a practical tool for reducing the burden of manual review and field testing, while also underscoring the importance of human review for items with fairness concerns\.

Keywords:natural language processing, Transformers, artificial intelligence, item difficulty prediction, item response theory

## 1Introduction

Evaluating the quality of new items is expensive and time\-consuming, requiring content review and field testing\. This issue has grown since the emergence of automated item generation \(AIG\) using large language models \(LLMs\)\. AIG review papers consistently urge for methods that can quickly filter poor quality items\(Oluokeet al\.,[2026](https://arxiv.org/html/2608.06609#bib.bib13); Circiet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib16); Falcãoet al\.,[2022](https://arxiv.org/html/2608.06609#bib.bib15); Tanet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib14)\)\. Relevant literature has focused on using LLMs, natural language processing \(NLP\), or machine learning to predict individual qualities of items, including item difficulty\(AlKhuzaeyet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib89); Benedettoet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib81)\), content alignment with the assessment blueprint\(Fuet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib29)\), or differential item functioning\(Maeda and Lu,[2025](https://arxiv.org/html/2608.06609#bib.bib168)\)\. However, prior work has not addressed these concerns as a unified class of issues related to item quality\. To capture this broader problem space, we use the termautomated item evaluation\(AIE; Yu and Burke,[2026](https://arxiv.org/html/2608.06609#bib.bib11)\), extending its prior use from a much narrower scope\. In this study, AIE refers to the use of computational methods to automatically assess item quality without requiring manual expert review or field testing of the items under evaluation\.

The difficulty of AIE stems from the fact that there are countless reasons that items can be unsuited for operational use\. The Standards for Educational and Psychological Testing\(AERAet al\.,[2014](https://arxiv.org/html/2608.06609#bib.bib128)\)articulate the broader requirements that define a valid, reliable, and fair item\. Also,Haladyna and Downing \([1989](https://arxiv.org/html/2608.06609#bib.bib33)\)cataloged 43 item\-writing rules covering stem construction, option formatting, correct answer selection, and distractor quality\. Training and deploying a separate prediction model for each rule is unrealistic because model training requires large sample sizes and substantial computational resources\. Moreover, treating rules separately fails to leverage the fact that items may exhibit multiple issues simultaneously and does not directly support the goal of rejecting the most undesirable items\.

Therefore, a comprehensive AIE approach could be desirable\. One possible approach is to use an existing item bank by treating operational items as “accepted” and items permanently retired due to irreparable issues as “rejected”, then training a model to predict that outcome from item content alone\. Accepted items had to have gone through rigorous content expert review and field testing, so this single label becomes an all\-encompassing measure of item quality\. Although this approach requires a large established item bank, it may be the single most comprehensive AIE method for both AIG and human\-written items\.

To our knowledge, no studies have predicted item acceptance and rejection as an AIE method\. The purpose of this study is to train a transformer language model to classify items as accepted or rejected for operational use, using item status data from a large\-scale standardized testing program\. To enhance predictive power, we augment the data using item critiques generated by an LLM\. To provide insight into model behavior beyond overall classification performance, we use historical item developer comments to evaluate classifier sensitivity by rejection reason\. We also include a sentiment analysis to further understand how LLM critique sentiment is related to prediction model behavior\.

## 2Related Works

### 2\.1\(Automated\) Item Evaluation

Item evaluation is considered as an important step during test development\(AERAet al\.,[2014](https://arxiv.org/html/2608.06609#bib.bib128); Haladyna and Downing,[1989](https://arxiv.org/html/2608.06609#bib.bib33)\)\.Gorgun and Bulut \([2025](https://arxiv.org/html/2608.06609#bib.bib3)\)summarized existing item evaluation approaches into three categories: metric\-based evaluations, post\-hoc analysis, and human evaluations\. Metric\-based approaches typically focus on comparing automatically generated items with a set of reference items\(Gorgun and Bulut,[2025](https://arxiv.org/html/2608.06609#bib.bib3)\)\. Other metric\-based evaluation approaches include NLP\-based techniques, such as word count, TF\-IDF, word embeddings, cosine similarity for identifying enemy items, and measures of readability and text complexity\(Yu and Burke,[2026](https://arxiv.org/html/2608.06609#bib.bib11); Aminiet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib7)\)\. However, despite their potential efficiency, metric\-based approaches are limited by their reliance on reference items\. This dependence can significantly restrict the content domain of generated items, as valid items may be incorrectly penalized simply because theyappeardifferent from reference items at the level of linguistic features\(Gorgun and Bulut,[2025](https://arxiv.org/html/2608.06609#bib.bib3)\)\. Post\-hoc analysis approaches typically involve field testing, in which real item response data are collected from a representative sample of examinees to assess item quality\. The most commonly examined aspects are statistical item properties, such as item difficulty prediction\(AlKhuzaeyet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib89); Benedettoet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib81)\)and item discrimination\(e\.g\., Maeda and Lu,[2026](https://arxiv.org/html/2608.06609#bib.bib34); Hanet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib83); Yanevaet al\.,[2020](https://arxiv.org/html/2608.06609#bib.bib36)\)\. Less commonly examined areas include differential item functioning\(Maeda and Lu,[2025](https://arxiv.org/html/2608.06609#bib.bib168)\)\. However, this approach can be costly and time\-consuming, as it requires substantial resources to collect response data and obtain reliable statistical estimates\. Finally, human evaluators can identify problems in items that may be difficult to detect using computational or statistical approaches\. However, human evaluation can be subjective and time\-consuming, especially when many items must be reviewed\. Another limitation is that evaluators need to be familiar with the relevant content domain, which can make it difficult to identify suitable experts for the evaluation process\.

The term “automated item evaluation” or a similar term has been used occasionally typically in the AIG literature\(Gorgun and Bulut,[2025](https://arxiv.org/html/2608.06609#bib.bib3); Prentzas and Binopoulou,[2025](https://arxiv.org/html/2608.06609#bib.bib12); Yu and Burke,[2026](https://arxiv.org/html/2608.06609#bib.bib11); Shin and Gierl,[2024](https://arxiv.org/html/2608.06609#bib.bib9); Wanget al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib10)\)\. Despite its sparse usage and variation in scope in the past literature, definitions of AIE appear to share a common theme: the use of computational methods to automatically assess some facets of item quality without requiring manual expert review or field testing of the items under evaluation\. Under this definition, only the first approach described byGorgun and Bulut \([2025](https://arxiv.org/html/2608.06609#bib.bib3)\)meets the criteria for AIE, as it eliminates the need to rely on field testing or human evaluators\.

However, although not proposed byGorgun and Bulut \([2025](https://arxiv.org/html/2608.06609#bib.bib3)\)themselves, the approach they described could be considered a fourth category: LLM\-based evaluation, which is another AIE approach\. This approach is the closest precedent to our comprehensive evaluation framework, as it trained an LLM to identify low\-quality AIG items using human review labels\. However, this approach is severely limited because it provides only a coarse\-grained evaluation based on human review labels, classifying items simply asgoodorbad\. Note that, asPeláneket al\.\([2022](https://arxiv.org/html/2608.06609#bib.bib8)\)argues, classifying items asgoodorbadmay be a fundamentally ill\-defined problem, as items that warrant the attention of content creators depend on the specific context and are at least partly subjective\. Furthermore, items may be problematic for many different reasons, including linguistic or psychometric characteristics that sometimes cannot be captured by human evaluators and vice versa\. Therefore, it is important for us to consider different aspects of item quality at once\.

It is worth highlighting that LLM\-based evaluation has made possible the automatic assessment of item\-content alignment with assessment blueprints, a less commonly examined area of AIE\(Fuet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib29)\)\. This is an important consideration in exam design in its own right, as it directly affects content validity\. This suggests that LLM\-based evaluation could shed light on aspects of items that were difficult to evaluate automatically prior to LLMs\.

The major limitation of these previous approaches is that they tend to focus on only one aspect of item quality at a time\. In practice, however, different dimensions of item quality may be interrelated\. For example, a difficult mathematics item may also contain difficult keywords, and such relationships cannot be fully leveraged when these issues are evaluated separately\. Furthermore, relying solely on computational or statistical methods may overlook problems that can only be identified through human judgment\. For these reasons, it is preferable to construct a model that can evaluate multiple dimensions of item quality simultaneously\. For example,Peláneket al\.\([2022](https://arxiv.org/html/2608.06609#bib.bib8)\)proposed a system that uses an outlier detection mechanism to identify computationally or statistically deviant items, thereby flagging potentiallybaditems\. Similarly,Yanevaet al\.\([2020](https://arxiv.org/html/2608.06609#bib.bib36)\)predicted whether MCQs has acceptable item difficulty and discrimination in a high\-stakes medical exam, framing the problem as binary classification using linguistic features and word embeddings extracted from item text\. However, the linguistic features both studies considered were mostly text complexity measures, which by themselves cannot account for the semantic content of items, something that LLMs can leverage\. On the other hand, our hybrid approach can account for these different aspects of item quality by using a dataset that includes rejected items with a broad range of documented reasons for rejection\. Rather than relying solely on human evaluators or linguistic features, this approach fine\-tunes an LLM to distinguish between acceptable and unacceptable items while implicitly learning from multiple sources of item\-quality evidence by taking into account the content of items\.

### 2\.2Transformer language models for text classification

The rise of Transformer language models increased demand for AIE research by making large\-scale AIG feasible, while simultaneously expanding the computational tools useful for AIE\. Traditionally, prediction from text in the assessment context has relied on expert judgment\(Wauterset al\.,[2012](https://arxiv.org/html/2608.06609#bib.bib68)\), syntactic features such as word count and term frequency\(Benedettoet al\.,[2020](https://arxiv.org/html/2608.06609#bib.bib60)\), or semantic features like word embeddings\(Hsuet al\.,[2018](https://arxiv.org/html/2608.06609#bib.bib69)\)\. More recently, Transformer\-based language models have substantially improved prediction accuracy\(Liet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib70); Hanet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib83); Maeda,[2025](https://arxiv.org/html/2608.06609#bib.bib131)\)\. Introduced byVaswaniet al\.\([2017](https://arxiv.org/html/2608.06609#bib.bib104)\), the Transformer architecture replaced recurrent networks with parallel attention mechanisms, enabling efficient training and scalable modeling of long\-range dependencies\. Models such as BERT\(Devlinet al\.,[2019](https://arxiv.org/html/2608.06609#bib.bib127)\)are pre\-trained on large corpora and fine\-tuned for downstream tasks\. Text is tokenized into embeddings and passed through encoder layers to produce contextually enriched representations\.

In this study, we use DeBERTaV3\-large\(Heet al\.,[2021](https://arxiv.org/html/2608.06609#bib.bib101)\)\. DeBERTa extends BERT\(Devlinet al\.,[2019](https://arxiv.org/html/2608.06609#bib.bib127)\)and RoBERTa\(Liuet al\.,[2019](https://arxiv.org/html/2608.06609#bib.bib67)\)\. DeBERTa outperforms earlier models by disentangling content and positional information within its attention mechanism, where each token is represented by separate content and position vectors, allowing the model to capture relationships between words more effectively\. The V3 variant further incorporates replaced token detection, yielding a 304\-million\-parameter model with strong benchmark performance that even outperforms the newer ModernBERT in natural language understanding\(Warneret al\.,[2024](https://arxiv.org/html/2608.06609#bib.bib140)\)\.

### 2\.3Using LLM Critiques for AIE

Recently, language models have been used to extract reasoning or rationales to augment a task\-specific model\(Henrichsen and Krebs,[2025](https://arxiv.org/html/2608.06609#bib.bib20); Hsiehet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib19); Scarlatoset al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib55)\)\. For an example in the assessment context, GPT\-4o model\(OpenAI,[2024](https://arxiv.org/html/2608.06609#bib.bib17)\)was used to generate the reasoning steps required to reach each multiple choice item option, and the responses were inputted into the longformer encoder language model to predict item difficulty\(Fenget al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib21)\)\. In another example, LLMs were given the role of content experts to make absolute and pairwise item comparisons to estimate item difficulty\(Kolesnikovaet al\.,[2026](https://arxiv.org/html/2608.06609#bib.bib26)\)\.

We use a similar data augmentation technique in this study using Qwen3\(Qwen Team,[2025](https://arxiv.org/html/2608.06609#bib.bib40)\)\. The Qwen3 model family includes a series of compact dense variants spanning 0\.6B, 1\.7B, 4B, and 8B parameters, all released publicly under the Apache 2\.0 license\. These models follow the broader trend of developing capable small language models \(SLMs\) suited to resource\-constrained settings, where low memory footprint and fast inference are primary constraints\(Luet al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib25)\)\. A key capability distinguishing these models from earlier SLMs is the integration of athinking modeand anon\-thinking modewithin a single model\. Rather than maintaining separate chat and reasoning model variants, users can toggle between slow, multi\-step reasoning and fast, context\-driven response generation at inference time\.

Empirical results reported inQwen Team \([2025](https://arxiv.org/html/2608.06609#bib.bib40)\)indicate that the Qwen3\-8B, 4B, and 1\.7B base models each outperform Qwen2\.5 models of the next larger size class on standard benchmarks, reflecting substantial efficiency gains from the distillation\-centric training approach\.

## 3Methods

### 3\.1Item Data

Included in the study were 52,759 items designed for English language arts \(ELA\) and mathematics state summative assessments for grades 3–11\. Of these, 34% have been rejected, defined as permanently removed from future operational use \(30% for ELA, 39% for math\)\. Rejection can occur at any stage of item development, before or after field testing, or after some operational use\. Rejection reasons span a range of categories including content and standards alignment issues, psychometric reasons, accessibility and sensitivity flags, or issues unrelated to the item content, such as data integrity problems \(see[Appendix A](https://arxiv.org/html/2608.06609#A1)for full category descriptions\)\.

Items that had not been rejected and had not yet been field tested were excluded from the study, as their operational suitability remains unknown\. Items rejected prior to field testing were retained, as their unsuitability for operations is established regardless of psychometric properties\. In total, 14% of items in the study are items rejected prior to field testing\. The remaining 86% of items were field tested among grade\-matched students across multiple states in the United States\.

To prepare each item for modeling, item text was concatenated with a\\nseparator between prompts and any available answer options\. Both selection\-based \(e\.g\., multiple choice, multiple select\) and constructed response \(e\.g\., short answer, extended response\) items were included in the study\. Any reading or listening passages were excluded from the item text data due to their extensive lengths\. Any images, figures, or tables in the items were also excluded\. Metadata were exluded as well, which included the scoring rubric and correct answer keys\. Data were partitioned randomly into approximately 80% training, 10% validation, and 10% test data\. Item groups with multiple items associated with the same stimulus were always included in the same data group\.

### 3\.2Prediction Approaches

Five prediction approaches were compared\. We used a single NVIDIA A10G Tensor Core 24GB graphics processor for all model inferences and training\.

#### 3\.2\.1Zero\-Shot Baseline

Qwen3\-0\.6B\(Qwen Team,[2025](https://arxiv.org/html/2608.06609#bib.bib40)\)was prompted to classify each item as “accept” or “reject” directly, without any task\-specific fine\-tuning \(see[Appendix B](https://arxiv.org/html/2608.06609#A2)for prompt\)\. The subject, grade level, and the aligned common core standards for the item were included in the prompt\.

#### 3\.2\.2Raw Item Text Fine\-Tuned Model

Item text was tokenized and passed into DeBERTaV3\-large\(Heet al\.,[2021](https://arxiv.org/html/2608.06609#bib.bib101)\), fine\-tuned as a binary classifier using Low\-Rank Adaptation \(LoRA;Huet al\.[2022](https://arxiv.org/html/2608.06609#bib.bib80); LoRA rank = 32, LoRA Alpha = 64, LoRA dropout = 0\.05, dropout = 0\.1, batch size = 8, max token length = 512, weight decay = \.01, learning rate = 5e\-5, epochs = 3\)\. LoRA is a parameter\-efficient fine\-tuning method that inserts trainable low\-rank matrices into selected layers while keeping the pretrained weights frozen, substantially reducing memory and computational costs\. We added a binary classification head with a sigmoid function to convert logits into rejection probabilities, followed by cross\-entropy loss \(CEL\) to minimize the distance between predicted and true labels\.

There were approximately double the number of accepted items compared to rejected items\. This class imbalance was addressed by incorporating class weights into the binary CEL\. Weights were computed as inversely proportional to class frequency, such that the minority class received proportionally greater influence on the gradient updates during training\. This approach preserves the full training sample while discouraging the model from defaulting to the majority class, effectively rebalancing the loss contribution of each class without altering the composition of the training data\. By increasing the relative penalty for misclassifying rejected items, class weighting was expected to make the model more attentive to rejected items\. We retained the state with the lowest validation CEL out of all epochs\.

#### 3\.2\.3Item Critique Fine\-Tuned Model

To augment the raw item text, Qwen3\-0\.6B\(Qwen Team,[2025](https://arxiv.org/html/2608.06609#bib.bib40)\)with thinking mode enabled was prompted to generate a two sentence critique or praise of each item’s quality\. The subject, grade level, and the aligned common core standards for the item were included in the prompt\. See[Appendix C](https://arxiv.org/html/2608.06609#A3)for the prompt and[Appendix D](https://arxiv.org/html/2608.06609#A4)for example critiques\. The resulting critique text was then used as input to DeBERTaV3\-large in place of the raw item text, with the same fine\-tuning procedure as the text\-only model \(LoRA rank = 50, LoRA Alpha = 100, LoRA dropout = 0\.05, dropout = 0\.1, batch size = 8, max token length = 512, weight decay = \.01, learning rate = 5e\-5, epochs = 3\)\.

#### 3\.2\.4Text \+ Critique Fusion Model

The fusion model combines representations from both DeBERTaV3\-large models described above\. The 1024\-dimensional output of each model’s final layer was extracted and concatenated\. The DeBERTa weights were frozen, and only the fusion layers were trained\. We added a 128\-dimensional intermediate layer followed by a binary classification head with a sigmoid function and a weighted CEL \(see Figure[1](https://arxiv.org/html/2608.06609#S3.F1)\)\.

Figure 1:Automated item evaluation model using raw text \+ LLM Critique fusion\.![Refer to caption](https://arxiv.org/html/2608.06609v1/figure_model.png)

*Note\.*Item text is processed directly through the first Deberta\. Qwen is used to generate item critiques, then entered into a separate Deberta\. Two DeBERTas are combined in the final fusion model\.

#### 3\.2\.5Stand\-Alone Subject Models

Given that ELA and math are very distinct subjects, we included an alternative approach of building separate models for these subjects\. All raw text\-only, critique\-only, and fusion models were re\-trained using this approach\.

### 3\.3Evaluation

The area under the ROC curve \(AUC\) was used as a cutoff threshold\-independent measure of the model’s overall discriminative ability\. Classification thresholds were fixed to \.50 by default\. Using this threshold, we report accuracy \(proportion of correctly classified items\), precision \(proportion of flagged items that are truly rejected\), sensitivity \(proportion of rejected items correctly flagged\), specificity \(proportion of viable items correctly passed\), and F1 score \(harmonic mean of precision and sensitivity\)\. Similar to outlier detection, this paper treats rejected cases as positive cases for evaluation purposes\.

#### 3\.3\.1Rejection Reason Classification

Item developers often left comments when moving an item to a permanent rejection status\. Comment quality varied widely, ranging from absent or uninformative \(e\.g\., “Moving to rejected”\) to detailed explanations\. Multiple comments for the same item were concatenated and treated as a single record\. Using a combination of Claude Sonnet 4\.6\(Anthropic,[2025](https://arxiv.org/html/2608.06609#bib.bib22)\)and manual fixes, comments from rejected items were cleaned and classified into 10 rejection reason categories: content, psychometric, data review, bias, abandoned, incomplete, passage, scoring, non\-content, and no data \(see[Appendix A](https://arxiv.org/html/2608.06609#A1)for definitions\)\. These labels were used to assess how well the best\-performing model detected items across each rejection reason\. The rejection reason data were only used for test sets\. Only 19% of rejected test items had usable rejection reason data\. Other items were classified in the “no data” category because the comments were missing \(80%\) or were ambiguous \(1%\)\.

### 3\.4Rejection Rate by Difficulty and Discrimination

We examined the Spearman correlations between the fusion model predictions and several item difficulty and discrimination metrics, including proportion correct, item\-total correlation, and item response theory \(IRT\) parameters\. All items were calibrated using the two\-parameter logistic \(2PL\) model\(Birnbaum,[1968](https://arxiv.org/html/2608.06609#bib.bib88)\)for dichotomous items or the generalized partial credit model \(GPCM;Muraki[1992](https://arxiv.org/html/2608.06609#bib.bib52)\) for polytomous items\. For GPCM items, the item difficulty parameter was centered to represent average difficulty of the thresholds\. Therefore, both item discrimination \(aa\) and difficulty \(bb\) parameters were comparable between 2PL and GPCM\. The data spanned two subjects and multiple grades, but we standardizedθ\\thetato have a mean of zero and standard deviation of one within each subject\-grade combination, effectively removing the vertical scale for this analysis\. Therefore, including proportion correct, item\-total correlation, all item statistics are relative to the subject and grade\. For example, a low proportion correct for an ELA grade 3 item is difficult for grade 3 students\.

### 3\.5Sentiment Analysis of Critiques

To examine if sentiment can partly explain how the Qwen3 critiques improved prediction, we conducted a sentiment analysis\. We computed sentiment using SiEBERT\(Hartmannet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib1)\), a RoBERTa\-large model fine\-tuned for binary sentiment classification across 15 heterogeneous English text sources, chosen for its cross\-domain generalization relative to models trained on a single text type\. For each critique, we extracted the difference between the positive and negative class logits, yielding a continuous log\-odds sentiment score in which higher values indicate more positive sentiment\. We then computed Spearman correlations between critique sentiment and predicted rejection probability of the text\-only modelPt​e​x​tP\_\{text\}, fusion modelPf​u​s​i​o​nP\_\{fusion\}and the differenceP​Δ=Pf​u​s​i​o​n−Pt​e​x​tP\\Delta=P\_\{fusion\}\-P\_\{text\}\.

## 4Results

All results are based on test data\. Model performance is summarized in Table[1](https://arxiv.org/html/2608.06609#S4.T1)\. Based on F1 score, the zero\-shot approach was clearly the worst performer \(F1=\.23=\.23\)\. The fusion model consistently outperformed both the raw text\-only and critique\-only models \(Accuracy=\.75=\.75, F1=\.64=\.64, AUC=\.80=\.80, precision=\.63=\.63, sensitivity=\.64=\.64, specificity=\.81=\.81\)\. In all cases, prediction for math was considerably more accurate than ELA\. The stand\-alone fusion models separated by subject \(ELA and math\) did not meaningfully change performance metrics\. Note that a fusion model with non\-weighted CEL had essentially the same performance and was not worth reporting separately \(Accuracy=\.75=\.75, F1=\.63=\.63, AUC=\.80=\.80, precision=\.62=\.62, sensitivity=\.65=\.65, specificity=\.79=\.79\)\. However, the weighted model offers an interpretive advantage: because class weighting shifts the optimal decision boundary near 0\.5, the classification threshold aligns with the intuitive probability midpoint\. Subsequent analyses focus on evaluating the fusion model performance in detail\.

Table 1:Classification Performance by Model and Subject*Note\.*ELA = English language arts, p = proportion of items predicted as reject, AUC = area under the curve\. Cutoff threshold for models were fixed to \.5\.

### 4\.1Improving Sensitivity by Lowering the Cutoff Threshold

Lowering the cutoff threshold below \.5 increases sensitivity to flag non\-viable items at the cost of specificity \(see Figure[2](https://arxiv.org/html/2608.06609#S4.F2)\)\. For instance, a threshold of \.25 raises the overall sensitivity to \.90 while reducing specificity to \.42 and F1 to \.60\. Math would have a sensitivity of \.91 \(specificity = \.56, F1 = \.70\), while ELA would have a sensitivity of \.88 \(specificity = \.31, F1 = \.51; see[Appendix E](https://arxiv.org/html/2608.06609#A5)\)\. A low detection threshold may be preferable in AIG, where producing items is far less costly than evaluating them\.

Figure 2:Cutoff threshold analysis for fusion model \(raw text \+ critique\)![Refer to caption](https://arxiv.org/html/2608.06609v1/thresh.png)

*Note\.*ELA = English language arts\. Lowering the cutoff threshold below \.5 increases sensitivity to flag rejected items at the cost of specificity\. A lower cutoff threshold may be preferable in contexts such as automated item generation where item generation has a low cost\.

### 4\.2Item Rejection Reasons

Predicted rejection probability by rejection reason was estimated on truly rejected items in the test data \(see Figure[3](https://arxiv.org/html/2608.06609#S4.F3)\)\. Items that were abandoned \(sensitivity=\.75=\.75\) or had psychometric issues \(sensitivity=\.66=\.66\) had one of the highest sensitivities for the fusion model, especially for math\. Compared to the text\-only model, incorporating item critiques in the fusion model improved detection across nearly all rejection categories\. Low sensitivity \(\.16\) to detect non\-content rejections was expected as they typically reflect back\-end data integrity issues unrelated to item text\. Detection rate increased substantially for incomplete math items \(Δ\\Deltasensitivity=\.31=\.31\) and those with content issues \(Δ\\Deltasensitivity=\.33=\.33\)\. However, the fusion model struggled to detect concerns regarding bias, sensitivity, fairness, and accessibility \(sensitivity=\.27=\.27\), most of which were ELA \(91%\)\.

Figure 3:Boxplots of predicted rejection probability by rejected reason for items truly labeled as rejected\.![Refer to caption](https://arxiv.org/html/2608.06609v1/rejectionreason.png)

*Note\.*ELA = English language arts\. S=Sensitivity \(using≥\.5\\geq\.5probability threshold\), N=number of items, Scoring = issues with rubrics, answer keys, scoring logic, or scoring metadata, Abandoned = items without a suitable exam placement, Passage = rejection driven by the passage set rather than the item itself, Psychometric = psychometric issues like low discrimination and extreme difficulty, Incomplete = items that never reached field\-testing due to stalling in development, Data review = questionable psychometric qualities requiring content expert review, including C\-level differential item functioning, Content = content errors like standard misalignment, Bias = content accessibility, bias, fairness, or sensitivity concerns, Non\-content = issues like missing, incompatible, or corrupt data that is not expected to be detectable by text\-based models, \(no data\) = items rejected without comments or with unclear reasons\. Only items truly labeled as rejected are shown\. For full rejection reason details, see[Appendix A](https://arxiv.org/html/2608.06609#A1)\.

### 4\.3Item Difficulty and Discrimination

Spearman correlations between predicted rejection probability and several psychometric statistics was strongest with proportion correct \(r=−\.30r=\-\.30\), indicating that the model assigned higher rejection probabilities to more difficult items\. Correlations with total item correlation \(r=−\.06r=\-\.06\), IRT discrimination \(r=−\.07r=\-\.07\), and the item intercept parameter \(−1\.7​a​b\-1\.7ab;r=−\.19r=\-\.19\) were weaker\. Overall, the model is sensitive to rejecting items that may be too difficult for the target population \(see Figure[4](https://arxiv.org/html/2608.06609#S4.F4)\)\. Note that 6\.4% of items had missing item statistics because they were rejected prior to field test analyses\. For these items, sensitivity was 66%, which was the same as the other items\.

Figure 4:Item difficulty and discrimination by fusion model\-predicted rejection probability\.![Refer to caption](https://arxiv.org/html/2608.06609v1/stats2.png)

*Note\.*r = Spearman correlations\.

### 4\.4Critique Sentiment

Correlations between critique sentiment andPt​e​x​tP\_\{text\}andPf​u​s​i​o​nP\_\{fusion\}was significant for only mathPf​u​s​i​o​nP\_\{fusion\}\(r=−\.05r=\-\.05,p<\.05p<\.05\)\. Negative critique sentiment was associated with slightly increasedP​ΔP\\Delta, across all subject and status label groups\. The Spearman correlation between sentiment andP​ΔP\\Deltawas−\.05\-\.05for ELA \(p<\.01p<\.01\) and−\.14\-\.14for math \(p<\.0001p<\.0001\)\. Disaggregating by status label, correlations were not significantly different from zero for ELA items \(p\>\.05p\>\.05\), and−\.11\-\.11for accepted and−\.18\-\.18for rejected math items \(p<\.0001p<\.0001\)\. Therefore, the association was stronger for math than for ELA, and strongest among math items that had a true rejected status\. Results were consistent with the interpretation that the critiques contributed information beyond the item text itself, where negative sentiment increased rejection rates\. But overall, sentiment was a minor characteristic of how critiques were related to the model behavior\.

## 5Discussion

We began the study with the goal of building a comprehensive AIE model that could identify unwanted items across as many aspects of item quality as possible\. We achieved this by training a transformer language model to classify items as accepted or rejected for operational use, using item status data from a large\-scale standardized testing program\. Data augmentation using item critiques generated by Qwen3 improved prediction accuracy across nearly all rejection reasons\. The result was a single AIE model that is nearly comprehensive, much more convenient than most other AIE methods that target a single aspect of item quality\. Our approach showed practical promise for large\-scale assessment pipelines where both the evaluation and discard of items carry substantial cost\.

We want to emphasize that AIE is useful for both human\-written and AI\-generated items, though the purpose and strategy of its use can vary\. In AIG contexts, where item generation is inexpensive, a lower detection threshold such as \.25 may be preferable\. At the cost of specificity \(\.42\) and F1 \(\.60\), sensitivity of our fusion model rises to \.90, catching most future rejections before they happen\. This can save a substantial amount of time spent on content expert review\. In more traditional item development contexts, the default threshold of \.5 may serve as an effective early warning to revise the item\. The predictions were considerably more accurate for math than ELA, which may suggest fundamental differences in how items are rejected between these subjects\. Potentially, ELA item rejections were often related to their passages, which were excluded from the current study\.

Among all item statistics examined, proportion correct \(item difficulty\) showed the strongest relationship with rejection\. In spite of not directly using item statistics in the training, the model especially tended to flag items that were too difficult\. This shows the value of item difficulty prediction research\(AlKhuzaeyet al\.,[2024](https://arxiv.org/html/2608.06609#bib.bib82); Benedettoet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib81)\), but the modest correlation we found \(r=−\.30r=\-\.30\) suggests that difficulty prediction alone is insufficient, as items are rejected for many reasons beyond difficulty\.

Several aspects of this study are novel\. First, the approach of fusing an encoder representation of raw item text with an encoder representation of a separate decoder\-generated critique is relatively new, though a handful of studies have taken similar approaches\(Henrichsen and Krebs,[2025](https://arxiv.org/html/2608.06609#bib.bib20); Hsiehet al\.,[2023](https://arxiv.org/html/2608.06609#bib.bib19); Scarlatoset al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib55); Fenget al\.,[2025](https://arxiv.org/html/2608.06609#bib.bib21)\)\. This method leverages the decoder’s capacity to draw on broad world knowledge when generating critiques\. Decoders are not designed to output numerical estimates reliably, so we rely on the encoder for precise, task\-specific prediction\. Our approach could likely improve by using a larger decoder\. The Qwen3\-0\.6B model we used is very small \(i\.e\., 600 million parameters\) compared to the most powerful proprietary multimodal models rumored to have trillions of parameters\(Li,[2026](https://arxiv.org/html/2608.06609#bib.bib6)\)\. The caveat is that these proprietary models may require additional security measures unlike models that can be ran locally like Qwen\. The field of AIE could therefore benefit from testing more powerful decoders combined with efficient, precisely fine\-tuned encoders\.

Another novel aspect of this study was the use of item developer comments\. We are not aware of any other paper that has used item developer comments to help build a prediction model in the educational assessment context\. The closest precedent may beMa \([2025](https://arxiv.org/html/2608.06609#bib.bib39)\), who applied DistilBERT and machine learning to examinee comments collected after test administration, building a model to identify comments most relevant for item review\. It established that free\-text commentary contains information for item quality decisions\. We considered using item developer comments in the training data but decided against it, as only 19% of our data had rejection comments\. Nevertheless, we could benefit from further exploration of how item developer comments can be leveraged for AIE\.

Our model takes a nearly comprehensive approach to AIE, as the key outcome of judging item quality is whether it is acceptable for operational use\. However, we cannot fully claim comprehensiveness for several reasons\. For example, item quality partly depends on the other items already in the bank\. We do not want two or more items that are too similar to each other, and detecting this would require a similarity analysis\(Peng,[2020](https://arxiv.org/html/2608.06609#bib.bib5)\)\. Further, the distribution of item difficulty within a bank or test should typically be balanced\(van der Linden and Pashley,[2000](https://arxiv.org/html/2608.06609#bib.bib4)\)\. Evaluating these require comparing the new items to the current item bank\.

Finally, we acknowledge that some rejection reasons are inherently unpredictable from item content alone, including item exposure, rejection due to other items within the same passage set, data integrity issues, or dependence on future events not yet reflected in the data\. Incorporating full item metadata including the passage text and scoring rubric into the prediction could partly resolve this, but some item quality issues will always depend on unpredictable future events\.

One of the biggest concerns with our fusion model was its difficulty identifying items flagged for bias, sensitivity, fairness, or accessibility concerns, possibly because these judgments require contextual and cultural knowledge that is difficult to encode from item text alone\. This suggests that human review remains essential for detecting these issues in particular\.

## 6Conclusion

We introduced a novel approach to automated item evaluation \(AIE\) by training a transformer language model to predict item acceptance or rejection using item status data from a large\-scale standardized testing program\. Augmenting item text with LLM\-generated critiques improved prediction accuracy across nearly all rejection reasons, demonstrating that combining encoder and decoder representations is a promising direction for AIE\. Unlike prior work that has focused on predicting individual item properties such as difficulty, our approach targets the operational decision that ultimately determines an item’s fate, offering a single, near\-comprehensive quality metric applicable to both human\-written and AI\-generated items\. As AIG continues to expand the volume of items requiring review, scalable AIE methods like the one presented here offer practical promise for reducing the cost and burden of item evaluation in large\-scale assessment programs\.

## References

- Standards for Educational and Psychological Testing\.American Educational Research Association,Washington, DC\.Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1)\.
- S\. AlKhuzaey, F\. Grasso, T\. R\. Payne, and V\. Tamma \(2023\)Text\-based question difficulty prediction: a systematic review of automatic approaches\.International Journal of Artificial Intelligence in Education,pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1)\.
- S\. AlKhuzaey, F\. Grasso, T\. R\. Payne, and V\. Tamma \(2024\)Text\-based question difficulty prediction: a systematic review of automatic approaches\.International Journal of Artificial Intelligence in Education34\(3\),pp\. 862–914\.Cited by:[§5](https://arxiv.org/html/2608.06609#S5.p3.1)\.
- M\. Amini, B\. Ahmadi, X\. Xiong, Y\. Zhang, and C\. Qiao \(2025\)Prompting strategies for language model\-based item generation in k\-12 education: bridging the gap between small and large language models\.External Links:2508\.20217,[Link](https://arxiv.org/abs/2508.20217)Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1)\.
- Anthropic \(2025\)Claude Sonnet 4\.6\.Note:[https://www\.anthropic\.com](https://www.anthropic.com/)Large language modelCited by:[§3\.3\.1](https://arxiv.org/html/2608.06609#S3.SS3.SSS1.p1.1)\.
- L\. Benedetto, A\. Cappelli, R\. Turrin, and P\. Cremonesi \(2020\)R2DE: a nlp approach to estimating irt parameters of newly generated questions\.InProceedings of the tenth international conference on learning analytics & knowledge,pp\. 412–421\.Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- L\. Benedetto, P\. Cremonesi, A\. Caines, P\. Buttery, A\. Cappelli, A\. Giussani, and R\. Turrin \(2023\)A survey on recent approaches to question difficulty estimation from text\.ACM Computing Surveys55\(9\),pp\. 1–37\.Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1),[§5](https://arxiv.org/html/2608.06609#S5.p3.1)\.
- A\. Birnbaum \(1968\)Some latent trait models and their use in inferring an examinee’s ability\.InStatistical theories of mental test scores,F\. M\. Lord and M\. R\. Novick \(Eds\.\),pp\. 397–479\.Cited by:[§3\.4](https://arxiv.org/html/2608.06609#S3.SS4.p1.3)\.
- R\. Circi, J\. Hicks, and E\. Sikali \(2023\)Automatic item generation: foundations and machine learning\-based approaches for assessments\.Frontiers in EducationVolume 8 \- 2023\.External Links:[Link](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2023.858273),[Document](https://dx.doi.org/10.3389/feduc.2023.858273),ISSN 2504\-284XCited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1)\.
- J\. Devlin, M\. W\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InNAACL HLT 2019 \- 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \- Proceedings of the Conference,Vol\.1\.Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p2.1)\.
- F\. Falcão, P\. Costa, and J\. M\. Pêgo \(2022\)Feasibility assurance: a review of automatic item generation in medical assessment\.Advances in Health Sciences Education27\(2\),pp\. 405–425\.Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1)\.
- W\. Feng, P\. Tran, S\. Sireci, and A\. Lan \(2025\)Reasoning and sampling\-augmented mcq difficulty prediction via LLMs\.arXiv preprint arXiv:2503\.08551\.External Links:[Link](https://arxiv.org/abs/2503.08551)Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p1.1),[§5](https://arxiv.org/html/2608.06609#S5.p4.1)\.
- Y\. Fu, H\. Jiao, T\. Zhou, N\. Zhang, M\. Li, Q\. Xu, S\. Peters, and R\. W\. Lissitz \(2025\)Text\-based approaches to item alignment to content standards in large\-scale reading & writing tests\.External Links:[Link](https://arxiv.org/abs/2509.26431)Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p4.1)\.
- G\. Gorgun and O\. Bulut \(2025\)Instruction\-tuned large\-language models for quality control in automatic item generation: a feasibility study\.Educational Measurement: Issues and Practice44\(1\),pp\. 96–107\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1111/emip.12663),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1111/emip.12663),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1111/emip\.12663Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p2.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p3.1)\.
- T\. M\. Haladyna and S\. M\. Downing \(1989\)Taxonomy of multiple\-choice item\-writing rules\.Applied Measurement in Education2\(1\),pp\. 37–50\.External Links:[Document](https://dx.doi.org/10.1207/s15324818ame0201%5F3)Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1)\.
- S\. Han, F\. Rijmen, A\. A\. Boykin, and S\. Lottridge \(2025\)Leveraging fine\-tuned large language models in item parameter prediction\.InProceedings of the Artificial Intelligence in Measurement and Education Conference \(AIME\-Con\): Full Papers,pp\. 250–264\.Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- J\. Hartmann, M\. Heitmann, C\. Siebert, and C\. Schamp \(2023\)More than a feeling: accuracy and application of sentiment analysis\.International Journal of Research in Marketing40\(1\),pp\. 75–87\.External Links:[Document](https://dx.doi.org/10.1016/j.ijresmar.2022.05.005)Cited by:[§3\.5](https://arxiv.org/html/2608.06609#S3.SS5.p1.3)\.
- P\. He, J\. Gao, and W\. Chen \(2021\)DeBERTaV3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.arXiv preprint arXiv:2111\.09543\.External Links:[Link](https://arxiv.org/abs/2111.09543)Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p2.1),[§3\.2\.2](https://arxiv.org/html/2608.06609#S3.SS2.SSS2.p1.1)\.
- M\. Henrichsen and R\. Krebs \(2025\)Two\-stage reasoning\-infused learning: improving classification with LLM\-generated reasoning\.arXiv preprint arXiv:2507\.00214\.External Links:[Link](https://arxiv.org/abs/2507.00214)Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p1.1),[§5](https://arxiv.org/html/2608.06609#S5.p4.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.arXiv preprint arXiv:2305\.02301\.External Links:[Link](https://arxiv.org/abs/2305.02301)Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p1.1),[§5](https://arxiv.org/html/2608.06609#S5.p4.1)\.
- F\. Hsu, H\. Lee, T\. Chang, and Y\. Sung \(2018\)Automated estimation of item difficulty for multiple\-choice tests: an application of word embedding techniques\.Information Processing & Management54\(6\),pp\. 969–984\.Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.06609#S3.SS2.SSS2.p1.1)\.
- D\. Kolesnikova, K\. Fedyanin, A\. D\. Hofman, M\. J\. Brinkhuis, and M\. Bolsinova \(2026\)Estimating item difficulty with large language models as experts\.arXiv preprint arXiv:2605\.18562\.Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p1.1)\.
- B\. Li \(2026\)Incompressible knowledge probes: estimating black\-box LLM parameter counts via factual capacity\.arXiv preprint arXiv:2604\.24827\.External Links:[Link](https://arxiv.org/abs/2604.24827)Cited by:[§5](https://arxiv.org/html/2608.06609#S5.p4.1)\.
- M\. Li, H\. Jiao, T\. Zhou, N\. Zhang, S\. Peters, and R\. W\. Lissitz \(2025\)Item difficulty modeling using fine\-tuned small and large language models\.Educational and Psychological Measurement85\(6\),pp\. 1065–1090\.Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)Roberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p2.1)\.
- Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, X\. Zhang, N\. D\. Lane, and M\. Xu \(2025\)Small language models: survey, measurements, and insights\.arXiv preprint arXiv:2409\.15790\.External Links:[Link](https://arxiv.org/abs/2409.15790)Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p2.1)\.
- Y\. Ma \(2025\)Analyzing examinee comments using DistilBERT and machine learning to ensure quality control in exam content\.arXiv preprint arXiv:2504\.06465\.External Links:[Link](https://arxiv.org/abs/2504.06465)Cited by:[§5](https://arxiv.org/html/2608.06609#S5.p5.1)\.
- H\. Maeda and Y\. Lu \(2025\)Finding words associated with DIF: predicting differential item functioning using LLMs and explainable AI\.Journal of Educational Measurement62\(4\),pp\. 883–906\(en\)\.External Links:ISSN 1745\-3984,[Link](https://onlinelibrary.wiley.com/doi/abs/10.1111/jedm.70017),[Document](https://dx.doi.org/10.1111/jedm.70017)Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1)\.
- H\. Maeda and Y\. Lu \(2026\)Multimodal test item parameter prediction from text, images, and metadata: fusing together AI vision and language models\.Educational and Psychological Measurement\.External Links:[Document](https://dx.doi.org/10.1177/00131644261460779),[Link](https://doi.org/10.1177/00131644261460779)Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1)\.
- H\. Maeda \(2025\)Field\-testing multiple\-choice questions with AI examinees: english grammar items\.Educational and Psychological Measurement85\(2\),pp\. 221–244\.External Links:[Document](https://dx.doi.org/10.1177/00131644241281053),[Link](https://doi.org/10.1177/00131644241281053),https://doi\.org/10\.1177/00131644241281053Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- E\. Muraki \(1992\)A generalized partial credit model: application of an em algorithm\.ETS Research Report Series1992\(1\),pp\. i–30\.Cited by:[§3\.4](https://arxiv.org/html/2608.06609#S3.SS4.p1.3)\.
- O\. M\. Oluoke, A\. M\. Gorbacheva, and O\. A\. Monday \(2026\)Evaluating LLM\-generated assessment items: a JBI\-guided critical appraisal and checklist\.International Journal of Evaluation and Research in Education\.Note:In pressCited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1)\.
- OpenAI \(2024\)GPT\-4o system card\.arXiv preprint arXiv:2410\.21276\.External Links:[Link](https://arxiv.org/abs/2410.21276)Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p1.1)\.
- R\. Pelánek, T\. Effenberger, A\. Kukučka,et al\.\(2022\)Towards design\-loop adaptivity: identifying items for revision\.Journal of Educational Data Mining14\(3\),pp\. 1–25\.Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p3.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p5.1)\.
- F\. Peng \(2020\)Automatic enemy item detection using natural language processing\.Ph\.D\. Thesis,University of Illinois Chicago\.External Links:[Document](https://dx.doi.org/10.25417/uic.14134559.v1),[Link](https://doi.org/10.25417/uic.14134559.v1)Cited by:[§5](https://arxiv.org/html/2608.06609#S5.p6.1)\.
- J\. Prentzas and A\. Binopoulou \(2025\)Explainable artificial intelligence approaches in primary education: a review\.Electronics14\(11\)\.External Links:[Link](https://www.mdpi.com/2079-9292/14/11/2279),ISSN 2079\-9292,[Document](https://dx.doi.org/10.3390/electronics14112279)Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p2.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p2.1),[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p3.1),[§3\.2\.1](https://arxiv.org/html/2608.06609#S3.SS2.SSS1.p1.1),[§3\.2\.3](https://arxiv.org/html/2608.06609#S3.SS2.SSS3.p1.1)\.
- A\. Scarlatos, N\. Fernandez, C\. Ormerod, S\. Lottridge, and A\. Lan \(2025\)Smart: simulated students aligned with item response theory for question difficulty prediction\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 25082–25105\.Cited by:[§2\.3](https://arxiv.org/html/2608.06609#S2.SS3.p1.1),[§5](https://arxiv.org/html/2608.06609#S5.p4.1)\.
- J\. Shin and M\. J\. Gierl \(2024\)Automated short\-response scoring for automated item generation in science assessments\.InThe Routledge International Handbook of Automated Essay Evaluation,pp\. 504–534\.Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p2.1)\.
- B\. Tan, N\. Armoush, E\. Mazzullo, O\. Bulut, and M\. Gierl \(2025\)A review of automatic item generation techniques leveraging large language models\.International Journal of Assessment Tools in Education12\(2\),pp\. 317–340\.Cited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1)\.
- W\. J\. van der Linden and P\. J\. Pashley \(2000\)Item selection and ability estimation in adaptive testing\.InComputerized Adaptive Testing: Theory and Practice,pp\. 1–25\.External Links:ISBN 978\-0\-306\-47531\-3,[Document](https://dx.doi.org/10.1007/0-306-47531-6%5F1),[Link](https://doi.org/10.1007/0-306-47531-6_1)Cited by:[§5](https://arxiv.org/html/2608.06609#S5.p6.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.arXiv preprint arXiv:1706\.03762\.External Links:[Link](https://arxiv.org/abs/1706.03762)Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- Y\. Wang, M\. Gopalakrishnan, and Y\. Bergner \(2025\)Using generated rubrics to provide a window into item evaluation with multi\-agent LLMs\.InInternational Conference on Artificial Intelligence in Education,pp\. 203–217\.Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p2.1)\.
- B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. Poli \(2024\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.arXiv preprint arXiv:2412\.13663\.External Links:[Link](https://arxiv.org/abs/2412.13663)Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p2.1)\.
- K\. Wauters, P\. Desmet, and W\. Van Den Noortgate \(2012\)Item difficulty estimation: an auspicious collaboration between data and judgment\.Computers & Education58\(4\),pp\. 1183–1193\.Cited by:[§2\.2](https://arxiv.org/html/2608.06609#S2.SS2.p1.1)\.
- V\. Yaneva, L\. A\. Ha, P\. Baldwin, and J\. Mee \(2020\)Predicting item survival for multiple choice questions in a high\-stakes medical exam\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 6812–6818\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.841/),ISBN 979\-10\-95546\-34\-4Cited by:[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p5.1)\.
- M\. C\. Yu and M\. I\. Burke \(2026\)Automatic item generation, evaluation, and scale construction of non\-cognitive measures with generative language models\.InArtificial Intelligence for I\-O Psychologists: Research and Applications,I\. Thompson, G\. P\. Yankov, and I\. Hernandez \(Eds\.\),External Links:ISBN 9780197807279,[Document](https://dx.doi.org/10.1093/9780197807309.003.0010),[Link](https://doi.org/10.1093/9780197807309.003.0010),https://academic\.oup\.com/book/0/chapter/561205002/chapter\-pdf/68115481/isbn\-9780197807309\-book\-part\-10\.pdfCited by:[§1](https://arxiv.org/html/2608.06609#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.06609#S2.SS1.p2.1)\.

## Appendix Appendix ARejection Reason Definitions

1. 1\.Content: content issue or error, including but not limited to \(1\) misalignment with the blueprint or standards, \(2\) unrealistic, inconsistent, or confusing scenarios, or \(3\) basic issues with the text or image such as grammar errors or missing information\. A large portion of these items are rejected at a content review event just prior to field testing\.
2. 2\.Psychometric: The following statistical criteria result in immediate rejection without manual review: proportion correct below \.03, item\-total correlation at or below \.05, a distractor with a higher item\-total correlation than the correct answer, poor inter\-rater reliability on hand\-scored items \(exact agreement less than 75%, 65%, or 55% on items with 2, 3, or greater score points, respectively\), or polytomous items where mean estimatedθ\\thetadoes not increase monotonically across score levels\. Items can also be rejected for less severe psychometric issues, though these require review and situational judgment: low or negative discrimination, too easy, too difficulty, insufficient sample size, overexposure, parameter drift, or differential item functioning\.
3. 3\.Data review: These are items with questionable psychometric qualities that needed to be reviewed by content experts\. Items flagged with C\-level differential item functioning are always reviewed by content experts\.
4. 4\.Bias: Content bias, sensitivity, fairness, or accessibility reasons
5. 5\.Abandoned: Items that could not find a suitable use in an exam\. Many tend to be extra items in passage sets\. Sometimes, the item itself has no content issues\.
6. 6\.Incomplete: Items stuck in the item development process\. Never reached field\-testing\.
7. 7\.Passage: The entire passage set \(i\.e\., stimulus or testlet\) was rejected\. Sometimes, the item itself has no content issues\.
8. 8\.Scoring: Any issue related to scoring rubric, answer key, scoring logic, or scoring meta data\. This can be a combination of content, data integrity, and psychometric issues\.
9. 9\.Non\-content: Issues unrelated to the text or images of the item itself, such as missing metadata, corrupt or malformed data, format incompatibilities, import errors, or expired copyright permissions\. Because these are primarily data integrity issues, text\-based models are not expected to detect them\.
10. 10\.\(no data\): Items rejected without comments or unclear reasons\.

## Appendix Appendix BQwen3 Zero\-Shot Classification Prompt

Below is an example python script for generating the Qwen3\-0\.6B zero\-shot classification prompt\.

SYSTEM\_PROMPT=\(

"Youareanexpertassessmentitemreviewer\.Youwillbegivenanassessmentitem\.Respondwithexactlyoneword:’accept’or’reject’\.Respond’accept’iftheitemisusableforassessmentas\-is\.Respond’reject’iftheitemhasissuesthatgobeyondasimplefix,suchas:fundamentalambiguity,significantbiasorsensitivityconcerns,clearmisalignmenttothestatedstandard,orbroken/unfixableformatting\.Donotincludeanyexplanation,punctuation,oradditionaltext\."

\)

defmake\_messages\(row\):

return\{

"messages":\[

\{"role":"system","content":SYSTEM\_PROMPT\},

\{"role":"user","content":f"Thisisa\{row\[’subject’\]\}subject,grade\{row\[’grade’\]\}item:\\n\{row\[’text’\]\}\\n\{row\[’standards’\]\}\\nAnyrelevantimages,figures,ortableshavebeenexcluded\."

\}\]\}

## Appendix Appendix CQwen3 Item Critique Prompt

Below is an example python script for generating the Qwen3\-0\.6B item critique prompt\.

SYSTEM\_PROMPT=\(

"Youareanexpertassessmentitemreviewerwithdeepknowledgeofpsychometricsanditemdevelopmentbestpractices\.Youwillbegivenanassessmentitem\.Reviewtheitemforcommonqualityissuessuchas:ambiguity,itemdifficulty,culturalordemographicbiasandsensitivity,contentalignmenttostandards,andflawedgrammarorformatting\.Writeaconcise2sentencesummary:iftheitemhasissues,namethemplainlyandbrieflyexplainwhytheymatter;iftheitemlookssound,saysoandnoteitsstrongestquality\.Donotusebulletpointsorheaders\.Bedirectandspecific\-avoidvaguepraiseorvaguecriticism\."

\)

defmake\_messages\(row\):

return\{

"messages":\[

\{"role":"system","content":SYSTEM\_PROMPT\},

\{"role":"user","content":f"Thisisa\{row\[’subject’\]\}subject,grade\{row\[’grade’\]\}item:\\n\{row\[’text’\]\}\\n\{row\[’standards’\]\}\\nAnyrelevantimagesandfiguresortableshavebeenexcluded\.\\nWriteaconcise2sentencesummaryabouttheitemquality,includingbutnotlimitedtoitsitemdifficulty,biasandsensitivity,andcontentalignmenttostandards\."

\}\]\}

## Appendix Appendix DExample Qwen3 Critiques

Table 2:Example Critiques Generated using Qwen3\-0\.6B\.*Note\.*Status = true status label, Sentiment = sentiment logit \(positive = positive sentiment, negative = negative sentiment\),Pf​u​s​i​o​nP\_\{fusion\}= fusion model probability of rejection,P​ΔP\\Delta=Pf​u​s​i​o​nP\_\{fusion\}minus text\-only model probability of rejection, ELA = English language arts\. Two critiques from accepted and rejected ELA and math items were randomly selected, with the requirement that one had a near\-zeroP​ΔP\\Delta, and another hadP​Δ\>\.3P\\Delta\>\.3\.

## Appendix Appendix EClassification with \.25 Cutoff

Table 3:Classification Performance by Model with \.25 Cutoff Threshold*Note\.*ELA = English language arts, p = proportion of items predicted as reject, AUC = area under the curve\. Cutoff threshold for models were fixed to \.25\.