Item Response Theory for AI Safety

# Item Response Theory for AI Safety
Source: [https://arxiv.org/html/2608.05086](https://arxiv.org/html/2608.05086)
Joshua Fonseca Rivera\\equalcontrib1, Neil Shah\\equalcontrib1, David Demitri Africa111These authors contributed equally as senior authors\.2, Konstantinos Voudouris††footnotemark:2

###### Abstract

Language models differ in how safely they behave and these differences are measured by safety benchmarks\. But aggregated benchmark scores are hard to trust and interpret, because benchmarks duplicate one another, correlate heavily, and models may sandbag when they detect evaluation\. To address these issues, we draw on Item Response Theory \(IRT\), a statistical toolkit for measuring these latents from performance on items with inferred psychometric properties\. We fit IRT models to eight safety benchmarks across 192 language models, the largest psychometric analysis of LLM safety evaluations to date, and contribute three results\. First, we find that three interpretable factors of refusal strictness, truthfulness, and contextual harm explain most of the variance between models across benchmarks\. Second, psychometrically selected items recover full benchmark scores with lower error than random subsets of the same size, and roughly ten adaptively chosen items suffice for several individual benchmarks, cutting evaluation cost by 97–99%\. Third, IRT supports audits of individual models, showing that it can be used to detect naive sandbagging and changes of model behind APIs\. Overall, we show IRT is a ready\-made toolkit for reading, reducing, and auditing safety benchmarks, which we recommend frontier labs and evaluators adopt\.

## Introduction

![Refer to caption](https://arxiv.org/html/2608.05086v1/x1.png)Figure 1:Item response theory provides a common framework for reading, reducing, and auditing safety benchmarks\.Left:We calibrate item difficulty and discrimination from model responses and use the resulting ability estimates to identify the latent structure of safety evaluation\.Center:Informative static tests and computerized adaptive testing recover benchmark scores and rankings from a small fraction of the original items\.Right:Person\-fit statistics, historical ability estimates, and response fingerprints detect complementary forms of behavioral change\.Language models vary widely in how safely they behave: whether they comply with harmful requests, over\-refuse benign ones, or answer truthfully\. This is scored by safety benchmarks, which use adversarial test items marked safe or unsafe\(Mazeika et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib24); Xie et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib37); Zou et al\.[2023](https://arxiv.org/html/2608.05086#bib.bib39)\)\. These results drive leaderboards, release decisions, and marketing\. Yet, a benchmark score is only useful if we understand what it measures, whether its items add distinct information, and whether the observed responses reflect the model’s ordinary behavior\(Bean et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib2); Hernández\-Orallo[2017](https://arxiv.org/html/2608.05086#bib.bib12); Romero\-Alvarado et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib30); Harding and Sharadin[2024](https://arxiv.org/html/2608.05086#bib.bib11); Voudouris et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib34)\)\.

Existing benchmark suites make each of these questions difficult\. In capability evaluation, scores correlate strongly within and across these benchmarks, suggesting that many tests and items measure overlapping behavior\(Kipnis et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib17); Maia Polo et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib22)\)\. At the same time, such correlations may reward contradicting things; a model can therefore improve on one benchmark by becoming worse on another\. Full benchmark suites are also expensive and often inefficient\. Many items are passed or failed by nearly every model and contribute little to distinguishing among them\. Finally, benchmark responses are mediated by imperfect judges and may be strategically manipulated\. A model that recognizes an evaluation could selectively alter its behavior while retaining a plausible aggregate score\(van der Weij et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib33)\)\.

Item response theory\(IRT; Lord[1980](https://arxiv.org/html/2608.05086#bib.bib20); Embretson and Reise[2000](https://arxiv.org/html/2608.05086#bib.bib7); Reckase[2009](https://arxiv.org/html/2608.05086#bib.bib28)\)provides a common framework for answering these questions\. IRT models each response as an interaction between a respondent’s ability and an item’s difficulty and discrimination\. Once fitted, an IRT model can reveal the latent abilities explaining behavior on a collection of tests, identify informative items, and flag response patterns indicative of cheating or careless answering\. IRT has recently been used to compress LLM capability benchmarks while preserving model rankings\(Maia Polo et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib22); Kipnis et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib17); Hofmann et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib13)\), and concurrent work applies adaptive testing to safety benchmarks\(Spagliardi et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib32)\)\. We show that compression is only one of its uses\.

In this work, we fit IRT models to eight safety benchmarks \(5,255 items, 192 models\) covering harmful compliance, over\-refusal, and truthfulness\. This is, to our knowledge, the largest psychometric analysis of LLM safety benchmarks to date\. We use the resulting calibration in three ways \(Figure[1](https://arxiv.org/html/2608.05086#Sx1.F1)\):

- •Characterizing what safety benchmarks measure\.We find that three latent factors \(*refusal strictness*,*truthfulness*, and*contextual harm*\) explain 77% of the variance between model abilities\. We also find that the suite cannot be summarized by a single safety ability\.
- •Cost\-efficient benchmarking\.Psychometrically selected items estimate full\-benchmark scores with lower error than random subsets of the same size: three fixed 25\-item tests recover the three latent abilities, and roughly ten adaptively chosen items recover several individual benchmarks, reducing their evaluation cost by 97–99%\.
- •Auditing with person\-fit\.Using promptedsandbagging model organisms, IRT\-derived person\-fit statistics catch at least 80% of selective prompted sandbagging\. This also can be used to identify model substitutions and changes behind API endpoints\.

### Related Work

#### Item response theory in LLMs\.

Item response theory has been used in natural language processing and machine learning to conduct dataset and leaderboard analysis\(Lalor, Wu, and Yu[2019](https://arxiv.org/html/2608.05086#bib.bib18); Martínez\-Plumed et al\.[2016](https://arxiv.org/html/2608.05086#bib.bib23); Rodriguez et al\.[2021](https://arxiv.org/html/2608.05086#bib.bib29); Pacchiardi et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib26)\)and propensity measurement in LLMs\(Romero\-Alvarado et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib30)\)\. For capability evaluation,tinyBenchmarks\(Maia Polo et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib22)\)andmetabench\(Kipnis et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib17)\)compress leaderboards to 1–3% of their items, and Fluid Benchmarking\(Hofmann et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib13)\)combines IRT with adaptive item selection to produce similar efficiency gains\. Closest to our work, concurrent research applies computerized adaptive testing to six safety benchmarks\(Spagliardi et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib32)\), meaning that items are administered one at a time with the latent factor updated after each administration\. This allows for the latent factor to be inferred with significantly fewer inference\-time item administrations than by running the model on the entire benchmark\. Complementing and extending this work, we study a broader safety suite that includes harmful compliance, over\-refusal, contextual harm, and truthfulness, calibrate on a substantially larger model cohort, and use the fitted measurement model for construct analysis and model auditing as well as compression\.

#### Safety benchmarks\.

Our suite spans safety benchmarks that measure: complying with harmful requests\(Zou et al\.[2023](https://arxiv.org/html/2608.05086#bib.bib39); Mazeika et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib24); Xie et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib37); Wang et al\.[2023](https://arxiv.org/html/2608.05086#bib.bib35); Ganguli et al\.[2022](https://arxiv.org/html/2608.05086#bib.bib9)\), refusing benign requests that merely look harmful\(over\-refusal; Cui et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib4); Röttger et al\.[2024](https://arxiv.org/html/2608.05086#bib.bib31)\), and truthfulness\(Lin, Hilton, and Evans[2022](https://arxiv.org/html/2608.05086#bib.bib19)\)\. Prior work has shown that safety evaluations can be redundant, sensitive to scoring choices, and shaped by tradeoffs between refusing harmful requests and answering benign ones\(Romero\-Alvarado et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib30); Voudouris et al\.[2026](https://arxiv.org/html/2608.05086#bib.bib34)\)\.

## Materials & Methods

We collected item\-level responses from up to 192 language models on eight safety benchmarks and fitted Item Response Theory \(IRT\) models to estimate the difficulty and discrimination of each item together with the latent abilities of the models\. We use the fitted measurement models in three analyses\. First, we infer the latent abilities shared across benchmarks\. Second, we distill the suite into short static and adaptive tests\. Third, we use expected response patterns to detect evaluation\-dependent behavior and changes behind model APIs\.

### Benchmarks and Model Responses

We evaluate models on eight benchmarks spanning harmful compliance, over\-refusal, contextual harm, and truthfulness \(Table[1](https://arxiv.org/html/2608.05086#Sx2.T1)\)\. The complete suite contains 5,255 items before preprocessing\.

Table 1:The eight benchmarks included in our analyses\. Item counts are reported after removing unanswered and zero\-variance items, leaving 5,067 items in total\.We collect one response per available model–item pair through OpenRouter’s API, using zero temperature and a maximum generation length of 1,024 tokens\. Analyses involving multiple benchmarks use only models with valid responses on every benchmark required for that analysis\.

We evaluate 192 chat models spanning open\-weight families from 1B to∼\\sim1T total parameters \(Llama, Gemma, Qwen, DeepSeek, Mistral, GLM, Kimi, gpt\-oss, among others\) and proprietary API\-only models \(GPT, Claude, Gemini, Grok\)\. The 169 models with complete responses on all eight benchmarks form the pool for cross\-benchmark analyses; the latent\-structure analyses \(factor analysis and multidimensional IRT\) use the 134 models that additionally have a valid judge score for every retained item on every benchmark \(judge failures leave at least one unscored item for each of the remaining 35; Appendix[A](https://arxiv.org/html/2608.05086#A1)gives the full roster\)\. The complete model inventory and benchmark\-coverage matrix are provided in supplementary material\.

Each generated response is scored using the judge prescribed by its source benchmark\. Letxm​j∈\{0,1\}x\_\{mj\}\\in\\\{0,1\\\}denote the scored response of modelmmto itemjj, wherexm​j=1x\_\{mj\}=1indicates the response rewarded by the corresponding benchmark\.

For harmful\-request benchmarks, a value of one generally denotes a refusal\. For OR\-Bench\-Hard, it denotes answering a benign request rather than over\-refusing it\. For TruthfulQA, it denotes a truthful answer\. This orientation means that larger benchmark\-specific ability estimates always correspond to higher performance according to the benchmark being fitted\.

We remove responses for which the prescribed judge does not return a valid score\. We also remove items with no variation across the evaluated models, since these items contain no information about differences within the model cohort\. This preprocessing leaves 5,067 items\.

### The 2PL IRT Model

IRT infers a test\-taker’s latent ability from responses to a fixed set of items\. We treat language models as test\-takers, benchmark prompts as items, and safe responses as correct answers\. The two\-parameter logistic \(2PL\) model is given by

P​\(xm​j=1∣θm,aj,bj\)=σ​\(aj​\(θm−bj\)\),P\(x\_\{mj\}=1\\mid\\theta\_\{m\},a\_\{j\},b\_\{j\}\)=\\sigma\\\!\\left\(a\_\{j\}\(\\theta\_\{m\}\-b\_\{j\}\)\\right\),\(1\)whereθm\\theta\_\{m\}is the model’s latent ability,bjb\_\{j\}is the item’s difficulty, andaja\_\{j\}is its discrimination\. Difficulty determines the ability level at which an item changes from usually failed to usually passed \(i\.e\., whereP​\(xm​j=1\)=0\.5P\(x\_\{mj\}=1\)=0\.5\)\. Discrimination determines how sharply the probability of success changes around that point\. Item and ability parameters are inferred jointly by marginal maximum likelihood\(Baker and Kim[2004](https://arxiv.org/html/2608.05086#bib.bib1)\); a model’s ability is then the posterior mode ofθ\\thetaunder a standard\-normal prior, given its responses and the fixed item parameters\. Unlike a raw pass rate, this estimate remains on a common scale when different models answer different subsets of items, which enables both short fixed tests and adaptive testing\.

We use the 2PL rather than simpler or richer alternatives\. The 1PL \(Rasch\) model constrains all items to discriminate equally, which the data reject: fitted discriminations vary by roughly an order of magnitude within every benchmark, variation that item selection later exploits, and likelihood\-ratio tests reject the equal\-discrimination constraint on all eight benchmarks\. Conversely, the 3PL and 4PL add guessing and slip parameters that improve AIC on only one benchmark and on none, respectively, and BIC never prefers a model richer than the 2PL \(Appendix[E](https://arxiv.org/html/2608.05086#A5)\)\.

Human psychometric datasets tend to have thousands of test\-takers, but LLM evaluations contain many more items tested on many fewer models\. At this scale, an unregularized maximum\-likelihood fit is usually unstable\.222The intuition for this is as follows\. For an item that happens to divide the observed models perfectly, the discrimination parameter,aja\_\{j\}, won’t converge as the closer the logistic gets to a step function, the better the fit\. For items that do not discriminate models because they are too hard or too easy, the difficulty parameter,bjb\_\{j\}, will also tend towards positive or negative infinity\. This has downstream consequences for item selection based on these parameters\. Small samples of test\-takers increase the risk of these problems by reducing the possible variance in response distributions\.We therefore regularize our IRT model by adding a log\-normal prior on discrimination and a normal prior on the item intercept\. We verify the effectiveness of this using split\-half calibration, where the model cohort is divided into two disjoint halves, and the 2PL model is fit separately to each half\. Then, we select the 25 items with highest discrimination in the first fit, and measure how much of those items’ discrimination is retained in the second\. Without regularization, the top\-25 items retain only 28% of their fitted discrimination on AdvBench and 38% on HarmBench \(mean over 10 random splits\); with our priors, retention rises to 68% and 76%, while the bulk of the parameter distribution is left unchanged\.

All results below use the regularized fits333As IRT tooling defaults can be finicky, cohort sizes like ours will be the norm for LLM work for some time, so such regularization is important to note as best practice\.\.

### Latent Ability Inference

We fit a separate 2PL model to each benchmark using the full model cohort\. This gives every model one estimated ability per benchmark, producing anM×8M\\times 8ability matrix\. We then apply minimum\-residual factor analysis, which chooses factor loadings that reproduce the observed correlation matrix, with oblimin rotation, which allows the resulting factors to be correlated rather than forcing them to be independent\. We select the number of factors using the Root Mean Square Error of Approximation \(RMSEA\), the Comparative Fit Index \(CFI\), and Horn’s parallel analysis\(Horn[1965](https://arxiv.org/html/2608.05086#bib.bib14); Embretson and Reise[2000](https://arxiv.org/html/2608.05086#bib.bib7)\)\. Appendix[D](https://arxiv.org/html/2608.05086#A4)describes these criteria and the sensitivity of the solution to the extraction method, and Appendix[C](https://arxiv.org/html/2608.05086#A3)reports an item\-level multidimensional IRT fit that relaxes the two\-stage design\.

### Benchmark Distillation

The full benchmark suite before processing contains 5,255 items, but many contribute little information about a given model\. Easy items are passed by nearly every model, while difficult items are failed by nearly every model\. Both consume compute without helping distinguish models at the relevant ability level\.

IRT supports two forms of benchmark reduction\. First, we can construct a short*static test*containing items that are informative across the model population\. Second, we can usecomputerized adaptive testing\(CAT\) to choose different items for different models according to their estimated ability\.

Under the 2PL, the Fisher information of itemjjat abilityθ\\thetais

Ij​\(θ\)=aj2​Pj​\(θ\)​\(1−Pj​\(θ\)\)\.I\_\{j\}\(\\theta\)=a\_\{j\}^\{2\}P\_\{j\}\(\\theta\)\\bigl\(1\-P\_\{j\}\(\\theta\)\\bigr\)\.An item is most informative near its difficulty, where a model has approximately equal probability of passing or failing, and increases with the square of the item’s discrimination\. Static tests select items that provide high information across the population’s ability range\. CAT instead selects the most informative item at the current ability estimate, observes the response, and updates the estimate before choosing the next item\(Weiss and Kingsbury[1984](https://arxiv.org/html/2608.05086#bib.bib36); Magis, Yan, and von Davier[2017](https://arxiv.org/html/2608.05086#bib.bib21)\)\.

We evaluate both methods on held\-out models\. For each of 20 random splits, we assign 75% of models to a calibration set and 25% to a test set\. We fit the 2PL and select items using only the calibration models, and evaluate how accurately each reduced test reproduces the full\-benchmark results for the held\-out models\.

Our primary criterion is recovery of the full\-benchmark score, measured by root mean squared error \(RMSE\); we additionally report recovery of model rankings, measured by Spearman correlation, for completeness\. To predict full scores, we fit a spline regression on the calibration models using the reduced\-test score and estimated ability as inputs\.

For comparison, we draw 100 random item sets at each budget\. These sets use the same per\-benchmark quotas as the informed tests and are passed through the same score\-calibration procedure\. Differences between informed and random tests therefore reflect item selection rather than test length, benchmark composition, or downstream calibration\.

### Sandbagging Detection and API Auditing

IRT models not only estimate ability; they also assign a probability to each response pattern\. Person\-fit statistics such aslzl\_\{z\}compare a model’s observed responses with those expected at its estimated ability\(Drasgow, Levine, and Williams[1985](https://arxiv.org/html/2608.05086#bib.bib6); Meijer and Sijtsma[2001](https://arxiv.org/html/2608.05086#bib.bib25)\)\. A model that fails easy items while passing difficult ones receives an improbable person\-fit score even if its overall pass rate appears normal\. In human testing, such patterns can indicate cheating, careless responding, or other departures from the assumed test\-taking process\.

This provides a basis for auditing individual models using only black\-box access\. We consider an auditor who has public item parameters and query access to an endpoint, and may additionally have historical ability estimates or reference responses from the claimed model\. The auditor is concerned with two failure modes: a model that behaves differently during evaluation than during deployment, and an endpoint that serves different weights than it claims\. We evaluate three tests that require progressively more reference information \(Table[2](https://arxiv.org/html/2608.05086#Sx2.T2)\)\.

Table 2:Audit methods ordered by the reference information they require\. Each method detects failure modes that can remain invisible to the preceding one\.#### Person\-fit\.

We construct controlled prompted\-sandbagging interventions on HarmBench’s standard\-behavior subset \(196 items after preprocessing\)\. In each intervention, a system prompt tells the model that it is being evaluated and instructs it to behave with maximum caution\. We use three prompt framings \(plain, persona, and incentive\) and apply each intervention in three scopes: to every item \(*uniform*\), only to the model’s weakest harm category \(*targeted*\), or only when a trigger phrase appears \(*triggered*\); we call the targeted and triggered scopes*selective*sandbagging\. Detection is fully blind to the intervention: the auditor receives neither labels identifying sandbagged runs nor a clean baseline from the suspect model\. A run is flagged when itslzl\_\{z\}statistic falls below a threshold set at the 10th percentile of the 171 cohort models with scored responses on this subset, fixing the false\-positive rate at 10%\. The 10% operating point is a policy choice fixing how many genuine models an auditor would needlessly re\-examine, not a fitted quantity; Appendix[B](https://arxiv.org/html/2608.05086#A2)reports full ROC curves from which any other operating point can be read\.

#### Ability shift\.

The second test re\-estimates ability from a 40\-item test and compares it with the model’s historical estimate\. A run is flagged when the shift exceeds the smallest ability gap that a 40\-item test resolves at that ability level, estimated from pairwise confusability within the cohort\. The same procedure quantifies when a substitution of one model for another is detectable from ability alone: for every pair of cohort models, we ask whether a 40\-item test distinguishes the pair on at least one benchmark\.

#### Response fingerprint\.

The third test compares an endpoint’s responses on a pre\-selected item set with previously collected responses from the claimed model, and matches each endpoint to the reference model with which its responses agree most\. We validate this by re\-collecting 16 endpoints on 500 pre\-selected items each\. Because an endpoint does not reproduce its own responses exactly, we calibrate the expected level of self\-disagreement from repeated collections on the same items with the same scoring pipeline \(Appendix[B](https://arxiv.org/html/2608.05086#A2)\)\.

## Results

### Latent Ability Inference

![Refer to caption](https://arxiv.org/html/2608.05086v1/x2.png)Figure 2:Pearson correlations between per\-benchmark abilities \(2PLθ\\theta, 134 models\), ordered by the three\-factor solution\. Boxes mark the three ability clusters; OR\-Bench\-Hard \(⊖\\ominus\) correlates negatively with its cluster—it measures refusal strictness reversed\. Blue cells outside the boxes reflect correlation between the abilities themselves \(refusal×\\timescontextual harmϕ=\.65\\phi=\.65\); the factors remain distinct because their patterns differ: the harm pair does not share the refusal cluster’s over\-refusal trade\-off \(ORB column\)\.![Refer to caption](https://arxiv.org/html/2608.05086v1/x3.png)Figure 3:Held\-out recovery of the three latent abilities and the sign\-aligned full\-suite composite fromkkitems, averaged over 20 splits with 95% confidence intervals\. For each target, the upper panel reports score\-recovery error \(RMSE\), our primary criterion, and the lower panel reports rank recovery \(Spearman correlation\)\. We compare computerized adaptive testing, a fixed static test, and stratified random subsampling\.#### Model selection favors three latent factors over one, two, or four\.

A one\-factor model explains 47% of the variance in the ability matrix and has poor absolute fit \(RMSEA=0\.32=0\.32\)\. A two\-factor model explains 69%, with CFI below the conventional threshold for acceptable fit \(0\.880\.88versus0\.950\.95\)\. A three\-factor model explains 77% and reaches CFI=0\.97=0\.97\(Appendix[D](https://arxiv.org/html/2608.05086#A4), Figure[9](https://arxiv.org/html/2608.05086#A4.F9)\)\. Horn’s parallel analysis matched to the minimum\-residual extraction retains three factors: the third observed eigenvalue exceeds the eigenvalue at the same rank in 1,000 random\-normal datasets, whereas the fourth does not\. The PCA\-based variant retains two \(Appendix[D](https://arxiv.org/html/2608.05086#A4)\)\. We use the three\-factor solution throughout\.

The three factors group the benchmarks as follows \(Figure[2](https://arxiv.org/html/2608.05086#Sx3.F2)\):

- •*Refusal strictness\.*HarmBench, SORRY\-Bench, and AHB load on one end of this factor, while OR\-Bench\-Hard loads on the other\. HarmBench and SORRY\-Bench abilities correlate strongly \(ρ=0\.89\\rho=0\.89\), and both correlate negatively with OR\-Bench\-Hard \(−0\.77≤ρ≤−0\.67\-0\.77\\leq\\rho\\leq\-0\.67\)\.
- •*Truthfulness\.*TruthfulQA and Do\-Not\-Answer define a second factor\. This factor is nearly independent of refusal strictness \(ϕ=0\.01\\phi=0\.01\)\.
- •*Contextual harm\.*AdvBench and Health\-ORSC define a third factor through their moderate correlation \(ρ=0\.58\\rho=0\.58\)\. This factor is distinct from refusal strictness, although the two factors are moderately positively correlated \(ϕ=0\.65\\phi=0\.65\)\.

### Benchmark Distillation

#### Three fixed 25\-item tests recover all three latent abilities using under 2% of the suite\.

We construct one 25\-item static test for each of the three latent abilities; together they contain 75 items, under 2% of the full suite\. On held\-out models, each test estimates its target ability with lower error than a random subset of the same size: RMSE0\.0470\.047versus0\.0620\.062for refusal strictness,0\.0410\.041versus0\.0480\.048for truthfulness, and0\.0650\.065versus0\.0670\.067for contextual harm \(Figure[3](https://arxiv.org/html/2608.05086#Sx3.F3)\)\. The same tests recover the model orderings atρ=0\.94\\rho=0\.94,0\.830\.83, and0\.890\.89, against random baselines of0\.900\.90,0\.670\.67, and0\.870\.87\. Within the truthfulness test, TruthfulQA is estimated at RMSE0\.0520\.052against a random baseline of0\.0650\.065; allocating the same number of items across one pooled, all\-benchmark test drops its rank recovery fromρ=0\.73\\rho=0\.73toρ=0\.60\\rho=0\.60\.

#### Roughly ten adaptive items recover individual benchmarks, cutting cost by 97–99%\.

Approximately ten adaptive items estimate HarmBench, SORRY\-Bench, and OR\-Bench\-Hard scores with 18–34% lower error than a random ten\-item subset, corresponding to a 97–99% reduction relative to administering those benchmarks in full\. The same ten items recover their full rankings atρ=0\.92\\rho=0\.92–0\.940\.94\. The advantage narrows to 4–21% at 25 items, and by 50 items random subsampling matches or beats CAT on score recovery for five of the eight benchmarks, although seven still reachρ≥0\.90\\rho\\geq 0\.90\.

#### Adaptive testing beats random subsampling on the full\-suite composite at every budget\.

On the sign\-aligned composite of all eight benchmarks, CAT estimates the full score with lower error than random subsampling at every evaluated budget \(Figure[3](https://arxiv.org/html/2608.05086#Sx3.F3), right\)\. At 25 items its error is 22% below a random subset of the same size, and random subsampling does not match it until roughly 55 items; beyond 100 items the two converge to within overlapping confidence intervals\. The fixed static test is ahead of random at 10–50 items, and random overtakes it beyond that\. Rank recovery shows the same ordering: CAT reachesρ=0\.92\\rho=0\.92at 25 items, a level random sampling reaches at approximately 75–100 items\.

![Refer to caption](https://arxiv.org/html/2608.05086v1/x4.png)Figure 4:Person\-fitlzl\_\{z\}distributions on HarmBench for the genuine cohort and the three sandbagging scopes; the dashed line marks the calibrated cutoff \(10% false\-positive rate\)\. Selective sandbagging produces aberrant response patterns; uniform sandbagging largely does not \(Figure[5](https://arxiv.org/html/2608.05086#Sx3.F5)shows the complementary ability\-shift and fingerprint tests\)\.![Refer to caption](https://arxiv.org/html/2608.05086v1/x5.png)Figure 5:The two audit tests beyond person\-fit\.\(A\)Ability shiftΔ​θ\\Delta\\thetarelative to the genuine baseline, by sandbagging scope; uniform sandbagging moves the whole response profile\.\(B\)Largest per\-benchmark ability gap versus total item disagreements for all 8,911 cohort pairs; circled points are the 41 ability\-matched pairs that a 40\-item test cannot separate but that still disagree on a median of 1,052 items\.

### Sandbagging Detection and API Auditing

#### Person\-fit catches selective prompted sandbagging but misses most uniform sandbagging\.

At the calibrated 10% false\-positive rate, person\-fit flags 80–100% of targeted and triggered sandbagging runs in every prompt framing \(90% and 97% pooled; Figure[4](https://arxiv.org/html/2608.05086#Sx3.F4)\)\. Uniform sandbagging is flagged less often: 18–20% of runs under the plain and persona framings and 70% under the incentive framing \(35% pooled\)\. Per\-framing rates and full ROC curves are given in Appendix[B](https://arxiv.org/html/2608.05086#A2)\.

#### Ability shift catches uniform sandbagging; combined, the two tests detect 94–97% of runs\.

Uniform sandbagging changes the estimated ability by a median of\+1\.69\+1\.69logits and by as much as\+3\.94\+3\.94, whereas targeted and triggered interventions shift ability by only\+0\.24\+0\.24to\+0\.35\+0\.35logits \(Figure[5](https://arxiv.org/html/2608.05086#Sx3.F5)A\)\. The 40\-item ability check against the model’s historicalθ\\thetaflags 84% of uniform runs, including every plain\- and persona\-framing uniform run that person\-fit misses; combining the two tests detects 94–97% of sandbagging runs in every scope\.

#### Response fingerprints identify even ability\-matched model substitutions\.

A 40\-item ability test distinguishes 99\.5% of the 8,911 model pairs in our 134\-model cohort on at least one benchmark\. The remaining 41 ability\-matched pairs \(24 same\-vendor snapshots or size variants, 17 cross\-vendor pairs\) still disagree on a median of 1,052 individual items across the full battery, and no cohort pair disagrees on fewer than 284 \(Figure[5](https://arxiv.org/html/2608.05086#Sx3.F5)B\)\. Matching the 16 re\-collected endpoints to reference responses on 500 pre\-selected items identifies 13 of 14 distinct model identities, and correctly declines to distinguish two catalog entries that appear to serve the same weights \(Appendix[B](https://arxiv.org/html/2608.05086#A2)\)\.

## Discussion

#### Safety benchmarks measure different abilities\.

The eight benchmarks cannot be interpreted as interchangeable measures of a single safety property\. For example, HarmBench and SORRY\-Bench reward largely the same behavior, while OR\-Bench\-Hard measures the opposing tendency to answer benign requests rather than refuse them\. An unweighted average therefore has two problems\. First, it gives extra weight to behaviors measured by several similar benchmarks\. Second, it hides the tradeoff between refusing harmful requests and avoiding unnecessary refusals\. A model can raise its aggregate safety score by refusing more requests indiscriminately, even when this makes it less useful on benign inputs\. Any combined score therefore encodes a substantive choice about how these outcomes should be valued\. Evaluators should report distinct abilities when benchmarks separate, justify the weights used in any aggregate, and test whether a new benchmark measures anything not already captured by the suite\.

#### Distilled tests enable frequent evaluation\.

Reducing several benchmarks by 97–99% makes it more practical to evaluate models throughout development rather than only before release\. Short tests could be used to compare training interventions, monitor individual checkpoints, and detect changes introduced by fine\-tuning, quantization, system prompts, or minor model updates\. They could also allow small laboratories and independent auditors to evaluate more models with limited compute\. Similar methods may help expand evaluation in low\-resource languages and settings, although each new setting would require its own calibration and validation\. These efficiencies do not extend the scope of the underlying benchmark\. A reduced test shows that the full benchmark can be reproduced with fewer items; it does not show that the benchmark predicts safety in deployment or that the same items will remain informative for future models\.

#### Auditing requires multiple tests\.

No single audit detects every kind of behavioral change\. Person\-fit catches selective prompted sandbagging by identifying response patterns that are inconsistent with a model’s estimated ability, but it is less effective against uniform sandbagging, which can resemble ordinary behavior at a different ability level\. Historical ability estimates detect these broader shifts, while response fingerprints distinguish models with similar aggregate abilities but different item\-level responses\. Together, these methods form a practical audit ladder: public item parameters support person\-fit, historical estimates reveal drift, and retained responses test whether an endpoint continues to behave like the system originally evaluated\. This is especially useful for hosted APIs, where provider routing, quantization, inference backends, and silent serving changes can alter behavior under the same model identifier\(Khoriaty[2026](https://arxiv.org/html/2608.05086#bib.bib16); Pape, Evertz, and Schönherr[2026](https://arxiv.org/html/2608.05086#bib.bib27); Kharinaev et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib15); Gao, Liang, and Guestrin[2025](https://arxiv.org/html/2608.05086#bib.bib10)\)\. These tests establish behavioral continuity rather than the identity of the underlying weights\. Fingerprint comparisons must therefore be calibrated against ordinary variation across repeated measurements of the same endpoint, and our experiments do not show that the methods would detect a model specifically trained to evade them\.

#### Safety evaluation requires psychometric validation\.

Safety benchmark scores already inform decisions with consequences, such as decisions to release models or pick one over another\. Psychometrics exists because human testing faced the same situation—important decisions resting on noisy, manipulable measurements, such as the case of evaluating psychomotor and cognitive limits for individuals in the military\(Draheim et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib5)\)\. Its response was to make validity, reliability, and test security objects of study in their own right\. Our analyses apply the first steps of that program, and none of their conclusions are visible in raw pass rates\. The factor names we assign are summaries of loading patterns, not necessarily validated constructs\. Using them, or any benchmark score, as grounds for consequential safety claims requires the convergent and predictive validation evidence that psychometrics requires, and safety evaluation has barely scratched the surface\. We hope this work encourages both the adoption of these tools and the involvement of researchers trained in measurement\.

## Limitations

Our cross\-benchmark analyses use up to 169 models, of which 134 enter the latent\-structure analyses; this is large for an LLM study but smaller than conventional human\-testing datasets that form the basis of most psychometric analysis\. This may also influence the interpretation of latent abilities, which we summarize and label using empirical loading patterns but cannot cleanly adjudicate between two or three dimensions\. Lastly, our sandbagging detection is evaluated on prompted model organisms rather than fine\-tuned ones, which might be easier to catch than scheming models that can vary their responses more adversarially across datasets\.

## Conclusion

Psychometrics offers safety evaluation more than compression: a measurement model that says what benchmarks measure, which aggregates are real, how to evaluate at 1–3% of the cost without losing ranking fidelity, and when an individual model’s responses should not be believed\. We provide the calibration, the map, the reduced tests, and the audit tools for eight widely used safety benchmarks, and release all artifacts for the community to build on\.

## Acknowledgments

We would like to thank Diogo Cruz, Vamshi Bonagiri, Cameron Holmes, and others for helpful feedback and discussion\. We thank the Supervised Program for Alignment Research for support\. We would also like to thank the UK AI Security Institute and the Department for Science, Innovation, and Technology more broadly for their support\.

## References

- Baker and Kim \(2004\)Baker, F\. B\.; and Kim, S\.\-H\. 2004\.*Item response theory: Parameter estimation techniques*\.CRC press\.
- Bean et al\. \(2025\)Bean, A\. M\.; Kearns, R\. O\.; Romanou, A\.; Hafner, F\. S\.; Mayne, H\.; Batzner, J\.; Foroutan, N\.; Schmitz, C\.; Korgul, K\.; Batra, H\.; Deb, O\.; Beharry, E\.; Emde, C\.; Foster, T\.; Gausen, A\.; Grandury, M\.; Han, S\.; Hofmann, V\.; Ibrahim, L\.; Kim, H\.; Kirk, H\. R\.; Lin, F\.; Liu, G\. K\.\-M\.; Luettgau, L\.; Magomere, J\.; Rystrøm, J\.; Sotnikova, A\.; Yang, Y\.; Zhao, Y\.; Bibi, A\.; Bosselut, A\.; Clark, R\.; Cohan, A\.; Foerster, J\.; Gal, Y\.; Hale, S\. A\.; Raji, I\. D\.; Summerfield, C\.; Torr, P\. H\. S\.; Ududec, C\.; Rocher, L\.; and Mahdi, A\. 2025\.Measuring what Matters: Construct Validity in Large Language Model Benchmarks\.arXiv:2511\.04703\.
- Bock and Aitkin \(1981\)Bock, R\. D\.; and Aitkin, M\. 1981\.Marginal Maximum Likelihood Estimation of Item Parameters: Application of an EM Algorithm\.*Psychometrika*, 46\(4\): 443–459\.
- Cui et al\. \(2024\)Cui, J\.; Chiang, W\.\-L\.; Stoica, I\.; and Hsieh, C\.\-J\. 2024\.OR\-Bench: An Over\-Refusal Benchmark for Large Language Models\.arXiv:2405\.20947\.
- Draheim et al\. \(2025\)Draheim, C\.; Sibley, C\.; Herdener, N\.; Cochrane, A\.; Melick, S\. R\.; Strong, K\.; and Coyne, J\. T\. 2025\.High\-Stakes Psychomotor Ability Assessment: A Military Selection Case Study of Practice Effects in Airplane Tracking Tasks\.*Cognitive Research: Principles and Implications*, 10\(1\): 82\.
- Drasgow, Levine, and Williams \(1985\)Drasgow, F\.; Levine, M\. V\.; and Williams, E\. A\. 1985\.Appropriateness Measurement with Polychotomous Item Response Models and Standardized Indices\.*British Journal of Mathematical and Statistical Psychology*, 38\(1\): 67–86\.
- Embretson and Reise \(2000\)Embretson, S\. E\.; and Reise, S\. P\. 2000\.*Item Response Theory for Psychologists*\.Mahwah, NJ: Lawrence Erlbaum Associates\.
- Galisai et al\. \(2026\)Galisai, M\.; Cifani, S\.; Giarrusso, F\.; Bisconti, P\.; Prandi, M\.; Pierucci, F\.; Sartore, F\.; and Nardi, D\. 2026\.Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety\.arXiv:2604\.18487\.
- Ganguli et al\. \(2022\)Ganguli, D\.; Lovitt, L\.; Kernion, J\.; Askell, A\.; Bai, Y\.; Kadavath, S\.; Mann, B\.; Perez, E\.; Schiefer, N\.; Ndousse, K\.; et al\. 2022\.Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned\.arXiv:2209\.07858\.
- Gao, Liang, and Guestrin \(2025\)Gao, I\.; Liang, P\.; and Guestrin, C\. 2025\.Model Equality Testing: Which Model Is This API Serving?In*International Conference on Learning Representations*\.
- Harding and Sharadin \(2024\)Harding, J\.; and Sharadin, N\. 2024\.What is it for a Machine Learning Model to Have a Capability?*The British Journal for the Philosophy of Science*\.
- Hernández\-Orallo \(2017\)Hernández\-Orallo, J\. 2017\.Evaluation in artificial intelligence: from task\-oriented to ability\-oriented measurement\.*Artificial Intelligence Review*, 48\(3\): 397–447\.
- Hofmann et al\. \(2025\)Hofmann, V\.; Heineman, D\.; Magnusson, I\.; Lo, K\.; Dodge, J\.; Sap, M\.; Koh, P\. W\.; Wang, C\.; Hajishirzi, H\.; and Smith, N\. A\. 2025\.Fluid Language Model Benchmarking\.In*Proceedings of the Conference on Language Modeling*\.
- Horn \(1965\)Horn, J\. L\. 1965\.A Rationale and Test for the Number of Factors in Factor Analysis\.*Psychometrika*, 30\(2\): 179–185\.
- Kharinaev et al\. \(2025\)Kharinaev, A\.; Moskvoretskii, V\.; Shvetsov, E\.; Studenikina, K\.; Mikhail, B\.; and Burnaev, E\. 2025\.Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models\.*arXiv preprint arXiv:2502\.15799*\.
- Khoriaty \(2026\)Khoriaty, M\. 2026\.Not Pinning Your OpenRouter Provider Might Invalidate Your Research\.https://www\.lesswrong\.com/posts/KsyoSAyBRXtwzSugg/not\-pinning\-your\-openrouter\-provider\-might\-invalidate\-your\.LessWrong\.
- Kipnis et al\. \(2025\)Kipnis, A\.; Voudouris, K\.; Schulze Buschoff, L\. M\.; and Schulz, E\. 2025\.metabench: A Sparse Benchmark of Reasoning and Knowledge in Large Language Models\.In*Proceedings of the Thirteenth International Conference on Learning Representations*\.ArXiv:2407\.12844\.
- Lalor, Wu, and Yu \(2019\)Lalor, J\. P\.; Wu, H\.; and Yu, H\. 2019\.Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing*\.
- Lin, Hilton, and Evans \(2022\)Lin, S\.; Hilton, J\.; and Evans, O\. 2022\.TruthfulQA: Measuring How Models Mimic Human Falsehoods\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*, 3214–3252\.
- Lord \(1980\)Lord, F\. M\. 1980\.*Applications of Item Response Theory to Practical Testing Problems*\.Hillsdale, NJ: Lawrence Erlbaum Associates\.
- Magis, Yan, and von Davier \(2017\)Magis, D\.; Yan, D\.; and von Davier, A\. A\. 2017\.*Computerized Adaptive and Multistage Testing with R*\.Springer\.
- Maia Polo et al\. \(2024\)Maia Polo, F\.; Weber, L\.; Choshen, L\.; Sun, Y\.; Xu, G\.; and Yurochkin, M\. 2024\.tinyBenchmarks: Evaluating LLMs with Fewer Examples\.In*Proceedings of the 41st International Conference on Machine Learning*\.
- Martínez\-Plumed et al\. \(2016\)Martínez\-Plumed, F\.; Prudêncio, R\. B\.; Martínez\-Usó, A\.; and Hernández\-Orallo, J\. 2016\.Making sense of item response theory in machine learning\.In*ECAI 2016*, 1140–1148\. IOS Press\.
- Mazeika et al\. \(2024\)Mazeika, M\.; Phan, L\.; Yin, X\.; Zou, A\.; Wang, Z\.; Mu, N\.; Sakhaee, E\.; Li, N\.; Basart, S\.; Li, B\.; Forsyth, D\.; and Hendrycks, D\. 2024\.HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal\.In*Proceedings of the 41st International Conference on Machine Learning*\.
- Meijer and Sijtsma \(2001\)Meijer, R\. R\.; and Sijtsma, K\. 2001\.Methodology Review: Evaluating Person Fit\.*Applied Psychological Measurement*, 25\(2\): 107–135\.
- Pacchiardi et al\. \(2025\)Pacchiardi, L\.; Voudouris, K\.; Slater, B\.; Martínez\-Plumed, F\.; Hernández\-Orallo, J\.; Zhou, L\.; and Schellaert, W\. 2025\.PredictaBoard: Benchmarking LLM score predictability\.In*Findings of the Association for Computational Linguistics: ACL 2025*, 15245–15266\.
- Pape, Evertz, and Schönherr \(2026\)Pape, D\.; Evertz, J\.; and Schönherr, L\. 2026\.The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility\.*arXiv preprint arXiv:2605\.19537*\.
- Reckase \(2009\)Reckase, M\. D\. 2009\.*Multidimensional Item Response Theory*\.Springer\.
- Rodriguez et al\. \(2021\)Rodriguez, P\.; Barrow, J\.; Hoyle, A\. M\.; Lalor, J\. P\.; Jia, R\.; and Boyd\-Graber, J\. 2021\.Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards?In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics*, 4486–4503\.
- Romero\-Alvarado et al\. \(2026\)Romero\-Alvarado, D\.; Martínez\-Plumed, F\.; Pacchiardi, L\.; Save, H\.; Pawar, S\. M\.; Mehrbakhsh, B\.; Casares, P\. A\. M\.; Slater, B\.; Bova, P\.; Romero, P\.; et al\. 2026\.Capabilities Ain’t All You Need: Measuring Propensities in AI\.*arXiv preprint arXiv:2602\.18182*\.
- Röttger et al\. \(2024\)Röttger, P\.; Kirk, H\. R\.; Vidgen, B\.; Attanasio, G\.; Bianchi, F\.; and Hovy, D\. 2024\.XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics*\.
- Spagliardi et al\. \(2026\)Spagliardi, F\.; Silva, M\.; Datta, A\.; Zhou, A\.; Bonagiri, V\.; and Cruz, D\. 2026\.Efficient Safety Benchmarking via Item Response Theory\.arXiv:2606\.20626\.
- van der Weij et al\. \(2024\)van der Weij, T\.; Hofstätter, F\.; Jaffe, O\.; Brown, S\. F\.; and Ward, F\. R\. 2024\.AI Sandbagging: Language Models Can Strategically Underperform on Evaluations\.arXiv:2406\.07358\.
- Voudouris et al\. \(2026\)Voudouris, K\.; Thalmann, M\.; Kipnis, A\.; Hernández\-Orallo, J\.; and Schulz, E\. 2026\.Measuring What AI Systems Might Do: Towards A Measurement Science in AI\.*arXiv preprint arXiv:2603\.00063*\.
- Wang et al\. \(2023\)Wang, Y\.; Li, H\.; Han, X\.; Nakov, P\.; and Baldwin, T\. 2023\.Do\-Not\-Answer: A Dataset for Evaluating Safeguards in LLMs\.arXiv:2308\.13387\.
- Weiss and Kingsbury \(1984\)Weiss, D\. J\.; and Kingsbury, G\. G\. 1984\.Application of Computerized Adaptive Testing to Educational Problems\.*Journal of Educational Measurement*, 21\(4\): 361–375\.
- Xie et al\. \(2024\)Xie, T\.; Qi, X\.; Zeng, Y\.; Huang, Y\.; Sehwag, U\. M\.; Huang, K\.; He, L\.; Wei, B\.; Li, D\.; Sheng, Y\.; Jia, R\.; Li, B\.; Li, K\.; Chen, D\.; Henderson, P\.; and Mittal, P\. 2024\.SORRY\-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors\.arXiv:2406\.14598\.
- Zhang et al\. \(2026\)Zhang, Z\.; Huang, L\.; Wu, G\.; et al\. 2026\.Health\-ORSC\-Bench: A Benchmark for Measuring Over\-Refusal and Safety Completion\.arXiv:2601\.17642\.
- Zou et al\. \(2023\)Zou, A\.; Wang, Z\.; Carlini, N\.; Nasr, M\.; Kolter, J\. Z\.; and Fredrikson, M\. 2023\.Universal and Transferable Adversarial Attacks on Aligned Language Models\.arXiv:2307\.15043\.

## Appendix AModels Evaluated

Table 3:Full model roster \(192 models, OpenRouter identifiers with the developer prefix omitted\)\.†marks the 23 models excluded from the cross\-benchmark analysis cohort \(missing from at least one benchmark\); the remaining 169 models form the cohort used for all cross\-benchmark analyses\. The full response and score data are provided in the supplementary code and data package\.DevelopernnModelsAlibaba \(Qwen\)36qwen\-2\.5\-72b\-instruct,qwen\-2\.5\-7b\-instruct,qwen\-2\.5\-coder\-32b\-instruct,qwen\-plus,qwen\-plus\-2025\-07\-28,qwen2\.5\-vl\-72b\-instruct,qwen3\-14b,qwen3\-235b\-a22b,qwen3\-235b\-a22b\-2507,qwen3\-30b\-a3b,qwen3\-30b\-a3b\-instruct\-2507,qwen3\-30b\-a3b\-thinking\-2507,qwen3\-32b,qwen3\-8b,qwen3\-coder,qwen3\-coder\-30b\-a3b\-instruct,qwen3\-coder\-flash,qwen3\-coder\-next,qwen3\-coder\-plus,qwen3\-next\-80b\-a3b\-instruct,qwen3\-vl\-235b\-a22b\-instruct,qwen3\-vl\-30b\-a3b\-instruct,qwen3\-vl\-32b\-instruct,qwen3\-vl\-8b\-instruct,qwen3\.5\-122b\-a10b,qwen3\.5\-27b,qwen3\.5\-35b\-a3b,qwen3\.5\-397b\-a17b,qwen3\.5\-9b,qwen3\.5\-flash\-02\-23,qwen3\.5\-plus\-02\-15,qwen3\.5\-plus\-20260420,qwen3\.6\-27b,qwen3\.6\-35b\-a3b,qwen3\.6\-flash,qwen3\.6\-plusAmazon3nova\-2\-lite\-v1,nova\-lite\-v1,nova\-micro\-v1Anthropic6claude\-3\-haiku,claude\-3\.5\-haiku†,claude\-haiku\-4\.5,claude\-opus\-4†,claude\-sonnet\-4†,claude\-sonnet\-4\.5Arcee AI1trinity\-miniByteDance5seed\-1\.6†,seed\-1\.6\-flash†,seed\-2\.0\-lite,seed\-2\.0\-mini,ui\-tars\-1\.5\-7bCohere3command\-a†,command\-r\-08\-2024,command\-r7b\-12\-2024DeepSeek11deepseek\-chat,deepseek\-chat\-v3,deepseek\-chat\-v3\-0324,deepseek\-chat\-v3\.1,deepseek\-r1\-distill\-llama\-70b,deepseek\-r1\-distill\-qwen\-32b†,deepseek\-v3\.1\-terminus,deepseek\-v3\.2,deepseek\-v3\.2\-exp,deepseek\-v4\-flash,deepseek\-v4\-proEssential AI1rnj\-1\-instruct†Google16gemini\-2\.0\-flash\-001†,gemini\-2\.0\-flash\-lite\-001†,gemini\-2\.5\-flash,gemini\-2\.5\-flash\-lite,gemini\-2\.5\-flash\-lite\-preview\-09\-2025,gemini\-2\.5\-pro,gemini\-3\-flash\-preview,gemini\-3\.1\-flash\-lite,gemini\-3\.1\-flash\-lite\-preview,gemma\-2\-27b\-it,gemma\-3\-12b\-it,gemma\-3\-27b\-it,gemma\-3\-4b\-it,gemma\-3n\-e4b\-it,gemma\-4\-26b\-a4b\-it,gemma\-4\-31b\-itGryphe1mythomax\-l2\-13bIBM2granite\-4\.0\-h\-micro,granite\-4\.1\-8bInclusionAI2ling\-2\.6\-1t,ling\-2\.6\-flashInflection2inflection\-3\-pi†,inflection\-3\-productivity†Kwaipilot1kat\-coder\-pro\-v2Liquid AI2lfm\-2\-24b\-a2b,lfm\-2\-24b\-a2b\-20260224Meta11llama\-3\-70b\-instruct†,llama\-3\-8b\-instruct,llama\-3\.1\-70b\-instruct,llama\-3\.1\-8b\-instruct,llama\-3\.2\-11b\-vision\-instruct,llama\-3\.2\-1b\-instruct,llama\-3\.2\-3b\-instruct,llama\-3\.3\-70b\-instruct,llama\-4\-maverick,llama\-4\-scout,llama\-guard\-4\-12bMicrosoft3phi\-4,phi\-4\-mini\-instruct,wizardlm\-2\-8x22bMiniMax6minimax\-01,minimax\-m2,minimax\-m2\-her,minimax\-m2\.1,minimax\-m2\.5,minimax\-m2\.7Mistral AI19codestral\-2508,devstral\-2512,devstral\-small†,ministral\-14b\-2512,ministral\-3b\-2512,ministral\-8b\-2512,mistral\-7b\-instruct\-v0\.1†,mistral\-large,mistral\-large\-2512,mistral\-medium\-3,mistral\-medium\-3\.1,mistral\-nemo,mistral\-saba,mistral\-small\-24b\-instruct\-2501,mistral\-small\-2603,mistral\-small\-3\.1\-24b\-instruct,mistral\-small\-3\.2\-24b\-instruct,mixtral\-8x22b\-instruct,voxtral\-small\-24b\-2507Moonshot AI4kimi\-k2,kimi\-k2\-0905,kimi\-k2\.5,kimi\-k2\.6Nex AGI1deepseek\-v3\.1\-nex\-n1†Nous Research5hermes\-2\-pro\-llama\-3\-8b†,hermes\-3\-llama\-3\.1\-405b,hermes\-3\-llama\-3\.1\-70b,hermes\-4\-405b,hermes\-4\-70bNVIDIA4llama\-3\.3\-nemotron\-super\-49b\-v1\.5,nemotron\-3\-nano\-30b\-a3b,nemotron\-3\-super\-120b\-a12b,nemotron\-nano\-9b\-v2†OpenAI17gpt\-3\.5\-turbo,gpt\-3\.5\-turbo\-0613†,gpt\-3\.5\-turbo\-16k,gpt\-4\-turbo,gpt\-4\.1†,gpt\-4\.1\-mini,gpt\-4\.1\-nano,gpt\-4o,gpt\-4o\-mini,gpt\-4o\-mini\-2024\-07\-18,gpt\-5\-mini,gpt\-5\-nano,gpt\-5\.4\-nano,gpt\-oss\-120b,gpt\-oss\-20b,o3\-mini,o4\-miniPerceptron1perceptron\-mk1Perplexity1sonarPrime Intellect1intellect\-3†Reka AI2reka\-edge,reka\-flash\-3Sao10K1l3\-lunaris\-8bStepFun2step\-3\.5\-flash,step\-3\.7\-flashTencent2hunyuan\-a13b\-instruct,hy3\-previewTheDrummer1rocinante\-12bUpstage1solar\-pro\-3xAI3grok\-4\.20,grok\-4\.3,grok\-build\-0\.1Xiaomi3mimo\-v2\-flash†,mimo\-v2\.5,mimo\-v2\.5\-proZ\.ai12glm\-4\-32b†,glm\-4\.5,glm\-4\.5\-air,glm\-4\.5v,glm\-4\.6,glm\-4\.6v,glm\-4\.7,glm\-4\.7\-flash,glm\-5,glm\-5\-turbo,glm\-5\.1,glm\-5v\-turbo
## Appendix BSandbagging and API Auditing Details

### Detection Operating Characteristics

All detection rates in the main\-text audit results use a single operating point: a run is flagged whenlz≤−0\.87l\_\{z\}\\leq\-0\.87, the 10th percentile of the 171 genuine cohort models, i\.e\. a 10% false\-positive rate\. Figure[6](https://arxiv.org/html/2608.05086#A2.F6)A shows the full ROC curves\. By framing \(plain, persona, incentive\), person\-fit flags 91% \(10/11\), 100% \(10/10\), and 80% \(8/10\) of targeted runs; 100% \(10/10\), 100% \(10/10\), and 90% \(9/10\) of triggered runs; and 18% \(2/11\), 20% \(2/10\), and 70% \(7/10\) of uniform runs\. The pooled rates in the main text aggregate these unequal run counts\.

The detections are not an artifact of the item budget\. Repeating the audit with a 100\-item subsample of HarmBench’s standard\-category pool instead of all 196 post\-preprocessing items leaves the plain\-framing operating points essentially unchanged: targeted runs are flagged at 11/11 \(versus 10/11 at the full budget\) and uniform runs at 1/11 \(versus 2/11; Figure[6](https://arxiv.org/html/2608.05086#A2.F6)B\)\.

![Refer to caption](https://arxiv.org/html/2608.05086v1/x6.png)Figure 6:Person\-fit detection of sandbagging on HarmBench\.\(A\)ROC curves per sandbagging scope with the three prompt framings pooled; dots mark the calibrated cutoff \(lz≤−0\.87l\_\{z\}\\leq\-0\.87, 10% false\-positive rate over 171 genuine models\)\.\(B\)The same audit at a 100\-item budget \(dashed\) versus the full 196\-item budget \(solid\), plain framing\.
### Fingerprint Null Calibration and Drift

An endpoint does not reproduce its own responses exactly, so fingerprint audits need a null distribution for self\-disagreement\. Under a stable classifier judge \(Do\-Not\-Answer\), endpoints disagree with themselves on a median of 2\.8% of items across repeated collections\. Under a threshold\-sensitive similarity judge \(TruthfulQA\), the median rises to 25%\. This variability depends on provider routing, generation stochasticity, item difficulty, and the judging procedure, so the null must be calibrated with repeated collections on the same audit items and the same scoring pipeline \(Figure[7](https://arxiv.org/html/2608.05086#A2.F7)\)\.

Against these nulls, cross\-model disagreement is well separated from self\-disagreement for 13 of the 14 distinct identities in the re\-collection experiment\. The two catalog entries that the procedure declines to separate agree with one another \(8\.0–9\.4% disagreement\) more closely than either agrees with itself across repeated runs \(10\.0–10\.2%\), consistent with both entries serving the same weights\.

The same reference responses support drift audits\. One endpoint disagrees with its own reference responses from two months earlier on 33\.6% of TruthfulQA items, against a resampling null of 2\.0%\. This change is inconsistent with ordinary endpoint variability under the calibrated null and indicates that the behavior associated with the model identifier changed between collections\.

![Refer to caption](https://arxiv.org/html/2608.05086v1/x7.png)Figure 7:Fingerprint self\-disagreement nulls versus cross\-model disagreement under a classifier judge \(Do\-Not\-Answer\) and a similarity judge \(TruthfulQA\), with the alias pair and the drift case marked\.

## Appendix CFitting Multidimensional IRT Models

Our main analysis is two\-stage\. We fit a unidimensional 2PL to each benchmark, obtain one ability per model per benchmark, and factor\-analyze the resulting134×8134\\times 8ability matrix\. A natural alternative is multidimensional IRT \(MIRT\): fitP​\(xm​j=1\)=σ​\(aj⊤​θm\+dj\)P\(x\_\{mj\}=1\)=\\sigma\(a\_\{j\}^\{\\top\}\\theta\_\{m\}\+d\_\{j\}\)with a vector\-valued abilityθm\\theta\_\{m\}directly to the item matrix, so the latent structure is estimated in one step\. This appendix explains why the two\-stage design estimates the same quantity and reports a direct MIRT fit that reaches the same conclusions\.

### The Two\-Stage Design Is a Constrained MIRT Model

Consider the MIRT model in which each item loads only on its own benchmark’s dimension, with the eight dimensions freely correlated:θm∼𝒩​\(0,Φ\)\\theta\_\{m\}\\sim\\mathcal\{N\}\(0,\\Phi\), whereΦ\\Phiis the8×88\\times 8ability correlation matrix\. Within a benchmark, responses follow a unidimensional 2PL; across benchmarks, all dependence runs throughΦ\\Phi\. Under this model, calibrating each benchmark separately loses no information about item parameters, because the per\-benchmark likelihoods are the joint model’s marginals\. The second\-stage factor analysis then estimates the structure ofΦ\\Phifrom the ability estimates\.

The two\-stage estimate ofΦ\\Phiconverges to the truth as the number of items per benchmark grows, and our benchmarks contribute 200–1,318 items each\. The residual measurement error in the ability estimates attenuates the second\-stage correlations toward zero\. This biases the analysis*against*the shared structure we report, so the factor correlations reported in the main text are conservative\. The same separation of within\-benchmark calibration from between\-benchmark structure is used by metabench\(Kipnis et al\.[2025](https://arxiv.org/html/2608.05086#bib.bib17)\)\.

One assumption carries the argument: no item loads on more than one ability\. If an item rewarded two abilities at once—say, a harm item that also rewards truthfulness—the per\-benchmark ability would blend the two, and the second\-stage factors could be distorted\. An unconstrained MIRT fit tests exactly this assumption, so we run one\.

### An Unconstrained MIRT Fit Reaches the Same Conclusions

We fit exploratory MIRT models with unconstrained loadings andθ∼𝒩​\(0,ID\)\\theta\\sim\\mathcal\{N\}\(0,I\_\{D\}\)to the134×5,055134\\times 5\{,\}055item matrix \(the items retaining response variation within this cohort\) atD=1D=1to44\. We compare dimensionalities by 5\-fold cross\-validated held\-out log\-likelihood over models: item parameters are calibrated on the training folds, and we evaluate the marginal likelihood of held\-out models’ complete response patterns without refitting any per\-model parameters\. Estimation uses Bock–Aitkin full\-information EM\(Bock and Aitkin[1981](https://arxiv.org/html/2608.05086#bib.bib3)\)on a pruned Gauss–Hermite grid, echelon constraints for rotational identification, and a𝒩​\(0,1\.52\)\\mathcal\{N\}\(0,1\.5^\{2\}\)prior on loadings—the multidimensional analogue of the calibration priors in Materials & Methods\. The implementation reproduces our unidimensional 2PL exactly atD=1D=1and matches an independent MIRT package on synthetic two\-factor data \(loading congruence1\.0001\.000\)\.

Table 4:Item\-level MIRT model comparison on the134×5,055134\\times 5\{,\}055matrix \(bold marks the best value per criterion\)\. Held\-out likelihood improves by14,71814\{,\}718nats atD=2D\{=\}2and6,7946\{,\}794atD=3D\{=\}3, then flattens \(\+2,286\+2\{,\}286atD=4D\{=\}4\); BIC is minimized atD=2D\{=\}2\. In\-sample AIC never turns, which is why an out\-of\-sample criterion is necessary with up to∼25\{\\sim\}25k free parameters\.![Refer to caption](https://arxiv.org/html/2608.05086v1/x8.png)Figure 8:Item\-level MIRT model comparison: cross\-validated held\-out log\-likelihood\(A\)and BIC\(B\)across latent dimensionalityDD\. Held\-out likelihood improves sharply toD=3D\{=\}3and only marginally atD=4D\{=\}4; BIC is minimized atD=2D\{=\}2\.The unconstrained fit agrees with the two\-stage analysis on the points of substance \(Table[4](https://arxiv.org/html/2608.05086#A3.T4)\)\. Both criteria reject a single dimension decisively\. BIC is minimized atD=2D=2, and the held\-out gains shrink from14,71814\{,\}718nats atD=2D\{=\}2to6,7946\{,\}794atD=3D\{=\}3and2,2862\{,\}286atD=4D\{=\}4, consistent with the two\-stage conclusion that at least two dimensions are robust and a third adds modest further structure\. Item loadings aggregated by benchmark recover the benchmark\-level structure, including the bipolar refusal\-strictness axis, which appears already atD=1D=1\(HarmBench\+0\.92\+0\.92, SORRY\-Bench\+0\.90\+0\.90versus OR\-Bench\-Hard−0\.64\-0\.64\)\. A permutation parallel analysis\(Horn[1965](https://arxiv.org/html/2608.05086#bib.bib14)\)on the item correlation spectrum gives an upper bound of 22 components; permutation bounds over\-count when models are far fewer than items, and the eigenvalue spectrum itself shows two dominant components with a visible third step\.

We report the two\-stage analysis in the main text for two reasons\. Its per\-benchmark abilities are directly interpretable, and its estimates are stable at our cohort size, where the unconstrained fit spends up to∼25\{\\sim\}25k parameters on 134 response patterns\. The item\-level fit serves as the robustness check: the simple\-structure constraint is not driving the conclusions\.

## Appendix DLatent Factor Explorations

### Factor\-Count Selection Criteria

![Refer to caption](https://arxiv.org/html/2608.05086v1/x9.png)Figure 9:Model selection for the number of latent abilities\.\(A\)CFI first exceeds the conventional0\.950\.95threshold at three factors\.\(B\)Horn’s parallel analysis \(minimum\-residual\): three eigenvalues exceed the 95th percentile of matched random data\. Shading marks the selected three\-factor solution\.We retain factors using three standard criteria\. RMSEA measures absolute misfit of the factor model to the observed correlation matrix, with values below0\.060\.06conventionally taken as close fit\. CFI compares the fitted model against an independence baseline, with values above0\.950\.95conventionally taken as acceptable\. Horn’s parallel analysis retains a factor when its observed eigenvalue exceeds the 95th percentile of eigenvalues at the same rank in random data of matched size\.

On our eight\-indicator ability matrix, the criteria do not agree perfectly\. CFI and minimum\-residual parallel analysis both select three factors\. The PCA variant of parallel analysis, which tends toward under\-extraction, retains two\. RMSEA improves monotonically with each added factor but reaches close fit at no solution, a known limitation with few indicators\. We therefore treat multidimensionality, with at least two latent abilities, as the robust conclusion, and use the three\-factor solution to describe the additional structure supported by CFI and parallel analysis\.

### Composite Scores Require Aligned Directions

The latent structure also determines which benchmark averages define coherent quantities\.

Suppose a composite combines benchmarks that all increase with the same underlying ability\. Models that score highly on one member should tend to score highly on the others, and a small subset of items should preserve the resulting model ranking\. If some benchmarks point in opposite directions, averaging them cancels part of the shared signal\. A related tension appears between AHB and TruthfulQA, whose model abilities correlate atρ=−0\.56\\rho=\-0\.56: high performance on one safety benchmark can therefore predict lower performance on another\.

Table 5:A composite can be recovered from few items exactly when its members agree in sign\. Signedρ¯\\bar\{\\rho\}is the mean pairwise ability correlation among members \(after any flips\); recovery is held\-out Spearman from 25 adaptively chosen items \(20 splits\)\. Rank correlation between the columns:0\.930\.93\.Table[5](https://arxiv.org/html/2608.05086#A4.T5)shows this relationship across seven composites\. For each composite, we calculate the mean signed correlation among its constituent benchmark abilities\. We then measure how well 25 adaptively selected items recover the full composite\. The rank correlation between sign agreement and recovery is0\.930\.93\.

The refusal benchmarks provide a matched comparison\. The raw refusal composite combines HarmBench, SORRY\-Bench, and OR\-Bench\-Hard without changing their orientations\. Because OR\-Bench\-Hard measures the opposite end of refusal strictness, its signal partially cancels the other two\. The resulting composite has a mean signed correlation of−0\.18\-0\.18and a recovery correlation ofρ=0\.79\\rho=0\.79\.

We then reverse OR\-Bench\-Hard so that all three benchmarks point toward the same end of the refusal\-strictness axis\. The constituent benchmarks remain unchanged; only the orientation of one score differs\. The mean signed correlation rises to0\.780\.78, and recovery rises toρ=0\.96\\rho=0\.96, the highest value among the composites considered\.

The same adjustment improves the full eight\-benchmark composite\. Flipping OR\-Bench\-Hard before averaging improves every item\-selection method we test\. Under random subsampling, recovery from 25 items increases fromρ=0\.65\\rho=0\.65toρ=0\.82\\rho=0\.82\. In this setting, correcting the composite produces a larger gain than replacing random sampling with any of the tested item\-selection algorithms\.

The order of operations therefore matters\. Before optimizing which items to administer, one must first determine whether the target score combines compatible dimensions and whether its components are oriented consistently\. A more efficient estimate of an incoherent composite remains an incoherent composite\.

## Appendix EWhy the 2PL

We compare the nested 1PL/2PL/3PL/4PL ladder on each benchmark separately, fitting each model by marginal maximum likelihood on the same cleaned matrices used for the main calibration \(per\-benchmark gap\-free models, post\-preprocessing items\)\. The asymptote parameters of the 3PL and 4PL carry the fitting library’s default Beta priors to keep them in the unit interval, so those two fits are MAP rather than ML and their criteria are approximate; the 2PL→\\to3PL and 3PL→\\to4PL likelihood\-ratio tests also place the constrained parameter on its boundary, which makes theχ2\\chi^\{2\}reference conservative\. BIC uses the number of models asnn, the standard convention for marginal\-likelihood fits\.

Table 6:Model comparison for the nested PL ladder, per benchmark\. Negative values favor the alternative over the 2PL \(bold: the single case\)\. The 1PL→\\to2PL likelihood\-ratio test rejects the 1PL on every benchmark atp<10−109p<10^\{\-109\}and is omitted from the table\.Three conclusions follow \(Table[6](https://arxiv.org/html/2608.05086#A5.T6)\)\. First, the 1PL is rejected decisively: the likelihood\-ratio test rejects the equal\-discrimination constraint on all eight benchmarks, and AIC agrees everywhere\. BIC alone prefers the 1PL on seven benchmarks, which reflects its penalty oflog⁡\(n\)\\log\(n\)per parameter atn≈170n\\approx 170models applied to 200–1,300 additional discrimination parameters rather than a substantive endorsement: equal discriminations contradict the roughly order\-of\-magnitude spread in fitted values, and would eliminate the information weighting that the item selection in our distillation analyses relies on\.

Second, the 3PL improves AIC on exactly one benchmark, HarmBench \(−60\-60, LRTp≈10−28p\\approx 10^\{\-28\}\), plausibly because judge leniency on some items acts like a nonzero floor on the pass probability\. The marginal AHB test \(p=\.03p=\.03\) is not corroborated by AIC\. Third, the 4PL is never preferred, and on several benchmarks its fitted likelihood falls below the 3PL’s, which is impossible at a maximum\-likelihood optimum; the added asymptotes make the optimization fail rather than the fit better, the expected outcome of models with up to∼5,000\{\\sim\}5\{,\}000parameters on∼170\{\\sim\}170response patterns\.

The unregularized 2PL and richer fits also reach the iteration cap on most benchmarks, the divergent\-discrimination pathology that motivates the calibration priors in Materials & Methods; it affects all rungs above the 1PL alike and does not change any verdict\. We therefore use the 2PL throughout, regularized as described there\. A pass\-probability floor for judge\-mediated benchmarks may nonetheless be worth revisiting at larger cohort sizes\.