Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

# Gated Against One Model, Open to the Next:Option-Only Solvability in Legal Multiple-Choice Benchmarks
Source: [https://arxiv.org/html/2608.15428](https://arxiv.org/html/2608.15428)
###### Abstract

Multiple\-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question\. Measuring that gap takes care: a model answeringAto most items scores above chance wherever the key sits atA, and reads as recognition when it is not\. We measure it onUA\-JudgeExam: 11,990 four\-option items with official keys, published by Ukraine’s Higher Qualification Commission of Judges\.

Shown the options and*no question*, Claude Haiku 4\.5 scores0\.3830\.383against chance, and the leak is concentrated:11\.8%11\.8\\%of items are answered blind on all eight option orders, against0\.20\.2items expected by chance\. It is not quotation: search over280,059280\{,\}059editions of Ukrainian legislation recovers0\.1280\.128\. Gating those out retains8,1288\{,\}128items, on which the gating model itself now scores0\.2040\.204— and GPT\-5\.6, which took no part in the selection, still answers0\.5150\.515of them with the question hidden\. Scoring twelve held\-out models on the whole set and subtracting each one’s answer\-position habit, only two keep an excess: GPT\-5\.6 at\+0\.265\+0\.265, Sonnet 4\.6 at\+0\.081\+0\.081\. Without it the ranking misleads: Llama 3\.1 8B scores0\.2920\.292blind, above every model but those two, purely by answeringAto92%92\\%of items\.

The gate does select something real: on the items it rejected, eleven of twelve models score0\.5180\.518–0\.7890\.789, every interval clear of what the same model scores on the items it kept\. But that signal is one model’s, and filtering on it does not transfer upward\. Neither is visible on a 400\-item sample, where nine models read as “statistically at chance”\. Rewriting distractors instead overshoots to0\.1680\.168, below chance and as exploitable\. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters\. Item format decides whether the problem can arise; capability decides how much is extracted\. We release the corpus, the predictions and the harness\.

Keywords:benchmark validity, multiple\-choice evaluation, shortcut learning, legal NLP, Ukrainian, contamination

## 1Introduction

A multiple\-choice benchmark reports one number: how often the model picks the key\. That number is treated as evidence about the model’s competence in the domain\. It is only such evidence to the extent that the question is load\-bearing — that the model could not have found the key from the options alone\.

The obvious repair, when a bank fails that test, is to filter it: drop the items a model can answer without the question and keep the rest\. This paper’s central finding is that the repair does not hold\. We filtered a state\-published judicial examination bank until the filtering model scored0\.2040\.204blind on what survived, below chance; GPT\-5\.6, which took no part in the selection, still answers0\.5150\.515of those same items with the question hidden\. A bank cleaned against one model is not clean for a better one, and we found no version of the procedure that made it so\.

This is not a hypothetical concern wherever the options are self\-contained propositions, as they are in professional licensing material\. Distractors are written by humans under time pressure, and a wrong option is wrong for a reason: it names the wrong authority, states the wrong deadline, or asserts something no statute says\. A reader who knows the domain may then be able to recognise the one option that reads like real law without ever seeing the question — and, as we show, so can a model, though we do not test human readers\. To the extent that happens, the benchmark measures recognition of well\-formed legal propositions rather than the reasoning it claims to measure\. Where the options are instead pointers into the stem — “i and iii” — the concern cannot arise, and we show that this distinction, rather than subject matter, is what separates the benchmarks we test\.

We make that quantity the object of study rather than a footnote\. Our contributions:

1. 1\.Filtering does not transfer\.Dropping the items one model can guess retains67\.8%67\.8\\%of the bank and drives the gating model itself to0\.2040\.204on what survives — yet GPT\-5\.6 still answers0\.5150\.515of the cleaned set with the question hidden\. Rewriting distractors instead overshoots below chance\. Neither repair produces a neutral set \(§[6](https://arxiv.org/html/2608.15428#S6), §[8](https://arxiv.org/html/2608.15428#S8)\)\.
2. 2\.A measurement protocol, and an estimator that separates habit from content\.The blind condition is run over option\-order permutations with the selecting model held separate from the reporting one\. Forcing the key into each slot in turn then makes the four accuracies sum to one whenever choice is independent of content, so their mean is chance by construction and any excess is content\. Measured that way, ten of twelve held\-out models extract nothing at all, and neither prompt phrasing, option labelling nor inference\-time reasoning accounts for what the other two extract \(§[5](https://arxiv.org/html/2608.15428#S5), §[6\.1](https://arxiv.org/html/2608.15428#S6.SS1)–§[6\.3](https://arxiv.org/html/2608.15428#S6.SS3)\)\.
3. 3\.A benchmark that does not leak, and why\.The same probe on LEXam returns chance; every one of its items uses reference\-style options, none longer than 33 characters\. Option format, not subject matter, decides whether the problem can exist \(§[7](https://arxiv.org/html/2608.15428#S7)\)\.
4. 4\.A resource\.UA\-JudgeExam: 11,990 items with official state\-issued keys, extraction verified by an independent second path, plus the gated8,1288\{,\}128\-item subset and every prediction behind the tables \(§[3](https://arxiv.org/html/2608.15428#S3)\)\.

## 2Related Work

#### Answering without the question\.

The probe we use is not new\.[Balepur et al\. 2024](https://arxiv.org/html/2608.15428#bib.bib1)prompt models with the choices alone on three MCQA datasets and beat a majority baseline in 11 of 12 cases, and find no evidence that this stems from memorisation, nor that priors over individual choices fully explain it\.[Cho et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib3)attack the same concern from the scoring side, proposing a metric that isolates how much the question contributes to a model’s decision\. Both establish that the phenomenon exists in general\-domain multiple choice\. We take it as given and ask the question that follows for a practitioner holding a benchmark and reaching for the obvious fix: how large is the effect on professional legal exam material, does filtering remove it, does a filter built with one model protect against another, and what property of an item set decides whether the problem arises at all\.

#### Legal benchmarks\.

Legal NLP benchmarks are predominantly English\([Guha et al\. 2023](https://arxiv.org/html/2608.15428#bib.bib5);[Chalkidis et al\. 2022](https://arxiv.org/html/2608.15428#bib.bib2);[Hendrycks et al\. 2021](https://arxiv.org/html/2608.15428#bib.bib7)\), and LEXTREME\([Niklaus et al\. 2023](https://arxiv.org/html/2608.15428#bib.bib9)\)broadens coverage to 24 languages, though its tasks are classification and token\-level extraction rather than multiple choice, so the blind condition does not apply to it\. The[Östling et al\. 2023](https://arxiv.org/html/2608.15428#bib.bib10)corpus is the closest analogue to our resource on the legal and ethical side — a national court corpus released under restrictions worked out with institutional review, where we can release in full only because the bank is a state document outside copyright\. Exam\-derived benchmarks inherit the item\-writing conventions of the professions that produced them, including the distractor\-writing conventions this paper is about: LEXam\([Fan et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib4)\)builds 340 law exams into a reasoning benchmark, and[Katz et al\. 2024](https://arxiv.org/html/2608.15428#bib.bib8)evaluate against a professional licensing examination directly\. We measure the blind condition on LEXam’s four\-choice split in §[7](https://arxiv.org/html/2608.15428#S7)and find no leakage at all, for a reason that turns out to be about how its options are written rather than about its subject matter\.

#### Position and selection bias\.

A separate line of work shows that models are sensitive to*where*an option sits\.[Zheng et al\. 2024](https://arxiv.org/html/2608.15428#bib.bib14)document a systematic preference for particular option IDs across 20 models and trace it to token bias — extra probability mass on the tokensA/B/C/Dthemselves — and debias by permuting option contents;[Pezeshkpour and Hruschka 2024](https://arxiv.org/html/2608.15428#bib.bib11)show the same sensitivity to option order\. That literature and this paper measure different things which are easy to confuse in a single\-order experiment, since a model that merely favours one slot will also score above chance whenever the key happens to sit there\. §[6\.2](https://arxiv.org/html/2608.15428#S6.SS2)separates them by forcing the key into each slot in turn, and finds the two orthogonal: the model with the strongest slot preference in our set is not the one that leaks most\. Our labelling sweep also bears on the mechanism, since relabelling the options in Cyrillic or with digits leaves the aversion to the first slot essentially unchanged\.

#### Shortcuts and artifacts\.

Work on annotation artifacts in natural language inference established that models exploit hypothesis\-only signal\([Gururangan et al\. 2018](https://arxiv.org/html/2608.15428#bib.bib6);[Poliak et al\. 2018](https://arxiv.org/html/2608.15428#bib.bib12)\); the blind condition is the multiple\-choice analogue\. The closest study in law is[Watson et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib13), who ask whether legal judgment prediction forecasts anything or merely reads outcome\-revealing language left in the judgment, on 33,158 UK Employment Tribunal claims\. Their answer and ours diverge in a way worth stating\. They remove the leaking*features*and the task survives: Macro\-F1 falls negligibly, so a real signal was there underneath\. We remove the leaking*items*and the benchmark does not survive in the same sense — the surviving set is clean for the model that selected it and not for a stronger one\. Deleting a cue from every instance and deleting the instances that carry it are not the same operation, and only the first leaves the measurement intact\.

## 3UA\-JudgeExam

### 3\.1Source

Ukraine’s Higher Qualification Commission of Judges \(ВККС\) publishes the complete question bank used for the anonymous written testing of candidates for appellate\-court judgeships, together with the key for every item\. We use the bank published by Commission decision of 15 July 2024, No\. 221/зп\-24: five documents totalling 1,672 pages, covering general legal knowledge and the administrative, commercial, criminal, and civil specialisations\. Each item has exactly four options, of which the document marks exactly oneправильна\(correct\) and threeнеправильна\(incorrect\)\.

The bank is published as an annex to a decision \(ршення\) of the Commission\. Article 8\(1\)\(3\) of the Law of Ukraine*On Copyright and Related Rights*\(No\. 2811\-IX\) places outside copyright protection “acts of state authorities … official documents of a political, legislative, administrative and judicial character \(laws, decrees, resolutions,*decisions*, state standards and the like\)”\. On that basis we redistribute the bank with attribution to the issuing decision\. The items are questions on points of law and contain no personal data\.

### 3\.2What an item looks like

Table 1:Itemvkks\-2024\-commercial\-2237, with an English gloss; the key, C, is in bold\. The options are self\-contained legal propositions, which is the property that makes the blind condition meaningful\. Article 31 of the Civil Code grants a person under 14 the right*самостйно вчиняти дрбн побутов правочини*— so D substitutes*рзн*\(“various”\) for*дрбн*\(“minor”\), which is not legal language, and A substitutes*майнов*\(“property”\) for the*особист немайнов*\(“personal non\-property”\) rights the article actually grants\. Telling these apart requires knowing the code, not reading the question\.Q\.Як д ма право самостйно вчиняти фзична особа, яка НЕ досягла 14 рокв?*Which acts may a natural person under 14 perform independently?*A\) Здйснювати майнов права на результати нтелектуально дяльност, що охороняються законом*Exercise property rights in protected intellectual\-property results*B\) Розпоряджатися банквським вкладом, унесеним нею на сво м’я*Dispose of a bank deposit made in their own name*C\) Вчиняти дрбн побутов правочини*Perform minor everyday transactions*D\) Вчиняти рзн побутов правочини*Perform various everyday transactions*This is the format that leaks\. Contrast it with LEXam \(§[7](https://arxiv.org/html/2608.15428#S7)\), whose four\-choice items offer options like “i und iii” — pointers into a list of statements given in the stem, which carry no legal content of their own\.

### 3\.3Extraction and its verification

The documents are ruled tables in PDF\. We extract them with a table parser and verify the result with a*second, independent*path that reconstructs rows from the table’s horizontal ruling lines, columns from its vertical ones, and cell contents from word coordinates — sharing no code with the first\. Comparison is insensitive to whitespace and line\-break hyphenation, because the two paths join wrapped words differently\.

Extraction yields11,990well\-formed items, plus 385 items the Commission itself marks as withdrawn \(Виключено ршенням\) and 40 that fail the four\-options\-one\-key invariant\. On a 200\-item stratified pilot the second path confirms the question text, the four options in order, and the key in0\.9950\.995of items\. Over the full bank it confirms the question text in0\.98960\.9896, option order in0\.98880\.9888, andthe answer key in0\.98410\.9841; the 192 items \(1\.6%1\.6\\%\) that fail at least one check are listed by identifier in the release\.

Five items are malformed in the source itself and we ship them as published rather than repairing them silently: in one the fourth option is blank in the PDF, and four carry two identical options\. None of them falls in the 400\-item sample used for the probes below, which contains no empty options\.

One extraction detail is worth reporting because it is a silent failure mode: in 942 items the cell holding the item number does not extract, which naively merges two adjacent items into one eight\-option record\. Detecting the merge by the appearance of a new question stem while options are already accumulating recovers those items; the recovered numbering contains zero duplicates within any of the five documents, which is the namespace the Commission numbers in and the check that the recovery is sound\.

## 4The Solvability Profile

Before asking what models score, we establish what trivial strategies score\. Unless noted, all numbers are over the full bank of 11,990 items, on the options as published\.

Table 2:What can be recovered without reasoning over the question\. Chance is0\.2500\.250\.Surface features carry almost nothing: the correct option is on average2\.62\.6characters longer than a distractor, and no length or position heuristic clears0\.3010\.301\. The key sits at A in28\.9%28\.9\\%of items, at B and at C in24\.8%24\.8\\%each and at D in21\.5%21\.5\\%, so a positional habit buys a little; we control for it explicitly in §[6](https://arxiv.org/html/2608.15428#S6)\.

### 4\.1Lexical search is not the mechanism

A plausible story is that the correct option is copied from a statute while distractors are invented, so a text search decides the item\. We tested this against the full text of280,059 current editionsof Ukrainian legislation \(2\.3562\.356billion characters\)\. For each option we ask whether its first twelve tokens occur verbatim anywhere in that corpus\.

Over the full bank the correct option is found verbatim in0\.5800\.580of items and distractors in0\.4380\.438— a real but weak signal\. The decisive statistic is what a searcher can do with it: exactly one option is found in19\.6%19\.6\\%of items, and there the match is the key65\.1%65\.1\\%of the time, for an overall accuracy of0\.1280\.128\. One\- and two\-token options — bare figures such as “1095” — occur verbatim94\.6%94\.6\\%of the time \(10,63710\{,\}637of47,96047\{,\}960options\)\. They inflate the raw match rates and, because they match for every option at once,*depress*what a searcher can do with them, so we also restrict to the8,0248\{,\}024items whose four options are all substantive \(three tokens or more\)\. There the searcher commits on25\.9%25\.9\\%of items and is right66\.4%66\.4\\%of the time when it does, for0\.1720\.172overall — still below chance\.

We ran this two ways: as phrase queries against a full\-text index, and as a single pass over an exported corpus with an inverted index of the option phrases\. The two implementations, on different machines with no shared code, agree on97\.11%97\.11\\%of individual option lookups over a shared 1,000\-item sample\. They do not agree exactly on what a searcher gets: the Postgres route is the more conservative, at0\.1120\.112against0\.1360\.136on those items\. That gap is not the128128timed\-out Postgres queries: dropping every item they touch leaves it at0\.1040\.104against0\.1270\.127\. It is a genuine recall difference,109109phrases found only by the offline index against33found only by Postgres\. We report the higher of the two throughout, so the figure above is the one least favourable to our argument\.

Critically, blind accuracy does*not*concentrate on items whose key is verbatim statutory text\. On the 199 items of the pilot for which both paths ran, Haiku scores0\.4330\.433on verbatim keys against0\.3470\.347on the rest \(n=97n=97and7575; it returned no parseable letter on 27 of the 199\) and Sonnet0\.3660\.366against0\.4140\.414\(n=112n=112and8787\) — the two models disagree even on the direction, and every interval overlaps\. The leak is not quotation\. It is plausibility\.

## 5The Blind Gate

### 5\.1Protocol

For each item we present the four options with the question withheld and ask for a single letter, repeating over eight random permutations of option order\. Under the null of no leakage each trial is correct with probability0\.250\.25, so eight trials give a binomial reference:P⁡\(X≥5\)=0\.027P\(X\\geq 5\)=0\.027\. We reject an item at five or more hits, and additionally require at least six of the eight trials to have produced a parseable letter, so that an item is never certified clean on the strength of two or three responses\. We do not reject on the lower tail — zero hits out of eight occurs for10\.0%10\.0\\%of clean items — and treat below\-chance behaviour as a set\-level diagnostic instead\.

Two caveats on that reference\. The eight orders are drawn independently rather than sampled without replacement, so an item sees6\.96\.9distinct orders on average and all eight only27%27\\%of the time; at temperature 0 a repeated order repeats its answer, which makes the trials slightly less independent than the binomial assumes\. Recomputing the tail against seven effective trials moves the expected number of items answered blind eight times out of eight from0\.20\.2to0\.70\.7, against the1,4191\{,\}419observed, so the conclusion is unaffected and the threshold is marginally conservative\. Separately, the run did not persist per\-item hit counts and seeded from Python’s salted string hash, so the histogram in Figure[1](https://arxiv.org/html/2608.15428#S5.F1)is recoverable from our logs but the selection is not bit\-for\-bit reproducible from the release; the accepted set itself is published in full\.

Two design points matter for the numbers to mean anything\. First, the gate runs on the item as published, not on a rewritten variant\. Second,the model that selects is not the model that reports: gating uses Haiku 4\.5 and every reported blind figure comes from models that took no part in selection\.

### 5\.2Results

Over the full bank the pooled blind rate during gating is0\.3830\.383\. The distribution of hits per item is strongly bimodal: 3,517 items \(29\.3%29\.3\\%\) are never answered blind in eight attempts, while 1,419 \(11\.8%11\.8\\%\) are answered blind in all eight — an event with probability1\.5×10−51\.5\\times 10^\{\-5\}under chance\. Leakage is concentrated in a minority of items, which is what makes selection a plausible remedy at all\.

1,419 items answered blind every time;chance predicts 0\.201000200030004000012345678Blind hits out of 8 permutationsItemsObservedChance \(binomial\)Figure 1:How often each item is answered blind, over eight permutations of option order, against what chance predicts\. The bank does not behave like a set of items with a uniform small leak: 3,517 items are never answered blind and 1,419 are answered blind every time, whereBinomial⁡\(8,0\.25\)\\mathrm\{Binomial\}\(8,0\.25\)predicts0\.20\.2items in the latter group\. That concentration is what makes selection worth attempting\.Figure[1](https://arxiv.org/html/2608.15428#S5.F1)shows the shape\. Of the8,1378\{,\}137items at four hits or fewer, nine fall to the minimum\-trials rule, so the gate retains8,128 items \(67\.8%67\.8\\%\)\. Retention varies by specialisation: administrative74\.3%74\.3\\%, commercial70\.6%70\.6\\%, civil64\.6%64\.6\\%, criminal63\.2%63\.2\\%, general61\.4%61\.4\\%\.

On those8,1288\{,\}128items the gating model now scores0\.2040\.204blind\. By its own measure the bank is clean\. Section[6](https://arxiv.org/html/2608.15428#S6)asks what that measure is worth to anyone else, and the answer is the reason this paper exists\.

## 6Does the Repair Transfer?

The gate was built with a single model\. We test whether the items it kept are clean for anyone else, on a 600\-item sample stratified by gate outcome — 400 accepted and 200 rejected — scored by thirteen models from eight vendors under a uniform protocol\. The sample is not a random draw from the bank, so the figures below are conditional on gate outcome rather than bank\-wide averages\. All models were reached through Amazon Bedrock in August 2026 with greedy decoding wherever the provider accepts it\. The blind condition on accepted items was subsequently re\-run on the*entire*accepted set with a 4,096\-token budget; the two runs agree closely where they overlap\. On the items both scored, ten of thirteen models differ by less than0\.0130\.013and the largest difference is Pixtral’s0\.0170\.017on the233233items it shares; Sonnet and Haiku return identical predictions on all400400\. The differences between the columns below are therefore sampling, not protocol\. Pixtral Large is quota\-bound on its inference profile and was re\-run on the whole accepted set\. One warning for anyone subsampling the released file: the bank is stored in the order of the five source documents, and the documents*are*the specialisations, so its first2,0002\{,\}000items are74%74\\%administrative\. Stratify rather than slice\.

Table 3:Blind \(options only\) and full accuracy, ordered by full\-condition accuracy\. Chance is0\.2500\.250\.Blind figures on the accepted set are measured on all8,1288\{,\}128items; the rejected\-set and full\-condition columns remain on the 600\-item sample, so their intervals are the wider ones\. Two models are set apart because their numbers are not evidence of recognition: Haiku is the gate’s own model, and Llama 3\.1 8B answersAto92%92\\%of items, which on its own earns0\.2870\.287here \(Table[4](https://arxiv.org/html/2608.15428#S6.T4)\)\. Blindnnis8,1288\{,\}128except DeepSeek R17,9767\{,\}976, Llama 3\.1 8B7,8867\{,\}886, Nova Micro8,0398\{,\}039, Ministral8,1268\{,\}126and Pixtral8,1258\{,\}125; fullnnis400400except DeepSeek R1352352, Pixtral365365, Nova Micro388388and Llama 3\.1 8B202202\.ModelVendorBlind, acceptedBlind, rejectedFull acc\.Full rej\.GPT\-5\.6OpenAI0\.515\[\.504,\.526\]0\.789 \[\.727,\.840\]0\.9580\.960Sonnet 4\.6Anthropic0\.320\[\.310,\.330\]0\.730 \[\.665,\.787\]0\.7400\.905DeepSeek R1DeepSeek0\.257 \[\.247,\.266\]0\.665 \[\.593,\.729\]0\.6680\.845Nova ProAmazon0\.232 \[\.223,\.241\]0\.645 \[\.577,\.708\]0\.5800\.739Qwen3 32BAlibaba0\.244 \[\.234,\.253\]0\.665 \[\.597,\.727\]0\.5520\.720Pixtral LargeMistral0\.239 \[\.230,\.248\]0\.602 \[\.531,\.669\]0\.5480\.718Llama 3\.3 70BMeta0\.237 \[\.228,\.247\]0\.610 \[\.541,\.675\]0\.5400\.790Gemma 3 12BGoogle0\.230 \[\.221,\.239\]0\.595 \[\.526,\.661\]0\.4720\.745Ministral 8BMistral0\.239 \[\.229,\.248\]0\.540 \[\.471,\.608\]0\.4630\.740Nova 2 LiteAmazon0\.225 \[\.216,\.234\]0\.545 \[\.476,\.613\]0\.4600\.660Nova MicroAmazon0\.233 \[\.224,\.242\]0\.518 \[\.448,\.586\]0\.4360\.658Llama 3\.1 8BMeta*0\.292*\[\.282,\.302\]0\.337 \[\.274,\.406\]0\.4260\.531Haiku 4\.5Anthropic*0\.204*\[\.195,\.213\]0\.800 \[\.739,\.850\]0\.5750\.805Four readings follow, and the first is the answer to the question in the heading\.

#### A gate built with one model does not protect against a better one\.

The gate ran until Haiku 4\.5 scored0\.2040\.204blind on the items it kept — below chance, its own leakage selected away\. GPT\-5\.6, which took no part in that selection, answers0\.5150\.515of those same items with the question hidden, and Sonnet 4\.60\.3200\.320\. Neither figure is a positional artefact: corrected for each model’s own answer\-position habit their excess is\+0\.265\+0\.265and\+0\.081\+0\.081, the only two excesses in the set \(Table[4](https://arxiv.org/html/2608.15428#S6.T4)\)\. This is a demonstration rather than an inference, and it is the paper’s central result: filtering a bank against one model leaves the strongest models in the set reading the options exactly as before\.

#### Position bias explains almost all of the rest\.

Unlike the gate, the sweep presents each item once in its published order, so a model with a positional habit could score above chance without reading anything\. Those habits are large and vendor\-specific: Ministral picksAin2%2\\%of blind trials, Pixtral picksBin46%46\\%, Sonnet 4\.6 picksDin42%42\\%, and Llama 3\.1 8B picksAin92%92\\%\. So for each model we compute what its own answer\-position distribution alone would earn against the gold\-position distribution of the accepted set, which isA\-heavy \(0\.2900\.290,0\.2450\.245,0\.2410\.241,0\.2230\.223\)\. That position\-only expectation runs0\.2370\.237–0\.2870\.287across models, and subtracting it is what separates recognition from habit\.

Table 4:What is left after each model’s own positional habit is subtracted\. Blind accuracy is measured on all8,1288\{,\}128accepted items\. “Habit” is the score that model’s answer\-position distribution alone earns against the gold positions of this set\. Only two models have an excess worth the name\.ModelABCDBlindHabitExcessGPT\-5\.60\.240\.270\.270\.220\.5150\.250\+0\.265\+0\.265Sonnet 4\.60\.080\.200\.300\.420\.3200\.238\+0\.081\+0\.081DeepSeek R10\.120\.240\.290\.350\.2570\.242\+0\.015\+0\.015Qwen3 32B0\.140\.270\.300\.300\.2440\.244\+0\.000\+0\.000Ministral 8B0\.020\.250\.450\.290\.2390\.238\+0\.001\+0\.001Llama 3\.3 70B0\.080\.280\.310\.330\.2370\.240−0\.003\-0\.003Nova Micro0\.090\.310\.220\.390\.2330\.240−0\.007\-0\.007Nova Pro0\.170\.280\.270\.280\.2320\.246−0\.014\-0\.014Gemma 3 12B0\.080\.280\.400\.230\.2300\.242−0\.012\-0\.012Nova 2 Lite0\.030\.300\.290\.390\.2250\.237−0\.012\-0\.012Pixtral Large0\.100\.460\.250\.200\.2390\.244−0\.005\-0\.005Llama 3\.1 8B0\.920\.000\.070\.010\.2920\.287\+0\.005\+0\.005Haiku 4\.50\.160\.440\.220\.190\.2040\.247−0\.043\-0\.043Two models keep a real excess: GPT\-5\.6 at\+0\.265\+0\.265and Sonnet 4\.6 at\+0\.081\+0\.081\. Every other held\-out model lands within0\.0150\.015of what its habit alone would earn; the gate’s own model sits0\.0430\.043*below*its habit, which is what selecting on it was supposed to do\. Llama 3\.1 8B is the instructive case: it scores0\.2920\.292blind, comfortably above chance and above four models we do not call leaky, purely because it answersAto92%92\\%of items and the key sits atAin29%29\\%of them\. Its excess is\+0\.005\+0\.005\. Read without the habit column it would look like a third leaker; it is not reading the options at all\.

#### The gate found something real, for almost everyone\.

On rejected items eleven of the twelve held\-out models score0\.5180\.518–0\.7890\.789, far above chance, and the accepted\-versus\-rejected gap is positive for all twelve\. The exception is Llama 3\.1 8B at0\.3370\.337, which is what a model that answersAto everything scores on a set where the key is atAmore often than not\. For every model that reads the options at all, one model’s blind failures are not idiosyncratic: they identify items that leak to the rest\.

#### The selection also seems to carry a signature of the model that made it\.

The gate keeps the items Haiku 4\.5 could*not*answer blind — items where Haiku’s preferred option is a distractor\. Any model that shares Haiku’s preferences inherits that preference, and with it the wrong answer\. Agreement with Haiku’s blind pick runs0\.420\.42–0\.520\.52for ten of the twelve where independence would give0\.250\.25, and it predicts the damage: agreement correlates with the accepted\-to\-rejected gap atr=0\.892r=0\.892over twelve models \(Table[5](https://arxiv.org/html/2608.15428#S6.T5)\)\. That correlation leans on one point\. Llama 3\.1 8B agrees with Haiku*less*than chance, at0\.1820\.182, because it is not reading the options, and it has much the smallest gap,0\.0450\.045; drop it andrrfalls to0\.5770\.577over the remaining eleven, which is the honest strength of the relationship among models that do read\. GPT\-5\.6, the other model outside the band at0\.3490\.349, is also the one the gate damages least\.

Table 5:Agreement with the gating model’s blind pick, and the cost of that agreement\. Independence would put agreement at0\.2500\.250\. The gap is blind accuracy on rejected items minus blind accuracy on accepted items; the more a model answers like the gate, the more the gate’s selection costs it\.This is the finding that a 400\-item sample could not have produced\. At that size every interval was wide enough to cover chance, and eleven models read as “at chance” — a tidy result that concealed both facts above: that nine of them extract nothing at all once habit is subtracted, and that what they do extract is shaped by the model that did the selecting\.

#### Leakage scales with capability\.

Ranking the eleven held\-out models by full\-condition accuracy reproduces their ranking by blind accuracy closely \(Table[3](https://arxiv.org/html/2608.15428#S6.T3)\)\. Pearsonr=0\.916r=0\.916\(95% CI\[0\.703,0\.978\]\[0\.703,0\.978\]\); leaving out any single model gives0\.8380\.838–0\.9430\.943, and rank correlation is0\.7730\.773\.

That correlation is partly mechanical: thexx\-axis is full accuracy on the*same*items as theyy\-axis, so item\-level noise is shared\. Replacing thexx\-axis with an independent capability measure — full accuracy on the*rejected*items, a disjoint set — givesr=0\.813r=0\.813\(95% CI\[0\.415,0\.950\]\[0\.415,0\.950\]\): still positive, with zero outside the interval, and stable under leave\-one\-out \(0\.8090\.809–0\.8540\.854\)\. Note what the association now rests on: with the blind axis compressed into a band of0\.0320\.032for nine of the eleven, it is carried mostly by the two models that leak\. Among the nine alone, on the same disjoint axis, it isr=0\.713r=0\.713— suggestive, but over a band narrower than the intervals of the quantity it is meant to explain\. We report an observed association\.

The more robust statement is a ratio, and Figure[2](https://arxiv.org/html/2608.15428#S6.F2)shows its shape\. Blind accuracy is a fixed fraction of full accuracy across the capability range:0\.3840\.384–0\.5380\.538, mean0\.4630\.463\. That arithmetic holds, but it should not be read as “a model recovers46%46\\%of its competence without the question”\. For nine of the eleven the blind score is their positional habit and nothing else, so the fraction describes where habit happens to land, not partial competence\. The reading survives only for the two models with a real excess\.

0\.40\.40\.450\.450\.50\.50\.550\.550\.60\.60\.650\.650\.70\.70\.750\.750\.80\.80\.850\.850\.90\.90\.950\.95110\.20\.20\.30\.30\.40\.40\.50\.50\.60\.6chancenine models at their own habitSonnet 4\.6GPT\-5\.6Full\-condition accuracy on gate\-accepted itemsBlind accuracy \(options only\)Figure 2:Blind accuracy against full\-condition accuracy on the gate\-accepted items, eleven held\-out models from eight vendors\. Bars are 95% Wilson intervals on the blind estimate, now measured on the whole accepted set rather than a 400\-item sample, which is why they are short; the horizontal position still carries the sample’s error\. The plotted correlation,r=0\.916r=0\.916, shares items between the two axes; against an independent capability estimate on disjoint items it isr=0\.813r=0\.813\(§[6](https://arxiv.org/html/2608.15428#S6)\)\. The gate was built with a model absent from this plot, and Llama 3\.1 8B is absent because it answersAto92%92\\%of items\. The bracketed nine sit within0\.0150\.015of their own positional habit \(Table[4](https://arxiv.org/html/2608.15428#S6.T4)\)\.The implication is the paper’s main point, and it comes in two parts that should not be conflated\. The first we observe directly: a bank filtered until one model is at chance is not clean for a stronger one\. Our accepted set was gated against Haiku 4\.5, and GPT\-5\.6 answers0\.5150\.515of it blind\. That is a demonstration, not an inference\.

The second is a prediction\. If blind accuracy keeps tracking capability beyond the range we tested, then filtering against the models available today will not hold against the models available later, and each round of filtering will remove the items that discriminate best\. We have eleven models spanning full\-condition accuracy from0\.4360\.436to0\.9580\.958; whether the relationship continues above that is untested, and we state it as a conjecture rather than a result\.

### 6\.1The numbers are not an artefact of how we asked

Every blind figure above comes from one prompt, so we re\-ran the blind condition on the 400 gate\-accepted items under five phrasings: the original; a minimal form with no framing at all \(“choose one option”\); the original with the word*legal*removed, to test whether naming the domain is itself what primes recognition; an English instruction over Ukrainian options; and an explicit “this is a guessing task” framing\. Two leaking models and one that sits at chance were run on all five\.

Table 6:Blind accuracy under five prompt phrasings, 400 gate\-accepted items\. Chance is0\.2500\.250\. These figures are on a 400\-item subset of the accepted set, not the whole of it, so they sit above the full\-scale numbers in Table[3](https://arxiv.org/html/2608.15428#S6.T3)— for these three models the subset runs0\.0090\.009to0\.0240\.024leakier than the set it was drawn from, and every model in it inherits that\. What the table is for is the*spread*within a model, which the shared sample does not distort\.Three things follow\. The leak survives every phrasing: Sonnet 4\.6’s worst variant is0\.3100\.310with a lower confidence bound of0\.2670\.267, still above chance, and GPT\-5\.6’s worst is0\.4630\.463\. Naming the domain helps a little and explains almost nothing — dropping*legal*costs0\.0330\.033for Sonnet and0\.0340\.034for GPT\-5\.6, leaving both far above chance\. And the control behaves: Nova Pro stays at chance under all five phrasings, every interval covering0\.2500\.250, so the probe does not manufacture signal where there is none\. The within\-model spread,0\.0330\.033–0\.0610\.061, is far smaller than the0\.180\.18that separates Sonnet from GPT\-5\.6 on these items, so which models leak is not a function of how we asked\.

That covers the blind prompt, which leaves three other things fixed at one realisation each, all of them underneath the ratio: the wording of the*full*condition, which is its denominator; the labels on the options, which are LatinA\)–D\)set over Ukrainian text, so that a letter prior would be indistinguishable from recognition of content; and the order of the options\. We vary the first two here and the third in §[6\.2](https://arxiv.org/html/2608.15428#S6.SS2), moving one factor at a time from the configuration used throughout\.

Table 7:Presentation robustness on the same 400 gate\-accepted items\. Blind chance is0\.2500\.250\. Marker rows change only the labels; the Latin rows repeat the baseline for reference\. No model answered with a Latin letter when shown Cyrillic labels, so the conventions are not being conflated\.Neither matters\. The full condition moves by0\.0050\.005–0\.0370\.037across five phrasings, so the denominator of the ratio is not a property of our prompt: recomputed against each of them, the blind\-to\-full ratio spans0\.4520\.452–0\.4640\.464for Sonnet and0\.5440\.544–0\.5470\.547for GPT\-5\.6, well inside the0\.3840\.384–0\.5380\.538range we report across models\. And the labels are inert in both conditions and for all three models, the largest shift being0\.0270\.027\. The Latin alphabet was not doing the work\.

### 6\.2Position bias is a different thing from the leak

The blind numbers so far use the option order as printed in the bank\. A model that simply favours one slot would score above chance whenever the key happened to be there, which in a single\-order measurement is indistinguishable from recognising the option’s content\. So we forced the key into each of the four slots in turn, keeping the distractors in their relative order: 400 items×\\times4 placements×\\times3 models, blind\.

This yields a test that needs no modelling assumption\. If a model’s choice were independent of what the options say, thenP⁡\(correct∣key in slot​j\)P\(\\text\{correct\}\\mid\\text\{key in slot \}j\)is justqjq\_\{j\}, the rate at which it reaches for slotjj; and because theqjq\_\{j\}sum to one, the four accuracies must sum to one as well\. Their mean is then exactly chance,0\.2500\.250, however lopsided the preference\. Whatever exceeds that is content\. Since each item contributes four correlated observations, we take the interval at item level rather than treating1,6001\{,\}600responses as independent\.

Nova Pro \(sum 1\.08\)Sonnet 4\.6 \(sum 1\.43\)GPT\-5\.6 \(sum 2\.01\)ABCDABCDABCD0\.00\.20\.40\.6Slot holding the keyBlind accuracyFigure 3:Blind accuracy with the key forced into each slot \(bars\) against how often the model picks that slot at all \(points\)\. Under content\-independence the two coincide and the bars sum to1\.001\.00\. Nova Pro has a real slot preference and no gap; Sonnet has the largest preference in the set and a clear gap; GPT\-5\.6 has almost no preference and the largest gap of all\.Figure[3](https://arxiv.org/html/2608.15428#S6.F3)shows the three cases and Table[8](https://arxiv.org/html/2608.15428#S6.T8)gives the numbers\. Sonnet 4\.6 turns out to carry a severe positional prior — it picks the last slot0\.4500\.450of the time and the first0\.0770\.077— which drags its blind accuracy from0\.1220\.122with the key atAto0\.6500\.650with the key atD, a spread of0\.5280\.528, an order of magnitude larger than anything in Table[7](https://arxiv.org/html/2608.15428#S6.T7)\. But the accuracies sum to1\.4271\.427, not1\.0001\.000, and the lift over the preference is positive in every slot, so content is being read on top of the prior\. Nova Pro, the control, has a genuine preference of its own and no content: it sums to1\.0851\.085and its position\-free estimate,0\.2710\.271, does not separate from chance\. GPT\-5\.6 is the opposite of Sonnet, picking the four slots almost uniformly and scoring near0\.500\.50wherever the key sits\.

Table 8:Blind solvability with position bias removed by construction: the mean over the four forced placements, with an item\-level95%95\\%interval on 400 items\. The single\-order column is the same 400\-item measurement, not the full\-scale figure of Table[3](https://arxiv.org/html/2608.15428#S6.T3)\.Two things follow\. Order does not carry the result: on this subset the position\-free estimates sit within0\.0220\.022of the single\-order values measured on the same items, and for Sonnet the single\-order figure is if anything the conservative one, so nothing here depends on where the key happened to be printed\. And position bias and leak exploitation are orthogonal\. The model with the largest slot preference is not the one that leaks most, and the model with almost none leaks most of all\. This matters for remedies: permuting options, the standard fix for position bias and the one our own gate relies on, does nothing whatever about content that is recognisable on its own\.

The labelling sweep of §[6\.1](https://arxiv.org/html/2608.15428#S6.SS1)also speaks to the mechanism\.[Zheng et al\. 2024](https://arxiv.org/html/2608.15428#bib.bib14)trace selection bias to token bias — probability mass attached to the option\-ID tokens themselves — which predicts that the preference should move when the tokens change\. It largely does not\. Sonnet 4\.6 picks the first slot0\.0920\.092of the time under LatinA–D,0\.0980\.098under CyrillicА–Гand0\.0900\.090under digits1–4; Nova Pro likewise stays in a narrow band \(0\.1800\.180,0\.1470\.147,0\.1820\.182\)\. What the labels do change is the size of the pull towards the last slot, which for Sonnet runs0\.4150\.415,0\.4770\.477and0\.3100\.310\. On our items the bias is positional first and token\-dependent second\.

### 6\.3Reasoning at inference time is not what makes options solvable

Our models run at vendor defaults, so the set mixes models that reason before answering with models that do not — as[Fan et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib4)also do, and for the same reason: reasoning cannot be switched off in GPT\-5\.6, which rejectsreasoning\_efforton Bedrock as it rejectstemperature\. That leaves an objection open: perhaps the0\.180\.18separating GPT\-5\.6 from Sonnet blind is inference\-time compute rather than what they read from the options\. What cannot be equalised downwards can be moved upwards and measured\.

Table 9:Reasoning interventions on the 400 gate\-accepted items\.pp\-values are McNemar tests on the paired items against that model’s direct\-answer row\. The extended\-thinking row is reported for completeness only: Anthropic’s API refuses temperature 0 when thinking is enabled, so that row moves two things at once\.ModelConditionBlindppFullOut tok\.Sonnet 4\.6direct answer0\.343—0\.7384Sonnet 4\.6extended thinking on0\.343—0\.78559Sonnet 4\.6step\-by\-step prompt0\.3880\.1390\.805619Nova Prodirect answer0\.255—0\.578260Nova Prostep\-by\-step prompt0\.2280\.2840\.578530DeepSeek R12,048\-token budget0\.253—0\.668—DeepSeek R14,096\-token budget0\.231—0\.6481,363Four measurements agree\. Enabling extended thinking leaves Sonnet’s blind accuracy exactly where it was, though that alone proves little — the model declines the budget, spending 59 output tokens of an allowed 2,000\. Forcing the reasoning at the prompt, as LEXam does for every model, makes it reason at 619 tokens against 4, and moves blind accuracy from0\.3430\.343to0\.3880\.388, which a paired test does not separate from noise\. The same instruction on the same items raises the*full*condition from0\.7380\.738to0\.8050\.805\(p=0\.001p=0\.001\)\. Reasoning that demonstrably helps when the question is present does not measurably help when it is absent, which is what one should expect: with no question there is nothing to reason towards\. Nor does it manufacture the ability — Nova Pro under the same instruction stays at chance\.

Across models the association fails in both directions: DeepSeek R1 spends1,3631\{,\}363output tokens per blind item and scores0\.2310\.231, at chance, while GPT\-5\.6 also reasons and scores0\.5240\.524on this subset\. The two reasoning models sit at the two extremes, with a conventional model between them\. Inference\-time reasoning is neither necessary nor sufficient for option\-only solvability\.

Re\-running R1 at LEXam’s 4,096\-token budget rather than ours also resolves the one real parsing casualty in this paper: on the blind condition over accepted items, unparsed responses fall from18\.0%18\.0\\%to1\.5%1\.5\\%— the13\.1%13\.1\\%quoted elsewhere pools all four of its cells — while the estimates move only within their intervals \(0\.253→0\.2310\.253\\to 0\.231blind on328328then394394parsed items,0\.668→0\.6480\.668\\to 0\.648full\)\. The truncation cost coverage; it was not biasing the numbers\.

## 7A Benchmark That Does Not Leak, and Why

Filtering failed, and rewriting distractors failed\. The one thing that works is not a repair at all but a property some banks have from the start\. If option\-only solvability were a general property of legal exam items, it should appear in other exam\-derived benchmarks\. We ran the same blind probe on the four\-choice split of LEXam\([Fan et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib4)\), 1,655 items from Swiss university law exams, with the two models that leaked most on our bank\.

Table 10:The same blind probe on LEXam’s four\-choice split\. Chance is0\.2500\.250\.nncounts responses from which a letter could be parsed; GPT\-5\.6 produced 34 unparseable answers in the full condition and none in the blind one\.Both models sit at chance, with the upper confidence bound below0\.2500\.250\. The same GPT\-5\.6 that recovers0\.5150\.515of our gate\-accepted items blind recovers nothing here\.

every LEXam optionfits in 33 characters0\.000\.250\.500\.751\.00050100150Option length \(characters\)Cumulative share of optionsUA\-JudgeExamLEXam 4\-choiceOptions: propositions vs pointerschance0\.00\.20\.40\.6GPT\-5\.6Sonnet 4\.6Blind accuracySame probe, same modelsFigure 4:Left: the cumulative distribution of option length in the two banks\. LEXam’s curve is complete at 33 characters — its longest option anywhere — while59\.6%59\.6\\%of our options are longer than that\. Ours are legal propositions that can be judged on their own; LEXam’s are pointers into a list given in the stem\. Right: what the same two models score with the question hidden\. The design choice on the left produces the difference on the right\.The reason is visible in the items \(Figure[4](https://arxiv.org/html/2608.15428#S7.F4)\)\. In*every one*of LEXam’s1,6551\{,\}655four\-choice items, all four options are*references*to statements enumerated in the stem — “i und iii”, “ii and iii”, “none of the statements”\. The median option is 9 characters against 42 in our bank, and the point is sharper than the median: the longest option anywhere in the split is 33 characters,“ii, iii, iv, v, vii, viii, und ix”\. No legal proposition of any kind appears in an option position\. A reference carries no content of its own, so there is nothing for a reader to recognise when the stem is hidden\. LEXam is immune to this failure by construction, not by filtering\.

This cuts two ways\. It validates the probe: an instrument that reports leakage should report chance where chance is the ground truth, and it does\. And it relocates our finding\. Option\-only solvability is not a property of legal exams, nor of any particular jurisdiction or language\. It is a property of*item format*: options that are self\-contained legal propositions can be judged on their own, and options that are pointers cannot\. Banks written in the first style — which includes most professional licensing material we are aware of, and ours — need the blind baseline reported\. Banks written in the second do not\.

## 8Remedies: One Clear Failure, Two Cautions

Filtering is not the only repair one might try, so we tried the others\. None of them lands on chance either, which is the same failure in a different guise: an item set that a model gets wrong at a predictable rate is not a neutral item set\.

#### Swapping in real answers overshoots\.

If distractors leak because they read as implausible law, replace them with propositions that are real law: the correct answers of other items, chosen from the same specialisation, matched on answer type and length, with the key’s position re\-randomised\. Blind accuracy for Sonnet 4\.6 moves from0\.3860\.386\[0\.321,0\.455\]\[0\.321,0\.455\]to0\.168\\mathbf\{0\.168\}\[0\.122,0\.226\]\[0\.122,0\.226\]\(n=197n=197, intervals disjoint\) while full accuracy is essentially preserved \(0\.792→0\.7460\.792\\to 0\.746\)\. The mechanism is real and controllable — and the result is useless\. A set on which a model scores0\.1680\.168blind is as exploitable as one on which it scores0\.3860\.386; an adversary inverts it\. Selecting donors by similarity to the*question*makes the three distractors mutually coherent and leaves the key as the odd one out; a variant selecting donors by similarity to the*key*inverts the artifact instead of removing it\. We could not find a donor rule that lands on chance\.

#### Model\-written distractors may leak to the model that wrote them\.

Generating distractors with Sonnet 4\.6, verifying with a second model that each is definitively wrong, and gating the result yields items on which Sonnet 4\.6’s blind accuracy is0\.3810\.381\[0\.296,0\.473\]\[0\.296,0\.473\], against0\.2740\.274\[0\.201,0\.363\]\[0\.201,0\.363\]on the human\-written originals of the same 113 items\. The point estimate moves in the direction one would fear — the generator’s own family recognising its own writing — but atn=113n=113the intervals overlap and we cannot call it established\. We report it because the design risk is cheap to avoid: do not generate distractors with a model from a family you intend to evaluate\.

#### No evidence that negation items drive the effect\.

It is natural to suspect items phrased “which is*not*…”, where the key is the odd one out by construction\. Sonnet 4\.6 scores0\.2500\.250\[0\.138,0\.411\]\[0\.138,0\.411\]blind on the 36 negation items in the pilot and0\.4160\.416\[0\.343,0\.493\]\[0\.343,0\.493\]on the 161 others\. The point estimates point away from the hypothesis, but 36 items give an interval too wide to separate the two, so this rules the explanation neither in nor out\.

## 9Limitations

#### The central claim is a demonstration, not a law\.

It rests on one bank and one gating model: a set filtered against Haiku 4\.5 is not clean for GPT\-5\.6\. It does not follow that every filter fails against every stronger model, only that this one did, decisively, and that nothing in the procedure prevents it\. A practitioner who wants a clean set should gate with several models from different vendors and accept the lower yield\.

#### What we measured at full scale, and what we did not\.

Only the blind condition on accepted items was re\-run on all8,1288\{,\}128; the rejected\-set and full\-condition columns still rest on the 600\-item sample, with half\-widths near±0\.05\\pm 0\.05and±0\.07\\pm 0\.07\. Because every model was scored on the same 400\-item draw, their errors are correlated and the whole column moved together when we measured the full set — by0\.0100\.010on average, which no per\-model interval predicted\. The presentation and position checks of §[6\.1](https://arxiv.org/html/2608.15428#S6.SS1)–§[6\.2](https://arxiv.org/html/2608.15428#S6.SS2)likewise cover one item set and three models rather than all eleven\.

#### Coverage is uneven across models\.

Llama 3\.1 8B returns no parseable letter on47\.5%47\.5\\%of full\-condition calls, and a larger token budget does not help; its full\-condition figure rests on 202 of 400 items, which is why it is set apart and excluded from the correlation\. DeepSeek R1 \(13\.1%13\.1\\%\) and Pixtral Large \(5\.9%5\.9\\%\) were budget\-bound and recover at4,0964\{,\}096tokens\. The other nine models parse above99\.5%99\.5\\%\.

#### What the design cannot tell us\.

We have no human blind baseline, so we cannot say whether a Ukrainian lawyer shown only the options would score near the pooled0\.3830\.383we report for the gating model, which would make the effect a property of the item\-writing genre rather than of models\. The capability association rests on eleven models, none below0\.4360\.436full\-condition accuracy, and the claim that item format governs the effect rests on a comparison of two benchmarks\. Extraction fidelity is verified end\-to\-end on 200 items and by aggregate on the rest\. Finally, GPT\-5\.6 reaches0\.9580\.958on the accepted set: for the strongest model tested the filtered benchmark is close to saturated, and its remaining discriminative value sits in the models below it\.

## 10Conclusion

Multiple\-choice benchmarks should report what a model scores without the question next to what it scores with it — and should subtract, from that score, what the model’s own answer\-position habit would have earned anyway\. The raw blind number is not interpretable alone\. On our gate\-accepted set Llama 3\.1 8B scores0\.2920\.292blind — above every held\-out model except the two that actually leak, and three and a half points above DeepSeek R1 — entirely because it answersAto92%92\\%of items and the key sits atAin29%29\\%of them\. Corrected for habit, its excess is\+0\.005\+0\.005, and only two of twelve models read anything out of the options at all — but those two read a great deal, GPT\-5\.6 recovering0\.5150\.515of a set that was filtered until another model could not\.

The headline of that measurement is what filtering does not buy\. Our gate ran until Haiku 4\.5 scored0\.2040\.204on the items it kept; GPT\-5\.6, which took no part in the selection, answers0\.5150\.515of the same items with the question hidden\. Filtering against one model does not neutralise a bank for a better one, and there is no version of the procedure we tested that does\. The surviving items also appear to carry a signature of the model that selected them — agreement with its blind choices tracks how far the selection sets another model back — though that relationship rests on twelve points and one of them carries it, so we offer it as an observation to test rather than a result\.

Both of these are invisible at the sample sizes ordinarily used\. On 400 items nine models read as “statistically at chance”, a tidy result that concealed the habit correction and the rest\. Scoring every model on one shared sample also correlates their errors, so the whole column can move together — and did, by0\.0100\.010, when we measured the full set\.

The same probe on LEXam returns chance, for a reason that is a design choice rather than an accident: its options point at statements in the stem instead of standing alone\. Where options must be self\-contained legal propositions, publish the habit\-corrected blind baseline per model alongside the headline score\. Where the format allows reference\-style options, the problem does not arise\.

## Appendix APrompts

The prompts below are in Ukrainian, matching the language of the bank; the LEXam comparison in §[7](https://arxiv.org/html/2608.15428#S7)uses the same two prompts in English, since that split is German and English\. Options are presented asA\)–D\)in the order given, except in the gate, where the order is permuted per trial\.

#### Blind condition\.

Нижче наведено чотири варанти вдповд на приховане юридичне питання\. Питання вам не показано\. Оберть той варант, який наймоврнше правильним\. Вдповдайте ЛИШЕ одню лтерою: A, B, C або D\.

#### Full condition\.

Питання квалфкацйного тестування для кандидатв на посаду судд\. Оберть диний правильний варант\. Вдповдайте ЛИШЕ одню лтерою: A, B, C або D\.

#### Blind condition, LEXam\.

Below are four answer options to a hidden legal exam question\. The question is not shown to you\. Choose the option most likely to be correct\. Reply with ONE letter only: A, B, C or D\.

#### Full condition, LEXam\.

Legal exam question\. Choose the single correct option\. Reply with ONE letter only: A, B, C or D\.

Vendors answer in three shapes — a bare letter, an explicitВдповдь: Xfollowed by prose, or a paragraph of reasoning ending in the letter — so the parser tries an explicit\-marker match, then a leading letter, then the last standalone letter\. Unparsed rates are below0\.5%0\.5\\%for nine of the thirteen models\. The exceptions, on the 600\-item sweep, are DeepSeek R1 \(13\.1%13\.1\\%\), Llama 3\.1 8B \(25\.5%25\.5\\%, and47\.5%47\.5\\%in the full condition alone\), Pixtral Large \(5\.9%5\.9\\%\) and Nova Micro \(2\.4%2\.4\\%\)\. Unparsed responses are excluded from that model’s denominators rather than scored as wrong\.

## Appendix BModels

All models were reached through Amazon Bedrock in August 2026\. Snapshot identifiers are given in full, since several of these aliases will move\. The lower half of the conventional group — Qwen3 32B, Gemma 3 12B, Ministral 8B, Nova Micro and Llama 3\.1 8B — was added to extend the capability range downwards; three of those overlap with the small open\-source group evaluated by[Fan et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib4), which makes the two model sets partially comparable\.

Table 11:Model identifiers, whether the provider accepted a temperature setting, and whether the model reasons before answering at its default settings\. The region column is where the 600\-item sweep ran; the full\-scale blind run sharded Qwen3, Gemma, Ministral and Pixtral across several regions to work around per\-profile quotas, which is recorded per call in the release\. Following[Fan et al\. 2026](https://arxiv.org/html/2608.15428#bib.bib4), reasoning and conventional models are grouped rather than equalised: reasoning cannot be switched off in GPT\-5\.6, which rejectsreasoning\_efforton Bedrock as it rejectstemperature\.ModelBedrock identifierRegionGreedyReasons*Reasoning models*GPT\-5\.6global\.openai\.gpt\-5\.6\-soleu\-central\-1noyesDeepSeek R1us\.deepseek\.r1\-v1:0us\-east\-1yesyes*Conventional models*Sonnet 4\.6eu\.anthropic\.claude\-sonnet\-4\-6eu\-central\-1yesnoHaiku 4\.5eu\.anthropic\.claude\-haiku\-4\-5\-20251001\-v1:0eu\-central\-1yesnoNova Proeu\.amazon\.nova\-pro\-v1:0eu\-central\-1yesnoNova 2 Liteeu\.amazon\.nova\-2\-lite\-v1:0eu\-central\-1yesnoPixtral Largeeu\.mistral\.pixtral\-large\-2502\-v1:0eu\-central\-1yesnoLlama 3\.3 70Bus\.meta\.llama3\-3\-70b\-instruct\-v1:0us\-east\-1yesnoQwen3 32Bqwen\.qwen3\-32b\-v1:0us\-east\-1yesnoGemma 3 12Bgoogle\.gemma\-3\-12b\-itus\-east\-1yesnoMinistral 8Bmistral\.ministral\-3\-8b\-instructus\-east\-1yesnoNova Microeu\.amazon\.nova\-micro\-v1:0eu\-central\-1yesnoLlama 3\.1 8Bus\.meta\.llama3\-1\-8b\-instruct\-v1:0us\-east\-1yesnoGPT\-5\.6 rejects thetemperatureparameter outright, so its requests omit it; every other model was run at temperature 0\. The 600\-item sweep used a 2,048\-token output budget and the full\-scale blind run 4,096; the budget matters for two models for opposite reasons\. Nova Pro is merely verbose: it writes a paragraph of justification around the letter, some 260 output tokens of which nearly all is visible text\. DeepSeek R1 spends its tokens on a reasoning trace the response body does not contain, averaging1,3631\{,\}363output tokens per blind item over the full\-scale run for a handful of visible characters\. It is the model the budget bound hardest — at 2,048 tokens it failed to reach a letter on13\.1%13\.1\\%of calls, at 4,096 on1\.9%1\.9\\%, the estimates moving only within their intervals — but not the only one: Pixtral Large fell from5\.9%5\.9\\%to1\.2%1\.2\\%on the same change\. Because the set mixes models that reason at inference time with models that do not, §[6\.3](https://arxiv.org/html/2608.15428#S6.SS3)measures what that mixture is worth rather than leaving it as a caveat\.

## Data and Code

The corpus \(11,990 items\), the gated subset \(8,128\), the 600\-item cross\-vendor sample, all 9,600 blind and full predictions from the first sweep and 7,200 from the small\-model extension, the negative\-result runs, the21,60021\{,\}600calls of prompt, labelling and position ablation, the reasoning controls, the105,664105\{,\}664\-call full\-scale blind run, and the extraction, verification and gating code are released at[https://huggingface\.co/datasets/overthelex/ua\-judge\-exam](https://huggingface.co/datasets/overthelex/ua-judge-exam)\.

## References

- Balepur et al\. \[2024\]Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger\.Artifacts or abduction: How do LLMs answer multiple\-choice questions without the question?In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 10308–10330, 2024\.URL[https://aclanthology\.org/2024\.acl\-long\.555/](https://aclanthology.org/2024.acl-long.555/)\.
- Chalkidis et al\. \[2022\]Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras\.LexGLUE: A benchmark dataset for legal language understanding in English\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*, pages 4310–4330, 2022\.doi:10\.18653/v1/2022\.acl\-long\.297\.
- Cho et al\. \[2026\]Gyeongje Cho, Yeonkyoung So, and Jaejin Lee\.Choices speak louder than questions\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.URL[https://arxiv\.org/abs/2502\.18798](https://arxiv.org/abs/2502.18798)\.
- Fan et al\. \[2026\]Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus\.LEXam: Benchmarking legal reasoning on 340 law exams\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.URL[https://arxiv\.org/abs/2505\.12864](https://arxiv.org/abs/2505.12864)\.
- Guha et al\. \[2023\]Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas\-Wood, Austin Peters, et al\.LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models\.In*Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track*, 2023\.URL[https://arxiv\.org/abs/2308\.11462](https://arxiv.org/abs/2308.11462)\.
- Gururangan et al\. \[2018\]Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R\. Bowman, and Noah A\. Smith\.Annotation artifacts in natural language inference data\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\)*, pages 107–112, 2018\.
- Hendrycks et al\. \[2021\]Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball\.CUAD: An expert\-annotated NLP dataset for legal contract review\.In*Proceedings of the 35th Conference on Neural Information Processing Systems, Datasets and Benchmarks Track*, 2021\.URL[https://arxiv\.org/abs/2103\.06268](https://arxiv.org/abs/2103.06268)\.
- Katz et al\. \[2024\]Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo\.GPT\-4 passes the bar exam\.*Philosophical Transactions of the Royal Society A*, 382\(2270\), 2024\.doi:10\.1098/rsta\.2023\.0254\.
- Niklaus et al\. \[2023\]Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis\.LEXTREME: A multi\-lingual and multi\-task benchmark for the legal domain\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 3016–3054, 2023\.doi:10\.18653/v1/2023\.findings\-emnlp\.200\.
- Östling et al\. \[2023\]Andreas Östling, Holli Sargeant, Huiyuan Xie, Ludwig Bull, Alexander Terenin, Leif Jonsson, Mns Magnusson, and Felix Steffek\.The Cambridge Law Corpus: A dataset for legal AI research\.In*Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track*, 2023\.URL[https://arxiv\.org/abs/2309\.12269](https://arxiv.org/abs/2309.12269)\.
- Pezeshkpour and Hruschka \[2024\]Pouya Pezeshkpour and Estevam Hruschka\.Large language models sensitivity to the order of options in multiple\-choice questions\.In*Findings of the Association for Computational Linguistics: NAACL 2024*, pages 2006–2017, 2024\.URL[https://aclanthology\.org/2024\.findings\-naacl\.130/](https://aclanthology.org/2024.findings-naacl.130/)\.
- Poliak et al\. \[2018\]Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme\.Hypothesis only baselines in natural language inference\.In*Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics \(\*SEM\)*, pages 180–191, 2018\.
- Watson et al\. \[2026\]Joe Watson, Joana Ribeiro de Faria, Marcus Tomalin, Mns Magnusson, Huiyuan Xie, Hao Tian Yeung, Christine Carter, Jonathan Rutherford, and Felix Steffek\.Shortcut learning in legal judgment prediction: Empirical evidence from the UK employment tribunal\.*arXiv preprint arXiv:2607\.04261*, 2026\.URL[https://arxiv\.org/abs/2607\.04261](https://arxiv.org/abs/2607.04261)\.
- Zheng et al\. \[2024\]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang\.Large language models are not robust multiple choice selectors\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.URL[https://openreview\.net/forum?id=shr9PXz7T0](https://openreview.net/forum?id=shr9PXz7T0)\.