字面锁定：测试语言如何改变类型化决策模型对孟加拉语习语的字面吸引

# How Test Language Shifts Literal Attractionin Typed Decision Models on Bengali Idioms
Source: [https://arxiv.org/html/2610.22526](https://arxiv.org/html/2610.22526)
\[ Extension = \.otf, UprightFont = \*\-regular, BoldFont = \*\-bold, ItalicFont = \*\-italic, BoldItalicFont = \*\-bolditalic \] \[ Script = Bengali, Language = Bengali \]

###### Abstract

Decision models, non\-generative scorers that report a probability over caller\-supplied options and emit no text, are increasingly deployed as routers, classifiers and guardrails\. Because they generate nothing, rationales and chain\-of\-thought do not apply\. We audit three: an encoder scorer, a decoder scorer and a closed API, entirely through interventions on their input, using expert\-annotated Bengali idioms carrying a gold figurative and a gold literal gloss\. Dropping an idiom’s own literal gloss into the option list draws every arm toward it, taking65%65\\%to96%96\\%of each arm’s remaining Bengali errors\. A model×\\timeslanguage matrix over one shared item set shows how far this depends on the test language: every arm is1\.91\.9to6\.46\.4times more captured in Bengali than in English on the same paired entries, the commercial model’s apparent resistance is a property of its test language, and capability does not order the failure\. Calibration degrades in all three arms, and on the open arms confidence inverts, so gating on it is worse than gating at random\. On a semantics\-free option list, where accuracy is at chance, the best\-calibrated arm still reports mean confidence0\.820\.82\. Code and data are available at[https://github\.com/sakhadib/literal\-lock\-in](https://github.com/sakhadib/literal-lock-in)\.

††∗\*Equal contribution\.![Refer to caption](https://arxiv.org/html/2610.22526v1/fig_bn_teaser_2.png)Figure 1:The model takes the words\.Laya\-ml,Lod\-lille andJev\-1\.13 keep these three markers throughout the paper\. The Bengali idiom\\bnঅগ্নিবল \(*agnibal*\) is built from\\bnঅগ্নি \(*fire*\) and\\bnবল \(*strength*\) and means intense hunger\. With both readings in the option list, all three models select the literal gloss in Bengali, atp=1\.00p=1\.00,0\.970\.97and0\.360\.36\. Asked as the paired English entry, with the English glosses of the same four options, Lod\-lille and Jev\-1\.13 recover and Laya\-ml does not\. Across the3,2653\{,\}265paired items literal attraction falls from0\.7310\.731,0\.9180\.918and0\.3040\.304in Bengali to0\.3790\.379,0\.2410\.241and0\.0470\.047in English \(§[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)\)\. Every item in that pool carries an English parenthetical inside its Bengali literal option by construction, this one included; §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6)measures what that cue is worth and shows capture without it\.## 1Introduction

A*decision model*is given a question and a list of candidate options and returns a probability distribution over exactly those options\. It does not sample and it produces no text\. We call the interface*typed*because the caller declares the answer type and the option set in the request, and the response is guaranteed to be a distribution over exactly those options: the output cannot be malformed, only wrong\. This class has grown quickly as infrastructure for routing, intent classification, content gating and tool selection, precisely because it is cheap, bounded and returns a number a caller can threshold on\. An ecosystem analysis of 2,170 public projects finds it already embedded as a reusable decision component across domains\([Ling et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib47)\), and proposals now place these models at the adjudication points of autonomous security harnesses\([dos Santos Barbosa, 2026](https://arxiv.org/html/2610.22526#bib.bib48)\)\. The threshold is load\-bearing\.

That bounded interface is also an explainability problem\. There is no rationale to inspect, no chain of thought to measure for faithfulness\([Jacovi and Goldberg, 2020](https://arxiv.org/html/2610.22526#bib.bib29);[Turpin et al\., 2023](https://arxiv.org/html/2610.22526#bib.bib30)\), and in the commercial case no weights at all\. What is left is the behavioral surface: the caller controls the question and the option list, and sees a probability vector\. Any audit of this class has to be built out of that\.

The obvious fallback, attributing the decision to input tokens\([Ribeiro et al\., 2016](https://arxiv.org/html/2610.22526#bib.bib31);[Lundberg and Lee, 2017](https://arxiv.org/html/2610.22526#bib.bib32)\), is both unreliable here\([Jain and Wallace, 2019](https://arxiv.org/html/2610.22526#bib.bib33);[Adebayo et al\., 2018](https://arxiv.org/html/2610.22526#bib.bib34);[Hooker et al\., 2019](https://arxiv.org/html/2610.22526#bib.bib35)\)and hard to read\. How far an idiom’s parts map onto components of its figurative meaning varies by idiom and is not annotated in the corpus we use\([Nunberg et al\., 1994](https://arxiv.org/html/2610.22526#bib.bib7);[Gibbs et al\., 1989](https://arxiv.org/html/2610.22526#bib.bib6)\); for an expression at the non\-decomposable end, a Shapley value over its words is computable but has no evident linguistic referent\. We would rather intervene on the expression as a unit than report shares we cannot interpret \(Appendix[D](https://arxiv.org/html/2610.22526#A4)\)\.

We take the option set as the main intervention surface instead\. Bengali*bāgdhārā*\(idioms\) are well suited to this because the expert resource we use ships, for each idiom, both a gold figurative meaning and a gold*literal*gloss: a ready\-made, human\-certified counterfactual distractor that no annotation effort of ours produced\. Dropping that gloss into the option list turns a benchmark question into a targeted probe: does the model read the expression as an idiom, or as its words? Figure[1](https://arxiv.org/html/2610.22526#S0.F1)shows one item answered both ways\.

We contribute \(i\) a meaning\-choice instrument for decision models in which the*distractor family*is the independent variable and the item, instruction andkkare held fixed, requiring no new annotation \(§[4](https://arxiv.org/html/2610.22526#S4)\); \(ii\) the Literal Attraction Rate, measured as a full model×\\timeslanguage matrix, which shows how far capture depends on the test language and that capability does not order it \(§[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)\); \(iii\) evidence that calibration degrades in all three arms, and that on the two open arms confidence becomes anti\-diagnostic, with confidence\-gated selective prediction worse than random gating \(§[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)\); \(iv\) a large dissociation in option\-order dependence between the two scoring designs, with a methodological consequence for anyone evaluating this class \(§[5\.4](https://arxiv.org/html/2610.22526#S5.SS4)\); and \(v\) a semantics\-free control, run on all three arms and needing no annotation of any kind, bounding what each model reports when the options carry no information at all \(§[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)\)\.

## 2Related Work

#### The decision\-model class\.

The evaluation literature is still forming, and reports quality somewhat below large models at far lower cost, with no independent accuracy advantage from the typed readout itself\([Tang and Zheng, 2026](https://arxiv.org/html/2610.22526#bib.bib50);[Rao and Callison\-Burch, 2026](https://arxiv.org/html/2610.22526#bib.bib42);[Ibrahim and Zaki, 2026](https://arxiv.org/html/2610.22526#bib.bib41);[Guo et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib44);[Yu and Yao, 2026](https://arxiv.org/html/2610.22526#bib.bib46)\)\. Two strands bear on ours\.[Ibrahim and Zaki \(2026\)](https://arxiv.org/html/2610.22526#bib.bib41)report decision\-model*calibration exceeding most LLM baselines*, and[Li et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib43)build the confidence\-gated cascade that follows from it; both depend on the returned confidence being diagnostic, and §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)–§[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)give conditions under which it is not\. Concurrently[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)find the decision follows an option’s*name*rather than its bound definition, and[Xu \(2026\)](https://arxiv.org/html/2610.22526#bib.bib49)flip 61\.4% of correct decisions with short, fluent context additions chosen to redirect them\. Appendix[B](https://arxiv.org/html/2610.22526#A2)sets out how each relates to our design\.

#### Idioms in NLP\.

Idiom resources and tasks are well established\([Haagsma et al\., 2020](https://arxiv.org/html/2610.22526#bib.bib1);[Tayyar Madabushi et al\., 2022](https://arxiv.org/html/2610.22526#bib.bib3);[Chakrabarty et al\., 2022](https://arxiv.org/html/2610.22526#bib.bib2)\), including for Bengali\([Sakhawat et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib51)\), and the difficulty of figurative language for distributional models is long\-noted\([Shwartz and Dagan, 2019](https://arxiv.org/html/2610.22526#bib.bib4)\)\. Psycholinguistics frames the phenomenon as competition between a literal and a figurative route whose resolution depends on decomposability and context\([Cacciari and Tabossi, 1988](https://arxiv.org/html/2610.22526#bib.bib5);[Gibbs et al\., 1989](https://arxiv.org/html/2610.22526#bib.bib6);[Titone and Connine, 1999](https://arxiv.org/html/2610.22526#bib.bib8)\), and[Oh et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib9)give mechanistic evidence that the competition is real inside a transformer: causal tracing recovers two pathways carrying the figurative and literal readings simultaneously, with early layers promoting the figurative one and a direct route keeping the literal one available\.

Ourd\-litintervention is the behavioral counterpart of that picture rather than a replication of it: they can ask*how*the literal route stays available inside a model they can open, we can only ask*how often*it wins in one that emits no text and, in one arm, exposes no weights \(Appendix[B](https://arxiv.org/html/2610.22526#A2)\)\.

#### Evaluating option scorers\.

Option\-order sensitivity and selection bias are documented for prompted generative LMs\([Pezeshkpour and Hruschka, 2024](https://arxiv.org/html/2610.22526#bib.bib11);[Wei et al\., 2024](https://arxiv.org/html/2610.22526#bib.bib12);[Zheng et al\., 2024](https://arxiv.org/html/2610.22526#bib.bib10);[Zhao et al\., 2021](https://arxiv.org/html/2610.22526#bib.bib13)\)\.[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)relay an independent evaluation harness reporting option\-order sensitivity in Laya; that report is relayed rather than theirs, and we could not determine from it which checkpoint it covers, so we do not read it as establishing the behavior of the multilingual checkpoint we audit\. What §[5\.4](https://arxiv.org/html/2610.22526#S5.SS4)adds is the comparison of option\-marker against independent\-option scoring over the full permutation set, and the caveat that permuting a list also relabels it, which that literature shows is itself a source of bias\([Zheng et al\., 2024](https://arxiv.org/html/2610.22526#bib.bib10)\)\.

#### Calibration and selective prediction\.

Proper scoring and calibration measurement are standard\([Brier, 1950](https://arxiv.org/html/2610.22526#bib.bib15);[Gneiting and Raftery, 2007](https://arxiv.org/html/2610.22526#bib.bib16);[Guo et al\., 2017](https://arxiv.org/html/2610.22526#bib.bib17);[Nixon et al\., 2019](https://arxiv.org/html/2610.22526#bib.bib18);[Kumar et al\., 2019](https://arxiv.org/html/2610.22526#bib.bib19)\), as is the observation that calibration degrades under distribution shift\([Ovadia et al\., 2019](https://arxiv.org/html/2610.22526#bib.bib22);[Jiang et al\., 2021](https://arxiv.org/html/2610.22526#bib.bib23)\)\. Selective prediction gives the operational test\([Geifman and El\-Yaniv, 2019a](https://arxiv.org/html/2610.22526#bib.bib20);[Geifman and El\-Yaniv, 2019b](https://arxiv.org/html/2610.22526#bib.bib21);[Traub et al\., 2024](https://arxiv.org/html/2610.22526#bib.bib24)\)\. We report the combination that matters for deployment: accuracy*decreasing*with confidence, and a confidence gate underperforming a random one\.

#### Behavioral testing\.

Our design follows behavioral checklists, contrast sets and counterfactual minimal pairs\([Ribeiro et al\., 2020](https://arxiv.org/html/2610.22526#bib.bib25);[Gardner et al\., 2020](https://arxiv.org/html/2610.22526#bib.bib26);[Kaushik et al\., 2020](https://arxiv.org/html/2610.22526#bib.bib27);[Naik et al\., 2018](https://arxiv.org/html/2610.22526#bib.bib28);[Jia and Liang, 2017](https://arxiv.org/html/2610.22526#bib.bib14)\); the contribution is carrying that tradition into a model class with no text output, where it is one of the few instruments left\.

#### Low\-resource evaluation\.

Bengali has strong encoders and corpora\([Bhattacharjee et al\., 2022](https://arxiv.org/html/2610.22526#bib.bib36);[Kakwani et al\., 2020](https://arxiv.org/html/2610.22526#bib.bib37)\), and cross\-lingual benchmarks establish the transfer gap\([Hu et al\., 2020](https://arxiv.org/html/2610.22526#bib.bib38);[Ruder et al\., 2021](https://arxiv.org/html/2610.22526#bib.bib39);[Adelani et al\., 2021](https://arxiv.org/html/2610.22526#bib.bib40)\); §[5\.7](https://arxiv.org/html/2610.22526#S5.SS7)localizes our failure to the Bengali side rather than to figurativeness\.

## 3Dataset

#### Source corpus\.

We use*Bangla Bagdhara*\([Sakhawat et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib51)\), a corpus of 10,361 Bengali idioms annotated under a 19\-field schema by expert consensus\.111The published count is 10,361 and the distributed record file holds 10,359; the two missing rows are lost in the corpus’s own export rather than in our loader, which parses all 10,359 without error, so we cannot recover them\. Of the 10,359, 8,876 carry every field the instrument requires\.Each row carries the idiom, a gold figurative meaning in Bengali, an expert literal gloss, semantic tags, alternative surface forms certified as meaning\-preserving, and a gold carrier sentence\. We add no annotation; every intervention below is assembled from fields that already exist\.

Its own benchmark on the same underlying task reports no model among 30 multilingual and instruction\-tuned LLMs above 50% accuracy, against 83\.4% for human annotators\. Option construction differs, so those figures are not directly comparable to ours, but they locate the difficulty in the material rather than in our instrument\.

## 4Methodology

Figure 2:The instrument in one view\.The audit proceeds from theBangla Bagdharacorpus through six stages:\(1\) corpus, which supplies 10,361 expert\-annotated idioms with gold figurative and literal glosses;\(2\) distractor family, the variable E1 manipulates, withd\-litandd\-hardinjecting the expert literal gloss;\(3\) item prompt, which fixes the typed question,k=4k=4options, instructions, and per\-item order seed;\(4\) models, comprising Laya\-ml, Lod\-lille, and Jev\-1\.13;\(5\) scoring, which records the four\-option probability vector,arg⁡max\\arg\\max, and confidence without requiring generated text; and\(6\) metrics, which evaluate accuracy, LAR, ECE, AURC, agreement, and order stability\. The figure’s visual encoding distinguishes the varied distractor family, literal\-bearing conditions, models under test, and measured outcomes\.Figure[2](https://arxiv.org/html/2610.22526#S4.F2)sets out the instrument end to end: what the corpus supplies, what each condition changes, and what comes back\.

### 4\.1The meaning\-choice task

Given an idiom, choose its gold figurative meaning fromk=4k=4candidates\. Chance is0\.2500\.250throughout\. Thedistractor family is the independent variable; the item, the instruction,kk, and per\-item option\-order randomization \(seeded by item id and experiment\) are held fixed across families, so a paired test across families compares one item under different option lists\.

### 4\.2Distractor families and measures

The five families ared\-rand\(other idioms’ glosses at random\),d\-surf\(glosses sharing surface tokens with the gold\),d\-sem\(glosses from idioms sharing semantic tags\),d\-lit\(one distractor replaced by the idiom’s*own*expert literal gloss\) andd\-hard\(all three pressures at once\)\. Appendix[C](https://arxiv.org/html/2610.22526#A3)gives each sampling rule, and Table[6](https://arxiv.org/html/2610.22526#A2.T6)one complete item in the original script\.

For the families containing a literal gloss, withℓi\\ell\_\{i\}the literal option andgig\_\{i\}the gold, the Literal Attraction Rate is

lar=1N∑i=1N\[argmaxopi\(o\)=ℓi\]\.\\textsc\{lar\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\!\\left\[\\arg\\max\_\{o\}p\_\{i\}\(o\)=\\ell\_\{i\}\\right\]\.laris not1−accuracy1\-\\text\{accuracy\}: it isolates the errors that are specifically a literal reading, and has no floor, since a model that never reads literally scores00while still being wrong\.

### 4\.3Interventions and statistics

Seven interventions vary one thing each: the distractor family \(E1\), the idiom’s surface form \(E2\), the idiom as a unit \(E3\), the task language \(E4\), nothing \(E5, derived from E1\), the option text \(E6\) and the option order \(E7\)\. Appendices[C](https://arxiv.org/html/2610.22526#A3)and[A](https://arxiv.org/html/2610.22526#A1)give what each holds fixed and the typed question they share\.

All intervals are 95% percentile bootstrap over items \(2,000 resamples\); paired comparisons use McNemar’s exact test; ECE uses 15 equal\-width bins and AURC is reported against a random\-gate baseline\. The two open arms were built independently from a written contract, and on all 4,440 overlapping rows the gold key, gold position, idiom string and literal\-distractor key agree at1\.0001\.000, so every paired test is on identical questions \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\)\.

### 4\.4Models

Three models span the design space \(Table[4](https://arxiv.org/html/2610.22526#A1.T4)\); none has an accompanying paper, so we identify them by their released artifacts\.222Laya\-ml:convaiinnovations/laya, subfoldermultilingual, on the Hugging Face Hub; the same repository ships the English\-only checkpoint used as the control in §[5\.7](https://arxiv.org/html/2610.22526#S5.SS7)\.Lod\-lille:mrn\-dk/lod\-lille\-0\.6B, same hub\.Jev\-1\.13:typesafe/jev\-1\.13, served over OpenRouter’s/api/alpha/decisionsendpoint, with no weights, training data or architecture published\.Laya\-mlis a bidirectional encoder scoring all options together from marker positions\.Lod\-lilleis a decoder with independent\-option scoring, each option attending only to the shared state, the question and itself from the same start position, which its model card reports as order\-independent by construction infp32; §[5\.4](https://arxiv.org/html/2610.22526#S5.SS4)finds that reordering and relabeling the list together still move its scores\.Jev\-1\.13is a closed commercial model behind a decisions API whose documentation names English as its primary and best\-supported language; we know its interface and nothing else\. Appendix[A](https://arxiv.org/html/2610.22526#A1)gives the call each one receives\.

## 5Analysis

### 5\.1Literal capture, and what it tracks

Table[1](https://arxiv.org/html/2610.22526#S5.T1)gives the ladder for the two open arms on the 888 items they share; Figure[3](https://arxiv.org/html/2610.22526#S5.F3)plots Laya\-ml’s full set alongside \(Appendices[E](https://arxiv.org/html/2610.22526#A5),[G](https://arxiv.org/html/2610.22526#A7)\)\. Lod\-lille is the stronger model on every non\-literal condition, by up to\+0\.133\+0\.133\. Insert the idiom’s own literal gloss and the ordering*reverses*: Lod\-lille falls to0\.0660\.066, half of Laya\-ml’s already\-below\-chance0\.1360\.136, and the same holds ford\-hard\.

Figure 3:The distractor ladder \(left\) and literal attraction rate \(right\) for the two open arms\.Laya\-ml \(n=8,876n=8\{,\}876\),Lod\-lille \(n=888n=888\); the black dashed line marks the four\-option chance level \(0\.2500\.250\) in both panels; error bars are 95% bootstrap CIs\. Surface\- and tag\-matched distractors cost a few points; the idiom’s own literal gloss \(d\-lit\) collapses both models far below chance, and the*stronger*model harder\.DistractorsLayaLod𝚫\\boldsymbol\{\\Delta\}random0\.3510\.484\+0\.133∗∗∗\+0\.133^\{\*\*\*\}surface0\.3390\.412\+0\.073∗∗∗\+0\.073^\{\*\*\*\}tag0\.3250\.446\+0\.120∗∗∗\+0\.120^\{\*\*\*\}\+\+literal0\.1360\.066−0\.070∗∗∗\-0\.070^\{\*\*\*\}hard0\.1490\.063−0\.086∗∗∗\-0\.086^\{\*\*\*\}lar\(lit\.\)0\.6580\.885lar\(hard\)0\.6150\.855meanpp\(lit\.\)0\.4600\.726meanpp\(gold\)0\.1930\.127Table 1:The distractor ladder and literal attraction\.Every cell is the 888 items the two arms share, so the printed values andΔ\\Deltahave one base \(Δ\\Deltais computed before rounding\); Laya\-ml’s full\-set figures are in Appendix[E](https://arxiv.org/html/2610.22526#A5)and differ by at most0\.0160\.016\(Appendix[O](https://arxiv.org/html/2610.22526#A15)\)\. McNemar is paired on those items \(∗∗∗p<0\.001p<0\.001\)\.Laya\-ml,Lod\-lille; the darker fill marks the higher value in a row\. The sign ofΔ\\Deltareverses at the literal boundary: the stronger arm leads on the first three families and trails on the two containing a literal gloss\.Within this pair, then,larmeasures something accuracy does not: a model can be selected as “better” on a conventional benchmark while being worse at what that benchmark proxies for\. Laya\-ml and Lod\-lille answer byte\-identical questions in the same language, so the crossover between them is a paired result\. The error is not diffuse: the decomposition underd\-litis gold / literal / other=0\.136=0\.136/0\.6580\.658/0\.2060\.206for Laya\-ml and0\.0660\.066/0\.8850\.885/0\.0480\.048for Lod\-lille, both on those 888 items, and on the clearest of them Laya\-ml placesp=1\.000p=1\.000on the literal gloss\. Laya\-ml’s full\-set decomposition,0\.1510\.151/0\.6440\.644/0\.2050\.205, is in Appendix[E](https://arxiv.org/html/2610.22526#A5)\.

#### Capture tracks the language, not capability\.

The arms above ran the literal condition in Bengali, while the commercial arm’s headlinelar=0\.047\\textsc\{lar\}=0\.047is*English*, so the two are not comparable as they stand\. We therefore ran the full model×\\timeslanguage matrix on the 3,265\-item shared pool, every cell built by one function from one seed \(Table[14](https://arxiv.org/html/2610.22526#A9.T14), Figure[5](https://arxiv.org/html/2610.22526#A9.F5)\); Jev\-1\.13’s English cell lands at0\.04720\.0472\[0\.0401,0\.0545\]\[0\.0401,0\.0545\], matching its value in the English ladder of Appendix[E](https://arxiv.org/html/2610.22526#A5)\.

Two things follow\. First,every arm is far more captured in Bengali than in English on the same paired entries:0\.731→0\.3790\.731\\rightarrow 0\.379,0\.918→0\.2410\.918\\rightarrow 0\.241and0\.304→0\.0470\.304\\rightarrow 0\.047, all paired McNemarp<10−100p<10^\{\-100\}\. Second,capture is not monotone in capability: Lod\-lille, the stronger open arm on random distractors, is the more captured in Bengali, and Jev\-1\.13 the least\. The open arms fall below chance here; Jev\-1\.13 does not, but loses0\.3900\.390of accuracy against its own English run\. The pool is not neutral and its Bengali literal option carries a cue the English one lacks; on ladder items without it, Bengali capture is still0\.5810\.581and0\.8710\.871\(Appendices[I](https://arxiv.org/html/2610.22526#A9),[J](https://arxiv.org/html/2610.22526#A10)\)\.

#### The capability ordering\.

One scored condition is matched across all three arms, Bengali idiom with random distractors, and on it the commercial model is far ahead,0\.7360\.736against0\.3670\.367and0\.4840\.484\(Appendix[L](https://arxiv.org/html/2610.22526#A12)\)\. That is the ordering the calibration results below should be read against, and the one capture does not follow\.

### 5\.2Confidence is anti\-diagnostic

Table[2](https://arxiv.org/html/2610.22526#S5.T2)gives the mean probability on the chosen option, split by outcome, on the literal\-bearing families\. Both open models are*more confident when captured than when correct*, Lod\-lille extremely so \(0\.7870\.787against0\.4530\.453\), and their reliability curves slope*downward*underd\-lit\(Figure[4](https://arxiv.org/html/2610.22526#A5.F4), Appendix[F](https://arxiv.org/html/2610.22526#A6)\)\.

Table 2:Confidence by outcome, pooled over the two literal\-bearing familiesd\-litandd\-hard, with calibration below\. Bold marks, per model, whether it is more confident when correct or when captured\. Each column is on that arm’s own items, so compare pattern rather than level\. Both open arms are more confident when captured, and their confidence\-gated AURC is*worse*than the random\-gate baseline on the row below it, which inverts selective prediction\. Both AURC rows are thed\-litcondition, and the random\-gate value is the expected AURC of a confidence signal drawn at random on the same items\.ECE depends on the binning\([Nixon et al\., 2019](https://arxiv.org/html/2610.22526#bib.bib18)\), so we also report the binning\-free multiclass Brier score:1\.0301\.030and1\.3911\.391for the open arms underd\-litagainst0\.7500\.750for a uniform four\-way distribution, so their reported probabilities are worse than declining to choose\. Equal\-mass bins move no ECE by more than0\.0110\.011\(Appendix[F](https://arxiv.org/html/2610.22526#A6)\)\.

Selective prediction therefore inverts on both open arms: restricting Laya\-ml’sd\-litto its most confident 5% gives0\.0070\.007accuracy against0\.1510\.151at full coverage, and Lod\-lille answers every item in its most confident 35% incorrectly \(Appendix[F](https://arxiv.org/html/2610.22526#A6)\)\. It is not an artifact of one confidence signal: Lod\-lille’s*separate confidence head*, which estimates whether the answer is in the option list at all, shows the same inversion \(0\.6310\.631against0\.2950\.295over both literal\-bearing families\), and it survives AUGRC\([Traub et al\., 2024](https://arxiv.org/html/2610.22526#bib.bib24)\), which matters here because AURC aggregates risk conditioned on the accepted set and so over\-weights failures made at high confidence, which is exactly where our inversion lives\. For a random confidence signal AUGRC is half the error rate, so the baseline needs no simulation: underd\-litthe open arms score0\.4480\.448and0\.4910\.491against0\.4250\.425and0\.4670\.467, worse than random on that measure as on AURC\.

Calibration degrades in all three arms; only the inversion is two\-arm\. Jev\-1\.13 looks well calibrated in English \(ECE0\.0140\.014, Brier0\.1070\.107\) and is not on the same items in Bengali \(ECE0\.1350\.135, Brier0\.6040\.604\), where its advantage over a random gate falls from5\.1×5\.1\\timesto1\.4×1\.4\\times\. It is not inverted, though, staying more confident when correct \(0\.7240\.724\) than when captured \(0\.6440\.644\), so the sign depends on the arm\. This does not contradict[Ibrahim and Zaki \(2026\)](https://arxiv.org/html/2610.22526#bib.bib41): in\-distribution calibration need not survive a targeted distractor shift, and it fails in the costly direction, since a confidence\-gated cascade\([Li et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib43)\)accepts without review exactly the tail where the open arms are*least*accurate \(Appendix[B](https://arxiv.org/html/2610.22526#A2)\)\.

### 5\.3Semantics\-free control, on all three arms

Theo4ablation replaces every option’s text with a bare ordinal label,\\bnঅর্থ1–4for the Bengali arms andmeaning1–4for the commercial one, so the four options are distinguishable but carry no meaning\. With the gold position randomized and no option carrying information, expected accuracy is0\.250\.25and whatever confidence survives above it is overconfidence, with no modeling assumption needed to read it\. It needs no gloss and no literal annotation, so it is the one condition we can put to all three arms without the pool restriction; the commercial arm’s run is in English, so the labels differ in language from the other two while carrying the same absence of meaning \(Figure[7](https://arxiv.org/html/2610.22526#A11.F7)\)\.

The ordering reverses against Table[2](https://arxiv.org/html/2610.22526#S5.T2)\. The best\-calibrated model under real options is the worst when the options carry nothing: Jev\-1\.13 reports0\.8150\.815mean confidence at0\.2540\.254accuracy, an overconfidence of0\.5610\.561, which we give as a gap rather than a binned ECE because the gap is a lower bound on ECE and needs no binning choice to defend\. Lod\-lille is the only one whose confidence falls close to the0\.250\.25a calibrated model would report \(0\.2820\.282\)\.

Jev\-1\.13 exposes a second field meant for exactly this case, a confidence that the answer is present in the option list at all, and it does not rescue the reading: ono4it reports0\.7540\.754against0\.9460\.946with full glosses\. The reference is not the0\.250\.25chance level of the top probability, since this field reports presence, so on a list where the answer provably is not present an informed value is near zero\. The pattern matches[Zhong et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib52)on a different task, with a scope condition that cuts both ways and that Appendix[B](https://arxiv.org/html/2610.22526#A2)sets out: a*verification*\-typed call recovers what the choice type does not, and we have not tested whether that holds here\. What the control shows is what a caller of this answer type is told, which is why we read it as the most portable finding: it needs no annotation and no matched language\. McNemar between the two open arms undero4is not significant \(p=0\.76p=0\.76\), so the instrument’s null behaves alike in both\.

### 5\.4Option order is architecture\-specific

We run every item under all4\!=244\!=24orderings of a fixed option set and record the chosen gloss’s identity, recovered asperm⁡\[slot⁡\(y^\)\]\\mathrm\{perm\}\[\\mathrm\{slot\}\(\\hat\{y\}\)\]; Appendix[Q](https://arxiv.org/html/2610.22526#A17)explains why the slot letter is the wrong unit for the*answer*\.

One caveat applies to everything here\. Permuting the list does two things at once: it moves each gloss and it changes which letter names it\. They cannot be separated in this design, and the literature gives reason to think the second matters, since[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)rebind option names to fixed definitions and find the decision follows the name, while[Zheng et al\. \(2024\)](https://arxiv.org/html/2610.22526#bib.bib10)trace most selection bias in multiple choice to token\-level priors on the option IDs\. What we measure is therefore sensitivity to display order and labeling*together*, which is what a caller varies when they reorder a list, not a clean effect of position\. Separating them needs a condition that moves each gloss while it keeps its own letter, which we did not run\.

Table[21](https://arxiv.org/html/2610.22526#A17.T21)gives six measures of that sensitivity, each on an item sample drawn separately per arm\. The marker\-based scorer changes its chosen gloss on83\.7%83\.7\\%of items against10\.7%10\.7\\%for the independent\-option scorer and13\.3%13\.3\\%for the commercial one, a7\.8×7\.8\\timesdifference in the rate of answer changes and a12×12\\timesdifference in the within\-item range ofpp\(gold\)\. Each scoring design is represented by exactly one model, so the design cannot be separated from the backbone or the capability of the model carrying it; we read the pattern as consistent with the scoring architecture rather than as demonstrated to follow from it\.

#### The invariance claim does not survive as stated\.

Lod\-lille is documented as order\-independent by construction, each option being scored from the same start position so that no option sees another\. Mapping every slot’s probability back to the option it displayed, a single option’s own score travels4\.0×10−24\.0\\times 10^\{\-2\}at the median item and never stays below10−410^\{\-4\}, orders of magnitude abovefp32noise \(Appendix[Q](https://arxiv.org/html/2610.22526#A17)\)\. Given the caveat above we state the conclusion narrowly: a caller who permutes an option list does not get identical probabilities back\. Whether the residue is positional or comes from the letters being reassigned we cannot say, and the second would be no less a failure of the guarantee than the first\.

A slot preference follows, measured over all five families of the full ladder, where the gold position is randomized: Laya\-ml takes slot B37\.2%37\.2\\%of the time against14\.6%14\.6\\%for slot A \(Appendix[Q](https://arxiv.org/html/2610.22526#A17)\)\. Here too slot and letter covary, so it is as consistent with a prior on the tokenBas with a preference for the second position\. Either way, an evaluation of this class that fixes option order is partly measuring something other than the options\. All results here randomize order per item, seeded by item id and experiment, and we would treat that as a default rather than a robustness check\.

### 5\.5The option side is read lexically, and the idiom is not held as a unit

Two interventions probe how each side of the item is read, and both carry scope conditions we state with the result\.

On the option side, scrambling the words inside every option costs almost nothing: writing*signal*for accuracy above chance as a fraction of the full\-gloss condition’s, the three arms retain0\.9280\.928,0\.9440\.944and1\.0051\.005of it\. The companion truncation is weaker than it looks, since the median Bengali gloss is four words and truncation to six is a no\-op on77\.8%77\.8\\%of options\. The scramble is the result, and it converges with[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)from the other direction; the scope condition is that the series runs on random distractors, where recognizing the right words already suffices, so it shows composition is not*required*here rather than that these models cannot compose \(Appendix[P](https://arxiv.org/html/2610.22526#A16)\)\.

On the idiom side, E3 perturbs the expression while holding the option list fixed\. It covers only idioms of three words or more, so every manipulation changes the string on99\.7%99\.7\\%of items or better\. Accuracy cannot carry the argument, because the condition isd\-litand the intact accuracies on E3’s own items,0\.1570\.157and0\.0720\.072, already sit below chance\. Agreement with the intact answer has no such floor, and that is where the claim rests: under word\-shuffling the open arms return the same option on0\.7420\.742and0\.9170\.917of items, at least as often as under meaning\-preserving rewrites \(0\.6570\.657,0\.6770\.677; Appendix[N](https://arxiv.org/html/2610.22526#A14)\)\. The one manipulation that moves both is the gold carrier sentence,\+0\.080\+0\.080and\+0\.117\+0\.117, so context rather than the idiom string carries the usable signal\. That cuts both ways, since[Xu \(2026\)](https://arxiv.org/html/2610.22526#bib.bib49)flip 61\.4% of correct decisions with short fluent additions chosen to redirect them \(Appendix[M](https://arxiv.org/html/2610.22526#A13)\)\.

### 5\.6What the item construction could explain

The literal gloss is an annotated field rather than a string we wrote, and in this corpus it is often lexically close to the idiom\. Three properties could let a lexical matcher select it without reading anything\. The idiom is a single word on42\.3%42\.3\\%of items, and on18\.0%18\.0\\%the literal gloss contains the idiom verbatim\. In the matrix pool it is the only option written partly in Latin script, since no Bengali figurative gloss in the corpus contains a Latin character\. And it is longer than the gold gloss on67%67\\%of items, though longest of all four on only40\.2%40\.2\\%\.

Two of the three buy measurable capture \(Table[3](https://arxiv.org/html/2610.22526#S5.T3)\): verbatim overlap on both arms, and the Latin\-script cue on Laya\-ml, with Lod\-lille’s interval covering zero\. Length is the exception and needs separate handling, since the subset below does not remove it\. Splitting on whether the literal option is the longest of the four moveslarby−0\.034\-0\.034\[−0\.055,−0\.013\]\[\-0\.055,\-0\.013\]on Laya\-ml and\+0\.020\+0\.020\[−0\.024,\+0\.064\]\[\-0\.024,\+0\.064\]on Lod\-lille, so Laya\-ml is slightly*less*captured when it is: a real asymmetry in the material, not a route to capture\.

Restricting to the2,4802\{,\}480items that are multiword, whose gloss does not contain the idiom and carries no parenthetical, removes both lexical cues and narrows the length gap to36\.636\.6against32\.432\.4characters\.laris still0\.5600\.560and0\.8910\.891against a chance level of0\.2500\.250, and the length split stays null; Appendix[J](https://arxiv.org/html/2610.22526#A10.SS0.SSS0.Px5)has the rest, includinglarby idiom length\.

Table 3:Two lexical cues are real; neither carries the result\.Laya\-ml,Lod\-lille\. Each cue would let a lexical matcher pick the literal option without reading it\. Verbatim overlap moveslaron both arms and the Latin\-script cue on Laya\-ml only, its Lod\-lille interval covering zero\. The last block keeps items that are multiword, whose literal gloss does not contain the idiom and carries no English parenthetical, so neither lexical cue is available:larstays far above the0\.2500\.250chance level on both arms\. Length is not removed by that subset and is reported separately in §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6), where it does not predict capture\. The commercial arm cannot be split this way, since its only Bengali literal run is the matrix pool, where every item carries an English parenthetical by construction\.
### 5\.7Failure localizes to Bengali, in every arm

E4 asks the same question of an ordinary benchmark item rather than a literal\-bearing one: hold the model fixed, swap the task language, keep the distractors random\. Every arm moves by a large margin, Laya\-ml\+0\.246\+0\.246, Lod\-lille\+0\.380\+0\.380and Jev\-1\.13\+0\.230\+0\.230, each paired on that arm’s own shared items \(all McNemarp<0\.001p<0\.001\)\. Lod\-lille reaches0\.8640\.864in English against0\.4840\.484in Bengali, so the figurative concepts appear largely available to it and the Bengali route is where the loss occurs, consistent with its documented English\-only training evaluation\. Laya\-ml’s English\-only checkpoint on Bengali script sits at0\.2830\.283, barely above chance, against0\.3710\.371for the multilingual one on the same items, so the scorer does read Bengali where the checkpoint can\. Here too the English condition presents the annotator’s nearest English idiom rather than a translation \(Appendices[K](https://arxiv.org/html/2610.22526#A11),[J](https://arxiv.org/html/2610.22526#A10)\)\.

## 6Discussion

Three cautions follow for anyone deploying this class\. A returned probability is worth little without a measurement of what the model returns on a degenerate option list, a case that arrives from an upstream bug or an empty retrieval rather than from an adversary\. A benchmark score is not by itself evidence against literal capture, since on our open arms the two move in opposite directions\. And an evaluation that fixes option order is partly reporting position and labeling, which §[5\.4](https://arxiv.org/html/2610.22526#S5.SS4)shows cannot be separated here\. None of this rebuts the cascade designs, whose cost and latency advantages are established\([Tang and Zheng, 2026](https://arxiv.org/html/2610.22526#bib.bib50);[Rao and Callison\-Burch, 2026](https://arxiv.org/html/2610.22526#bib.bib42)\); it bounds them, since the property they rest on degrades first and silently\.

## 7Conclusion

Decision models are being placed at decision points, and the interface that makes them attractive is also what makes them hard to audit: no rationale, no chain of thought, and in the commercial case no weights\. We have argued that the caller’s input is an adequate substitute, since every result here comes from changing the option list or the expression it asks about and reading the probability vector back, which needs no new annotation and works unchanged on a closed, text\-free model\.

One substituted option is enough\. The idiom’s own expert literal gloss draws all three arms toward the literal reading, taking65%65\\%to96%96\\%of what each arm gets wrong in Bengali, and the model×\\timeslanguage matrix separates two explanations that are easy to conflate: capture falls by0\.260\.26to0\.680\.68in every arm when the paired entries are asked in English, so the commercial model’s apparent resistance belongs to its test language rather than to the model\. Capability does not order the failure: the middle arm by accuracy is the most captured in Bengali\. Nor is any of this visible to the caller: confidence inverts on the open arms under the literal shift, calibration degrades on all three, and on an option list stripped of semantics the arms still report between0\.280\.28and0\.820\.82\.

What the design cannot settle is set out in Limitations, and the cues that could have produced this result are measured in §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6)rather than assumed away\. We release the instrument and the per\-item outputs\.

## Limitations

All three models are evaluated zero\-shot, and this is the limitation we would weigh most heavily after the one below\. Decision models are trained on a particular set of decision tasks, and the open arms’ own documentation reports that base checkpoints are weak on tasks outside that set\. Some part of what we measure may therefore be out\-of\-distribution task transfer rather than literal capture specifically\. Theo4control bounds this from one side, since it shows both open arms at chance accuracy with no detectable difference between them, and thed\-randtod\-litcontrast is within\-model and within\-item, which removes any fixed task\-transfer offset\. Neither argument rules the confound out\. A single lightly fine\-tuned arm would settle it, and we have not run one\.

Laya\-ml’s run had its per\-bucket temperatures unset, so its ECE figures are raw and we do not describe the arms as representing distinct calibration philosophies: both open arms are effectively untempered and comparable on that axis, and Jev\-1\.13’s internal calibration is unknown to us\. Laya\-ml also ran withbf16requested on a Turing card, which has nobf16tensor\-core path, so the arithmetic falls back tofp32CUDA cores \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\); a reader reproducing on later hardware is not reproducing the same numerics\.

Outside the capture matrix, which it ran on the full3,2653\{,\}265\-item pool, Lod\-lille ran a fixed 10% subset \(888 items\), so its intervals there are roughly±3\\pm 3points and its stratified cells are thin\. Appendix[O](https://arxiv.org/html/2610.22526#A15)bounds the subset bias at0\.0160\.016and shows it changes no sign or ordering\. E3 on the Lod\-lille arm is 180 items and several of its perturbation contrasts are not individually significant\. E7 covers 992 / 299 / 300 items rather than the full sets, drawn separately per arm, so its cross\-arm comparison is unpaired\.

Three of the interventions are weaker than their headings suggest, and we would rather say so than have it found\. E7 cannot separate position from labeling, because permuting a list reassigns the letters with it; the condition that would separate them moves each gloss while it keeps its own letter, and we did not run it\. E3 runs underd\-lit, where intact accuracy is already below chance, so its accuracy rows have little room to fall and the conclusion rests on the intact\-answer agreement instead; running it underd\-randwould let accuracy carry the argument too\. E6 runs underd\-rand, where matching the right words is already enough to answer, so it shows composition is not required rather than that it is unavailable; the scramble underd\-litord\-semis the test that would separate those, and it is unrun\. None of the three affects §[5\.1](https://arxiv.org/html/2610.22526#S5.SS1), §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)or §[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)\.

The matrix in §[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)runs on the 3,265 items that carry a usable English literal gloss, which is not a neutral sample: for Laya\-ml, in\-poollaris higher than out\-of\-pool whiled\-randaccuracy is unchanged, so the absolute rates there are higher than they would be corpus wide\. That difference is not generic pool selection\. Pool membership requires an English parenthetical inside the Bengali literal gloss, and Appendix[J](https://arxiv.org/html/2610.22526#A10)shows the parenthetical is itself worth\+0\.165\+0\.165oflaron Laya\-ml, which is most of the in\-pool elevation\. The language comparison is paired within item and within model, so the selection affects the levels rather than the direction\. The matrix also covers the literal condition only; we did not run the other distractor families in both languages\. The confidence results in §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)remain two\-arm, because the commercial arm never shows the inversion, not because it was unmeasured: §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)reports its Bengali calibration from the matrix outputs\.

The two language cells are also not symmetric in construction, and this is the limitation we would flag hardest\. They are paired entries rather than identical items: the Bengali state is the idiom, while the English state is the annotator’s nearest English idiom rather than a translation of it\. The two literal glosses are drawn from one annotated field in different ways, which leaves the Bengali literal option both lexically closer to its own state and, because it retains its English parenthetical, the only option in the list written partly in Latin script\. Appendix[J](https://arxiv.org/html/2610.22526#A10)sets out all four asymmetries we could find and tests the two that recorded outputs can reach\. Restricted to the1,4291\{,\}429items whose Bengali literal gloss shares no content word with the idiom, the capture gap is\+0\.358\+0\.358,\+0\.610\+0\.610and\+0\.310\+0\.310against headline gaps of\+0\.352\+0\.352,\+0\.677\+0\.677and\+0\.256\+0\.256\. Restricted to the ladder items whose literal gloss carries no English parenthetical at all,5,4855\{,\}485for Laya\-ml and560560for Lod\-lille, Bengali capture is still0\.5810\.581and0\.8710\.871, against a chance level of0\.2500\.250\. Neither cue accounts for the result, though the second is worth a real\+0\.165\+0\.165on one arm and we report it as such\. What we cannot do without new runs is make the two cells symmetric by construction; a translation\-paired corpus, or a Bengali cell with the parenthetical stripped, would settle what we can only bound\.

More broadly, this is one corpus, one language, two small open models and one commercial endpoint\. The semantics\-free control is the only result here that we would expect to transfer as stated; the rest should be read as findings about these systems on this material, and as an instrument other people can point at theirs\.

Distractors are drawn from the same corpus, so difficulty is corpus\-relative rather than absolute\. Jev\-1\.13 is a commercial endpoint and may change without notice; our results are a snapshot, and we report the exact call count and cost in Appendix[R](https://arxiv.org/html/2610.22526#A18)so the snapshot is at least auditable\.

## Ethics Statement

The corpus is a published, expert\-annotated resource of idioms\([Sakhawat et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib51)\); it contains no personal data and no content about identifiable individuals, and it is distributed under CC BY\-SA 4\.0\. Our interventions rewrite option lists and perturb idiom strings, and produce no new annotation and no generated text\.

The work audits two openly released models and one commercial API\. We report failure modes in all three, including the vendor’s, and we have taken care to separate what is measured from what is inferred\. Where a reading of our own data would have been more flattering than the data supports, we say so in the text: §[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)states the pool selection effect against our cleanest number, and Appendix[J](https://arxiv.org/html/2610.22526#A10)reports a construction cue that accounts for part of the headline effect on one arm\. Commercial endpoints are mutable, so the Jev\-1\.13 results should be read as a dated snapshot rather than a standing property of the product\.

Bengali is under\-served relative to its speaker population, and we think an audit that locates a failure in the language rather than in the task is the more useful kind to publish\. The instrument, the per\-item outputs and the analysis code are released under CC BY\-SA 4\.0, the corpus’s own license, and the two open checkpoints are used under the terms their model cards declare; Appendix[R](https://arxiv.org/html/2610.22526#A18)is the run record they correspond to\.

## References

- J\. Adebayo, J\. Gilmer, M\. Muelly, I\. Goodfellow, M\. Hardt, and B\. KimSanity checks for saliency maps\.InAdvances in Neural Information Processing Systems 31,External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1)\.
- Adelaniet al\.\(2021\)D\. I\. Adelani, J\. Abbott, G\. Neubig,et al\.MasakhaNER: named entity recognition for African languages\.Transactions of the Association for Computational Linguistics\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00416),[Link](https://aclanthology.org/2021.tacl-1.66/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px6.p1.1)\.
- Bhattacharjeeet al\.\(2022\)A\. Bhattacharjee, T\. Hasan, W\. Ahmad, K\. S\. Mubasshir, M\. S\. Islam, A\. Iqbal, M\. S\. Rahman, and R\. ShahriyarBanglaBERT: language model pretraining and benchmarks for low\-resource language understanding evaluation in Bangla\.InFindings of the Association for Computational Linguistics: NAACL 2022,Seattle, United States,pp\. 1318–1327\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-naacl.98),[Link](https://aclanthology.org/2022.findings-naacl.98/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px6.p1.1)\.
- Brier \(1950\)G\. W\. BrierVerification of forecasts expressed in terms of probability\.Monthly Weather Review\.External Links:[Document](https://dx.doi.org/10.1175/1520-0493%281950%29078%3C0001%3AVOFEIT%3E2.0.CO%3B2),[Link](https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Cacciari and Tabossi \(1988\)C\. Cacciari and P\. TabossiThe comprehension of idioms\.Journal of Memory and Language\.External Links:[Document](https://dx.doi.org/10.1016/0749-596X%2888%2990014-9),[Link](https://doi.org/10.1016/0749-596X(88)90014-9)Cited by:[Appendix A](https://arxiv.org/html/2610.22526#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Chakrabartyet al\.\(2022\)T\. Chakrabarty, A\. Saakyan, D\. Ghosh, and S\. MuresanFLUTE: figurative language understanding through textual explanations\.InProceedings of EMNLP 2022,External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.481),[Link](https://aclanthology.org/2022.emnlp-main.481/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- dos Santos Barbosa \(2026\)J\. A\. dos Santos BarbosaCalibrated decision models for autonomous penetration\-testing harnesses: JEV and Laya as System One decision layers for LLM\-driven pentest agents\.External Links:2609\.28940,[Link](https://arxiv.org/abs/2609.28940)Cited by:[§1](https://arxiv.org/html/2610.22526#S1.p1.1)\.
- Gardneret al\.\(2020\)M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant,et al\.Evaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Online,pp\. 1307–1323\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.117),[Link](https://aclanthology.org/2020.findings-emnlp.117/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px5.p1.1)\.
- Geifman and El\-Yaniv \(2019a\)Y\. Geifman and R\. El\-YanivSelectiveNet: a deep neural network with an integrated reject option\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 2151–2159\.External Links:[Link](https://proceedings.mlr.press/v97/geifman19a.html)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Geifman and El\-Yaniv \(2019b\)Y\. Geifman and R\. El\-YanivSelectiveNet: a deep neural network with an integrated reject option\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 2151–2159\.External Links:[Link](https://proceedings.mlr.press/v97/geifman19a.html)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Gibbset al\.\(1989\)R\. W\. Gibbs, N\. P\. Nayak, and C\. CuttingHow to kick the bucket and not decompose: analyzability and idiom processing\.Journal of Memory and Language\.External Links:[Document](https://dx.doi.org/10.1016/0749-596X%2889%2990014-4),[Link](https://doi.org/10.1016/0749-596X(89)90014-4)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p3.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Gneiting and Raftery \(2007\)T\. Gneiting and A\. E\. RafteryStrictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association\.External Links:[Document](https://dx.doi.org/10.1198/016214506000001437),[Link](https://doi.org/10.1198/016214506000001437)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of ICML 2017,External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Guoet al\.\(2026\)R\. Guo, Y\. Liu, G\. Deng, Y\. Li, L\. Zhao, Y\. Wu, S\. Chen, Y\. Zhang, and L\. Y\. ZhangJust ask Jev: reinforcement learning for calibrated decisions as a zero\-shot detector of AI alignment failures\.External Links:2609\.29429,[Link](https://arxiv.org/abs/2609.29429)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1)\.
- Haagsmaet al\.\(2020\)H\. Haagsma, J\. Bos, and M\. NissimMAGPIE: a large corpus of potentially idiomatic expressions\.InProceedings of the Twelfth Language Resources and Evaluation Conference,External Links:[Link](https://aclanthology.org/2020.lrec-1.35/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Hookeret al\.\(2019\)S\. Hooker, D\. Erhan, P\. Kindermans, and B\. KimA benchmark for interpretability methods in deep neural networks\.InAdvances in Neural Information Processing Systems 32,External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/fe4b8556000d0f0cae99daa5c5c5a410-Abstract.html)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1)\.
- Huet al\.\(2020\)J\. Hu, S\. Ruder, A\. Siddhant, G\. Neubig, O\. Firat, and M\. JohnsonXTREME: a massively multilingual multi\-task benchmark for evaluating cross\-lingual generalization\.InProceedings of ICML 2020,External Links:[Link](https://proceedings.mlr.press/v119/hu20b.html)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px6.p1.1)\.
- Ibrahim and Zaki \(2026\)H\. Ibrahim and Y\. ZakiEvaluating decision models for text annotation in computational social science\.External Links:2609\.24574,[Link](https://arxiv.org/abs/2609.24574)Cited by:[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px5.p1.1),[Appendix F](https://arxiv.org/html/2610.22526#A6.p2.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2610.22526#S5.SS2.p4.1)\.
- Jacovi and Goldberg \(2020\)A\. Jacovi and Y\. GoldbergTowards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4198–4205\.External Links:[Link](https://aclanthology.org/2020.acl-main.386/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.386)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p2.1)\.
- Jain and Wallace \(2019\)S\. Jain and B\. C\. WallaceAttention is not explanation\.InProceedings of NAACL 2019,External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1357),[Link](https://aclanthology.org/N19-1357/)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1)\.
- Jia and Liang \(2017\)R\. Jia and P\. LiangAdversarial examples for evaluating reading comprehension systems\.InProceedings of EMNLP 2017,External Links:[Document](https://dx.doi.org/10.18653/v1/D17-1215),[Link](https://aclanthology.org/D17-1215/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px5.p1.1)\.
- Jianget al\.\(2021\)Z\. Jiang, J\. Araki, H\. Ding, and G\. NeubigHow can we know when language models know? on the calibration of language models for question answering\.Transactions of the Association for Computational Linguistics9,pp\. 962–977\.External Links:[Link](https://aclanthology.org/2021.tacl-1.57/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00407)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Kakwaniet al\.\(2020\)D\. Kakwani, A\. Kunchukuttan, S\. Golla, G\. N\.C\., A\. Bhattacharyya, M\. M\. Khapra, and P\. KumarIndicNLPSuite: monolingual corpora, evaluation benchmarks and pre\-trained multilingual language models for Indian languages\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4948–4961\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.445/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.445)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px6.p1.1)\.
- Kaushiket al\.\(2020\)D\. Kaushik, E\. Hovy, and Z\. C\. LiptonLearning the difference that makes a difference with counterfactually\-augmented data\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Sklgs0NFvr)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px5.p1.1)\.
- Kumaret al\.\(2019\)A\. Kumar, P\. S\. Liang, and T\. MaVerified uncertainty calibration\.InAdvances in Neural Information Processing Systems 32,External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/f8c0c968632845cd133308b1a494967f-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2026\)Y\. Li, Y\. Miao, R\. Krishnan, and R\. PadmanJEV\-as\-a\-Judge: accept when confident, escalate when unsure\.External Links:2609\.26550,[Link](https://arxiv.org/abs/2609.26550)Cited by:[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px5.p1.1),[Appendix F](https://arxiv.org/html/2610.22526#A6.p2.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2610.22526#S5.SS2.p4.1)\.
- Linget al\.\(2026\)G\. Ling, M\. Xue, and Z\. YeJev in the wild: a data\-driven analysis of the Jev model’s functionality, applications and ecosystem\.External Links:2609\.30216,[Link](https://arxiv.org/abs/2609.30216)Cited by:[§1](https://arxiv.org/html/2610.22526#S1.p1.1)\.
- Lundberg and Lee \(2017\)S\. M\. Lundberg and S\. LeeA unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems 30,External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1)\.
- Naiket al\.\(2018\)A\. Naik, A\. Ravichander, N\. Sadeh, C\. Rose, and G\. NeubigStress test evaluation for natural language inference\.InProceedings of COLING 2018,External Links:[Link](https://aclanthology.org/C18-1198/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px5.p1.1)\.
- Nixonet al\.\(2019\)J\. Nixon, M\. W\. Dusenberry, L\. Zhang, G\. Jerfel, and D\. TranMeasuring calibration in deep learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) Workshops,pp\. 38–41\.External Links:[Link](https://openaccess.thecvf.com/content_CVPRW_2019/html/Uncertainty_and_Robustness_in_Deep_Visual_Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.html)Cited by:[Appendix F](https://arxiv.org/html/2610.22526#A6.SS0.SSS0.Px2.p1.1),[Table 10](https://arxiv.org/html/2610.22526#A6.T10),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1),[§5\.2](https://arxiv.org/html/2610.22526#S5.SS2.p2.1)\.
- Nunberget al\.\(1994\)G\. Nunberg, I\. A\. Sag, and T\. WasowIdioms\.Language70\(3\),pp\. 491–538\.External Links:[Document](https://dx.doi.org/10.1353/lan.1994.0007)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p3.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1)\.
- Ohet al\.\(2026\)S\. Oh, X\. Huang, M\. Pink, M\. Hahn, and V\. DembergTug\-of\-war between idioms’ figurative and literal interpretations in LLMs\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Rabat, Morocco,pp\. 2942–2958\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.135),[Link](https://aclanthology.org/2026.eacl-long.135/)Cited by:[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Ovadiaet al\.\(2019\)Y\. Ovadia, E\. Fertig, J\. Ren, Z\. Nado, D\. Sculley, S\. Nowozin, J\. Dillon, B\. Lakshminarayanan, and J\. SnoekCan you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/8558cb408c1d76621371888657d2eb1d-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1)\.
- Pezeshkpour and Hruschka \(2024\)P\. Pezeshkpour and E\. HruschkaLarge language models sensitivity to the order of options in multiple\-choice questions\.InFindings of NAACL 2024,External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.130),[Link](https://aclanthology.org/2024.findings-naacl.130/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px3.p1.1)\.
- Rao and Callison\-Burch \(2026\)D\. Rao and C\. Callison\-BurchJEV vs\. LLMs as rubric judges: cheaper, faster, and wrong in the same places\.External Links:2609\.29769,[Link](https://arxiv.org/abs/2609.29769)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.22526#S6.p1.1)\.
- Ribeiroet al\.\(2016\)M\. T\. Ribeiro, S\. Singh, and C\. Guestrin”Why should i trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ’16,New York, NY, USA,pp\. 1135–1144\.External Links:ISBN 9781450342322,[Link](https://doi.org/10.1145/2939672.2939778),[Document](https://dx.doi.org/10.1145/2939672.2939778)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p3.1)\.
- Ribeiroet al\.\(2020\)M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. SinghBeyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4902–4912\.External Links:[Link](https://aclanthology.org/2020.acl-main.442/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.442)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px5.p1.1)\.
- Ruderet al\.\(2021\)S\. Ruder, N\. Constant, J\. Botha, A\. Siddhant, O\. Firat,et al\.XTREME\-R: towards more challenging and nuanced multilingual evaluation\.InProceedings of EMNLP 2021,External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.802),[Link](https://aclanthology.org/2021.emnlp-main.802/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px6.p1.1)\.
- Sakhawatet al\.\(2026\)A\. Sakhawat, S\. A\. Parveen, M\. R\. Amin, S\. A\. Mahmud, M\. S\. Islam, and T\. KhatunWhen words don’t mean what they say: figurative understanding in Bengali idioms\.InProceedings of the Fifteenth Language Resources and Evaluation Conference,S\. Piperidis, N\. Bel, H\. van den Heuvel, N\. Ide, S\. Krek, and A\. Toral \(Eds\.\),Palma de Mallorca, Spain,pp\. 6870–6879\.External Links:[Document](https://dx.doi.org/10.63317/546w2cys6m6t),[Link](https://aclanthology.org/2026.lrec-1.546/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2610.22526#S3.SS0.SSS0.Px1.p1.1),[Ethics Statement](https://arxiv.org/html/2610.22526#Sx2.p1.1)\.
- Shwartz and Dagan \(2019\)V\. Shwartz and I\. DaganStill a pain in the neck: evaluating text representations on lexical composition\.Transactions of the Association for Computational Linguistics7,pp\. 403–419\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00277),[Link](https://aclanthology.org/Q19-1027/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Sunet al\.\(2026\)Y\. Sun, J\. Xu, J\. Shi, and Z\. YangType\-safe is not error\-free: a constrained decision head follows the option name, not the rubric bound to it\.External Links:2609\.26758,[Link](https://arxiv.org/abs/2609.26758)Cited by:[Appendix A](https://arxiv.org/html/2610.22526#A1.SS0.SSS0.Px4.p1.1),[Appendix P](https://arxiv.org/html/2610.22526#A16.SS0.SSS0.Px1.p1.1),[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px3.p1.1),[§5\.4](https://arxiv.org/html/2610.22526#S5.SS4.p2.1),[§5\.5](https://arxiv.org/html/2610.22526#S5.SS5.p2.1)\.
- Tang and Zheng \(2026\)L\. Tang and Y\. ZhengTyped decision models: an early evidence audit and evaluation checklist\.External Links:2609\.32160,[Link](https://arxiv.org/abs/2609.32160)Cited by:[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.22526#S6.p1.1)\.
- Tayyar Madabushiet al\.\(2022\)H\. Tayyar Madabushi, E\. Gow\-Smith, M\. Garcia, C\. Scarton, M\. Idiart, and A\. VillavicencioSemEval\-2022 task 2: multilingual idiomaticity detection and sentence embedding\.InProceedings of the 16th International Workshop on Semantic Evaluation \(SemEval\-2022\),Seattle, United States,pp\. 107–121\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.semeval-1.13),[Link](https://aclanthology.org/2022.semeval-1.13/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Titone and Connine \(1999\)D\. A\. Titone and C\. M\. ConnineOn the compositional and noncompositional nature of idiomatic expressions\.Journal of Pragmatics\.External Links:[Document](https://dx.doi.org/10.1016/S0378-2166%2899%2900008-9),[Link](https://doi.org/10.1016/S0378-2166(99)00008-9)Cited by:[Appendix A](https://arxiv.org/html/2610.22526#A1.SS0.SSS0.Px4.p1.1),[Appendix D](https://arxiv.org/html/2610.22526#A4.p3.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px2.p1.1)\.
- Traubet al\.\(2024\)J\. Traub, T\. J\. Bungert, C\. T\. Lüth, M\. Baumgartner, K\. H\. Maier\-Hein, L\. Maier\-Hein, and P\. F\. JägerOvercoming common flaws in the evaluation of selective classification systems\.InAdvances in Neural Information Processing Systems 37,External Links:[Document](https://dx.doi.org/10.52202/079017-0076),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/047c84ec50bd8ea29349b996fc64af4b-Abstract-Conference.html)Cited by:[Table 10](https://arxiv.org/html/2610.22526#A6.T10),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px4.p1.1),[§5\.2](https://arxiv.org/html/2610.22526#S5.SS2.p3.1)\.
- Turpinet al\.\(2023\)M\. Turpin, J\. Michael, E\. Perez, and S\. R\. BowmanLanguage models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems 36,External Links:[Document](https://dx.doi.org/10.52202/075280-3275),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/ed3fea9033a80fea1376299fa7863f4a-Abstract-Conference.html)Cited by:[Appendix D](https://arxiv.org/html/2610.22526#A4.p2.1),[§1](https://arxiv.org/html/2610.22526#S1.p2.1)\.
- Weiet al\.\(2024\)S\. Wei, C\. Wu, H\. Huang, and H\. ChenUnveiling selection biases: exploring order and token sensitivity in large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 5598–5621\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.333),[Link](https://aclanthology.org/2024.findings-acl.333/)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px3.p1.1)\.
- Xu \(2026\)Z\. XuJevOut: natural context can flip decision models\.External Links:2609\.30243,[Link](https://arxiv.org/abs/2609.30243)Cited by:[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1),[§5\.5](https://arxiv.org/html/2610.22526#S5.SS5.p3.1)\.
- Yu and Yao \(2026\)G\. Yu and Y\. YaoVisual Jev: accurate and efficient decisions from shared visual context\.External Links:2609\.25845,[Link](https://arxiv.org/abs/2609.25845)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2021\)Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. SinghCalibrate before use: improving few\-shot performance of language models\.InProceedings of ICML 2021,External Links:[Link](https://proceedings.mlr.press/v139/zhao21c.html)Cited by:[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px3.p1.1)\.
- Zhenget al\.\(2024\)C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. HuangLarge language models are not robust multiple choice selectors\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2309.03882)Cited by:[Appendix Q](https://arxiv.org/html/2610.22526#A17.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.22526#S2.SS0.SSS0.Px3.p1.1),[§5\.4](https://arxiv.org/html/2610.22526#S5.SS4.p2.1)\.
- Zhonget al\.\(2026\)J\. Zhong, M\. Li, and Y\. LaiWhen the right answer is missing: an arithmetic\-dependent rejection bottleneck in Jev\.External Links:2609\.39496,[Link](https://arxiv.org/abs/2609.39496)Cited by:[Appendix B](https://arxiv.org/html/2610.22526#A2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2610.22526#S5.SS3.p3.1)\.

## Appendix AThe typed question

Every call in this paper sends one typedchoicequestion and receives one probability vector\. Nothing else is exchanged: there is no prompt template, no system message, no sampling parameter and no generated text\. This is the whole interface, which is why every intervention in this paper acts on one of the two things the caller supplies: the state, or the option set\.

Table 4:The three model arms\.Item counts are each arm’s main run: the full ladder for Laya\-ml, a fixed 10% random subset for Lod\-lille, and the English pool for Jev\-1\.13\. Every arm also ran the3,2653\{,\}265\-item capture matrix in full\. Parameter counts are the model cards’\. Appendix[O](https://arxiv.org/html/2610.22526#A15)bounds the subset’s drift at0\.0160\.016\.#### Request\.

For the commercial arm the body posted to/api/alpha/decisionsis the following; the instruction string is the one in Table[5](https://arxiv.org/html/2610.22526#A1.T5)\.

> ``` { "model": "typesafe/jev-1.13", "state": ¡idiom or carrier sentence¿, "questions": { "q": { "type": "choice", "instructions": ¡instruction string¿, "criteria": { "A": ¡gloss¿, "B": ¡gloss¿, "C": ¡gloss¿, "D": ¡gloss¿ } } } } ```

The two open arms are called locally with the same four fields: the idiom as state, the instruction string, and the four option texts keyedA–D\. The option order is shuffled per item by a generator seeded with the item id and the experiment seed, so within an experiment the same item receives the same permutation in both open arms and in every family\. The commercial arm’s blocks use their own seeds \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\), so its permutations differ\.

#### Response\.

> ``` { "type": "choice", "choice": "B", "confidence": 0.75, "probabilities": { "A": 0.00, "B": 0.84, "C": 0.16, "D": 0.00 } } ```

We assert on every call that the returned keys are a subset of\{A,B,C,D\}and that the probabilities sum to11within10−210^\{\-2\}; no call in any run violated either\.

#### Two confidence signals, and which one we use\.

Note thatconfidenceis not the maximum probability: in the example above it is0\.750\.75whilemaxo⁡p⁡\(o\)=0\.84\\max\_\{o\}p\(o\)=0\.84\. It is a separate head whose documented job is to estimate whether the correct answer is present in the option list at all\. Lod\-lille ships the same arrangement\. We log it asconf\_headand report it where it matters \(§[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)\), but every cross\-model number in this paper uses the maximum probability, because it is the only confidence signal all three arms define identically\. Where the two disagree the head is not the more diagnostic of the pair: on Lod\-lille’s captured items mean head confidence is0\.6310\.631against0\.2950\.295on correct ones, the same inversion the top probability shows\.

#### Answer types we did not use\.

The interface also acceptsscore\(a bounded numeric judgment\) andbooleanquestions\. We usechoiceexclusively and deliberately\. Ascorequestion would ask the model to rate a gloss in isolation, which destroys the property the instrument depends on: that the gold and the literal reading compete inside a single forward pass, exactly as the psycholinguistic account describes\([Cacciari and Tabossi, 1988](https://arxiv.org/html/2610.22526#bib.bib5);[Titone and Connine, 1999](https://arxiv.org/html/2610.22526#bib.bib8)\)\. Abooleanquestion would do the same and additionally reintroduce a yes/no label whose name carries meaning of its own, which is the failure[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)document\. Keeping every experiment in one answer type also means the five distractor families differ in nothing but their option text\.

Table 5:The two instruction strings\. Each is one sentence and is byte\-identical across families, experiments and arms; the Bengali string is the direct equivalent of the English one\. Which string a call receives follows the language of the cell, on every arm: Bengali cells receive the Bengali string and English cells the English one, including the open arms’ English cells in E4 and in the capture matrix\. The open arms’ task configuration records only the Bengali string because their English cells set the English one in the run code\. The matched condition of Appendix[L](https://arxiv.org/html/2610.22526#A12)therefore shares its instruction across the three arms, and differs only in the sampled distractors, which the commercial arm drew under its own seed namespace\.

## Appendix BPositioning against concurrent work

The decision\-model literature is a few weeks old at the time of writing, and several papers describe failures adjacent to ours\. This appendix states what each one establishes, what it does not, and why our design is complementary rather than duplicative\.

#### Mechanism versus behavior\.

[Oh et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib9)apply causal tracing to autoregressive transformers and recover two pathways carrying an idiom’s figurative and literal readings at once: early layers and specific attention heads promote the figurative interpretation while a direct route keeps the literal one available, and disambiguating context is integrated from the first layer onward\. That is mechanistic evidence that the competition ourd\-litcondition exploits is real inside a model, and it is what makes our outcome measure interpretable rather than arbitrary: when a decision model places its mass on the literal gloss, there is a documented sense in which both readings were present and one won\.

The two designs answer different questions, and the difference is the reason both are needed\. Causal tracing requires weight access, token\-level activations and a generative forward pass to intervene on\. Two of our three arms release weights and the third does not, but none of the three emits a token stream to trace, and the commercial one is reachable only through an HTTP interface\. Their method cannot be pointed at the systems that are currently being deployed as routers and guardrails; ours can, at the cost of saying nothing about where inside the model the competition is resolved\. They establish*how*the literal route survives; we measure*how often*it takes over, in which language, and with what confidence attached\.

#### Option names and option definitions\.

[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)hold a typed model’s option definitions fixed and permute which name is bound to which, and find decision\-flip rates up to 70\.4 points higher for semantically loaded labels than for neutral ones: the model follows the name rather than the definition it was given\. They note that type\-safety conceals this, because every output remains structurally valid\.

Ouro\-series \(§[5\.5](https://arxiv.org/html/2610.22526#S5.SS5)\) manipulates the complementary half\. We hold the option names fixed atA–Dand destroy the internal structure of the text bound to them, chiefly by shuffling words within each option \(truncation is the weaker probe, Appendix[P](https://arxiv.org/html/2610.22526#A16)\), and find accuracy almost unchanged\. Read together, the two results point the same way from either side: within an option, these models are sensitive to lexical identity, and on random distractors their decisions do not depend on composed meaning\. That falls short of showing they cannot compose, since composition is not needed on that condition \(Appendix[P](https://arxiv.org/html/2610.22526#A16)\)\. Neither result alone would establish that, because theirs is consistent with a model that reads definitions carefully but weights names more heavily, and ours is consistent with a model that reads definitions as unordered bags\.

#### Context that flips decisions\.

[Xu \(2026\)](https://arxiv.org/html/2610.22526#bib.bib49)append short, fluent, task\-preserving context and redirect 61\.4% of initially correct decisions, with targeted flip rates of 64\.9–73\.2% across systems\. Our E3 carrier\-sentence condition \(§[5\.5](https://arxiv.org/html/2610.22526#S5.SS5)\) is the benign case of the same mechanism: embedding an idiom in its gold example sentence*raises*accuracy by0\.0800\.080and0\.1170\.117on the two open arms\. The two findings are the same sensitivity with opposite signs, and the uncomfortable implication is that a caller cannot tell them apart from the interface\. A model whose answer improves when helpful context is appended is, necessarily, a model whose answer moves when context is appended at all\.

#### Rejection, and the answer type\.

[Zhong et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib52)report the failure closest to ouro4control: asked to choose among candidates that contain no correct answer, Jev rejects on about7%7\\%of cases, while selecting a correct candidate almost perfectly when one is present\. Two scope conditions matter\. Their items are paired arithmetic problems with an explicit rejection option, which ours are not, so the agreement is between a reading and a result rather than a replication\. And they find the same model rejects correctly on about99%99\\%of those cases when asked instead for a Boolean verification of each candidate in turn\. That is a different answer type from the choice type we hold fixed throughout \(Appendix[A](https://arxiv.org/html/2610.22526#A1)\), and it is a genuine possibility that a verification\-typed call would recover what the choice type does not\. Our claim is about what the choice interface reports, which is the interface these systems are deployed behind; it is not a claim that the underlying model cannot tell\.

#### Calibration and cascades\.

[Ibrahim and Zaki \(2026\)](https://arxiv.org/html/2610.22526#bib.bib41)evaluate decision models against 19 language models on 18 computational social science tasks, and report a median deficit of 11\.6 F1 at 44 times lower cost, with confidence calibration*exceeding*most baselines; they recommend decision models as a cheap first stage that routes uncertain items onward\.[Li et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib43)build that system, accepting high\-confidence verdicts and escalating the rest, and exceed a frontier model’s accuracy at 41% of its cost\.

We reproduce the favorable calibration for the commercial arm on its own language \(§[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)\), so this is not a contradiction\. The boundary we add has two parts\. First, in\-distribution calibration does not survive the shifts we apply: on the open arms it fails under a targeted distractor shift, and on the commercial arm, which stays calibrated under that shift in English, it fails under the change of language\. On the open arms it fails in the direction that costs a cascade most: on the literal\-bearing families the high\-confidence tail is where the open arms are*least*accurate, so a confidence gate forwards exactly the items that should have been escalated\. Second, and independent of any distribution assumption, the semantics\-free control \(§[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)\) shows what each arm reports when the option list provably carries no information\. A gate calibrated on real options has no defense against that case, and a degenerate option list is an ordinary consequence of an upstream bug or an empty retrieval\.

#### What none of this settles\.

All of the above, ours included, is zero\-shot evaluation of base checkpoints\.[Tang and Zheng \(2026\)](https://arxiv.org/html/2610.22526#bib.bib50)review 28 papers published within nine days of the first widely used release and conclude that the typed readout has shown no independent accuracy advantage, with the demonstrated gains in latency, cost and cascade placement\. Whether any of these failure modes survives task\-specific fine\-tuning is, as far as we can determine, unmeasured by anyone so far\.

Table 6:One item exactly as received by the models\.The state is the Bengali idiom\\bnওজন \(*ojon*\), literally “physical weight,” with the English equivalent*carry weight*\. The instruction is identical across items and distractor families\. The four option texts are the actual strings reconstructed with the builder and seed used for the run and checked against the recorded gold and literal keys\. The probabilities are the returned values\. Laya\-ml places0\.990\.99on the literal option, which denotes the physical weight of an object, and0\.000\.00on the gold figurative gloss; Jev\-1\.13 places0\.820\.82on the gold option\.

## Appendix CWhat each experiment changes

Table[7](https://arxiv.org/html/2610.22526#A3.T7)states, for each experiment, the one thing that varies and the things held fixed\. Reading down the “varies” column follows the argument: each row isolates one candidate explanation for the failure measured in E1\.

Table 7:The seven interventions\. E5 is the calibration and selective\-prediction analysis of §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2): it issues no calls of its own and is computed from the E1 outputs\. E4’s two cells are paired corpus rows rather than the same strings: the English cell presents the annotator’s English equivalent idiom and the English glosses, so what is held fixed is the row and the family, not the state \(Appendix[J](https://arxiv.org/html/2610.22526#A10)\)\. Seeds are the Bengali arms’; the commercial arm uses a disjoint set \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\) because its blocks were run in a different order\.#### E1, the ladder\.

d\-randsamples three other idioms’ figurative glosses uniformly\.d\-surfsamples glosses sharing at least one surface token with the gold\.d\-semsamples glosses from idioms sharing at least one semantic tag\.d\-litreplaces one sampled distractor with the idiom’s own expert literal gloss\.d\-hardapplies all three pressures at once\. Duplicates are removed and the slate is topped up from the random pool if fewer thankkdistinct options survive\.

#### E3, perturbing the unit\.

E3 runs on the literal\-bearing option set, so its accuracies are comparable tod\-litrather than tod\-rand: on the same 1,778 items Laya\-ml scores0\.1760\.176underd\-litand0\.3530\.353underd\-rand, against0\.1570\.157for the intact idiom here\. The0\.1760\.176and0\.1570\.157are the same model on the same items under the same family, and they differ because E1 and E3 draw their random distractors from separate seed namespaces \(1000 and 3000\), so the two option lists share the literal gloss and the gold but not the two sampled distractors\. Comparisons are therefore made within an experiment, never across one\.v0is the intact idiom\.v1shuffles its words,v2reverses them,v3deletes one word at random,v4substitutes one word with a word sampled from another idiom, andv5replaces the idiom with its gold carrier sentence\. The option list is held byte\-identical across all six, so the only difference reaching the model is the state string\.

#### E6, ablating the options\.

o0is the full gloss\.o1truncates every option to its first six words\.o2shuffles the words within each option\.o3replaces each gloss with its counterpart in the other language\.o4replaces every option with a bare label carrying no semantics, which pins accuracy at chance by construction and is the control of §[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)\.

## Appendix DWhy interventions on the option set

This appendix explains the design choice the rest of the paper rests on, for readers who want the reasoning rather than the result\.

A decision model takes a question and a list of candidate options and returns a probability over exactly those options\. It does not sample and it emits no text\. Three of the usual explainability routes are therefore closed or compromised\. There is no rationale to inspect and no chain of thought whose faithfulness could be measured\([Jacovi and Goldberg, 2020](https://arxiv.org/html/2610.22526#bib.bib29);[Turpin et al\., 2023](https://arxiv.org/html/2610.22526#bib.bib30)\)\. Weights are available for two of our arms but not for the commercial one, so no method that requires opening every model, including probing and causal tracing, can be applied uniformly across the three\([Oh et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib9)\)\. And input attribution, the usual fallback\([Ribeiro et al\., 2016](https://arxiv.org/html/2610.22526#bib.bib31);[Lundberg and Lee, 2017](https://arxiv.org/html/2610.22526#bib.bib32)\), is not only unreliable here\([Jain and Wallace, 2019](https://arxiv.org/html/2610.22526#bib.bib33);[Adebayo et al\., 2018](https://arxiv.org/html/2610.22526#bib.bib34);[Hooker et al\., 2019](https://arxiv.org/html/2610.22526#bib.bib35)\)but hard to interpret for this material\.

The last point needs stating carefully, because it is what motivates everything else, and the standard shorthand for it is too strong\. Idioms are not uniformly non\-compositional\.[Nunberg et al\. \(1994\)](https://arxiv.org/html/2610.22526#bib.bib7)argue the opposite of the slogan usually attributed to them: idioms vary in how far their parts can be mapped onto components of the figurative reading, and many are*idiomatically decomposable*, with*spill the beans*distributing its meaning across*spill*and*beans*in a way that*kick the bucket*does not\. Psycholinguistic work finds decomposability measurable and consequential for processing\([Gibbs et al\., 1989](https://arxiv.org/html/2610.22526#bib.bib6);[Titone and Connine, 1999](https://arxiv.org/html/2610.22526#bib.bib8)\)\.

What follows is weaker than ill\-posedness but enough for our purposes\. A Shapley value over the tokens of an idiom is always computable, and for a non\-decomposable expression there is no reason to expect the resulting per\-token shares to correspond to anything linguistically real: the quantity is defined by the attribution game rather than by a decomposition the expression actually supports, and decomposability is a property of the individual idiom that we do not have annotated\. The numbers will still arrive, and they will still read as an explanation\. We therefore do not build on them, and intervene on the interface instead, which requires no assumption about where an idiom’s meaning sits\.

What remains is the interface itself\. The caller controls the question and the option list and observes a probability vector\. So we intervene there\. The corpus supplies, for every idiom, both a gold figurative gloss and an expert literal gloss, which means a counterfactual distractor is already annotated and we create no new labels: dropping the literal gloss into the option list turns a benchmark item into a targeted probe of whether the expression was read as an idiom or as its words\. Holding the item, the instruction andkkfixed while varying only the distractor family makes the comparison within\-item and within\-model, so a difference between two families cannot be explained by item difficulty, by the prompt, or by anything about the model that does not change between the two calls\.

The same logic produces the other five interventions; E5 is an analysis of E1’s outputs rather than an intervention\. If the failure were about the idiom’s surface form rather than its meaning, E2 would show it \(Appendix[N](https://arxiv.org/html/2610.22526#A14)\)\. If the model held the expression as a stored unit, breaking the unit in E3 would hurt\. If the problem were figurativeness rather than the language, E4 would not move\. If the model read options as sentences rather than as bags of words, E6 would collapse under scrambling\. If the scorer were order\-independent as documented, E7 would return the same gloss under all 24 permutations\. Each experiment is an attempt to explain the E1 result away, and the paper reports which attempts succeed\.

## Appendix EThe full distractor ladder

Table[8](https://arxiv.org/html/2610.22526#A5.T8)gives every family for every arm\. Two caveats on the commercial column: its numbers are English, and itsd\-surfandd\-semrows run on the larger 8,761\-item English pool rather than the 3,265\-item literal pool, because those families need no literal gloss\.

#### What the ladder is for\.

The five families form an ordered difficulty scale in which only the distractor construction changes\.d\-randdraws three other idioms’ figurative glosses uniformly, so a wrong answer requires confusing two unrelated meanings\.d\-surfdraws glosses belonging to idioms that share a word with the target, which makes the distractors lexically adjacent to the state but not to the gold\.d\-semdraws from idioms sharing a topical tag, which makes them semantically adjacent to the gold\.d\-litreplaces one distractor with the target’s own expert literal gloss\.d\-hardcombines the literal gloss with one surface and one semantic neighbor\. Reading down the first four rows therefore separates three different kinds of hardness, and the point of the ladder is that they are not interchangeable\.

#### The literal family is not merely the hardest\.

Measured as a drop fromd\-rand, the lexical and semantic neighbors cost Laya\-ml0\.0290\.029and0\.0340\.034, and Lod\-lille0\.0720\.072and0\.0380\.038\. The literal gloss costs them0\.2160\.216and0\.4180\.418: roughly six to eleven times as much as either neighbor family, from a single substituted option\. A benchmark that stopped atd\-sem, which is where most distractor\-hardness work stops, would have recorded these two models as moderately sensitive to distractor difficulty and would have missed the effect entirely\. That is the argument for the family being the independent variable rather than a robustness check\.

#### The commercial column does something different\.

Jev\-1\.13’s ordering is not the same shape\. Its worst family in English isd\-semat0\.8290\.829, and the literal family at0\.9310\.931is*easier*for it than the semantic one, which inverts the open arms’ ordering\. Those two rows come from different pools, so that comparison is indicative; the paired version is on the3,2653\{,\}265items bothd\-randandd\-litshare, where the literal gloss costs it only0\.0370\.037\(0\.9680\.968to0\.9310\.931\) against0\.2160\.216and0\.4180\.418on the open arms\. Read alone, that looks like a model which has simply solved the literal distractor\. Read against the Bengali column of Table[14](https://arxiv.org/html/2610.22526#A9.T14), where the same model on the same construction is captured at0\.3040\.304, it reads instead as a model for which the literal distractor is easy*in English*\. Only the matrix distinguishes the two, which is why we report the English ladder and the matrix together rather than either alone\.

#### Whyd\-hardadds little\.

d\-hardkeepsd\-lit’s literal gloss and replaces its two random distractors with harder neighbors, yet costs only a further0\.0070\.007and0\.0030\.003on the open arms\. Once the literal gloss is present it absorbs essentially all of the available error: the remaining two slots could be filled with anything without changing the outcome much\. This is also why we reportlaron both families and treatd\-litas the headline, since it is the cleaner single\-variable manipulation\.

FamilyLaya\-mlLod\-lilleJev\-1\.13bnbnend\-rand0\.3670\.4840\.968d\-surf0\.3380\.4120\.915d\-sem0\.3330\.4460\.829d\-lit0\.1510\.0660\.931d\-hard0\.1440\.0630\.854lard\-lit0\.6440\.8850\.047lard\-hard0\.6300\.8550\.043Table 8:Accuracy by distractor family, and literal attraction on the two families that contain a literal gloss\. Chance is0\.2500\.250\. The commercial column is English and is the reason §[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)runs the matrix: compare it with the Bengali column of Table[14](https://arxiv.org/html/2610.22526#A9.T14), where the same model is captured at0\.3040\.304\.Figure 4:Reliability, 15 equal\-width bins\. Panels left to right:Laya\-ml,Lod\-lille,Jev\-1\.13\.circles are the random\-distractor condition, squares in each panel’s own color the\+\+literal one, and the gray dashed diagonal perfect calibration\. Underd\-litthe two open models’ curves bend*downward*: the model is most wrong where it reports being most sure\. Jev\-1\.13, in English, stays on the diagonal\.

## Appendix FSelective prediction in detail

Selective prediction asks what happens if the caller answers only the items the model is most confident about\. A useful confidence signal makes accuracy rise as coverage falls\. Table[9](https://arxiv.org/html/2610.22526#A6.T9)reports accuracy at low coverage for the two open arms, on the random\-distractor condition and on the literal\-bearing one\.

Table 9:Accuracy under a confidence gate\.The two model arms are named in the column\-group headers and also carry their usual tints,cyan for Laya\-ml andmagenta for Lod\-lille, so the table reads without color\. On random distractors, both arms behave as a caller would hope: restricting to the most confident 5% roughly doubles accuracy\. On the literal\-bearing condition, the curve inverts\. Laya\-ml falls from0\.1510\.151at full coverage to0\.0070\.007in its most confident 5%, while Lod\-lille answersevery oneof its most confident 35% incorrectly\. Gating on confidence is therefore worse than not gating, and worse than gating at random\.This is the result with the most direct operational consequence\. A cascade that forwards high\-confidence answers and escalates the rest\([Li et al\., 2026](https://arxiv.org/html/2610.22526#bib.bib43);[Ibrahim and Zaki, 2026](https://arxiv.org/html/2610.22526#bib.bib41)\)forwards precisely the items on which these models are wrong, and escalates the ones they would have got right\.

#### How to read the table\.

Each row fixes a coverage level and reports accuracy on the retained items, so a row is not a subset of the row above it in any model\-independent way: the items retained at0\.050\.05coverage are whichever items*that*model ranked highest, which differ between the two arms and between the two families\. The comparison the table supports is therefore vertical within a column, not horizontal across them\. The quantity to watch is the sign of the gradient\. In the twod\-randcolumns it is the sign a caller expects, with one small reversal at the top of Lod\-lille’s \(0\.9770\.977at5%5\\%,0\.9890\.989at10%10\\%\), and the magnitude is substantial: Lod\-lille reaches0\.9770\.977in its most confident twentieth against0\.4840\.484at full coverage\. In the twod\-litcolumns it is reversed, and the reversal is monotone rather than noisy, which is what makes it an inversion of the mechanism rather than a weak signal\. The four zeros in the last column are literal: across the311311items making up Lod\-lille’s most confident35%35\\%underd\-lit, the model is correct on none\.

#### Calibration without binning\.

Table[10](https://arxiv.org/html/2610.22526#A6.T10)reports the same distributions under three metrics\. The point of giving three is that each fails differently\. ECE with equal\-width bins is the field standard and is also the metric[Nixon et al\. \(2019\)](https://arxiv.org/html/2610.22526#bib.bib18)show to be sensitive to the binning: with 15 equal\-width bins most of the mass can land in one or two bins, and an estimator averaging over near\-empty bins can move with the bin count\. Replacing them with 15 equal\-mass bins, which is their adaptive variant, changes nothing material here: the largest shift across the nine cells is0\.0110\.011, and the ordering of the arms is identical\. So the calibration failure is a property of the distributions, not of our bin edges\.

The Brier score is the stronger statement because it needs no bins at all\. It is strictly proper, so no reporting strategy beats reporting one’s true beliefs, and it has a fixed reference point: a model that returns0\.250\.25on all four options scores exactly0\.7500\.750\. Underd\-randboth open arms sit below that reference, which is what a model carrying some signal should do\. Underd\-litboth rise above it, to1\.0301\.030and1\.3911\.391\. This is worth stating plainly because it is stronger than miscalibration: on the literal\-bearing condition, a caller integrating these models’ full reported distributions is worse off than a caller who replaced them with a flat prior\. Jev\-1\.13 in English stays far below the reference throughout, and the one cell where it approaches a visible calibration cost isd\-hard\(0\.2200\.220, ECE0\.0470\.047\), which withd\-litis one of the two conditions containing the literal gloss\.

Table 10:Three calibration metrics on the same recorded distributions\.Laya\-ml,Lod\-lille,Jev\-1\.13\. ECE uses 15 equal\-width bins, ACE the adaptive equal\-mass variant of[Nixon et al\. \(2019\)](https://arxiv.org/html/2610.22526#bib.bib18); the largest gap between them across these nine cells is0\.0110\.011, so the ECE figures elsewhere in the paper are not artifacts of the binning\. Brier is the multiclass score and needs no bins; bold marks cells above0\.7500\.750, its value for a uniform four\-way distribution, where the model’s reported distribution is less useful to a caller than no preference at all\. AUGRC is the generalized risk\-coverage area of[Traub et al\. \(2024\)](https://arxiv.org/html/2610.22526#bib.bib24), which does not condition risk on the accepted set and so does not over\-weight high\-confidence failures; therandcolumn is its analytic random baseline, half the error rate, and bold marks the AUGRC cells that are worse than it\. No new model calls were made; every metric here is a re\-reading of the released per\-item probabilities\.

## Appendix GStratified results

Literal capture is not concentrated in rare or obscure idioms, which would be the comfortable explanation\. Splitting Laya\-ml’sd\-litcondition by the corpus’s own frequency annotation giveslar0\.6710\.671on*very common*idioms \(n=1,146n=1\{,\}146\),0\.6460\.646on*common*\(n=6,200n=6\{,\}200\),0\.6150\.615on*rare*\(n=1,507n=1\{,\}507\) and0\.5650\.565on*very rare*\(n=23n=23\)\. If anything the effect is strongest on the most familiar expressions, which is the opposite of a long\-tail story and consistent with these being the idioms whose literal readings are most available\.

The gradient is monotone across the three populated strata and spans0\.0560\.056, which is small against the effect itself but points the wrong way for a familiarity account\. We would not build on the*very rare*cell: atn=23n=23its interval is wide enough to be consistent with anything, and we report it only so the stratification is complete\.

The corpus’scultural\_significancefield is unusable as a stratifier: 10,337 of 10,359 rows carry the same value\. We mention this because it is the field a reader would most want stratified, and its absence is a property of the resource rather than a choice of ours\.

manipulationp⁡\(gold\)p\(\\text\{gold\}\)string sent to the modelE3: the state changesidiom:\\bnঅগাধ জলের মাছ \(*Still waters run deep*\)intact0\.21\\bnঅগাধ জলের মাছword shuffle0\.19\\bnমাছ জলের অগাধreversed0\.19\\bnমাছ জলের অগাধdelete one word0\.24\\bnঅগাধ জলেরsubstitute one word0\.24\\bnঅগাধ জলের ফ্যাin its carrier sentence0\.07\\bnসে অগাধ জলের মাছ, কখনো বুঝতে পারি না তার পরিকল্পনা কী।E6: the options changegold option rewritten; the other three are rewritten identicallyfull gloss\\bnখুব চালাক লোক; সুচতুর ব্যক্তি; যার মনোভাব বোঝা যায় নাtruncated to 6 words\\bnখুব চালাক লোক সুচতুর ব্যক্তি যারwords scrambled\\bnখুব লোক সুচতুর চালাক যায় ব্যক্তি না যার বোঝা মনোভাবEnglish glossA very clever person; a shrewd individual; someone whose mindset or intentions are hard to fathomsemantics removed\\bnঅর্থ 1Table 11:What the interventions do to real Bengali text\.The table shows the exact strings used for the idiom\\bnঅগাধ জলের মাছ \(*still waters run deep*, literally “a fish of deep water”\); the tinted string in each block is the unmodified one the rest of that block is derived from\.*E3*changes the state supplied to the model; the reported probabilities are Laya\-ml’s, on the gold figurative gloss\. E6 rewrites the options rather than the state, so a singlep⁡\(gold\)p\(\\text\{gold\}\)is not defined for its rows and the column is empty there\. Shuffling and reversing the expression leave the probability essentially unchanged, whereas the carrier sentence changes it\.*E6*changes the option text while holding its role fixed; after the full gloss, the rows truncate it, scramble its words, translate it, and finally remove its content, ending with the semantics\-free control\. The other three options undergo the same transformation as the gold option\. The E6 tokenizer splits on punctuation and rejoins on spaces, so the semicolons of the full gloss do not survive into the truncated and scrambled rows; the scrambled row is a strict permutation of the full gloss’s ten tokens and can be checked as one\.
## Appendix HOutput distributions

The three arms do not merely differ in accuracy; they place their probability mass very differently, and the shape of that mass is what makes the confidence results of §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)possible\. Table[12](https://arxiv.org/html/2610.22526#A8.T12)summarizes the returned vectors\.

Table 12:Distribution of the returned maximum probability, and normalized entropy of the full vector \(1=1=uniform\)\. The literal\-bearing family makes both open arms*sharper*, not flatter: Lod\-lille’s mean peak rises from0\.4470\.447to0\.7450\.745and the share of near\-deterministic answers from0\.0020\.002to0\.1710\.171, while its accuracy falls from0\.4840\.484to0\.0660\.066\. The model is not confused by the literal gloss; it is convinced by it\.That last point is worth stating separately, because it is the mechanism behind the inverted reliability curves\. A model that found the literal option merely distracting would spread mass and lose accuracy together\. Both open arms do the opposite: adding the literal gloss*concentrates*the distribution onto it\. On 21 items Laya\-ml returnsp=1\.000p=1\.000on the literal gloss andp=0\.000p=0\.000on the gold, to three decimal places\.

#### How to read the entropy column\.

Normalized entropy is the entropy of the returned four\-way vector divided bylog⁡4\\log 4, so11is uniform and00is a point mass\. It is reported alongside the maximum because the two can move independently and the pair identifies the shape\. A model that is uncertain between two options and has ruled out the other two has a high maximum and a middling entropy; a model hedging across all four has a low maximum and a high entropy\. Both open arms underd\-randsit near0\.890\.89, which is a mild preference over an almost flat vector: at0\.440\.44mean maximum they are closer to guessing than their accuracy suggests\. Underd\-litLod\-lille’s entropy falls to0\.4970\.497while its accuracy falls to0\.0660\.066, so it moves from hedging to committing at precisely the point where committing is wrong\.

Jev\-1\.13 is a different instrument\. Its entropy is0\.0680\.068underd\-randand85\.2%85\.2\\%of its answers exceed0\.950\.95, so it is nearly a hard classifier that happens to expose probabilities\. That matters for how its calibration figures should be read: a model returning1\.0001\.000almost always has little room to be miscalibrated in the ECE sense while it is accurate, and correspondingly no room to signal doubt when it is not\. Its low ECE in Table[2](https://arxiv.org/html/2610.22526#S5.T2)and its0\.8150\.815confidence on the semantics\-free control \(§[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)\) are the same property seen from two sides\.

#### Worked examples\.

Table[13](https://arxiv.org/html/2610.22526#A8.T13)gives five items in the original script with the annotators’ own glosses and English equivalents\. They are drawn from the extreme tail rather than sampled, items where the mass on the literal gloss is at or near1\.0001\.000, and we flag the selection because the tail is what an aggregatelarof0\.6440\.644does not convey\. All five also come from the2,4802\{,\}480items of §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6)on which neither lexical cue is available, so none of them is a case where the idiom and its literal gloss are the same string or where the literal option is the only one in Latin script\. What they add is the*kind*of expression involved:\\bnজলের দাম \(*joler dam*\), the price of water, means dirt cheap, and\\bnদু কথা \(*du kotha*\), two words, means a trifling remark\. These are everyday constructions, and the literal reading the model selects is in every case the annotators’ own rather than one we wrote\.

Table 13:Items where Laya\-ml places almost all probability on the literal gloss\.All five are drawn from the2,4802\{,\}480items on which none of the lexical construction cues of §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6)is available: each idiom is multiword, its literal gloss does not contain the idiom string, and the gloss carries no English parenthetical, so no option in the list carries Latin script\. Each idiom does still share a content word with its literal gloss, the hook that Table[16](https://arxiv.org/html/2610.22526#A10.T16)tests and finds does not carry capture\. Glosses and English equivalents are the corpus’s own; the English in parentheses is ours, for readability\. Both the literal and the figurative gloss were in the option list, and the last column is the mass placed on the literal one underd\-liton the full Bengali run\. On every one of these the model gave the gold gloss at most0\.0110\.011\.

## Appendix IThe capture matrix

Table[14](https://arxiv.org/html/2610.22526#A9.T14)gives the six cells of §[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)with the accuracy column alongsidelar; Table[15](https://arxiv.org/html/2610.22526#A9.T15)gives the bootstrap intervals on thoselarvalues and the paired within\-model language effect\. All cells aren=3,265n=3\{,\}265and were built by one function from seed 21000\. The language effect is positive on every arm and the intervals on the two language cells are disjoint in every case\. Jev\-1\.13’s Englishlarhere agrees with its English\-ladder value to four decimal places,0\.04720\.0472, which is why we treat the two constructions as measuring the same thing\. Its accuracy in that cell is0\.9260\.926against0\.9310\.931in Table[8](https://arxiv.org/html/2610.22526#A5.T8): same model, same items, same family, but the two runs draw their two sampled distractors from different seed namespaces \(12000 and 21000\), so the option lists share the gold and the literal gloss and differ in the rest\. Capture is insensitive to that and accuracy is mildly sensitive, which is what one would expect\.

Figure 5:Literal attraction by model and task language, all six cells on the same 3,265 items \(error bars 95% bootstrap CIs; dashed line the four\-option chance level\)\.Laya\-ml,Lod\-lille,Jev\-1\.13; solid bars Bengali, hatched bars English\. The Bengali ordering does not follow capability, and Jev\-1\.13’s English value matches the0\.0470\.047of the English ladder\. The cells are in Table[14](https://arxiv.org/html/2610.22526#A9.T14)and their intervals in Appendix[I](https://arxiv.org/html/2610.22526#A9)\.#### Why this table is the one we defend hardest\.

Everywhere else in the paper a comparison moves one thing and holds the rest fixed within one arm\. Here the design is a full crossing, so the same3,2653\{,\}265items, the same distractor draw and the same seed produce all six cells, and both margins are readable\. The language margin is large and uniform in sign: every arm is more captured in Bengali, by\+0\.352\+0\.352,\+0\.677\+0\.677and\+0\.256\+0\.256\. The capability margin is not even monotone\. Lod\-lille, which beats Laya\-ml by\+0\.133\+0\.133on random distractors over the 888 items they share \(Table[1](https://arxiv.org/html/2610.22526#S5.T1)\), is the*most*captured arm in Bengali, and the strongest arm is the least captured in both languages\. A single ordering of these three models therefore cannot predict their capture, while knowing the test language can\.

#### Reading accuracy againstlar\.

The two columns per language are not redundant\. Accuracy counts items answered correctly;larcounts items where the chosen option was specifically the idiom’s own literal gloss, which is one of the three available errors\. The gap between them is the residual error on other distractors\. For Lod\-lille in Bengali, accuracy0\.0440\.044andlar0\.9180\.918leave0\.0380\.038for everything else, so the literal gloss accounts for almost the whole error budget rather than merely enlarging it\. For Jev\-1\.13 in Bengali, accuracy0\.5360\.536andlar0\.3040\.304leave0\.1600\.160, so even its errors are less literal\-dominated\. This is the sense in which capture is a specific failure and not a restatement of accuracy\.

#### What the table cannot settle\.

Three things\. The pool is the36\.8%36\.8\\%of the corpus that carries a usable English gloss, so the levels are pool levels \(Appendix[J](https://arxiv.org/html/2610.22526#A10)\)\. The two language cells are paired entries rather than translations, and are asymmetric in construction in four ways we enumerate in the same appendix\. And the matrix covers the literal condition only; we did not cross the other four families with language, so we cannot say whether the language effect is specific to literal capture or general to distractor difficulty\. The first of these affects levels, the second we test in part, since the recorded outputs reach two of the four asymmetries, and the third is simply unrun\.

Table 14:The literal condition as a model×\\timeslanguage matrix, all six cells measured on the same 3,265 items with one construction and one seed \(n=3,265n=3\{,\}265per cell; 95% CIs in Appendix[I](https://arxiv.org/html/2610.22526#A9)\)\. Bold marks the highestlarper language\. Every arm is far more captured in Bengali than in English, and the Bengali ordering does not follow capability: Jev\-1\.13, the strongest arm, is the least captured in both languages, while Lod\-lille, stronger than Laya\-ml on random distractors \(Table[1](https://arxiv.org/html/2610.22526#S5.T1)\), is the more captured of the two in Bengali\.Table 15:Bootstrap intervals on thelarcells of Table[14](https://arxiv.org/html/2610.22526#A9.T14)and the paired within\-model language effect\. Intervals are 2,000\-resample percentile intervals on the same3,2653\{,\}265items;Δ\\Deltais the Bengali\-minus\-English difference on those paired items, McNemarp<10−100p<10^\{\-100\}in all three rows\.

## Appendix JHow the two language cells were built

The capture matrix is the result we lean on hardest, so this appendix states exactly how its two language cells were constructed, lists every asymmetry between them that we could find, and reports the tests of the two that the existing outputs can reach\.

#### Where the glosses come from\.

Neither literal gloss is a translation produced by us or by a machine\. The corpus annotates one field per idiom,literal\_meaning, written in Bengali with an English parenthetical alongside it, so a single expert\-authored field carries both readings\. The Bengali cell uses that field as written\. The English cell uses the parenthetical alone, recovered by a regular expression that accepts a parenthesized span only if it contains at least three ASCII letters and is at least60%60\\%ASCII alphabetic, which excludes Bengali asides and bare numerals\. Of8,8768\{,\}876items,3,3913\{,\}391carry such a parenthetical,3,3523\{,\}352of those also carry an English equivalent idiom, and3,2653\{,\}265survive a final cut requiring the English literal gloss to be lexically distinct from the English gold gloss \(Jaccard<0\.34<0\.34\)\. That pool of3,2653\{,\}265is the matrix\. The provenance matters because a machine\-translated English gloss would make the English cell a measurement of the translator; an annotator\-written one does not\.

#### The four asymmetries\.

We can state these precisely, which is better than asserting the cells are matched\.

*\(i\) The state differs in kind\.*The Bengali state is the idiom itself\. The English state is the annotator’s nearest English idiom, taken from thesimilar\_in\_englishfield, not a translation of the Bengali one\. The cells are therefore paired entries, and we use that phrase in the paper rather than “identical items”\.

*\(ii\) The Bengali literal option is bilingual\.*Because the Bengali cell uses the annotated field as written, and pool membership requires that field to contain an English parenthetical, every one of the3,2653\{,\}265Bengali literal options contains its own English gloss inside it, averaging4444characters against2424for the English literal option\. This is the most direct of the four, because it supplies a cue that has nothing to do with reading the expression: no Bengali figurative gloss in the corpus contains any Latin character, so whenever the literal option carries a parenthetical it is the*only*option in the list written partly in Latin script\. It is also usually the longer of the two glosses, though not reliably the longest of the four: that holds on40\.2%40\.2\\%of items, and §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6)shows length does not predict capture\. “Select the option containing English” would reproduce the Bengali capture numbers for all three arms and would also predict the English cell showing less capture, because there the cue is absent\. We test it below\.

*\(iii\) The distinctness cut applies to one side only\.*The Jaccard<0\.34<0\.34filter between gold and literal gloss was applied in English and not in Bengali\. Measured after the fact, the filter turns out not to bind: mean gold\-versus\-literal Jaccard is0\.0290\.029in Bengali against0\.0520\.052in English, so the Bengali side is the more separated of the two and would have passed the cut almost everywhere\. This asymmetry is procedural rather than consequential\.

*\(iv\) The literal gloss echoes the state more often in Bengali\.*With \(ii\), this is one of the two that could plausibly manufacture the result\. The Bengali literal gloss shares at least one content word with the Bengali idiom on56\.2%56\.2\\%of pool items; the English literal gloss shares one with the English equivalent idiom on25\.1%25\.1\\%\. A model doing order\-free lexical matching on the option side, which §[5\.5](https://arxiv.org/html/2610.22526#S5.SS5)shows all three partly are, has a surface hook into the Bengali literal option that it lacks in English\.

#### Testing asymmetry \(iv\)\.

The existing per\-item outputs are enough to test it without running anything new\. Split the pool on whether the Bengali literal gloss shares a content word with the idiom and readlarin each stratum \(Table[16](https://arxiv.org/html/2610.22526#A10.T16), upper block\)\. If the hook were driving capture, the overlap stratum would be far more captured in every arm\. It is not: the effect is\+0\.026\+0\.026for Laya\-ml,\+0\.130\+0\.130for Lod\-lille and−0\.129\-0\.129for Jev\-1\.13\. The sign is not even consistent across arms, and on the commercial arm the hook is associated with*less*capture, which is the opposite of the predicted direction\.

The sharper test restricts both language cells to the1,4291\{,\}429items where the hook is absent and re\-reads the gap there \(lower block\)\. It barely moves:\+0\.358\+0\.358,\+0\.610\+0\.610and\+0\.310\+0\.310, against headline gaps of\+0\.352\+0\.352,\+0\.677\+0\.677and\+0\.256\+0\.256\. For two of the three arms the gap is slightly*larger*once the hook is removed\. We read this as showing that asymmetry \(iv\) does not account for the Bengali–English difference, not as showing the cells are equivalent; a corpus paired by translation rather than by annotator judgment would settle what we can only bound here\.

Table 16:The content\-word hook does not carry the language gap\.Columns are Laya\-ml, Lod\-lille and Jev\-1\.13\. “Hook” means the Bengali literal gloss shares at least one content word with the idiom, one of the two asymmetries most likely to inflate Bengali capture\. Upper block: capture is not reliably higher when the hook is present, and on the commercial arm it is lower\. Lower block: restricting both language cells to hook\-absent items leaves the capture gap essentially unchanged, and larger for two of three arms\. The script is released with the per\-item outputs\.
#### Testing asymmetry \(ii\)\.

The capture matrix cannot test this, because every item in its pool carries a parenthetical by construction\. The Bengali arm of E1 can\. It runs thed\-litfamily over all8,8768\{,\}876items, and on5,4855\{,\}485of them the literal gloss carries no parenthetical at all, so the Latin\-script cue is simply absent\. If the cue were what produces capture, those items should show little of it\.

They show a great deal of it \(Table[17](https://arxiv.org/html/2610.22526#A10.T17)\)\. Two things follow, and they point in opposite directions\.

First, the mechanism is real, at least on the marker\-based scorer\. Laya\-ml’slaris0\.7460\.746where the cue is present against0\.5810\.581where it is absent, a difference of\+0\.165\+0\.165with a bootstrap interval of\[\+0\.145,\+0\.184\]\[\+0\.145,\+0\.184\]; on Lod\-lille it is\+0\.037\+0\.037with an interval straddling zero\. Part of the Bengali capture we report is therefore attributable to a cue rather than to a reading of the expression, and we would not have found that without looking\.

Second, it is not what produces the result\. With the cue entirely absent, Laya\-ml still selects the literal gloss on58\.1%58\.1\\%of items and Lod\-lille on87\.1%87\.1\\%, against a chance level of0\.2500\.250and against English\-cell capture of0\.3790\.379and0\.2410\.241\. The effect is also specific: on the three families that contain no literal option, the same split moves accuracy by at most0\.0420\.042with inconsistent sign, so the parenthetical stratum is not simply the easier or harder half of the corpus\.

Table 17:The Latin\-script cue is real but does not carry the result\.Laya\-ml,Lod\-lille\. Items are split by whether the idiom’s literal gloss carries an English parenthetical, which is the only thing in a Bengali option list written in Latin script\. Upper block: the cue is worth\+0\.165\+0\.165of capture on the marker scorer and little on the independent\-option scorer, yet capture without it is still0\.5810\.581and0\.8710\.871against a chance level of0\.2500\.250\. Lower block: the same split moves accuracy on families containing no literal option by at most0\.0420\.042, and on Laya\-ml by a consistent1\.41\.4to3\.03\.0points, five to twelve times less than what it buys on the literal family\. The commercial arm cannot be split this way, since its only Bengali literal run is the pool, where every item carries a parenthetical\.This bounds the cue rather than removing it\. It covers the two open arms and not the commercial one, and the no\-cue stratum is a different item set from the pool, so these numbers are not a clean subtraction from the matrix\. The experiment that would settle it is to strip the parenthetical from the Bengali literal option and re\-run the Bengali cell, which we have not done\.

#### Item construction, and what it could explain\.

The literal gloss is an annotated field rather than a string we wrote, and in this corpus it is often lexically close to the idiom\. Of8,8768\{,\}876items,3,7523\{,\}752\(42\.3%42\.3\\%\) have a single\-word idiom,3,3333\{,\}333have two words and the rest three or more\. On1,3031\{,\}303items \(14\.7%14\.7\\%\) the Bengali part of the literal gloss is identical to the idiom, and on1,5991\{,\}599\(18\.0%18\.0\\%\) it contains the idiom string verbatim\. The literal option averages41\.141\.1characters against31\.731\.7for the gold gloss and is the longer of the two on67%67\\%of items, although against all three alternatives it is the longest on only40\.2%40\.2\\%\. Some entries are also better described as polysemous words than as idioms: for\\bnচারপায়া \(*charpaya*\), “four\-legged” and “rope cot” are two senses of one word rather than a literal and a figurative reading of a phrase\. We did not filter these out, because the corpus’s own annotation is what defines the literal and figurative options and we did not want to substitute our judgment for it, but a reader should know they are in the pool\.

larby idiom length is not what a string\-matching account predicts\. For Laya\-ml it runs0\.6350\.635,0\.6840\.684,0\.5950\.595and0\.5730\.573across one\-, two\-, three\- and four\-or\-more\-word idioms, peaking in the middle rather than on the single\-word items where the idiom and its gloss are most often the same string\. Verbatim overlap does matter:laris0\.7670\.767where the gloss contains the idiom against0\.6160\.616where it does not \(\+0\.151\+0\.151, bootstrap\[\+0\.128,\+0\.175\]\[\+0\.128,\+0\.175\]\), and0\.9860\.986against0\.8660\.866on Lod\-lille \(\+0\.120\+0\.120,\[\+0\.089,\+0\.150\]\[\+0\.089,\+0\.150\]\)\. Removing both lexical cues at once, which leaves the2,4802\{,\}480items that are multiword, carry no verbatim overlap and carry no English parenthetical, leaveslarat0\.5600\.560and0\.8910\.891and accuracy at0\.1870\.187and0\.0790\.079\(Table[3](https://arxiv.org/html/2610.22526#S5.T3)\)\.

Length is not removed by that subset, so we test it on its own\. Reconstructing eachd\-litoption list under the run’s own seed, the literal option is the longest of the four on40\.2%40\.2\\%of all items and averages41\.141\.1characters against31\.531\.5for the other three\. Inside the cue\-free subset both gaps narrow, to30\.6%30\.6\\%and36\.636\.6against32\.432\.4characters, so the subset does attenuate length without eliminating it\. It does not need to: splitting on whether the literal option is the longest giveslar0\.6230\.623against0\.6570\.657on Laya\-ml, a difference of−0\.034\-0\.034\[−0\.055,−0\.013\]\[\-0\.055,\-0\.013\]in the*opposite*direction to the cue, and0\.8970\.897against0\.8770\.877on Lod\-lille,\+0\.020\+0\.020\[−0\.024,\+0\.064\]\[\-0\.024,\+0\.064\]\. Within the cue\-free subset the two differences are−0\.026\-0\.026\[−0\.069,\+0\.016\]\[\-0\.069,\+0\.016\]and\+0\.056\+0\.056\[−0\.025,\+0\.130\]\[\-0\.025,\+0\.130\], neither distinguishable from zero\. Length is a genuine asymmetry in the material and not a route to capture, which is why we report it separately rather than claiming the subset removes it\. The worked examples in Table[13](https://arxiv.org/html/2610.22526#A8.T13)are drawn from this subset so that none of them rests on verbatim overlap or the Latin\-script cue\. All five do share a content word with their literal gloss, which is asymmetry \(iv\), and Table[16](https://arxiv.org/html/2610.22526#A10.T16)shows that overlap does not carry capture\.

#### What this appendix does not establish\.

Asymmetry \(i\) remains untested: we cannot ask what the arms would do on an English state that is a literal translation rather than an idiomatic equivalent without running that condition\. We flag it in Limitations rather than dismissing it\.

## Appendix KCross\-lingual arms

Figure[6](https://arxiv.org/html/2610.22526#A11.F6)plots the E4 conditions discussed in §[5\.7](https://arxiv.org/html/2610.22526#S5.SS7)\. E4 differs from the capture matrix in two ways that are easy to conflate, so we state them\. E4 uses thed\-randfamily, with no literal gloss anywhere in the option list, and it asks what the language costs on an ordinary benchmark item\. The matrix usesd\-litand asks what the language does to literal capture specifically\. The two therefore measure different things and the figure should not be read as a version of Table[14](https://arxiv.org/html/2610.22526#A9.T14)\.

#### What the four Laya\-ml bars are\.

Laya\-ml has two checkpoints in play\. The first two bars are the multilingual checkpoint on Bengali and on English, which is the within\-model language contrast\. The third and fourth use the vendor’s English\-only checkpoint, and exist as a control rather than as a result: the third runs it on the English condition, where it should and does perform, and the fourth runs it on Bengali script, where it sits at0\.2830\.283, barely above the0\.2500\.250chance line\. That fourth bar does one specific job\. A checkpoint documented as unable to read the script fails here in the documented way, while the multilingual checkpoint on the same items reaches0\.3710\.371, so the scoring harness does pass Bengali through to a model that can read it\. What the bar cannot do is rule out a harness that depresses Bengali scores without erasing them, since an English\-only checkpoint would fail on Bengali either way; it shows the harness is not erasing the language, not that the harness contributes nothing to the deficit\.

#### The gains are not uniform, and the ordering is informative\.

Lod\-lille gains the most \(\+0\.380\+0\.380\), Laya\-ml and Jev\-1\.13 roughly similar amounts \(\+0\.246\+0\.246and\+0\.230\+0\.230\)\. Lod\-lille is the arm whose documentation states English\-only training evaluation, and it is also the arm whose Bengali accuracy \(0\.4840\.484\) is closest to chance relative to its English accuracy \(0\.8640\.864\)\. The natural reading is that its figurative knowledge is largely intact and the Bengali interface is where it is lost\. We cannot confirm that without probing the model, which we do not do here, and we note the alternative: a model may also be worse at Bengali figurative semantics specifically, which would produce the same external behavior\.

#### Why the error bars matter here\.

The three arms run at very differentnn, so the intervals differ by a factor of three in width\. Lod\-lille’s English condition in particular runs on its10%10\\%subset\. All three gaps clear McNemar atp<0\.001p<0\.001on paired items, which is the test to read rather than the overlap of the bars, since the bars are unpaired marginals and a paired effect can be significant while marginal intervals overlap\.

Figure 6:Accuracy by task language, grouped by model:Laya\-ml \(four bars\),Lod\-lille,Jev\-1\.13\.Solidbars are Bengali input,hatchedbars English; Laya\-ml’s third and fourth bars use the English\-only checkpoint, the fourth being the control\. Error bars 95% bootstrap CIs, dashed line chance\. Every model gains between 23 and 38 points when the paired English entry is presented instead of the Bengali idiom\.Figure 7:The semantics\-free control\. Per model, thebar is accuracy \(95% bootstrap CI\) and the colored bar is mean reported confidence \(Laya\-ml,Lod\-lille,Jev\-1\.13\); the dashed line is the four\-option chance level\. No option carries information, so the gap between the two bars is overconfidence:0\.2500\.250,0\.0490\.049and0\.5610\.561\. Jev\-1\.13’s condition is English\.

## Appendix LThe matched three\-way condition

Table 18:The common baseline across all three arms\.The condition is matched across models: a Bengali idiom followed by its Bengali figurative gloss with random distractors, the same instruction string and the samekk\. It is not byte\-identical, because the commercial arm drew its sampled distractors under a disjoint seed namespace \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\), so the three option lists share the gold and differ in the three distractors\. Each row is on that arm’s own items, sonndiffers\. The pairwise differences are computed on the items each pair shares, which is why they do not equal the differences between rows: Jev–Laya\+0\.369\+0\.369, Jev–Lod\+0\.259\+0\.259and Lod–Laya\+0\.133\+0\.133\(McNemar, allp<10−9p<10^\{\-9\}\)\.Figure[8](https://arxiv.org/html/2610.22526#A12.F8)plots Table[18](https://arxiv.org/html/2610.22526#A12.T18), the one condition that is matched across all three arms: Bengali idiom, Bengali glosses, random distractors, four options, gold position randomized, one instruction string\. The sampled distractors are not the same three, because the commercial arm’s blocks ran under their own seed namespace \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\); what is held is the construction, not the byte string\.

#### Why only one condition qualifies\.

Every other comparison in the paper is scoped by something the three arms do not share\. The literal condition needs an annotated literal gloss, which exists in Bengali for the whole corpus but in English only for the3,2653\{,\}265\-item pool\. The confidence results need a comparable confidence signal; the maximum probability is the only one all three define identically \(Appendix[A](https://arxiv.org/html/2610.22526#A1)\), and it is uncalibrated on Laya\-ml, whose shipped per\-bucket temperatures were not applied \(Appendix[R](https://arxiv.org/html/2610.22526#A18)\)\. Beyond it the arms expose different auxiliary signals, Lod\-lille’s separate confidence head and Jev\-1\.13’s presence\-of\-answer field, which we report where they bear on a result \(§[5\.2](https://arxiv.org/html/2610.22526#S5.SS2), §[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)\)\. The order experiment needs the full permutation set, which we ran at differentnnper arm\. This condition needs none of that, so it is the one place where a bare capability ordering can be read without a caveat attached\.

#### What the ordering is for\.

It exists to be contradicted later\. The ordering is Jev\-1\.130\.7360\.736, Lod\-lille0\.4840\.484, Laya\-ml0\.3670\.367, with every pairwise McNemar significant atp<10−9p<10^\{\-9\}, so it is not a close call\. Three later results do not follow it\. Capture is highest on the middle arm \(Table[14](https://arxiv.org/html/2610.22526#A9.T14)\)\. Calibration is best on the strongest arm by ECE and worst on the strongest arm on the semantics\-free control, where it reports0\.8150\.815confidence at0\.2540\.254accuracy\. Order stability is highest on the two arms with the independent\-option and commercial designs, which are the middle and the strongest arm, and lowest on the weakest\. Without a baseline ordering none of those non\-correspondences would be statable, which is why a table this simple earns its own appendix\.

#### The one caveat\.

Jev\-1\.13’s cell runs on8,7618\{,\}761items against Lod\-lille’s888888, and the three arms are not the same size, architecture or license\. The condition is matched on the*task*, not on the models, and a capability ordering across three systems chosen for architectural contrast is not a benchmark ranking\. We use it as a reference point internal to this paper and would not quote it as one\.

Figure 8:The matched three\-way comparison\. Bars left to right:Laya\-ml,Lod\-lille,Jev\-1\.13\. Black dashed line is chance, error bars 95% bootstrap CIs\. This capability ordering is the baseline against which the calibration failures in §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)–§[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)should be read\.

## Appendix MUnit integrity

Figure[9](https://arxiv.org/html/2610.22526#A13.F9)plots the E3 perturbations described in §[5\.5](https://arxiv.org/html/2610.22526#S5.SS5)\. The options are byte\-identical across all six variants, so the only difference reaching the model is the state string\. Table[11](https://arxiv.org/html/2610.22526#A7.T11)shows the six states for one idiom exactly as they were logged\.

#### Reading Table[11](https://arxiv.org/html/2610.22526#A7.T11)\.

The table is the one place in the paper where the actual strings reaching the model are shown, and it is worth reading closely because two of its rows look like typesetting errors and are not\. The*word shuffle*and*reversed*rows carry the same string: for a three\-word expression a particular shuffle and the reversal coincide, and the harness logged both conditions independently and happened to produce the same state\. We print what was logged\. Equally, the*substitute one word*row replaces the final word with an unrelated token, producing a string that is not Bengali in any idiomatic sense, and the model’s probability on the gold gloss*rises*from0\.210\.21to0\.240\.24\. That single cell is the clearest illustration of §[5\.5](https://arxiv.org/html/2610.22526#S5.SS5): damaging the expression beyond recognition does not cost the model anything, because the expression was not what it was reading\.

The one row that moves is the carrier sentence, and it moves*downward*here \(0\.21→0\.070\.21\\rightarrow 0\.07\) even though context raises accuracy on average by0\.0800\.080and0\.1170\.117\. A single item is not evidence against the aggregate; we include it rather than hunting for an item that agrees with the mean, because the aggregate is the claim and the item is an illustration\.

The lower block shows the E6 forms applied to the gold option\. The scrambled row is worth comparing word by word with the full gloss: it contains the same lexical inventory in an order no Bengali reader would accept, and costs the models almost nothing \(Table[20](https://arxiv.org/html/2610.22526#A16.T20)\)\. The last row is the semantics\-free control, and the table makes concrete what that control removes, which is everything except the option’s label\.

#### What each perturbation is supposed to break\.

The six states form a graded attack on the expression’s integrity\.*Intact*is the baseline\.*Word shuffle*destroys word order while preserving the lexical inventory, so a model storing the expression as an ordered unit should fail while a model matching its words should not\.*Reversed*is the specific shuffle that is maximally far from the original in order while being deterministic rather than sampled\.*Delete one word*removes material, which should hurt any model relying on the full form\.*Substitute one word*replaces material with an unrelated token, producing a string that is not the idiom at all and is not idiomatic Bengali\.*Carrier sentence*adds the corpus’s own example sentence around the intact idiom, which adds disambiguating context without touching the expression\.

#### Reading the two panels together\.

Accuracy on the left and intact\-agreement on the right answer different questions, and either alone is misleading\. Accuracy can stay flat because the perturbation does nothing, or because it changes many answers in compensating directions\. Agreement distinguishes those: it counts the share of items whose answer is unchanged from the intact state, so a model that is unaffected scores high on both while a model that is churning scores flat accuracy and low agreement\. Laya\-ml agrees with itself on0\.7420\.742of items under shuffling and Lod\-lille on0\.9170\.917\. For Lod\-lille the perturbation is close to invisible\. For Laya\-ml it is not negligible, but it is no larger than what a meaning\-preserving rewrite does: the annotators’ certified alternative forms leave Laya\-ml’s answer unchanged on only0\.6570\.657of pairs \(Appendix[N](https://arxiv.org/html/2610.22526#A14)\)\. Scrambling the expression’s word order moves this model no more than paraphrasing it faithfully, which is the opposite of what a model holding the expression as an ordered unit would show\.

#### What this rules out, and what it does not\.

It rules out the hypothesis that these models hold the idiom as a stored lexical unit whose retrieval is keyed on its exact form, because breaking the form does not cost them the answer\. It does not establish what they do instead\. The state string is read: §[5\.6](https://arxiv.org/html/2610.22526#S5.SS6)shows a literal gloss that contains the idiom verbatim is chosen more often\. What the shuffle shows is that its word order carries little, which fits a model that reads the expression as a set of words\. The carrier sentence is the only manipulation that moves either arm, by\+0\.080\+0\.080and\+0\.117\+0\.117, and it is the one that adds context rather than altering the expression\. Our reading is that the usable figurative signal is in the surrounding material rather than in the expression’s form, and we state it as a reading\.

#### What E3 can and cannot show\.

Three scope conditions bound it, and we state them rather than leaving them to be inferred\.

*The perturbations are real\.*E3 selects idioms of three words or more under the same tokenizer the distractor builder uses\. The corpus holds1,7911\{,\}791such idioms and the logged outputs cover1,7781\{,\}778of them,1,1361\{,\}136of three words,495495of four and147147longer; the1313missing are absent from the logged output rather than excluded by the rule, and at0\.7%0\.7\\%of the condition they move no reported figure\. No manipulation is a no\-op on a one\- or two\-word string\. Measured against the logged intact state, the shuffle changes the string on99\.9%99\.9\\%of items, reversal on99\.7%99\.7\\%, deletion and substitution on100%100\\%\. The one genuine collision is between shuffle and reversal, which coincide on16\.0%16\.0\\%of items because a three\-word reversal is one of only six permutations; we report both rows rather than merging them\.

*Accuracy is floored\.*The condition isd\-lit, where intact accuracy is already0\.1570\.157and0\.0720\.072, below the0\.2500\.250chance level\. A perturbation that destroyed the figurative reading entirely could not push accuracy much lower, so the flat accuracy rows are weak evidence on their own and we do not rest the conclusion on them\. Agreement with the intact answer has no such floor: it can run from00to11regardless of how accurate either answer is, and it is0\.7420\.742and0\.9170\.917under shuffling\. That is the number the “not a unit” reading rests on\. Running E3 underd\-rand, where there is headroom above chance, would let accuracy carry the argument too, and we did not\.

*Power\.*Lod\-lille’s E3 runs on180180items, so several of its individual perturbation contrasts are not separately significant and we do not claim them\. The claim is the pattern across six conditions and two arms, and the one contrast that is individually large in both\.

Figure 9:Unit integrity: the idiom is perturbed, the options held fixed\.Laya\-ml,Lod\-lille; error bars 95% bootstrap CIs, dashed line chance \(left panel\)\.*Left:*accuracy\.*Right:*agreement with the answer on the intact idiom\. Scrambling and reversing the expression does not meaningfully hurt either model, which is what a model reading the expression as a set of words would show\.

## Appendix NSurface invariance

The corpus ships alternative surface forms certified by annotators as meaning\-preserving: a variant spelling, a clipped form, or a near\-synonymous rewording recorded in thealternative\_idiomsfield by the same experts who wrote the glosses\. These are not perturbations of ours\. A model reading meaning rather than form should return the same answer on both members of such a pair\. E2 presents each pair under thed\-hardfamily, so the literal gloss is in the option list for both members and the comparison is made under the condition where capture is strongest\.

Laya\-ml agrees with itself on0\.6570\.657of3,1433\{,\}143pairs \(2,4232\{,\}423items\) and Lod\-lille on0\.6770\.677of310310pairs \(234234items\): about a third of the time, a rewrite the annotators certified as meaning\-preserving changes the answer\. Mean Jensen–Shannon divergence between the two returned probability vectors is0\.0390\.039and0\.0850\.085, on a scale where00is identical and11is disjoint support\.

#### Reading agreement against divergence\.

The two numbers look inconsistent and are not\. Small mean divergence with frequent answer changes is the signature of decisions taken on thin margins: the rewrite moves the distribution only slightly, but enough to cross the argmax boundary\. Laya\-ml’s mean peak probability on these pairs is0\.5350\.535, so its top two options are often close and a small shift reorders them \(Appendix[H](https://arxiv.org/html/2610.22526#A8)gives the full distributions\)\. Lod\-lille is the harder case at0\.7370\.737: its distributions are sharp, yet it still changes answer on a third of pairs, so for that arm the thin\-margin reading covers part of the instability rather than all of it\.

#### Why this is reported as a control rather than a headline\.

E2 is the experiment designed to explain the E1 result away, and it does not\. If capture were an artifact of the particular surface form the corpus happens to record, presenting the annotators’ alternative form should recover the figurative reading\. It does not\. Accuracy does move, and significantly:0\.138→0\.1930\.138\\rightarrow 0\.193for Laya\-ml and0\.035→0\.1580\.035\\rightarrow 0\.158for Lod\-lille \(McNemar exactp<0\.001p<0\.001on both\), so the surface form is not irrelevant\. These canonical\-form accuracies are lower than thed\-hardrows of Table[8](https://arxiv.org/html/2610.22526#A5.T8)\(0\.1440\.144and0\.0630\.063\) because E2 runs only on the items that carry an annotated alternative form,2,4232\{,\}423of8,8768\{,\}876and234234of888888, and that subset is harder for both arms; the comparison here is within the pair, so the level does not affect it\. But both forms sit far below the0\.2500\.250chance level on both arms, so the alternative form rescues neither model\. We read the two results together: these scorers are genuinely unstable under meaning\-preserving rewriting, which is worth reporting on its own, and that instability is neither large enough nor in the right direction to be the mechanism behind alarof0\.6440\.644to0\.8850\.885\.

#### Caveat on the pairs\.

Items vary in how many alternative forms they carry, so the pair counts exceed the item counts and items with more variants are weighted more heavily\. We report the pair\-level figure because it is the quantity a caller experiences; averaging within item first gives0\.6670\.667and0\.6970\.697, so the pair weighting*lowers*agreement by1\.01\.0and1\.91\.9points, which is why the pair\-level figure is the more conservative of the two\.

## Appendix OSubset representativeness

Lod\-lille ran a fixed10%10\\%random subset, drawn once with seed909909and reused for every experiment on that arm except the capture matrix, which it ran on the full3,2653\{,\}265\-item pool, so its subset cells are comparable to each other\. The question this appendix answers is whether they are comparable to the other arm’s full\-set cells\. Because the subset is a subset of the same corpus, we can measure the answer directly rather than assume it: Table[19](https://arxiv.org/html/2610.22526#A15.T19)runs Laya\-ml, which has the full set, on those same888888items and compares it with itself\.

#### Why this is the right test\.

An alternative would be to compare Lod\-lille on the subset against Laya\-ml on the full set and attribute any difference to sampling, which conflates the sample with the model\. Holding the model fixed and varying only the sample isolates the sampling term\. The drift in Table[19](https://arxiv.org/html/2610.22526#A15.T19)measures how well the subset stands in for the full corpus\. It does not enter the paired cross\-arm comparisons, which are made on the shared 888 items where both arms see the same subset\.

#### What the drift column shows\.

The largest absolute drift across five families is0\.0160\.016, ond\-rand\. Three families drift down and two up, but the two upward drifts are negligible \(\+0\.001\+0\.001,\+0\.004\+0\.004\), so the fair summary is a slight downward lean of under two points rather than no lean at all\. Two of the five subset estimates fall marginally outside the full\-set bootstrap interval, which atn=888n=888againstn=8,876n=8\{,\}876is the expected rate rather than a signal\. Critically, no family’s drift is large enough to reorder the ladder:d\-litandd\-hardremain far below the other three on both the full set and the subset, and thed\-randtod\-litgap, which is the quantity the paper reports, changes by0\.0010\.001\.

Table 19:Laya\-ml on the 10% subset against its own full set, drift computed before rounding\. Maximum drift is0\.0160\.016, and the larger drifts are downward, a slight lean of under two points\. Two of five subset estimates fall marginally outside the full\-set bootstrap CI, which is expected atn=888n=888and changes no sign or ordering\. The full\-set figures are therefore reported as the precise estimates, and Lod\-lille’s intervals are reported wider throughout\.

## Appendix PFull option ablation

Table[20](https://arxiv.org/html/2610.22526#A16.T20)gives the numbers behind Figure[10](https://arxiv.org/html/2610.22526#A16.F10)\. The five rows hold the state, the instruction and the gold position fixed and rewrite every option in the list the same way, so a row is a statement about what the option text needs to contain for the decision to survive\.

#### The first three rows are the finding\.

Truncating each option to its first six words costs0\.0040\.004,0\.0240\.024and0\.0110\.011across the three arms\. Scrambling the words*inside*each option, which leaves a string no reader of either language would accept as a sentence, costs0\.0080\.008,0\.0130\.013and nothing at all; on Jev\-1\.13 the scrambled row is marginally*above*the full gloss, which is within noise but makes the point that the manipulation is invisible to it\. Taken together: what these models need from an option is its lexical inventory, and the order and syntax of that inventory are close to free\. This is the behavioral half of the result[Sun et al\. \(2026\)](https://arxiv.org/html/2610.22526#bib.bib45)reach from the other direction by permuting which option name is bound to which definition\.

#### The fourth row is a separate effect\.

Presenting the options in the other language, with the state unchanged, costs Laya\-ml0\.0180\.018and Jev\-1\.130\.0070\.007but Lod\-lille0\.0950\.095, which is five to thirteen times as much\. The arm that loses most from a language mismatch between state and options is the arm whose documentation reports English\-only training evaluation, and the one with the largest cross\-lingual gap in E4\. We note the correspondence; with three models we cannot test it\.

#### The fifth row is the control, and it is the row that validates the instrument\.

Replacing every option with a bare label leaves0\.2560\.256,0\.2330\.233and0\.2540\.254, against a chance level of0\.2500\.250\. This matters more than it looks\. It shows that the four preceding rows are measuring option semantics rather than some artifact of the harness, because when the semantics are removed the measurement collapses to exactly where it should\. It also shows the gold\-position randomization is working: a harness with a position leak would score above chance here\. And it is the condition §[5\.3](https://arxiv.org/html/2610.22526#S5.SS3)builds on, because it is the one place where accuracy is known a priori, at0\.250\.25, so any reported confidence above that is unambiguously excess\.

#### Caveats\.

Three bound what theo\-series shows\. First, truncation to six words is a weaker manipulation than it sounds: the median Bengali gloss is four words, so truncation leaves77\.8%77\.8\\%of options untouched, and the English options, median seven words, are cut on58\.0%58\.0\\%\. The scramble is the informative row: it is necessarily a no\-op on the5\.7%5\.7\\%of options that are a single word, and otherwise changes the option’s word order\. Second, the whole series runs on random distractors, where identifying the right words already suffices to answer; a model that composes meaning and one that only matches words would both score well\. So these rows show that composition is not*required*on this condition, not that these models cannot compose\. Running the scramble underd\-litord\-sem, where the distractors are lexically close and composition would pay, is the test that would separate the two, and we did not run it\. Third, Jev\-1\.13’s column is English throughout, so its rows are not comparable in level to the other two, only in shape\. The three arms’*full gloss*rows differ slightly from the correspondingd\-randrows of Table[8](https://arxiv.org/html/2610.22526#A5.T8)because E6 draws its own distractor sample, under seed60006000on the open arms and1600016000on the commercial arm, whose E6 also runs on873873items rather than3,2653\{,\}265; the differences are0\.0020\.002to0\.0080\.008and we report both rather than reconciling them silently\.

Table 20:Option\-side ablation\.The semantics\-free row removes meaningful option content and yields near\-chance accuracy in all three arms\. Jev\-1\.13’s column is English\.Figure 10:Option\-side ablation\. Panels left to right:Laya\-ml,Lod\-lille,Jev\-1\.13\. In each panel the fifth,bar is the semantics\-free control of §[5\.3](https://arxiv.org/html/2610.22526#S5.SS3); the whole Jev\-1\.13 panel is hatched because its task is inEnglish\. Error bars are 95% bootstrap CIs and the black dashed line is chance\. Truncating and word\-scrambling the option text cost almost nothing in any of the three models\.

## Appendix QOrder sensitivity and position bias

This appendix reports the per\-slot selection shares behind §[5\.4](https://arxiv.org/html/2610.22526#S5.SS4), then sets out the permutation audit that establishes the residual instability of the independent\-option scorer is real rather than an artifact of our harness\.

#### What is held fixed\.

Each item here is one idiom with one fixed set of four option texts\. We present that same set 24 times, once per permutation, and change nothing else: the instruction string, the state,kk, the decoding configuration and the option texts themselves are byte\-identical across the 24 calls, and no call is retried\. The only quantity that varies is which slot, and therefore which letter, each option occupies\. Anything that moves across the 24 calls is attributable to that reordering, which cannot separate position from labeling \(§[5\.4](https://arxiv.org/html/2610.22526#S5.SS4)\): the condition is clean in what it holds fixed, not in what it can attribute\.

#### Item samples\.

The three arms’ item samples were drawn separately,992992for Laya\-ml,299299for Lod\-lille and300300for Jev\-1\.13, and barely overlap: no item is common to all three, and any two share at most3333\. Every cross\-arm figure in Tables[21](https://arxiv.org/html/2610.22526#A17.T21)and[22](https://arxiv.org/html/2610.22526#A17.T22)is therefore a comparison of rates on different items, not a paired one\. At these sizes the gap between83\.7%83\.7\\%and10\.7%10\.7\\%is far outside sampling error, but item composition is not controlled across arms\.

#### Recovering option identity\.

The harness records, per call, the permutation as a four\-digit string in which positionssholds the index, in the canonical option list, of the option displayed in slotss\. A chosen slot must be mapped back through that permutation before it can be compared across calls\. Counting distinct slot*letters*instead inflates instability to near unity for any scorer, because a letter necessarily changes whenever the same gloss is moved; on Laya\-ml that metric reads1\.0001\.000against the0\.8370\.837reported here\. It is an easy substitution to make and it is not what we want to measure, so we flag it\. All numbers in the paper use the mapped gloss identity\.

Table 21:Option\-order sensitivity\.Each item is evaluated under all 24 orderings of the same four options \(n=992/299/300n=992/299/300\)\.Laya,Lod,Jev; bold marks the largest value in each row\.*pp\(gold\) range*is the within\-item range of the gold option’s probability across the 24 orderings, averaged over items;*selection spread*is the gap between the most and least chosen slot, pooled over orderings, and is00under no positional preference\. The marker\-based scorer changes its chosen gloss on83\.7%83\.7\\%of items against10\.7%10\.7\\%for the independent\-option scorer\. The three item samples are drawn separately per arm, so cross\-arm comparisons are unpaired\. Lod is documented as order\-independent by construction; the permutation audit below shows it is not, which is why its entries here are small rather than zero\.
#### The permutation audit\.

Table[22](https://arxiv.org/html/2610.22526#A17.T22)reports a quantity that the summary measures in Table[21](https://arxiv.org/html/2610.22526#A17.T21)cannot: not whether the*answer*changes, but how far a*single option’s own score*travels as that option is moved around the list\. This is the quantity the independent\-option design makes a claim about\. Scoring every option from the same start position, so that no option attends to any other, should make an option’s score a function of that option and the state alone, so the spread should sit at the level of floating\-point accumulation noise, which forfp32summation over sequences of this length is on the order of10−710^\{\-7\}\. It does not\. Lod\-lille’s median item has an option whose score moves4\.0×10−24\.0\\times 10^\{\-2\}, five orders of magnitude higher, and not one of its 299 items falls below10−410^\{\-4\}\. Two consequences follow\. First, the10\.7%10\.7\\%answer\-change rate in Table[21](https://arxiv.org/html/2610.22526#A17.T21)is not a tie\-breaking or slot\-indexing artifact of our harness: ties would require near\-identical scores, and the scores are not near\-identical, so what moves is the scoring itself\. Second, the documented guarantee is not one an evaluation can rely on, at least in the configuration we were able to run\. We take the narrow reading throughout: the independent\-option design is a large improvement over the marker\-based one on this axis, and it is not the categorical property its documentation describes\.

Table 22:Permutation audit\.Per item, the largest distance a single option’s own score travels across the 24 orderings of its list, after mapping each slot back to the option it displayed\.Laya\-ml,Lod\-lille,Jev\-1\.13\. Bold marks the largest value in each row\. An order\-independent scorer should sit atfp32noise, around10−710^\{\-7\}; the*share*row gives the fraction of items at or below10−410^\{\-4\}, a threshold three orders of magnitude more permissive than that\. Only Jev\-1\.13 has any items at that threshold, and then fewer than a quarter; its probabilities are reported to two decimal places, so for that arm the row means identical*as reported*rather than identical in the underlying computation\. The script that produces this table is released with the per\-item outputs\.
#### The commercial arm\.

Jev\-1\.13 makes no public invariance claim, and its behavior is the least like either open arm\. It returns identical distributions on23\.0%23\.0\\%of items, with the caveat above that its probabilities reach us rounded to two decimal places, so this describes the reported values rather than the computation behind them\. Against that, its mean spread \(0\.1390\.139\) is three times Lod\-lille’s and its maximum \(0\.8200\.820\) approaches Laya\-ml’s\. The distribution is bimodal rather than merely noisier, which is what one would expect from a scorer that is close to deterministic on easy items and switches regime on hard ones\. We cannot test that reading without weights, and we do not claim it\.

#### Position bias\.

Figure[12](https://arxiv.org/html/2610.22526#A17.F12)gives the distributional summaries behind Table[21](https://arxiv.org/html/2610.22526#A17.T21), both from the permutation sweep\. Figure[11](https://arxiv.org/html/2610.22526#A17.F11)is the one panel in this appendix drawn from a different run: it reports per\-slot selection shares over the full ladder condition, pooled over its families, on8,8768\{,\}876items for Laya\-ml,888888for Lod\-lille and8,7618\{,\}761for Jev\-1\.13, because position is randomized per item there and the estimate is far better powered than the sweep’s992992items would allow\. The sweep agrees in direction and is milder,32\.7%32\.7\\%against18\.8%18\.8\\%on slot B and slot A for Laya\-ml\. Because the gold option is placed uniformly at random in both, the expectation under no positional bias is exactly0\.250\.25in every slot for every arm, so neither needs a baseline model: the distance from the dashed line*is*the bias\. Laya\-ml’s profile is not the primacy gradient usually reported for prompted generative scorers\([Zheng et al\., 2024](https://arxiv.org/html/2610.22526#bib.bib10)\); it peaks on slot B at37\.2%37\.2\\%against14\.6%14\.6\\%for slot A, a2\.5×2\.5\\timesswing in selection share from position alone, and is not a monotone function of position at all\. We note the shape and decline to explain it; with one model per scoring design, we have no way to separate a property of the marker scheme from a property of this particular backbone\.

Figure 11:Share of selections falling on each option slot, over the full ladder condition rather than the permutation sweep\. Panels left to right:Laya\-ml,Lod\-lille,Jev\-1\.13\. The gold option is placed uniformly at random, so the black dashed line at0\.250\.25is the expectation under no positional bias and any departure from it is bias\. Laya\-ml peaks on slot B, taking it37\.2%37\.2\\%of the time against14\.6%14\.6\\%for slot A; the other two arms sit close to the line throughout, with a mild and monotone preference for earlier slots\.Figure 12:Order sensitivity\.*Left:*how many distinct glosses each model selects across the 24 orderings of one option set; within each group bars runLaya\-ml,Lod\-lille,Jev\-1\.13, and a value of 1 means perfect stability\.*Right:*per model, the colored bar is the share of items whose chosen gloss is unstable and thebar is the mean within\-item range ofpp\(gold\)\. Laya\-ml is unstable on 83\.7% of items with a meanpp\(gold\) swing of0\.3460\.346; the other two are near\-stable\.

## Appendix RRun record

#### Environment\.

Python 3\.13\.15 throughout\. The two open arms ran on one Tesla T4 \(15,360 MiB\): Laya\-ml withamp\_dtypeset tobf16andmax\_len1024 at 27\.2 ms per call, Lod\-lille infp32at 86\.1 ms per call\. The T4 is Turing, whose tensor cores takefp16and integer inputs only, so there is no nativebf16path and nofp32ortf32tensor\-core path either: the autocast context accepts the setting and the arithmetic falls back to ordinaryfp32CUDA cores, withbf16rounding only at the cast boundaries\. We report the configured value because that is what the run record holds, and flag it because a reader comparing against an Ampere or later card is not comparing the same numerics\.

The commercial arm needs no accelerator and ran on CPU againsttypesafe/jev\-1\.13\. That alias is mutable: at the time of our run it resolved to the20260917build, and a later caller using the same alias may not reach the same weights\. Every commercial figure in this paper should be read against that build\.

#### Release\.

#### Temperatures\.

Laya\-ml’s run recordedtemperature = \[1\.0, 1\.0, 1\.0\]withtemperature\_by\_optionsempty, so the per\-bucket temperatures it ships were not applied and its ECE figures are raw\. Lod\-lille shipstemperature: 1\.0by design, its authors having reported that fitting hurt held\-out calibration\. Both open arms are therefore untempered and comparable on that axis\. The commercial arm’s internal calibration is not observable\.

#### Seeds\.

The Bengali arms use a per\-experiment seed \(E1 1000, E2 2000, E3 3000, E4 4000, E6 6000, E7 7000\) added to the item id, so each item’s option permutation is reproducible and identical across arms\. The split seed is 20261001\. The commercial arm uses a disjoint block seed set \(order audit 17000, literal 12000, ceiling 11000, Bengali control 14000, options 16000\) because its blocks ran in a different order; the permutation for a given item is still a pure function of seed and id\. Lod\-lille’s 10% subset was drawn once with seed909909\.

#### Call volume and cost\.

The commercial arm issued 56,533 calls for $0\.9348 at $0\.042 per million input tokens with output free\. The blocks are the order audit \(300300items×24\\times\\,24permutations=7,200=7\{,\}200\), literal capture on the shared pool \(3,265×33\{,\}265\\times 3families,d\-rand,d\-litandd\-hard,=9,795=9\{,\}795\), the English ceiling \(8,761×38\{,\}761\\times 3families,d\-rand,d\-surfandd\-sem,=26,283=26\{,\}283\), the Bengali control \(8,7618\{,\}761, one family\) and the option ablation \(873×5873\\times 5option forms=4,365=4\{,\}365\), which is 56,404 retained rows\. The option ablation runs on873873items rather than the888888of the shared subset because this block is English and1515of the888888carry no English equivalent idiom to use as the state; the873873are exactly the intersection of the subset with the8,7618\{,\}761items that do\. The remaining 129 calls are the endpoint probe and the response\-contract checks made before each block\. Transient 429 and 5xx responses were retried with exponential backoff to a 30 s ceiling; no call was dropped\.

#### Statistics\.

All intervals are 95% percentile bootstrap over items, 2,000 resamples, fixed seed 0\. ECE uses 15 equal\-width bins\. Paired comparisons use McNemar’s exact test via the binomial distribution\. AURC is reported against a random\-gate baseline computed on the same items\. Differences and ratios are computed from unrounded values, so they can differ in the last digit from differences of the printed figures\.

#### What is released\.

The per\-item outputs for every cell of every experiment \(one row per item per condition, carrying the full probability vector, the predicted key, the gold key, the literal key and the gold position\), the five split manifests, the three run environment records quoted above, the task configurations containing the verbatim instruction strings, the analysis scripts that turn those outputs into every table in this paper, the figure sources, and, beside each figure, a CSV of the exact plotted values\.

#### Pairing\.

Because the two open arms were reconstructed independently from a written contract rather than from shared code, the released outputs admit a direct check: on the 4,440 overlapping rows the gold key, gold position, idiom string and literal\-distractor key agree at1\.0001\.000\. That check is itself a released artifact, and it is what licenses every paired test in the paper\.

#### The matrix run\.

The six cells of §[5\.1](https://arxiv.org/html/2610.22526#S5.SS1)were run as one job, with their own seed namespace \(21000\) so no option permutation collides with the runs above\. Every cell covers the same 3,265 items and is built by a single function; the two languages share one distractor draw per item and are shuffled independently from the same generator\. Laya\-ml ran both languages on the multilingual checkpoint, so only the language changes and the checkpoint does not\. Lod\-lille ran the full pool here rather than its 10% subset, so its matrix cells rest on3,2653\{,\}265items\. The commercial arm issued 6,531 calls for $0\.1152\. Its English cell reproduces its English\-ladder value to four decimal places \(0\.04720\.0472\), which is the check that the two constructions measure the same thing\.

#### Known gaps\.

E2 was not run on the commercial arm, a gap in coverage rather than a constraint of language or cost\. Cost was not the constraint: the whole commercial arm came to $1\.05, the $0\.9348 of the main blocks plus the $0\.1152 of the matrix run\. The matrix covers the literal condition only; the other distractor families were not run in both languages\. The confidence*inversion*remains a two\-arm result, although §[5\.2](https://arxiv.org/html/2610.22526#S5.SS2)reports the commercial arm’s Bengali calibration from the matrix outputs\.