Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction

# Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction
Source: [https://arxiv.org/html/2608.12426](https://arxiv.org/html/2608.12426)
Mariya I\. Vasileva

###### Abstract

Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints—reasoning structure, safety boundaries, output schemas\. Individual constraints are handled proficiently, but the compositional regime, where many must hold jointly, remains poorly characterized: how rapidly does performance degrade, what governs the degradation, and can the collapse be mitigated? We introduceConstraint Saturation Evaluation \(CSE\), a procedurally generated benchmark that systematically varies the number of simultaneous constraints \(kk\), with every constraint scored by a deterministic, rule\-based verifier and zero LLM\-judge involvement: 15 models, 36 constraint types, 369,753 checks atk=1−12k\{=\}1\{\-\}12\. Three findings emerge\. First, per\-constraint pass rate decays gradually and predictably, while the chance of satisfying*all*kkconstraints collapses—a model passing individual constraints at∼41%\{\\sim\}41\\%atk=8k\{=\}8succeeds on all eight just 5\.7% of the time\. Second, constraints do not degrade equally: structural constraints lose2×2\{\\times\}more baseline capability per added constraint than lexical ones, ordered by a comprehension\-maintenance gap that separates constraints requiring sustained tracking from binary decisions immune to composition\. Third, failures are nearly independent, which is what makes the accumulation multiplicative; the residual coupling that does exist tracks shared output features rather than pairwise interference—a wrong sentence count fails every constraint that reads it\. Weak coupling means there is no pairing to exploit: no selection or arrangement of constraints mitigates the collapse\. Three targeted ablations find little room at inference time: pre\-generation planning does not move the threshold at all, while post\-hoc self\-correction and best\-of\-5 retries delay it by only one to two constraints\. Only raising the per\-constraint pass rate helps\. Reliable instruction following breaks down beyond 5–6 simultaneous constraints: probe\-level success falls below 50% at 7 constraints for the strongest model, and at 3 or fewer for 12 of 15\. All verifiers, probes, model outputs, and code are released for reproducibility\.

## 1Introduction

Language models deployed in complex applications must now adhere to multiple explicit constraints simultaneously—reasoning structure, safety boundaries, domain\-specific protocols, output schemas—within a single response\. Models handle individual constraints well, but performance under composition is poorly understood: does it degrade gracefully, or is there a sharp transition? Three specific questions remain open: what is the functional form of the degradation, is there a structure governing which constraints fail first, and does the collapse arise from pairwise interference between constraints or from something simpler? A fourth question follows from any answer to these: can the collapse be mitigated?

Several benchmarks address pieces of this question\. IFEval[52](https://arxiv.org/html/2608.12426#bib.bib1)pioneered deterministic verification across 25 constraint types but evaluates atk≤3k\{\\leq\}3\. COLLIE[47](https://arxiv.org/html/2608.12426#bib.bib3)demonstrated through a formal constraint grammar that composition increases difficulty, but uses fixed compositions rather than varyingkk\. FollowBench[19](https://arxiv.org/html/2608.12426#bib.bib2)extends tok=5k\{=\}5—the closest existing per\-kkanalysis—but relies on GPT\-4 as judge\. RECAST[13](https://arxiv.org/html/2608.12426#bib.bib5)pushes to 13\+ constraints but reports at four coarse tiers; CCTU[49](https://arxiv.org/html/2608.12426#bib.bib10)finds no model exceeds 20% under strict multi\-constraint adherence\. InfoBench[37](https://arxiv.org/html/2608.12426#bib.bib4)observes that models fail number and linguistic constraints preferentially—a hierarchy we extend acrosskk\. Outside text, ConceptMix[46](https://arxiv.org/html/2608.12426#bib.bib17)reports that text\-to\-image models drop from 83% to 8% as concept count grows from 1 to 7\. In compositional generalization more broadly[21](https://arxiv.org/html/2608.12426#bib.bib47),[9](https://arxiv.org/html/2608.12426#bib.bib45)show exponential decay with composition depth, and[34](https://arxiv.org/html/2608.12426#bib.bib46)quantify a compositionality gap persisting at∼40%\{\\sim\}40\\%regardless of scale\. Benchmarks that rely on LLM\-based evaluation[19](https://arxiv.org/html/2608.12426#bib.bib2);[13](https://arxiv.org/html/2608.12426#bib.bib5);[46](https://arxiv.org/html/2608.12426#bib.bib17)face a circularity at highkk: the judge itself struggles with compositional constraint violations in exactly the regime where models fail most\. The pattern is otherwise consistent across benchmarks, modalities, and tasks: composition is hard\. What is missing is the functional form—where capability breaks, what drives the breakdown, and whether the trajectory is predictable\. An extended discussion of all related work, including training methods, mechanistic explanations, cross\-modal parallels, and phase transition theory, appears in[AppendixB](https://arxiv.org/html/2608.12426#A2)\.

We introduceConstraint Saturation Evaluation \(CSE\), designed to address all three\. Every constraint is verified by a deterministic function rather than an LLM judge—a design choice that makes CSE’s measurements invariant to the same compositional degradation it studies\. CSE evaluates 36 deterministically verifiable constraint types spanning 8 processing dimensions, 4,527 probes atk=1k\{=\}1to1212, 15 models across 8 families \(369,753 constraint checks\)\.

We measure the co\-failure structure directly and find failures nearly independent, which makes joint success approximately the product ofkkper\-constraint rates: a mild per\-constraint decay compounds into a steep collapse\. Prior work assumes interference\. The rates, their 2%\-to\-28% spread across models, and the failure ordering are all experimental results\.

We address three open questions:

1. 1\.Is the decay predictable?Per\-constraint pass rate follows a multiplicative model \(72\.0%×0\.922k−172\.0\\%\\times 0\.922^\{k\-1\}, held\-out MAE=0\.2\{=\}0\.2pp\), but probe\-level success collapses below 2% byk=9k\{=\}9: a model passing individual constraints at∼41%\{\\sim\}41\\%atk=8k\{=\}8succeeds on all eight just 5\.7% of the time\. Probe\-level success drops below 50% at just 7 constraints for the strongest model, and at 3 or fewer for 12 of 15 \([Section4](https://arxiv.org/html/2608.12426#S4)\)\.
2. 2\.Do all constraints degrade equally?Structural constraints lose2\.0×2\.0\{\\times\}more baseline capability per added constraint than lexical ones \(95% CI:\[1\.9,2\.3\]\[1\.9,2\.3\]\), and the comprehension\-maintenance gap \(ρ=−0\.584\\rho\{=\}\{\-\}0\.584\) predicts degradation rate: constraints requiring sustained tracking during generation degrade fastest, while binary decisions are immune \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px6)\)\.
3. 3\.What drives the collapse—interference or accumulation?Constraint failures are weakly correlated through shared output features \(meanφ=\+0\.067\\varphi\{=\}\{\+\}0\.067\), not pairwise interference\. This near\-multiplicative structure is what makes the decay in \(1\) predictable—and it means the only lever for improvement is per\-constraint reliability, not constraint pairing \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)\)\.

## 2Related Work

Several benchmarks address pieces of this question\. IFEval[52](https://arxiv.org/html/2608.12426#bib.bib1)and COLLIE[47](https://arxiv.org/html/2608.12426#bib.bib3)establish deterministic verification of instruction\-following constraints, with COLLIE demonstrating through a formal constraint grammar that composition increases difficulty—but IFEval evaluates atk≤3k\\leq 3, and COLLIE uses fixed compositions rather than systematically varyingkk, precluding trajectory analysis\. FollowBench[19](https://arxiv.org/html/2608.12426#bib.bib2)introduces a multi\-level mechanism that incrementally adds one constraint per level up tok=5k\{=\}5and tracks both hard and soft satisfaction rates across levels—the closest existing per\-kkanalysis—but relies on GPT\-4 as judge for open\-ended constraints and does not characterize the functional form of the observed degradation\. RECAST[13](https://arxiv.org/html/2608.12426#bib.bib5)pushes constraint density to 13\+ per instance and proposes reinforcement learning from verifiable constraints, but reports performance at four coarse difficulty tiers rather than per\-kktrajectories, precluding decay analysis\. In constrained tool use, CCTU[49](https://arxiv.org/html/2608.12426#bib.bib10)finds no model exceeds 20% task completion under strict multi\-constraint adherence\. Outside text entirely, ConceptMix[46](https://arxiv.org/html/2608.12426#bib.bib17)reports that text\-to\-image models drop from 83% to 8% as concept count grows from 1 to 7\. The pattern is consistent across benchmarks and modalities: composition is hard\. What is missing is a characterization of the*functional form*of the degradation—where capability breaks, what drives the breakdown, and whether the trajectory is predictable\. An extended discussion of all related work, including training methods, mechanistic explanations, cross\-modal parallels, and phase transition theory, appears in[AppendixB](https://arxiv.org/html/2608.12426#A2)\.

## 3Constraint Saturation Evaluation \(CSE\)

### 3\.1Constraint taxonomy

CSE defines 36 constraint types across 8 dimensions \([Table2](https://arxiv.org/html/2608.12426#S4.T2)\), selected to be deterministically verifiable and to span a range of processing demands\. Constraints range from local pattern matching \(lipogram: avoid a letter; forbidden word: exclude a token\) through document\-level structure \(exact paragraph count, monotonic sentence length\) to relational reasoning requiring structured output \(scene graphs, logic grids\) and self\-referential reasoning \(a sentence correctly reporting its own word count\)\. Whether this design choice produces a measurable hierarchy in degradation rates is tested in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px6)\. All 36 verifiers are fully deterministic, each returning both a binary pass/fail judgment and a continuous scores∈\[0,1\]s\\in\[0,1\]representing partial compliance\.

Each constraint entered the pool through a four\-stage vetting pipeline—deterministic verifiability, parameter calibration, composability analysis, and pilot validation \([AppendixK](https://arxiv.org/html/2608.12426#A11)\)\. The composability stage is critical for a compositional benchmark: a constraint that individually tests an interesting capability but locks down multiple output dimensions \(sentence count*and*word count*and*ordering\) inherits the union of those dimensions’ incompatibilities and would distort the composition bias at highkk\. Composability profiles for all 38 candidates and a case study comparing an accepted constraint against a rejected one appear in[AppendixK](https://arxiv.org/html/2608.12426#A11)\.

Following[19](https://arxiv.org/html/2608.12426#bib.bib2), we report two complementary satisfaction metrics\. For a set ofmmprobes, each containingkkconstraints with binary outcomessij∈\{0,1\}s^\{j\}\_\{i\}\\in\\\{0,1\\\}:

Strict CSR \(sCSR\):

sCSR=1m​∑i=1m∏j=1ksij\\text\{sCSR\}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\prod\_\{j=1\}^\{k\}s^\{j\}\_\{i\}\(1\)measures the fraction of probes where*all*kkconstraints are simultaneously satisfied\.

Marginal CSR \(mCSR\):

mCSR=1m​k​∑i=1m∑j=1ksij\\text\{mCSR\}=\\frac\{1\}\{mk\}\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{k\}s^\{j\}\_\{i\}\(2\)measures the average per\-constraint satisfaction rate\.

Atk=1k\{=\}1, sCSR and mCSR are identical\. Their divergence at higherkkis the phase transition: mCSR decays gradually \(each constraint fails somewhat more often under load\), but sCSR collapses because it is the*product*ofkkmarginal rates\. sCSR is the deployment\-relevant metric—in practice, all constraints must hold simultaneously—and is where the phase transition is visible\. mCSR shows that per\-constraint competence is not the bottleneck: models that pass individual constraints at 45% may succeed on complete probes only 12% of the time\.

We additionally report the mean continuous scores¯i∈\[0,1\]\\bar\{s\}\_\{i\}\\in\[0,1\]for each constraint, which credits near\-misses \(e\.g\., a model producing 48 words when the target is 50 scores∼0\.96\{\\sim\}0\.96but passes=\{=\}False\)\. The*comprehension\-maintenance gap*is

Δi=s¯i−mCSRi,\\Delta\_\{i\}=\\bar\{s\}\_\{i\}\-\\text\{mCSR\}\_\{i\},\(3\)where a largeΔ\\Deltaindicates that a model partially complies—demonstrating comprehension of the constraint—but fails to maintain full satisfaction\.

We define the*compositional half\-life*k∗k^\{\*\}for each model as the smallestkkat which sCSR drops below 50%—the number of simultaneous constraints a model can reliably handle\.

### 3\.2Probe generation

CSE probes are generated procedurally \([Algorithm1](https://arxiv.org/html/2608.12426#alg1)\)\. For each constraint countkk, the composer drawskk\-subsets from the constraint pool, rejects unsatisfiable combinations via the compatibility checker \([AppendixI](https://arxiv.org/html/2608.12426#A9)\), instantiates parameters from fixed calibrated ranges, shuffles constraint presentation order, and assembles the prompt\. The composer targets a fixed number of satisfiable probes perkklevel, resampling new combinations when rejections occur rather than accepting reduced yield\. At lowkk\(≤5\{\\leq\}5\), all\(\|𝒞\|k\)\\binom\{\|\\mathcal\{C\}\|\}\{k\}subsets are materialized and shuffled for uniform coverage; atk≥6k\{\\geq\}6, combinations are sampled uniformly at random with a per\-combination reuse cap\.

#### Compatibility checker\.

The compatibility checker enforces satisfiability through unconditional pair blocks, parameter\-dependent arithmetic checks, and cross\-constraint propagation \([AppendixI](https://arxiv.org/html/2608.12426#A9)\)\. Rejection rates increase withkk: from∼32%\{\\sim\}32\\%atk=4k\{=\}4to∼98%\{\\sim\}98\\%atk=12k\{=\}12; the resampling loop compensates by drawing additional combinations from the\(36k\)\\binom\{36\}\{k\}pool \(∼24\{\\sim\}24M compatible atk=12k\{=\}12out of\(3612\)≈1\.25\\binom\{36\}\{12\}\\approx 1\.25B total;[Table13](https://arxiv.org/html/2608.12426#A16.T13)\)\. The resulting composition bias at highkk—surviving probes exclude incompatible constraint types—is analyzed and controlled for in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px2)\.

#### Parameter control\.

All constraint parameters are sampled from fixed calibrated ranges held constant acrosskk\(e\.g\., word count targets of 40–60, lipogram bans on letters with English frequency rank 5–15\), so that the phase transition measures compositional burden, not parameter variation\. A Kruskal\-Wallis test confirms no significant variation in mean per\-constraint difficulty acrosskkstrata \(H=6\.23H\{=\}6\.23,p=0\.96p\{=\}0\.96\)\.

#### Impossible probes\.

Additionally, 444 deliberately impossible probes test constraint*prioritization*under logical impossibility\. Each probe contains a constraint pair with a clean proof of unsatisfiability—e\.g\., L1 \(avoid letter ‘e’\) paired with L2 \(include “telephone”\)—and measures which constraint the model sacrifices when compliance is structurally unachievable\. Probe types span direct contradictions between constraint pairs, hand\-designed hierarchy tests \(e\.g\., L1 vs R2 with all\-‘r’ items, directly testing whether the depth\-of\-processing ordering extends from degradation rates to sacrifice priorities\), and hidden impossibilities embedded atk=4−8k\{=\}4\{\-\}8to test whether impossibility*detection*degrades under compositional load alongside constraint*satisfaction*\. Full probe inventory and analysis plan appear in[AppendixR](https://arxiv.org/html/2608.12426#A18)\.

#### Range ofkk\.

We evaluate atk=1k\{=\}1to1212\. Atk=12k\{=\}12, each probe draws 33% of the constraint pool and the incompatibility rejection rate reaches∼98%\{\\sim\}98\\%, narrowing the diversity of surviving combinations; extending further yields diminishing combinatorial coverage\. The design splits the range into a training region \(k=1−8k\{=\}1\{\-\}8\) used for model fitting and a held\-out region \(k=9−12k\{=\}9\{\-\}12\) used for validation \([AppendixP](https://arxiv.org/html/2608.12426#A16)\)\.

### 3\.3Extraction and verification

Models receive the full prompt containing a topic and allkkconstraint instructions in natural language; no chain\-of\-thought scaffolding, decomposition hints, or constraint\-ordering cues are provided\. For example, ak=4k\{=\}4probe might read:

> Write a short essay about renewable energy\.Your response must not contain the letter ‘e’ anywhere\. Your response must contain exactly 3 paragraphs, each containing at least 2 sentences\. Every word must be between 1 and 8 characters long\. Your response must include all of the following words: “solar”, “carbon”, “wind”\.

Responses are wrapped in<answer\>tags; only content inside the tags is scored\. The extractor uses the*last*matched tag pair \(handling mid\-response revisions\), accepts open\-without\-close tags so that constraints satisfied before truncation still count, and falls back to the full response when tags are absent\. Tag compliance is≥90%\{\\geq\}90\\%for 12 of 15 models; the three exceptions \(Gemini Pro 47%, Kimi K2\.6 67%, DeepSeek V4 Pro 73%\) are scored via the fallback\. Before verification, markdown artifacts \(bold, italic, headings\) that do not affect constraint semantics but interfere with character\-level verifiers are stripped; structural formatting \(bullets, code blocks, tables\) is preserved\.

All 36 verifiers are fully deterministic, each returning both a binary pass/fail judgment and a continuous scores∈\[0,1\]s\\in\[0,1\]representing partial compliance \([Table2](https://arxiv.org/html/2608.12426#S4.T2); full specifications in[AppendixH](https://arxiv.org/html/2608.12426#A8)\)\. A probe passes only if allkkconstraints pass simultaneously—strict conjunction, no partial credit\.

### 3\.4Evaluated models

We evaluate 15 models across 8 families \([Table1](https://arxiv.org/html/2608.12426#S4.T1)\): GPT \(5\.2, 5\.4 Pro, 5\.5\), Claude \(4\.5 Sonnet, 4\.6 Opus, 4\.7 Opus\), Gemini \(3\.1 Flash\-Lite, 3\.1 Pro Preview\), Llama \(3\.1 70B, 3\.1 405B, 4 Maverick\), Qwen 3 235B Instruct, DeepSeek V4 Pro, Kimi K2\.6, and Grok 4\.1\.

## 4Experiments

Table 1:Evaluated models ranked by sCSR \([Equation1](https://arxiv.org/html/2608.12426#S3.E1)\) across 314,108 satisfiable constraint checks \(temperature=0\{=\}0, greedy decode\)\. Columns: sCSR \(all\-pass probe rate\), mCSR \(per\-constraint rate,[Equation2](https://arxiv.org/html/2608.12426#S3.E2)\), Score \(continuous compliance,[Equation3](https://arxiv.org/html/2608.12426#S3.E3)\),k∗k^\{\*\}\(compositional half\-life: smallestkkwhere sCSR drops below 50%\), Tag \(answer\-tag compliance,[Section3\.3](https://arxiv.org/html/2608.12426#S3.SS3)—low fill indicates chain\-of\-thought leakage into scored content, itself an instruction\-following failure\)\. Row shading:blue==compositionally robust \(k∗≥5k^\{\*\}\\geq 5\),orange==cannot reliably handle even one constraint \(k∗=1k^\{\*\}=1\); the 12 unshaded models cluster atk∗=2−4k^\{\*\}=2\{\-\}4\. Ranking inversions between sCSR and mCSR reveal the compositionality penalty: Gemini Pro ranks \#3 by sCSR but \#11 by mCSR \([AppendixL](https://arxiv.org/html/2608.12426#A12)\)\.#### Phase transition and decay model\.

All 15 models exhibit the same qualitative pattern: high performance atk=1k\{=\}1\(70\.7% aggregate sCSR\), steep decline throughk=4−7k\{=\}4\{\-\}7, then near\-zero sCSR atk≥10k\{\\geq\}10\([AppendixE](https://arxiv.org/html/2608.12426#A5)\)\. This shape is consistent across all 8 families despite different architectures, scales, and training approaches\. The overall mCSR across 314,108 satisfiable constraint checks is 42\.8% \(369,753 total checks including 55,645 on impossible probes; see[AppendixR](https://arxiv.org/html/2608.12426#A18)\)\. GPT\-5\.5 is a notable outlier at 79\.8% mCSR / 64\.2% sCSR—the highest\-performing model by a wide margin—while the remaining 14 models cluster between 15% and 67% mCSR\. The gap between mCSR and sCSR is model\-dependent: Llama 405B shows 46\.4% per\-constraint competence but only 12\.4% probe\-level success \(34\.0pp penalty\), while GPT\-5\.5 sustains 79\.8% to 64\.2% \(15\.6pp\)—a 2\.2×\\timesrange across these two models, widening to 12×\\timesacross the full panel \(Gemini Pro at 2\.7pp to Llama 405B at 34\.0pp\), reflecting differences in how correlated each model’s constraint failures are \([AppendixJ](https://arxiv.org/html/2608.12426#A10)\)\. Probe\-level success becomes essentially impossible past a threshold, even though individual constraints are still satisfied at reasonable rates—this divergence between mCSR and sCSR is the phase transition\. The completekk\-by\-model table appears in[AppendixE](https://arxiv.org/html/2608.12426#A5)\.

#### Difficulty confound control\.

We test whether the phase transition could be an artifact if higher\-kkprobes systematically draw harder constraints rather than imposing genuine compositional burden via three independent controls\. First, intrinsic difficulty—measured as each constraint’sk=1k\{=\}1\-only pass rate to isolate single\-constraint capability—shows no significant variation acrosskkstrata \(Kruskal\-WallisH=6\.23H\{=\}6\.23,p=0\.96p\{=\}0\.96\), and the compositional penalty is identical whether difficulty is measured atk=1k\{=\}1or marginally across allkk\(Δ=0\.0\\Delta\{=\}0\.0pp at everykk\)\. Second, stratifying probes into difficulty terciles within eachkkreveals that easy, medium, and hard probes all converge to the same floor byk=5−6k\{=\}5\{\-\}6—even after excluding the three compositionally immune constraints \(L1, L5, W4\)\. Third, constraint presentation order is randomized within each probe, and position shows negligible correlation with pass rate \(mean\|ρ\|=0\.029\|\\rho\|\{=\}0\.029;[AppendixN](https://arxiv.org/html/2608.12426#A14)\); IRT calibration preserves model rankings exactly \(1PLρ=1\.000\\rho\{=\}1\.000on 2,952 discriminating probes;[AppendixL](https://arxiv.org/html/2608.12426#A12)\)\. This suggests that the transition is driven by how many constraints are imposed, not by which ones or in what order\.

#### Two\-regime decay model\.

If constraint failures are largely uncorrelated \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)\), the aggregate sCSR should followP⁡\(k\)=a⋅rk−1P\(k\)=a\\cdot r^\{k\-1\}\. We test this by fitting onk=1−8k\{=\}1\{\-\}8and evaluating on held\-outk=9−12k\{=\}9\{\-\}12\([Figure1](https://arxiv.org/html/2608.12426#S4.F1)\)\. At the probe level \(sCSR\), the decay is steep:P⁡\(k\)=1\.000⋅e−0\.376​k\+0\.003P\(k\)=1\.000\\cdot e^\{\-0\.376k\}\+0\.003, with sCSR dropping below 4% byk=9k\{=\}9and below 2% byk=11k\{=\}11\. At the per\-constraint level, the aggregate decay ismCSR​\(k\)=72\.0%×0\.922k−1\\text\{mCSR\}\(k\)=72\.0\\%\\times 0\.922^\{k\-1\}\(held\-out MAE=0\.2=0\.2pp\)—each additional constraint reduces the average pass rate to 92\.2% of the previous level\.

The multiplicative model captures the aggregate but masks a per\-model floor: 10 of 15 models plateau above the prediction atk=12k\{=\}12\(mean underprediction−5\.1\-5\.1pp\), sustained by compositionally immune constraints that require no tracking during generation \(L5: 89\.7%, N2: 83\.4%, L2: 80\.1%;[AppendixF](https://arxiv.org/html/2608.12426#A6)\)\. Per\-model decay factors spanr=0\.721r=0\.721\(DeepSeek V4 Pro\) tor=0\.977r=0\.977\(Claude 4\.7 Opus\):*slow\-decay*models \(r\>0\.93r\>0\.93; Claude 4\.7, GPT\-5\.5, Llama family, Claude 4\.5, Flash\-Lite, Qwen, Grok\) sustain 30–53% atk=12k\{=\}12;*medium\-decay*models \(r≈0\.86r\\approx 0\.86; GPT\-5\.2, GPT\-5\.4, Gemini Pro\) follow the multiplicative prediction; and*fast\-decay*models \(r<0\.77r<0\.77; DeepSeek, Kimi\) collapse but stay off zero via immune constraints\. Per\-kkpredictions versus actuals appear in[AppendixF](https://arxiv.org/html/2608.12426#A6)\.

Figure 1:Compositional collapse: mCSR vs\. sCSR with multiplicative fit\.Blue: per\-constraint mCSR—models still get each individual instruction right∼41%\{\\sim\}41\\%of the time atk=8k\{=\}8, declining gently as72\.0%×0\.922k−172\.0\\%\\times 0\.922^\{k\-1\}\(0\.2pp held\-out MAE, dashed teal\)\.Orange: probe\-level sCSR—the chance of satisfying*all*kkconstraints simultaneously, which drops below 4% byk=9k\{=\}9and below 2% byk=11k\{=\}11because it is the product ofkkmarginal rates\. Even atk=8k\{=\}8, models still pass individual constraints 40\.7% of the time yet satisfy all eight simultaneously only 5\.7% of the time \(35pp gap\)—the divergence between the two curves is the phase transition\.Gray: per\-model mCSR trajectories \(n=15n\{=\}15\), revealing model\-dependent floors—10/15 models plateau above the multiplicative prediction atk=12k\{=\}12\.
#### Co\-failure structure\.

Prior work on compositional generalization implicitly assumes that constraint*interactions*drive compositional difficulty—that specific pairs conflict, interfere, or compete for shared resources[44](https://arxiv.org/html/2608.12426#bib.bib6);[29](https://arxiv.org/html/2608.12426#bib.bib43)\. We test this directly by computing pairwiseφ\\varphicoefficients across 601 constraint pairs \(29 of\(362\)=630\\binom\{36\}\{2\}\{=\}630excluded for insufficient co\-occurrence\), conditioned onkkto avoid the confound that everything fails at highkk\. Eachφ\\varphiis computed per\-model first \(to prevent weak models from inflating co\-failure rates\), then aggregated via median across models\.

Pairwise correlations are weak: the meanφ=\+0\.067\\varphi=\+0\.067and 322 pairs \(54%\) fall within\|φ\|≤0\.05\|\\varphi\|\\leq 0\.05\. Only 1 pair shows negativeφ\\varphi\(O4−\{\-\}R2,φ=−0\.055\\varphi\{=\}\{\-\}0\.055\), confirming that constraint selection cannot mitigate the transition—the only lever is per\-constraint reliability\.

The elevatedφ\\varphipairs share*output features*rather than exhibiting pairwise interference\. The top three—F1−\{\-\}F2 \(φ=0\.517\\varphi\{=\}0\.517\), L4−\{\-\}S2 \(φ=0\.501\\varphi\{=\}0\.501\), N1−\{\-\}N3 \(φ=0\.490\\varphi\{=\}0\.490\)—all depend on the same structural output: document format \(F1/F2\), sentence count \(L4/S2\), or number count \(N1/N3\)\. When a model produces the wrong number of sentences, every sentence\-dependent constraint fails simultaneously—not because L4 interferes with S2, but because both read from the same output feature\. Within\-dimension pairs show marginally higher coupling \(mean\|φ\|=0\.087\|\\varphi\|\{=\}0\.087\) than across\-dimension pairs \(0\.0670\.067\), but this difference does not reach significance \(Mann\-Whitneyp=0\.227p\{=\}0\.227\) and is driven by the numerical–numerical \(φ¯=0\.275\\bar\{\\varphi\}\{=\}0\.275\) and format–format \(φ¯=0\.241\\bar\{\\varphi\}\{=\}0\.241\) clusters\. Grouping constraints by shared output feature rather than by dimension confirms that within\-cluster\|φ\|\|\\varphi\|\(0\.1130\.113\) exceeds across\-cluster\|φ\|\|\\varphi\|\(0\.0580\.058\) atp<10−6p<10^\{\-6\}: the co\-failure structure reflects output\-feature coupling, not cognitive interference \([AppendixM](https://arxiv.org/html/2608.12426#A13)\)\. Constraint failures compound near\-multiplicatively—the residual correlations are mechanical, average out across the diverse constraint pool, and do not degrade the decay model’s 0\.2pp predictive accuracy\.

Figure 2:Compositional synergy decay\.Top: actual sCSR \(%\)\.Bottom\(mirrored\): predicted sCSR under independence\. Shaded gap==synergy \(actual−\-predicted\)\. Synergy drops from\+26\+26pp atk=1k\{=\}1to∼1\{\\sim\}1pp atk=12k\{=\}12: models outperform the independence prediction at lowkkbut converge to it under heavy compositional load\. Dashed tercile curves stratify probes by per\-constraint difficulty \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px5)\)\.
#### Ruling out compliance mode\.

If models entered a focused “compliance mode” when facing easy constraints—concentrating resources to satisfy allkksimultaneously—easy probes would show artificially high sCSR, inflating the synergy beyond what independence predicts\. We test this by stratifying probes at eachkkinto terciles ranked by mean per\-constraint difficulty: easy\-tercile probes \(composed of high\-pass\-rate constraints such as L5, L2, N2\) start at 94% sCSR atk=1k\{=\}1; hard\-tercile probes \(S4, M2, S1, W5\) start at 44%\. For each tercile, we compute synergy—actual sCSR minus the product of the constituent constraints’ marginal pass rates—which measures how much better models perform than the independence baseline predicts\. If compliance mode were real, easy probes would show disproportionately high synergy from behavioral focus\. The opposite holds: easy\-anchor probes show*more*synergy \(\+33\+33pp\) than hard\-anchor probes \(\+18\+18pp\), but this is an arithmetic consequence of higher marginals producing a more pessimistic product\-of\-marginals baseline, not a behavioral signal\. Both terciles converge to∼1\.5%\{\\sim\}1\.5\\%sCSR byk≈9−12k\{\\approx\}9\{\-\}12regardless of starting point—initial difficulty is irrelevant at saturation \([Figure2](https://arxiv.org/html/2608.12426#S4.F2)\)\. The synergy itself decays concavely, not linearly \(superlinearity improvement: 84% probe\-level, 99% constraint\-level\), confirming that the transition is driven by compositional load, not by constraint\-specific difficulty or model strategy\.

#### Degradation hierarchy\.

If constraint failures are largely uncorrelated and the transition is purely combinatorial, does every constraint degrade at the same rate? We compute per\-constraint retentionret​\(k\)=mCSR​\(k\)/mCSR​\(k=1\)\\text\{ret\}\(k\)=\\text\{mCSR\}\(k\)/\\text\{mCSR\}\(k\{=\}1\)—the fraction of baseline performance preserved at eachkk—and fit a linear degradation slope\. The compositional half\-lifet0\.5t\_\{0\.5\}is the smallestkkat which retention drops below 50%; constraints that never reach this threshold withink=1−12k\{=\}1\{\-\}12are markedt0\.5\>12t\_\{0\.5\}\{\>\}12in[Table2](https://arxiv.org/html/2608.12426#S4.T2)\.

Structural and ordering constraints degrade2\.0×2\.0\{\\times\}faster than lexical constraints \(retention\-normalized dimension\-level slopes: structural−0\.073\-0\.073, lexical−0\.036\-0\.036; 95% CI on ratio:\[1\.9,2\.3\]\[1\.9,2\.3\];[Table2](https://arxiv.org/html/2608.12426#S4.T2)\)\. Lexical constraints are the slowest degraders across all 36 constraints and 15 models\.

The dimension hierarchy, however, is an emergent consequence of a more fundamental variable: the comprehension\-maintenance gapΔ\\Delta\(score−\-mCSR\)\. This gap measures the difference between a model’s ability to*understand*a constraint \(score\) and its ability to*maintain*compliance throughout the response \(mCSR\)\. The gap predicts degradation rate withρ=−0\.584\\rho\{=\}\{\-\}0\.584\(p=0\.0002p\{=\}0\.0002, 95% CI:\[−0\.767,−0\.302\]\[\-0\.767,\-0\.302\], bootstrap\), a stronger predictor than baseline difficulty \(ρ=\+0\.566\\rho\{=\}\{\+\}0\.566\) or dimension assignment\. Constraints requiring sustained attention—counting words \(W1,Δ=50\.2\\Delta\{=\}50\.2pp\), tracking letter avoidance \(L1,Δ=48\.6\\Delta\{=\}48\.6pp\), maintaining word length ranges \(W2,Δ=50\.1\\Delta\{=\}50\.1pp\)—degrade fastest\. Binary constraints that require no sustained tracking—including a mandatory word \(L2,Δ=2\.1\\Delta\{=\}2\.1pp\), producing a JSON structure \(R1,Δ=0\.2\\Delta\{=\}0\.2pp\)—are immune\. Composition depletes the same finite resource that sustained constraint maintenance requires\.

Table 2:Constraint profiles \(36 constraints, 8 dimensions ordered fastest→\\toslowest degradation\)\.⋆\\starindicates compositionally immune constraints \(retention\>80%\{\>\}80\\%atk≥8k\{\\geq\}8\)\.t0\.5\>12t\_\{0\.5\}\{\>\}12indicates retention never drops below 50% within the evaluatedk=12k\{=\}12range\. Dimension\-level slopes reported in text are observation\-weighted aggregates; per\-constraint slopes here are constraint\-weighted, so simple averaging will not reproduce the dimension\-level values\. Full specifications in[Table7](https://arxiv.org/html/2608.12426#A8.T7)\.Dim\.Constraint𝒌=𝟏\\boldsymbol\{k\{=\}1\}mCSR𝚫\\boldsymbol\{\\Delta\}Slope𝒕0\.5\\boldsymbol\{t\_\{0\.5\}\}%↑\\uparrow%↑\\uparrowpp↓\\downarrowret\.↑\\uparrowkk↑\\uparrowStructuralS1 Word count3914\+41−\.065\-\.0655S2 Sent\. count10054\+20−\.060\\mathbf\{\-\.060\}8S3 Paragraph count10040\+20−\.065\-\.0655S4 Character count198\+31−\.075\-\.0757S5 Line count8937\+20−\.066\-\.0666S6 Palindromic sent\.4214\+12−\.062\-\.0624OrderingO1 Monotonic length5229\+34−\.062\-\.0627O2 Alphabetical9144\+27−\.060\\mathbf\{\-\.060\}7O3 Alternating length8738\+30−\.067\-\.0675O4 Relaxed growth6430\+14−\.061\-\.0616Word\-levelW1 Unique words4728\+50−\.085\-\.0857W2 Word length range8247\+50−\.069\-\.0698W3 Min words/sent\.9853\+25−\.048\\mathbf\{\-\.048\}6W4 Unique bigrams⋆\\,\\star5152\+40−\.056\-\.056\>\\boldsymbol\{\>\}12W5 Consonant clusters4218\+17−\.059\-\.0597MetaM1 Hidden message3920\+22−\.074\-\.0746M2 Self\-counting2213\+34−\.081\-\.0818M3 Divisibility2422\+36−\.073\\mathbf\{\-\.073\}\>\\boldsymbol\{\>\}12FormatF1 Markdown table9464\+5−\.045\\mathbf\{\-\.045\}12F2 Bullet list9967\+3−\.045\\mathbf\{\-\.045\}12F3 Code blocks9167\+3−\.051\-\.05110F4 Unique closers9959\+31−\.047\-\.0479RelationalR1 Scene graph9368\+0−\.031\\mathbf\{\-\.031\}\>\\boldsymbol\{\>\}12R2 Transitive ordering6536\+22−\.045\-\.0456R3 Logic grid8567\+3−\.037\-\.037\>\\boldsymbol\{\>\}12R4 Sentence bridge8630\+23−\.063\-\.0634NumericalN1 Integer sum4934\+19−\.082\-\.0829N2 No digits9983\+16−\.035\\mathbf\{\-\.035\}\>\\boldsymbol\{\>\}12N3 Distinct numbers5234\+22−\.077\-\.07710N4 Equations8866\+8−\.047\-\.04712LexicalL1 Lipogram⋆\\,\\star4751\+49−\.052\-\.052\>\\boldsymbol\{\>\}12L2 Mandatory words9880\+2−\.032\-\.032\>\\boldsymbol\{\>\}12L3 Sentence initial9348\+17−\.065\-\.0658L4 Acrostic9950\+19−\.058\-\.0586L5 Forbidden word⋆\\,\\star9890\+10−\.011\\mathbf\{\-\.011\}\>\\boldsymbol\{\>\}12L6 Vowel/cons\. ratio8164\+19−\.030\-\.030\>\\boldsymbol\{\>\}12Three constraints have retention\>80%\>80\\%atk≥8k\{\\geq\}8: L1 \(lipogram, 95%\), L5 \(forbidden word, 89%\), and W4 \(no repeated bigrams, 86%\)\. L1’s apparent immunity is a survivorship effect: among genuine content failures \(n=4,022n\{=\}4\{,\}022\), 70\.6% of violations occur in the first 20% of the response \(median position 0\.078\)—models collide with the forbidden letter almost immediately or not at all, a vocabulary collision at generation onset rather than a capacity that erodes under compositional load\. Full failure mode decomposition, per\-model profiles, and control analyses appear in[AppendixG](https://arxiv.org/html/2608.12426#A7)\.

#### Constraint prioritization under impossibility\.

CSE includes 444 impossible probes \(55,645 constraint checks across 15 models\), each with a proof of unsatisfiability \([AppendixR](https://arxiv.org/html/2608.12426#A18)\)\. Zero probe×\\timesmodel pairs pass all constraints on any impossible probe: the deterministic verifiers are airtight\.

When forced to choose which constraint to violate, models reveal deterministic sacrifice priorities \([Figure3](https://arxiv.org/html/2608.12426#S4.F3)\)\. Three rules emerge across all 15 models, unanimous and stable acrosskk\([SectionR\.2](https://arxiv.org/html/2608.12426#A18.SS2)\)\.

Figure 3:Constraint sacrifice under impossibility\. Three deterministic rules hold across all 15 models: concrete inclusion beats abstract avoidance \(L2: 73%, L1: 0%\), prohibition beats requirement \(N2: 91%, N1: 0%\), and natural prose beats imposed structure \(O3: 56%, O4: 0%\)\. The hierarchy inversion \(L1 sacrificed for R2\) suggests models preserve whichever constraint required greater planning investment\. Full analysis in[SectionR\.2](https://arxiv.org/html/2608.12426#A18.SS2)\.

## 5Interventions

The multiplicative structure makes differential predictions about what should help\. If the collapse is set by per\-constraint reliability rather than by disorganised generation, then restructuring the prompt should not shift it, while supplying additional attempts should buy a bounded improvement—bounded because, were failures on a given probe independent across resamples,NNretries would recover1−\(1−p\)N1\-\(1\-p\)^\{N\}\. We test three interventions, each on three models:pre\-generation planning\(a plan\-first scaffold prepended to the prompt, run against a token\-matched no\-scaffold control\),post\-hoc self\-correction\(a second pass that re\-prompts the model with its own output to check and revise\), andbest\-of\-5 retries\(five independent generations, the probe passing if any one satisfies all constraints\)\. Throughout we report the*compositional half\-life*k∗k^\{\*\}: the smallestkkat which probe\-level success \(sCSR\) falls below 50%\.[Table3](https://arxiv.org/html/2608.12426#S5.T3)gives the per\-model results\.

Table 3:Effect of each intervention on the compositional half\-lifek∗k^\{\*\}\. Planning compares scaffold against a token\-matched control; self\-correction and best\-of\-5 compare against the single\-pass baseline\. Effect column: change in per\-constraint decay factorrrfor planning, change in sCSR for self\-correction, and peak observed sCSR lift against the lift that independent resampling predicts for best\-of\-5\.#### Planning does nothing; output budget does\.

Aggregate per\-constraint decay is unchanged—control93\.0%×0\.948k−193\.0\\%\\times 0\.948^\{k\-1\}against scaffold90\.2%×0\.952k−190\.2\\%\\times 0\.952^\{k\-1\}—andΔ​k∗\\Delta k^\{\*\}is inconsistent in sign, with the strongest model regressing\. Scaffold compliance was 100%, 100%, and 56%, so the null is not models ignoring the instruction\. The output budget, by contrast, moves the threshold on its own: DeepSeek’s decay factor rises from0\.7300\.730to0\.8770\.877andk∗k^\{\*\}from 3 to 5 between the baseline and the token\-matched control, with no scaffold involved\.

#### Retries and correction help, and then plateau\.

Both recover one to two constraints, and self\-correction returns most where per\-constraint reliability is worst—the ordering the mechanism predicts, since a correction pass rescues borderline probes\. The instructive number is the last column of[Table3](https://arxiv.org/html/2608.12426#S5.T3): best\-of\-5 delivers roughly two\-fifths of the lift that independent resampling predicts, despite draws being near\-fully distinct \(4\.98–4\.99 of 5\)\. The model repeats the same failure across draws\.

#### Two senses of independence\.

This bounded ceiling does not contradict[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)\. Independence there holds*across constraints within a response*, which is what makes sCSR multiplicative\. Best\-of\-NNprobes independence*across resamples of one probe*, which is where the correlation lies\. Both hold: which constraints fail within a response is close to independent, while whether a given probe fails is largely deterministic across draws\.

#### What moves the threshold\.

Adding attempts or a correction pass buys one to two constraints; restructuring the prompt buys none; none removes the collapse\. Per\-constraint reliability remains the operative lever, with retries, a correction pass, and an adequate output budget as deployment\-time palliatives\. Each intervention was run on three models rather than all 15; GPT\-5\.5 rejects the temperature parameter and is therefore absent from the retry experiment; and the self\-correction comparison is directional rather than strictly paired, as a small number of second\-pass generations failed\.

## 6Discussion

#### A combinatorial consequence\.

The decay does not require constraint interference to arise—it follows from a simple arithmetic fact: the product of many numbers slightly less than 1\.0 converges to zero\. Unlikekk\-SAT phase transitions[30](https://arxiv.org/html/2608.12426#bib.bib15);[31](https://arxiv.org/html/2608.12426#bib.bib14), our decay requires no interaction structure; it is pure multiplicative accumulation\. The primary lever to shift the transition rightward is improving per\-constraint reliability; there is no shortcut through constraint selection\. Inference\-time interventions such as constrained decoding[27](https://arxiv.org/html/2608.12426#bib.bib34);[26](https://arxiv.org/html/2608.12426#bib.bib35), attention steering[29](https://arxiv.org/html/2608.12426#bib.bib43), and structured generation frameworks[45](https://arxiv.org/html/2608.12426#bib.bib52);[3](https://arxiv.org/html/2608.12426#bib.bib48)can enforce some constraints externally, potentially shifting what counts as the model’s “native” capability vs\. the system’s\. Whether such scaffolding delays the onset or merely raises the floor—and whether chain\-of\-thought or decomposition prompting would let models manage more constraints by planning explicitly—remains open\. CSE deliberately evaluates raw generation without scaffolding to isolate the model’s intrinsic compositional capacity; measuring the effect of scaffolding is a natural follow\-up\. The difficulty\-stratification analysis \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px2)\) provides indirect evidence that the transition is not driven by constraint familiarity: probes composed entirely of easy constraints—those models handle at 70–85% in isolation—show the same qualitative collapse atk=5−6k\{=\}5\{\-\}6as probes with hard constraints, suggesting the bottleneck is compositional capacity rather than per\-constraint knowledge\.

#### Post\-training shapes failure mode, not compositional capacity\.

Refusal strategies vary dramatically across model families: GPT\-5\.2 and GPT\-5\.4 refuse∼50%\{\\sim\}50\\%of satisfiable probes they fail on, while Llama, Grok, and Flash\-Lite refuse<1%\{<\}1\\%\. Within the GPT family, refusal decreases as capability improves \(GPT\-5\.5: 17%\), suggesting that stronger models learn to distinguish genuine impossibility from difficulty\. This pattern is consistent with RLHF\-trained refusal calibration[33](https://arxiv.org/html/2608.12426#bib.bib33)rather than a universal difficulty signal\. Post\-training also shapes tag compliance: reasoning\-heavy models \(Claude 4\.7, DeepSeek\) spend tokens on chain\-of\-thought that either exhausts the output budget or bypasses answer tags, suppressing measured performance\. These are instruction\-following failures in their own right, but they reflect alignment training choices rather than intrinsic compositional limits—the same model with a larger token budget \(Claude 4\.7 at 16,384 vs 4,096 tokens\) jumps from rank \#10 to \#2 without any change to its compositional capacity\.

#### Deployment implications\.

Production systems assuming linear degradation will experience unexpected failures: a system reliable with 3 constraints may fail significantly with 5\. Per\-model decay factors \(r=0\.721r=0\.721to0\.9770\.977\) enable estimation of the maximumkkat which a model meets a target compliance rate—for 90%, most models handle at most 1–2 constraints; the best \(GPT\-5\.5,k∗=7k^\{\*\}\{=\}7\) is limited to∼6\{\\sim\}6\. Continuous compliance metrics overestimate true satisfaction by up to 50pp near the transition boundary \(W1: 78\.6% score vs\. 28\.4% mCSR;[Equation3](https://arxiv.org/html/2608.12426#S3.E3)\); strict binary verification is essential\.

#### Scale does not predict compositional performance\.

Counter to typical scaling expectations, we observe multiple ranking inversions: Gemini Pro ranks \#2 atk=1k\{=\}1\(90\.5% single\-constraint pass rate\) but \#11 overall \(42\.1% mCSR\); Flash\-Lite ranks \#6 atk=1k\{=\}1\(78\.1%\) but \#3 overall \(56\.6%\)\. Single\-constraint competence does not predict compositional robustness—the models that understand constraints best in isolation are not necessarily the ones that compose them most reliably\.

#### A capacity parallel\.

The 5–6 constraint ceiling is strikingly close to human working memory limits: Miller’s7±27\{\\pm\}2chunks[32](https://arxiv.org/html/2608.12426#bib.bib51)and Cowan’s more conservative4±14\{\\pm\}1[7](https://arxiv.org/html/2608.12426#bib.bib49)\. We do not claim a shared mechanism—LLMs do not have a working memory bottleneck in the cognitive sense—but the convergence suggests that compositional constraint satisfaction may face a capacity limit that is loosely independent of the substrate\. Whether this reflects a fundamental information\-theoretic bound on simultaneous constraint tracking or a coincidence of current architectures is an open question\.

## 7Conclusion

CSE reveals that compositional instruction following degrades via a predictable multiplicative pattern, with reliable performance breaking down beyond 5–6 simultaneous constraints across all 15 models tested\. A depth\-of\-processing hierarchy governs which constraints fail first—structural constraints degrade2×2\{\\times\}faster than lexical ones—driven by processing demands rather than intrinsic difficulty\. Three directions would advance this work: measuring whether inference\-time scaffolding shifts the transition rightward, testing invariance to constraint novelty, and measuring how constraint load degrades primary\-task accuracy\.

## Limitations

#### Conjunctive composition only\.

CSE evaluates pure conjunction: allkkconstraints must hold simultaneously\. Other composition structures—sequential chaining, conditional branching, nested dependencies[44](https://arxiv.org/html/2608.12426#bib.bib6)—may exhibit different decay profiles and are left to future work\.

#### Verifiability selection bias\.

The 30 constraint types are chosen to be deterministically verifiable, which systematically excludes semantic, pragmatic, and discourse\-level constraints \(coherence, informativeness, tone consistency, factual accuracy\)\. This is the design tradeoff that enables CSE’s measurement invariance—the same compositional degradation that affects model generation would compromise an LLM\-based judge in exactly the regime we study\.

#### Ecological validity\.

Thek=12k\{=\}12regime is deliberately a stress test, not a simulation of typical deployment\. Real\-world systems typically combine 2–6 high\-level constraints \(format, length, tone, content domain, safety boundaries\) plus implicit norms\. The practical implication of CSE’s findings lies atk=5−6k\{=\}5\{\-\}6—where the transition occurs—which falls within the range of realistic deployment constraint counts\.

#### Adversarial implications\.

The phase transition reveals a potential attack vector: an adversary aware that models reliably fail beyondk=5−6k\{=\}5\{\-\}6constraints could craft prompts with many simultaneous constraints to overwhelm safety\-related constraints, exploiting the finding that relational and structural constraints are dropped before surface\-level ones\. Characterizing this vulnerability is a prerequisite for defending against it\.

#### Specification gap exploitation\.

A systematic audit of all passing responses identifies four specification gaps that models exploit under compositional pressure \([Figure4](https://arxiv.org/html/2608.12426#Sx1.F4)\)\. Empty code blocks \(F3, 82\.4% of passes—models emit delimiter pairs with no content, ranging from 50\.7% for Claude 4\.5 to 100% for GPT\-5\.2/5\.5\) and palindromic sentence copying \(S6, 19\.4%—models duplicate sentences reversed to guarantee matched word counts, with strong model separation: Llama 405B at 83\.3% vs\. Claude models at≤8%\{\\leq\}8\\%\) are the highest\-prevalence patterns\. Two\-sentence trivial satisfaction of contradictory ordering constraints \(O1\+O3, 6 cases\) and unicode escape substitution to bypass word uniqueness \(W1, 10 cases\) are quantitatively negligible\. None of these gaps compromise the benchmark’s core findings: the difficulty\-stratified analysis \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px2)\) confirms that all terciles converge to the same floor, including after excluding compositionally immune constraints, and the same robustness extends to exploitable ones—no single constraint drives any result\. The gaps reveal that models under compositional load adopt mechanical shortcuts over genuine compliance—itself a finding about resource allocation under pressure—and all four are straightforward to close in future iterations\.

Figure 4:Three specification gap exploitations\.\(a\)GPT\-5\.5 on CSE\-E\-k6\-4693: satisfies F3 \(2 code blocks\) with four empty‘‘‘delimiters and W1 \(unique words\) via unicode escapes \(\\u0070etfor “pet”\)\.\(b\)Llama 405B on CSE\-E\-k1\-0003: satisfies S6 \(palindromic word counts\) by copying sentences 1–3 as sentences 5–7 reversed, guaranteeing identical word counts without generating original content\.\(c\)DeepSeek V4 Pro on CSE\-IMP\-A4\-02: satisfies both O1 \(monotonic\) and O3 \(alternating\) with exactly 2 sentences—a short sentence followed by a long one—exploiting the absence of a minimum sentence count requirement\.

## References

- Anet al\.\(2025\)K\. An, L\. Sheng, G\. Cui, S\. Si, N\. Ding, Y\. Cheng, and B\. ChangULTRA\-IF: advancing instruction following from the wild\.arXiv preprint arXiv:2502\.04153\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Balkıret al\.\(2026\)E\. Balkır, A\. Pernthaller, M\. Basaldella, J\. Hernández\-Orallo, and N\. CollierConfident rankings with fewer items: adaptive LLM evaluation with continuous scores\.arXiv preprint arXiv:2601\.13885\.Cited by:[Appendix L](https://arxiv.org/html/2608.12426#A12.SS0.SSS0.Px3.p1.1),[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px1.p1.1)\.
- Beurer\-Kellneret al\.\(2023\)L\. Beurer\-Kellner, M\. Fischer, and M\. VechevPrompting is programming: a query language for large language models\.InProceedings of the 44th ACM SIGPLAN International Conference on Programming Language Design and Implementation \(PLDI\),External Links:[Document](https://dx.doi.org/10.1145/3591300)Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2022\)H\. Chen, H\. Li, D\. Chen, and K\. NarasimhanControllable text generation with language constraints\.arXiv preprint arXiv:2212\.10466\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Chenget al\.\(2025\)J\. Cheng, X\. Liu, C\. Wang, X\. Gu, Y\. Lu, D\. Zhang, Y\. Dong, J\. Tang, H\. Wang, and M\. HuangSPaR: self\-play with tree\-search refinement to improve instruction\-following in large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Cohenet al\.\(2024\)V\. Cohen, G\. Nangue Tasse, N\. Gopalan, S\. James, M\. Gombolay, R\. Mooney, and B\. RosmanCompositional instruction following with language models and reinforcement learning\.Transactions on Machine Learning Research\.Note:arXiv:2501\.12539Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px4.p1.1)\.
- Cowan \(2001\)N\. CowanThe magical number 4 in short\-term memory: a reconsideration of mental storage capacity\.Behavioral and Brain Sciences24\(1\),pp\. 87–114\.External Links:[Document](https://dx.doi.org/10.1017/S0140525X01003922)Cited by:[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px5.p1.1)\.
- Donget al\.\(2025\)G\. Dong, K\. Lu, C\. Li, T\. Xia, B\. Yu, C\. Zhou, and J\. ZhouSelf\-play with execution feedback: improving instruction\-following capabilities of large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Dziriet al\.\(2023\)N\. Dziri, X\. Lu, M\. Sclar, X\. L\. Li, L\. J\. Jiang, B\. Y\. Lin, P\. West, C\. Bhagavatula, R\. Le Bras, J\. D\. Hwang, S\. Sanyal, S\. Welleck, X\. Ren, A\. Ettinger, Z\. Harchaoui, and Y\. ChoiFaith and fate: limits of transformers on compositionality\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2608.12426#S1.p2.1)\.
- Faveroet al\.\(2025\)A\. Favero, A\. Sclocchi, F\. Cagnetta, P\. Frossard, and M\. WyartHow compositional generalization and creativity improve as diffusion models are trained\.InForty\-second International Conference on Machine Learning,Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px2.p1.1)\.
- Ferrazet al\.\(2024\)T\. P\. Ferraz, K\. Mehta, Y\. Lin, H\. Chang, S\. Oraby, S\. Liu, V\. Subramanian, T\. Chung, M\. Bansal, and N\. PengLLM self\-correction with DeCRIM: decompose, critique, and refine for enhanced following of instructions with multiple constraints\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Ganet al\.\(2025\)H\. Gan, B\. Tinaz, M\. S\. Sepehri, Z\. Fabian, and M\. SoltanolkotabiConceptMix\+\+: leveling the playing field in text\-to\-image benchmarking via iterative prompt optimization\.arXiv preprint arXiv:2507\.03275\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Guoet al\.\(2025\)Z\. Guo, W\. Liu, M\. Xie, J\. Xu, Z\. Huang, M\. Tian, J\. Xu, Y\. Shen, Q\. Qian, M\. Wu, X\. Wang, C\. Lv, H\. Wang, H\. Yao, X\. Zheng, and X\. HuangRECAST: expanding the boundaries of LLMs’ complex instruction following with multi\-constraint data\.arXiv preprint arXiv:2505\.19030\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1),[§2](https://arxiv.org/html/2608.12426#S2.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.Training compute\-optimal large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30016–30030\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px2.p1.1)\.
- Hsiehet al\.\(2023\)C\. Hsieh, J\. Zhang, Z\. Ma, A\. Kembhavi, and R\. KrishnaSugarCrepe: fixing hackable benchmarks for vision\-language compositionality\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Huanget al\.\(2024\)I\. Huang, W\. Lin, M\. J\. Mirza, J\. A\. Hansen, S\. Doveh, V\. I\. Butoi, R\. Herzig, A\. Arbelle, H\. Kuehne, T\. Darrell, C\. Gan, A\. Oliva, R\. Feris, and L\. KarlinskyConMe: rethinking evaluation of compositional reasoning for modern VLMs\.arXiv preprint arXiv:2406\.08164\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Huanget al\.\(2023\)K\. Huang, C\. Duan, K\. Sun, E\. Xie, Z\. Li, and X\. LiuT2I\-CompBench\+\+: an enhanced and comprehensive benchmark for compositional text\-to\-image generation\.arXiv preprint arXiv:2307\.06350\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Jianget al\.\(2024a\)Q\. Jiang, J\. Zeng, Q\. He, J\. Liang, and Y\. XiaoFrom complex to simple: enhancing multi\-constraint complex instruction following ability of large language models\.InFindings of the Association for Computational Linguistics: EMNLP,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Jianget al\.\(2024b\)Y\. Jiang, Y\. Wang, X\. Zeng, W\. Zhong, L\. Li, F\. Mi, L\. Shang, X\. Jiang, Q\. Liu, and W\. WangFollowBench: a multi\-level fine\-grained constraints following benchmark for large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1),[§2](https://arxiv.org/html/2608.12426#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.12426#S3.SS1.p3.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px2.p1.1)\.
- Lake and Baroni \(2018\)B\. M\. Lake and M\. BaroniGeneralization without systematicity: on the compositional skills of sequence\-to\-sequence recurrent networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.12426#S1.p2.1)\.
- Liet al\.\(2024\)B\. Li, Z\. Lin, D\. Pathak, J\. Li, Y\. Fei, K\. Wu, T\. Ling, X\. Xia, P\. Zhang, G\. Neubig, and D\. RamananGenAI\-Bench: evaluating and improving compositional text\-to\-visual generation\.arXiv preprint arXiv:2406\.13743\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Liet al\.\(2025\)P\. Li, X\. Tang, S\. Chen, Y\. Cheng, R\. Metoyer, T\. Hua, and N\. V\. ChawlaAdaptive testing for LLM evaluation: a psychometric alternative to static benchmarks\.arXiv preprint arXiv:2511\.04689\.Cited by:[Appendix L](https://arxiv.org/html/2608.12426#A12.SS0.SSS0.Px3.p1.1)\.
- Lianget al\.\(2025\)Z\. Liang, Z\. Hou, and X\. WangFine\-grained constraint generation\-verification for improved instruction\-following\.InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics \(GEM2\),Vienna, Austria,pp\. 862–879\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)W\. Liu, Y\. He, Y\. Li, H\. Huang, C\. Hu, J\. Liu, S\. Li, W\. Su, and B\. ZhengAIR: complex instruction generation via automatic iterative refinement\.arXiv preprint arXiv:2502\.17787\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Luet al\.\(2022\)X\. Lu, S\. Welleck, P\. West, L\. Jiang, J\. Kasai, D\. Khashabi, R\. Le Bras, L\. Qin, Y\. Yu, R\. Zellers, N\. A\. Smith, and Y\. ChoiNeuroLogic A\*esque decoding: constrained text generation with lookahead heuristics\.InProceedings of NAACL,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2021\)X\. Lu, P\. West, R\. Zellers, R\. Le Bras, C\. Bhagavatula, and Y\. ChoiNeuroLogic decoding: \(un\)supervised neural text generation with predicate logic constraints\.InProceedings of NAACL,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2025\)Y\. Lu, Z\. Bai, C\. Yuan, H\. Jiang, and X\. WangEnhancing complex instruction following for large language models with mixture\-of\-contexts fine\-tuning\.arXiv preprint arXiv:2505\.11922\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Lvet al\.\(2026\)F\. Lv, Y\. Qin, R\. Hong, J\. Liang, J\. Wu, K\. Li, X\. Sun, and C\. ZhangAttend to the active: structure\-aware dynamic attention in LLMs for compositional instruction following\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Mertenset al\.\(2006\)S\. Mertens, M\. Mézard, and R\. ZecchinaThreshold values of random K\-SAT from the cavity method\.Random Structures & Algorithms28\(3\),pp\. 340–373\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Mézard and Montanari \(2009\)M\. Mézard and A\. MontanariInformation, physics, and computation\.Oxford University Press\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Miller \(1956\)G\. A\. MillerThe magical number seven, plus or minus two: some limits on our capacity for processing information\.Psychological Review63\(2\),pp\. 81–97\.External Links:[Document](https://dx.doi.org/10.1037/h0043158)Cited by:[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px5.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px2.p1.1)\.
- Presset al\.\(2023\)O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. LewisMeasuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP,Cited by:[§1](https://arxiv.org/html/2608.12426#S1.p2.1)\.
- Pyatkinet al\.\(2025\)V\. Pyatkin, S\. Malik, V\. Graf, H\. Ivison, S\. Huang, P\. Dasigi, N\. Lambert, and H\. HajishirziGeneralizing verifiable instruction following\.arXiv preprint arXiv:2507\.02833\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px2.p1.1)\.
- Qiet al\.\(2024\)Y\. Qi, H\. Peng, X\. Wang, B\. Xu, L\. Hou, and J\. LiConstraint back\-translation improves complex instruction following of large language models\.arXiv preprint arXiv:2410\.24175\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Qinet al\.\(2024\)Y\. Qin, K\. Song, Y\. Hu, W\. Yao, S\. Cho, X\. Wang, X\. Wu, F\. Liu, P\. Liu, and D\. YuInFoBench: evaluating instruction following ability in large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1)\.
- Renet al\.\(2025\)H\. Ren, Y\. Liang, B\. Su, C\. Sun, H\. Lu, K\. Zhang, and C\. WeiLexInstructEval: lexical instruction following evaluation for large language models\.arXiv preprint arXiv:2511\.17561\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Sakaiet al\.\(2025\)Y\. Sakai, H\. Kamigaito, and T\. WatanabeRevisiting compositional generalization capability of large language models considering instruction following ability\.arXiv preprint arXiv:2506\.15629\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px1.p1.1)\.
- Sunet al\.\(2024a\)H\. Sun, L\. Liu, J\. Li, F\. Wang, B\. Dong, R\. Lin, and R\. HuangConifer: improving complex constrained instruction\-following ability of large language models\.arXiv preprint arXiv:2404\.02823\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Sunet al\.\(2024b\)K\. Sun, K\. Huang, X\. Liu, Y\. Wu, Z\. Xu, Z\. Li, and X\. LiuT2V\-CompBench: a comprehensive benchmark for compositional text\-to\-video generation\.arXiv preprint arXiv:2407\.14505\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Thrushet al\.\(2022\)T\. Thrush, R\. Jiang, M\. Bartolo, A\. Singh, A\. Williams, D\. Kiela, and C\. RossWinoground: probing vision and language models for visio\-linguistic compositionality\.InProceedings of CVPR,Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler,et al\.Emergent abilities of large language models\.Transactions on Machine Learning Research\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px2.p1.1)\.
- Wenet al\.\(2024\)B\. Wen, P\. Ke, X\. Gu, L\. Wu, H\. Huang, J\. Zhou, W\. Li, B\. Hu, W\. Gao, J\. Xu,et al\.ComplexBench: benchmarking complex instruction following with multi\-constraint composition\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px2.p1.1),[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4.p1.1),[Conjunctive composition only\.](https://arxiv.org/html/2608.12426#Sx1.SS0.SSS0.Px1.p1.1)\.
- Willard and Louf \(2023\)B\. T\. Willard and R\. LoufEfficient guided generation for large language models\.arXiv preprint arXiv:2307\.09702\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2307.09702)Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.12426#S6.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2024\)X\. Wu, D\. Yu, H\. Huang, O\. Russakovsky, and S\. AroraConceptMix: a compositional image generation benchmark with controllable difficulty\.arXiv preprint arXiv:2408\.14339\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1),[§2](https://arxiv.org/html/2608.12426#S2.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, H\. Chen, A\. W\. Hanjie, R\. Yang, and K\. R\. NarasimhanCOLLIE: systematic construction of constrained text generation tasks\.InProceedings of ICLR,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1),[§2](https://arxiv.org/html/2608.12426#S2.p1.1)\.
- Yeet al\.\(2025\)J\. Ye, C\. Huang, Z\. Chen, W\. Fu, C\. Yang, L\. Yang, Y\. Wu, P\. Wang, M\. Zhou, X\. Yang, T\. Gui, Q\. Zhang, Z\. Shi, J\. Fan, and X\. HuangA multi\-dimensional constraint framework for evaluating and improving instruction following in large language models\.arXiv preprint arXiv:2505\.07591\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px3.p1.1)\.
- Yeet al\.\(2026\)J\. Ye, G\. Zhang, W\. Fu, T\. Gui, Q\. Zhang, and X\. HuangCCTU: a benchmark for tool use under complex constraints\.arXiv preprint arXiv:2603\.15309\.Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1),[§2](https://arxiv.org/html/2608.12426#S2.p1.1)\.
- Zenget al\.\(2025\)J\. Zeng, Q\. He, Q\. Ren, J\. Liang, Y\. Xiao, W\. Zhou, Z\. Sun, and F\. YuOrder matters: investigate the position bias in multi\-constraint instruction following\.arXiv preprint arXiv:2502\.17204\.Cited by:[Appendix N](https://arxiv.org/html/2608.12426#A14.p1.1),[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px1.p1.1)\.
- Zhonget al\.\(2025\)Z\. Zhong, A\. Raghunathan, and N\. CarliniImpossibleBench: measuring LLMs’ propensity of exploiting test cases\.arXiv preprint arXiv:2510\.20270\.Cited by:[§B\.2](https://arxiv.org/html/2608.12426#A2.SS2.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.1](https://arxiv.org/html/2608.12426#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.12426#S1.p2.1),[§2](https://arxiv.org/html/2608.12426#S2.p1.1)\.

## Appendix AReproducibility

All probes, deterministic verifiers, raw model outputs, and evaluation code will be released upon publication\. The total evaluation comprises 67,905 API calls \(15 models×\\times4,527 probes\) at temperature=0\{=\}0with greedy decoding\. Four of 15 evaluated models are open\-weight and fully reproducible: Llama 3\.1 70B and 405B Instruct, Llama 4 Maverick, and Qwen 3 235B Instruct\. Proprietary model outputs are included in the release for verification\. The probe composer, compatibility checker, calibrated parameter sampling distributions, and all 36 verification functions are included in the released code\.

## Appendix BRelated Work \(Expanded\)

This section provides an expanded version of the related work discussion summarized in[Section1](https://arxiv.org/html/2608.12426#S1)\.

### B\.1Compositional Instruction Following

#### Evaluation benchmarks\.

IFEval[52](https://arxiv.org/html/2608.12426#bib.bib1)established the paradigm of verifiable instruction following with 25 constraint types and deterministic programmatic checks, but probes typically containk≤3k\\leq 3constraints—insufficient to observe compositional degradation\. A model scoring 80%\+ on IFEval could in principle collapse atk=5−8k\{=\}5\{\-\}8on CSE, exposing degradation invisible to proficiency\-oriented evaluation\. Adaptive IRT methods[2](https://arxiv.org/html/2608.12426#bib.bib26)extend the evaluation paradigm by enabling confident model rankings with fewer test items, though they require more examinees than CSE’s 15\-model panel\. InfoBench[37](https://arxiv.org/html/2608.12426#bib.bib4)decomposes each instruction into atomic verifiable criteria and evaluates per\-criterion satisfaction via its Decomposed Requirements Following Rate \(DRFR\)\. GPT\-4 achieves∼89%\{\\sim\}89\\%DRFR overall, yet more than 10% of requirements remain unfulfilled even for this strongest model, and no model fully satisfies number or linguistic constraints\. Performance is highest on content and style, intermediate on format, and lowest on number and linguistic dimensions—a constraint\-type hierarchy that foreshadows CSE’s depth\-of\-processing finding, where structural constraints degrade2\.0×2\.0\{\\times\}faster than lexical ones\. InfoBench operates at the single\-instruction level; CSE extends this analysis systematically acrossk=1k\{=\}1to1212, revealing how per\-criterion gaps compound into the phase transition\. FollowBench[19](https://arxiv.org/html/2608.12426#bib.bib2)scales tok=5k\{=\}5and introduces the Consistent Satisfaction Level \(CSL\) metric for tracking partial compliance across difficulty levels, but relies on GPT\-4 as a judge for several constraint types, introducing evaluation variance\. CSL measures which satisfaction level a model can sustain, which is conceptually adjacent to CSE’skk\-curve—but CSE isolates where the transition occurs rather than tracking cumulative levels, and uses fully deterministic verification throughout\. COLLIE[47](https://arxiv.org/html/2608.12426#bib.bib3)is the closest prior work: it pioneered grammar\-based constraint specification with formal CFG definitions and corpus\-grounded extraction ensuring natural solutions exist, and demonstrated that compositional constraints are harder than individual ones—a finding we confirm and extend\. Three key differences: \(1\) COLLIE uses fixed constraint compositions while CSE systematically varieskkfrom 1–12, enabling the decay characterization that is this paper’s central contribution; \(2\) COLLIE reports that composition is hard but does not quantify*where*capability breaks or what functional form the degradation takes; \(3\) CSE’s dual scoring \(binary \+ continuous\) reveals gaps between partial compliance and strict satisfaction that are absent from COLLIE’s binary\-only evaluation\. COLLIE is stronger in formal rigor \(CFG grammar\) and in its pass@kkevaluation design; CSE contributes the systematic scaling analysis and the mechanistic decomposition of why composition fails\.

#### Complex constraint benchmarks\.

RECAST[13](https://arxiv.org/html/2608.12426#bib.bib5)pushes constraint counts tok=13\+k\{=\}13\{\+\}with 19 constraint types and proposes RLVC training with constraint\-specific reward signals, but does not report per\-kkperformance trajectories, precluding decay analysis\. ComplexBench[44](https://arxiv.org/html/2608.12426#bib.bib6)models constraint composition*structure*—And, Chain, Selection, Nesting—rather than count, and is thus orthogonal to our analysis of how many constraints can be jointly satisfied\. IFBENCH[35](https://arxiv.org/html/2608.12426#bib.bib7)demonstrates that models overfit to IFEval’s 25 known constraint types and proposes IF\-RLVR training for generalization—a different axis \(unseen constraints\) from our focus \(composition scaling\)\. CCTU[49](https://arxiv.org/html/2608.12426#bib.bib10)evaluates constrained tool use, finding that no model exceeds 20% task completion under strict multi\-constraint adherence—a ceiling remarkably close to CSE’s probe\-level floor atk≥9k\{\\geq\}9, despite a completely different task domain, suggesting the phase transition may reflect a general property of LLM compositional processing\.

#### Training methods\.

Conifer[40](https://arxiv.org/html/2608.12426#bib.bib12)generates multi\-level constraints with GPT\-4 and employs curriculum tuning, while AutoIF[8](https://arxiv.org/html/2608.12426#bib.bib13)implements self\-dialogue with execution\-based verification; AIR[25](https://arxiv.org/html/2608.12426#bib.bib30)and MiSO[28](https://arxiv.org/html/2608.12426#bib.bib37)extend automatic refinement and mixture\-of\-contexts fine\-tuning, respectively, to complex instruction generation—all focus on training rather than diagnostic evaluation\. MulDimIF[48](https://arxiv.org/html/2608.12426#bib.bib8)explores how constraint*presentation mode*\(example, listing, incorporation\) affects following and proposes GRPO training; LexInstructEval[38](https://arxiv.org/html/2608.12426#bib.bib9)introduces a formal grammar for lexical constraints with bilingual evaluation, finding that procedural depth degrades performance—complementary to our finding that constraint*count*does so\. UltraIF[1](https://arxiv.org/html/2608.12426#bib.bib29)scales to 100K\+ instructions via automatic constraint decomposition; SPaR[5](https://arxiv.org/html/2608.12426#bib.bib31)uses self\-play with tree\-search refinement for similar scaling\. DeCRIM[11](https://arxiv.org/html/2608.12426#bib.bib36)takes a complementary approach, decomposing multi\-constraint instructions into individual constraints and iteratively refining compliance; related decomposition strategies include Complex\-to\-Simple[18](https://arxiv.org/html/2608.12426#bib.bib38)and constraint back\-translation[36](https://arxiv.org/html/2608.12426#bib.bib28)—approaches whose effectiveness CSE’s independence finding predicts, since failures accumulate per\-constraint rather than through pairwise interference\. FiGV[24](https://arxiv.org/html/2608.12426#bib.bib50)synthesizes constraint\-augmented training data with hybrid LLM/function verification at the constraint level, confirming that deterministic verification of individual constraints produces higher\-quality training signal than holistic LLM judgment—a principle CSE applies to evaluation\.[4](https://arxiv.org/html/2608.12426#bib.bib32)formalize controllable generation as satisfaction of language constraints, bridging constrained decoding and instruction following\. Neurologic Decoding[27](https://arxiv.org/html/2608.12426#bib.bib34);[26](https://arxiv.org/html/2608.12426#bib.bib35)approaches constraint satisfaction from the inference side via lexically\-constrained beam search; more recent structured generation frameworks[45](https://arxiv.org/html/2608.12426#bib.bib52);[3](https://arxiv.org/html/2608.12426#bib.bib48)can enforce format and keyword constraints at the decoding level, removing them from the model’s compositional burden entirely\. CSE evaluates raw generation without such scaffolding to isolate intrinsic compositional capacity; measuring how constrained decoding or chain\-of\-thought prompting shifts the transition is a natural follow\-up \([Section6](https://arxiv.org/html/2608.12426#S6)\)\.

#### Mechanistic explanations\.

ATA[29](https://arxiv.org/html/2608.12426#bib.bib43)identifies a mechanistic explanation for compositional degradation: models diffusely attend across all sub\-tasks in the input, and inactive sub\-tasks cause interference via attention distraction—corroborating our phase transition with performance drops from 60% to 48% as sub\-task count increases\. Their inference\-time attention steering recovers 6–10pp without retraining, demonstrating that the transition is partially addressable through architectural intervention\. In reinforcement learning, CERLLA[6](https://arxiv.org/html/2608.12426#bib.bib41)demonstrates that compositional value function representations with Boolean task algebra \(AND, OR, NOT\) enable sample\-efficient learning of 162 language\-conditioned tasks\. CSE’s constraint composition is purely conjunctive \(allkkconstraints must hold simultaneously\), but the shared insight is that compositional structure—whether in policy space or constraint space—determines scalability\.

### B\.2Constraint Scaling and Cross\-Modal Parallels

#### Interaction vs\. accumulation\.

A central question in compositional evaluation is whether difficulty arises from constraint*interactions*\(specific pairs that conflict\) or from*accumulation*\(each additional constraint independently reducing success probability\)\. ComplexBench[44](https://arxiv.org/html/2608.12426#bib.bib6)models structural interactions through nesting and branching but does not measure pairwise statistical independence\.[50](https://arxiv.org/html/2608.12426#bib.bib11)demonstrate that constraint ordering affects performance by up to 25%, establishing that presentation matters—we find this effect is orthogonal to constraint count \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)\)\. Ordered CommonGen[39](https://arxiv.org/html/2608.12426#bib.bib42)evaluates whether LLMs can generate sentences with concepts in a specified order across 36 models, finding that even the best model follows the specified order only 75% of the time on a single ordering constraint\. CSE’s ordering constraints show similarly lowk=1k\{=\}1rates \(O1 monotonic length: 28\.9%, O2 alphabetical: 43\.7%\), and these per\-constraint imperfections are precisely what compound multiplicatively to produce the phase transition\.

#### Phase transitions and scaling laws\.

Our finding that constraint failures are weakly correlated connects to the theoretical literature on phase transitions in constraint satisfaction\. Randomkk\-SAT exhibits a sharp satisfiability threshold driven by clause density and interaction structure[30](https://arxiv.org/html/2608.12426#bib.bib15);[31](https://arxiv.org/html/2608.12426#bib.bib14)\. Our transition is mechanistically simpler: it arises from near\-multiplicative accumulation of failure probabilities without requiring interaction structure\. This parallels neural scaling laws[20](https://arxiv.org/html/2608.12426#bib.bib39);[14](https://arxiv.org/html/2608.12426#bib.bib40), which established that simple parametric models \(power laws in compute and data\) predict performance with practical accuracy\.[10](https://arxiv.org/html/2608.12426#bib.bib44)provide theoretical support for hierarchical compositional learning: diffusion models learn grammar rules level\-by\-level, with sample complexityPℓ∝mℓ\+1P\_\{\\ell\}\\propto m^\{\\ell\+1\}for level\-ℓ\\ellrules—higher\-level \(more global\) rules require polynomially more data\. This predicts that constraints requiring deeper compositional processing should be more fragile, consistent with our finding that structural constraints degrade2\.0×2\.0\{\\times\}faster than lexical constraints under compositional load \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px6)\)\. Our floor modelP⁡\(k\)=a⋅e−β​k\+cP\(k\)=a\\cdot e^\{\-\\beta k\}\+cserves an analogous role for constraint composition: three parameters, fitted onk=1−8k\{=\}1\{\-\}8, predict held\-outk=9−12k\{=\}9\{\-\}12within 0\.2pp—comparable predictive precision from a comparably simple functional form\. The “emergent abilities” literature[43](https://arxiv.org/html/2608.12426#bib.bib16)documents sharp capability transitions as a function of model scale; we find analogous transitions as a function of compositional load\.

#### Cross\-modal compositional decay\.

Earlier work on vision\-language compositionality[42](https://arxiv.org/html/2608.12426#bib.bib21);[15](https://arxiv.org/html/2608.12426#bib.bib22)establishes that VLMs struggle with compositional attribute binding even atk=2k\{=\}2;[16](https://arxiv.org/html/2608.12426#bib.bib23)show that these failures persist under controlled evaluation designs\. The decay pattern we observe in text is consistent with cross\-modal findings in generation tasks\. ConceptMix[46](https://arxiv.org/html/2608.12426#bib.bib17)reports that text\-to\-image models drop from 83% atk=1k\{=\}1concepts to 8% atk=7k\{=\}7—implyingr≈0\.65r\\approx 0\.65per added concept, steeper than CSE’s aggregater=0\.922r\{=\}0\.922but following the same multiplicative functional form\. ConceptMix\+\+[12](https://arxiv.org/html/2608.12426#bib.bib18)shows that prompt optimization can recover up to 20% of lost compositional capability, suggesting the limitation is partially in prompt understanding rather than solely in generation\. T2I\-CompBench\+\+[17](https://arxiv.org/html/2608.12426#bib.bib19)documents degradation across attribute binding, spatial relationships, and generative numeracy\. T2V\-CompBench[41](https://arxiv.org/html/2608.12426#bib.bib20)shows temporal composition failures in video generation\. GenAI\-Bench[22](https://arxiv.org/html/2608.12426#bib.bib24)evaluates compositional text\-to\-visual generation, reporting degradation with compositional complexity in image, video, and 3D generation\. These are all constrained*generation*tasks, and the consistency of the decay pattern across text\-to\-text, text\-to\-image, and text\-to\-video generation suggests the transition may be a general property of compositional generation rather than a modality\-specific limitation\. ImpossibleBench[51](https://arxiv.org/html/2608.12426#bib.bib27)tests whether models detect infeasible constraint combinations in coding tasks; CSE includes deliberate impossible probes and finds zero hallucinated compliance—models either correctly refuse or fail on the constraint text itself\.

## Appendix CPer\-Constraint mCSR Heatmap

[Figure5](https://arxiv.org/html/2608.12426#A3.F5)shows per\-constraint mCSR for all 36 constraints across 15 models\.

![Refer to caption](https://arxiv.org/html/2608.12426v1/constraint_heatmap_pill.png)Figure 5:Per\-constraint mCSR by model \(36 constraints×\\times15 models\)\. GPT\-5\.5 dominates across all dimensions \(79\.8% overall\), followed by Claude 4\.7 Opus \(67\.0%\) and Flash\-Lite \(56\.6%\)\. The format–relational cluster is visible as a correlated block \(F1–F3, R1, R3\): models that fail structured output fail relational reasoning simultaneously—output\-feature coupling, not cognitive interference \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)\)\. Structural and meta constraints \(S1, S4, S6, M2\) are uniformly hard \(≤15%\{\\leq\}15\\%\), while lexical constraints \(L5 forbidden word: 90%, L2 mandatory words: 80%\) and numerical \(N2 no digits: 83%\) sustain the asymptotic floor\. DeepSeek V4 Pro and Kimi K2\.6 show the fast\-decay pattern: high variance across constraints but uniformly low rates outside the immune cluster\.
## Appendix DConstraint Profiles

[Figure6](https://arxiv.org/html/2608.12426#A4.F6)visualizes three properties of each constraint side by side: isolated difficulty \(k=1k\{=\}1mCSR\), compositional vulnerability \(degradation slope\), and the comprehension\-maintenance gap\.

Figure 6:Constraint profiles for all 36 constraints, grouped by dimension\.Left:k=1k\{=\}1mCSR \(single\-constraint difficulty in isolation\)\.Center:per\-constraint degradation slope \(more negative = faster collapse under compositional load; warm = fast, cool = slow\)\.Right:comprehension\-maintenance gapΔi=s¯i−mCSRi\\Delta\_\{i\}=\\bar\{s\}\_\{i\}\-\\text\{mCSR\}\_\{i\}\([Equation3](https://arxiv.org/html/2608.12426#S3.E3)\): filled circles show mCSR \(strict binary satisfaction\), open circles show continuous score \(partial compliance\), and the connecting bar isΔ\\Delta\. Constraints withΔ\>20\\Delta\>20pp are labeled\. A large gap indicates that models partially comply—demonstrating comprehension—but fail strict verification \(e\.g\., W1 unique words: score 78\.6%, mCSR 28\.4%,Δ=50\\Delta\{=\}50pp\)\. A near\-zero gap indicates all\-or\-nothing behavior with no partial\-credit regime \(e\.g\., R1 scene graph: score 67\.9%, mCSR 67\.7%,Δ=0\.2\\Delta\{=\}0\.2pp\)\. Stars mark compositionally immune constraints \(retention\>80%\{\>\}80\\%atk≥8k\{\\geq\}8\): L1, L5, W4\.
## Appendix EComplete Phase Transition Table

[Table4](https://arxiv.org/html/2608.12426#A5.T4)reports per\-model mCSR at eachkkfor all 15 models, sorted byk=1k\{=\}1performance\. GPT\-5\.5 leads at everykkvalue; the per\-model decay factorsrr\([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px3)\) compress each trajectory into a single number\.

Table 4:Per\-model mCSR \(%\) by constraint countkk, sorted byk=1k\{=\}1\.Boldmarks the best model at eachkk\. The rightmost columnrris the per\-model decay factor fromrate​\(k\)=a⋅rk−1\\text\{rate\}\(k\)=a\\cdot r^\{k\-1\}, fitted onk=1−8k\{=\}1\{\-\}8: the fraction of performance retained per additional constraint \(r=0\.95r\{=\}0\.95means 5% loss per constraint\)\. All 15 models start between 48–98% atk=1k\{=\}1and converge toward a model\-dependent floor atk≥8k\{\\geq\}8\.rrranges from0\.7210\.721\(DeepSeek, loses 28% per constraint\) to0\.9770\.977\(Claude 4\.7, loses 2%\)\. Row shading indicates decay archetype: unshaded is slow decay \(r\>0\.93r\{\>\}0\.93, 10 models\),medium\(0\.80≤r≤0\.930\.80\{\\leq\}r\{\\leq\}0\.93\),fast\(r<0\.80r\{<\}0\.80\)\.
## Appendix FDecay Model Predictions and Cross\-Validation

#### Aggregate decay model\.

The per\-constraint aggregate decay is well\-described by a pure multiplicative model:rate​\(k\)=72\.0%×0\.922k−1\\text\{rate\}\(k\)=72\.0\\%\\times 0\.922^\{k\-1\}\(held\-out MAE=0\.2=0\.2pp onk=9−12k\{=\}9\{\-\}12\)\. A three\-parameter floor model \(P⁡\(k\)=a⋅e−β​k\+cP\(k\)=a\\cdot e^\{\-\\beta k\}\+c, all\-data fit:a=0\.782a\{=\}0\.782,β=0\.082\\beta\{=\}0\.082,c=0\.0%c\{=\}0\.0\\%\) does not improve on this aggregate fit \(MAE=0\.2=0\.2pp\), because averaging across 15 models with diverse decay profiles \(r=0\.72−0\.98r=0\.72\{\-\}0\.98\) smooths the per\-model floors into a continuous decline\.

#### Cross\-validation\.

The floor parameterccis unstable under cross\-validation: it collapses to 0\.0% in 11 of 12 LOO folds and inflates to 9\.4% whenk=1k\{=\}1is held out, indicating that 12 data points do not robustly constrain a three\-parameter model at the aggregate level\. Under multiple train/test splits, the floor model performs comparably to, but does not outperform, the multiplicative baseline\.

#### The floor is per\-model, not aggregate\.

The aggregate curve masks per\-model floors because between\-model variance in decay rate \(r=0\.72−0\.98r=0\.72\{\-\}0\.98\) dominates the signal\. Atk=12k\{=\}12, the multiplicative model systematically underpredicts 10 of 15 models \(mean error:−5\.1\-5\.1pp\), indicating that these models plateau above the multiplicative prediction\. Even the weakest models \(DeepSeek V4 Pro, Kimi K2\.6\) sustain∼11%\{\\sim\}11\\%per\-constraint rates atk=12k\{=\}12despite multiplicative predictions of1−3%1\{\-\}3\\%—the compositionally immune constraints keep them off zero\. Two models \(Claude 4\.6 Opus, Claude 4\.7 Opus\) are overpredicted, decaying*faster*than the multiplicative model at highkk\. The floor is a per\-model phenomenon, visible when models are analyzed individually but masked in the cross\-model aggregate\.

#### Independent floor derivation\.

Compositionally immune constraints are identified by two criteria: degradation slope\>−0\.02\>\{\-\}0\.02\(nearly flat acrosskk\) andk≥8k\{\\geq\}8pass rate\>5%\>5\\%\(non\-trivially contributing to the floor\)\. Four constraints meet both criteria: L5 \(forbidden word, 87\.4% atk≥8k\{\\geq\}8\), L1 \(lipogram, 45\.0%\), M3 \(word count divisibility, 18\.3%\), and M2 \(self\-counting, 7\.6%\)\. S4 meets the slope criterion but has a trivially low rate \(4\.5% atk≥8k\{\\geq\}8\)\. W4 \(no repeated bigrams\) narrowly misses the slope threshold \(−0\.021\-0\.021\) but sustains 43\.7% atk≥8k\{\\geq\}8\(86% retention\), qualifying as immune under the retention criterion used in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px6)\. The independently predicted floor from these immune constraints isc=34\.5%c\{=\}34\.5\\%\(meank≥8k\{\\geq\}8rate of the four immune constraints, weighted by their representation\)—far above the fittedc=0\.0%c\{=\}0\.0\\%\. The gap confirms that the immune constraints’ per\-constraint rates do not translate directly to an aggregate floor: their contribution is diluted by the 32 non\-immune constraints whose rates converge toward zero, and by the between\-model variance that smooths the aggregate curve\.

#### Floor estimate comparison\.

The fitted and LOO estimates converge near zero; the independent estimate overshoots because it counts immune constraint rates without accounting for their dilution in the 36\-constraint aggregate\. The floor is real per\-constraint \(L5 at 87%, L1 at 45%, W4 at 44%, M3 at 18% atk≥8k\{\\geq\}8\) and per\-model \(10/15 underpredicted atk=12k\{=\}12\), but is not a feature of the aggregate decay curve with 15 models\.

## Appendix GFailure Mode Details

This appendix provides per\-model breakdowns supporting the failure mode analysis in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px6)\.

Table 5:L1 \(lipogram\) failure categorization by model \(n=5,267n\{=\}5\{,\}267total failures across 15 models\)\. “Genuine” = model attempts the constraint and violates it in content; “Refusal” = model identifies infeasibility and the forbidden letter appears in the explanation text; “Meta” = constraint self\-narration \(restating the rule using the forbidden letter\); “Case” = case\-folding artifact\. The refusal rate varies widely: GPT\-5\.2 and GPT\-5\.4 refuse∼90%\{\\sim\}90\\%of failed probes, while open\-weight models \(Llama, Grok, Maverick\) refuse<1%\{<\}1\\%\.Table 6:Model failure strategy profiles for genuine L1 failures only \(n=4,022n\{=\}4\{,\}022\)\. Median = normalized position of first violation \(0 = response start, 1 = end\)\.Pink= immediate strategy \(median<0\.05<0\.05\): violations occur at response onset\.Blue= gradual strategy \(median≥0\.10\{\\geq\}0\.10\): violations distributed across the response\. GPT\-5\.5 is excluded \(8 genuine failures, insufficient for profiling\)\.#### Classifier design\.

Accurately separating meta\-preamble from genuine failures required iterative refinement\. An initial narrow classifier \(keying on explicit preamble markers\) identifies only 8 meta\-preamble cases\. Manual inspection reveals contamination: 4 of 5 top\-ranked “genuine” failures are constraint self\-narration \(“Constraint L1 requires no letter ‘k’ ”\)\. We expand the classifier with a constraint\-echo regex matching patterns such as “no letter ‘X’ ”, “avoiding the letter”, and “Constraint \[L1\]” within 60 characters of the violation, reclassifying matches in the first 150 characters as meta\-preamble\. This moves 106 cases from genuine to meta\-preamble \(8→\\to114\), primarily from Claude models \(\+50, \+22, \+23 for the three variants\)\. The 150\-character threshold prevents reclassifying violations deep in the response where constraint references may be legitimate\.

## Appendix HFull Constraint Specifications

[Table7](https://arxiv.org/html/2608.12426#A8.T7)lists all 36 constraints with exact instruction text and verifier specifications\. Constraints marked with⋆\\starwere added through the design pipeline described in[AppendixK](https://arxiv.org/html/2608.12426#A11)\. One candidate lexical constraint \(anagram embedding\) was rejected during this pipeline because its verifier requires factual knowledge about which letter rearrangements are valid English words, violating the structural\-verification\-only design principle\.

Table 7:Complete constraint taxonomy \(36 constraints, 8 dimensions\)\. TheInstructioncolumn shows the text injected into probe prompts; parameters in braces are instantiated from fixed calibrated ranges\. TheVerifiercolumn summarizes the deterministic check applied to each response\. Constraints marked⋆\\starare new in CSE\.Table 8:Complete constraint taxonomy \(continued\)\.Table 9:Complete constraint taxonomy \(continued\)\.
## Appendix IProbe Generation and Compatibility Verification

Algorithm 1Probe generation with compatibility\-guaranteed resampling1:Pool

𝒞\\mathcal\{C\}, targets

\{\(k,gk,nk\)\}\\\{\(k,g\_\{k\},n\_\{k\}\)\\\}, topics

𝒯\\mathcal\{T\}, param ranges

Θ\\Theta
2:

nkn\_\{k\}primary

\+\+\(gk−nk\)\(g\_\{k\}\-n\_\{k\}\)reserve probes at each

kk
3:foreach

\(k,gk,nk\)\(k,g\_\{k\},n\_\{k\}\)do

4:

𝒫k←∅\\mathcal\{P\}\_\{k\}\\leftarrow\\emptyset
5:while

\|𝒫k\|<gk\|\\mathcal\{P\}\_\{k\}\|<g\_\{k\}do

6:

S←S\\leftarrowdraw

kk\-subset from

𝒞\\mathcal\{C\}⊳\\trianglerightExhaustive or random

7:

θ←\\theta\\leftarrowsample parameters for

SSfrom

Θ\\Theta
8:Shuffle constraint order in

SS
9:if

¬CheckCompat​\(S,θ\)\\neg\\,\\textsc\{CheckCompat\}\(S,\\theta\)then

10:continue

11:endif

12:

𝒫k←𝒫k∪\{Probe​\(S,θ,NextTopic​\(𝒯\)\)\}\\mathcal\{P\}\_\{k\}\\leftarrow\\mathcal\{P\}\_\{k\}\\cup\\\{\\textsc\{Probe\}\(S,\\theta,\\textsc\{NextTopic\}\(\\mathcal\{T\}\)\)\\\}
13:endwhile

14:Mark first

nkn\_\{k\}as primary, remainder as reserve

15:endfor

16:return

⋃k𝒫k∪ImpossibleProbes​\(𝒞\)\\bigcup\_\{k\}\\mathcal\{P\}\_\{k\}\\;\\cup\\;\\textsc\{ImpossibleProbes\}\(\\mathcal\{C\}\)

Whenkkconstraints are sampled into a single probe, certain parameter combinations create provably unsatisfiable requirements\. To prevent structurally impossible probes from entering the benchmark, the probe composer applies a multi\-layered compatibility checker \(check\_compatibility\(\)\) that rejects constraint combinations before probe generation\. We describe each category of structural conflict and the corresponding check\.

#### Hard incompatibilities\.

Nine constraint pairs are unconditionally incompatible regardless of parameters and are blocked as a static reject list: F1\+L1 \(JSON keys contain the forbidden letter\), M2\+W1 \(the self\-counting phrase “This text contains exactlyNNwords” inherently repeats common words\), O1\+O3 \(monotonic and alternating sentence lengths contradict\), N1\+N2 and N3\+N2 \(embedding numerical digits vs\. forbidding all digits\), F3\+R2 \(bullet list explicitly forbids numbered lists, transitive ordering requires them\), and N1\+R2, N3\+R2, N2\+R2 \(R2’s numbered list format “1\. first, 2\. second, …” injects digit sequences into the response text; numerical constraint verifiers that scan for digits viaextract\_numbers\(\)count these list markers as integers, corrupting N1’s count/sum, N3’s distinct count, and violating N2’s no\-digits rule\)\.

#### Sentence count consistency\.

Three constraints independently impose an exact sentence count: S2 \(explicit parameter\), L4 \(acrostic word length, e\.g\.,DREAM→\\to5 sentences\), and M1 \(hidden message word length, e\.g\.,SPARK→\\to5 sentences\)\. A fourth constraint, S3 \(paragraph structure\), imposes a*minimum*sentence count:target\_paragraphs×min\_sentences\_per\_para\\texttt\{target\\\_paragraphs\}\\times\\texttt\{min\\\_sentences\\\_per\\\_para\}\(default:×2\\times 2\)\. If any two of S2, L4, M1 co\-occur with different counts, or if S3’s minimum exceeds the count required by S2, L4, or M1, the probe is rejected\. For example, S3 with 3 paragraphs requires≥6\\geq 6sentences, but all M1 hidden words \(BRAVE,LIGHT,DREAM,SPARK,STORM\) have 5 letters, making S3\+M1 impossible whenever the paragraph count exceeds 2\.

#### Word budget feasibility\.

When S1 \(exact word count\), W3 \(minimum words per sentence\), and a sentence\-count constraint \(S2, L4, or M1\) co\-occur, the probe is rejected if the product of sentence count and minimum words per sentence exceeds the word budget\. For example, S1=50\{\}=50words, S2=6\{\}=6sentences, W3=10\{\}=10minimum words per sentence requires≥60\\geq 60words but only 50 are allowed\.

#### Alternating length feasibility\.

O3 \(alternating short/long sentences\) defines a maximum word count for short sentences \(short\_max\)\. If W3’s minimum words per sentence exceedsshort\_max, no short sentence can satisfy both constraints simultaneously, and the probe is rejected\.

#### Forbidden letter propagation\.

L1 \(lipogram\) forbids a single letter\. Several constraints force that letter into the output: M1 requires a word*starting with*each letter of its hidden word \(a word starting with ‘v’ necessarily contains ‘v’\); L4 requires sentences*starting with*each letter of the acrostic word \(same logic\); L3 requires every sentence to start with a word beginning with a specific letter\. If L1’s forbidden letter appears in M1’s or L4’s target word, or equals L3’s required letter, the probe is rejected\. Similarly, if L1’s forbidden letter appears in any of L2’s mandatory words, the probe is rejected\.

#### Word length range conflicts\.

W2 \(word length range\) restricts every word to\[min\_len,max\_len\]\[\\texttt\{min\\\_len\},\\texttt\{max\\\_len\}\]characters\. Two constraints can force words outside this range: L2 \(mandatory words\) may require words longer thanmax\_len\(e\.g\., “algorithm” is 9 characters, exceeding a\[1,8\]\[1,8\]range\); M2 \(self\-counting phrase\) requires the fixed phrase “This text contains exactlyNNwords,” whose words “This” and “text” \(4 characters each\) may fall below a tightmin\_len\. Both interactions are checked at generation time\.

#### Lexical conflicts\.

L5 \(forbidden word\) and L2 \(mandatory words\) are checked for overlap: if the forbidden word appears in the required word list, the probe is rejected\.

#### Format exclusivity\.

F1 \(valid JSON\) requires the entire response to be a single JSON object with no text outside it\. This is structurally incompatible with any constraint that requires non\-JSON output formatting or whose verifier assumes prose with identifiable sentences\. Specifically, F1 is blocked from co\-occurring with 12 constraints in three categories:*\(i\)*output format conflicts: F2 \(markdown table\), F3 \(bullet list\), and R2 \(numbered list\), which demand non\-JSON structure;*\(ii\)*prose\-dependent verifiers: S2 \(sentence count\), S3 \(paragraph structure\), L3 \(sentence\-initial letter\), L4 \(acrostic\), M1 \(terminal acrostic\), O1 \(monotonic sentence length\), O2 \(alphabetical sentences\), O3 \(alternating sentence length\), and W3 \(minimum words per sentence\), all of which split the raw text into sentences—a JSON object contains no sentence boundaries in the prose sense, so these verifiers find zero sentences and fail deterministically\. F1 remains compatible with constraints that operate on raw text without assuming prose structure, including word and character counts \(S1, S4, S5\), lexical constraints \(L2, L5, L6\), numerical constraints \(N1–N3\), word\-level properties \(W1, W2\), meta\-constraints \(M2, M3\), and other JSON\-producing constraints \(R1, R3\), whose required keys can be merged into a single JSON object\.

Additionally, F3 \(bullet list\) explicitly states “Do not use numbered lists or other bullet formats,” which directly contradicts R2 \(transitive ordering\), whose instruction requires “Output them as a numbered list \(1\. first, 2\. second, etc\.\)\.” The F3\+R2 pair is blocked unconditionally\.

#### Word count divisibility\.

S1 \(exact word countNN\) and M3 \(word count divisible byDD\) are checked for arithmetic compatibility: the probe is rejected ifNmodD≠0N\\bmod D\\neq 0\.

#### Integer count consistency\.

N1 \(embed exactlyKKintegers summing toTT\) and N3 \(exactlyDDdistinct integers\) are checked: the probe is rejected ifK<DK<D, since the total number of integer occurrences cannot be less than the number of distinct values required\.

#### Validation\.

In total, the compatibility checker enforces 9 hard pair blocks, 12 F1 format\-exclusivity blocks, and 14 parameter\-dependent checks covering sentence counts, word budgets, letter propagation, word length ranges, word count divisibility, integer count consistency, and lexical conflicts\. All checks are deterministic and run inO⁡\(k2\)O\(k^\{2\}\)time during probe composition\. The rejection rate increases withkk: from∼32%\{\\sim\}32\\%atk=4k\{=\}4to∼98%\{\\sim\}98\\%atk=12k\{=\}12\. High\-kkprobes are drawn from the restricted subset of compatible combinations; we analyze and control for the resulting composition bias in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px2)\.

#### Probe sampling strategy\.

At lowkk, the space of possible constraint combinations is small enough to enumerate exhaustively: the probe generator materializes all\(\|𝒞\|k\)\\binom\{\|\\mathcal\{C\}\|\}\{k\}combinations, shuffles them uniformly, and iterates through them\. For the 36\-constraint pool used in CSE, this is feasible throughk=5k\{=\}5\(\(365\)=376,992\\binom\{36\}\{5\}\{=\}376\{,\}992combinations,∼120\{\\sim\}120MB\)\. Atk≥6k\{\\geq\}6, exhaustive materialization exceeds a 500 MB memory cap \(\(366\)=1,947,792\\binom\{36\}\{6\}\{=\}1\{,\}947\{,\}792combinations,∼0\.7\{\\sim\}0\.7GB;\(3610\)=254,186,856\\binom\{36\}\{10\}\{=\}254\{,\}186\{,\}856,∼140\{\\sim\}140GB\), so the generator switches to uniform random sampling: combinations are drawn uniformly at random with a reuse cap of 10 per combination\.

Both paths feed into the same acceptance pipeline: draw a candidate combination, check compatibility, generate the probe if compatible, reject and redraw if not\. The only difference is coverage uniformity: exhaustive shuffle guarantees each combination is visited at most once before repeating, while random sampling may revisit combinations \(though the reuse cap bounds this\)\. For the paper’s analyses, this distinction is immaterial: atk=6k\{=\}6we draw∼500\{\\sim\}500probes from2\.762\.76M possible combinations \(0\.018%0\.018\\%sample\), making the two strategies statistically indistinguishable\. The compatibility checker serves as the acceptance gate in both cases—a probe is valid or it is not, regardless of how the candidate combination was drawn\.

## Appendix JModel Notes and API Success Rates

[Table10](https://arxiv.org/html/2608.12426#A10.T10)reports per\-model API success rates\. All models were evaluated on 4,527 probes \(4,470 primary\+\+57 impossible\) at temperature=0\{=\}0with an output token budget of 4,096 tokens \(16,384 for Claude 4\.7 Opus, which requires extended reasoning\)\. GPT\-5\.5 completed 4,526 of 4,527 probes \(1 empty response excluded\)\. Claude 4\.7 Opus had 340 content\-safety refusals from Anthropic’s integrity filter and 57 transient API errors; all 397 are excluded from scoring \(treated identically to missing probes, not injected as failures\), as they reflect infrastructure\-level safety filtering rather than intrinsic compositional capacity\. Claude 4\.5 Sonnet \(2 refusals\) and Claude 4\.6 Opus \(1 refusal\) are similarly excluded\. All remaining models achieved≥99\.9%\{\\geq\}99\.9\\%API success\.

Table 10:API success rates by model\. “Refusals” are integrity\-filter rejections excluded from scoring \(the model never saw the prompt\)\. “Errors” are transient API failures \(rate limits, timeouts, empty responses\), also excluded\. Both are treated as missing probes—not as failures—to measure intrinsic compositional capacity independent of infrastructure behavior\.#### Compositionality penalty anomalies\.

Two models show near\-zero gaps between mCSR and sCSR: DeepSeek V4 Pro \(22\.4% marginal, 22\.4% strict, 0\.0pp penalty\) and Gemini 3\.1 Pro \(42\.1% marginal, 39\.4% strict, 2\.7pp penalty\)\. Both are tag\-compliance artifacts, not genuine compositional resilience\. DeepSeek’s 72\.9% answer\-tag compliance means that∼27%\{\\sim\}27\\%of responses fail tag extraction entirely—all constraints are scored on raw text including chain\-of\-thought, producing correlated all\-fail outcomes that affect mCSR and sCSR equally\. When extraction succeeds, DeepSeek attempts constraints genuinely, but the binary extraction\-success/extraction\-failure mode dominates the statistics and collapses the mCSR–sCSR gap\. Gemini Pro \(46\.9% tag compliance\) shows the inverse of the Llama pattern: Llama models have 100% tag compliance, so every probe receives clean evaluation and constraint failures are distributed across different constraints in different probes \(high mCSR, low sCSR\)\. Gemini Pro’s low tag compliance suppresses its mCSR \(many individual constraint checks fail because the verifier scores reasoning text\), but the probes where it produces clean answers tend to pass all constraints simultaneously, yielding relatively high sCSR\. This connects to the co\-failure analysis \([Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)\): tag\-extraction failure acts as a shared output feature that induces correlated failures across all constraints within a probe, identical in mechanism to the sentence\-count coupling identified among constraint pairs\.

#### Bimodal constraint profiles\.

Grok 4\.1 shows an unusually flat trajectory \(r=0\.948r\{=\}0\.948\) despite ranking 14th atk=1k\{=\}1\(54\.9%\)\. Its per\-constraint profile is bimodal: 7 constraints are near\-zero even atk=1k\{=\}1\(W5: 0\.1%, S4: 0\.0%, M1: 0\.2%\), while 7 are near\-ceiling \(L5: 98\.3%, R1: 98\.5%, F3: 90\.2%\)\. With little in between to erode, the trajectory is dominated by immune constraints maintaining their rates, and the∼34%\{\\sim\}34\\%floor atk=12k\{=\}12is almost entirely the immune cluster\.

## Appendix KConstraint Design Methodology

Designing constraints for a*compositional*benchmark is harder than for a single\-constraint benchmark: every candidate must be checked against all existing constraints for structural incompatibilities—both unconditional and parameter\-dependent—and constraints that lock down multiple output dimensions simultaneously create disproportionate composition bias at highkk\. We formalize the vetting process as a four\-stage pipeline\.

#### Stage 1: Deterministic verifiability\.

The constraint is rejected if its pass/fail verdict cannot be computed as a deterministic function of the response text and constraint parameters, without LLM judgment\. This is the foundational design choice: it excludes semantic and pragmatic constraints but makes CSE’s measurements invariant to the same compositional degradation it studies \([Section1](https://arxiv.org/html/2608.12426#S1)\)\. All 36 constraints in the final pool pass this stage; rejected candidates included “maintain formal tone throughout” \(requires stylistic judgment\), “each paragraph must introduce a novel argument” \(requires semantic novelty detection\), and “the response must be factually accurate” \(requires knowledge verification\)\.

#### Stage 2: Composability analysis\.

Each surviving candidate is assessed on four axes\.*Hard conflicts:*how many existing constraints are unconditionally incompatible regardless of parameters?*Parameter\-dependent conflicts:*how many are incompatible for some parameter values but not others?*Output dimensions constrained:*how many aspects of the output \(sentence count, word count, ordering, format, vocabulary\) does the constraint lock down?*High\-kksurvival:*what fraction of randomk=12k\{=\}12constraint sets including this constraint would pass the compatibility checker?

Constraints that constrain multiple output dimensions simultaneously inherit the incompatibilities of each dimension\. For example, a candidate Fibonacci sentence length constraint \(“the word count of consecutive sentences must follow the Fibonacci sequence starting from 3, 4”\) locks down sentence count, per\-sentence word count, and sentence ordering—essentially fusing S2\+\+S1\+\+O1 into a single constraint\. It inherits 2 hard conflicts \(O3 alternating length\) and 6 parameter\-dependent conflicts \(S2, S1, W3, L4, M1, S3\), giving it POOR composability\. In contrast, R4 \(cross\-sentence word bridge: “the last word of each sentence must be the first word of the next”\) constrains a single novel dimension \(inter\-sentence word dependency\) with 0 hard conflicts and 2 parameter\-dependent ones \(L4, L3\), giving it GOOD composability\. R4 was accepted; the Fibonacci candidate was softened into O4 \(relaxed growth: “each sentence must contain more words than the combined word count of the previous two”\), which reduces the hard conflicts from 2 to 1 and the parameter\-dependent conflicts from 6 to 3\.

#### Stage 3: Difficulty calibration\.

Each constraint is piloted atk=1k\{=\}1on 3 models spanning the performance range\. Parameters are adjusted to target 20–80% mCSR atk=1k\{=\}1, ensuring the constraint is neither trivially satisfiable \(uninformative at allkk\) nor near\-impossible \(unable to contribute to the decay signal\)\. Constraints that floor \(<5%<5\\%\) or ceiling \(\>95%\>95\\%\) after parameter tuning are rejected\. The fixed MEDIUM difficulty level used across allkkvalues is calibrated at this stage\.

#### Case study: W6 \(word length alternation\) — rejected at Stage 3\.

W6 required words to alternate between short \(≤N\{\\leq\}Ncharacters\) and long \(≥M\{\\geq\}Mcharacters\)\. Three calibration attempts failed: strict alternation with a 3\-character gap between bands \(0% mCSR\), strict alternation with no gap \(0% mCSR\), and a 75% tolerance threshold \(0% mCSR, 54% score\)\. The root cause is distributional: English word lengths are∼\{\\sim\}2:1 skewed toward short words—2–5 character words constitute∼\{\\sim\}65% of the lexicon—and the skew is even more pronounced in generated text, where high\-frequency function words \(*the*,*and*,*for*,*is*\) dominate\. Strict alternation requires roughly equal representation in alternating positions, which is structurally infeasible against this distribution\. At 0% mCSR, W6 would contribute no decay signal and would poison co\-occurring constraints in theφ\\varphianalysis with guaranteed failures\.

#### Stage 4: Pool entry\.

The constraint enters the 36\-type pool with its calibrated parameter ranges, verified composability profile, and deterministic verifier\. The full pool spans 8 dimensions \([Table2](https://arxiv.org/html/2608.12426#S4.T2)\), from surface\-level lexical patterns \(L1 lipogram, L5 forbidden word\) through document structure \(S3 paragraph count, O1 monotonic sentence length\) to relational reasoning \(R1 scene graph, R3 logic grid\)\.

## Appendix LIRT Validation

A 1PL \(Rasch\) Item Response Theory model fitted on 2,952 informative probes \(excluding 1,131 all\-pass or all\-fail probes that do not discriminate between models\) converges in 234 iterations and recovers the identical model ranking \(Spearmanρ=1\.000\\rho=1\.000, MAE=0\.0000\{=\}0\.0000\)\. IRT ability estimates confirm the tier structure visible in raw sCSR: GPT\-5\.5 is a massive outlier \(θ=\+4\.3\\theta=\+4\.3, 85\.0% raw pass rate\), Claude 4\.7 Opus and Gemini 3\.1 Pro form a second tier \(θ=\+1\.8\\theta=\+1\.8to\+2\.4\+2\.4\), and a dense middle pack clusters aroundθ≈0\\theta\\approx 0\. At the other extreme, Kimi K2\.6 \(θ=−2\.9\\theta=\-2\.9, 6\.3%\) occupies a distinct lower tier\. The perfect rank correlation confirms that raw sCSR is not distorted by item difficulty confounds—IRT calibration preserves the ranking exactly, validating sCSR as a sufficient metric without adaptive testing infrastructure\. The 28% of probes uninformative for IRT ranking are not wasted: they define the endpoints of the phase transition curve and power constraint\-level analyses operating on all 369,753 checks\.

#### mCSR vs\. sCSR ranking divergence\.

IRT validates the sCSR ranking, but the sCSR ranking itself diverges substantially from the mCSR ranking reported in[Table1](https://arxiv.org/html/2608.12426#S4.T1)\.[Table11](https://arxiv.org/html/2608.12426#A12.T11)shows the full comparison\. The most striking mover is Gemini Pro, which ranks \#11 by mCSR \(42\.1%\) but \#3 by sCSR \(39\.4%\)—it passes fewer individual constraints but chains them together more reliably than models with higher per\-constraint rates\. Flash\-Lite drops from \#3 \(mCSR\) to \#5 \(sCSR\); Claude 4\.5 drops from \#5 to \#9\. The mid\-tier models \(Llama 405B, Llama 70B, Qwen, Grok\) cluster tightly on sCSR \(11–15%\) despite spreading across 42–46% on mCSR—their constraint failures are distributed across different constraints in different probes, so per\-constraint competence does not translate to probe\-level success\. This divergence is the compositionality penalty: mCSR measures whether a model can satisfy constraints individually, while sCSR measures whether it can satisfy them simultaneously\.

Table 11:Model rankings by mCSR \(per\-constraint\) vs\. sCSR \(probe\-level\)\. Rank changes≥3\{\\geq\}3positions are highlighted\. The divergence quantifies how much compositional precision differs from per\-constraint competence\.
#### Geometric mean ranking\.

A natural follow\-up: what if we require models to rank high on*both*per\-constraint and probe\-level metrics? The geometric meanmCSR×sCSR\\sqrt\{\\text\{mCSR\}\\times\\text\{sCSR\}\}penalizes imbalance—a model must score well on both to rank highly\.[Table12](https://arxiv.org/html/2608.12426#A12.T12)shows the result\.

Table 12:Model rankings under three metrics: mCSR \(per\-constraint breadth\), sCSR \(probe\-level precision\), and their geometric mean \(balanced\)\. Rank changes≥3\{\\geq\}3positions between mCSR and geometric mean are highlighted\. The top 2 models are unambiguous under any metric; the middle tier reshuffles entirely depending on whether breadth or precision is valued\.The largest movers reveal what drives the reshuffling\. Gemini Pro jumps from mCSR rank \#11 to geometric rank \#4: its balanced profile \(42\.1/39\.4, 2\.7pp penalty\) is rewarded by the geometric mean\. However, this balance is a tag compliance artifact—47% of Gemini Pro responses fail tag extraction entirely, suppressing both metrics equally rather than reflecting genuine compositional strength\. Flash\-Lite drops from \#3 to \#5: its high per\-constraint competence \(56\.6%\) is dragged down by poor probe\-level success \(24\.1%\)\. Llama 405B drops from \#6 to \#11: the 34\.0pp compositionality penalty is devastating under a metric that requires both dimensions to be high\. GPT\-5\.4 Pro rises from \#12 to \#7: moderate on both metrics \(35\.6/23\.2\), with no extreme gap\.

The top 2 models \(GPT\-5\.5, Claude 4\.7\) are unambiguous under any metric\. The middle tier is where the ranking depends entirely on whether one values breadth \(mCSR\) or precision \(sCSR\)\. The geometric mean is a principled compromise, but in practice it primarily surfaces the tag compliance confound \(Gemini Pro, DeepSeek\) rather than adding independent insight—models whose tag extraction suppresses both metrics equally appear “balanced” without being compositionally strong\.

#### Caveats\.

M=15M\{=\}15models is below classical IRT recommendations \(N≥200N\\geq 200for stable 2PL estimation\)\. Recent work on adaptive IRT for LLM evaluation[2](https://arxiv.org/html/2608.12426#bib.bib26);[23](https://arxiv.org/html/2608.12426#bib.bib25)shows that continuous scores and adaptive item selection can improve ranking confidence with fewer items, but requires more examinees than CSE’s 15\-model panel\. We also tried a 2PL model with L2\-regularized discrimination parameters \(λ=0\.5\\lambda\{=\}0\.5, centered atα=1\.0\\alpha\{=\}1\.0\)\. It does not converge despite 333 iterations and 2M function evaluations, produces pathological parameter scales \(θ\\thetaspanning\[−15\.9,\+11\.1\]\[\-15\.9,\+11\.1\],β\\betaspanning\[−31\.9,\+25\.2\]\[\-31\.9,\+25\.2\]\), and degrades the ranking \(ρ=0\.986\\rho=0\.986\)\. The regularization successfully controlledα\\alpha\(mean=0\.67\{=\}0\.67, std=0\.54\{=\}0\.54, max=1\.70\{=\}1\.70—no longer hitting the bound\), but the optimizer compensated by inflatingθ\\thetaandβ\\beta, a classic sign of an underdetermined model withM=15M\{=\}15examinees\. The 2PL’s log\-likelihood improvement over the 1PL is expected from doubling the parameter count and does not indicate better model identification\. We use IRT as a validation check \(which it passes: perfect rank preservation with a converged optimizer\) rather than as a primary analysis\.

## Appendix MCo\-Failure Analysis: Additional Detail

Within\- vs\. across\-dimension\.Within\-dimension co\-failure \(mean\|φ\|=0\.087\|\\varphi\|=0\.087\) exceeds across\-dimension \(0\.0670\.067\), but this difference does not reach significance \(Mann\-Whitneyp=0\.227p=0\.227\) and is driven by the numerical–numerical \(φ¯=0\.275\\bar\{\\varphi\}\{=\}0\.275\) and format–format \(φ¯=0\.241\\bar\{\\varphi\}\{=\}0\.241\) clusters\. Structural–structural \(φ¯=0\.039\\bar\{\\varphi\}=0\.039\), lexical–lexical \(φ¯=0\.047\\bar\{\\varphi\}=0\.047\), and most other within\-dimension pairs are at noise level\. Constraints within the same cognitive dimension do not interfere with each other more than constraints across dimensions, outside the shared structured\-output dependency\.

No shortcut through constraint selection\.The co\-failure analysis contains only 1 synergistic pair \(φ<−0\.05\\varphi<\-0\.05\): O4–R2 atφ=−0\.055\\varphi\{=\}\{\-\}0\.055\. If “compatible” constraint pairs existed—combinations that are easier together than independently—they would appear as negativeφ\\varphivalues\. They essentially do not\.

Co\-occurrence coverage acrosskk\.Eachφ\\varphicoefficient requires both constraints to appear together in the same probe\. At constraint countkk, the probability that two specific constraints co\-occur equals\(34k−2\)/\(36k\)=k⁡\(k−1\)/\(36×35\)\\binom\{34\}\{k\{\-\}2\}/\\binom\{36\}\{k\}=k\(k\{\-\}1\)/\(36\\times 35\), which is0\.16%0\.16\\%atk=2k\{=\}2and0\.95%0\.95\\%atk=4k\{=\}4\. Consequently, theφ\\varphianalysis is best powered atk≥4k\{\\geq\}4, where the expected co\-occurrence count per pair exceeds 3 for probe sets of size 200\. Atk=2−3k\{=\}2\{\-\}3, most pairs have zero or one co\-occurrence—a structural property of any design that drawskk\-subsets from a 36\-type pool, not a feature of sample size that can be overcome by generating more probes at feasible scale\. Our independence estimates are therefore computed atk≥4k\{\\geq\}4and aggregated acrosskklevels\. This scope restriction is conservative: lowerkkimposes less compositional pressure and fewer competing demands on the model’s generation, so if constraints do not interfere atk≥4k\{\\geq\}4they are unlikely to do so atk=2−3k\{=\}2\{\-\}3\.

Output\-feature cluster analysis\.Constraints can be grouped by the output feature they depend on rather than by linguistic dimension:*sentence\_structure*\(S2, S3, S5, S6, L3, L4, M1, O1–O4, R4, F4, W3\),*word\_inventory*\(W1, W2, W4, L1, L2, L5\),*number\_content*\(N1–N4\),*document\_format*\(F1–F3\),*relational\_json*\(R1, R3\),*global\_statistics*\(S1, S4, M2, M3, L6, W5\), and*ordered\_list*\(R2\)\. Within\-cluster constraint pairs show mean\|φ\|=0\.113\|\\varphi\|=0\.113, while across\-cluster pairs show\|φ\|=0\.058\|\\varphi\|=0\.058\(Mann\-Whitneyp<10−6p<10^\{\-6\}\)\. This confirms that the elevatedφ\\varphivalues in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px4)reflect shared structural dependencies on the same output feature—when a model produces the wrong number of sentences, every sentence\-dependent constraint fails simultaneously—rather than pairwise constraint interference\. Grouping constraints by linguistic dimension \(structural, lexical, etc\.\) does not predict which pairs co\-fail \(within\-dimension mean\|φ\|=0\.087\|\\varphi\|=0\.087vs across\-dimension0\.0670\.067,p=0\.227p=0\.227\)\. Grouping by output feature—sentence count, number count, document format—does \(p<10−6p<10^\{\-6\}\)\. Co\-failure is mechanical: when sentence splitting breaks, everything that reads sentences breaks together, regardless of which dimension those constraints belong to\. L4 \(lexical\) and S2 \(structural\) share a sentence\-count dependency \(φ=0\.501\\varphi\{=\}0\.501\); L1 \(lexical\) and L5 \(lexical\) share a dimension but not an output feature \(φ=−0\.039\\varphi\{=\}\{\-\}0\.039\)\.

## Appendix NConstraint Ordering Position Effect

[50](https://arxiv.org/html/2608.12426#bib.bib11)demonstrate that constraint ordering affects performance by up to 25%, raising the concern that CSE’s degradation hierarchy could reflect prompt position rather than processing demands\. To eliminate this confound by design, CSE randomizes the presentation order of constraints within each probe: after constraint selection and parameter instantiation, the constraint list is uniformly shuffled before prompt assembly \([Algorithm1](https://arxiv.org/html/2608.12426#alg1)\)\. This breaks any correlation between constraint type and prompt position, ensuring that the ordering regression measures position effects rather than type effects\.

We verify that the randomization achieved its goal by regressing per\-constraint pass rate on position within the prompt \(1st constraint, 2nd, …,kkth\)\. Within eachkklevel, the correlations are negligible: mean\|ρ\|=0\.029\|\\rho\|=0\.029acrossk=2−12k\{=\}2\{\-\}12, with only 5 of 11kkvalues reachingp<0\.05p<0\.05\. Crucially, the directions are inconsistent—positive atk=3k\{=\}3\(ρ=\+0\.046\\rho\{=\}\{\+\}0\.046\) andk=11k\{=\}11\(ρ=\+0\.043\\rho\{=\}\{\+\}0\.043\), negative atk=5k\{=\}5\(ρ=−0\.073\\rho\{=\}\{\-\}0\.073\) andk=10k\{=\}10\(ρ=−0\.056\\rho\{=\}\{\-\}0\.056\)—indicating no systematic primacy or recency bias\. The largest per\-kkeffect \(k=5k\{=\}5,ρ=−0\.073\\rho=\-0\.073\) explains<0\.5%<0\.5\\%of variance\. Within eachkklevel, position explains<0\.1%<0\.1\\%of variance \(r2≈0\.001r^\{2\}\\approx 0\.001\), confirming that the degradation hierarchy reflects constraint processing demands rather than presentation order\.

## Appendix OSynergy at Lowkk

An unexpected finding: actual sCSRs*exceed*independence predictions at allk=1−11k\{=\}1\{\-\}11\(from\+26\.4\+26\.4pp atk=1k\{=\}1to\+1\.1\+1\.1pp atk=11k\{=\}11, 95% CI excludes zero\), transitioning to approximate independence only atk=12k\{=\}12\(\+1\.0\+1\.0pp\)\. This is the opposite of interference—models perform*better*than chance when juggling small numbers of constraints, and the synergy persists deeper than in CSE\-30 \(which reached independence atk=5k\{=\}5\)\. With 36 constraints \(vs\. 30\), there is more room for constraint co\-satisfaction before the multiplicative regime dominates\. We test whether hard constraints force models into a more careful “compliance mode” that benefits all constraints, by partitioning probes by the difficulty of their harder constraint\. Easy\-anchor probes show*more*synergy \(\+32\.6\+32\.6pp\) than hard\-anchor probes \(\+17\.7\+17\.7pp\)—the opposite of the compliance\-mode prediction\. The synergy is a base\-rate artifact rather than a behavioral strategy\. Does the synergy disappear at a constant rate, or does it decay in a curve? We fit both a linear model \(synergy​\(k\)=a\+b​k\\text\{synergy\}\(k\)=a\+bk\) and a quadratic model \(synergy​\(k\)=a\+b​k\+c​k2\\text\{synergy\}\(k\)=a\+bk\+ck^\{2\}\) to the penalty curve\. The*superlinearity improvement*—the fraction of the linear model’s residual error that the quadratic term eliminates—is 98\.8% at the constraint level \(83\.7% at the probe level\): the decay is strongly concave, not linear\. Synergy drops sharply fromk=1k\{=\}1tok=4k\{=\}4\(\+26\+26pp→\\to\+14\+14pp\), then tapers gradually throughk=8−12k\{=\}8\{\-\}12\(\+4\+4pp→\\to\+1\+1pp\)\. This concave shape is consistent with the floor model: at lowkkthere is substantial synergy to lose, while at highkkthe system is already near the independence floor and little synergy remains to erode\.

## Appendix PWhyk=12k\{=\}12Is the Upper Bound

We evaluate atk=1k\{=\}1to1212rather than extending to higher values\. This upper bound is determined by three converging considerations: signal exhaustion, constraint diversity erosion, and scientific question coverage\.

#### Signal exhaustion\.

At the per\-constraint level, the aggregate decay is well\-described by a multiplicative model \(held\-out MAE=0\.2\{=\}0\.2pp;[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px3)\)\. Byk=12k\{=\}12, the aggregate mCSR is 29\.2%\. At the probe level, sCSR reaches<4%\{<\}4\\%byk=9k\{=\}9and 1\.4% atk=12k\{=\}12—additionalkkvalues measure probe\-level performance at the noise floor\. Thek=1−8k\{=\}1\{\-\}8range characterizes the decay;k=9−12k\{=\}9\{\-\}12validates it\. All three contributions—the decay characterization, the hierarchy, and the co\-failure structure—are fully resolved withink=1−12k\{=\}1\{\-\}12\.

#### Constraint diversity erosion\.

Atk=12k\{=\}12, each probe draws 12 of 36 constraints \(33% of the pool\)\. Askkincreases, probes become increasingly homogeneous: the fraction of compatible combinations shrinks, and surviving probes are drawn from an increasingly narrow subset\.

The incompatibility rejection rate increases withkk, as shown in[Table13](https://arxiv.org/html/2608.12426#A16.T13)\. While the combinatorial pool grows rapidly \(\(3612\)≈1\.25​B\\binom\{36\}\{12\}\\approx 1\.25\\text\{B\}\), the fraction of compatible combinations drops to∼2%\{\\sim\}2\\%atk=12k\{=\}12\. The resampling loop \([Algorithm1](https://arxiv.org/html/2608.12426#alg1)\) compensates by drawing additional combinations until the target count is reached—feasible at allkkvalues because the absolute number of compatible combinations remains large \(∼24​M\{\\sim\}24\\text\{M\}atk=12k\{=\}12\)\. The composition bias at highkk—surviving probes exclude incompatible constraint types—is analyzed and controlled for in[Section4](https://arxiv.org/html/2608.12426#S4.SS0.SSS0.Px2)\.

Table 13:Combinatorial pool size and estimated incompatibility rejection rate at eachkkfor\|𝒞\|=36\|\\mathcal\{C\}\|\{=\}36\. “Compatible” is the estimated number of constraint combinations passing the full compatibility checker \(Monte Carlo estimate, 200,000 samples perkk\)\. The resampling loop draws from this pool until the target probe count is reached\.
#### Pool size as the fundamental constraint\.

Thek=12k\{=\}12ceiling is ultimately determined by the constraint pool size \(\|𝒞\|=36\|\\mathcal\{C\}\|\{=\}36\)\. A larger pool would permit higherkkvalues with maintained diversity and lower rejection rates\. We chose\|𝒞\|=36\|\\mathcal\{C\}\|\{=\}36to balance three requirements: \(1\) sufficient diversity across 8 processing dimensions for the degradation hierarchy, \(2\) all constraints deterministically verifiable with no LLM\-judge involvement, and \(3\) feasible pilot calibration \(36 verifiers×\\timesparameter sweeps×\\times2 validation phases\)\.

## Appendix QVerifier Audit and Known Edge Cases

We conducted an exhaustive audit of all 30 constraint verifiers across all 55,962 reported failures \(82,628 total constraint checks, 59,897 failures after excluding impossible probes\)\. The audit identified 5,733 discrepancies \(10\.2% of failures\) across eight classes, ranked below by impact\. An independent code\-level audit of 25,000 checks against reimplemented verifiers confirmed 99\.98% agreement on non\-discrepant checks, and an LLM\-as\-judge evaluation of 300 stratified samples found 100% agreement\.

#### Class 1: Meta\-note contamination \(3,721 cases, 64\.9% of discrepancies\)\.

Models that satisfy a constraint in their content sometimes add metacommentary \(“Note: I avoided the letter ‘j’ ”\) that itself violates the constraint\. The verifier scores the entire response including the note\. For example, a model avoiding the letter ‘j’ writes a complete essay with zero ‘j’ in the content, then appends “Note: I wrote this without using the letter ‘j’ anywhere”—the note contains ‘j’\. This disproportionately penalizes models that are good at the constraint: GPT\-5\.4 Pro has 78% of its L1 \(lipogram\) failures from meta\-notes alone\. Affected constraints: W2 \(835\), W1 \(753\), M3 \(708\), N2 \(506\), L1 \(375\), S5 \(98\), L6 \(71\), N3 \(45\), S1 \(8\)\.Fix:wrap the final response in<answer\>tags and score only the tagged content\.

#### Class 2: Markdown formatting \(892 cases, 15\.6%\)\.

Models use markdown syntax \(\#,\*\*\) which the verifier reads as literal characters\. A heading\# Historycauses the verifier to read ‘\#’ as the first character instead of ‘H’\. Bold formatting\*\*D\*\*ecliningcauses ‘\*’ to be read as the first character instead of ‘D’\. For L4 \(acrostic\), models that highlight acrostic letters with bold formatting—e\.g\.,\*\*S\*\*olar\.\.\. \*\*T\*\*echnology\.\.\. \*\*O\*\*utput\.\.\.—produce a correct acrostic that the verifier reads as\*\*\*\*\*\. Affected constraints: L3 \(512\), L4 \(317\), S3 \(63\)\.Fix:instruct models to write in plain text without markdown\.

#### Class 3: Code fence interaction \(503 cases, 8\.8%\)\.

When F4 \(code blocks\) co\-occurs with sentence\-dependent constraints, the‘‘‘markers and code contents create false structural elements\. Fence markers are counted as sentences, lines, or paragraphs\. Code content \(variable names, JSON keys\) skews word counts, vowel ratios, and number counts\. For example, a model writes 5 prose sentences plus a code block; the verifier splits on periods inside the code, counting 8 “sentences\.” Affected constraints: S2 \(52\), S3 \(95\), S5 \(29\), O2, O3 \(34\), O1 \(20\), M1 \(13\), M3 \(92\), L6 \(7\), W1 \(36\), W2 \(46\)\.Fix:strip code block contents before running sentence/structure verifiers\.

#### Class 4: Numbered list sentence splitting \(379 cases, 6\.6%\)\.

Models write numbered lists \(“1\. First item\. 2\. Second item\.”\)\. The period after the digit \(“1\.”\) triggers the sentence splitter\(?<=\[\.\!?\]\)\\s\+, creating false sentence boundaries\. Each list item becomes two or more “sentences\.” For O2 \(alphabetical first words\), list indices create “sentences” starting with digits, which break alphabetical order\. Affected constraints: S2 \(39\), O2 \(∼\\simpart of 340\)\.Fix:define sentence boundaries explicitly in the prompt; optionally exclude\\d\+\\\.from the splitter\.

#### Class 5: R3 underdetermined puzzles \(234 cases, 4\.1%\)\.

106 of 189 R3 \(logic grid\) probes have multiple valid solutions because the generated clues do not uniquely determine the assignment\. The verifier accepts only one hardcoded solution\. Models that produce a correct alternative are marked as failures\. For example, clues fix Bob’s pet and Carol’s color, but leave Alice’s and Carol’s pets unconstrained—both \{Alice:cat, Carol:fish\} and \{Alice:fish, Carol:cat\} are valid\.Fix:brute\-force all valid solutions \(36 permutations\) and accept any match\. Fix the puzzle generator to produce uniquely solvable puzzles\.

#### Class 6: R2\+N2 word\-number interaction \(126 cases, 2\.2%\)\.

R2 \(transitive ordering\) requires a numbered list \(“1\. Mouse, 2\. Rabbit…”\)\. When N2 \(no digits\) co\-occurs, models write “one\. Mouse, two\. Rabbit…” to satisfy both constraints\. R2’s parser cannot extract word\-numbered lists—it looks for digit prefixes only\. The model’s ordering is correct, and it correctly avoids digits, but the verifier cannot recognize the format\.Fix:add word\-number parsing to R2’s list extractor\.

#### Class 7: F3/F4 format variants \(124 cases, 2\.2%\)\.

Models use a different but functionally equivalent format\. For F3 \(bullet list\), 101 models used asterisk bullets \(\* item\) instead of the required dash bullets \(\- item\)\. For F4 \(code blocks\), 23 models used indented code blocks \(4 spaces\) instead of fenced blocks \(‘‘‘\)\. These are arguably genuine failures since the constraint explicitly specifies the format, but could also be considered format\-variant compliance\.

#### Class 8: R1 relation format mismatch \(92 cases, 1\.6%\)\.

The probe prompt describes spatial relations as “A is above B\.” Models faithfully reproduce “is above” in their JSON output\. The verifier expects just “above” \(without the “is” prefix\)\. Exact string matching fails despite correct semantics\.Fix:strip common prefixes \(“is ”, “are ”\) before comparison; clarify expected format in the prompt\.

#### Summary\.

The answer\-tag mechanism \(wrapping the final response in<answer\>tags and scoring only tagged content\) eliminates Classes 1 and 2, which together account for 4,613 discrepancies \(80\.5% of all identified issues\)\. The remaining 20% require targeted verifier fixes\. Ten of 30 constraints \(S4, L2, L5, W3, F1, F2, M2, N1, and arguably F3, F4\) have zero discrepancies—their verifiers are fully reliable\.

#### CSE LLM\-judge spot\-check audit\.

To validate verifier accuracy on the expanded CSE benchmark \(36 constraints, 15 models, 223,643 failed constraint checks\), we conducted a systematic spot\-check using Claude Opus 4\.6 as an independent LLM judge across nine parallel audit passes—one per constraint category \(R2, S1, L1, N1/N3, W2/W4, O1/O2/O3, F1/F2/F4, R3, M1/M2/S6\)—sampling 10–20 near\-miss failures per pass \(failures with continuous score≥0\.80\{\\geq\}0\.80, indicating the model nearly satisfied the constraint\)\. For each sampled failure, the judge independently parsed the model’s response text, re\-evaluated the constraint, and compared its verdict against the stored verifier output; all flagged discrepancies were then manually verified by the authors to confirm or dismiss the finding, yielding four cross\-cutting issues ranked by impact: \(1\) theextract\_answer\(\)function matching the*first*rather than*last*<answer\>tag in responses where models quote the prompt instruction, affecting 7,581\+ scored rows across all constraints; \(2\) thesplit\_sentences\(\)function treating JSON objects, markdown tables, and bullet lists as prose sentences, producing 33–59% false failures among near\-miss ordering violations \(O1/O2/O3\) and 64 false S6 verdicts; \(3\) the R3 JSON parser spanning from the first\{to last\}across multi\-object responses, causing 1,051 false failures \(19\.2% of R3 failures; DeepSeek V4 Pro CSR would increase from 15% to 59% with the fix\); and \(4\) five R2 list\-parsing gaps \(inline multi\-item lines, parenthetical numbering, space\-delimited formats\) causing∼\{\\sim\}27% of sampled R2 failures to be false negatives where the model solved the ordering correctly but the parser could not extract it—Three of four issues were fixed and rescored before final analysis: \(1\)extract\_answer\(\)now usesrfind\(\)to match the last structural<answer\>tag, skipping quoted instruction text; \(2\) astrip\_non\_prose\(\)helper removes JSON blocks, markdown tables, and bullet lists before sentence splitting in the O1, O2, O3, S6, F4, and W4 verifiers; \(3\) the R3 JSON parser now uses bracket\-depth tracking to extract the first complete, schema\-matching JSON object rather than spanning from first\{to last\}\. The R2 parser gaps \(issue 4\) remain as a known limitation; the detailed per\-constraint audit with proposed code fixes is documented in the supplementary materials\.

## Appendix RImpossible Probe Experiment

Beyond measuring how performance*degrades*under composition, CSE includes impossible probes—constraint combinations with a proof of unsatisfiability—to study how models*prioritize*when compliance is structurally unachievable\.

Every impossible probe requires a clean logical proof of unsatisfiability; mechanical verifier conflicts \(e\.g\., code\-fence parsing interference\) are excluded because the model cannot reason about verifier internals\. This separates impossible probes \(which test constraint understanding and prioritization\) from merely difficult probes \(which test capacity\)\.

### R\.1Design

The 444 impossible probes \(19 types×\\times3 topic variants×\\timesup to 15 models==55,645 constraint checks\) span three categories of impossibility\.

*Binary contradictions*pair two constraints that are logically incompatible regardless of parameters: forbidden letter vs\. mandatory words containing that letter \(L1\+\{\+\}L2\), embedding digits vs\. forbidding digits \(N1\+\{\+\}N2\), alternating vs\. monotonic sentence lengths \(O1\+\{\+\}O3\), cross\-sentence word bridge vs\. all\-unique words \(R4\+\{\+\}W1\)\. Each has a clean proof: for example, R4 forces the last word of sentenceiito equal the first word of sentencei\+1i\{\+\}1, so any response with≥2\{\\geq\}2sentences must repeat at least one word, violating W1\.

*Parameter\-induced contradictions*use constraint pairs that are compatible for most parameter values but impossible under specific choices enforced at probe generation time: a sentence count of 3 paired with an acrostic word of length 5 \(S2\+\{\+\}L4\), a forbidden letter that appears in the required hidden\-message word \(L1\+\{\+\}M1\), or two constraints that independently fix sentence count to different values \(S2\+\{\+\}S6\)\.

*Hierarchy and load tests*probe whether sacrifice priorities change under compositional pressure\. One probe pairs forbidden letter ‘r’ with a transitive ordering puzzle whose items all contain ‘r’ \(*rabbit, raccoon, raven, rooster, ram, robin*\)—there is no partial solution, so the model must choose which constraint to abandon entirely\. Another embeds the L1\+\{\+\}L2 contradiction among 2, 4, or 6 additional compatible filler constraints, testing whether the sacrifice ordering is stable askkincreases from 4 to 8\. A four\-way probe \(forbidden letter ‘e’\+\+mandatory words\+\+unique words\+\+exact word count\) tests whether the strongest sacrifice priority erodes under multi\-constraint tension\.

### R\.2Results

Across 55,645 constraint checks, 28\.2% of individual constraints were satisfied on impossible probes\. Zero probe×\{\\times\}model pairs passed all constraints on any impossible probe—the deterministic verifiers correctly reject every impossible probe for every model\.

#### Constraint survival ranking\.

[Table14](https://arxiv.org/html/2608.12426#A18.T14)reports per\-constraint pass rates on impossible probes, ordered by survival rate\. Binary constraints that require no sustained tracking dominate the top \(L5: 92\.7%, N2: 69\.5%, L2: 60\.6%\), while precise\-count constraints requiring sustained maintenance cluster at the bottom \(S4: 1\.1%, S6: 2\.9%, M2: 3\.0%, S1: 3\.2%\)\.

Table 14:Per\-constraint survival rate on 444 impossible probes\.
#### Sacrifice rules\.

Three deterministic rules emerge across all 15 models:

1. 1\.Concrete inclusion\>\{\>\}abstract avoidance\.L2 \(mandatory words\) survives at 73–100%; L1 \(lipogram\) is sacrificed at 0–7%\. Models universally keep the named word and accept the letter violation\.
2. 2\.Prohibition\>\{\>\}requirement\.N2 \(no digits\) survives at 84–91%; N1 and N3 are sacrificed at 0%, N4 at 4%\. Models find it easier to omit than to include\.
3. 3\.Natural structure\>\{\>\}imposed structure\.O3 \(alternating\) survives at 56%; O1 \(monotonic\) at 7% and O4 \(growth\) at 0%\. Alternating sentence lengths approximate natural prose more closely than strict monotonic or Fibonacci growth\.

#### Hierarchy inversion under binary choice\.

When forbidden letter ‘r’ is paired with a transitive ordering puzzle whose items all contain ‘r’, the model must abandon one constraint entirely\. R2 \(relational, the fastest degrader under compositional load\) survived at 33%, while L1 \(lexical, compositionally immune\) was sacrificed at 7%—the opposite of the degradation hierarchy\. The same inversion appears when cross\-sentence bridge \(R4\) is paired with unique words \(W1\): R4 survives at 62%, W1 is sacrificed at 2%\. Both cases are suggestive of a sunk\-cost mechanism: the model commits computation to the harder constraint \(solving the ordering puzzle, maintaining the bridge pattern\) and preserves it over the simpler one \(letter avoidance, word uniqueness\)\. These results rest onN=3N\{=\}3probes each and require replication\.

#### Stability under compositional load\.

When the L1\+\{\+\}L2 contradiction is embedded among 2–6 additional compatible constraints \(k=4k\{=\}4tok=8k\{=\}8\), the sacrifice ordering is stable: L2 survives at 82–89% and L1 is sacrificed at 0–4% regardless ofkk\. In a four\-way probe \(forbidden letter ‘e’\+\+mandatory words\+\+unique words\+\+exact word count\), L2 survives at 100% and L1 at 0%—the strongest sacrifice priority is unanimous even under multi\-constraint tension\. Compositional load compresses sacrifice margins slightly but does not alter the ordering\.

#### Budget\-exhaustion regime\.

On the 389 impossible probes where the model must triage across many jointly unsatisfiable constraints—rather than choosing between two that directly contradict—a second sacrifice regime emerges that*preserves*the degradation hierarchy rather than inverting it\. Binary constraints that require no sustained tracking survive at the highest rates \(L5: 90–100%, N2: 56–83%, L2: 53–73%\)\. One\-shot structural constraints also survive \(F1: 41–60%, R1: 50–60%\): format and relational JSON output require a single planning decision rather than sustained maintenance\. The consistently sacrificed constraints are precisely those with the highest sustained\-tracking demand \(S4: 0–7%, S6: 0–6%, M2: 0–6%, S1: 2–7%\), matching the degradation hierarchy from[Table2](https://arxiv.org/html/2608.12426#S4.T2)\.

#### Two regimes of sacrifice\.

The sacrifice ordering depends on the choice structure\. Under*binary choice*—where one of two constraints must go—models preserve whichever constraint required greater planning investment, inverting the degradation hierarchy\. Under*multi\-constraint triage*—where the model must decide what it can fit from a jointly unsatisfiable set—the sacrifice ordering preserves the degradation hierarchy: the hardest constraints are dropped first, the easiest survive\.

## Appendix SComprehension vs\. Execution: A Case Study

[Figure7](https://arxiv.org/html/2608.12426#A19.F7)presents a side\-by\-side comparison of two models responding to the same probe \(CSE\-E\-k1\-0003, constraint S6: palindromic sentence structure\)\. The probe requires exactly 7 sentences whose word counts form a palindrome—sentences 1 and 7, 2 and 6, 3 and 5 must have the same word count\.

Claude 4\.7 Opus reasons about the constraint explicitly: it drafts sentences, counts words, catches errors \(“S2: 11\. Need 10\. Fix…”\), revises, and verifies\. Its chain\-of\-thought trace shows genuine comprehension of the palindromic structure\. Yet the final answer has word counts \[8, 9, 12, 20, 12, 10, 9\]—pairs \(1,7\) are 8 vs\. 9, \(2,6\) are 9 vs\. 10\. Off by one word on two of three pairs \(score = 0\.33\)\.

Llama 405B bypasses reasoning entirely\. It copies sentences 1–3 as sentences 5–7 in reverse order, producing word counts \[13, 14, 27, 57, 27, 14, 13\]—a perfect palindrome \(score = 1\.0\)\. The sentences are not merely matched in word count; they are*identical*, which mechanically guarantees equal counts\.

This is not a violation of the constraint\. The specification requires palindromic*word counts*, not unique sentence content\. Llama’s strategy exploits a specification gap: copying is the most reliable way to guarantee exact word\-count matches\. Preventing this would require an additional constraint \(“all sentences must be distinct”\), which would itself interact with W4 \(no repeated bigrams\) and significantly alter the compositional landscape\.

This case study illustrates that the comprehension\-maintenance gap is not solely about tracking difficulty—it is also about*strategy selection*\. Models that find mechanical shortcuts to satisfy constraints outperform models that reason about them explicitly\. Claude 4\.7’s failure is not one of understanding \(it comprehends the palindrome structure perfectly\) but of execution \(it cannot maintain exact word counts across revisions\)\. The “dumber” strategy wins precisely because it eliminates the execution bottleneck entirely\.

Figure 7:Comprehension vs\. execution on probe CSE\-E\-k1\-0003 \(S6: palindromic sentence word counts,k=1k\{=\}1\)\.Left:Claude 4\.7 Opus drafts, counts, and revises—demonstrating full comprehension of the palindrome structure—but miscounts by one word on 2/3 pairs \(score 0\.33\)\.Right:Llama 405B copies sentences 1–3 as 5–7 reversed, mechanically guaranteeing a perfect palindrome \(score 1\.0\) without explicit reasoning\. The constraint requires palindromic word counts, not unique content; Llama’s copy strategy is a legitimate exploitation of this specification gap\. Models that find mechanical shortcuts to satisfy constraints outperform models that reason about them explicitly—the comprehension\-maintenance gap is about strategy selection, not just tracking difficulty\.