When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning

# When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning
Source: [https://arxiv.org/html/2608.09942](https://arxiv.org/html/2608.09942)
Tughanbulut Kurtulushtkurtul1@stu\.vistula\.edu\.pl Faculty of Computer Engineering Vistula University, Warsaw, Poland ORCID: 0009\-0009\-4861\-7126

###### Abstract

It is widely assumed that chain\-of\-thought \(CoT\) prompting universally improves LLM reasoning\. We investigate this assumption through the conceptual framework of theHd​pH\_\{\\\!dp\}bandwidth bound\(Chen et al\.,[2024](https://arxiv.org/html/2608.09942#bib.bib1)\)\. While the formal bound applies only asymptotically – at astronomically large prompt lengths – it identifies a fundamental architectural bottleneck: serial computation whose depth exceeds a transformer’s single\-forward\-pass capacity must be externalised, precisely what CoT does\. Our central empirical finding is a within\-benchmark serial\-depth gradient: single\-pass \(no\-CoT\) accuracy degrades monotonically with per\-item serial depth, while CoT is approximately depth\-invariant\. We measure CoT effects across three instruction\-tuned models \(Qwen\-2\.5\-7B/32B, Llama\-3\.1\-8B\) and five standard NLP benchmarks at practical context lengths\. The depth\-class hypotheses inspired by the framework are otherwise supported\. On high\-depth P\-complete tasks \(GSM8K, MATH\), CoT provides a massive\+54\+54to\+68\+68percentage\-point recovery gap across all three models\. Conversely, on shallow TC0tasks \(MMLU, ARC\-Challenge\), forcing CoT reasoning is structurally redundant: it yields approximately zero benefit \(Δ∈\[0\.0,\+4\.6\]\\Delta\\in\[0\.0,\+4\.6\]pp across all six cells, no Bonferroni\-significant negative effect\); these high TC0baselines \(up to 95% on ARC\) may, however, reflect pretraining contamination rather than genuinely shallow computation, so this null is not a clean architectural test\. Tasks in the intermediate class𝐋\\mathbf\{L\}\(HumanEval\) exhibit a strict model\-size\-dependent transition:\+23\.2\+23\.2pp for the 32B model,\+9\.1\+9\.1pp for the 8B,−28\.7\-28\.7pp for the 7B\. The pooled cross\-benchmark depth–recovery correlation is Spearmanρ=0\.661\\rho=0\.661\(p=0\.007p=0\.007,n=15n=15\), with 9 of 15 benchmark\-level McNemar tests significant after Bonferroni correction\. Our findings, pre\-registered on OSF, indicate that chain\-of\-thought is not a universal reasoning enhancer but acts as abandwidth bypass– helping serial computation that strains single\-pass capacity while remaining redundant for tasks that already fit\.

## 1Introduction

A central question in LLM theory is*which*computational problems a transformer can solve within a single forward pass\.Chen et al\. \([2024](https://arxiv.org/html/2608.09942#bib.bib1)\)formalise this via the*multi\-party autoregressive communication model*\. Their main result \(Theorem 1\.1\) is the first*unconditional*lower bound for multi\-layer decoder\-only transformers: anLL\-layer transformer withHHattention heads of head dimensionddand precisionppcannot solveLL\-sequential function composition wheneverHd​p=H×d×p≤n2−4​LH\_\{\\\!dp\}=H\\times d\\times p\\leq n^\{2^\{\-4L\}\}, wherennis the prompt length\. Equivalently,Hd​pH\_\{\\\!dp\}is the key bandwidth parameter governing how much serial computation a single forward pass can perform\. As a direct corollary, the paper provides a*provable advantage of chain\-of\-thought*: tasks that are exponentially hard for a single forward pass become exponentially easier once intermediate steps are externalised to the output stream\. Figure[1](https://arxiv.org/html/2608.09942#S2.F1)\(in Section 2\) illustrates this contrast schematically\.

Despite its theoretical elegance, theHd​pH\_\{\\\!dp\}bound describes an asymptotic regime\. Inverting its condition, the smallest prompt length at which the failure guarantee is non\-vacuous for a given model isn⋆=Hd​p24​Ln^\{\\star\}=H\_\{\\\!dp\}^\{\\,2^\{4L\}\}, which for the models studied here ranges fromn⋆≈101034\.4n^\{\\star\}\\approx 10^\{10^\{34\.4\}\}\(Qwen\-7B\) ton⋆≈101077\.8n^\{\\star\}\\approx 10^\{10^\{77\.8\}\}\(Qwen\-32B\) – far beyond any physically realizable sequence length\. It is therefore unknown whether this architectural bottleneck has any bearing on the benchmarks routinely used to compare deployed LLMs, whose prompts span only a few hundred to a few thousand tokens\.

This paper tests that gap\.We \(1\) map five widely\-used NLP benchmarks onto the CC primitives fromChen et al\. \([2024](https://arxiv.org/html/2608.09942#bib.bib1)\), \(2\) derive qualitative, depth\-class hypotheses motivated by theHd​pH\_\{\\\!dp\}bound for each benchmark, and \(3\) test whether two conditions – direct\-answer \(no\-CoT\) versus chain\-of\-thought \(CoT\) – produce the accuracy patterns the serial\-depth account implies\.

The pre\-registered hypothesis had two parts\. On the positive side: high\-depth \(P\-complete\) benchmarks should show large CoT recovery gains because their high serial depth should exceed single\-pass capacity\. On the negative side: low\-depth \(TC0\) benchmarks should be CoT\-insensitive or actively penalised, because additional serial tokens add noise without unlocking computation the model lacks bandwidth for\. We find the positive hypothesis supported across all three models and both math benchmarks\. The negative hypothesis, with correctly extracted answers \(see Appendix[B](https://arxiv.org/html/2608.09942#A2)\), is not supported on MMLU or ARC, where CoT is approximately neutral across all six cells \(Δ∈\[0\.0,\+4\.6\]\\Delta\\in\[0\.0,\+4\.6\]pp\)\. The only*negative*cell that survives correct extraction is HumanEval/Qwen\-7B \(−28\.7\-28\.7pp\), which is a class𝐋\\mathbf\{L\}benchmark rather than a TC0one\. These findings converge with the large\-scale meta\-analysis ofSprague et al\. \([2025](https://arxiv.org/html/2608.09942#bib.bib17)\), who show across 100\+ papers and 20 datasets that CoT helps mainly on math and symbolic reasoning, with little to no benefit on non\-symbolic tasks\.

## 2Background

### 2\.1TheHd​pH\_\{\\\!dp\}Bound

![Refer to caption](https://arxiv.org/html/2608.09942v1/figure1_theorem.png)Figure 1:Schematic of the Chen, Peng & Wu \(2024\) theorem, included as conceptual motivation\.Top: in the single\-forward\-pass regime a depth\-LLtask is bottlenecked at theHd​p=H×d×pH\_\{dp\}=H\\times d\\times pbit neck, and the theorem guarantees failure wheneverHd​p≤n2−4​LH\_\{dp\}\\leq n^\{2^\{\-4L\}\}\.Bottom: chain\-of\-thought externalises each step to the output stream, giving each layer a freshHd​pH\_\{dp\}channel and making depth\-LLtasks exponentially easier\. The bound is asymptotic and binds only at astronomically largenn, far beyond the context lengths we test; the figure illustrates the theoretical mechanism, not a claim about empirical behaviour\.Let a decoder\-only transformer haveLLlayers,HHattention heads per layer, head dimensiondd, and numerical precisionppbits \(16 for FP16\)\. The*bandwidth parameter*is:

Hd​p=H×d×p\.H\_\{\\\!dp\}\\;=\\;H\\times d\\times p\.Chen et al\. \([2024](https://arxiv.org/html/2608.09942#bib.bib1)\)prove unconditionally that anLL\-layer decoder\-only transformer fails atLL\-sequential function composition wheneverHd​p≤n2−4​LH\_\{\\\!dp\}\\leq n^\{2^\{\-4L\}\}\. Crucially, this is an asymptotic bound\. Inverting the condition, the smallest prompt length at which the guarantee is non\-vacuous for a model of bandwidthHd​pH\_\{\\\!dp\}isn⋆=Hd​p24​Ln^\{\\star\}=H\_\{\\\!dp\}^\{\\,2^\{4L\}\}\.111Setting the threshold equal to the model’s bandwidth,n2−4​L=Hd​pn^\{2^\{\-4L\}\}=H\_\{\\\!dp\}, and solving givesn⋆=Hd​p24​Ln^\{\\star\}=H\_\{\\\!dp\}^\{\\,2^\{4L\}\}\.For the models tested this ranges fromn⋆≈101034\.4n^\{\\star\}\\approx 10^\{10^\{34\.4\}\}\(Qwen\-7B,L=28L\{=\}28\) ton⋆≈101077\.8n^\{\\star\}\\approx 10^\{10^\{77\.8\}\}\(Qwen\-32B,L=64L\{=\}64\) – versus then≈102n\\approx 10^\{2\}–10310^\{3\}tokens of our benchmarks\. The formal bound therefore does not bind at any physically realizable context length, and we treat it as a conceptual motivation rather than a literal predictor\. It nonetheless identifies a real architectural property: decoder\-only transformers possess a serial\-depth bottleneck, and tasks requiring more serial depth than the single\-forward\-pass bandwidth permits must externalise intermediate computation – precisely what chain\-of\-thought does\. We hypothesise that this bottleneck governs empirical reasoning failures even at practical sequence lengths, making it the mechanistic motivation for the CoT recovery gains we measure on P\-complete benchmarks\.

This architectural constraint was anticipated empirically byNye et al\. \([2022](https://arxiv.org/html/2608.09942#bib.bib14)\), who noted that a model“is asked to perform these tasks in one forward pass \[and\] cannot adapt the amount of compute”, and showed that routing intermediate steps through an explicit scratchpad recovers accuracy on multi\-step arithmetic that fails under direct prediction\. TheHd​pH\_\{\\\!dp\}bound offers a conceptual explanation for why the scratchpad helps precisely on tasks requiring serial depth\.

### 2\.2CC Primitives and Benchmark Mapping

We assign each benchmark a coarse heuristic mapping into the CC\-primitive taxonomy ofChen et al\. \([2024](https://arxiv.org/html/2608.09942#bib.bib1)\), intended solely as a vehicle for deriving pre\-registered, depth\-class hypotheses:

Table 1:Benchmark\-to\-primitive mapping \(*heuristic*\)\. The “Heuristic primitive” column is the initial single\-primitive label used to derive pre\-registered hypotheses\. An LLM\-as\-judge inter\-rater check \(§[4](https://arxiv.org/html/2608.09942#S4)\) yields pooledκ=0\.293\\kappa=0\.293, indicating that real benchmarks are mixtures of primitives rather than instances of any one\. The load\-bearing column for the hypotheses tested \(H1–H4\) is*Depth class*, not the specific primitive label; the depth\-class hypotheses are robust to primitive relabeling within the same class\.kk\-sequential composition \(GSM8K, MATH\) requires chainingkkarithmetic operations where each depends on the previous – a P\-complete task\.Cobbe et al\. \([2021](https://arxiv.org/html/2608.09942#bib.bib4)\)define GSM8K problems as requiring 2–8 sequential steps, directly quantifying thekkrange\. Critically, they also provide their own empirical confirmation of the serial\-depth bottleneck: finetuning a 6B model to output the final answer*without*intermediate steps drops accuracy from 20\.6% to 5\.2% – a 75% relative collapse caused purely by removing the intermediate computation steps\. This is precisely the no\-CoT vs\. CoT contrast we study, and their result directly foreshadows our finding that P\-complete benchmarks degrade sharply under single\-pass generation\. Set disjointness and sparse parity \(MMLU, ARC\) require only constant\-depth circuits and are in TC0; they benefit little from serial scratchpad use\. MMLU\(Hendrycks et al\.,[2021a](https://arxiv.org/html/2608.09942#bib.bib8)\)spans 57 subjects from elementary mathematics to professional law, evaluated zero\-shot or few\-shot from pretraining knowledge alone\. We provisionally treat MMLU as predominantly set\-disjointness\-like \(TC0\): most questions require selecting the answer option whose content overlaps with a fact stored in the model, with no serial composition between options\. This is a coarse aggregate assignment; the per\-item picture is reported in §[4](https://arxiv.org/html/2608.09942#S4)\. This classification is supported within the MMLU paper itself: Hendrycks et al\. find that GPT\-3 performs worst on calculation\-heavy STEM subjects \(which require sequential arithmetic – a P\-complete operation\) and best on verbal knowledge subjects \(factual retrieval – TC0\)\. The aggregate MMLU signal is therefore dominated by the TC0majority of subjects, implying no CoT benefit\. This expectation is independently confirmed bySprague et al\. \([2025](https://arxiv.org/html/2608.09942#bib.bib17)\), who show that 95% of MMLU’s total CoT performance gain is attributable to questions whose text or model response contains an equals sign – i\.e\., the math\-related minority – while non\-math MMLU questions receive no reliable benefit from CoT\.Clark et al\. \([2018](https://arxiv.org/html/2608.09942#bib.bib3)\)constructed the ARC Challenge partition to contain*only*questions that both a retrieval\-based solver and a word\-co\-occurrence solver answer incorrectly; in 2018, no baseline – including neural models trained on SQuAD and SNLI – significantly outperformed a random baseline \(≈25%\\approx 25\\%\) on the Challenge Set\. Our models achieve 82–95% accuracy under no\-CoT on the same partition\. We do not interpret this gap as evidence for any particular cause: modern instruction\-tuned models genuinely are far more capable than 2018 baselines, and pretraining exposure to ARC is also plausible given the benchmark’s ubiquity\. We return to this in §[5](https://arxiv.org/html/2608.09942#S5)\. HumanEval\(Chen et al\.,[2021](https://arxiv.org/html/2608.09942#bib.bib2)\)evaluates Python function synthesis from docstrings\. We provisionally label the dominant primitive as*pointer chasing*– the model must resolve a chain of variable bindings and function calls in order\. The per\-item judge \(§[4](https://arxiv.org/html/2608.09942#S4)\) more often labels HumanEval items askk\-composition; both pointer chasing andkk\-composition involve serial dependency, and pointer chasing is computable in log\-space \(class𝐋\\mathbf\{L\}\), placing the benchmark intermediate between TC0and P\-complete\. We hypothesise a moderate, model\-size\-dependent CoT benefit on this class\. This classification is corroborated byChen et al\. \([2021](https://arxiv.org/html/2608.09942#bib.bib2)\)themselves: they report that Codex accuracy on synthetic chaining tasks degrades by a factor of 2–3 per additional operation and that the model fails to bind operations to variables correctly as chain length grows – the signature of a serial\-depth bottleneck on sequential computation\.

##### What is load\-bearing\.

Throughout the remainder of this paper, the load\-bearing column of Table[1](https://arxiv.org/html/2608.09942#S2.T1)isDepth class\(P\-complete, TC0, class𝐋\\mathbf\{L\}\), not the specific primitive label\. The primitive labels are heuristic, per\-item\-noisy, and – as the LLM\-as\-judge check in §[4](https://arxiv.org/html/2608.09942#S4)shows \(κ=0\.293\\kappa=0\.293\) – not stable across graders\. The depth classes are coarse, hold up to primitive relabeling within the same class, and are what hypotheses H1–H4 are derived from\.

## 3Experimental Setup

### 3\.1Models

We evaluate three instruction\-tuned decoder\-only transformer models spanning a1\.43×1\.43\\timesrange ofHd​pH\_\{\\\!dp\}bandwidth \(Table[2](https://arxiv.org/html/2608.09942#S3.T2)\)\.

Table 2:Model architectures andHd​pH\_\{\\\!dp\}values \(p=16p=16for FP16\)\.HH= query heads; Llama\-3\.1\-8B uses GQA with 8 KV heads, but output bandwidth per token is determined by query heads × head dim\. All models are instruction\-tuned variants without internal reasoning phases; the single\-pass condition we test applies directly to their single\-forward\-pass computation\.
### 3\.2Conditions

no\-CoT: a direct\-answer system prompt with a short output cap: 32 tokens for GSM8K, MMLU, and ARC; 64 for MATH \(to allow\\boxed\{\}formatting\); and 256 for HumanEval \(the minimum required to emit a viable function body\)\. All caps are≥8×\\geq 8\\timessmaller than the CoT budget\. For GSM8K, MATH, MMLU, and ARC the short caps prevent the model from externalising multi\-step reasoning to the output stream, directly operationalising the single\-forward\-pass condition; the 256\-token HumanEval cap is a necessarily looser operationalisation \(a function body cannot be written in a few tokens\), a caveat we revisit in §[5](https://arxiv.org/html/2608.09942#S5)\.

CoT: a standard chain\-of\-thought prompt\(Wei et al\.,[2022b](https://arxiv.org/html/2608.09942#bib.bib20)\)with a 2048\-token cap\. Intermediate steps are externalised to the output stream, bypassing the single\-pass constraint\. This is mechanistically equivalent to the*scratchpad*ofNye et al\. \([2022](https://arxiv.org/html/2608.09942#bib.bib14)\): both route intermediate state through the output stream rather than requiring it to be compressed into residual activations within a single forward pass\.

##### Why the short\-output cap denies externalised computation\.

We use “single\-pass condition” as shorthand: generatingmmoutput tokens is strictlymmsequential forward passes, so the short caps do not enforce a literal single forward pass\. What they enforce is the absence of an*externalised scratchpad*– the model cannot write intermediate state to the output stream and read it back over many tokens, which is the operational contrast the scratchpad/Hd​pH\_\{\\\!dp\}account concerns\. For instruction\-tuned models \(which lack a hidden reasoning phase, unlike explicit reasoning models such as DeepSeek\-R1 or o1\), the short caps deny this externalisation channel\. A reader might object that the cap simply provides insufficient output length to write the final answer\. Three observations rule this out\. First, TC0tasks \(MMLU, ARC\) achieve 68–95% accuracy under the 32\-token cap, demonstrating that 32 tokens is sufficient when the computation fits in a single pass\. Second, the cross\-benchmark depth–recovery correlation \(Spearmanρ=0\.661\\rho=0\.661across 15 cells; §[4](https://arxiv.org/html/2608.09942#S4)\) shows the CoT gain scales with assigned CC depth, not with output length\. Third, the HumanEval direction reversal between Qwen\-7B \(−28\.7\-28\.7pp\) and Qwen\-32B \(\+23\.2\+23\.2pp\) holds the cap fixed while varying model scale, so the effect is not an output\-length artefact\. Reasoning models with explicit hidden chain\-of\-thought phases would require a different experimental design and are outside our scope\.

### 3\.3Data, Inference, and Pre\-registration

We sample 800 items sequentially from each benchmark’s test split \(164 for HumanEval, full split\)\. All inference is run at temperature = 0\.0 \(greedy decoding\) in FP16 via vLLM\(Kwon et al\.,[2023](https://arxiv.org/html/2608.09942#bib.bib11)\)on rented GPU instances \(Vast\.ai\)\. Total compute cost: under $5 USD\.

##### Pre\-registration\.

This study was formally pre\-registered on the Open Science Framework \(OSF Registries\) on 9 May 2026 under license CC\-BY 4\.0\. The H1–H4 hypotheses were derived from theHd​pH\_\{\\\!dp\}bound prior to any data collection; production inference was partially complete at the time of registration \(Qwen\-2\.5\-7B accuracy on GSM8K and MMLU had been observed, with no stratified analysis performed\), as disclosed in the registration’s foreknowledge statement\. The registration is available at:

- •Registration DOI:10\.17605/OSF\.IO/92JDK
- •
- •
- •

The pre\-registration document specifies: \(i\) the four primary hypotheses \(H1–H4\) tested in §[4](https://arxiv.org/html/2608.09942#S4); \(ii\) the McNemar exact test with Bonferroni correction atα=0\.05/15\\alpha=0\.05/15as the primary statistical procedure; \(iii\) the Spearman rank correlation as the secondary depth–recovery test; \(iv\) the inter\-rater Cohen’sκ≥0\.7\\kappa\\geq 0\.7target for LLM\-as\-judge primitive validation; and \(v\) the benchmark\-to\-primitive mapping used to derive hypotheses\. Of these pre\-registered analyses, only theκ\\kappatarget was not met; this deviation is reported transparently in §[4](https://arxiv.org/html/2608.09942#S4)and §[5](https://arxiv.org/html/2608.09942#S5)\.

### 3\.4Scoring

GSM8K and MMLU/ARC use exact\-match numeric and letter extraction respectively, with regular\-expression parsers robust to special\-token leakage\. MATH uses a brace\-balanced\\boxed\{\}extractor with numeric equivalence for fractions\. HumanEval is functionally scored: the model output is parsed for the longest fenced‘‘‘pythonblock \(or, if absent, treated as a free\-form completion appended to the function prompt\), then executed against the canonical test suite in a 10 s subprocess sandbox\. All 984 HumanEval inference records \(164 items×\\times3 models×\\times2 conditions\) are scored; results match the pass@1 figures reported in Table[3](https://arxiv.org/html/2608.09942#S4.T3)\. All parsers were fixed and rows re\-scored before any analysis; Appendix[B](https://arxiv.org/html/2608.09942#A2)documents two scoring artefacts present in an earlier preprint \(Zenodo, May 2026\) and their corrections\. Format error rate across all non\-HumanEval scored inferences: 0\.90% under CoT \(86/9,600\) and 0\.46% under no\-CoT \(44/9,600\); excluded from accuracy calculations but retained in the database\.

##### Inter\-rater agreement \(Cohen’sκ\\kappa\)\.

The pre\-registration committed to validating the per\-benchmark single\-primitive labels via an independent LLM\-as\-judge\. We used Gemma\-2\-27B\-it \(google/gemma\-2\-27b\-it, December 2024 release; family\-independent of all test models\) at temperature 0, prompted with neutral CC\-theory definitions and required to output a one\-sentence reasoning step plus a JSON\(primitive, depth\)judgement for 200 sampled items per benchmark \(964 total; 0 parse failures\)\. For HumanEval the canonical implementation was provided alongside the function spec so that the judge could observe the actual computational structure rather than only the docstring\.

## 4Results

### 4\.1Main Effect

Figure[2](https://arxiv.org/html/2608.09942#S4.F2)shows accuracy under both conditions for all three models\. The math\-side hypothesis holds across all three models: P\-complete benchmarks \(GSM8K, MATH\) show large positive CoT recovery gaps \(\+54\+54to\+68\+68pp\)\. The stronger negative TC0hypothesis does not hold: with correctly extracted answers, CoT is approximately neutral on MMLU and ARC\-Challenge across all six \(model, benchmark\) cells \(Δ∈\[0\.0,\+4\.6\]\\Delta\\in\[0\.0,\+4\.6\]pp\)\. HumanEval shows the hypothesised model\-size\-dependent transition\.

![Refer to caption](https://arxiv.org/html/2608.09942v1/figure1_main.png)Figure 2:Accuracy by CoT condition and CC primitive\. GreenΔ\\Delta= CoT gain; redΔ\\Delta= CoT penalty\.n=800n=800per cell\.Table[3](https://arxiv.org/html/2608.09942#S4.T3)reports accuracy with 95% Wilson confidence intervals\. The mean CoT recovery gap across models is\+60\.6\+60\.6pp for GSM8K,\+60\.5\+60\.5pp for MATH,\+3\.2\+3\.2pp for MMLU,\+1\.4\+1\.4pp for ARC, and\+1\.2\+1\.2pp for HumanEval \(averaging−28\.7\-28\.7,\+9\.1\+9\.1, and\+23\.2\+23\.2across the three models\)\.

Table 3:Accuracy \(%\) with 95% Wilson CIs \[in brackets\]\.Δ\\Delta= CoT−\-no\-CoT\.n=800n=800per cell for GSM8K/MATH/MMLU/ARC;n=164n=164for HumanEval \(full split\)\.
### 4\.2Statistical Tests

H1 \(CoT effect; high\-kkcollapse\)\.We first test, at the benchmark level, whether CoT changes accuracy at all: McNemar’s exact test, Bonferroni\-correctedα=0\.05/15=0\.0033\\alpha=0\.05/15=0\.0033for the 15 benchmark\-level tests \(5 benchmarks×\\times3 models\)\. The depth\-stratified collapse that H1 concerns is tested at the bin level below\.9 of 15 are significant\.The 9 significant cells are the six P\-complete cells \(GSM8K and MATH across all three models, all withb≫cb\\gg candp<10−117p<10^\{\-117\}\), Qwen\-7B’s MMLU \(b=80b\{=\}80,c=44c\{=\}44,p=1\.6×10−3p=1\.6\\times 10^\{\-3\}, in the direction CoT\>\>no\-CoT\), and two HumanEval cells \(Qwen\-7B and Qwen\-32B\)\. The six*non\-significant*cells are all on MMLU or ARC where the framework hypothesised no CoT benefit: Qwen\-7B/ARC \(b=39b\{=\}39,c=39c\{=\}39,p=1\.00p=1\.00– a literal tie\), Llama\-8B/MMLU \(b=84b\{=\}84,c=61c\{=\}61,p=0\.067p=0\.067\), Llama\-8B/ARC \(b=61b\{=\}61,c=38c\{=\}38,p=0\.027p=0\.027\), Qwen\-32B/MMLU \(b=46b\{=\}46,c=27c\{=\}27,p=0\.034p=0\.034\), Qwen\-32B/ARC \(b=20b\{=\}20,c=14c\{=\}14,p=0\.39p=0\.39\), as well as Llama\-8B/HumanEval \(b=25b\{=\}25,c=10c\{=\}10,p=0\.017p=0\.017\)\.Partially confirmed:the math\-side hypothesis holds across the board; the TC0hypothesis \(no CoT penalty\) holds for all six MMLU/ARC cells, but the framework’s stronger claim that CoT should*actively hurt*TC0tasks is not supported \(no cell shows a significant negative direction at Bonferroniα\\alpha\)\. The HumanEval cells follow the hypothesised model\-size\-dependent pattern\.

The pre\-registration further specified 60 bin\-level tests \(5 benchmarks×\\times3 models×\\times4 depth bins\)\. With the updated per\-item depth labels \(calculator\-step counts for GSM8K, AST nesting depth for HumanEval, equation\-count proxy for MATH\), 42 of these 60 cells are populated; the remaining 18 are structurally empty because MMLU and ARC heuristic depth labels collapse tok=1k\{=\}1, consistent with their TC0classification\. Of the 42 computable bin\-level tests, 25 are significant at the Bonferroni\-correctedα=0\.05/60=8\.3×10−4\\alpha=0\.05/60=8\.3\\times 10^\{\-4\}\(adjusted using the pre\-registered denominator\)\. The non\-significant cells are concentrated in HumanEval atk=1k\{=\}1andk≥7k\{\\geq\}7\(small bin counts:n=1n\{=\}1ton=21n\{=\}21\), in Llama\-8B/HumanEval where the CoT effect is small \(Δ=\+9\.1\\Delta=\+9\.1pp\), and in all five MMLU/ARC \(k=1k\{=\}1\) cells that are non\-significant at the benchmark level above\.

H2 \(recovery gap∝k\\propto k\):Spearman rank correlation between assigned CC depth class \(P\-complete benchmarks and HumanEval codedk=4k=4,222HumanEval is coded at the P\-complete depth \(k=4k\{=\}4\) because the per\-item judge predominantly labels its items askk\-composition \(§[4](https://arxiv.org/html/2608.09942#S4)\); this is the conservative choice, as it sets the strictest depth expectation for the class\-𝐋\\mathbf\{L\}benchmark\.TC0codedk=1k=1\) and CoT recovery gap, computed across all five benchmarks\. Pooledρ=0\.661\\rho=0\.661,p=0\.007p=0\.007\(n=15n=15\)\. Per\-model: Llama\-8Bρ=0\.866\\rho=0\.866\(p=0\.058p=0\.058,n=5n=5\), Qwen\-32Bρ=0\.866\\rho=0\.866\(p=0\.058p=0\.058\), Qwen\-7Bρ=0\.289\\rho=0\.289\(p=0\.638p=0\.638\) – Qwen\-7B’s gradient is broken by its HumanEval penalty \(−28\.7\-28\.7pp, hypothesised positive at depthk=4k\{=\}4\)\.Pooled gradient confirmed; per\-model gradient holds for the two larger models only\.

H3 \(MMLU: no\-CoT≥\\geqCoT\):One\-sided McNemar testsp=1\.00p=1\.00,0\.980\.98,0\.990\.99for Qwen\-7B, Llama\-8B, Qwen\-32B respectively – all non\-significant\. The direction is in fact reversed \(CoT slightly outperforms no\-CoT by\+2\.4\+2\.4to\+4\.6\+4\.6pp\), but no individual cell reaches Bonferroni significance\.Falsified\.H3 was the most directly model\-mechanism\-tied of the four hypotheses: it held that the bandwidth bypass would*actively hurt*TC0benchmarks because additional serial tokens add noise without unlocking computation the model lacks bandwidth for\. The data does not support this hypothesis; the bandwidth\-bypass mechanism appears to be one\-sided \(helping high\-depth tasks\) rather than two\-sided\.

H4 \(GSM8K largest positive gap\):Bootstrap 95% CIs on the CoT recovery gap overlap between GSM8K and MATH for all models \(e\.g\., Qwen\-7B: GSM8K \[64\.4, 71\.5\] pp vs MATH \[62\.7, 70\.1\] pp\)\. GSM8K has the largest average gap \(\+60\.6 pp vs MATH \+60\.5 pp\) but the two benchmarks are statistically indistinguishable – consistent with both being deepkk\-composition tasks of similar assigned depth\. HumanEval/Qwen\-32B was originally reported as exceeding both in an earlier draft, but correct parsing puts it atΔ=\+23\.2\\Delta=\+23\.2pp\.Not distinguishable from MATH; see §[5](https://arxiv.org/html/2608.09942#S5)\.

##### Cohen’sκ\\kappa\(pre\-registered deviation\)\.

Pooled Cohen’sκ\\kappabetween the per\-benchmark heuristic primitive labels and Gemma\-2\-27B\-it’s per\-item primitives isκ=0\.293\\kappa=0\.293, well below the pre\-registered targetκ≥0\.7\\kappa\\geq 0\.7\. Per\-benchmark agreement: GSM8K 96%, MATH 87%, MMLU 59%, ARC 1\.5%, HumanEval 4\.3%\. The judge’s recorded reasoning \(released alongside the labels\) shows the disagreement is principled rather than noisy: it labels most ARC items asset\_disjointness\(factual recall\) rather thansparse\_parity, reflecting that ARC items are answerable by retrieval; it labels most HumanEval items ask\_composition\(sequential function build\-up\) rather thanpointer\_chasing, reflecting that the canonical solutions of most HumanEval problems chain operations rather than dereference references through state\. We interpret this as evidence that real benchmarks are mixtures of CC primitives rather than instances of a single primitive, consistent with recent work on benchmark heterogeneity\. The depth\-based hypotheses tested in H1–H4 are independent of primitive labels: the failure pattern depends on per\-item serial depth, not on which primitive a benchmark exemplifies\. We retain the heuristic primitive labels as a coarse taxonomy \(cf\. Table[1](https://arxiv.org/html/2608.09942#S2.T1)\) but report theκ\\kappadeviation transparently\. The full judge labels and reasoning text are included in the released artefact for re\-analysis\.

### 4\.3Per\-benchmark Depth Gradient

We stratify each benchmark by per\-item CC depth \(k=1k\{=\}1, 2–3, 4–6,≥7\\geq\{7\}\)\. The depth proxy is benchmark\-specific: equation count for MATH, calculator\-step count for GSM8K, AST nesting depth for HumanEval; MMLU and ARC items collapse tok=1k\{=\}1under any structural depth metric and are excluded from the gradient analysis \(see §[5](https://arxiv.org/html/2608.09942#S5)\)\. Table[4](https://arxiv.org/html/2608.09942#S4.T4)shows the same monotone pattern across all three benchmarks for which depth is defined\. The Qwen\-32B MATH cell illustrates the canonical pattern: no\-CoT accuracy degrades from 45\.5% atk=1k\{=\}1to 15\.4% atk≥7k\{\\geq\}7\(a 30 pp range\), while CoT stays in the 82–88% band – externalising computation compensates for depth regardless ofkk\. The GSM8K and HumanEval panels show the same direction of effect with smaller dynamic range\.

As a pre\-registered secondary analysis, mean CoT output length \(completion tokens\) increases monotonically with bin on MATH: 558 tokens atkk=1, 586 atkk=2–3, 662 atkk=4–6, and 839 atkk≥\\geq7\. A per\-item Spearman rank correlation betweenkkand completion\-token count yieldsρ=0\.221\\rho=0\.221,p<10−26p<10^\{\-26\}\(n=2,317n=2\{,\}317items\), confirming that the model allocates more computation to deeper problems even within MATH\.

Table 4:Accuracy \(%\) by CC depth bin for Qwen\-32B across the three benchmarks with non\-trivial depth diversity\. No\-CoT degrades monotonically withkkon all three; CoT is approximately bin\-invariant\. HumanEvalk≥7k\{\\geq\}7hasn=1n\{=\}1and is omitted\. Bin counts: MATHn=231n\{=\}231/336/175/52; GSM8Kn=48n\{=\}48/444/294/14; HumanEvaln=21n\{=\}21/113/29/1\.
### 4\.4Model Size and No\-CoT Accuracy

Within the Qwen family, no\-CoT accuracy on GSM8K rises with model size: 23\.1% \(Qwen\-2\.5\-7B,Hd​pH\_\{\\\!dp\}=57K\) to 39\.9% \(Qwen\-2\.5\-32B,Hd​pH\_\{\\\!dp\}=82K\), qualitatively consistent with larger models performing more serial computation within a single pass\. The cross\-family comparison is confounded: Llama\-3\.1\-8B \(nominalHd​pH\_\{\\\!dp\}=65K\) scores only 13\.8%, plausibly because Grouped\-Query Attention \(8 KV heads against 32 query heads\) reduces its effective per\-token bandwidth\. With only three models and a family confound, we do not claim thatHd​pH\_\{\\\!dp\}quantitatively predicts accuracy; we report the trend as suggestive and qualitatively in line with the serial\-depth account\. Figure[3](https://arxiv.org/html/2608.09942#S4.F3)shows the depth\-gradient effect \(panel a\) and the model\-size trend \(panel b\)\.

![Refer to caption](https://arxiv.org/html/2608.09942v1/figure2_depth_scaling.png)Figure 3:No\-CoT accuracy, serial depth, and model size\.\(a\)MATH accuracy stratified by CC depth bin for all three models \(95% Wilson bands from per\-item data,n=324n\{=\}324–2,0222\{,\}022per bin\)\. No\-CoT \(solid\) collapses monotonically as serial depth grows; CoT \(dashed\) is approximately depth\-invariant across the entire range, separating the two regimes by 50–75 pp at the deepest bin\.\(b\)On GSM8K, no\-CoT accuracy rises with model size within the Qwen family \(Llama\-8B is off\-trend, plausibly because GQA reduces its effective per\-token bandwidth\), while CoT recovers a near\-uniform 70–94% irrespective of architecture\. With three models and a family confound we read panel \(b\) as suggestive, not as a quantitative law\.
### 4\.5Memorization Separation \(GSM\-Symbolic\)

An alternative reading of the GSM8K gap is item\-level memorization: models recall pretraining\-seen answers under no\-CoT, and CoT plays no architectural role\. We test this by re\-running all three models on GSM\-Symbolic\(Mirzadeh et al\.,[2024](https://arxiv.org/html/2608.09942#bib.bib13)\), which substitutes names and numerals in GSM8K templates while preserving arithmetic depth, holding the protocol fixed \(800 items per cell, both conditions, greedy decoding\)\.

Per\-cell deltas stay within±10\\pm 10pp \(mean−4\.3\-4\.3pp no\-CoT,−1\.4\-1\.4pp CoT; Table[5](https://arxiv.org/html/2608.09942#S4.T5)\) – well within the GSM8K→\\toGSM\-Symbolic drops reported inMirzadeh et al\. \([2024](https://arxiv.org/html/2608.09942#bib.bib13)\)\(their Fig\. 3 spans−0\.3\-0\.3to−9\.2\-9\.2pp\)\. The CoT recovery gap is preserved in direction and magnitude on every model: Qwen\-7B\+68\.0→\+65\.4\+68\.0\\to\+65\.4, Llama\-8B\+59\.9→\+65\.4\+59\.9\\to\+65\.4, Qwen\-32B\+53\.9→\+59\.8\+53\.9\\to\+59\.8pp\. Memorization would flatten the gap under perturbation; it does not\. The depth\-dependent pattern is therefore an architectural effect, not retrieval of memorized items\.

Table 5:GSM8K vs\. GSM\-Symbolic accuracy \(n=800n\{=\}800items per cell, Wilson 95% CIs\)\.All six per\-cell deltas fall within±10\\pm 10pp; the CoT recovery gap on the math side is preserved on every model\. The positive \(math\-side\) hypothesis of theHd​pH\_\{\\\!dp\}framework is not an artefact of GSM8K memorisation\.

## 5Discussion

The math\-side hypothesis holds; the TC0hypothesis does not\.Our pre\-registered hypotheses posited a two\-sided effect: CoT should help on high\-depth tasks and hurt on low\-depth tasks\. With correctly extracted answers, only the first half is borne out\. P\-complete benchmarks \(GSM8K, MATH\) recover\+54\+54to\+68\+68pp across all three models, significant beyond any reasonable correction\. TC0benchmarks \(MMLU, ARC\) are essentially flat: CoT changes accuracy by0\.00\.0to\+4\.6\+4\.6pp across all six \(model, benchmark\) cells\. The framework’s positive hypothesis \(CoT bypasses bandwidth on high\-depth tasks\) is supported; the negative hypothesis \(additional serial tokens*hurt*TC0tasks\) is not\. This narrows the framework’s empirical content to a one\-sided claim and converges with the large\-scale meta\-analysis ofSprague et al\. \([2025](https://arxiv.org/html/2608.09942#bib.bib17)\), who report that CoT helps mainly on math and symbolic reasoning across 14 models and 20 datasets, with little to no effect on non\-symbolic tasks\.

Why the TC0hypothesis may have failed\.Several non\-exclusive explanations are consistent with the data\.*Ceiling effects:*no\-CoT accuracy on ARC is 82–95% across the three models\. Any negative CoT effect must come out of a small remaining headroom; the bandwidth\-bypass mechanism would need to specifically disrupt ceiling\-level performance, which is a stronger claim than the positive \(recovery\) version\.*Instruction\-tuning produces a two\-mode model:*our models are RLHF\-tuned to follow instructions for both “answer directly” and “think step by step”\. A direct\-answer prompt under a 32\-token cap and a CoT prompt under a 2048\-token cap may both produce competent answers for items that fit in a single forward pass, with neither mode forcing the model into a regime where serial expansion is harmful\.*Mixture\-of\-primitives:*the pre\-registered LLM\-as\-judge analysis \(§[4](https://arxiv.org/html/2608.09942#S4),κ=0\.293\\kappa=0\.293\) already indicated that real benchmarks are mixtures rather than pure instances of a single CC primitive\. If individual MMLU and ARC items vary internally in computational depth, the aggregate CoT effect averages across items where CoT helps \(deep\) and items where CoT is neutral \(shallow\), with the net at approximately zero\.

Alternative explanations: artefacts and contamination\.Gururangan et al\. \([2018](https://arxiv.org/html/2608.09942#bib.bib6)\)show that NLI benchmarks contain surface\-level artifacts exploitable by hypothesis\-only classifiers \(67% accuracy on SNLI without the premise\)\. An analogous effect may operate here: the high no\-CoT accuracy on MMLU \(68–82%\) and ARC \(82–95%\) could partly reflect models exploiting lexical shortcuts rather than performing genuine set\-disjointness or sparse parity computation\.

A third possibility is data contamination\.Magar & Schwartz \([2022](https://arxiv.org/html/2608.09942#bib.bib12)\)show that pretrained models can*exploit*\(not merely memorize\) test labels seen during pretraining, with exploitation increasing with duplication frequency and model size\. MMLU and ARC are among the most widely distributed NLP benchmarks and are almost certainly present in the pretraining corpora of all three models tested\. The scale of the no\-CoT accuracy on ARC is striking:Clark et al\. \([2018](https://arxiv.org/html/2608.09942#bib.bib3)\)report that in 2018 no system significantly outperformed random \(≈\\approx25%\) on the ARC Challenge Set – a partition specifically designed to resist surface\-level methods – yet our models achieve 82–95% under no\-CoT on the same questions\. A more than triple accuracy jump over the strongest 2018 baselines on a dataset designed to be hard for retrieval and co\-occurrence is consistent with – but does not by itself demonstrate – pretraining exposure\. Modern models also reflect much larger pretraining corpora, better architectures, and instruction tuning, any of which could account for substantial portions of the gap\.Hendrycks et al\. \([2021a](https://arxiv.org/html/2608.09942#bib.bib8)\)conducted a memorisation check in 2021: prompt\-compression entropy was not positively correlated with accuracy for GPT\-3, suggesting exact question memorisation was minimal at that time\. However, that analysis predates the instruction\-tuning era; our models were trained on corpora assembled in 2024–2025, in which MMLU has become a standard evaluation target ubiquitous in model\-card reports and leaderboards, making exposure far more likely than in 2021\. Notably, unspecialized crowd workers \(MTurk\) achieve only 34\.5% on MMLU while expert\-level accuracy is≈\\approx89\.8%\(Hendrycks et al\.,[2021a](https://arxiv.org/html/2608.09942#bib.bib8)\); our models’ 68–82% no\-CoT accuracy places them well above unspecialized humans, suggesting genuine knowledge acquisition from pretraining – but does not rule out additional exploitation of memorised label associations for specific subjects\.

We consider the annotation\-artifact and contamination accounts*complementary*to the architectural one rather than competing: both help explain why no\-CoT accuracy on TC0benchmarks is already so close to ceiling \(82–95% on ARC\), which in turn leaves very little headroom for any architectural effect \(positive or negative\) to manifest at the aggregate level\. Disentangling these accounts requires benchmarks explicitly designed to resist lexical shortcuts and verified to be absent from pretraining corpora\. We leave this to future work\.

The one significant CoT penalty \(HumanEval/Qwen\-7B\)\.The single Bonferroni\-significant negative effect in our data is Qwen\-7B on HumanEval \(−28\.7\-28\.7pp,p=4×10−9p=4\\times 10^\{\-9\}\): chain\-of\-thought*lowers*the smallest model’s code accuracy\. We flag this as a genuine but unexplained finding rather than fold it into the serial\-depth account, for two reasons\. First, the no\-CoT HumanEval condition permits up to 256 tokens to emit a function body, so it is not a clean single\-pass contrast; it is better read as “write code directly” versus “explain first, then write code,” which the bandwidth framing does not address\. Second, the penalty appears only for the weakest model, consistent with the broader finding that CoT is not universally beneficial and can reduce accuracy in some settings\(Sprague et al\.,[2025](https://arxiv.org/html/2608.09942#bib.bib17)\)\. A mechanistic account of why externalised reasoning harms smaller code models is left to future work\.

H4 deviation\.GSM8K and MATH have statistically indistinguishable CoT recovery gaps\. Both arekk\-composition tasks; their similar gaps are consistent with theory and strengthen H2 rather than undermining it\.

Limitations\.*Instruction\-tuning confound:*All models are instruction\-tuned variants trained to produce CoT\-style reasoning\. The short no\-CoT caps suppress this trained behaviour\. The math\-side recovery effect within the same models is therefore established under conditions that genuinely deny the model the externalisation channel; whether the absence of a TC0penalty would survive a more aggressive denial of CoT\-style reasoning under no\-CoT \(e\.g\. explicit “answer in one token” instructions\) is untested\.*Test\-time compute confound:*The no\-CoT \(short\-cap\) and CoT \(2048\-token\) conditions differ in both whether intermediate state is externalised*and*the total inference\-time compute available\. Because a decoder\-only transformer can perform additional serial computation only by emitting tokens, these two factors are intrinsically entangled, and our design does not separate them; the results are equally consistent with an inference\-time compute\-scaling account, and we do not claim a mechanism distinct from externalised test\-time computation\.*TC0baseline validity:*No\-CoT accuracy on MMLU/ARC is high \(68–95%\), consistent with pretraining contamination or lexical shortcuts \(§[5](https://arxiv.org/html/2608.09942#S5)\)\. We therefore cannot attribute the absence of a CoT effect on these benchmarks to the architectural account specifically; the TC0null is confounded with ceiling and contamination effects\.*Asymptotic bound:*Hd​pH\_\{\\\!dp\}is an asymptotic lower bound; threshold values are treated as ordinal, not exact\.*Scope:*Decoder\-only dense transformers only; MoE and SSM architectures are excluded\.*English\-only benchmarks\.**Heuristic depth labelling:*The per\-item depth estimator \(calculator\-step count for GSM8K, equation\-count proxy for MATH, AST nesting depth for HumanEval\) produces non\-trivial bin diversity for those three benchmarks\. MMLU and ARC items collapse tok=1k\{=\}1under any structural depth metric we tried, because their per\-item textual structure does not expose a numeric depth proxy\. We treat this collapse as consistent with their TC0classification rather than as a measurement artefact, but bin\-stratified analyses for MMLU and ARC are correspondingly limited to thek=1k\{=\}1stratum\.*Theoretical framing \(pre\-registered deviation\):*The pre\-registration framed H1–H4 as predictions of theHd​pH\_\{\\\!dp\}bound\. On closer analysis the bound is asymptotic: its failure guarantee is non\-vacuous only for prompt lengthsn⋆=Hd​p24​Ln^\{\\star\}=H\_\{\\\!dp\}^\{\\,2^\{4L\}\}\(herelog10⁡n⋆≳1034\\log\_\{10\}n^\{\\star\}\\gtrsim 10^\{34\}\), so it does not bind at the context lengths we test\. We therefore reframe it as conceptual motivation rather than a literal predictor\. The hypotheses and all registered analyses are unchanged from registration; only their theoretical interpretation is refined\.*Cohen’sκ\\kappadeviation:*The pre\-registered LLM\-as\-judge inter\-rater check \(described in the Scoring section\) yieldedκ=0\.293\\kappa=0\.293, well below the pre\-registeredκ≥0\.7\\kappa\\geq 0\.7target\. The judge’s per\-item labels indicate that the heuristic single\-primitive\-per\-benchmark labels are too coarse: ARC items are predominantly judgedset\_disjointnessrather than the pre\-registeredsparse\_parity, and HumanEval items are predominantly judgedk\_compositionrather thanpointer\_chasing\. We retain the heuristic primitive taxonomy as a coarse descriptor \(cf\. Table[1](https://arxiv.org/html/2608.09942#S2.T1)\) and report this as a pre\-registered deviation\. The depth\-based hypotheses tested in H1–H4 are not affected by primitive labels\.*Pre\-registration completion:*The memorization\-separation analysis, originally deferred, was completed via GSM\-Symbolic perturbations\(Mirzadeh et al\.,[2024](https://arxiv.org/html/2608.09942#bib.bib13)\)and is reported in Section[4\.5](https://arxiv.org/html/2608.09942#S4.SS5)\. All other pre\-registered analyses \(functional HumanEval scoring with the full 984\-row execution sweep; the bin\-level McNemar tests on real per\-item depth labels for GSM8K/HumanEval; the LLM\-as\-judgeκ\\kappacheck\) were completed for this version of the paper\. All main\-text numerical claims are recomputed byverify\_paper\_stats\.py; Section[4\.5](https://arxiv.org/html/2608.09942#S4.SS5)claims are reproduced by scripts in thememorization\_check/directory, released as a separate archive\.

## 6Related Work

Transformer expressivity\.Hahn \([2020](https://arxiv.org/html/2608.09942#bib.bib7)\)proved \(Corollary 2 therein\) that hard\-attention transformers cannot compute PARITY or model 2DYCK, and extends these limitations to soft attention under smoothness assumptions on activations\. The PARITY result is direct theoretical precedent for our ARC\-Challenge “sparse parity” primitive classification: transformers are provably unable to compute parity regardless of scale, which is consistent with both the high no\-CoT baseline and the absence of CoT recovery we observe on ARC\.Pérez et al\. \([2021](https://arxiv.org/html/2608.09942#bib.bib15)\)proved Turing\-completeness for the Transformer under hard attention and*arbitrary\-precision*activations, but explicitly note that“Transformers with fixed precision are not Turing complete\.”Since our models operate at FP16 \(fixed precision\), they are precisely in the non\-Turing\-complete regime – creating the theoretical space for serial\-depth limitations to manifest\.Chen et al\. \([2024](https://arxiv.org/html/2608.09942#bib.bib1)\)close this gap by proving the first unconditional lower bounds for multi\-layer decoder\-only transformers via the autoregressive communication model, withHd​pH\_\{\\\!dp\}as the governing bandwidth parameter under realistic \(FP16\) precision\.

Chain\-of\-thought\.Wei et al\. \([2022b](https://arxiv.org/html/2608.09942#bib.bib20)\)established CoT as a general reasoning enhancement, andWei et al\. \([2022a](https://arxiv.org/html/2608.09942#bib.bib19)\)showed CoT is itself an*emergent ability*that only surpasses direct\-answer prompting on GSM8K at∼68\{\\sim\}68B parameters \(base LaMDA; Figure 3A therein\)\. Our instruction\-tuned 8B model already gains\+59\.9\+59\.9pp from CoT, consistent with RLHF shifting the emergence threshold downward\. Critically,Wei et al\. \([2022a](https://arxiv.org/html/2608.09942#bib.bib19)\)frame CoT as uniformly beneficial once it emerges; our results show its benefit is concentrated on P\-complete \(math\) tasks and is approximately zero on TC0tasks \(MMLU, ARC\) across all three models tested – a distinction their scaling analysis does not capture\.Cobbe et al\. \([2021](https://arxiv.org/html/2608.09942#bib.bib4)\)show that finetuning a 6B model to answer GSM8K*without*intermediate steps collapses accuracy from 20\.6% to 5\.2%, with the authors explicitly attributing this to the model having no mechanism to route intermediate computation outside a single forward pass – the same bottleneck formalised by theHd​pH\_\{\\\!dp\}bound\.Nye et al\. \([2022](https://arxiv.org/html/2608.09942#bib.bib14)\)introduced the scratchpad, showing that polynomial evaluation improves from 8\.8% to 20\.1% \(few\-shot\) when a model writes intermediate steps rather than answering directly, with even larger gains for addition on out\-of\-distribution lengths\. Their core mechanistic observation – that the model“cannot adapt the amount of compute”within a single forward pass – is the empirical precursor to theHd​pH\_\{\\\!dp\}bound\. Crucially, Nye et al\. find scratchpad helps most on tasks requiring sequential multi\-step computation, directly foreshadowing our result that only P\-complete benchmarks benefit from externalising computation, while TC0tasks do not\. In the code\-generation domain,Chen et al\. \([2021](https://arxiv.org/html/2608.09942#bib.bib2)\)provide independent empirical evidence of the same bottleneck: Codex’s pass rate on synthetic problems built from chained string\-manipulation primitives drops by a factor of 2–3 with every additional chained operation \(their Figure 11\), and the model fails to bind operations to variables once chain length exceeds a few steps\. These findings predate theHd​pH\_\{\\\!dp\}bound and are consistent with it: single forward\-pass bandwidth limits serial computation regardless of domain\. Subsequent work\(Kojima et al\.,[2022](https://arxiv.org/html/2608.09942#bib.bib10); Wang et al\.,[2023](https://arxiv.org/html/2608.09942#bib.bib18)\)extended CoT to zero\-shot and ensemble settings but likewise treats it as a universal gain\.Sprague et al\. \([2025](https://arxiv.org/html/2608.09942#bib.bib17)\)conduct the most comprehensive empirical challenge to this universality assumption: a meta\-analysis of 100\+ papers and experiments across 20 datasets and 14 models show that CoT helps mainly on math and symbolic reasoning, with negligible or negative effects on commonsense, knowledge, and soft\-reasoning tasks\. They further decompose CoT’s benefit into*planning*\(translating a problem into a formal specification\) and*execution*\(performing intermediate symbolic steps\), showing that much of CoT’s gain comes from execution – where it nonetheless underperforms external symbolic solvers\. Our results converge with theirs on the positive side: CoT’s benefit on math is supported by both theHd​pH\_\{\\\!dp\}bandwidth\-bypass account and their planning/execution decomposition\. On the negative side, our corrected data show that the absence of a CoT benefit on TC0tasks is not accompanied by a systematic CoT penalty, which constrains the form of any account that would attribute the null to the bound itself rather than to ceiling effects, instruction\-tuning, or task heterogeneity\.

Benchmark analysis\.Gururangan et al\. \([2018](https://arxiv.org/html/2608.09942#bib.bib6)\)showed that crowdsourced NLI benchmarks contain annotation artifacts exploitable without genuine inference – hypothesis\-only models reach 67% on SNLI\.Magar & Schwartz \([2022](https://arxiv.org/html/2608.09942#bib.bib12)\)showed that models can*exploit*\(not merely memorize\) contaminated test labels seen during pretraining, with exploitation growing with duplication frequency and model size; critically, memorization does not guarantee exploitation, implicating specific training dynamics\. Both confounds are directly relevant to the high no\-CoT scores we observe on TC0benchmarks \(§[5](https://arxiv.org/html/2608.09942#S5)\)\. We introduce CC primitive labeling as a complementary lens for anticipating*which*benchmarks benefit from which architectural capabilities, orthogonal to both artifact and contamination analyses\.

## 7Conclusion

We investigate whether the serial\-depth bottleneck identified by theHd​pH\_\{\\\!dp\}bandwidth bound\(Chen et al\.,[2024](https://arxiv.org/html/2608.09942#bib.bib1)\)governs behaviour on standard NLP benchmarks, even though the formal bound binds only at astronomically large context lengths \(n⋆=Hd​p24​Ln^\{\\star\}=H\_\{\\\!dp\}^\{\\,2^\{4L\}\}\)\. Across three instruction\-tuned models and five benchmarks, the framework’s positive hypothesis is supported: P\-complete tasks \(GSM8K, MATH\) show\+54\+54to\+68\+68pp CoT recovery gaps across all three models, and HumanEval shows the hypothesised model\-size\-dependent transition \(\+23\.2\+23\.2pp for Qwen\-32B;−28\.7\-28\.7pp for Qwen\-7B\)\. The framework’s stronger negative hypothesis – that CoT should*actively hurt*TC0tasks – is not supported: with correctly extracted answers, CoT is approximately neutral on MMLU and ARC across all six \(model, benchmark\) cells \(Δ∈\[0\.0,\+4\.6\]\\Delta\\in\[0\.0,\+4\.6\]pp\)\. The pooled cross\-benchmark depth–recovery correlation is Spearmanρ=0\.661\\rho=0\.661\(p=0\.007p=0\.007,n=15n=15\); 9 of 15 pre\-registered McNemar tests are significant; pre\-registered H3 \(MMLU no\-CoT≥\\geqCoT\) is falsified\.

All data, code, judge labels, and inference logs are released to support independent verification\. TheHd​pH\_\{\\\!dp\}framework is a useful one\-sided account of where CoT will help; whether it can also anticipate where CoT actively harms requires benchmarks designed to resist lexical shortcuts and verified to be absent from pretraining corpora\.

#### AI Usage Disclosure

Claude Opus 4\.7 \(Anthropic\) was used during the preparation of this manuscript for code generation, literature search, and editing of draft text\. All scientific claims, experimental design, data collection, analysis, and conclusions are the sole responsibility of the author\. No AI\-generated content was used as a primary source or cited as evidence\.

## References

- Chen et al\. \(2024\)Lijie Chen, Binghui Peng, and Hongxun Wu\.Theoretical limitations of multi\-layer transformer, 2024\.URL[https://arxiv\.org/abs/2412\.02975](https://arxiv.org/abs/2412.02975)\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, et al\.Evaluating large language models trained on code, 2021\.URL[https://arxiv\.org/abs/2107\.03374](https://arxiv.org/abs/2107.03374)\.
- Clark et al\. \(2018\)Peter Clark, Isaac Cowhey, Oren Etzioni, et al\.Think you have solved question answering? Try ARC, the AI2 reasoning challenge, 2018\.URL[https://arxiv\.org/abs/1803\.05457](https://arxiv.org/abs/1803.05457)\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al\.Training verifiers to solve math word problems, 2021\.URL[https://arxiv\.org/abs/2110\.14168](https://arxiv.org/abs/2110.14168)\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori et al\.The Llama 3 herd of models, 2024\.URL[https://arxiv\.org/abs/2407\.21783](https://arxiv.org/abs/2407.21783)\.
- Gururangan et al\. \(2018\)Suchin Gururangan, Swabha Swayamdipta, Omer Levy, et al\.Annotation artifacts in natural language inference data\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\)*, pp\. 107–112, New Orleans, Louisiana, 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/N18\-2017\.URL[https://aclanthology\.org/N18\-2017/](https://aclanthology.org/N18-2017/)\.
- Hahn \(2020\)Michael Hahn\.Theoretical limitations of self\-attention in neural sequence models\.*Transactions of the Association for Computational Linguistics*, 8:156–171, January 2020\.doi:10\.1162/tacl\\\_a\\\_00306\.URL[https://doi\.org/10\.1162/tacl\_a\_00306](https://doi.org/10.1162/tacl_a_00306)\.
- Hendrycks et al\. \(2021a\)Dan Hendrycks, Collin Burns, Steven Basart, et al\.Measuring massive multitask language understanding\.In*International Conference on Learning Representations*, 2021a\.URL[https://openreview\.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ)\.
- Hendrycks et al\. \(2021b\)Dan Hendrycks, Collin Burns, Saurav Kadavath, et al\.Measuring mathematical problem solving with the MATH dataset, 2021b\.URL[https://arxiv\.org/abs/2103\.03874](https://arxiv.org/abs/2103.03874)\.
- Kojima et al\. \(2022\)Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa\.Large language models are zero\-shot reasoners\.In*Advances in Neural Information Processing Systems*, volume 35, 2022\.URL[https://arxiv\.org/abs/2205\.11916](https://arxiv.org/abs/2205.11916)\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th Symposium on Operating Systems Principles*, SOSP ’23, pp\. 611–626, Koblenz, Germany, 2023\. ACM\.doi:10\.1145/3600006\.3613165\.URL[https://doi\.org/10\.1145/3600006\.3613165](https://doi.org/10.1145/3600006.3613165)\.
- Magar & Schwartz \(2022\)Inbal Magar and Roy Schwartz\.Data contamination: From memorization to exploitation\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pp\. 157–165, Dublin, Ireland, May 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.acl\-short\.18\.URL[https://aclanthology\.org/2022\.acl\-short\.18/](https://aclanthology.org/2022.acl-short.18/)\.
- Mirzadeh et al\. \(2024\)Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar\.GSM\-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024\.URL[https://arxiv\.org/abs/2410\.05229](https://arxiv.org/abs/2410.05229)\.
- Nye et al\. \(2022\)Maxwell Nye, Anders Johan Andreassen, Guy Gur\-Ari, et al\.Show your work: Scratchpads for intermediate computation with language models\.In*Deep Learning for Code Workshop*, 2022\.URL[https://openreview\.net/forum?id=HBlx2idbkbq](https://openreview.net/forum?id=HBlx2idbkbq)\.
- Pérez et al\. \(2021\)Jorge Pérez, Pablo Barceló, and Javier Marinkovic\.Attention is Turing\-Complete\.*Journal of Machine Learning Research*, 22\(75\):1–35, 2021\.URL[http://jmlr\.org/papers/v22/20\-302\.html](http://jmlr.org/papers/v22/20-302.html)\.
- Qwen Team \(2024\)Qwen Team\.Qwen2\.5 technical report, 2024\.URL[https://arxiv\.org/abs/2412\.15115](https://arxiv.org/abs/2412.15115)\.
- Sprague et al\. \(2025\)Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, et al\.To CoT or not to CoT? Chain\-of\-thought helps mainly on math and symbolic reasoning\.In*International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2409\.12183](https://arxiv.org/abs/2409.12183)\.
- Wang et al\. \(2023\)Xuezhi Wang, Jason Wei, Dale Schuurmans, et al\.Self\-consistency improves chain of thought reasoning in language models\.In*International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=1PL1NIMMrw](https://openreview.net/forum?id=1PL1NIMMrw)\.
- Wei et al\. \(2022a\)Jason Wei, Yi Tay, Rishi Bommasani, et al\.Emergent abilities of large language models\.*Transactions on Machine Learning Research*, 2022a\.URL[https://openreview\.net/forum?id=yzkSU5zdwD](https://openreview.net/forum?id=yzkSU5zdwD)\.
- Wei et al\. \(2022b\)Jason Wei, Xuezhi Wang, Dale Schuurmans, et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.In*Advances in Neural Information Processing Systems*, volume 35, 2022b\.URL[https://arxiv\.org/abs/2201\.11903](https://arxiv.org/abs/2201.11903)\.

## Appendix AReleased Artifacts

The OSF project \([https://osf\.io/hteuj](https://osf.io/hteuj), DOI10\.17605/OSF\.IO/92JDK\) and the Zenodo deposit \(DOI10\.5281/zenodo\.20294033\) include: the 20,184\-record SQLite inference database \(31 MB; raw outputs, extracted answers, gold labels, correctness flags, and completion\-token counts\); all system/user prompts; the regex,\\boxed\{\}, and HumanEval functional\-execution scorers; per\-item CC depth labels \(GSM8K calculator steps, MATH equation counts, HumanEval AST depths\); the Gemma\-2\-27B\-it judge labels and reasoning text \(964 items\);verify\_paper\_stats\.py, which reproduces every main\-text number in under 30 seconds;rescore\_mmlu\_arc\.py\(Appendix[B](https://arxiv.org/html/2608.09942#A2)\) andverify\_parser\_bug\.py, the rescoring script and the diagnostic that caught the v1 parser artefact; thememorization\_check/scripts reproducing §[4\.5](https://arxiv.org/html/2608.09942#S4.SS5); the pre\-registration document; and figure\-generation code for Figures[1](https://arxiv.org/html/2608.09942#S2.F1)–[3](https://arxiv.org/html/2608.09942#S4.F3)\.

## Appendix BScoring Artefact Corrections

This is a corrected manuscript that addresses two independent scoring artefacts present in an early preprint draft \(released on Zenodo on 19 May 2026\)\. Both artefacts artificially suppressed no\-CoT baselines\.

##### MMLU and ARC \(First Correction\)\.

The initial draft reported large negative CoT effects on MMLU and ARC\-Challenge, framed as a “CoT Sign Reversal\.” The MMLU/ARC scorer used the regex\\b\(\[A\-D\]\)\\band returned the*first*standalone capital letter found\. For CoT outputs that enumerate options, this regex reliably returnsAregardless of the model’s final answer\. A final\-answer\-aware parser locates the last “Answer” marker, and extracts the case\-sensitive letter\. With proper scoring, CoT is approximately neutral on MMLU and ARC across all six cells\.

##### HumanEval \(Second Correction\)\.

An intermediate draft reported a very low no\-CoT baseline \(15\.9%\) for Qwen\-32B on HumanEval\. A subsequent audit revealed that the functional execution script \(score\_humaneval\.py\) did not strip stop tokens \(e\.g\.<\|assistant\|\>\) from the raw generated text\. During functional execution, these tags causedSyntaxErrortracebacks for otherwise correct code, artificially suppressing the no\-CoT accuracy\. With tags properly stripped via regex, the true Qwen\-32B no\-CoT baseline is 62\.2%, and the CoT benefit is\+23\.2\+23\.2pp\.

##### What the corrections change\.

The central “CoT Sign Reversal” framing of the initial draft on MMLU and ARC is not supported by the corrected data\. The math\-side findings \(GSM8K and MATH CoT recovery gaps of\+54\+54to\+68\+68pp\) and the GSM\-Symbolic memorisation control are unchanged\. The HumanEval model\-size\-dependent crossover is preserved, but with smaller effect magnitudes \(\+23\.2\+23\.2pp for Qwen\-32B,−28\.7\-28\.7pp for Qwen\-7B\)\. The pre\-registered H1 \(originally 15/15 significant\) becomes 9/15; H3 \(MMLU no\-CoT≥\\geqCoT\) is falsified; the depth–recovery Spearman correlation reduces fromρ=0\.850\\rho=0\.850toρ=0\.661\\rho=0\.661\(still significant atp=0\.007p=0\.007\)\.