面向微型Transformer算术推理的算法草稿本与课程分层方法

# Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
Source: [https://arxiv.org/html/2610.09003](https://arxiv.org/html/2610.09003)
## Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny TransformersThanks:Code, model weights, and benchmark datasets are available at[https://github\.com/SoKasl/AIMathLab](https://github.com/SoKasl/AIMathLab)\.

###### Abstract

Autoregressive Large Language Models \(LLMs\) frequently struggle with deterministic multi\-step algorithmic tasks such as multi\-digit multiplication and long division\. In this paper, we investigate the mechanics of multi\-step arithmetic in compact “Tiny” Transformers \(∼\\sim10\.6M non\-embedding parameters, 49\.3M total\) trained on synthetic data across four basic operations \(\+\+,−\-,×\\times,//\) unrolled as step\-by\-step scratchpads\.

First, we establish the necessary training foundations: \(1\) dataloader sequence padding creates an 83% gradient starvation artifact that collapses accuracy from 40% to 1%, remediated via continuous sequence packing; \(2\) linguistic pretraining is an essential prerequisite \(≤2\.0%\\leq 2\.0\\%without it\); and \(3\) modern architectural primitives \(RoPE, RMSNorm, SwiGLU\) and Sparse Mixture of Experts \(MoE\) substantially improve additive reasoning over baseline GPT\-2\.

Second, we demonstrate that algorithmic scratchpad formulation directly dictates success\. Introducing a deterministicDigit\-by\-Digit Long Divisionscratchpad within a 4\-stageHierarchical Developmental Curriculumdramatically elevates single\-digit division from 4\.0% to86\.7% accuracyon a 4,000\-problem held\-out benchmark\. In contrast, multi\-digit multiplication remained challenging: detailed error analysis revealed that while the model correctly computed single\-digit sub\-products and place\-value zeros, our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi\-digit terms in a single step without pairwise intermediate accumulation\. Finally, we identify two key boundaries: performance collapses to 0\.00% on unseen 4\-digit operands, and unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86\.7% down to 0\.00%\.

## 1Introduction

Autoregressive Transformers have achieved widespread empirical success across diverse natural language processing domains\[[15](https://arxiv.org/html/2610.09003#bib.bib1),[3](https://arxiv.org/html/2610.09003#bib.bib2)\]\. However, their ability to perform exact mathematical reasoning, particularly multi\-digit multiplication and division, remains fragile\[[20](https://arxiv.org/html/2610.09003#bib.bib3),[4](https://arxiv.org/html/2610.09003#bib.bib12),[6](https://arxiv.org/html/2610.09003#bib.bib6)\]\. Scaling laws suggest that increasing model capacity to hundreds of billions of parameters mitigates these reasoning deficits\. Nevertheless, recent investigations into small language models \(SLMs\), such asTinyStories\[[7](https://arxiv.org/html/2610.09003#bib.bib5)\], indicate that specialized inductive biases, targeted data engineering, and explicit step\-by\-step unrolling can induce complex structured behaviors at miniature scales\.

In standard autoregressive language modeling, token generation is governed by causal next\-token prediction:

P⁡\(x1,x2,…,xT\)=∏t=1TP⁡\(xt∣x<t;θ\)P\(x\_\{1\},x\_\{2\},\\dots,x\_\{T\}\)=\\prod\_\{t=1\}^\{T\}P\(x\_\{t\}\\mid x\_\{<t\};\\theta\)\(1\)When applied directly to multi\-digit arithmetic \(e\.g\., predicting983×43=42269983\\times 43=42269\), next\-token prediction requires the network to evaluate a complex composite function within the fixed forward\-pass depth of its attention and feed\-forward layers\. To evaluate whether autoregressive transformers can execute deterministic algorithms without being confounded by linguistic sentence comprehension \(such as story word problems in GSM8K\[[4](https://arxiv.org/html/2610.09003#bib.bib12)\]\), we formulate a controlled experimental laboratory focused purely on canonical arithmetic operations \(\+\+,−\-,×\\times,//\) unrolled as sequential state transitions\.

In this work, we investigate whether compact Transformer models \(∼\\sim10\.6M non\-embedding parameters\) can reliably execute multi\-step arithmetic algorithms\. We construct a controlled experimental testbed to address four interconnected questions:

1. 1\.Linguistic Pretraining as an Inductive Foundation:Does natural language pretraining act as a necessary syntactic scaffold for acquiring formal arithmetic reasoning syntax, and how does pretraining duration scale downstream convergence?
2. 2\.Architectural Inductive Biases:Do modern architectural upgrades—Rotary Position Embeddings \(RoPE\)\[[19](https://arxiv.org/html/2610.09003#bib.bib8)\], Root Mean Square Layer Normalization \(RMSNorm\)\[[21](https://arxiv.org/html/2610.09003#bib.bib10)\], SwiGLU activations\[[18](https://arxiv.org/html/2610.09003#bib.bib9)\], and Sparse Mixture of Experts \(MoE\)\[[17](https://arxiv.org/html/2610.09003#bib.bib11)\]—expand arithmetic capacity at comparable active compute?
3. 3\.Data Engineering and Gradient Density:How do sequence batching mechanics \(padding vs\. packing\) affect gradient flow and algorithmic convergence?
4. 4\.Algorithmic Scratchpad Decomposition & Curriculum Staging:Why do direct FOIL and partial\-quotient scratchpads\[[14](https://arxiv.org/html/2610.09003#bib.bib4),[20](https://arxiv.org/html/2610.09003#bib.bib3)\]fail on multi\-digit multiplication and division, and can algorithmic reformulation paired with hierarchical curriculum staging overcome these limitations?

## 2Related Work

#### Arithmetic Reasoning and Scratchpads\.

Early investigations demonstrated that standard transformers struggle with direct arithmetic computation\[[13](https://arxiv.org/html/2610.09003#bib.bib20)\]\. Nye et al\.\[[14](https://arxiv.org/html/2610.09003#bib.bib4)\]introduced scratchpads—intermediate execution steps that allow language models to emit scratch memory before producing final outputs\. Chain\-of\-Thought \(CoT\) prompting\[[20](https://arxiv.org/html/2610.09003#bib.bib3)\]popularized this technique for few\-shot prompting in frontier models\. However, Dziri et al\.\[[6](https://arxiv.org/html/2610.09003#bib.bib6)\]showed that transformers perform computation via subgraph matching rather than true algorithmic execution, exhibiting severe failure modes when composite problem graphs exceed the network’s effective reasoning horizon\.

#### Arithmetic in Small Transformers\.

Most relevant to our investigation is the work of Lee et al\.\[[10](https://arxiv.org/html/2610.09003#bib.bib14)\], who demonstrated that compact \(∼\\sim10M parameter\) NanoGPT models can master addition and subtraction when provided with reversed digit formatting and linguistic pretraining\. Our work builds upon and substantially extends their findings in several key dimensions:

1. 1\.Algorithmic Reform of Long Division:Lee et al\. focused on addition, subtraction, and multi\-digit multiplication\. We investigate multi\-digit division and demonstrate that traditional partial\-quotient guessing fails in tiny models, whereas refactoring division into a deterministic digit\-by\-digit quotient descent algorithm achieves high accuracy\.
2. 2\.Hierarchical Curriculum Staging:Rather than training on mixed datasets simultaneously, we formulate an explicit 4\-stage developmental progression based on algorithmic sub\-task dependencies\.
3. 3\.Mixture of Experts Routing:We examine sparse conditional routing \(MoE\) in compact models to study whether parameter modularity benefits multi\-task arithmetic\.
4. 4\.Large\-Scale Generalization & Drift Evaluation:We evaluate 4,000 held\-out samples to establish exact length generalization boundaries and analyze catastrophic developmental drift\.

#### Length Generalization and Positional Inductive Biases\.

Extrapolating learned arithmetic procedures to operands longer than those seen during training remains an open challenge\[[1](https://arxiv.org/html/2610.09003#bib.bib15)\]\. Kazemnejad et al\.\[[8](https://arxiv.org/html/2610.09003#bib.bib16)\]showed that positional encodings fundamentally govern length extrapolation in transformers, finding that NoPE \(no positional embeddings\) or relative representations often outperform standard learned embeddings\. We systematically evaluate RoPE\[[19](https://arxiv.org/html/2610.09003#bib.bib8)\]on multi\-digit operands up to 4 digits\.

#### Curriculum Learning and Catastrophic Forgetting\.

Curriculum learning, introduced by Bengio et al\.\[[2](https://arxiv.org/html/2610.09003#bib.bib19)\], posits that presenting training samples in order of increasing difficulty accelerates optimization\. In sequential multi\-task settings, however, neural networks suffer from catastrophic interference\[[11](https://arxiv.org/html/2610.09003#bib.bib18),[9](https://arxiv.org/html/2610.09003#bib.bib17)\], where learning new tasks overwrites previously mastered representations unless protected by rehearsal buffers\[[16](https://arxiv.org/html/2610.09003#bib.bib21)\]\.

## 3Experimental Setup and Architectures

All models in our ablation suite are constrained to a compact budget ofL=6L=6transformer layers, hidden dimensiondmodel=384d\_\{\\text\{model\}\}=384,nheads=6n\_\{\\text\{heads\}\}=6, and context block sizeT=256T=256\.

#### Parameter Accounting\.

The 6\-layer transformer backbone comprises∼\\sim10\.6M non\-embedding parameters\. When coupled with the standard OpenAI GPT\-2 vocabulary \(V=50,257V=50,257\) embedding and unembedding matrices, the total parameter count is 49\.3M for dense configurations\. For the Sparse MoE configuration \(with 4 SwiGLU experts per layer\), the total parameter count is 70\.5M, though only 2 experts are dynamically activated per token \(maintaining active compute at approximately1\.5×1\.5\\timesdense FLOPs\)\.

### 3\.1Architectural Configurations

We examine four architectural configurations:

1. 1\.Baseline GPT\-2:Standard scaled dot\-product attention, learned absolute positional embeddingsWpos∈ℝT×dW\_\{\\text\{pos\}\}\\in\\mathbb\{R\}^\{T\\times d\}, Layer Normalization\[[15](https://arxiv.org/html/2610.09003#bib.bib1)\], and standard GELU feed\-forward networks with expansion ratio 4\.
2. 2\.FlashAttention Optimization:Incorporates IO\-aware exact FlashAttention\[[5](https://arxiv.org/html/2610.09003#bib.bib7)\]\. As an exact mathematical reformulation of scaled dot\-product attention, FlashAttention serves as a computational sanity check—yielding identical mathematical outputs while accelerating GPU/SRAM memory transfers fromO⁡\(T2\)O\(T^\{2\}\)toO⁡\(T\)O\(T\)\.
3. 3\.Modern Dense Transformer:Replaces legacy GPT\-2 components with modern state\-of\-the\-art primitives: - •Rotary Position Embeddings \(RoPE\):Replaces absolute positional embeddings with relative rotational transformations\[[19](https://arxiv.org/html/2610.09003#bib.bib8)\]applied directly to query and key projections: RΘ,md=diag​\(Rθ1,m,Rθ2,m,…,Rθd/2,m\)R\_\{\\Theta,m\}^\{d\}=\\text\{diag\}\\left\(R\_\{\\theta\_\{1\},m\},R\_\{\\theta\_\{2\},m\},\\dots,R\_\{\\theta\_\{d/2\},m\}\\right\)\(2\) - •RMSNorm:Eliminates mean\-centering operations, stabilizing activations via root\-mean\-square scaling\[[21](https://arxiv.org/html/2610.09003#bib.bib10)\]: RMSNorm​\(x\)=x1d​∑i=1dxi2\+ϵ⊙γ\\text\{RMSNorm\}\(x\)=\\frac\{x\}\{\\sqrt\{\\frac\{1\}\{d\}\\sum\_\{i=1\}^\{d\}x\_\{i\}^\{2\}\+\\epsilon\}\}\\odot\\gamma\(3\) - •SwiGLU Activations:Replaces standard two\-layer MLPs with gated linear units utilizing the Swish activation function\[[18](https://arxiv.org/html/2610.09003#bib.bib9)\]: SwiGLU​\(x\)=\(Swish​\(x​Wgate\)⊙\(x​Wup\)\)​Wdown\\text\{SwiGLU\}\(x\)=\\left\(\\text\{Swish\}\(xW\_\{\\text\{gate\}\}\)\\odot\(xW\_\{\\text\{up\}\}\)\\right\)W\_\{\\text\{down\}\}\(4\)
4. 4\.Sparse Mixture of Experts \(MoE\):Substitutes dense MLP blocks with a Sparse MoE layer\[[17](https://arxiv.org/html/2610.09003#bib.bib11)\]containingE=4E=4distinct SwiGLU experts with Top\-2 parameterized routing: H⁡\(x\)=∑i∈Top\-2​\(G​\(x\)\)G​\(x\)i⋅Experti​\(x\),G⁡\(x\)=Softmax​\(Top\-2​\(x​Wg\)\)H\(x\)=\\sum\_\{i\\in\\text\{Top\-2\}\(G\(x\)\)\}G\(x\)\_\{i\}\\cdot\\text\{Expert\}\_\{i\}\(x\),\\quad G\(x\)=\\text\{Softmax\}\(\\text\{Top\-2\}\(xW\_\{g\}\)\)\(5\)

#### KV\-Cached Inference Engine\.

For accelerated evaluation on large benchmarks, we implemented an autoregressive Key\-Value \(KV\) cache engine with position offset tracking\. During decoding, key and value states are cached across layers, reducing generation complexity per token fromO⁡\(N\)O\(N\)toO⁡\(1\)O\(1\)and achieving an inference throughput of∼\\sim8\.8 problems/sec on Apple Silicon\.

#### Training Hyperparameters\.

To ensure an exact, unconfounded comparison, all models were trained using identical, untuned hyperparameters: AdamW optimizer \(β1=0\.9,β2=0\.999\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.999, weight decay 0\.01\), learning rateη=3×10−4\\eta=3\\times 10^\{\-4\}without decay, batch size 4, and context length 256\.

## 4Task Formulation and Synthetic Data Generation

To evaluate algorithmic reasoning strictly independent of linguistic comprehension artifacts, we formulate our mathematical benchmark entirely around pure arithmetic equations rather than natural language story problems\. Each training instance follows a standardized prompt\-scratchpad format:

Q: What is A \[op\] B?\\nA: <thought\> \[Trace\] </thought\> \[Final Answer\]

By enforcing a structured thought delimiter \(<thought\> \.\.\. </thought\>\), the autoregressive objective transforms standard next\-token prediction into an unrolled simulation of algorithmic execution\. We generate synthetic instances covering four core operations:

### 4\.1Place\-Value Addition

Operand pairs \(a,b∈\[10,999\]a,b\\in\[10,999\]\) are decomposed by decimal place\-value \(hundreds, tens, ones\)\. The scratchpad unrolls intermediate grouping and summation:

Q:Whatis482\+357?

A:<thought\>482\+357=400\+300\+80\+50\+2\+7=700\+130\+9=839</thought\>839

### 4\.2Borrow\-State Subtraction

For subtractions \(a≥ba\\geq b\), the trace explicitly computes column\-wise differences from right to left, tracking borrow state propagation \(10​s10\\text\{s\}and100​s100\\text\{s\}\):

Q:Whatis824\-457?

A:<thought\>4ones\-7ones\(borrow10\)\-\>14\-7=7\.1tens\-5tens\(borrow100\)\-\>110\-50=60\.7hundreds\-4hundreds\-\>700\-400=300\.Answer=300\+60\+7=367</thought\>367

### 4\.3Multi\-Digit Multiplication \(FOIL Scratchpad\)

Multiplication is decomposed into polynomial partial products with base\-10 zero accounting \(e\.g\.,900×40→9×4=36900\\times 40\\rightarrow 9\\times 4=36with 3 zeros→36000\\rightarrow 36000\), followed by an unrolled summation:

Q:Whatis84\*32?

A:<thought\>84\*32=\(80\+4\)\*\(30\+2\)\.80\*30:8\*3=24with2zeros\-\>2400\.80\*2:8\*2=16with1zeros\-\>160\.4\*30:4\*3=12with1zeros\-\>120\.4\*2:8\.Sum:2400\+160\+120\+8=2688\.</thought\>2688

### 4\.4Deterministic Digit\-by\-Digit Long Division

Unlike prior recursive partial\-quotient estimation which guesses large multiples, our digit\-by\-digit algorithm explicitly models elementary school long division:

Q:Whatis3024/6?

A:<thought\>Divide3024by6:Take30:30/6=5\(6\*5=30\),rem=30\-30=0\.Bringdown2\-\>2:2/6=0\(6\*0=0\),rem=2\-0=2\.Bringdown4\-\>24:24/6=4\(6\*4=24\),rem=24\-24=0\.Finalquotient=504\.</thought\>504

## 5Data Engineering: The Token Density Collapse

During fine\-tuning, the mechanics of sequence batching substantially influence optimization dynamics\. We evaluate three distinct dataloader paradigms \(Figure[1](https://arxiv.org/html/2610.09003#S5.F1)\):

Streaming \(Windowing\) \.\.\.400\+110=510</thought\> Q: What is 82\-14? A: <thought\> 2 ones \- 4\.\.\. Density:100% Flaw:Truncated prompts\.

\(a\)Naive Streaming
Single\-Problem Padded Q: What is 23\+45? A: <thought\>\.\.\.68 \[EOS\] \[PAD\] \[PAD\] \[PAD\]\.\.\. Density:∼\\sim17% Flaw:83% gradient starvation\.

\(b\)Padded Instruction
Sequence Packed Q1:\.\.A1:\.\.\[EOS\] Q2:\.\.A2:\.\.\[EOS\] Q3:\.\.A3:\.\.\[EOS\]\[PAD\] Density:∼\\sim97% Benefit:High density \+ intact\.

\(c\)Sequence Packing

Figure 1:Comparison of data loading strategies for 256\-token context blocks\.### 5\.1The40%→1%40\\%\\rightarrow 1\\%Performance Collapse

In our initial experiments, switching from continuous streaming to an isolated single\-problem padded dataloader resulted in a sharp drop in arithmetic performance: overall accuracy dropped from40\.0%to1\.0%across all model architectures \(Table[1](https://arxiv.org/html/2610.09003#S5.T1)\)\.

Table 1:Empirical collapse under single\-instruction padded training\.Figure 2:Active token throughput in millions \(blue, left axis\) versus downstream arithmetic accuracy \(red, right axis\) across dataloader packaging strategies over 5,000 steps\. Padded training induces severe gradient starvation, collapsing performance\.Root Cause Analysis:Because a standard two\-digit addition problem comprises only∼\\sim45 tokens, padding it into a 256\-token block with masked cross\-entropy loss \(ignore\_index=−100\\text\{ignore\\\_index\}=\-100\) leaves 83% of the sequence empty\. Over 5,000 gradient steps, the model receives updates from only 900,000 active tokens—equivalent to fewer than 900 steps of dense training\. The model converges on syntactic formatting \(emitting valid thought tags and halting tokens\), but its internal arithmetic weights remain uncalibrated\.

### 5\.2Sequence Packing with Prompt Masking

We resolve this issue by implementingSequence Packing with Instruction Masking\. Multiple complete, intact equation\-scratchpad\-answer triples are concatenated sequentially within each 256\-token block until full\. Prompt tokens \(QQ\) for each problem are masked with target label−100\-100, while loss is computed exclusively on thought traces, final numerical answers, and⟨\|endoftext\|⟩\\langle\|\\text\{endoftext\}\|\\ranglestopping tokens:

ℒ=−∑t=1T𝟏\{yt≠−100\}logP\(xt∣x<t;θ\)\\mathcal\{L\}=\-\\sum\_\{t=1\}^\{T\}\\mathbf\{1\}\_\{\\\{y\_\{t\}\\neq\-100\\\}\}\\log P\(x\_\{t\}\\mid x\_\{<t\};\\theta\)\(6\)This delivers∼\\sim97% token density, eliminates truncated sequences, and restores full convergence\.

## 6Empirical Ablations and Optimization Dynamics

Table[2](https://arxiv.org/html/2610.09003#S6.T2)summarizes the complete ablation matrix across all ten experimental configurations evaluated on our standardized 100\-problem benchmark \(25 problems per operation: Addition, Subtraction, Multiplication, Division\)\.

Table 2:Master experimental ablation results across all 10 evaluation runs on the 100\-problem evaluation set\. Runs 1–4 evaluate baseline scratchpad configurations; Runs 9 and 10 introduce our deterministic digit\-by\-digit long division algorithm, elevating division from 4\.0% to 60\.0%\.### 6\.1Ablation 1: Validating the Necessity of Linguistic Pretraining

To evaluate whether mathematical alignment can emerge independently of linguistic scaffolding, we varied WikiText\-103\[[12](https://arxiv.org/html/2610.09003#bib.bib13)\]pretraining duration from 10 to 50,000 steps prior to arithmetic fine\-tuning\. Across all architectures, the model was subsequently fine\-tuned for 5,000 math steps using identical hyper\-parameters \(Table[3](https://arxiv.org/html/2610.09003#S6.T3)\)\.

Table 3:Impact of natural language pretraining duration on downstream arithmetic accuracy after 5,000 mathematical fine\-tuning steps\. Insufficient pretraining \(≤10\\leq 10steps\) results in near\-total failure across all architectures \(≤2\.0%\\leq 2\.0\\%\), while 20,000 steps provides the optimal syntactic foundation\.As shown in Table[3](https://arxiv.org/html/2610.09003#S6.T3), models initialized with only 10 steps of language exposure consistently fail to acquire arithmetic reasoning \(scoring≤2\.0%\\leq 2\.0\\%\)\. Inspection of generated traces indicates that without language pretraining, models cannot learn the hierarchical delimiter syntax required to generate closed thought traces \(<thought\> \.\.\. </thought\>\)\. Conversely, 20,000 steps establishes robust syntactic comprehension, yielding up to 36\.0% multi\-task accuracy\. Extending language training to 50,000 steps degrades performance \(dropping to 19\.0% in Modern Dense\), consistent with observations by Lee et al\.\[[10](https://arxiv.org/html/2610.09003#bib.bib14)\]that prolonged linguistic pretraining can constrain parameter plasticity for downstream arithmetic specialization in compact models\.

### 6\.2Ablation 2: Architectural Inductive Biases at Constant Scale

Comparing Runs 1–4 isolates the impact of modern transformer primitives at an identical non\-embedding parameter budget \(∼\\sim10\.6M\) and fixed training budget \(20,000 Wiki steps, 5,000 Math steps\), presented in Table[4](https://arxiv.org/html/2610.09003#S6.T4):

Table 4:Architectural ablation at constant∼\\sim10\.6M non\-embedding parameter scale\. Modern inductive biases \(RoPE, RMSNorm, SwiGLU\) and Sparse MoE provide a substantial boost over standard GPT\-2\.- •Exact FlashAttention Sanity Check \(Run 1 vs\. Run 2\):Incorporating FlashAttention yields statistically indistinguishable performance \(5\.0% vs\. 6\.0%\), confirming that exact tiling maintains mathematical parity with scaled dot\-product attention while speeding up execution\.
- •Legacy GPT\-2 to Modern Dense \(Run 1 vs\. Run 3\):Replacing absolute positional embeddings with RoPE, LayerNorm with RMSNorm, and standard GELU MLPs with SwiGLU raises Addition from 8\.0% to56\.0%and Subtraction from 12\.0% to76\.0%\. RoPE’s relative rotary encoding allows attention heads to track relative place\-value distance across multiple unrolled columns\.
- •Dense vs\. Sparse MoE \(Run 3 vs\. Run 4\):Introducing Top\-2 gating across 4 SwiGLU experts further increases Addition to60\.0%and Subtraction to84\.0%\. While MoE increases total parameter count \(70\.5M vs\. 49\.3M\), inference FLOPs remain comparable due to sparse Top\-2 routing\.

### 6\.3Ablation 3: Extended Training Plateau \(5,000 to 100,000 Steps\)

In Runs 5 and 6, we extended mathematical training from 5,000 to 100,000 steps \(a 20×\\timescompute increase\)\. While Addition and Subtraction achieved 100\.0% saturation, Multiplication and Division plateaued at8\.0%\. Between step 50,000 and 100,000, overall performance increased by only 2% \(from 52% to 54%, Figure[3](https://arxiv.org/html/2610.09003#S6.F3)\)\. This demonstrates that raw training duration cannot overcome the inductive bottlenecks of ill\-conditioned scratchpads\.

Figure 3:Left: Overall multi\-task accuracy plateau from 10,000 to 100,000 steps across Dense and MoE architectures, illustrating the sharp 55% ceiling\. Right: Per\-operation accuracy trajectory in Run 8 \(MoE\), demonstrating that Addition and Subtraction saturate at 100% while Multiplication remains at 0% and Division caps at 20%\.
### 6\.4Ablation 4: Hierarchical Curriculum and Long Division Advance

To resolve the plateau on division and multiplication, we designed a 4\-StageHierarchical Developmental Curriculumstructured around algorithmic sub\-task dependency:

1. 1\.Stage 1: Additive Foundations \(30k samples\):50% Addition, 50% Subtraction with explicit place\-value carry and borrow tracking\.
2. 2\.Stage 2: Single\-Digit Multiplication Table \(30k samples\):100% exhaustive coverage of the10×1010\\times 10multiplication table \(0×00\\times 0to9×99\\times 9, 40%\), combined with 60% Add/Sub rehearsal\.
3. 3\.Stage 3: Multi\-Digit FOIL with Zero\-Accounting \(40k samples\):60% Multi\-digit multiplication, coupled with 40% additive rehearsal\.
4. 4\.Stage 4: Digit\-by\-Digit Long Division \(50k samples\):45% Long Division, 35% Multi\-digit Multiplication, 10% Addition, 10% Subtraction\.

Replacing partial\-quotient estimation with deterministic digit\-by\-digit descent produced a substantial performance gain on development evaluations \(Runs 9 and 10 in Table[2](https://arxiv.org/html/2610.09003#S6.T2)\):

- •Modern Dense \(Run 9\): Division reached56\.0%\(14/25\) on the development test set, lifting overall accuracy to63\.0%\.
- •Sparse MoE \(Run 10\): Division reached60\.0%\(15/25\) at step 30,000, lifting overall accuracy to61\.0%\.

#### Interaction of Scratchpads and Curriculum Staging\.

Comparing Run 7 and Run 9 illustrates the necessary synergy between scratchpad formulation and curriculum staging\. In Run 7, applying a curriculum while retaining the ill\-conditioned partial\-quotient scratchpad yielded only 4\.0% division accuracy after 100,000 steps\. This demonstrates that curriculum staging cannot compensate for scratchpads whose intermediate steps require implicit multi\-digit estimation\. When paired with deterministic digit\-by\-digit quotient descent, however, the hierarchical curriculum enabled rapid convergence, reaching 56%–60% accuracy within 30,000 steps\.

Checkpoint Selection Disclosure:Checkpoints designated as “best” were saved based on peak overall accuracy on this 100\-problem developmental set\. Because a 25\-problem operational slice carries a wide 95% confidence interval \(≈±19%\\approx\\pm 19\\%\), this developmental metric serves primarily as a checkpoint selection signal; Section[7](https://arxiv.org/html/2610.09003#S7)provides the definitive, high\-powered evaluation across 4,000 independent samples\.

Figure 4:Learning trajectories across mathematical operations throughout the 4\-stage Hierarchical Developmental Curriculum for Modern Dense \(left, Run 9\) and Sparse MoE \(right, Run 10\)\. Notice the steep rise of Long Division \(red triangles\) to 56–60% as the training corpus transitions into Stage 4\.Figure 5:Long Division Performance: Comparison of Division accuracy across algorithmic paradigms in∼\\sim10M parameter networks\. Transforming division from partial quotient guessing to deterministic digit\-by\-digit descent produces a substantial improvement \(0–4%→\\rightarrow60\.0% on development tests\)\.

## 7Large\-Scale 4,000\-Problem Benchmark and Developmental Dynamics

While the 100\-problem evaluation set \(Table[2](https://arxiv.org/html/2610.09003#S6.T2)\) provides an initial benchmark for architectural screening, its sample size \(N=25N=25per operation\) carries a 95% Wilson score confidence interval of≈±19%\\approx\\pm 19\\%, introducing potential variance in fine\-grained comparisons\. To establish rigorous statistical certainty, isolate length generalization limits, and analyze late\-stage training dynamics, we generated a comprehensive 4,000\-problem balanced evaluation benchmark \(1,000 problems per arithmetic operation\)\.

### 7\.1Benchmark Design and Evaluation Protocol

The benchmark was synthesized deterministically using a fixed pseudo\-random seed \(seed=42\) with the following operational partitions:

- •Addition \(1,000 problems\):514 in\-distribution problems \(a,b∈\[10,999\]a,b\\in\[10,999\], matching the training range\) and 486 out\-of\-distribution problems \(a,b∈\[1000,4999\]a,b\\in\[1000,4999\], strictly unseen 4\-digit operands\)\.
- •Subtraction \(1,000 problems\):517 in\-distribution problems \(a,b∈\[10,999\]a,b\\in\[10,999\],a≥ba\\geq b\) and 483 out\-of\-distribution problems \(a,b∈\[1000,4999\]a,b\\in\[1000,4999\]\)\.
- •Multiplication \(1,000 problems\):2501×21\\times 2\-digit problems, 5002×22\\times 2\-digit problems, and 2503×23\\times 2\-digit problems\.
- •Division \(1,000 problems\):300 single\-digit divisor problems \(b∈\[2,9\]b\\in\[2,9\], dividends up to 5 digits\) and 700 multi\-digit divisor problems \(b∈\[10,99\]b\\in\[10,99\]\), constructed with exact integer quotients \(a=b×ca=b\\times c, remainder zero\)\.

In\-distribution instances were sampled independently from the same integer domain as the training distribution, whereas 4\-digit addition and subtraction instances tested unseen length regimes\. All evaluations were executed via autoregressive categorical sampling \(torch\.multinomialat temperatureT=1\.0T=1\.0\) up to a maximum context length ofT=256T=256tokens\. To extract the predicted numerical answer, the evaluator parses the integer immediately succeeding the</thought\>delimiter, falling back to the terminal numerical string if the tag is unclosed\.

### 7\.2Inference Acceleration via KV Caching

Generating multi\-step unrolled scratchpad reasoning traces across 16,000 total instances \(4 model checkpoints×\\times4,000 problems\) imposes significant computational overhead\. Standard autoregressive generation without key\-value caching recomputes past keys and values across all prior sequence positions at each generated token step, requiring repeated full\-context forward projections\.

By integrating a persistent key\-value cache \(K,V∈ℝB×nheads×T×dheadK,V\\in\\mathbb\{R\}^\{B\\times n\_\{\\text\{heads\}\}\\times T\\times d\_\{\\text\{head\}\}\}\), linear projection layers are evaluated exclusively for the newly emitted token\. While attending over past keys still requiresO⁡\(T\)O\(T\)attention operations per step, eliminating redundant query\-key\-value transformations across preceding tokens significantly reduced generation latency across all benchmark sweeps\.

### 7\.3Benchmark Results and Statistical Confidence

We evaluated two developmental snapshots for both Modern Dense \(Run 9\) and Sparse MoE \(Run 10\): \(1\) theStep 30,000 checkpoint\(peak multi\-task performance\), and \(2\) theStep 50,000 checkpoint\(final saturated training\)\. Results are detailed in Table[5](https://arxiv.org/html/2610.09003#S7.T5), and key developmental phenomena are illustrated in Figure[6](https://arxiv.org/html/2610.09003#S7.F6)\. Across the 4,000\-sample benchmark, the 95% Wilson score margin of error is bounded by±3\.1%\\pm 3\.1\\%for individual 1,000\-problem operations and±1\.5%\\pm 1\.5\\%overall\. The difference between Step 30,000 and Step 50,000 on single\-digit long division \(86\.67% vs\. 0\.00% in Modern Dense; 84\.33% vs\. 1\.67% in MoE\) is statistically significant \(Fisher’s exact test,p<10−15p<10^\{\-15\}\)\.

Table 5:Large\-scale 4,000\-problem benchmark comparing Step 30,000 \(peak multi\-task\) and Step 50,000 \(final\) checkpoints across Modern Dense and Sparse MoE architectures\. At Step 30,000, single\-digit long division achieves up to 86\.67% accuracy\. Prolonged training to Step 50,000 drives addition and subtraction toward 99\.4% saturation but induces catastrophic developmental drift on division\. Unseen 4\-digit addition completely collapses \(0\.00%\), defining a strict length generalization boundary\.Figure 6:Large\-Scale Benchmark Dynamics across 4,000 held\-out problems\. \(a\)Length Generalization Collapse:While models achieve near\-deterministic accuracy \(\>99\.2%\>99\.2\\%\) on in\-distribution operands \(≤3\\leq 3digits\), accuracy abruptly collapses to 0\.00% on unseen 4\-digit operands due to fixed positional and causal depth horizons\. \(b\)Catastrophic Developmental Drift:Single\-digit division performance peaks at Step 30,000 \(86\.67% Dense, 84\.33% MoE\), but prolonged un\-buffered training to Step 50,000 reduces division accuracy down to 0\.00% as late\-stage addition gradients overwrite parameters\.
### 7\.4Key Empirical Findings

The large\-scale evaluation exposes four critical empirical principles:

1. 1\.Single\-Digit Long Division Execution: At Step 30,000, both Modern Dense and Sparse MoE models reliably execute deterministic digit descent, scoring86\.67%and84\.33%respectively on 300 unseen single\-digit division problems \(b∈\[2,9\]b\\in\[2,9\], dividends up to 5 digits\)\. This confirms that tiny models can reliably execute multi\-step long division when the quotient search space is constrained to single\-digit trial descent\.
2. 2\.In\-Distribution Additive Saturation: By Step 50,000, in\-distribution addition and subtraction reach near\-perfect saturation, scoring99\.22%and99\.42%across over 1,000 combined test samples\.
3. 3\.The Length Generalization Boundary: Across both checkpoints and both architectures, 4\-digit addition collapses entirely to0\.00%\(0/486\)\. Because rotary positional embeddings and causal attention layers were parameterized over sequences bounded by 3\-digit operands during training, the network is unable to recursively propagate carry bits across longer unrolled sequence lengths without recurrence or explicit length\-curriculum data\.
4. 4\.Catastrophic Developmental Drift and Wrap\-Around: Continuing training from Step 30,000 to Step 50,000 without balanced experience replay erodes algorithmic division capabilities: single\-digit division plummets from 86\.67% to0\.00%in Dense and 84\.33% to1\.67%in MoE\. Analysis of the dataloader reveals that the sequential curriculum file wrapped around back to Stage 1 \(pure addition/subtraction\), overwriting network weights previously tuned for division\.

## 8Qualitative Error Analysis and Failure Modes

### 8\.1Multiplication Summation Bottleneck

In Runs 7 and 8, we trained models on a specialized curriculum dataset incorporating explicit base\-10 zero accounting \(e\.g\.,80×50→8×5=4080\\times 50\\rightarrow 8\\times 5=40, append two zeros→4000\\rightarrow 4000\)\. Despite this innovation, multi\-digit multiplication scored 0\.0% on strict end\-to\-end evaluation\.

To uncover why multiplication failed, we conducted a token\-by\-token inspection of the generated evaluation traces:

Listing 1:Actual generation trace from Run 9 \(Math Step 30,000\)Q:Whatis983\*43?

A:<thought\>

983\*43=\(900\+80\+3\)\*\(40\+3\)\.

900\*40:9\*4=36with3zeros\-\>36000\.\[CORRECT\]

900\*3:9\*3=27with2zeros\-\>2700\.\[CORRECT\]

80\*40:8\*4=32with2zeros\-\>3200\.\[CORRECT\]

80\*3:8\*3=24with1zeros\-\>240\.\[CORRECT\]

3\*40:3\*4=12with1zeros\-\>120\.\[CORRECT\]

3\*3:9\.\[CORRECT\]

Sum:36000\+2700\+3200\+240\+120\+9=43992\.

</thought\>43992

Expected:42269\|Extracted:43992\[Evaluated:FALSE\]

Trace Analysis:In our qualitative examination of generated multiplication traces, the model doesnotfail at single\-digit multiplication facts \(9×4=369\\times 4=36,8×4=328\\times 4=32,3×3=93\\times 3=9\), nor at base\-10 zero accounting\. Intermediate sub\-products and zero\-padding are correctly formed\. The dominant point of failure occurs at the final multi\-term summation line:

Sum=36000\+2700\+3200\+240\+120\+9=43992\\text\{Sum\}=36000\+2700\+3200\+240\+120\+9=43992\(7\)The prompt format required the model to aggregate up to nine multi\-digit terms in a single autoregressive step without intermediate scratchpad accumulation\. While a∼\\sim10\.6M parameter network reliably computes two\-operand addition with near\-perfect accuracy, computing a nine\-term summation in a single forward pass exceeds the representational depth of a 6\-layer transformer\. Because of an arithmetic summation error on this single step, the entire problem was evaluated as incorrect\.

## 9Discussion and Limitations

### 9\.1Pairwise Column\-Summation Scratchpads

Our analysis suggests an architectural direction for multi\-digit multiplication in tiny models\. Since qualitative traces indicate that models reliably generate individual sub\-products \(900×40=36000900\\times 40=36000\), the primary bottleneck is the unrolled multi\-term summation\. Refactoring the final stage into an explicit pairwise accumulation scratchpad \(e\.g\.,36000\+2700=3870036000\+2700=38700;38700\+3200=4190038700\+3200=41900, etc\.\) would align with the model’s demonstrated 2\-operand addition capability, potentially closing the multiplication gap\.

### 9\.2Catastrophic Drift and the Necessity of Rehearsal Buffers

A central finding from our large\-scale 4,000\-problem evaluation \(Section[7](https://arxiv.org/html/2610.09003#S7)\) is the vulnerability of compact networks to catastrophic representational drift\. Between Step 30,000 and Step 50,000, addition and subtraction saturated from∼\\sim93% to over 99\.2% in\-distribution, yet single\-digit division execution collapsed from 86\.67% to 0\.00% in Modern Dense and 84\.33% to 1\.67% in Sparse MoE\.

Because compact∼\\sim10\.6M parameter networks operate with minimal parameter redundancy, late\-stage gradient updates overwrite previously learned algorithmic representations when training batches lack balanced task diversity\. In continual learning and sequential task training, this phenomenon is conventionally mitigated throughrehearsal buffers\(or experience replay\)\[[16](https://arxiv.org/html/2610.09003#bib.bib21)\], where a fixed proportion of historical task distributions \(e\.g\., 20% division, 20% multiplication, 30% addition, 30% subtraction\) is persistently interleaved into every active training batch\. Our empirical results demonstrate that while hierarchical staging is essential for initial skill acquisition, maintaining composite algorithmic reasoning across developmental stages strictly requires permanent replay buffers\.

### 9\.3Tokenization Confound

A notable consideration in our experiments is the use of the standard OpenAI GPT\-2 Byte Pair Encoding \(BPE\) tokenizer \(V=50,257V=50,257\)\. Unlike character\-level or digit\-level tokenizers, standard BPE tokenizes numbers non\-uniformly: many 2\-digit and 3\-digit numbers are represented as single discrete tokens, whereas 4\-digit numbers are split irregularly into subword tokens\. This token boundary shift represents a plausible confounding factor in our out\-of\-distribution length generalization collapse at 4 digits, suggesting that future investigations into small arithmetic models should evaluate digit\-level tokenizers\.

### 9\.4Limitations

We explicitly acknowledge several limitations of our empirical study:

1. 1\.Cold\-Start Algorithmic Staging:Runs 9 and 10 demonstrate the efficacy of pairing digit\-by\-digit scratchpads with staged developmental curricula\. While Run 7 establishes that curriculum staging alone cannot overcome ill\-conditioned scratchpads \(plateauing at 4\.0% division\), future work should evaluate whether digit\-by\-digit quotient descent can be acquired effectively from a uniform cold\-start distribution, or whether hierarchical sub\-skill mastery is strictly prerequisite\.
2. 2\.Single\-Seed Evaluations:Due to computational constraints across ten extended runs, the experimental trajectories reflect a single random seed\. While the large\-scale 4,000\-problem benchmark provides strong statistical power on evaluation, multi\-seed training runs will be valuable to characterize seed\-level variance\.
3. 3\.Decoding Stochasticity:Evaluation traces were generated via categorical sampling \(T=1\.0T=1\.0\), introducing stochastic variation across individual problem evaluations compared to deterministic greedy decoding\.
4. 4\.Fixed Hyperparameters:All configurations used identical, untuned hyperparameters \(η=3×10−4\\eta=3\\times 10^\{\-4\}\)\. Performance across individual architectures may further improve with targeted learning rate schedules and batch size tuning\.
5. 5\.Untested Rehearsal Buffers:While the drift mechanism was isolated to curriculum wrap\-around, implementing dynamic replay buffers during training remains future work\.

## 10Conclusion

In this paper, we systematically investigated the arithmetic reasoning limits of compact \(∼\\sim10\.6M non\-embedding parameter\) Transformers\. We validated that linguistic pretraining and sequence\-packed token density are essential prerequisites for arithmetic convergence\. We isolated the specific failure mode in multi\-digit multiplication, showing that models reliably learn sub\-multiplications and zero\-padding but falter when required to aggregate up to nine terms simultaneously without pairwise intermediate accumulation\. Finally, we demonstrated that restructuring long division into a deterministic digit\-by\-digit developmental curriculum elevates division accuracy from 4% to60%on development tests and86\.7%on single\-digit divisors on a 4,000\-problem held\-out benchmark\. Our findings demonstrate that algorithmic unrolling aligned with transformer forward\-pass capacity enables multi\-step arithmetic execution in parameter\-constrained regimes\.

## References

- \[1\]C\. Anil, Y\. Wu, A\. Andreassen, A\. Lewkowycz, V\. Misra, V\. Ramasesh, A\. Slone, G\. Gur\-Ari, E\. Dyer, and B\. Neyshabur\(2022\)Exploring length generalization in large language models\.Advances in Neural Information Processing Systems35,pp\. 38546–38556\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px3.p1.1)\.
- \[2\]Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston\(2009\)Curriculum learning\.InProceedings of the 26th Annual International Conference on Machine Learning,pp\. 41–48\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px4.p1.1)\.
- \[3\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in Neural Information Processing Systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2610.09003#S1.p1.1)\.
- \[4\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2610.09003#S1.p1.1),[§1](https://arxiv.org/html/2610.09003#S1.p2.2)\.
- \[5\]T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\(2022\)FlashAttention: fast and memory\-efficient exact attention with io\-awareness\.Advances in Neural Information Processing Systems35,pp\. 16344–16359\.Cited by:[item 2](https://arxiv.org/html/2610.09003#S3.I1.i2.p1.1)\.
- \[6\]N\. Dziri, X\. Lu, M\. Sclar, X\. L\. Li, L\. Jiang, B\. Y\. Lin, S\. Welleck, P\. West, C\. Bhagavatula, R\. Le Bras, J\. Hwang, S\. Sanyal, X\. Ren, A\. Ettinger, Z\. Harchaoui, and Y\. Choi\(2023\)Faith and fate: limits of transformers on compositionality\.Advances in Neural Information Processing Systems36,pp\. 70244–70275\.Cited by:[§1](https://arxiv.org/html/2610.09003#S1.p1.1),[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]R\. Eldan and Y\. Li\(2023\)TinyStories: how small can language models be and still speak coherent english?\.arXiv preprint arXiv:2305\.07759\.Cited by:[§1](https://arxiv.org/html/2610.09003#S1.p1.1)\.
- \[8\]A\. Kazemnejad, I\. Padhi, K\. N\. Ramamurthy, P\. Das, and S\. Reddy\(2023\)The impact of positional encoding on length generalization in transformers\.Advances in Neural Information Processing Systems36,pp\. 36093–36129\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px3.p1.1)\.
- \[9\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px4.p1.1)\.
- \[10\]N\. Lee, K\. Sreenivasan, J\. D\. Lee, K\. Lee, and D\. Papailiopoulos\(2023\)Teaching arithmetic to small transformers\.arXiv preprint arXiv:2307\.03381\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px2.p1.1),[§6\.1](https://arxiv.org/html/2610.09003#S6.SS1.p2.1)\.
- \[11\]M\. McCloskey and N\. J\. Cohen\(1989\)Catastrophic interference in connectionist networks: the sequential learning problem\.Psychology of Learning and Motivation24,pp\. 109–165\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px4.p1.1)\.
- \[12\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2016\)Pointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§6\.1](https://arxiv.org/html/2610.09003#S6.SS1.p1.1)\.
- \[13\]R\. Nogueira, Z\. Jiang, and J\. Lin\(2021\)Investigating the limitations of transformers with simple arithmetic tasks\.arXiv preprint arXiv:2102\.13019\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]M\. Nye, A\. J\. Andreassen, G\. Gur\-Ari, H\. Michalewski, J\. Austin, D\. Bieber, D\. Dohan, A\. Lewkowycz, M\. Bosma, D\. Luan, C\. Sutton, and A\. Odena\(2021\)Show your work: scratchpads for intermediate computation with language models\.arXiv preprint arXiv:2112\.00114\.Cited by:[item 4](https://arxiv.org/html/2610.09003#S1.I1.i4.p1.1),[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language models are unsupervised multitask learners\.OpenAI Blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2610.09003#S1.p1.1),[item 1](https://arxiv.org/html/2610.09003#S3.I1.i1.p1.1)\.
- \[16\]A\. Robins\(1995\)Catastrophic forgetting, rehearsal and pseudorehearsal for neural networks\.Connection Science7\(2\),pp\. 123–146\.Cited by:[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px4.p1.1),[§9\.2](https://arxiv.org/html/2610.09003#S9.SS2.p2.1)\.
- \[17\]N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[item 2](https://arxiv.org/html/2610.09003#S1.I1.i2.p1.1),[item 4](https://arxiv.org/html/2610.09003#S3.I1.i4.p1.1)\.
- \[18\]N\. Shazeer\(2020\)GLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[item 2](https://arxiv.org/html/2610.09003#S1.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2610.09003#S3.I1.i3.I1.i3.p1.1)\.
- \[19\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[item 2](https://arxiv.org/html/2610.09003#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px3.p1.1),[1st item](https://arxiv.org/html/2610.09003#S3.I1.i3.I1.i1.p1.1)\.
- \[20\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35,pp\. 24824–24837\.Cited by:[item 4](https://arxiv.org/html/2610.09003#S1.I1.i4.p1.1),[§1](https://arxiv.org/html/2610.09003#S1.p1.1),[§2](https://arxiv.org/html/2610.09003#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]B\. Zhang and R\. Sennrich\(2019\)Root mean square layer normalization\.Advances in Neural Information Processing Systems32,pp\. 12360–12371\.Cited by:[item 2](https://arxiv.org/html/2610.09003#S1.I1.i2.p1.1),[2nd item](https://arxiv.org/html/2610.09003#S3.I1.i3.I1.i2.p1.1)\.