Recursive transformers for semiconductor thermo-mechanical reliability

# Recursive weight-sharing transformers for semiconductor thermo-mechanical reliability
Source: [https://arxiv.org/html/2607.27251](https://arxiv.org/html/2607.27251)
###### Abstract

Transformer\-based surrogate models are increasingly used to replace expensive first\-principles simulation in engineering design\. But conventional transformer architectures are often over parameterized for the small, low\-dimensional datasets typical of engineering design spaces, where large simulation data is expensive to generate\. Under these conditions, excess parameter capacity leads to overfitting rather than improved accuracy, while also incurring unnecessary memory and compute overhead — this motivates a shift towards architectures that focus onadditional computerather thanadditional learnable parameters\. This paper presents a hardware\-aware evaluation of three recursive transformer paradigms for surrogate thermo\-mechanical analysis of advanced packages: a\)Tiny Recursive Model, b\) our proposed Depth Recursive transformer, c\) and a simple recursive transformer\. We systematically compare their predictive performance \(Recall@K, Mean Reciprocal Rank\), parameter count, computational complexity \(FLOPs\), providing practical design guidelines for selecting recursive transformer architectures under resource\-constrained scenarios\. We validate this principle on two low\-dimensional engineering prediction tasks: 1\) thermo\-mechanical reliability analysis of advanced semiconductor packages, where stress and warpage from thermal cycling must be evaluated repeatedly across a design\-of\-experiments sweep under costly finite element analysis \(FEA\)\. 2\) Laplace PDE iterative numerical solver for capacitance field\. Overall, recursive weight\-sharing transformers provide an effective and generalizable trade\-off between prediction accuracy, parameter efficiency, and computational cost for small\-data engineering surrogate modeling — a regime for which conventional large\-parameter transformers are poorly suited — as demonstrated on advanced package reliability prediction and capactior electrostatic field modeling\.

## IIntroduction

Transformer models have demonstrated strong predictive performance across a wide range of engineering applications, including electronic design automation \(EDA\)\[[7](https://arxiv.org/html/2607.27251#bib.bib8)\], scientific machine learning\[[16](https://arxiv.org/html/2607.27251#bib.bib17)\], and digital twins\[[25](https://arxiv.org/html/2607.27251#bib.bib25)\]\. However, their increasing computational and memory requirements present significant challenges for deployment in hardware\-constrained environments, such as on\-chip inference, edge accelerators, and real\-time design optimization\[[22](https://arxiv.org/html/2607.27251#bib.bib23),[2](https://arxiv.org/html/2607.27251#bib.bib3)\]\. Consequently, substantial research has focused on reducing transformer computation through improvements at the algorithmic, architectural, and system levels\[[3](https://arxiv.org/html/2607.27251#bib.bib5),[8](https://arxiv.org/html/2607.27251#bib.bib9),[5](https://arxiv.org/html/2607.27251#bib.bib6),[11](https://arxiv.org/html/2607.27251#bib.bib12)\]\. Thermo\-mechanical simulation remains computationally expensive, motivating the use of transformer\-based surrogate models for rapid reliability prediction\. However, deploying such models within practical EDA workflows requires architectures with low computational complexity, compact memory footprints, and high inference throughput\. These requirements have motivated a broad range of transformer compute reduction techniques and can be broadly categorized into seven complementary paradigms according to the aspect of computation they optimize\. These includeEfficient Attention\[[14](https://arxiv.org/html/2607.27251#bib.bib15),[3](https://arxiv.org/html/2607.27251#bib.bib5)\],Token Reduction\[[21](https://arxiv.org/html/2607.27251#bib.bib21),[1](https://arxiv.org/html/2607.27251#bib.bib2)\],Conditional Computation\[[8](https://arxiv.org/html/2607.27251#bib.bib9),[6](https://arxiv.org/html/2607.27251#bib.bib7)\],Architectural Optimization\[[5](https://arxiv.org/html/2607.27251#bib.bib6),[15](https://arxiv.org/html/2607.27251#bib.bib16),[24](https://arxiv.org/html/2607.27251#bib.bib24),[23](https://arxiv.org/html/2607.27251#bib.bib22)\],Model Compression\[[11](https://arxiv.org/html/2607.27251#bib.bib12),[12](https://arxiv.org/html/2607.27251#bib.bib13),[13](https://arxiv.org/html/2607.27251#bib.bib14)\],Hardware Optimization\[[4](https://arxiv.org/html/2607.27251#bib.bib4),[20](https://arxiv.org/html/2607.27251#bib.bib20)\], andState Space Models\[[10](https://arxiv.org/html/2607.27251#bib.bib11),[9](https://arxiv.org/html/2607.27251#bib.bib10)\]\. Although these approaches pursue the common objective of improving computational efficiency, they address different computational bottlenecks, ranging from reducing attention complexity and sequence length to redesigning network architectures or replacing attention\-based models altogether\. Among these paradigms, this work focuses onArchitectural Optimization, where computational efficiency is achieved by redesigning the transformer itself\. In particular, recursive weight sharing enables deeper computation without a proportional increase in parameters\[[5](https://arxiv.org/html/2607.27251#bib.bib6),[24](https://arxiv.org/html/2607.27251#bib.bib24),[23](https://arxiv.org/html/2607.27251#bib.bib22)\], making it particularly attractive for resource\-constrained EDA applications\. Although numerous approaches have been proposed to reduce transformer computation, their suitability depends on the target application\. Efficient attention mechanisms primarily benefit long\-sequence models where self\-attention dominates the computational cost\[[14](https://arxiv.org/html/2607.27251#bib.bib15),[3](https://arxiv.org/html/2607.27251#bib.bib5)\], while token reduction techniques rely on large token sets to achieve meaningful computational savings\[[21](https://arxiv.org/html/2607.27251#bib.bib21),[1](https://arxiv.org/html/2607.27251#bib.bib2)\]\. Conditional computation and Mixture\-of\-Experts architectures improve scalability for large foundation models but introduce additional routing complexity and memory overhead\[[8](https://arxiv.org/html/2607.27251#bib.bib9),[6](https://arxiv.org/html/2607.27251#bib.bib7),[19](https://arxiv.org/html/2607.27251#bib.bib1)\]\. Model compression and hardware\-specific optimizations are generally applied after model design or target specific execution platforms\[[11](https://arxiv.org/html/2607.27251#bib.bib12),[13](https://arxiv.org/html/2607.27251#bib.bib14),[4](https://arxiv.org/html/2607.27251#bib.bib4)\]\. In contrast, our application involves compact transformer surrogates operating on short input sequences under strict memory and computational constraints\. Consequently, Architectural Optimization provides the most suitable design strategy, enabling computation to be reduced intrinsically through recursive weight sharing while preserving a compact, hardware\-friendly architecture\[[5](https://arxiv.org/html/2607.27251#bib.bib6),[24](https://arxiv.org/html/2607.27251#bib.bib24),[23](https://arxiv.org/html/2607.27251#bib.bib22)\]\.

![Refer to caption](https://arxiv.org/html/2607.27251v1/NEW_TRM_CHATGPT4.png)Figure 1:Tiny Recursive model![Refer to caption](https://arxiv.org/html/2607.27251v1/NEW_DEPTH_CHATGPT5.png)Figure 2:Depth Recursive model![Refer to caption](https://arxiv.org/html/2607.27251v1/COPILOT_SIMPLE.png)Figure 3:Simple Recursive model
## IIRecursive weight sharing transformers

The main concept of recursive transformer is to treat transformer as a reusable template \(e\.g\.RECUR\\operatorname\{RECUR\}\) throughout the model\. Multiple replicas of RECUR are used in a model, but they all share the single set of weights, which is updated once during each training iteration\.RECUR\\operatorname\{RECUR\}should be independent of models we choose to deploy in\. The simplest configuration ofRECUR\\operatorname\{RECUR\}is a one block transformer in Eqn\.[1](https://arxiv.org/html/2607.27251#S2.E1)which consists of Self\-AttentionATTN⁡\(⋅\)\\operatorname\{ATTN\}\(\\cdot\), Normalization layerNORM⁡\(⋅\)\\operatorname\{NORM\}\(\\cdot\), Feed\-Forward Network \(FFN\)\. Conversely, if we want a deeper transformer, we can defineRECUR\\operatorname\{RECUR\}using e\.g\. two transformer blocks\.

RECUR=\\displaystyle\\operatorname\{RECUR\}=\{\}ATTN\(⋅∣WQ,WK,WV,WO\)\\displaystyle\\operatorname\{ATTN\}\(\\cdot\\mid W\_\{Q\},W\_\{K\},W\_\{V\},W\_\{O\}\)\(1\)→ADD1\+NORM1\(⋅∣γ1,β1\)\\displaystyle\\rightarrow\\operatorname\{ADD\}\_\{1\}\+\\operatorname\{NORM\}\_\{1\}\(\\cdot\\mid\\gamma\_\{1\},\\beta\_\{1\}\)→FFN\(⋅∣W1,W2\)\\displaystyle\\rightarrow\\operatorname\{FFN\}\(\\cdot\\mid W\_\{1\},W\_\{2\}\)→ADD2\+NORM2\(⋅∣γ2,β2\)\.\\displaystyle\\rightarrow\\operatorname\{ADD\}\_\{2\}\+\\operatorname\{NORM\}\_\{2\}\(\\cdot\\mid\\gamma\_\{2\},\\beta\_\{2\}\)\.
### II\-ATiny Recursive Model

The Tiny Recursive Model \(TRM\) uses weight sharing RECUR to cap the parameters while perform AI reasoning on Sudoku and puzzle tasks shown in Fig\.[1](https://arxiv.org/html/2607.27251#S1.F1)\. Within each iteration,RECURZ\\operatorname\{RECUR\}\_\{Z\}updates the latent feature𝐳\\mathbf\{z\}by feeding the input embeddings𝐄∈ℝL×dt​o​k​e​n=\[𝐞1,…,𝐞L\]\\mathbf\{E\}\\in\\mathbb\{R\}^\{L\\times d\_\{token\}\}=\[\\mathbf\{e\}\_\{1\},\\ldots,\\mathbf\{e\}\_\{L\}\]–whereLLis number of tokens anddt​o​k​e​nd\_\{token\}is the embedding dimension–along with the previous latent feature𝐳\(n−1\)\\mathbf\{z\}^\{\(n\-1\)\}, and the previous output feature𝐲\(n−1\)\\mathbf\{y\}^\{\(n\-1\)\}\. The updated latent feature𝐳\(n\)\\mathbf\{z\}^\{\(n\)\}is then passed toRECURY\\operatorname\{RECUR\}\_\{Y\}, which updates the output representation𝐲\(n\)\\mathbf\{y\}^\{\(n\)\}while masking the input embeddings with zeros\. Repeating this coupled update forNNtimes defines theLATENT\\operatorname\{LATENT\}operation in Eqn\.[2](https://arxiv.org/html/2607.27251#S2.E2)\. This progressively refine both𝐳\(n\)\\mathbf\{z\}^\{\(n\)\}and𝐲\(n\)\\mathbf\{y\}^\{\(n\)\}\.

LATENT≜\(RECURY∘RECURZ\)N\\displaystyle\\operatorname\{LATENT\}\\triangleq\\left\(\\operatorname\{RECUR\}\_\{Y\}\\circ\\operatorname\{RECUR\}\_\{Z\}\\right\)^\{N\}\(2\)𝐳\(n\)=RECURZ⁡\(\[𝐞1,…,𝐞L,𝐳\(n−1\),𝐲\(n−1\)\]\),\\displaystyle\\mathbf\{z\}^\{\(n\)\}=\\operatorname\{RECUR\_\{Z\}\}\\\!\\left\(\[\\mathbf\{e\}\_\{1\},\\ldots,\\mathbf\{e\}\_\{L\},\\mathbf\{z\}^\{\(n\-1\)\},\\mathbf\{y\}^\{\(n\-1\)\}\]\\right\),𝐲\(n\)=RECURY⁡\(\[𝟎,…,𝟎,𝐳\(n\),𝐲\(n−1\)\]\)\.\\displaystyle\\mathbf\{y\}^\{\(n\)\}=\\operatorname\{RECUR\_\{Y\}\}\\\!\\left\(\[\\mathbf\{0\},\\ldots,\\mathbf\{0\},\\mathbf\{z\}^\{\(n\)\},\\mathbf\{y\}^\{\(n\-1\)\}\]\\right\)\.Finally to compute TRM \(Eqn\.[3](https://arxiv.org/html/2607.27251#S2.E3)\),LATENT\\operatorname\{LATENT\}is executed forTTtimes and fed into anotherRECUR\\operatorname\{RECUR\}\. Increasing either the innerNNor the outerTTloops increases the computation cost \(FLOPs\) without increasing the parameters through weight sharing\.

TRM​\(𝐄,𝐳i​n​i,𝐲i​n​i\)≜RECUR∘\(LATENT\)T=RECUR⁡\(𝐄,\[\(RECURY⁡\(𝟎,RECURZ⁡\(𝐄,𝐳,𝐲\),𝐲\)\)N\]T\)\.\\begin\{split\}&\\mathrm\{TRM\}\\\!\\left\(\\mathbf\{E\},\\mathbf\{z\}^\{ini\},\\mathbf\{y\}^\{ini\}\\right\)\\triangleq\\operatorname\{RECUR\}\\circ\\left\(\\operatorname\{LATENT\}\\right\)^\{T\}\\\\ &=\\operatorname\{RECUR\}\\\!\\left\(\\mathbf\{E\},\\left\[\\left\(\\operatorname\{RECUR\}\_\{Y\}\\\!\\left\(\\mathbf\{0\},\\operatorname\{RECUR\}\_\{Z\}\\\!\\left\(\\mathbf\{E\},\\mathbf\{z\},\\mathbf\{y\}\\right\),\\mathbf\{y\}\\right\)\\right\)^\{N\}\\right\]^\{T\}\\right\)\.\\end\{split\}\(3\)

### II\-BDepth Recursive Model

From our domain expertise in advanced packaging, we observe that the dataset exhibits a gradually changing depth\-like structure for each fixed design case\. Motivated by this observation, we repurpose RECUR by treating sequential depth as a state input while fully retaining the weight sharing objective in Fig\.[2](https://arxiv.org/html/2607.27251#S1.F2)We call this approach as the Depth Recursive Model \(DEPTH\)\. Inspired by Recurrent Neural Networks \(RNNs\), DEPTH adapts the notion of hidden states to weight\-dependent sequential depth states within RECUR\. Using the terminology of RNN for sequential modeling, in DEPTH \(Eqn\.[4](https://arxiv.org/html/2607.27251#S2.E4)\) we define sequential state asS\(t\)S^\{\(t\)\}, initial hidden state as embedding input𝐄=𝐇\(0\)\\mathbf\{E\}=\\mathbf\{H\}^\{\(0\)\}, and sequential \(or recursive\) outputH\(t\+1\)H^\{\(t\+1\)\}for deptht∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\.

DEPTH\(𝐄St⊈E,𝐒\(1\),…,𝐒\(T\)\)≜\(RECUR∘\(⋅\+𝐒\(t\)𝐖s\)\)T=RECUR⁡\(𝐄\+𝐒\(1\)​𝐖s\)→𝐇\(1\)→RECUR⁡\(𝐇\(1\)\+𝐒\(2\)​𝐖s\)→𝐇\(2\)⋯→RECUR⁡\(𝐇\(T−1\)\+𝐒\(T\)​𝐖s\)→𝐇\(T\)\.\\begin\{array\}\[\]\{l\}\\mathrm\{DEPTH\}\\\!\\Big\(\\mathbf\{E\}\_\{S^\{t\}\\nsubseteq E\},\\mathbf\{S\}^\{\(1\)\},\\ldots,\\mathbf\{S\}^\{\(T\)\}\\Big\)\\triangleq\\left\(\\operatorname\{RECUR\}\\circ\\left\(\\cdot\+\\mathbf\{S\}^\{\(t\)\}\\mathbf\{W\}\_\{s\}\\right\)\\right\)^\{T\}\\\\\[5\.69054pt\] =\\operatorname\{RECUR\}\\\!\\left\(\\mathbf\{E\}\+\\mathbf\{S\}^\{\(1\)\}\\mathbf\{W\}\_\{s\}\\right\)\\rightarrow\\mathbf\{H\}^\{\(1\)\}\\\\ \\rightarrow\\operatorname\{RECUR\}\\\!\\left\(\\mathbf\{H\}^\{\(1\)\}\+\\mathbf\{S\}^\{\(2\)\}\\mathbf\{W\}\_\{s\}\\right\)\\rightarrow\\mathbf\{H\}^\{\(2\)\}\\\\ \\cdots\\rightarrow\\operatorname\{RECUR\}\\\!\\left\(\\mathbf\{H\}^\{\(T\-1\)\}\+\\mathbf\{S\}^\{\(T\)\}\\mathbf\{W\}\_\{s\}\\right\)\\rightarrow\\mathbf\{H\}^\{\(T\)\}\.\\end\{array\}\(4\)

### II\-CSimple Recursive model

The simple recursive model \(SIMPLE\) in Fig\.[3](https://arxiv.org/html/2607.27251#S1.F3)is seen to repeatedly apply an identical recursive transformation\. Using \(Eqn\.[5](https://arxiv.org/html/2607.27251#S2.E5)\), SIMPLE applies RECUR forTTrecursive iterations to generate the outputH\(T\)H^\{\(T\)\}\.

SIMPLE⁡\(𝐄\)\\displaystyle\\operatorname\{SIMPLE\}\(\\mathbf\{E\}\)≜\(RECUR\)T\\displaystyle\\triangleq\\left\(\\operatorname\{RECUR\}\\right\)^\{T\}\(5\)=RECUR⁡\(𝐄\)→𝐇\(1\)\\displaystyle=\\operatorname\{RECUR\}\(\\mathbf\{E\}\)\\rightarrow\\mathbf\{H\}^\{\(1\)\}→RECUR⁡\(𝐇\(1\)\)→𝐇\(2\)\\displaystyle\\rightarrow\\operatorname\{RECUR\}\\left\(\\mathbf\{H\}^\{\(1\)\}\\right\)\\rightarrow\\mathbf\{H\}^\{\(2\)\}…→RECUR⁡\(𝐇\(T−1\)\)→𝐇\(T\)\\displaystyle\\dots\\rightarrow\\operatorname\{RECUR\}\\left\(\\mathbf\{H\}^\{\(T\-1\)\}\\right\)\\rightarrow\\mathbf\{H\}^\{\(T\)\}

## IIIExperimental Results

### III\-ABaselines and proposed models

To thoroughly evaluate the tracking and retrieval performance, we compare our approach against seven baseline configurations categorized across four distinct architectural paradigms:

- •VANILLA:Consists ofM1, which serves as a baseline 1\-Block Layer Normalization \(LN\) Transformer architecture\.
- •SIMPLE:ComprisesM2andM3for Simple Recursive Model, representing a 2\-Block Root Mean Square \(RMS\) recursive structure configured with recursive steps ofT=1T=1andT=3T=3, respectively\.
- •TRM:Represents the Tiny Recursive Model, whereM4denotes a 1\-Block\-RMS variant incorporating a latent variable \(ZZ\), andM5denotes a 1\-Block\-LN variant operating over joint latent configurations \(Z,YZ,Y\)\.T=5T=5andN=3N=3were used\.
- •DEPTH:Comprises Depth Recursive Model, whereM6specifies a 1\-Block\-RMS design andM7represents a 2\-Block\-RMS architecture\. The depth is adapted toT=16T=16for both the Stress10K and Warpage10K dataset\.

### III\-BDatasets description

We evaluate recursive models across three datasets with partial DOE factorial cases,N=f​a​c​t​o​r​sl​e​v​e​l​sN=factors^\{levels\}—the Stress10k and Warpage10k datasets \(N=54×16=10,000N=5^\{4\}\\times 16=10\{,\}000pairs\) generated by FEA simulation\. Synthetic capacitor electrostatic field dataset generated by physics\-informed neural network \(PINN\) \(N=222×15=7,265N=22^\{2\}\\times 15=7\{,\}265pairs\)\.

Stress10k, Warpage10k\[[18](https://arxiv.org/html/2607.27251#bib.bib19)\]: Advanced packaging design requires understanding how material properties and geometric configurations influence thermo\-mechanical behavior across spatial and depth\-wise dimensions\. Design variables –epoxy molding compound \(EMC\) coefficient of thermal expansion \(CTE\) and elastic modulus \(E\), die sizes \(S​i​z​ed​i​e​sSize\_\{dies\}\), inter\-die spacing \(G​a​pd​i​e​sGap\_\{dies\}\), and position within the package depth \(S\(t\)S^\{\(t\)\}\) –determine the resulting stress and warpage responses obtained from finite element analysis \(FEA\)\.

𝐱𝐬𝐭𝐫𝐞𝐬𝐬=\[EMCCTE,EMCE,Sizedies,Gapdies,S\(t\)\]∈ℝ5×1\.\\displaystyle\\mathbf\{x\_\{stress\}\}=\\begin\{bmatrix\}\\mathrm\{EMC\}\_\{\\mathrm\{CTE\}\},\\;\\mathrm\{EMC\}\_\{E\},\\;\\mathrm\{Size\}\_\{\\mathrm\{dies\}\},\\;\\mathrm\{Gap\}\_\{\\mathrm\{dies\}\},\\;S^\{\(t\)\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{5\\times 1\}\.\(6\)𝐈𝐬𝐭𝐫𝐞𝐬𝐬=\{𝐈\(1\),𝐈\(2\),…,𝐈\(16\)\}∈ℝ13×13×16\\displaystyle\\mathbf\{I\_\{stress\}\}=\\left\\\{\\mathbf\{I\}^\{\(1\)\},\\mathbf\{I\}^\{\(2\)\},\\dots,\\mathbf\{I\}^\{\(16\)\}\\right\\\}\\in\\mathbb\{R\}^\{13\\times 13\\times 16\}PINN\[[17](https://arxiv.org/html/2607.27251#bib.bib18)\]: The electrostatic field within an air\-filled capacitor can be modeled by establishing collocation and boundary points across a two\-dimensional grid\. While the Laplace equation governs the internal collocation points, the system’s boundary conditions define the perimeter\. In this study, we implement five distinct boundary conditions \(C1 through C5\) in\[[17](https://arxiv.org/html/2607.27251#bib.bib18)\]governed by boundary parameters \(a,b,da,b,d\)\. Rather than treating all boundary metrics as static values, we allow the parameters to vary dynamically\. Modifying these specific parameters significantly alters the resulting electrostatic field distribution, as demonstrated in Figure 1\.

𝐱P​I​N​N=\[a,b,d\(t\)\]∈ℝ3×1\.\\displaystyle\\mathbf\{x\}\_\{PINN\}=\\begin\{bmatrix\}\\mathrm\{a\},\\mathrm\{b\},d^\{\(t\)\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{3\\times 1\}\.\(7\)𝐈P​I​N​N=\{𝐈\(1\),𝐈\(2\),…,𝐈\(15\)\}∈ℝ13×13×15\\displaystyle\\mathbf\{I\}\_\{PINN\}=\\left\\\{\\mathbf\{I\}^\{\(1\)\},\\mathbf\{I\}^\{\(2\)\},\\dots,\\mathbf\{I\}^\{\(15\)\}\\right\\\}\\in\\mathbb\{R\}^\{13\\times 13\\times 15\}
![Refer to caption](https://arxiv.org/html/2607.27251v1/pareto_copilot_mmr_flops.png)Figure 4:Higher FLOPs means a GPU takes longer to run on a single request\.![Refer to caption](https://arxiv.org/html/2607.27251v1/pareto_copilot_mmr_params.png)Figure 5:Unlike FLOPs, we cannot ”slowly” load a model into VRAM while running it; the entire model must fit into VRAM before inference begins\.
### III\-CPareto Analysis

Figure[4](https://arxiv.org/html/2607.27251#S3.F4)compares MRR against FLOPs across the three datasets\. The Pareto frontiers differ across datasets: Stress10K progresses from M1→\\rightarrowM2→\\rightarrowM3→\\rightarrowM7, indicating that M3 remains a competitive intermediate operating point; PINN follows M1→\\rightarrowM2→\\rightarrowM6→\\rightarrowM7, where depth conditioning first becomes Pareto\-optimal\. Warpage10K transitions M2→\\rightarrowM7, suggesting that the intermediate recursive variants are dominated\. The TRM variants \(M4–M5\) do not lie on the Pareto frontier for any dataset, indicating a less favorable accuracy\-to\-computation trade\-off\. Figure[5](https://arxiv.org/html/2607.27251#S3.F5)compares MRR against Parameters\. Unlike the FLOP analysis, the Pareto frontiers consistently favor DEPTH\. For PINN, Stress10K and Warpage10K, the Pareto frontier consists exclusively of the DEPTH models \(M6–M7\), where additional parameters yield further accuracy improvements while remaining Pareto\-optimal\. The TRM variants \(M4–M5\) remain well inside the Pareto region despite their larger parameter count, indicating that increasing parameter alone does not translate into better MRR\.

## IVConclusion

The proposed DEPTH \(M6, M7\) deliver the strongest or near\-strongest retrieval accuracy while using among the fewest parameters and lowest FLOPs of all seven models evaluated\. The naive weight\-sharing recursion of SIMPLE \(M2,M3\) is FLOP\-efficient but accuracy\-limited relative to M7 at comparable compute, while the dual\-latent\-state recursion of TRM \(M4,M5\) is both parameter\- and FLOP\-expensive without a commensurate accuracy return\. These results support the central claim that explicitly uses depth as a recursively injected input in which each recursive output contributes an individual loss term during BPTT, is the primary factor behind the favorable accuracy\-per\-parameter and accuracy\-per\-FLOP trade\-offs achieved for package\-level Stress10k, Warpage10k, and PINN prediction\.

## References

- \[1\]\(2023\)Token merging: your vit but faster\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[2\]C\. Cioflanet al\.\(2024\)Distributed inference with minimal off\-chip traffic for transformers on low\-power mcus\.arXiv preprint arXiv:2412\.04372\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[3\]T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\(2022\)FlashAttention: fast and memory\-efficient exact attention with io\-awareness\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[4\]T\. Dao\(2024\)FlashAttention\-2: faster attention with better parallelism\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[5\]M\. Dehghaniet al\.\(2019\)Universal transformers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[6\]N\. Duet al\.\(2022\)GLaM: efficient scaling of language models with mixture\-of\-experts\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[7\]S\. Fan, H\. Lu, S\. Zhang, N\. Cao, X\. Zhang, and J\. Li\(2024\)Graph\-transformer\-based surrogate model for accelerated converter circuit topology design\.InProceedings of the 61st ACM/IEEE Design Automation Conference \(DAC\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[8\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[9\]A\. Gu and T\. Dao\(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[10\]A\. Gu, K\. Goel, and C\. Ré\(2022\)Efficiently modeling long sequences with structured state spaces\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[11\]S\. Han, J\. Pool, J\. Tran, and W\. J\. Dally\(2015\)Learning both weights and connections for efficient neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[12\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.InNIPS Deep Learning Workshop,Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[13\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[14\]A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret\(2020\)Transformers are rnns: fast autoregressive transformers with linear attention\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[15\]Z\. Lan, M\. Chen, S\. Goodman, K\. Gimpel, P\. Sharma, and R\. Soricut\(2020\)ALBERT: a lite bert for self\-supervised learning of language representations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[16\]Z\. Li, K\. Meidani, and A\. B\. Farimani\(2023\)Transformer for partial differential equations’ operator learning\.Transactions on Machine Learning Research\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[17\]K\. Lim, R\. Dutta, and M\. Rotaru\(2023\)Inverse prediction of capacitor multiphysics dynamic parameters using deep generative model\.In2023 IEEE 73rd Electronic Components and Technology Conference \(ECTC\),pp\. 1825–1831\.Cited by:[§III\-B](https://arxiv.org/html/2607.27251#S3.SS2.p2.4)\.
- \[18\]K\. Lim and J\. Lin\(2025\)Deep clustering based boundary\-decoder net for inter and intra layer stress prediction of heterogeneous integrated ic chip\.In2025 IEEE 75th Electronic Components and Technology Conference \(ECTC\),pp\. 239–245\.Cited by:[§III\-B](https://arxiv.org/html/2607.27251#S3.SS2.p2.3)\.
- \[19\]Mistral AI\(2024\)Mixtral of experts\.arXiv preprint\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[20\]NVIDIA\(2023\)TensorRT\-llm\.Note:GitHub RepositoryCited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[21\]Y\. Raoet al\.\(2021\)DynamicViT: efficient vision transformers with dynamic token sparsification\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[22\]S\. Sahaet al\.\(2025\)Vision transformers on the edge: a comprehensive survey of model compression and acceleration strategies\.arXiv preprint arXiv:2503\.02891\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[23\]Samsung SAIL Research\(2025\)Tiny recursive models\.arXiv preprint\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[24\]L\. Yanget al\.\(2024\)Looped transformers are better at learning learning algorithms\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.
- \[25\]F\. Zhanget al\.\(2026\)Transformer self\-attention encoder–decoder with multimodal deep learning for response time series forecasting and digital twin support in wind structural health monitoring\.Preprint\.Cited by:[§I](https://arxiv.org/html/2607.27251#S1.p1.1)\.