神经路由求解器中的决策机制解析

# Understanding Decision-Making Mechanisms in Neural Routing Solvers
Source: [https://arxiv.org/html/2609.36063](https://arxiv.org/html/2609.36063)
Mazdak Teymourian11footnotemark:1Affiliation:Sharif University of TechnologyMohammad IzadiAffiliation:\{fatemehaskarijirhandeh, mazdak\.tey\}@gmail\.comMahdieh Soleymani BaghshahAffiliation:\{izadi, soleymani\}@sharif\.edu

###### Abstract

Neural Combinatorial Optimization \(NCO\) has achieved strong empirical success, yet the internal mechanisms driving model decisions remain largely unexplored\. In this paper, we investigate three representative autoregressive NCO models spanning two encoder\-decoder configurations: AM and POMO \(heavy\-encoder, light\-decoder\), and LEHD \(light\-encoder, heavy\-decoder\)\. Through behavioral analyses, representation probing, and causal interventions, we examine how these models construct solutions and use internal representations during decoding\. Our results suggest that AM and POMO predominantly follow a persistent geometric pattern throughout solution construction, whereas LEHD contains linearly accessible information about multiple future actions\. Causal experiments further provide evidence for the role of future\-node representations in LEHD’s decision\-making\. We also observe that LEHD relies strongly on the current\-node representation for immediate local decisions, while the start\-node representation plays a broader navigational role over the subsequent route\. Cross\-instance alignment analyses additionally indicate that LEHD maps current\-node representations into a relatively shared latent region, which may provide a stable reference for evaluating subsequent decisions\. Across the Traveling Salesman Problem and the Capacitated Vehicle Routing Problem, these results reveal distinct decision\-making patterns across these architecturally distinct solvers and provide a foundation for more interpretable analyses of NCO solvers\. Code and additional visualizations are provided in the[https://github\.com/NCO\-Interpretability/NCO\-Interpretability](https://github.com/NCO-Interpretability/NCO-Interpretability)\.

## 1Introduction

The Traveling Salesman Problem \(TSP\)\([Grötschel and Holland, 1991](https://arxiv.org/html/2609.36063#bib.bib23)\)and the Vehicle Routing Problem \(VRP\)\([Dantzig and Ramser, 1959](https://arxiv.org/html/2609.36063#bib.bib19)\)are among the most celebrated combinatorial problems in computer science, with critical applications spanning transportation\([Pillac et al\., 2013](https://arxiv.org/html/2609.36063#bib.bib20)\), logistics\([Konstantakopoulos et al\., 2022](https://arxiv.org/html/2609.36063#bib.bib21)\), and drug discovery\([Liu et al\., 2017](https://arxiv.org/html/2609.36063#bib.bib22)\)\. Due to the NP\-hard nature of these routing problems, exact algorithms are computationally prohibitive at scale\. While classical heuristics like Concorde\([Applegate et al\., 2006](https://arxiv.org/html/2609.36063#bib.bib24)\)and LKH3\([Helsgaun, 2017](https://arxiv.org/html/2609.36063#bib.bib25)\)produce high\-quality solutions, they incur heavy computational overhead\. Consequently, Neural Combinatorial Optimization \(NCO\) has emerged as a significantly faster alternative capable of achieving near\-optimal or superior performance on tailored distributions\.

Figure 1:Comparison of POMO and LEHD tour construction strategies\.Left:POMO constructs the tour in a clockwise manner while oscillating between shallow and deep onion layers, with each oscillation highlighted in a distinct color\.Middle:LEHD local navigation across consecutive steps\. Colored dashed lines represent candidate multi\-step plans, which LEHD dynamically revises at each step\.Right:LEHD global navigation\. LEHD evaluates candidate branches based on the relative angular position of the start node relative to the current node\. It prioritizes the branch visiting nodes further from the start first \(orange branch\) over the branch visiting closer nodes first \(blue branch\), minimizing the final return cost to the start node\.Among NCO approaches, Transformer\-based architectures\([Vaswani et al\., 2017](https://arxiv.org/html/2609.36063#bib.bib11)\)represent a dominant paradigm\. Trained via Reinforcement Learning \(RL\), AM\([Kool et al\., 2019](https://arxiv.org/html/2609.36063#bib.bib8)\)employs a heavy encoder and a lightweight decoder \(HELD\) to construct tours by sequentially appending unvisited nodes to the current partial path\. For each instance, AM uses a learned policy to choose the starting node\. POMO\([Kwon et al\., 2021](https://arxiv.org/html/2609.36063#bib.bib7)\)also uses a HELD architecture and RL to train the model but it also leverages the rotational and structural symmetries inherent to cyclic routing problems and uses different starting positions for each instance, thus not requiring a policy to select a starting node\. However, AM and POMO struggle to generalize to problem instances significantly larger than those seen during training\. Addressing this bottleneck, LEHD\([Luo et al\., 2024](https://arxiv.org/html/2609.36063#bib.bib6)\)shifts to a Supervised Learning \(SL\) paradigm using an inverted architectural allocation, a light encoder \(1 layer\) paired with a heavy decoder \(6 layers\)\. At each step, LEHD receives the start node, current node, and unvisited set to predict the next step\. Beyond autoregressive appending,[Luo et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib4)recently proposed L2C\-Insert, an insertion\-based heuristic that decouples generation into node selection and placement phases, using an encoder\-decoder network to determine optimal insertion points within the partial tour\.

Despite the rapid evolution of NCO architectures, the internal mechanisms driving their predictions remain largely black boxes\. A few pioneering studies have begun peeling back these layers:[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib1)used linear probes to demonstrate that Euclidean distances are linearly decodable from internal representations and identified specific embedding dimensions critical to LEHD’s predictions, while[Narad et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib2)applied Sparse Autoencoders \(SAEs\) to show that latent neurons capture geometric features such as boundary detection and spatial clustering\. While insightful, these prior efforts offer limited mechanistic insights, focusing primarily on correlational analysis rather than establishing causality\. Consequently, a fundamental question remains open:what explicit, macro\-level solving strategies do these networks actually learn and execute?To bridge this gap, we present a comprehensive interpretability framework, combining behavioral probing, representation analysis, activation patching, and causal analysis, to systematically dissect the strategies of leading NCO models\. To the best of our knowledge, ours is the first work to employ causal interventions to explain the decision\-making mechanisms of these solvers\.

Using this framework, we analyze AM, POMO, and LEHD as representative construction paradigms, uncovering distinct spatial and mechanistic behaviors:

- •AM and POMO:Construct tours through a rigid spatial pattern, sweeping anti\-clockwise for AM and clockwise for POMO from the initial node while systematically oscillating between shallow and deep convex \(onion\) layers \(Figure[1](https://arxiv.org/html/2609.36063#S1.F1)\)\. As detailed in Section[4](https://arxiv.org/html/2609.36063#S4), this structural rigidity stems directly from training on uniform point distributions and shifts predictably when retrained on clustered instances\.
- •LEHD:Exhibits evidence of two complementary decision components which share similarities with Model Predictive Control \(MPC\)\([Richalet et al\., 1978](https://arxiv.org/html/2609.36063#bib.bib26)\)\. Locally, its final decoder layers encode explicit multi\-step trajectories over an effective horizon, executing the immediate step before dynamically revising its trajectory \(Figure[1](https://arxiv.org/html/2609.36063#S1.F1)\)\. Globally, LEHD utilizes the relative angular position of the start node to steer navigation toward distant unvisited clusters first, minimizing the eventual return cost\. We also find that LEHD maps different current nodes to a relatively shared latent embedding space across instances and steps, suggesting that it plans in a canonical, current\-centered latent reference frame\.

Our main contributions are summarized as follows:

- •We conduct a diverse set of experiments to provide a mechanistic explanation of how distinct NCO architectures construct routing solutions\.
- •We characterize the geometric structure of the tours constructed by AM and POMO, demonstrating the rigidity of their learned strategy and how the training data distribution influences it\.
- •We reveal that LEHD coordinates a local look\-ahead horizon with global spatial reference\-frame navigation, explaining its strong generalization capabilities across problem scales\.

Core findings generalize across both TSP and CVRP \(details in Appendix[F](https://arxiv.org/html/2609.36063#A6)\)\. Appendices A–G cover related work, extended results, extra visualizations, and experimental setups\.

Figure 2:Behavioral comparison of NCO solvers on TSP\-500 under seven synthetic distribution shifts\. AM, POMO, LEHD, L2C\-Insert, and cluster\-trained POMO are compared by optimality gap and convex\-hull order violations across Clustered, Expansion, Explosion, Grid, Implosion, Mixed, and Uniform distributions, together with inference runtime\. The results highlight differences in solution quality, geometric consistency, and computational efficiency under diverse instance geometries\.
## 2Behavioral Results and Model Comparison

We compare AM, POMO, and LEHD as representative append\-based solvers with different encoder–decoder allocations, and include L2C\-Insert as an insertion\-based reference\. Throughout the paper, all models are evaluated using greedy, single\-trajectory decoding without model\-specific test\-time enhancements, such as POMO’s rotational augmentation or LEHD’s RRC\. This evaluation protocol allows us to study the intrinsic decision\-making behavior of the learned policies without introducing additional inference\-time modifications that may obscure the underlying model mechanisms\.

All models are trained on Uniform TSP\-100 and evaluated in Figure[2](https://arxiv.org/html/2609.36063#S1.F2)on TSP\-500 across seven distributions, thereby testing both size and distribution generalization\. We report optimality gap, convex\-hull violation rate, and inference runtime on a logarithmic scale; a hull violation denotes failure to preserve the cyclic order of convex\-hull vertices, a necessary property of an optimal Euclidean TSP tour\.

L2C\-Insert incurs substantially higher inference costs without consistently improving solution quality over LEHD\. LEHD achieves competitive solution quality across diverse distributions while maintaining substantially lower inference cost than the insertion\-based baseline\. Therefore, we retain L2C\-Insert as an insertion\-based reference for behavioral comparison, while focusing our mechanistic analyses on AM, POMO, and LEHD due to their shared append\-based construction process, which enables a controlled comparison of their decision mechanisms\. Additional details are provided in Appendix[C](https://arxiv.org/html/2609.36063#A3)\.

\(a\)Mean onion layer oscillation\(b\)Mean intra\-cluster oscillation\(c\)Angle comparison per step
Figure 3:Tracking geometrical properties of generated tours\.\(a\)Onion layers are segmented intoBBbins to calculate the average number of oscillations between shallow and deeper layers, where higher values indicate a greater number of deep oscillations\.\(b\)Each cluster is partitioned into binwise onion layers to count high\-intensity oscillations within individual clusters\.\(c\)The average start\-to\-current node angle relative to each tour’s first edge is tracked throughout tour construction\. A decreasing angle indicates a clockwise rotation of the start\-to\-current displacement vector\.
## 3Experimental Framework

This section presents the experimental framework used to interpret NCO solvers\. Following the problem formulations in Appendix[A](https://arxiv.org/html/2609.36063#A1), we first characterize their observable geometric behavior and then examine the representational and causal mechanisms underlying their decisions\. The experimental settings are provided in Appendix[G](https://arxiv.org/html/2609.36063#A7)and the corresponding CVRP analyses are reported in Appendix[F](https://arxiv.org/html/2609.36063#A6)\. These analyses are subsequently synthesized in Section[4](https://arxiv.org/html/2609.36063#S4)and Section[5](https://arxiv.org/html/2609.36063#S5)to develop an interpretation of the distinct decision\-making strategies learned by AM, POMO, and LEHD\.

### 3\.1Geometric Trajectory Analysis

We characterize the generated solutions through two complementary geometric summaries: onion\-depth progression and start\-centered angular progression\. The first measures how a solver moves across nested convex layers of the point set, while the second describes how selected nodes are distributed angularly with respect to the first generated edge\.

#### 3\.1\.1Onion\-Depth Progression

We use onion decomposition to characterize the layer\-wise structure of TSP instances\([Chazelle, 1985](https://arxiv.org/html/2609.36063#bib.bib18);[de Berg et al\., 2000](https://arxiv.org/html/2609.36063#bib.bib17)\)\. It recursively removes the convex hull of the remaining points, partitioning them into nested layersρt\\rho\_\{t\}\(Appendix[A\.3](https://arxiv.org/html/2609.36063#A1.SS3)\)\. We then discretizeρt∈\[0,1\]\\rho\_\{t\}\\in\[0,1\]intoBBequal\-width binsbt=1\+min⁡\(B−1,⌊B​ρt⌋\)\.b\_\{t\}=1\+\\min\\left\(B\-1,\\left\\lfloor B\\rho\_\{t\}\\right\\rfloor\\right\)\.

Bin11is the outermost region and binBBthe deepest\. After removing consecutive duplicate bins, we count an oscillation when the trajectory leaves bin11, reaches at leastbmin=min⁡\(B,⌊τ​B⌋\+1\)b\_\{\\min\}=\\min\\left\(B,\\left\\lfloor\\tau B\\right\\rfloor\+1\\right\), and returns to bin11, whereτ∈\(0,1\]\\tau\\in\(0,1\]controls the required depth\. We report the average number of such oscillations across problem instances\.

#### 3\.1\.2Angular Progression

We measure start\-centered angular progression using the first generated edge as the reference axis,a=xπ1−xπ0a=x\_\{\\pi\_\{1\}\}\-x\_\{\\pi\_\{0\}\}, whereπ0\\pi\_\{0\}is the start node\. At stept∈\{1,…,N−1\}t\\in\\\{1,\\ldots,N\-1\\\}, we compare it withvt=xπt−xπ0v\_\{t\}=x\_\{\\pi\_\{t\}\}\-x\_\{\\pi\_\{0\}\}\. The signed angular position is computed asθt=atan2⁡\(ax​vty−ay​vtx,a⊤​vt\)\.\\theta\_\{t\}=\\operatorname\{atan2\}\\left\(a^\{x\}v\_\{t\}^\{y\}\-a^\{y\}v\_\{t\}^\{x\},a^\{\\top\}v\_\{t\}\\right\)\.

We unwrapθt\\theta\_\{t\}to remove artificial discontinuities at the\[−180∘,180∘\]\[\-180^\{\\circ\},180^\{\\circ\}\]boundary, allowing the angular progression to vary continuously beyond this interval\. We then compute the mean signed angle across instances at each decoding step, capturing systematic clockwise or counterclockwise progression around the start node\.

For CVRP, angles are measured within each route relative to its first edge, excluding depot returns\. Routes are aligned by normalized progress before aggregation\.

### 3\.2Future\-Action Planning Probes

We use linear probes to examine whether neural combinatorial solvers encode information about their own future decisions\. At decoding steptt, the frozen model has produced the partial solutionπ≤t=\[π0,π1,…,πt\]\\pi\_\{\\leq t\}=\[\\pi\_\{0\},\\pi\_\{1\},\\ldots,\\pi\_\{t\}\], and the current decoding state defines a feasible candidate set𝒜t\\mathcal\{A\}\_\{t\}\. For TSP, this set consists of the unvisited nodes,𝒜t=V∖\{π0,…,πt\}\\mathcal\{A\}\_\{t\}=V\\setminus\\\{\\pi\_\{0\},\\ldots,\\pi\_\{t\}\\\}\. For routing problems with additional constraints, such as CVRP,𝒜t\\mathcal\{A\}\_\{t\}contains all unvisited customers, regardless of current capacity constraints\. For a future horizonhh, the probing target is the action selected by the same frozen model at stept\+ht\+hduring its greedy rollout,yt,h=πt\+hy\_\{t,h\}=\\pi\_\{t\+h\}\. The probe therefore does not predict the optimal solution; instead, it tests whether the model’s current internal representation already contains information about its own future trajectory\. For each feasible candidatei∈𝒜ti\\in\\mathcal\{A\}\_\{t\}, we extract a 128\-dimensional candidate\-specific representationrt,i\(ℓ\)r\_\{t,i\}^\{\(\\ell\)\}from internal layerℓ\\ell\. A separate linear probe is trained for each layerℓ\\elland horizonhh, assigning one scalar score to every feasible candidate:st,i\(ℓ,h\)=Wℓ,h​rt,i\(ℓ\)\+bℓ,h\.s\_\{t,i\}^\{\(\\ell,h\)\}=W\_\{\\ell,h\}r\_\{t,i\}^\{\(\\ell\)\}\+b\_\{\\ell,h\}\.The probe is optimized using cross\-entropy loss\.

This formulation treats future\-action prediction as a candidate\-ranking problem\. If a linear probe can recover the model’s future actionπt\+h\\pi\_\{t\+h\}from the representation at steptt, then that representation contains linearly accessible information about the model’s subsequent decisions\. We apply the same probing protocol to the POMO, AM, and LEHD \(Figure[4\(b\)](https://arxiv.org/html/2609.36063#S3.F4.sf2)\), while the model\-specific representation choices are described in their corresponding analysis sections\.

\(a\)LEHD attention\(b\)Probe accuracy per planning horizon\(c\)Probe\-guided steering
Figure 4:\(a\)LEHD attention heatmap from the current node to the unvisited nodes\.\(b\)Probe accuracy across planning horizons for the decoder layers of LEHD, AM, and POMO\.\(c\)Probe\-guided counterfactual steering for LEHD on TSP\-100 atα=5\\alpha=5, compared against random\-direction and random\-horizon baselines\. Bars show the mean change in logit gap between the clean top\-two actions
### 3\.3Probe\-Guided Counterfactual Steering

Inspired by prior work combining planning probes with causal interventions and activation steering\([Bush et al\., 2025](https://arxiv.org/html/2609.36063#bib.bib34);[Li et al\., 2023](https://arxiv.org/html/2609.36063#bib.bib35);[Panickssery et al\., 2023](https://arxiv.org/html/2609.36063#bib.bib36);[Li et al\., 2022](https://arxiv.org/html/2609.36063#bib.bib37);[Nanda et al\., 2023](https://arxiv.org/html/2609.36063#bib.bib38)\), we investigate whether amplifying representations associated with an alternative future route can shift the model’s immediate action preference\.

At decoding steptt, letaaanda′a^\{\\prime\}denote the highest\- and second\-highest\-probability feasible actions, respectively\. We construct a counterfactual continuation by forcingπt\+1′=a′\\pi^\{\\prime\}\_\{t\+1\}=a^\{\\prime\}and then following the frozen model’s greedy policy\. We ask whether intervening on representations of nodes appearing at future steps along this counterfactual continuation can causally shift the model’s immediate action preference towarda′a^\{\\prime\}\.

For future horizonsh∈ℋh\\in\\mathcal\{H\}and decoder layersℓ∈ℒ\\ell\\in\\mathcal\{L\}, we steer the representation of the counterfactual future nodeyt,h′=πt\+h′y^\{\\prime\}\_\{t,h\}=\\pi^\{\\prime\}\_\{t\+h\}along its probe direction:r~t,yt,h′\(ℓ\)=rt,yt,h′\(ℓ\)\+α​Wℓ,h⊤/‖Wℓ,h‖22\\widetilde\{r\}\_\{t,y^\{\\prime\}\_\{t,h\}\}^\{\(\\ell\)\}=r\_\{t,y^\{\\prime\}\_\{t,h\}\}^\{\(\\ell\)\}\+\\alpha W\_\{\\ell,h\}^\{\\top\}/\\\|W\_\{\\ell,h\}\\\|\_\{2\}^\{2\}, whereα\>0\\alpha\>0controls intervention strength\. We jointly steer horizonsℋ=\{2,3,4,5\}\\mathcal\{H\}=\\\{2,3,4,5\\\}, without directly modifying the immediate candidatesaaora′a^\{\\prime\}\.

We then recompute the action logits without forcinga′a^\{\\prime\}and measure the change in its preference overaa:Δ​Gt=\[zsteered​\(a′\)−zsteered​\(a\)\]−\[zclean​\(a′\)−zclean​\(a\)\]\\Delta G\_\{t\}=\[z\_\{\\mathrm\{steered\}\}\(a^\{\\prime\}\)\-z\_\{\\mathrm\{steered\}\}\(a\)\]\-\[z\_\{\\mathrm\{clean\}\}\(a^\{\\prime\}\)\-z\_\{\\mathrm\{clean\}\}\(a\)\]\. Controls use norm\-matched random directions or an equal number of disjoint random future horizons\.

### 3\.4Node\-Role Representation Alignment

We measure the cross\-instance cosine similarity of node representations at each decoding step\. For each layer, decoding step, and node role, we compare the corresponding representations across problem instances\. This quantifies how consistently each functional role is represented across instances throughout decoding\.

### 3\.5Current and Start Node Contribution

We use activation patching \(Appendix[B\.2](https://arxiv.org/html/2609.36063#A2.SS2)\) to examine the roles of two state\-defining node representations during decoding: the current node and the start node\. The current node represents the local position from which the next decision is made, while the start node is a fixed element of the generated tour that remains available to the decoder throughout the rollout\. In both interventions below, we modify the encoder\-produced node embeddings before they are passed to the decoder\.

#### 3\.5\.1Node Attribution

We perform stepwise mean ablations of the start and current node representations\. At each decoding step, we compute a mean embedding over all clean encoded node representations collected from a reference set of instances at that step, and temporarily replace either the start\-node or current\-node representation with this mean\. This removes instance\-specific information while keeping the remaining decoding state unchanged\. We then measure the drop in the clean next\-node probability to assess the influence of each representation on the immediate decision\.

#### 3\.5\.2Targeted Intervention

In the start\-node patching experiment, we replace the encoded representation of the original start node with the representation of an already visited donor node, i\.e\.,Hπ0′=HdH^\{\\prime\}\_\{\\pi\_\{0\}\}=H\_\{d\}, whereπ0\\pi\_\{0\}is the original start node andddis the donor node\. This intervention changes the start\-node representation while leaving the decoding state otherwise unchanged\. We select donor nodes using three geometric conditions: maximum angle without distance constraints, maximum angle under similar distance, and similar angle with different distance\. These conditions allow us to separate the effects of angular direction and distance in the start\-node representation\.

\(a\)LEHD mean ablation\(b\)POMO vs\. LEHD start\-node patching\(c\)LEHD trajectory
Figure 5:Causal interventions on node representations\.\(a\)Drop in the clean next\-node probability after mean ablation of the current and start nodes\.\(b\)Effect of start\-node activation patching under three geometric donor conditions for LEHD and POMO\.\(c\)Example clean and patched LEHD trajectories showing route changes over 22 decoding steps\.

## 4Interpretability Analysis of AM and POMO

As noted in Section[1](https://arxiv.org/html/2609.36063#S1), both AM and POMO employ an asymmetrical architecture consisting of multiple encoder layers and a single decoder layer\. Because the current decoding step is omitted from the encoder embeddings and processed exclusively by the lightweight decoder, we hypothesize that AM and POMO’s construction strategy relies less on local geometric structures around the current node and more on a generalized global heuristic\. To test this hypothesis, we examine the geometric properties of tours generated by these models and compare them against alternative solvers\.

Following the methodology in Section[3\.1](https://arxiv.org/html/2609.36063#S3.SS1), we track the angular displacement of the current node relative to the start node throughout tour construction\. As shown in Figure[3\(c\)](https://arxiv.org/html/2609.36063#S2.F3.sf3), a clear behavioral divergence emerges: while baseline methods dynamically adjust their angular direction without an explicit directional bias,POMO exhibits a systematic tendency to construct tours in a clockwise manner while AM constructs in an anti\-clockwise manner\.

Furthermore,both of these models oscillate between different onion\-layer depths with substantially higher frequency and intensitythan competing solvers \(Figure[3\(a\)](https://arxiv.org/html/2609.36063#S2.F3.sf1)\)\. This spatial disparity becomes increasingly pronounced on larger problem scales, helping explain why AM and POMO underperform relative to LEHD and L2C\-Insert, both of which adapt more dynamically to local instance geometry \(see Section[5](https://arxiv.org/html/2609.36063#S5)\)\.

This observation aligns with findings by[Huang et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib5), who noted that lightweight decoder architectures yield less context\-aware decoding decisions, ultimately limiting performance\. To mitigate this constraint, light\-decoder architectures frequently employ multi\-start sampling at inference time, generating candidate tours from multiple initial nodes per instance and selecting the minimal\-cost path\. We argue that the efficacy of multi\-start sampling stems directly from counteracting POMO’s rigid geometric strategy\. However, because test\-time sampling provides only constrained flexibility compared to the innate adaptivity of greedy LEHD, the performance gap between the two models widens as instance sizes increase \(Appendix[C](https://arxiv.org/html/2609.36063#A3)\)\.

This structural rigidity also accounts for AM and POMO’s performance degradation on clustered distributions \(Figure[2](https://arxiv.org/html/2609.36063#S1.F2)\)\. Because these models are trained exclusively on uniform point distributions, their learned heuristics function adequately on uniform data but fail to generalize to clustered instances, forcing unnecessary inter\-cluster jumps that severely inflate total tour cost and hull violation\. Interestingly, while POMO suffers a larger overall performance gap, its rigid clockwise progression better preserves the cyclic order of convex\-hull nodes on near\-uniform distributions, while degrading sharply on clustered and mixed distributions \(Figure[2](https://arxiv.org/html/2609.36063#S1.F2)\)\. This also applies to AM\.

To evaluate how training data distributions influence POMO’s learned strategy, we retrained the model on clustered datasets from[Bi et al\. \(2023\)](https://arxiv.org/html/2609.36063#bib.bib10)and measured intra\-cluster layer oscillations\. As illustrated in Figure[3\(b\)](https://arxiv.org/html/2609.36063#S2.F3.sf2), this retrained variant oscillates significantly more within individual clusters than baseline methods—and even exceeds the oscillation intensity of the original uniform\-trained POMO\. This shift demonstrates that the training distribution directly governs the learned heuristic: when trained on clustered instances, POMO restricts its layer oscillations within a cluster before transitioning to the next\. This adaptation yields substantial performance gains across all evaluated instance distributions \(Figure[2](https://arxiv.org/html/2609.36063#S1.F2)\), suggesting that diversifying training data distributions offers a promising avenue for enhancing NCO generalization more broadly\. We trained AM and LEHD on clustered data as well, as depicted in Appendix[C\.1](https://arxiv.org/html/2609.36063#A3.SS1)\.

Remarkably, the angular progressions of AM and POMO persist even in their cluster\-trained variant, indicating that they apply their core strategy hierarchically within each cluster, effectively treating individual clusters as distinct subproblems\. This reinforces[Bi et al\. \(2023\)](https://arxiv.org/html/2609.36063#bib.bib10), demonstrating that light\-decoder models operate similarly when solving full instances or decomposed subproblems\.

## 5Interpretability Analysis of LEHD

While POMO’s routing strategy is relatively rigid, as discussed in Section[4](https://arxiv.org/html/2609.36063#S4), LEHD’s strategy is considerably more adaptive to the underlying geometry of the nodes\. Our analysis reveals that LEHD’s navigation paradigm comprises two distinct components: alocal navigationmechanism that optimizes paths across immediate local neighborhoods, and aglobal navigationmechanism that steers the local trajectory toward a broader, global target direction\. Below, we dissect each component in detail\.

### 5\.1Local Navigation

How does LEHD select its immediate next node—and does it plan ahead when doing so? Because LEHD computes attention over all nodes during decoding, we first visualized its attention heatmaps\. This revealed a striking pattern \(see Figure[4\(a\)](https://arxiv.org/html/2609.36063#S3.F4.sf1)\): at any given step, the current node attends heavily not only to the immediate next node but also to a sequence of nodes slated for addition in subsequent decoding steps\. This suggests that LEHD selects its next node by explicitly accounting for its planned trajectory over a longer horizon, a strategy that shows some resemblance to Model Predictive Control \(MPC\)\([Richalet et al\., 1978](https://arxiv.org/html/2609.36063#bib.bib26)\)\. In MPC, a future path is optimized up to a fixed predictive horizon, but only the first control step is executed; looking ahead prevents short\-sighted, purely greedy decisions\.

To test whether future trajectory information is actively encoded in the decoder, we trained linear classifier probes on the latent representations from the final two layers\. The full probing protocol is detailed in Section[3\.2](https://arxiv.org/html/2609.36063#S3.SS2)\. We define the horizon as the number of decoding steps between the current node and the node being predicted\.

Probing accuracies are shown in Figure[4\(b\)](https://arxiv.org/html/2609.36063#S3.F4.sf2), alongside AM and POMO as baselines\. For POMO and AM, we probe candidate\-wise products of decoder contexts and final encoder embeddings or logit keys, respectively\. At horizon 1 \(the immediate next node\), all models achieve near\-perfect accuracy, as expected: all three architectures use a linear projection over these candidate\-specific representations to compute the next\-node distribution\. At longer horizons, however, a stark divergence emerges\. LEHD predicts the node two steps ahead with approximately 80% accuracy, indicating that substantial future\-path information is explicitly retained in its latent space\. AM and POMO, by contrast, degrade sharply, especially on larger instances, implying that such information is absent or heavily obscured in their representations\. This gap persists up to horizons of 4–5 steps, beyond which LEHD’s probe accuracy also decays, delineating the upper bound of its effective look\-ahead horizon\. To rule out the possibility of the probes predicting the nearest neighbor, in Section[D\.1](https://arxiv.org/html/2609.36063#A4.SS1)we also evaluate them on instances where LEHD does not choose the nearest neighbor\.

Probing establishes thepresenceof future\-path information, but not that the model uses it\. To test causal relevance, we turn to the counterfactual steering described in Section[3\.3](https://arxiv.org/html/2609.36063#S3.SS3)\. As shown in Figure[4\(c\)](https://arxiv.org/html/2609.36063#S3.F4.sf3), slightly modifying the embeddings of nodes in the second most probable branch changes the model’s preference toward choosinga1′a^\{\\prime\}\_\{1\}as its immediate next action, without altering the embeddings of the immediate next nodes\. Random\-direction updates, or updates to nodes far along the horizon, leave LEHD’s decision unchanged, consistent with the bounded effective horizon identified by the probes\. We further assess potential off\-manifold effects of these interventions in Appendix[D\.3](https://arxiv.org/html/2609.36063#A4.SS3)\. This provides causal evidence that LEHD relies on its encoded look\-ahead trajectory to determine its local actions\. A complementary behavioral causal experiment in Appendix[D\.2](https://arxiv.org/html/2609.36063#A4.SS2)further examines POMO and LEHD and corroborates this conclusion\.

Table 1:TSP\-100 rollout\-order similarity comparison between LCS and Rev\. LCS\.

Table 2:Layer\-wise cross\-instance alignment of current, start, and random\-node representations in LEHD on Uniform TSP\-100\. Enc\. and Dec\. denote encoder and decoder layers, respectively\.

### 5\.2Global Navigation

While local navigation governs immediate look\-ahead dependencies, LEHD also relies on a global navigation component that steers the route toward a broader target direction\. To isolate the role of the start node in this mechanism, we first compare it with the current node using the mean\-ablation intervention described in Section[3\.5\.1](https://arxiv.org/html/2609.36063#S3.SS5.SSS1)\. As shown in Figure[5\(a\)](https://arxiv.org/html/2609.36063#S3.F5.sf1), ablating the current node produces a much larger drop in the clean next\-node probability than ablating the start node\. This indicates that immediate next\-node selection is governed primarily by the current local state, whereas the start node has a weaker direct effect on local transitions and may instead support higher\-level guidance\.

We examine this role more directly using the activation\-transfer intervention introduced in Section[3\.5\.2](https://arxiv.org/html/2609.36063#S3.SS5.SSS2), where the original start\-node representationSSis replaced by that of a donor nodeS′S^\{\\prime\}\. As shown in Figure[5\(b\)](https://arxiv.org/html/2609.36063#S3.F5.sf2),maximum angle without distance constraintsproduces the largest drop in the clean next\-node probability, followed bymaximum angle under similar distance, whilesimilar angle with different distanceproduces the smallest effect\. The first two conditions preserve a large angular displacement under different distance constraints, whereas the third substantially changes distance while approximately preserving direction\. This ordering suggests that the intervention effect is more strongly associated with the angular displacement∠⁡\(S,C,S′\)\\angle\(S,C,S^\{\\prime\}\), defined by the original start node, the current node, and the donor node, than with the distance betweenSSandS′S^\{\\prime\}\. In contrast, the corresponding POMO curves remain comparatively close to zero and exhibit little separation across the three donor conditions\. Thus, POMO’s immediate decisions appear largely insensitive to the directional information introduced through the patched start representation, whereas LEHD responds strongly to changes in this direction\. The corresponding results for AM, which selects its starting node through its learned policy, are discussed in Section[D\.4](https://arxiv.org/html/2609.36063#A4.SS4)\.

The qualitative example in Figure[5\(c\)](https://arxiv.org/html/2609.36063#S3.F5.sf3)illustrates the resulting reorientation\. In this instance, the patched start node lies at an angle of nearly180∘180^\{\\circ\}relative to the original start node, and the generated route changes its overall direction so that the remaining trajectory approaches the patched anchor through a shorter completion path\. To quantify this effect, Table[1](https://arxiv.org/html/2609.36063#S5.T1)compares the ordering of the unvisited nodes in the clean and patched rollouts\. LCS is the normalized longest common subsequence between the clean rollout order and the patched rollout order, and therefore measures how much of the original forward ordering is preserved\. Rev\. LCS instead compares the patched rollout order with the reversed clean rollout, and measures whether the intervention causes the remaining nodes to be visited in the opposite direction\.

For LEHD, increasing∠⁡\(S,C,S′\)\\angle\(S,C,S^\{\\prime\}\)consistently decreases LCS and increases Rev\. LCS\. At the largest angular displacements, reversed agreement exceeds forward agreement, indicating a substantial reordering of the future selection sequence toward the reverse direction\. In contrast, POMO maintains relatively high LCS and low Rev\. LCS across the full range of angular interventions, suggesting that its rollout order remains comparatively stable even under large directional perturbations\. Together, these results suggest that the start node provides global directional guidance in LEHD, but has much less influence in POMO\.

### 5\.3Latent Space

To examine whether LEHD organizes node representations according to their functional roles, we apply the cross\-instance alignment analysis described in Section[3\.4](https://arxiv.org/html/2609.36063#S3.SS4)\. As shown in Table[2](https://arxiv.org/html/2609.36063#S5.T2), the current\-node representation exhibits high cosine similarity across instances from decoder layer 2 onward\. Although the current node changes at every decoding step and occupies different geometric locations across instances, its representation remains strongly aligned at the same step\. This suggests that LEHD maps functionally equivalent current nodes into a shared latent region, potentially providing a consistent reference for subsequent decisions\. The start node, by contrast, remains fixed throughout the rollout, yet its alignment increases more gradually and peaks in deeper decoder layers, consistent with a broader navigational role becoming more strongly expressed later in the computation\. Randomly selected unvisited nodes show larger variability and less consistent alignment\. Overall, these results suggest that LEHD progressively organizes its latent space around distinct task\-specific node roles\. Additional visualizations are provided in Appendix[E](https://arxiv.org/html/2609.36063#A5)\.

## 6Conclusion and Future Work

In this work, we conducted an extensive and diverse set of experiments to provide a comprehensive understanding of the strategies underlying three of the most well\-known neural NCO solvers for routing problems\. Our results reveal that these models employ distinct strategies to solve TSP and CVRP instances\. These insights, and the methods used to derive them, can inform the design of better solvers and guide future work aimed at deepening our understanding of these models\.

Several promising future directions emerge from this study\. First, training on more diverse data distributions could improve robustness and generalization\. Second, LEHD could be trained to predict multiple nodes along the path at once instead of only the next node, enabling it to plan further ahead and potentially strengthening its routing capability\. Third, increasing the number of decoder layers relative to encoder layers appears to increase model flexibility and improve generalization\. While this work focuses on three representative NCO solvers, extending the analysis to other solvers is another interesting direction\.

## AI Use Statement

In this work, we used generative AI tools for implementing methods \(specifically, assisting in the implementation of some experiments\)\. We have not used generative AI tools for generating synthetic data sets, developing theoretical models or conceptual frameworks, formulating or proving mathematical claims, proposing or refining hypotheses, designing research methodology or experiments, supporting qualitative or thematic data analysis, interpreting results, or assisting with translation or dataset cleaning and reformatting\. Additionally, we used generative AI tools for creating or editing software code \(primarily for initial code scaffolding, such as generating plots or loading raw data\) and for editing the paper to improve readability \(proofreading and polishing the text of this manuscript\)\. We have reviewed all AI\-assisted work\. Specifically, all code generated with LLM assistance was rigorously reviewed, tested, and validated by multiple co\-authors to ensure its correctness and integration into our codebase, and all AI\-assisted text edits were reviewed by the authors for accuracy and intent\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

## References

- Alain and Bengio \(2018\)G\. Alain and Y\. BengioUnderstanding intermediate layers using linear classifier probes\.External Links:1610\.01644,[Link](https://arxiv.org/abs/1610.01644)Cited by:[§B\.2\.1](https://arxiv.org/html/2609.36063#A2.SS2.SSS1.p1.1)\.
- Applegateet al\.\(2006\)D\. L\. Applegate, R\. E\. Bixby, V\. Chvátal, and W\. J\. CookThe traveling salesman problem: a computational study\.Princeton University Press\.Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- Belinkov \(2021\)Y\. BelinkovProbing classifiers: promises, shortcomings, and advances\.External Links:2102\.12452,[Link](https://arxiv.org/abs/2102.12452)Cited by:[§B\.2\.1](https://arxiv.org/html/2609.36063#A2.SS2.SSS1.p1.1)\.
- Biet al\.\(2023\)J\. Bi, Y\. Ma, J\. Wang, Z\. Cao, J\. Chen, Y\. Sun, and Y\. M\. CheeLearning generalizable models for vehicle routing problems via knowledge distillation\.External Links:2210\.07686,[Link](https://arxiv.org/abs/2210.07686)Cited by:[§4](https://arxiv.org/html/2609.36063#S4.p6.1),[§4](https://arxiv.org/html/2609.36063#S4.p7.1)\.
- Bushet al\.\(2025\)T\. Bush, S\. Chung, U\. Anwar, A\. Garriga\-Alonso, and D\. KruegerInterpreting emergent planning in model\-free reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 82115–82197\.Cited by:[§3\.3](https://arxiv.org/html/2609.36063#S3.SS3.p1.1)\.
- Chazelle \(1985\)B\. ChazelleOn the convex layers of a planar set\.IEEE Transactions on Information Theory31\(4\),pp\. 509–517\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1985.1057060)Cited by:[§3\.1\.1](https://arxiv.org/html/2609.36063#S3.SS1.SSS1.p1.1)\.
- Dantzig and Ramser \(1959\)G\. B\. Dantzig and J\. H\. RamserThe truck dispatching problem\.Management Science6,pp\. 80–91\.Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- de Berget al\.\(2000\)M\. de Berg, M\. van Kreveld, M\. Overmars, and O\. SchwarzkopfComputational geometry: algorithms and applications\.Second edition,Springer\-Verlag\.External Links:[Link](http://www.cs.uu.nl/geobook/)Cited by:[§3\.1\.1](https://arxiv.org/html/2609.36063#S3.SS1.SSS1.p1.1)\.
- Geigeret al\.\(2025\)A\. Geiger, D\. Ibeling, A\. Zur, M\. Chaudhary, S\. Chauhan, J\. Huang, A\. Arora, Z\. Wu, N\. Goodman, C\. Potts,et al\.Causal abstraction: a theoretical foundation for mechanistic interpretability\.Journal of Machine Learning Research26\(83\),pp\. 1–64\.Cited by:[§B\.2\.2](https://arxiv.org/html/2609.36063#A2.SS2.SSS2.p1.1)\.
- Grötschel and Holland \(1991\)M\. Grötschel and O\. HollandSolution of large\-scale symmetric travelling salesman problems\.Math\. Program\.51\(1–3\),pp\. 141–202\.External Links:ISSN 0025\-5610Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- Heimersheim and Nanda \(2024\)S\. Heimersheim and N\. NandaHow to use and interpret activation patching\.External Links:2404\.15255,[Link](https://arxiv.org/abs/2404.15255)Cited by:[§B\.2\.2](https://arxiv.org/html/2609.36063#A2.SS2.SSS2.p1.1)\.
- Helsgaun \(2017\)K\. HelsgaunAn extension of the lin\-kernighan\-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems: technical report\.Roskilde Universitet\(English\)\.Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- Hochreiter and Schmidhuber \(1997\)S\. Hochreiter and J\. SchmidhuberLong short\-term memory\.Neural Comput\.9\(8\),pp\. 1735–1780\.External Links:ISSN 0899\-7667,[Link](https://doi.org/10.1162/neco.1997.9.8.1735),[Document](https://dx.doi.org/10.1162/neco.1997.9.8.1735)Cited by:[§B\.1\.2](https://arxiv.org/html/2609.36063#A2.SS1.SSS2.p1.1)\.
- Huanget al\.\(2025\)Z\. Huang, J\. Zhou, Z\. Cao, and Y\. XURethinking light decoder\-based solvers for vehicle routing problems\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4pRwkYpa2u)Cited by:[§4](https://arxiv.org/html/2609.36063#S4.p4.1)\.
- Jenneret al\.\(2024\)E\. Jenner, S\. Kapur, V\. Georgiev, C\. Allen, S\. Emmons, and S\. RussellEvidence of learned look\-ahead in a chess\-playing neural network\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§B\.2\.3](https://arxiv.org/html/2609.36063#A2.SS2.SSS3.p1.1)\.
- Kikutaet al\.\(2024\)D\. Kikuta, H\. Ikeuchi, K\. Tajiri, and Y\. NakanoRouteExplainer: an explanation framework for vehicle routing problem\.InAdvances in Knowledge Discovery and Data Mining,pp\. 30–42\.External Links:ISBN 9789819722594,ISSN 1611\-3349,[Link](http://dx.doi.org/10.1007/978-981-97-2259-4_3),[Document](https://dx.doi.org/10.1007/978-981-97-2259-4%5F3)Cited by:[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p3.1),[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p4.1)\.
- Konstantakopouloset al\.\(2022\)G\. D\. Konstantakopoulos, S\. P\. Gayialis, and E\. P\. KechagiasVehicle routing problem and related algorithms for logistics distribution: a literature review and classification\.Operational Research22\(3\),pp\. 2033–2062\.External Links:[Document](https://dx.doi.org/10.1007/s12351-020-00600-7),[Link](https://ideas.repec.org/a/spr/operea/v22y2022i3d10.1007_s12351-020-00600-7.html)Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- Koolet al\.\(2019\)W\. Kool, H\. van Hoof, and M\. WellingAttention, learn to solve routing problems\!\.External Links:1803\.08475,[Link](https://arxiv.org/abs/1803.08475)Cited by:[§B\.1\.2](https://arxiv.org/html/2609.36063#A2.SS1.SSS2.p1.1),[§1](https://arxiv.org/html/2609.36063#S1.p2.1)\.
- Kwonet al\.\(2021\)Y\. Kwon, J\. Choo, B\. Kim, I\. Yoon, Y\. Gwon, and S\. MinPOMO: policy optimization with multiple optima for reinforcement learning\.External Links:2010\.16011,[Link](https://arxiv.org/abs/2010.16011)Cited by:[§B\.1\.3](https://arxiv.org/html/2609.36063#A2.SS1.SSS3.p1.1),[§1](https://arxiv.org/html/2609.36063#S1.p2.1)\.
- Liet al\.\(2022\)K\. Li, A\. K\. Hopkins, D\. Bau, F\. Viégas, H\. Pfister, and M\. WattenbergEmergent world representations: exploring a sequence model trained on a synthetic task\.arXiv preprint arXiv:2210\.13382\.Cited by:[§3\.3](https://arxiv.org/html/2609.36063#S3.SS3.p1.1)\.
- Liet al\.\(2023\)K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. WattenbergInference\-time intervention: eliciting truthful answers from a language model\.Advances in neural information processing systems36,pp\. 41451–41530\.Cited by:[§3\.3](https://arxiv.org/html/2609.36063#S3.SS3.p1.1)\.
- Liuet al\.\(2017\)R\. Liu, X\. Li, and K\. S\. LamCombinatorial chemistry in drug discovery\.Current Opinion in Chemical Biology38,pp\. 117–126\.Note:Next Generation TherapeuticsExternal Links:ISSN 1367\-5931,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.cbpa.2017.03.017),[Link](https://www.sciencedirect.com/science/article/pii/S1367593117300534)Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- Luoet al\.\(2024\)F\. Luo, X\. Lin, F\. Liu, Q\. Zhang, and Z\. WangNeural combinatorial optimization with heavy decoder: toward large scale generalization\.External Links:2310\.07985,[Link](https://arxiv.org/abs/2310.07985)Cited by:[§B\.1\.4](https://arxiv.org/html/2609.36063#A2.SS1.SSS4.p1.1),[§1](https://arxiv.org/html/2609.36063#S1.p2.1)\.
- Luoet al\.\(2025\)F\. Luo, X\. Lin, M\. Zhong, F\. Liu, Z\. Wang, J\. Sun, and Q\. ZhangLearning to insert for constructive neural vehicle routing solver\.External Links:2505\.13904,[Link](https://arxiv.org/abs/2505.13904)Cited by:[§B\.1\.5](https://arxiv.org/html/2609.36063#A2.SS1.SSS5.p1.1),[§1](https://arxiv.org/html/2609.36063#S1.p2.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. J\. Andonian, and Y\. BelinkovLocating and editing factual associations in gpt\.InAdvances in neural information processing systems,Cited by:[§B\.2\.2](https://arxiv.org/html/2609.36063#A2.SS2.SSS2.p1.1)\.
- Nandaet al\.\(2023\)N\. Nanda, A\. Lee, and M\. WattenbergEmergent linear representations in world models of self\-supervised sequence models\.arXiv preprint arXiv:2309\.00941\.External Links:[Link](https://arxiv.org/abs/2309.00941)Cited by:[§3\.3](https://arxiv.org/html/2609.36063#S3.SS3.p1.1)\.
- Naradet al\.\(2025\)R\. Narad, L\. Boussioux, and M\. WagnerMechanistic interpretability for neural tsp solvers\.External Links:2510\.21693,[Link](https://arxiv.org/abs/2510.21693)Cited by:[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p2.1),[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p4.1),[§1](https://arxiv.org/html/2609.36063#S1.p3.1)\.
- Panicksseryet al\.\(2023\)N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. TurnerSteering llama 2 via contrastive activation addition, 2024\.URL https://arxiv\. org/abs/2312\.066813\.Cited by:[§3\.3](https://arxiv.org/html/2609.36063#S3.SS3.p1.1)\.
- Pillacet al\.\(2013\)V\. Pillac, M\. Gendreau, C\. Guéret, and A\. L\. MedagliaA review of dynamic vehicle routing problems\.European Journal of Operational Research225\(1\),pp\. 1–11\.External Links:ISSN 0377\-2217,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.ejor.2012.08.015),[Link](https://www.sciencedirect.com/science/article/pii/S0377221712006388)Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p1.1)\.
- Richaletet al\.\(1978\)J\. Richalet, A\. Rault, J\.L\. Testud, and J\. PaponModel predictive heuristic control: applications to industrial processes\.Automatica14\(5\),pp\. 413–428\.External Links:ISSN 0005\-1098,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0005-1098%2878%2990001-8),[Link](https://www.sciencedirect.com/science/article/pii/0005109878900018)Cited by:[2nd item](https://arxiv.org/html/2609.36063#S1.I1.i2.p1.1),[§5\.1](https://arxiv.org/html/2609.36063#S5.SS1.p1.1)\.
- Taufeequeet al\.\(2025\)M\. Taufeeque, P\. Quirke, M\. Li, C\. Cundy, A\. D\. Tucker, A\. Gleave, and A\. Garriga\-AlonsoPlanning in a recurrent neural network that plays sokoban\.External Links:2407\.15421,[Link](https://arxiv.org/abs/2407.15421)Cited by:[§B\.2\.3](https://arxiv.org/html/2609.36063#A2.SS2.SSS3.p1.1)\.
- Taufeequeet al\.\(2026\)M\. Taufeeque, A\. D\. Tucker, A\. Gleave, and A\. Garriga\-AlonsoPath channels and plan extension kernels: a mechanistic description of planning in a sokoban RNN\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=aAshH4kQ1v)Cited by:[§B\.2\.3](https://arxiv.org/html/2609.36063#A2.SS2.SSS3.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2609.36063#S1.p2.1)\.
- Vinyalset al\.\(2017\)O\. Vinyals, M\. Fortunato, and N\. JaitlyPointer networks\.External Links:1506\.03134,[Link](https://arxiv.org/abs/1506.03134)Cited by:[§B\.1\.1](https://arxiv.org/html/2609.36063#A2.SS1.SSS1.p1.1),[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p2.1)\.
- Wanget al\.\(2022\)K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. SteinhardtInterpretability in the wild: a circuit for indirect object identification in gpt\-2 small\.arXiv preprint arXiv:2211\.00593\.Cited by:[§B\.2\.2](https://arxiv.org/html/2609.36063#A2.SS2.SSS2.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3–4\),pp\. 229–256\.Cited by:[§B\.1\.2](https://arxiv.org/html/2609.36063#A2.SS1.SSS2.p1.1)\.
- Zhang and Nanda \(2023\)F\. Zhang and N\. NandaTowards best practices of activation patching in language models: metrics and methods\.arXiv preprint arXiv:2309\.16042\.Cited by:[§B\.2\.2](https://arxiv.org/html/2609.36063#A2.SS2.SSS2.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, Y\. Ma, Z\. Cao, and H\. C\. LauProbing neural combinatorial optimization models\.External Links:2510\.22131Cited by:[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p1.1),[§B\.3](https://arxiv.org/html/2609.36063#A2.SS3.p4.1),[§1](https://arxiv.org/html/2609.36063#S1.p3.1)\.

## Appendix APreliminaries

### A\.1Traveling Salesman Problem

Given a complete graphG=\(V,E\)G=\(V,E\)withnnnodes and edge costsci​jc\_\{ij\}, the TSP seeks a minimum\-cost Hamiltonian cycle visiting each node exactly once\. Formally, we seek a permutationπ=\[π0,π1,…,πn−1\]\\pi=\[\\pi\_\{0\},\\pi\_\{1\},\\ldots,\\pi\_\{n\-1\}\]that minimizes:

minπ⁡\(∑i=0n−2cπi,πi\+1\+cπn−1,π0\)\.\\min\_\{\\pi\}\\left\(\\sum\_\{i=0\}^\{n\-2\}c\_\{\\pi\_\{i\},\\pi\_\{i\+1\}\}\+c\_\{\\pi\_\{n\-1\},\\pi\_\{0\}\}\\right\)\.\(1\)

### A\.2Capacitated Vehicle Routing Problem

In the CVRP, a vehicle with capacityQQmust service customer demandsdi\>0d\_\{i\}\>0from a central depotv0v\_\{0\}\. The goal is to find a set of routesℛ\\mathcal\{R\}minimizing total travel cost while respecting capacity constraints:

minℛ\\displaystyle\\min\_\{\\mathcal\{R\}\}∑ℛk∈ℛ∑t=0\|ℛk\|−2crtk,rt\+1k,\\displaystyle\\sum\_\{\\mathcal\{R\}\_\{k\}\\in\\mathcal\{R\}\}\\sum\_\{t=0\}^\{\|\\mathcal\{R\}\_\{k\}\|\-2\}c\_\{r\_\{t\}^\{k\},r\_\{t\+1\}^\{k\}\},\(2\)s\.t\.\\displaystyle\\text\{s\.t\.\}∑t=1\|ℛk\|−2drtk≤Q,∀ℛk∈ℛ\.\\displaystyle\\sum\_\{t=1\}^\{\|\\mathcal\{R\}\_\{k\}\|\-2\}d\_\{r\_\{t\}^\{k\}\}\\leq Q,\\quad\\forall\\mathcal\{R\}\_\{k\}\\in\\mathcal\{R\}\.Each routeℛk=\[r0k,r1k,…,r\|ℛk\|−1k\]\\mathcal\{R\}\_\{k\}=\[r\_\{0\}^\{k\},r\_\{1\}^\{k\},\\ldots,r\_\{\|\\mathcal\{R\}\_\{k\}\|\-1\}^\{k\}\]starts and ends at the depot \(r0k=r\|ℛk\|−1k=v0r\_\{0\}^\{k\}=r\_\{\|\\mathcal\{R\}\_\{k\}\|\-1\}^\{k\}=v\_\{0\}\), and every customer is visited exactly once\.

### A\.3Onion Decomposition

Onion decomposition recursively removes the convex hull of the remaining points, partitioning them into nested layers\. A node has onion depthd⁡\(v\)=kd\(v\)=kif it is removed at peeling iterationkk; outer\-hull nodes have depth zero, while larger values indicate deeper layers\.

Letπt\\pi\_\{t\}denote the node selected at decoding steptt\. We normalize its depth as

ρt=d⁡\(πt\)max⁡\(1,maxv∈𝒱⁡d⁡\(v\)\)\.\\rho\_\{t\}=\\frac\{d\\left\(\\pi\_\{t\}\\right\)\}\{\\max\\left\(1,\\max\_\{v\\in\\mathcal\{V\}\}d\(v\)\\right\)\}\.\(3\)

## Appendix BRelated Work

### B\.1NCO Solvers

#### B\.1\.1Pointer Networks

The paradigm of using deep neural networks to learn heuristics for solving NP\-hard routing problems was pioneered by Pointer Networks\([Vinyals et al\., 2017](https://arxiv.org/html/2609.36063#bib.bib9)\)\. Unlike traditional sequence\-to\-sequence models that are constrained by a fixed output vocabulary, Pointer Networks employ a modified attention mechanism as a dynamic pointing layer, enabling the model to select outputs directly from variable\-length input sequences\.

#### B\.1\.2AM

While Pointer Networks were originally implemented using recurrent architectures such as LSTMs\([Hochreiter and Schmidhuber, 1997](https://arxiv.org/html/2609.36063#bib.bib16)\), the Attention Model \(AM\)\([Kool et al\., 2019](https://arxiv.org/html/2609.36063#bib.bib8)\)adopted a Transformer\-based architecture\. AM achieves permutation invariance, allowing the model to treat the input graph as an unordered spatial set\. To eliminate the need for ground\-truth optimal solutions during training, AM introduced the use of reinforcement learning through the REINFORCE algorithm\([Williams, 1992](https://arxiv.org/html/2609.36063#bib.bib15)\), together with a rollout baseline to reduce gradient variance\.

#### B\.1\.3POMO

Although AM demonstrated the effectiveness of Transformer\-based routing solvers, its training process still suffered from high variance\. Policy Optimization with Multiple Optima \(POMO\)\([Kwon et al\., 2021](https://arxiv.org/html/2609.36063#bib.bib7)\)addressed this limitation by exploiting the rotational and structural symmetries inherent in cyclic routing problems such as the TSP\. Instead of constructing a single trajectory, POMO generatesNNtrajectories for a given graph instance, each initialized from a different starting node\.

#### B\.1\.4LEHD

A persistent limitation of both AM and POMO is their difficulty in generalizing to graph sizes substantially larger than those encountered during training\. Unlike POMO, LEHD\([Luo et al\., 2024](https://arxiv.org/html/2609.36063#bib.bib6)\)adopts a supervised learning paradigm\. It employs a Transformer\-based architecture consisting of a one\-layer encoder and a six\-layer decoder\. At each decoding step, the model receives the starting node, the current node, and the set of unvisited nodes, and predicts the next node to append to the partial route\.

#### B\.1\.5L2C\-Insert

Both POMO and LEHD are construction heuristics that generate solutions through successive node appending\. Aiming to overcome the limitations of appending\-based strategies,[Luo et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib4)proposed L2C\-Insert, an insertion\-based solver\. The method consists of a node\-selection phase, in which a candidate node is chosen based on its proximity to the previously selected node, followed by an insertion phase\. During the insertion phase, a Transformer encoder\-decoder architecture determines the optimal position of the selected node within the current partial tour, thereby incrementally expanding the solution\.

### B\.2Mechanistic Interpretability

Mechanistic Interpretability \(MI\) seeks to uncover the internal computations performed by neural networks by identifying the information encoded in their representations and establishing causal relationships between model components and observed behavior\.

#### B\.2\.1Probing

Probing is one of the most widely used approaches for analyzing neural representations\. A probe is typically a simple linear classifier or regressor trained to predict a target property from a model’s hidden activations\. Strong probe performance suggests that information relevant to the target property is encoded in the examined representation\([Alain and Bengio, 2018](https://arxiv.org/html/2609.36063#bib.bib12)\)\. Consequently, probing has become a standard tool for investigating the information accessible at different layers of a neural network\([Belinkov, 2021](https://arxiv.org/html/2609.36063#bib.bib13)\)\.

#### B\.2\.2Activation Patching

While probing can reveal whether information is present in a representation, it cannot determine whether that information is causally used by the model\. Activation patching addresses this limitation by replacing activations from one input with the corresponding activations from another and measuring the resulting change in model behavior\. By analyzing the effects of such interventions, activation patching can identify components that are causally responsible for specific computations\([Meng et al\., 2022](https://arxiv.org/html/2609.36063#bib.bib27);[Wang et al\., 2022](https://arxiv.org/html/2609.36063#bib.bib28);[Zhang and Nanda, 2023](https://arxiv.org/html/2609.36063#bib.bib29);[Geiger et al\., 2025](https://arxiv.org/html/2609.36063#bib.bib30)\)\. The technique has been successfully applied to uncover computational circuits and information flow in Transformer\-based models\([Heimersheim and Nanda, 2024](https://arxiv.org/html/2609.36063#bib.bib14)\)\.

#### B\.2\.3Look\-Ahead and Planning in Sequential Decision\-Making Models

Outside of NLP, a closely related line of MI work asks whether networks trained end\-to\-end on sequential decision\-making tasks represent their own future actions, and whether these representations are causally used rather than merely correlated with behavior\.[Jenner et al\. \(2024\)](https://arxiv.org/html/2609.36063#bib.bib31)combine linear probing with activation patching on Leela Chess Zero, the strongest open\-source chess engine, and find that its policy network linearly encodes the optimal move several turns ahead of the current position, and that these representations are causally necessary for its output in certain board states\. In a parallel line of work on a different planning domain,[Taufeeque et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib32)train linear probes that decode a recurrent network’s future actions roughly 50 steps in advance while it plays Sokoban, and show via intervention on the hidden state that these probed representations causally steer the agent’s subsequent behavior;[Taufeeque et al\. \(2026\)](https://arxiv.org/html/2609.36063#bib.bib33)extend this analysis into a full circuit\-level account, localizing directional “path channels” that implement a bidirectional, plan\-extending search\. Both lines of work establish the same two\-step recipe that we adopt: probing to establish that a representation of a future decision exists, followed by causal intervention to establish that the model actually relies on it\. Our contribution is to bring this recipe to bear on NCO routing solvers, where a variable\-size, permutation\-sensitive candidate set and the absence of a fixed board or grid structure require a different probe formulation — candidate\-ranking over the feasible action set rather than a fixed square\- or move\-indexed classification target — than either the chess or Sokoban settings\.

### B\.3Interpretability of NCO Solvers

Despite the rapid progress of neural combinatorial optimization, the internal mechanisms underlying these solvers remain largely unexplored\. One of the first studies to investigate this question is the work of[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib1)\. Using probing techniques, they showed that Euclidean distance information is encoded in the internal representations of AM, POMO, and LEHD\. Furthermore, their analysis suggested that these models do not behave purely myopically when constructing solutions: their myopia\-avoidance probe is framed as a single\-step, binary classification between the globally optimal edge and the locally greedy \(nearest\-neighbor\) edge at the current decision, evaluated on the same two backbones we study\. This probe does not test whether the model encodes a specific candidate action at longer horizons, nor whether any such representation is causally used, both of which are the focus of our Future\-Action Planning Probes and the causal interventions\. Also by examining the weights of the probe they found that there exists two dimensions in the latent space of LEHD which have high importance in its node selection\.

More recently,[Narad et al\. \(2025\)](https://arxiv.org/html/2609.36063#bib.bib2)applied sparse autoencoders \(SAEs\) to a pointer network\-style model\([Vinyals et al\., 2017](https://arxiv.org/html/2609.36063#bib.bib9)\)and found evidence that the learned representations capture interpretable geometric concepts, including boundary detection and spatial clustering; the authors explicitly identify causal circuit analysis via activation patching as future work rather than something their study performs\.

Complementary to these mechanistic approaches,[Kikuta et al\. \(2024\)](https://arxiv.org/html/2609.36063#bib.bib3)proposed RouteExplainer, a post\-hoc explanation framework for VRP solutions\. RouteExplainer quantifies the influence of individual edges on the generated route, introduces a pipeline for generating counterfactual explanations, and leverages large language models \(LLMs\) to enhance the interpretability of the resulting explanations\.

Taken together, existing interpretability studies of NCO solvers are exclusively correlational — probing\([Zhang et al\., 2025](https://arxiv.org/html/2609.36063#bib.bib1)\), sparse autoencoders\([Narad et al\., 2025](https://arxiv.org/html/2609.36063#bib.bib2)\), post\-hoc explanation\([Kikuta et al\., 2024](https://arxiv.org/html/2609.36063#bib.bib3)\), — and none establishes a causal link between a specific internal representation and a specific routing decision\. To the best of our knowledge, the activation\-patching experiments in Section 4\.4 are the first causal account of decision\-making mechanisms in NCO routing solvers specifically, extending the look\-ahead\-probing\-and\-patching paradigm established outside NLP \(above\) to this domain for the first time\.

## Appendix CExtended Behavioral Results

This section extends the behavioral evaluation presented in the main paper\. We report detailed results for TSP and CVRP across multiple problem sizes and seven distribution shifts, and subsequently examine the rotational sensitivity of the models\. These experiments provide a broader comparison of solution quality, geometric consistency, scale generalization, and robustness under changes in instance geometry\.

### C\.1TSP Behavioral Results

We compare L2C\-Insert, LEHD, POMO, AM, and cluster\-trained variants of POMO and LEHD across TSP sizes from 20 to 1000 and seven distribution shifts\. We report the optimality gap relative to Concorde, the average number of edge crossings, and the convex\-hull order violation rate\. An optimal Euclidean TSP tour contains no edge crossings and visits convex\-hull vertices in their cyclic order; therefore, these two metrics capture complementary forms of geometric inconsistency\. Lower values are better, and the best result for each metric, including ties, is shown in bold\.

Tables[3](https://arxiv.org/html/2609.36063#A3.T3)–[8](https://arxiv.org/html/2609.36063#A3.T8)reveal a clear separation between tour quality and geometric regularity\. At smaller problem sizes, L2C\-Insert and standard LEHD are generally competitive, whereas POMO and AM tend to exhibit larger optimality gaps\. As the problem size increases, however, LEHD\-Cluster becomes increasingly strong and consistently outperforms standard LEHD\. This effect is especially pronounced on TSP\-500 and TSP\-1000, where LEHD\-Cluster achieves the best or near\-best optimality gaps across the tested distributions\.

Interestingly, this improvement appears to stem from cluster\-based training rather than from the architecture alone\. LEHD\-Cluster generalizes substantially better than standard LEHD as the problem size grows, including on unseen distributions\. One plausible explanation is that cluster training encourages a more decompositional strategy, where the model solves spatially coherent regions as smaller subproblems before connecting them\. We view this as a behavioral hypothesis rather than direct mechanistic evidence\.

Cluster training also substantially improves the large\-scale generalization of POMO, reducing its optimality gap across all seven distributions on TSP\-500 and TSP\-1000\. Together with the LEHD\-Cluster results, this suggests that clustered training can benefit both architectures by improving how they exploit spatial structure\.

The geometric metrics provide a complementary view\. POMO and POMO\-Cluster can exhibit relatively low convex\-hull violation rates despite substantially larger optimality gaps, indicating that preserving coarse geometric structure is not sufficient for producing a low\-cost tour\. AM shows a similar separation in several settings, where geometrically regular behavior does not necessarily translate into strong tour quality\. Overall, the results suggest that large\-scale generalization depends not only on preserving geometric constraints, but also on learning a solution strategy that can organize and coordinate decisions across increasingly large instances\.

Table 3:Solution quality and geometric consistency under greedy decoding on TSP\-20 across seven test distributions\.Table 4:Solution quality and geometric consistency under greedy decoding on TSP\-50 across seven test distributions\.Table 5:Solution quality and geometric consistency under greedy decoding on TSP\-100 across seven test distributions\.Table 6:Solution quality and geometric consistency under greedy decoding on TSP\-200 across seven test distributions\.Table 7:Solution quality and geometric consistency under greedy decoding on TSP\-500 across seven test distributions\.Table 8:Solution quality and geometric consistency under greedy decoding on TSP\-1000 across seven test distributions\.
### C\.2CVRP Behavioral Results

We compare L2C\-Insert, LEHD, POMO and AM across CVRP sizes from 20 to 1000 and seven test distributions\. We report the optimality gap relative to high\-quality reference solutions obtained using a modern implementation of Hybrid Genetic Search \(HGS\), together with the geometric metrics defined in Section[C\.1](https://arxiv.org/html/2609.36063#A3.SS1)\. For CVRP, edge crossings and convex\-hull order violations are evaluated separately within each vehicle route and then averaged across the routes and problem instances\. Lower values are better for all metrics, and the best result in each setting, including ties, is shown in bold\.

Tables[9](https://arxiv.org/html/2609.36063#A3.T9)–[14](https://arxiv.org/html/2609.36063#A3.T14)show that LEHD exhibits stronger scale generalization on the larger instances\. It achieves the lowest optimality gap across all seven distributions on CVRP\-200 and in six of the seven distributions on both CVRP\-500 and CVRP\-1000\. In contrast, L2C\-Insert generally produces fewer within\-route crossings and lower convex\-hull violation rates, indicating greater local geometric consistency\.

Table 9:Solution quality and within\-route geometric consistency under greedy decoding on CVRP\-20 across seven test distributions\.Table 10:Solution quality and within\-route geometric consistency under greedy decoding on CVRP\-50 across seven test distributions\.Table 11:Solution quality and within\-route geometric consistency under greedy decoding on CVRP\-100 across seven test distributions\.Table 12:Solution quality and within\-route geometric consistency under greedy decoding on CVRP\-200 across seven test distributions\.Table 13:Solution quality and within\-route geometric consistency under greedy decoding on CVRP\-500 across seven test distributions\.Table 14:Solution quality and within\-route geometric consistency under greedy decoding on CVRP\-1000 across seven test distributions\.
### C\.3Sensitivity

We evaluate the rotational robustness of L2C\-Insert, LEHD, POMO, and AM\. For modelmmand instanceii, letgi,α\(m\)g\_\{i,\\alpha\}^\{\(m\)\}denote the optimality gap after rotating the instance by angleα\\alpha\. We consider𝒜=\{0∘,60∘,120∘,180∘,210∘,270∘\}\\mathcal\{A\}=\\\{0^\{\\circ\},60^\{\\circ\},120^\{\\circ\},180^\{\\circ\},210^\{\\circ\},270^\{\\circ\}\\\}\. The rotation sensitivity of each instance is defined assi\(m\)=maxα∈𝒜⁡gi,α\(m\)−minα∈𝒜⁡gi,α\(m\)s\_\{i\}^\{\(m\)\}=\\max\_\{\\alpha\\in\\mathcal\{A\}\}g\_\{i,\\alpha\}^\{\(m\)\}\-\\min\_\{\\alpha\\in\\mathcal\{A\}\}g\_\{i,\\alpha\}^\{\(m\)\}\. We report the mean sensitivity over theMMevaluated instances asS\(m\)=1M​∑i=1Msi\(m\)S^\{\(m\)\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}s\_\{i\}^\{\(m\)\}\.

Lower values indicate greater rotational robustness\. Tables[15](https://arxiv.org/html/2609.36063#A3.T15)and[16](https://arxiv.org/html/2609.36063#A3.T16)show that L2C\-Insert generally has the lowest sensitivity at smaller problem sizes, while LEHD remains robust at larger scales\. On TSP\-500 and TSP\-1000, LEHD achieves the lowest sensitivity in five and four of the seven distributions, respectively\. POMO is generally more sensitive than LEHD, while AM exhibits particularly high sensitivity in several settings\. The best result in each row, including ties, is shown in bold\.

Table 15:Rotation sensitivity across test distributions for TSP\-20, TSP\-50, and TSP\-100\.Table 16:Rotation sensitivity across test distributions for TSP\-200, TSP\-500, and TSP\-1000\.

## Appendix DExtended Mechanistic Interpretability Results

### D\.1Controlling for Nearest\-Neighbor Heuristics in LEHD

To examine whether LEHD’s multi\-step future\-action probe performance is primarily driven by a simple nearest\-neighbor heuristic, we restrict the evaluation to decoding states where LEHD’s own greedy next action is not the nearest currently unvisited node\. Results for the final two decoder layers, L5 and L6, are reported in Table[17](https://arxiv.org/html/2609.36063#A4.T17)\.

Probe accuracy decreases under this restriction, but remains substantial for multiple future steps\. On TSP\-500, for instance, L5 achieves 56\.2% Top\-1 accuracy ath=2h=2and 30\.3% ath=3h=3, even though the model’s immediate action is explicitly non\-nearest\-neighbor\. The same qualitative pattern holds for TSP\-50 and TSP\-100\. Thus, the future\-action signal accessible from LEHD’s decoder representations is not reducible to a simple nearest\-neighbor rule\. These results indicate that the future\-action information encoded in LEHD’s decoder representations cannot be explained solely by a simple nearest\-neighbor heuristic and instead reflects richer information about the model’s subsequent trajectory\.

Table 17:Top\-1 future\-node probe accuracy \(%\) for LEHD on TSP\-50, TSP\-100, and TSP\-500\.*Own\-Trajectory Future*evaluates prediction of LEHD’s own future nodes from its greedy decoding trajectory\.*Own\-Trajectory Future \(Non\-NN\)*restricts evaluation to states where LEHD’s greedy next action is not the nearest currently unvisited node\.
### D\.2Behavioral\-Causal Node Perturbation

In addition to the counterfactual steering experiment described in Section[3\.3](https://arxiv.org/html/2609.36063#S3.SS3), we conduct a second causal experiment that does not rely on probes or manual modifications to the model’s embeddings\. Our setup compares the model’s greedy rollout branch,a1,a2,…a\_\{1\},a\_\{2\},\\ldots, with the greedy rollout starting from the second\-highest\-probability node,a1′,a2′,…a^\{\\prime\}\_\{1\},a^\{\\prime\}\_\{2\},\\ldots\. We perturb the coordinates of nodesa2′,a3′,…,a5′a^\{\\prime\}\_\{2\},a^\{\\prime\}\_\{3\},\\ldots,a^\{\\prime\}\_\{5\}, and then observe how the logit gap betweena1a\_\{1\}anda1′a^\{\\prime\}\_\{1\}changes at each steptt:

Δ​Gt=\[zsteered​\(a1′\)−zsteered​\(a1\)\]−\[zclean​\(a1′\)−zclean​\(a1\)\],\\Delta G\_\{t\}=\\left\[z\_\{\\mathrm\{steered\}\}\(a^\{\\prime\}\_\{1\}\)\-z\_\{\\mathrm\{steered\}\}\(a\_\{1\}\)\\right\]\-\\left\[z\_\{\\mathrm\{clean\}\}\(a^\{\\prime\}\_\{1\}\)\-z\_\{\\mathrm\{clean\}\}\(a\_\{1\}\)\\right\],wherezsteered​\(⋅\)z\_\{\\mathrm\{steered\}\}\(\\cdot\)andzclean​\(⋅\)z\_\{\\mathrm\{clean\}\}\(\\cdot\)denote the logits under the perturbed and clean inputs, respectively\.

We consider four perturbation categories:

- •Thetowardcategory shortens the distance between consecutive nodesai′a^\{\\prime\}\_\{i\}andai−1′a^\{\\prime\}\_\{i\-1\}using an interpolation coefficientα\\alpha: rai′←\(1−α\)​rai′\+α​rai−1′,r\_\{a^\{\\prime\}\_\{i\}\}\\leftarrow\(1\-\\alpha\)r\_\{a^\{\\prime\}\_\{i\}\}\+\\alpha r\_\{a^\{\\prime\}\_\{i\-1\}\},whererar\_\{a\}denotes the location vector of nodeaa\.
- •Similarly, theawaycategory increases the distance between consecutive nodesai′a^\{\\prime\}\_\{i\}andai−1′a^\{\\prime\}\_\{i\-1\}using an extrapolation coefficientα\\alpha\.
- •Therandom directioncategory perturbs the node in a random direction\.
- •Therandom nodecategory perturbs a node in a far horizon, e\.g\.,a10′a^\{\\prime\}\_\{10\}\.

In most scenarios, there is only one plausible next node, with selection probability≈1\\approx 1, and the second\-best node is clearly poor\. We therefore restrict our analysis to scenarios in which the second\-most\-probable node has a reasonable chance, defined as selection probability\>0\.20\>0\.20\. This makes the perturbations more meaningful\. A sample visualization is presented in Figure[6](https://arxiv.org/html/2609.36063#A4.F6)

\(a\)Before perturbation\(b\)After perturbation
Figure 6:A sample visualization of node perturbation\.\(a\)Shows the original instance\.\(b\)Shows the perturbed instance\.In Figure[7](https://arxiv.org/html/2609.36063#A4.F7), we observe that the gap between the most probable node, i\.e\., the original next node, and the second\-highest\-probability node in LEHD decreases in thetowardcategory, where the distance between consecutive nodes along that horizon is shortened\. This indicates that LEHD’s preference shifts toward the second\-most\-probable branch\. In contrast, we observe no meaningful change for the other perturbation categories, signaling that LEHD’s decision remains relatively unchanged in those cases\. Interestingly, this perturbation becomes even more effective as the model progresses through decoding\.

Figure 7:LEHDΔ​G\\Delta Gwith probability threshold\.We also conducted the same experiment for POMO, with results shown in Figure[8](https://arxiv.org/html/2609.36063#A4.F8)\. Although thetowardcategory also produces a larger effect than the other categories, its value is much smaller than that of LEHD\. This shows that LEHD is much more sensitive to the locations of nodes beyond the immediate next node than POMO is, highlighting the difference in look\-ahead capabilities discussed in Section[5](https://arxiv.org/html/2609.36063#S5)\. Moreover, the effect of perturbation is relatively constant at every decoding step in POMO, unlike in LEHD, further highlighting POMO’s lack of decoding\-step sensitivity\.

Figure 8:POMOΔ​G\\Delta Gwith probability threshold\.These experiments, together with the counterfactual steering experiment presented in Section[3\.3](https://arxiv.org/html/2609.36063#S3.SS3), provide causal evidence for the look\-ahead mechanism in LEHD\.

### D\.3Assessing Off\-Manifold Effects in Probe\-Guided Steering

A potential concern with the probe\-guided counterfactual steering experiment in Section[3\.3](https://arxiv.org/html/2609.36063#S3.SS3)is that the observed behavioral changes may result from pushing internal representations away from the model’s naturally occurring representation manifold, rather than from selectively modifying the future\-action information under investigation\. To assess the extent of such off\-manifold perturbations, we measure three complementary representation\-space diagnostics: the ratio of steered to clean representation norms, the cosine similarity between steered and clean representations, and the relative norm of the perturbation\. These quantities are computed for the targeted future\-node representations across the intervened decoder layers and horizons\. Table[18](https://arxiv.org/html/2609.36063#A4.T18)reports these diagnostics across steering strengths to assess whether the interventions substantially alter the model’s natural representation geometry\.

Table 18:Representation\-space diagnostics for probe\-guided counterfactual steering on Uniform TSP\-100 across future horizons and decoder layers\.
### D\.4Start\-Node Navigation in AM

Although all models are evaluated in a single\-trajectory setting, AM differs from POMO and LEHD in how the starting node is selected\. While POMO and LEHD begin from an arbitrary starting node, AM selects its starting node through its learned policy\. At the initial decoding step, AM uses two learnable placeholder embeddings for the first and current nodes, enabling start selection to be learned jointly with subsequent route construction\. AM and POMO also differ in their decoder conditioning: AM concatenates the first\- and current\-node embeddings and incorporates a separate graph\-level representation, whereas POMO combines their projected embeddings through addition\. These differences in start selection and learned decoder conditioning may contribute to AM’s greater sensitivity to start\-node interventions\.

As shown in Figure[9](https://arxiv.org/html/2609.36063#A4.F9), start\-node patching causes substantially smaller probability drops when node 0 is prescribed as the starting node rather than selected by AM’s policy\. This suggests that learned start selection may make the start\-node representation more influential in subsequent routing decisions, although the two settings also produce different tour trajectories\. One possible explanation for the contrasting trends in AM and LEHD is that AM relies more on start\-node information during early and middle tour construction, whereas LEHD may use it more heavily as the tour approaches completion\.

Figure 9:Start\-node intervention sensitivity in AM on Uniform TSP\-100\.\(a\) Policy\-selected start; \(b\) forced start at node 0, followed by greedy decoding\. Previously visited donors are selected by maximum angular separation \(red\), maximum angle with matched distance \(blue\), or similar direction with different distance \(green\)\. The y\-axis shows the mean drop in clean next\-action probability after patching\.Undermaximum\-angle donor selection without distance constraints, policy\-selected starts yield lower LCS and higher Rev\. LCS than forced starts in Phase I \(Table[19](https://arxiv.org/html/2609.36063#A4.T19)\), but this pattern is less consistent in Phase II \(Table[20](https://arxiv.org/html/2609.36063#A4.T20)\)\. Unlike LEHD \(Table[1](https://arxiv.org/html/2609.36063#S5.T1)\), AM retains higher forward\- than reverse\-order similarity even at large angles\. This may suggest a policy\-dependent route\-anchoring role in AM, compared with a possible global\-navigation reference in LEHD\.

Table 19:AM rollout\-order similarity on Uniform TSP\-100 under maximum\-angle donor selection\. Phase I: steps 20–45\.

Table 20:AM rollout\-order similarity on Uniform TSP\-100 under maximum\-angle donor selection\. Phase II: steps 46–99\.

## Appendix EVisualization

We present additional visualizations of tour\-construction patterns, routing behaviors, and internal representations of neural solvers\.

### E\.1Latent\-Space Organization

Figure[10](https://arxiv.org/html/2609.36063#A5.F10)provides a qualitative view of how LEHD organizes node\-role representations across decoder layers\. A single PCA basis is jointly fitted across all decoder layers, allowing direct comparison of their representations in a shared latent space\. The reported explained\-variance ratios correspond to this joint PCA\. The faint markers show instance\-level representations, while the larger markers denote their corresponding means\. From Layer 2 onward, current\-node representations from different instances become concentrated in a compact shared region, consistent with the cross\-instance node alignment reported in the main text\. In contrast, the start\-node representations remain more dispersed in the earlier layers, but their distinct region becomes increasingly apparent from Layer 4 onward, suggesting that the decoder progressively separates the start node as a dedicated global reference\. The mean future\-node representations also exhibit an approximately ordered trajectory, particularly in the intermediate layers\.

Figure 10:Layer\-wise PCA of LEHD node\-role representations\.Faint points show instance\-level representations across 128 Uniform TSP\-50 instances, while larger markers indicate their means\. Numbered markers denote LEHD’s next six actions under greedy decoding\.
### E\.2Geometric Tour\-Construction Patterns

Figures[11](https://arxiv.org/html/2609.36063#A5.F11)and[12](https://arxiv.org/html/2609.36063#A5.F12)provide representative examples of POMO and AM rollouts across increasing problem sizes\. Node colors indicate normalized onion depth, with outer nodes assigned lower values and interior nodes higher values\. In both models, the highlighted segments reveal outer\-to\-deep\-to\-outer excursions during tour construction\. In POMO, this pattern appears clearly already on TSP\-50 and persists, becoming more pronounced as the problem size increases to TSP\-100 and TSP\-200, consistent with the oscillatory behavior quantified by the onion\-depth analysis\. The corresponding AM examples show that similar excursions can also arise under the same visualization, providing a qualitative comparison of geometric tour\-construction patterns across the two models, without implying that the frequency or strength of these excursions is the same in AM and POMO\.

Figure[13](https://arxiv.org/html/2609.36063#A5.F13)qualitatively compares the radial organization of POMO, LEHD, and AM routes on Uniform CVRP\-200 and CVRP\-500\. Among the selected high\-backtracking examples, POMO exhibits the most pronounced radial reversals, producing intertwined trajectories that repeatedly move toward and away from the depot\. AM shows an intermediate pattern, with several routes displaying radial reversals alongside more structured outward and return segments\. In contrast, LEHD exhibits the most regular radial organization in these examples, with routes generally extending toward their peak\-radius nodes before returning to the depot\. The differences become particularly visible on CVRP\-500, where the longer routes make the contrasting geometric structures easier to observe\. These visualizations illustrate the differences in radial backtracking examined in the quantitative analysis\.

![Refer to caption](https://arxiv.org/html/2609.36063v1/pomo_uniform_onion_depth_triptych_colored_excursions_stronger_N50_N100_N200_B5.png)Figure 11:Onion\-depth structure of POMO tours\.Representative Uniform TSP\-50, TSP\-100, and TSP\-200 rollouts illustrate POMO’s repeated oscillation between outer and deeper onion layers\. Colored segments highlight example outer\-to\-deep\-to\-outer excursions, while node colors indicate normalized onion depth\.![Refer to caption](https://arxiv.org/html/2609.36063v1/am_uniform_onion_depth_triptych_colored_excursions_stronger_N50_N100_N200_B5.png)Figure 12:Onion\-depth structure of AM tours\.Selected Uniform TSP\-50, TSP\-100, and TSP\-200 rollouts illustrate outer\-to\-deep\-to\-outer excursions in AM\. Colored segments highlight example excursions, while node colors indicate normalized onion depth\.\(a\)Uniform CVRP\-200\.\(b\)Uniform CVRP\-500\.
Figure 13:Radial route structure of POMO, LEHD, and AM on CVRP\.Selected Uniform CVRP\-200 and CVRP\-500 solutions are shown, with POMO on the left, LEHD in the center, and AM on the right\. Each vehicle route is displayed in a distinct color; black stars indicate depot locations, and enlarged circular markers identify the farthest customer from the depot along each route\. In these selected examples, radial backtracking is most visually pronounced in POMO, followed by AM, while LEHD exhibits more regular outward\-then\-inward route structures\.
### E\.3Attention Patterns During Decoding

In Figure[4\(a\)](https://arxiv.org/html/2609.36063#S3.F4.sf1)of the main text, we visualized the attention heatmap for a single layer of LEHD to demonstrate that the model attends more strongly to nodes scheduled for future visits than to other unvisited nodes\. In Figure[14](https://arxiv.org/html/2609.36063#A5.F14), we provide a detailed layer\-wise breakdown of this behavior across all decoder layers\. The upcoming nodes in the ground\-truth sequence are highlighted with red dashed lines\. For visual clarity, self\-attention from the current node to itself has been omitted\. Notably, the attention distribution is more diffuse in the initial layer and becomes increasingly focused on upcoming nodes in subsequent layers, suggesting that future\-action planning primarily occurs in the deeper decoding layers\. Additionally, an animated visualization of the step\-wise attention dynamics across layers is provided in[https://github\.com/NCO\-Interpretability/NCO\-Interpretability](https://github.com/NCO-Interpretability/NCO-Interpretability)\.

![Refer to caption](https://arxiv.org/html/2609.36063v1/LEHD_50_uniform_attention_steps.png)Figure 14:Layer\-wise attention heatmaps of the current node during decoding\.Each panel illustrates the attention weight distribution assigned by the current node \(query\) to remaining unvisited nodes \(keys\) for each decoder layer at a representative decoding step\. Red dashed lines denote the nodes actually visited in subsequent steps\.

## Appendix FCVRP Results

In this section, we extend the interpretability experiments presented in Section[3](https://arxiv.org/html/2609.36063#S3)to the Capacitated Vehicle Routing Problem \(CVRP\)\. Unlike the TSP, where a solution consists of a single continuous tour, CVRP solutions comprise multiple distinct routes that each originate and terminate at a central depot\. Here, we evaluate whether the mechanistic and geometric properties observed in TSP generalize to individual routes within CVRP solutions\.

### F\.1Geometric Trajectory Analysis

For the TSP, we established that POMO constructs tours via an oscillating clockwise progression\. However, because individual routes in CVRP are substantially shorter than full TSP tours, even on large problem sizes, the average number of deep onion\-layer oscillations per route is naturally low across all methods, rendering this metric less effective for distinguishing geometric biases \(Figure[15\(a\)](https://arxiv.org/html/2609.36063#A6.F15.sf1)\)\.

To capture radial geometric distortion within shorter individual routes, we first identify the node furthest from the depot within a given route and use it as an anchor to bisect the route into two segments\. For the first segment, we compute the depot distance of each node and count how often the distancedecreasesbetween consecutive steps\. For the second segment, we count how often the distanceincreases\. Intuitively, this measures the degree of radial back\-and\-forth distortion along the route\. As shown in Figure[15\(b\)](https://arxiv.org/html/2609.36063#A6.F15.sf2)and visually illustrated in Figure[13](https://arxiv.org/html/2609.36063#A5.F13), POMO exhibits higher radial distortion than LEHD, L2C\-Insert, and HGS, while AM shows comparable distortion at smaller scales but substantially less at N=1000\. Furthermore, POMO maintains a consistent clockwise progression during route construction, mirroring its TSP behavior \(Figure[15\(c\)](https://arxiv.org/html/2609.36063#A6.F15.sf3)\)\. AM also exhibits a milder clockwise tendency on CVRP, in contrast to its counterclockwise progression on TSP\.

\(a\)Mean onion\-layer oscillations\(b\)Radial route distortion\(c\)Angular orientation progression
Figure 15:Geometric trajectory properties on CVRP routes\.\(a\)Average onion\-layer oscillations across problem scales; values remain low due to short individual route lengths\.\(b\)Step\-wise radial distortion along individual routes, showing heightened back\-and\-forth movement in POMO compared to baselines, with AM exhibiting intermediate distortion at larger scales\.\(c\)Mean current\-to\-depot angle relative to each route’s first edge, averaged within instances and then across instances, showing POMO’s persistent clockwise progression and a weaker clockwise tendency in AM\.
### F\.2Future\-Action Planning Probes

In Section[5](https://arxiv.org/html/2609.36063#S5), we showed that LEHD’s internal representations encode multi\-step future node sequences for the TSP\. To determine whether this look\-ahead capability extends to CVRP, we trained linear probes to predict upcoming nodes within the active route\. For CVRP, the probe ranks all unvisited customers, regardless of their immediate feasibility under the remaining vehicle capacity\. We exclude horizons that cross a depot return, ensuring that each evaluated target belongs to the same route as the current customer\. As shown in Figure[16\(a\)](https://arxiv.org/html/2609.36063#A6.F16.sf1), LEHD demonstrates strong predictive accuracy over short future horizons, closely replicating the planning characteristics observed on TSP\. In contrast, both POMO and AM retain only limited horizon\-predictive information, with probe accuracy dropping sharply beyond one\-step prediction\.

To investigate whether these future\-action representations also influence the current decision, we extend our probe\-guided steering experiment to CVRP\. At each selected state, we identify the model’s preferred next customerH1H\_\{1\}and its second\-best alternativeH1′H\_\{1\}^\{\\prime\}, then counterfactually forceH1′H\_\{1\}^\{\\prime\}and collect subsequent customersH2′,…,H5′H\_\{2\}^\{\\prime\},\\ldots,H\_\{5\}^\{\\prime\}within the same vehicle route\. We intervene on the representations of these future customers without directly modifyingH1H\_\{1\}orH1′H\_\{1\}^\{\\prime\}\. As shown in Figure[16\(b\)](https://arxiv.org/html/2609.36063#A6.F16.sf2), steering LEHD’s future\-node representations increases the logit gap in favor ofH1′H\_\{1\}^\{\\prime\}, with a larger mean effect than either the random\-direction or disjoint\-horizon control\. These findings provide evidence that LEHD’s accessible future\-action information is causally connected to its current routing decisions in CVRP\.

\(a\)Future\-action linear probing accuracy\(b\)Probe\-guided steering
Figure 16:Future\-action representations and causal steering on CVRP\.\(a\)Linear probing accuracy across planning horizons for LEHD, POMO, and AM, revealing differences in the accessibility of future\-action information from decoder representations\.\(b\)Change in the logit gap under probe\-guided steering and control interventions on CVRP\-100, measuring the effect of modifying future\-node representations on the current decision\.
### F\.3Current and Start Node Contribution

We next extend our causal intervention studies to CVRP routes\. As in the TSP setting, similar causal patterns emerge: Figure[17\(a\)](https://arxiv.org/html/2609.36063#A6.F17.sf1)reveals that LEHD’s immediate next\-node predictions are far more sensitive to mean\-ablating the current node representation than the depot node, while Figure[17\(b\)](https://arxiv.org/html/2609.36063#A6.F17.sf2)confirms that the angular orientation of the depot node guides global route navigation\.

\(a\)LEHD mean ablation impact\(b\)POMO vs\. LEHD start\-node patching
Figure 17:Causal interventions on node representations in CVRP\.\(a\)Probability drop in clean next\-node predictions following mean ablation of current versus depot node representations in LEHD\.\(b\)Effect of start/depot\-node activation patching under three geometric donor conditions, comparing directional sensitivity in LEHD and POMO\.Interestingly, while the Longest Common Subsequence \(LCS\) trend under depot\-node interventions decreases as the donor angle range widens \(matching the TSP pattern in Table 1\), the absolute LCS values are substantially lower for CVRP \(Table[21](https://arxiv.org/html/2609.36063#A6.T21)\)\. Intervening on the depot node often forces models to switch to entirely different customer clusters, resulting in non\-overlapping route memberships and low sequence alignment\. Furthermore, because CVRP routes are strictly constrained by vehicle capacity, LEHD cannot rely solely on start\-node angular orientation to minimize return costs; it must simultaneously optimize capacity constraints, keeping Reverse LCS values consistently low across all donor angle ranges\.

Table 21:CVRP\-500 route rollout\-order similarity under depot\-node activation patching\.Results evaluate maximum\-angle donor selection without distance constraints\. LCS measures forward\-order sequence agreement with clean rollouts, while Rev\. LCS measures agreement with reversed clean rollouts\.
### F\.4Node\-Role Representation Alignment

A central finding for the TSP was that LEHD projects current and start node representations into canonical, shared regions of the embedding space across decoding steps and instances \(Table 2\)\. We evaluate this cross\-instance representation alignment for CVRP in Table[22](https://arxiv.org/html/2609.36063#A6.T22)\. The results confirm that the same canonical representation mechanism operates in CVRP: intermediate and late decoder layers maintain high cosine similarities for the current and depot nodes compared to randomly selected customer nodes\.

Table 22:Layer\-wise cross\-instance alignment of node representations in LEHD on Uniform CVRP\-100\.Values denote mean cosine similarity \(±\\pmstd\) across different problem instances and decoding steps for current, depot, and random customer node representations\. Enc\. and Dec\. denote encoder and decoder layers\.

## Appendix GExperimental Settings

This section summarizes the implementation details and hyperparameter settings of the experiments\. All models were evaluated using greedy decoding without model\-specific test\-time augmentation\. All experiments were conducted on a single NVIDIA GeForce RTX 3090 GPU\.

### G\.1Onion\-Depth Analysis

For the onion\-depth analysis, we set the normalized depth threshold toτ=0\.5\\tau=0\.5\. For an onion decomposition discretized intoBBdepth bins, the minimum depth required for an oscillation is

bmin=min⁡\(B,⌊0\.5​B⌋\+1\)\.b\_\{\\min\}=\\min\\left\(B,\\left\\lfloor 0\.5B\\right\\rfloor\+1\\right\)\.\(4\)
An oscillation is counted when the compressed bin sequence leaves the outermost bin, reaches at leastbminb\_\{\\min\}, and subsequently returns to the outermost bin\. Consecutive repetitions of the same bin are removed before counting\.

### G\.2Targeted Start\-Node Intervention

For the targeted start\-node intervention, donor nodes were selected from the already visited portion of the partial tour\. LetSSdenote the original start node,CCthe current node, anddda candidate donor node\. We define

θd=∠⁡\(xS−xC,xd−xC\),rd=∥xd−xC∥2∥xS−xC∥2\.\\theta\_\{d\}=\\angle\\left\(x\_\{S\}\-x\_\{C\},\\,x\_\{d\}\-x\_\{C\}\\right\),\\qquad r\_\{d\}=\\frac\{\\lVert x\_\{d\}\-x\_\{C\}\\rVert\_\{2\}\}\{\\lVert x\_\{S\}\-x\_\{C\}\\rVert\_\{2\}\}\.\(5\)
The geometric constraints used for the three donor\-selection conditions are reported in Table[23](https://arxiv.org/html/2609.36063#A7.T23)\.

Table 23:Geometric constraints used for donor selection in the targeted start\-node intervention\.Thus, similar distance permits a deviation of at most20%20\\%from the original start\-to\-current distance\. The similar\-angle condition restricts the angular deviation to15∘15^\{\\circ\}, while requiring the donor distance to be at least30%30\\%smaller or larger\.

### G\.3Future\-Action Planning Probes

For each model layer and prediction horizon, we trained a separate linear candidate\-ranking probe while keeping the underlying routing model frozen\. The data\-split and optimization settings are summarized in Table[24](https://arxiv.org/html/2609.36063#A7.T24)\.

Table 24:Training settings for the future\-action planning probes\.
### G\.4Training Settings

For the standard benchmark experiments, we used the official checkpoints released by the authors to ensure that our baseline results faithfully match the original implementations\. To analyze the effect of the training data distribution, we retrained each model on clustered instances while keeping its architecture and original training paradigm unchanged\.

We did not switch any model to a different training paradigm \(e\.g\., RL to SL or SL to RL\)\. Such a switch is not straightforward and would change the method under comparison\. AM and POMO are originally RL methods: AM is trained with REINFORCE using a rollout baseline, while POMO uses REINFORCE with multiple starting nodes and a shared baseline\. These mechanisms are specific to the RL formulation, and adapting them to SL would require a non\-trivial redesign of the training objective and targets\. For LEHD, the original method is SL; the authors explicitly note that RL training is impractical because the heavy decoder structure incurs substantial memory and computational costs\. We therefore restrict our experiments to each model’s original training setting, which allows us to isolate the effect of the training distribution rather than confounding it with a change in the learning algorithm\.