SPARK: Susceptibility-Guided Profiling and Steering of Latent Reasoning States in Large Language Models

# SPARK: Susceptibility-Guided Profiling and Steering of Latent Reasoning States in Large Language Models
Source: [https://arxiv.org/html/2607.10296](https://arxiv.org/html/2607.10296)
Dongxu Zhang1Yiding Sun1Zihao Guo1Xiangyang Yang1Kai Tang2Lin Chen1Cheng Tan3Jihua Zhu1

###### Abstract

Reasoning failures in large language models \(LLMs\) are usually evaluated from final answers, but a wrong answer does not reveal why the model failed\. The same incorrect output may reflect missing capability, an unstable reasoning trajectory, or a failure to activate a reasoning state that is already available in the frozen model\. Existing prompting and benchmark\-based evaluation methods mostly operate at the output level, while generic activation\-steering methods typically apply global directions without diagnosing which examples require intervention\. In this paper, we introduceSPARK, which uses hidden\-state response to diagnose whether a model internally enters an effective reasoning state and to guide lightweight test\-time steering\. The key observation is that raw hidden\-state susceptibility is strongly confounded by prompt length, especially in programmatic and algorithmic reasoning where harder serialized instances naturally become longer\.SPARKtherefore uses length\-controlled susceptibility to separate input\-scale effects from residual reasoning activation, and combines this signal with cross\-layer coordination to select reasoning\-active anchors and under\-activated hard examples\. We useFRONTIER\-4\.5K as a controlled programmatic reasoning suite for latent profiling and difficulty\-aware analysis, and evaluateSPARK\-Steering on GSM8K and MATH\-500 with forward\-only benchmark profiling\. Our method improves Qwen3 series models consistently; on MATH\-500, accuracy rises from 82\.0% to 84\.6% for Qwen3\-4B and from 82\.4% to 85\.6% for Qwen3\-8B\. These results suggest that susceptibility can serve not only as a diagnostic signal for reasoning failures, but also as a practical guide for targeted test\-time intervention\.

Keywords:Large Language Models, Chain\-of\-Thought, Latent Reasoning States

![Refer to caption](https://arxiv.org/html/2607.10296v1/x1.png)Figure 1:Motivation and overview ofSPARK\.Benchmark evaluation observes correct and wrong answers and summarizes performance with accuracy curves, but leaves the latent reasoning state hidden\.SPARKcomplements this output\-level view with latent profiling and test\-time steering\. It diagnoses hidden\-state response, selects reasoning\-active anchors and under\-activated hard targets, and evaluates whether steering improves harder examples while preserving easier cases\.## 1Introduction

Large language models \(LLMs\) have made rapid progress on mathematical, symbolic, and algorithmic reasoning tasks\[wu2026intern,xu2026nanoresearch,huang2026cfms\], yet their reasoning failures remain difficult to interpret from final answers alone\[zhang2025ascot\]\. A model may solve easy instances reliably, become unstable on moderately difficult inputs, and fail sharply as problem structure grows\[tang2026mitigatinghallucinationsinterlayerconsistency,tang2026seememitigatinghallucinationslarge,guo2026fademitigatinghallucinationsreducing\]\. Benchmark accuracy captures this behavioral pattern only at the output level\[srivastava2023beyond,liang2022holistic\]\. It tells us when performance drops, but not what internal condition has changed\. This matters because two incorrect answers may correspond to different internal states\. In one case, the model may not possess the required capability\[hendrycks2021measuring\]\. In another, it may have partial capability but fail to activate the latent computation needed for the current input\[wei2022emergent\]\. Moreover, hard reasoning problems often become longer and more structurally dense\[zhang2026chain\], so weak hidden\-state response may reflect input scale rather than a true collapse of reasoning\.

Current evaluation practice creates a gap between observing failure and deciding how to intervene\. Accuracy curves can locate a behavioral capability boundary, but they do not specify what should be changed when a hard example fails\. Prompting treats the model largely as a black box\[wei2022chain,kojima2022large\], while fine\-tuning changes parameters and obscures whether the frozen model already contains a usable reasoning state\[hu2022lora,wang2026pointrft,sun2026tri\]\. A useful reasoning diagnostic should therefore be measurable during a forward pass, require no parameter updates, and provide an actionable signal for targeted test\-time intervention\.

We address this problem through hidden\-state susceptibility\. Inspired by susceptibility as response to a small perturbation\[goodfellow2014explaining,novak2018sensitivity\], we measure how strongly transformer hidden states react when small noise is applied to the input embeddings\. This yields a lightweight response signal for each problem, with the initial intuition that examples near a behavioral boundary may show amplified or unstable internal responses\. Our experiments show that this intuition must be refined\. In programmatic reasoning, raw susceptibility is strongly affected by prompt length\[press2021train,liu2024lost\], especially in algorithmic tasks where harder graph\-structured instances naturally become longer after serialization\. As difficulty increases, raw hidden\-state response can decrease largely because the input is longer; without length control, a metric that appears to describe reasoning may instead describe input scale\.

This observation motivates a length\-controlled view of susceptibility\. Instead of asking whether a problem has high or low raw response, we compare its response with what is expected among inputs of similar length\. The resulting residual signal better reflects reasoning activation after removing a major input scale confound\. A positive residual suggests stronger internal activation than length alone would predict, while a negative residual suggests under activation\. This distinction is especially important for hard failures, where weak raw response could otherwise be mistaken for a trivial consequence of long prompts\. We presentSPARKfor susceptibility guided profiling and steering of latent reasoning states\. Following the pipeline in Figure[1](https://arxiv.org/html/2607.10296#S0.F1), the framework first estimates the behavioral boundary from the accuracy curve over difficulty\. It then profiles latent states\[geva2021transformer,meng2022locating\]using length controlled susceptibility and cross\-layer coordination, separating reasoning\-active examples from hard examples that remain under activated after length correction\. This partition is based on corrected internal evidence rather than prompt length or raw response, and turns latent profiling into a selection mechanism for intervention\[belrose2023eliciting,ghandeharioun2024patchscopes\]\.

This diagnosis further enablesSPARK\-Steering, a targeted test\-time intervention for frozen models\[rimsky2024steering\]\. Instead of applying a generic activation direction,SPARKextracts a reasoning direction from active anchors, representing the latent shift from a baseline state toward an active reasoning state\. During inference, the direction is injected into selected layers\[turner2023steering,zou2023representation\]for under\-activated hard targets\. This makes steering diagnosis\-guided and selective, it aims to repair hard failures while preserving easier cases, without updating model parameters\.

We evaluateSPARKonFRONTIER\-4\.5K, a 4,500\-instance programmatic reasoning suite covering symbolic composition, logical inference, and algorithmic reasoning\. These tasks allow us to compare behavioral accuracy with internal response as difficulty changes\. Algorithmic reasoning is the primary diagnostic setting because it exhibits both a clear capability boundary and a strong length confound\. In this paper, our contributions are as follows:

- •We constructFRONTIER\-4\.5K, a reasoning benchmark with executable supervision and continuous difficulty annotations for joint behavioral and latent\-state analysis\.
- •We identify length as a critical confound in hidden\-state susceptibility analysis, showing that raw response signals can reflect input\-scale attenuation rather than reasoning activation, especially in algorithmic reasoning\.
- •To the best of our knowledge, we present the first susceptibility\-guided steering framework for implicit reasoning states\.SPARKextracts reasoning directions from active anchors and selectively injects them into under\-activated hard examples at test time\.
- •Extensive experiments show thatSPARKimproves GSM8K, MATH\-500, andFRONTIER, with stronger gains on harder reasoning cases and the importance of susceptibility\-guided direction and layer selection\.

## 2Related Work

### 2\.1Reasoning in Large Language Models\.

LLMs have shown strong performance on mathematical, symbolic, and commonsense reasoning benchmarks, while their success often changes sharply with model scale, prompt design, and task complexity\[zhang2026pointcot,hendrycks2021measuring,cobbe2021training\]\. A large body of work improves reasoning by eliciting intermediate computations in the output space\. Chain\-of\-thought \(CoT\) prompting encourages models to generate step\-by\-step rationales\[wei2022chain\], zero\-shot variants show that such behavior can be triggered by simple natural\-language instructions\[kojima2022large\], and self\-consistency improves robustness by marginalizing over multiple sampled reasoning paths\[zhang2026not\]\. Subsequent prompting methods further decompose hard problems into simpler subproblems or search over intermediate reasoning states, such as least\-to\-most prompting, Tree of Thoughts, and ReAct\-style reasoning with actions\[zhou2022least,yao2023tree,yao2022react\]\. Other work studies verifier\-guided reasoning and process supervision, emphasizing that correct final answers may depend on the stability of intermediate reasoning trajectories\[zelikman2022star,lightman2024let\]\. These methods primarily operate through prompts, demonstrations, decoding, or supervision over generated rationales\. In contrast, SPARK does not primarily aim to elicit longer or more explicit reasoning traces; it diagnoses whether the model internally enters a reasoning\-active state\.

### 2\.2Activation Steering and Test\-Time Control\.

Another line of work controls language models at inference time without updating their parameters\. Early approaches guide generation through external discriminators, gradients, or future\-token scoring\[yang2021fudge,krause2021gedi,dathathri2019plug\]\. More recent activation\-level methods directly modify internal states using directions derived from contrastive prompts or representation analyses\[li2023inference,rimsky2024steering\]\. Activation addition, representation engineering, inference\-time intervention, and contrastive activation addition show that adding a vector to selected hidden layers can steer attributes such as sentiment, honesty, toxicity, or refusal behavior without full fine\-tuning\[turner2023steering,zou2023representation\]\. These results suggest that some model behaviors are organized along manipulable directions in activation space\. SPARK\-Steering applies this idea to reasoning, but differs in two ways\. First, the steering direction is not chosen from a generic semantic contrast; it is extracted from reasoning\-active anchors selected by length\-controlled susceptibility and cross\-layer coordinationΦ\\Phi\. Second, the target of intervention is not a global style or safety attribute, but a specific failure mode: high\-difficulty examples whose hidden\-states are under\-activated relative to same\-length inputs\. This connects latent diagnosis with a causal test\-time intervention for reasoning\.

## 3Reasoning Benchmark

Our analysis requires a benchmark that exposes more than final answers\. To study behavioral boundaries together with latent activation, each instance should provide verified supervision, a controlled difficulty coordinate, and metadata about its generated structure\. We constructFRONTIER\-4\.5K, a programmatic reasoning benchmark with 4,500 generated instances for evaluating both output accuracy and hidden\-state response\.FRONTIER\-4\.5K contains three domains with 1,500 instances each\. Symbolic Composition requires multi\-step transformations over symbols and variable bindings; Logical Inference requires multi\-hop rule chaining with distractor facts; and Algorithmic Reasoning requires graph\- and procedure\-based queries such as connectivity, path existence, and shortest\-path problems\[velivckovic2022clrs,markeeva2024clrs\]\. All answers are produced by deterministic solvers, avoiding model\-based judging\.

Each instance includes a problem textxx, verified answeryy, domain label, generation parameters, token lengthTT, and normalized difficultyd∈\[0,1\]d\\in\[0,1\]\. Difficulty is derived from domain\-specific generation factors and normalized within each domain to ensure comparable ordering: composition depth and variable interactions for symbolic tasks, rule\-chain length and distractors for logical tasks, and graph size, edge density, and query complexity for algorithmic tasks\.

## 4Method

### 4\.1Problem Setup

Let𝒟=\{\(xi,yi,di\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\},d\_\{i\}\)\\\}\_\{i=1\}^\{N\}be a domain\-specific split from one downstream benchmark\. Each inputxix\_\{i\}has a verified answeryiy\_\{i\}and a normalized difficulty scoredi∈\[0,1\]d\_\{i\}\\in\[0,1\], where larger values indicate harder instances\. We use this set to estimate behavioral boundaries, fit length\-control models, and select reasoning\-active anchors\. A frozen language modelfθf\_\{\\theta\}produces an answery^i\\hat\{y\}\_\{i\}, and correctness is denoted by:

ci=𝟏​\[y^i=yi\]\.c\_\{i\}=\\mathbf\{1\}\[\\hat\{y\}\_\{i\}=y\_\{i\}\]\.\(1\)The token length ofxix\_\{i\}is denoted byTiT\_\{i\}\. Throughout this section, model parametersθ\\thetaare fixed\. All profiling signals are computed from forward passes only\.

The method has two stages\. First,SPARK Profilingestimates the behavioral boundary and diagnoses latent reasoning states using hidden\-state response\. Second,SPARK\-Steeringuses the diagnosed active states to construct a test\-time steering direction for hard under activated examples\.

### 4\.2Behavioral Capability Boundary

The behavioral boundary is estimated from the relation between difficulty and correctness\. We fit an accuracy curve:

p^​\(d\)≈P​\(c=1∣d\),\\hat\{p\}\(d\)\\approx P\(c=1\\mid d\),\(2\)using held out calibration examples\. The capability boundaryd⋆d^\{\\star\}is defined as the difficulty level where the fitted pass rate drops to 50 percent,d⋆=inf\{d∈\[0,1\]∣p^​\(d\)≤0\.5\}\.d^\{\\star\}=\\inf\\\{d\\in\[0,1\]\\mid\\hat\{p\}\(d\)\\leq 0\.5\\\}\.This boundary is a behavioral quantity\. It is estimated from correctness, not from hidden\-state metrics\.

![Refer to caption](https://arxiv.org/html/2607.10296v1/x2.png)Figure 2:Overview ofSPARK\.Given tasks organized by domain, difficulty, and token length, SPARK first profiles a frozen LLM to identify the behavioral boundary and latent activation patterns associated with successful reasoning\. It then selects active anchors and under\-activated hard examples to estimate critical reasoning directions, which are injected at test\-time through activation steering\. The resulting intervention is applied only at inference time\.
### 4\.3Latent Susceptibility

For an inputxx, letE​\(x\)E\(x\)be its input embedding sequence\. For layerll, letrl​\(x\)r\_\{l\}\(x\)be the representation used for latent analysis,rl​\(x\)r\_\{l\}\(x\)is a pooled hidden\-state from layerll, such as the final prompt token representation or a mean pooled representation over selected prompt positions\. The same pooling rule is used for all methods and controls\. To measure hidden\-state response, we add small Gaussian perturbations\[goodfellow2014explaining\]to the input embeddings\. For perturbation indexkk, define:

E~k​\(x\)=E​\(x\)\+ϵ​ηk,ηk∼𝒩​\(0,I\),\\widetilde\{E\}\_\{k\}\(x\)=E\(x\)\+\\epsilon\\eta\_\{k\},\\quad\\eta\_\{k\}\\sim\\mathcal\{N\}\(0,I\),\(3\)whereϵ\\epsiloncontrols perturbation scale\. Letrl​\(x;E~k\)r\_\{l\}\(x;\\widetilde\{E\}\_\{k\}\)denote the same pooled layer representation when the model is run with the perturbed embedding sequenceE~k​\(x\)\\widetilde\{E\}\_\{k\}\(x\)\. The layerwise susceptibility is defined as:

χl​\(x\)=1K​∑k=1K\(‖rl​\(x;E~k\)−rl​\(x\)‖2ϵ​\(‖rl​\(x\)‖2\+τ\)\)2,\\chi\_\{l\}\(x\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\frac\{\\\|r\_\{l\}\(x;\\widetilde\{E\}\_\{k\}\)\-r\_\{l\}\(x\)\\\|\_\{2\}\}\{\\epsilon\(\\\|r\_\{l\}\(x\)\\\|\_\{2\}\+\\tau\)\}\\right\)^\{2\},\(4\)whereKKis the number of perturbations andτ\\tauis a small numerical constant\. We aggregate the layerwise responses over a selected layer setℒχ\\mathcal\{L\}\_\{\\chi\}to obtain a sample\-level susceptibility:

χ​\(x\)=1\|ℒχ\|​∑l∈ℒχχl​\(x\)\.\\chi\(x\)=\\frac\{1\}\{\|\\mathcal\{L\}\_\{\\chi\}\|\}\\sum\_\{l\\in\\mathcal\{L\}\_\{\\chi\}\}\\chi\_\{l\}\(x\)\.\(5\)This quantity measures response intensity\. It should be interpreted as a response\-intensity measure rather than a difficulty score or a direct predictor of correctness\.

### 4\.4Length Controlled Susceptibility

Raw susceptibility can be affected by prompt length\. To separate length effects from reasoning state effects, we fit a length response model on the calibration split within each domain:

log⁡\(χ​\(x\)\+τ\)=a\+b​log⁡T​\(x\)\+ξx,\\log\(\\chi\(x\)\+\\tau\)=a\+b\\log T\(x\)\+\\xi\_\{x\},\(6\)whereT​\(x\)T\(x\)is token length,aaandbbare fitted coefficients, andξx\\xi\_\{x\}is the residual term\. The length controlled susceptibility for each example is defined as:

χLC​\(x\)=log⁡\(χ​\(x\)\+τ\)−a−b​log⁡T​\(x\)\.\\chi\_\{\\mathrm\{LC\}\}\(x\)=\\log\(\\chi\(x\)\+\\tau\)\-a\-b\\log T\(x\)\.\(7\)A positiveχLC\\chi\_\{\\mathrm\{LC\}\}means thatxxelicits stronger response than expected for inputs of similar length\. A negative value indicates residual under activation after accounting for length\.

### 4\.5Cross\-Layer Coordination

Susceptibility measures response strength, but strong response alone does not guarantee coherent reasoning\. We therefore measure whether hidden\-state changes are directionally coordinated across layers\. For a coordination layer setℒΦ=\{l1,…,lm\},\\mathcal\{L\}\_\{\\Phi\}=\\\{l\_\{1\},\\ldots,l\_\{m\}\\\},define the normalized layer update:

ul​\(x\)=rl​\(x\)−rl−1​\(x\)‖rl​\(x\)−rl−1​\(x\)‖2\+τ\.u\_\{l\}\(x\)=\\frac\{r\_\{l\}\(x\)\-r\_\{l\-1\}\(x\)\}\{\\\|r\_\{l\}\(x\)\-r\_\{l\-1\}\(x\)\\\|\_\{2\}\+\\tau\}\.\(8\)Cross\-layer coordination\[geva2021transformer\]is the average pairwise cosine similarity among these updates:

Φ​\(x\)=2m​\(m−1\)​∑1≤i<j≤muli​\(x\)⊤​ulj​\(x\)\.\\Phi\(x\)=\\frac\{2\}\{m\(m\-1\)\}\\sum\_\{1\\leq i<j\\leq m\}u\_\{l\_\{i\}\}\(x\)^\{\\top\}u\_\{l\_\{j\}\}\(x\)\.\(9\)HighΦ\\Phiindicates that multiple layers move in consistent directions, which is expected when the model enters a more stable and internally coherent reasoning state\.

### 4\.6SPK Index and State Profiling

The final profiling score combines length controlled response with coordination\. Letz​\(⋅\)z\(\\cdot\)denote standardization within the calibration split of the same domain\. The length controlled SPK score is defined as:

SPKLC​\(x\)=z​\(χLC​\(x\)\)\+λ​z​\(Φ​\(x\)\),\\mathrm\{SPK\}\_\{\\mathrm\{LC\}\}\(x\)=z\(\\chi\_\{\\mathrm\{LC\}\}\(x\)\)\+\\lambda z\(\\Phi\(x\)\),\(10\)whereλ\\lambdacontrols the contribution of cross\-layer coordination\. This score is used to construct two sets\. Reasoning\-active anchors𝒜\\mathcal\{A\}are examples that are solved correctly, lie near or below the behavioral boundary, and have highχLC\\chi\_\{\\mathrm\{LC\}\}and highSPKLC\\mathrm\{SPK\}\_\{\\mathrm\{LC\}\}\. Under\-activated target candidates𝒰\\mathcal\{U\}are hard examples near or above the behavioral boundary with lowχLC\\chi\_\{\\mathrm\{LC\}\}\. Baseline correctness is used to construct reasoning\-active anchors and diagnostic plots, but not to select test\-time target examples\. Quantile thresholds are chosen on the calibration split and kept fixed for evaluation\.

### 4\.7Critical Reasoning Direction

For each anchora∈𝒜a\\in\\mathcal\{A\}, we construct two inputs\. The base inputxa0x\_\{a\}^\{0\}contains only the target problem\. The active inputxa\+x\_\{a\}^\{\+\}contains SPK selected demonstrations followed by the same target problem\. Letrl​\(xa0\)r\_\{l\}\(x\_\{a\}^\{0\}\)andrl​\(xa\+\)r\_\{l\}\(x\_\{a\}^\{\+\}\)be their layer representations\. The unnormalized direction at layerllis:

v¯l=1\|𝒜\|​∑a∈𝒜\(rl​\(xa\+\)−rl​\(xa0\)\)\.\\bar\{v\}\_\{l\}=\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{a\\in\\mathcal\{A\}\}\\left\(r\_\{l\}\(x\_\{a\}^\{\+\}\)\-r\_\{l\}\(x\_\{a\}^\{0\}\)\\right\)\.\(11\)The injected direction is normalized as:

vl=v¯l‖v¯l‖2\+τ\.v\_\{l\}=\\frac\{\\bar\{v\}\_\{l\}\}\{\\\|\\bar\{v\}\_\{l\}\\\|\_\{2\}\+\\tau\}\.\(12\)This direction represents the average latent shift from a base state toward a reasoning\-active state\. A stricter control replacesxa0x\_\{a\}^\{0\}with a length matched random demonstration input, which tests whether the direction is caused by reasoning\-active content rather than added context length\.

ModelMethodGSM8KMATH\-500FRONTIER\-4\.5KAlphaAccuracyTokensAlphaAccuracyTokensAlphaSym\.Log\.Alg\.Qwen3\-0\.6B\\cellcolorbaselinegrayOriginal\\cellcolorbaselinegray0\\cellcolorbaselinegray55\.8%\\cellcolorbaselinegray184\\cellcolorbaselinegray0\\cellcolorbaselinegray52\.0%\\cellcolorbaselinegray656\\cellcolorbaselinegray0\\cellcolorbaselinegray55\.5%\\cellcolorbaselinegray48\.2%\\cellcolorbaselinegray12\.5%Steering0\.2556\.3%1840\.2552\.6%6620\.2556\.2%48\.6%14\.0%Steering0\.556\.0%1870\.552\.8%6450\.559\.2%49\.1%16\.8%Steering1\.055\.6%1861\.053\.4%6401\.058\.5%49\.8%17\.2%Qwen3\-4B\\cellcolorbaselinegrayOriginal\\cellcolorbaselinegray0\\cellcolorbaselinegray89\.7%\\cellcolorbaselinegray176\\cellcolorbaselinegray0\\cellcolorbaselinegray82\.0%\\cellcolorbaselinegray817\\cellcolorbaselinegray0\\cellcolorbaselinegray88\.1%\\cellcolorbaselinegray75\.8%\\cellcolorbaselinegray19\.8%Steering0\.2589\.9%1780\.2582\.6%8120\.2588\.4%76\.1%22\.0%Steering0\.590\.2%1780\.583\.4%8420\.592\.3%76\.5%24\.4%Steering1\.090\.5%1771\.084\.0%8431\.091\.6%76\.8%25\.7%Qwen3\-8B\\cellcolorbaselinegrayOriginal\\cellcolorbaselinegray0\\cellcolorbaselinegray92\.1%\\cellcolorbaselinegray213\\cellcolorbaselinegray0\\cellcolorbaselinegray82\.4%\\cellcolorbaselinegray896\\cellcolorbaselinegray0\\cellcolorbaselinegray88\.8%\\cellcolorbaselinegray76\.4%\\cellcolorbaselinegray20\.1%Steering0\.2592\.4%2150\.2583\.6%8850\.2589\.2%76\.9%23\.2%Steering0\.592\.6%2130\.584\.2%8730\.593\.1%77\.3%25\.6%Steering1\.093\.2%2101\.084\.0%9021\.092\.3%77\.6%26\.4%

Table 1:Main results ofSPARK\-Steering on GSM8K, MATH\-500, and FRONTIER\-4\.5K with Qwen3 series models\. Shaded rows denote the original baseline without steering\. For GSM8K and MATH\-500, we report the steering strengthα\\alpha, exact\-match accuracy, and average generated tokens\. For FRONTIER\-4\.5K, we report accuracy on symbolic \(Sym\.\), logical \(Log\.\), and algorithmic \(Alg\.\) reasoning subsets\. Bold numbers mark the best accuracy within each model and benchmark\.

## 5Experiments

We evaluateSPARK\-Steering from three perspectives: standard benchmark performance, sensitivity to steering strength and model family, and agreement between the observed gains and the proposed latent under\-activation mechanism\.

### 5\.1Experimental Setup

#### Benchmarks and models\.

We use GSM8K and MATH\-500 as reasoning benchmarks\[cobbe2021training,hendrycks2021measuring,lightman2024let\]\. GSM8K mainly contains arithmetic problems, while MATH\-500 contains more challenging problems\. We also evaluate on the held\-out split of our reasoning suiteFRONTIER\-4\.5K\. The main experiments use Qwen3 series models\[yang2025qwen3\]\. Experiments further test Llama\-3\.1\-8B\[grattafiori2024llama\]and DeepSeek\-R1\-Distill\-Qwen\-7B\[guo2025deepseek\], abbreviated as DeepSeek\-R1 in the following for readability\.

#### Steering protocol and metrics\.

SPARKis train\-free in the sense that no model parameters are updated and no task\-specific predictor is trained\. It is nevertheless data\-calibrated: before evaluation, we run the frozen model on training/calibration questions to estimate length\-response baselines, susceptibility distributions, reasoning\-active anchor pools, and benchmark\-matched steering layers\. Calibration labels are used to determine correctness for anchor selection and threshold calibration, while the reported accuracies are computed on held\-out evaluation questions\. The*Original*setting runs the frozen model without intervention\.*Steering*injects aSPARKdirection extracted from reasoning\-active anchors selected by length\-controlled susceptibility and cross\-layer coordination\. We report a sweep over steering strengthα\\alpharather than optimizing model parameters, and measure exact\-match accuracy together with average generated tokens\.

### 5\.2Main Results

Table[1](https://arxiv.org/html/2607.10296#S4.T1)reports the low\-alpha sweep on GSM8K, MATH\-500, and FRONTIER\-4\.5K\. Across Qwen3 model sizes, moderate steering improves accuracy without noticeably increasing generation length\. On GSM8K, where larger models are close to saturation, the gains are modest but stable\. Qwen3\-0\.6B improves from 55\.8% to 56\.3%, Qwen3\-4B from 89\.7% to 90\.5%, and Qwen3\-8B from 92\.1% to 93\.2%\. The gains are larger on MATH\-500, where examples require more sustained multi\-step reasoning\. Qwen3\-0\.6B improves from 52\.0% to 53\.4%, Qwen3\-4B from 82\.0% to 84\.0%, and Qwen3\-8B from 82\.4% to 84\.2%\. FRONTIER\-4\.5K shows the same trend in a controlled programmatic setting\. Steering consistently improves symbolic, logical, and algorithmic reasoning across model sizes, with the largest gains on the algorithmic subset\. Qwen3\-0\.6B improves from 12\.5% to 17\.2%, Qwen3\-4B from 19\.8% to 25\.7%, and Qwen3\-8B from 20\.1% to 26\.4%\. Together, these results suggest thatSPARK\-Steering is most useful when the model has partial reasoning ability but the original forward pass does not enter a sufficiently active latent state\. The average token counts remain in the same range, indicating that the improvement is not simply a consequence of longer generations\.

### 5\.3Analysis

#### Alpha sensitivity\.

Table[3](https://arxiv.org/html/2607.10296#S5.T3)extends the MATH\-500 sweep to larger steering strengths\. The response is clearly non\-monotonic, which is important for interpretingSPARKas a targeted latent intervention rather than arbitrary activation amplification\. Qwen3\-0\.6B reaches its best score atα=1\.0\\alpha=1\.0or1\.51\.5, improving from 52\.0% to 53\.4%\. Qwen3\-4B improves from 82\.0% to 84\.6%, with the best results atα=2\.5\\alpha=2\.5and3\.03\.0\. Qwen3\-8B obtains the largest gain, improving from 82\.4% to 85\.6% atα=2\.0\\alpha=2\.0andα=3\.0\\alpha=3\.0\. This dose\-response pattern suggests that moderate steering can move the hidden trajectory toward a more useful reasoning state, while overly strong steering can interfere with the model’s native computation\. The preferred steering strength also changes with model scale: larger Qwen3 models tolerate stronger intervention and obtain larger MATH\-500 gains\.

Table 2:Evaluation on MATH\-500\. The table reports original and steered performance \(%\) for Llama\-3\.1\-8B and DeepSeek\-R1 under different steering strengths, including exact\-match accuracy and average generated tokens\.Table 3:Large\-alpha sweep on MATH\-500 for Qwen3 models\. Each row reports the steering strengthα\\alpha, exact\-match accuracy \(%\), and average generated tokens for a single model\.![Refer to caption](https://arxiv.org/html/2607.10296v1/x3.png)Figure 3:Layer susceptibility and single\-layer steering dynamics\.The first three plots report the layerwise susceptibility spectrumχl\\chi\_\{l\}on the MATH\-500 benchmark inputs\. The x\-axis is normalized layer depth, the y\-axis is meanχl\\chi\_\{l\}on a log scale, and shaded regions denote SEM\. Red dashed lines mark the selected steering layerL⋆L^\{\\star\}, while circles mark susceptibility peaks\. The last plot reports held\-out MATH\-500 accuracy as steering strengthα\\alphavaries when injectingh←h\+α​vh\\leftarrow h\+\\alpha vatL⋆L^\{\\star\}\. The selected layers align with dominant susceptibility peaks, and the downstream accuracy changes remain bounded\.
#### Cross\-Family Generalization

Table[2](https://arxiv.org/html/2607.10296#S5.T2)tests whether the steering effect transfers beyond Qwen3\. On Llama\-3\.1\-8B, moderate steering improves MATH\-500 accuracy from 45\.6% to 48\.4% atα=0\.5\\alpha=0\.5, with only a moderate increase in generated tokens, which indicates that the learned direction is not tied to a single Qwen3 model size or architecture\. DeepSeek\-R1 exhibits a different sensitivity profile\. Small steering strengths still improve accuracy, from 85\.4% to 86\.4% atα=0\.25\\alpha=0\.25and 86\.2% atα=0\.5\\alpha=0\.5\. However, increasingα\\alphato 1\.0 sharply reduces accuracy to 32\.0% while increasing average generation length from 4138 to 7519 tokens\. We attribute this collapse to over\-steering: DeepSeek\-R1 already follows long native reasoning trajectories, and a large activation shift can disrupt these trajectories, making generation overly verbose and unstable\.

#### Layer Susceptibility and Single\-Layer Steering\.

We next examine whether the layers selected bySPARKcorrespond to meaningful susceptibility structure\. Figure[3](https://arxiv.org/html/2607.10296#S5.F3)shows the layerwise susceptibility spectrumχl\\chi\_\{l\}and the resulting MATH\-500 accuracy under single\-layer steering at the selected layerL⋆L^\{\\star\}\. Across Qwen3 model sizes, the selected steering layer consistently aligns with a dominant susceptibility peak\. For Qwen3\-0\.6B and Qwen3\-4B,L⋆L^\{\\star\}lies near the final layer, matching the sharp late\-layer rise ofχl\\chi\_\{l\}\. For Qwen3\-8B,L⋆L^\{\\star\}shifts to an earlier layer, suggesting that the most sensitive reasoning\-related layer can move with scale\. The downstream accuracy curves remain smooth and bounded asα\\alphavaries, supporting the view thatSPARKselects a controlled intervention site rather than injecting an arbitrary perturbation\.

![Refer to caption](https://arxiv.org/html/2607.10296v1/x4.png)Figure 4:Qualitative case study\.The base model reaches the correct setup but makes a local algebraic error when converting\(x3\+x−3\)2\(x^\{3\}\+x^\{\-3\}\)^\{2\}intox6\+x−6x^\{6\}\+x^\{\-6\}\.SPARK\-Steering preserves the setup and repairs this local transformation, changing the extracted answer from324324to the gold answer322322\. The example is scored by exact match after final\-answer extraction\.

### 5\.4Case Study

Figure[4](https://arxiv.org/html/2607.10296#S5.F4)shows a representative algebra example where the base model reaches the correct setup but fails at a local transformation\. Given a real root ofx2−3​x\+1=0x^\{2\}\-3x\+1=0, the task is to computex6\+x−6x^\{6\}\+x^\{\-6\}\. Both the original and steered responses correctly derivex\+x−1=3x\+x^\{\-1\}=3andx3\+x−3=18x^\{3\}\+x^\{\-3\}=18\. The original response then incorrectly setsx6\+x−6=\(x3\+x−3\)2=324x^\{6\}\+x^\{\-6\}=\(x^\{3\}\+x^\{\-3\}\)^\{2\}=324, omitting the cross term in\(x3\+x−3\)2=x6\+2\+x−6\(x^\{3\}\+x^\{\-3\}\)^\{2\}=x^\{6\}\+2\+x^\{\-6\}\. WithSPARK\-Steering atα=3\.0\\alpha=3\.0, the model preserves the setup but applies the corrected identity, producing182−2=32218^\{2\}\-2=322\. The extracted final answer changes from324324to the gold answer322322, and the example is scored after final\-answer extraction\.

### 5\.5Ablation Studies

Table 4:Ablation study on MATH\-500 with Qwen3\-8B\.Δ\\Deltadenotes the accuracy change over the original model\. Direction ablations test how the steering vector is constructed, while layer ablations test where the vector is injected\.Table[4](https://arxiv.org/html/2607.10296#S5.T4)compares different direction\-construction and layer\-selection choices\. Random directions give only a small gain, indicating that arbitrary activation noise does not explain the improvement\. Easy correct anchors and high raw\-χ\\chianchors improve over the baseline, but remain weaker than length\-controlled directions\. This supports the central role of length control: raw susceptibility captures useful response information, but it is still confounded by prompt length\. The best direction is obtained from highSPKLC\\mathrm\{SPK\}\_\{\\mathrm\{LC\}\}anchors, which combine length\-controlled response with cross\-layer coordination\. Layer ablations show the same pattern\. Fixed early, middle, or late layers provide partial gains, but susceptibility\-guided layer selection performs best\.

## 6Conclusion

This work studies reasoning failures in LLMs through the lens of latent state activation\. Final\-answer accuracy reveals where a model succeeds or fails, but it does not distinguish missing capability from a failure to activate an available reasoning state\. We show that hidden\-state susceptibility offers a useful internal response signal for this distinction, while also revealing an important caveat\. Raw susceptibility is strongly affected by prompt length, especially in algorithmic reasoning\. Length\-controlled susceptibility addresses this confound and provides a cleaner view of residual reasoning activation in hard examples\. Building on this diagnosis,SPARKconnects latent profiling with test\-time intervention\. It identifies reasoning\-active anchors, detects under\-activated hard targets, and uses the diagnosed active direction to steer hidden states while keeping model parameters frozen\. Experiments on standard benchmarks show thatSPARKimproves reasoning performance, with stronger gains on harder examples and limited disruption to easier ones\. These findings suggest that reasoning evaluation should not stop at measuring whether a model answers correctly\. A more informative evaluation also asks whether the internal computation needed for reasoning is activated, how this activation is shaped by input scale, and whether it can be guided during inference\.

## References