犹豫有其几何形态:面向稀疏激活引导的熵训练双曲探针
摘要
论文提出HEST方法,通过熵训练探针将大语言模型的隐藏状态嵌入到庞加莱球中,仅沿双曲测地线引导高熵(即犹豫)的token,从而在Qwen2.5-Math和Llama-3.1模型上提升数学推理准确率,而固定向量的欧式引导方式在这些模型上效果不佳。
arXiv:2610.02391v1 Announce Type: new
Abstract: When a large language model solves a mathematical problem, its reasoning is largely hierarchical, and the solution often branches at a few tokens where the next-token entropy is high. Such tree-like structure embeds in hyperbolic space with far lower distortion than in Euclidean space. Activation steering, however, usually edits the hidden states of a pretrained model by adding one fixed Euclidean vector at every token, even though most tokens of a solution are already determined by the context. We propose Hyperbolic Entropy Steering (HEST), which embeds the hidden states in the Poincar\'e ball with a lightweight probe whose only label is the model's own next-token entropy. Where this entropy exceeds a threshold, HEST moves the embedded state along the geodesic of steepest descent of a readout of the probe and maps the change back to the hidden state. For the Busemann readout of a learned ideal point, we prove that a step of fixed length lowers it by the same amount at every state. On three instruction-tuned models from the Qwen2.5-Math and Llama-3.1 families, HEST with the Busemann readout improves greedy accuracy on MATH-500 and GSM8K in five of six settings, by up to 1.8 points, whereas a contrastive steering vector added at every token lowers accuracy. With a Euclidean probe trained in the same way, this gain disappears on Qwen2.5-Math-1.5B-Instruct. The gains are largest on problems where the model hesitates often, and accuracy on the remaining problems is almost unchanged.
查看缓存全文
缓存时间: 2026/10/05 10:04
# Entropy-Trained Hyperbolic Probes for Sparse Activation Steering
Source: [https://arxiv.org/html/2610.02391](https://arxiv.org/html/2610.02391)
\\tl\_set:Ne\\outputfile
outputfile
## Hesitation Has a Geometry: Entropy\-Trained Hyperbolic Probes for Sparse Activation Steering
Zeyong ZhangAffiliation:Department of Data ScienceAffiliation:New Jersey Institute of TechnologyAffiliation:Newark, NJ, USAEmail:[zz63@njit\.edu](mailto:)Tung Sum Thomas KwokAffiliation:Department of Statistics and Data ScienceAffiliation:University of California, Los AngelesAffiliation:Los Angeles, CA, USAEmail:[tk1018@ucla\.edu](mailto:)Tengfei MaAffiliation:Department of Biomedical InformaticsAffiliation:Stony Brook UniversityAffiliation:Stony Brook, NY, USAEmail:[tengfei\.ma@stonybrook\.edu](mailto:)Mengjia Xu††thanks:Corresponding author\.Affiliation:Department of Data ScienceAffiliation:New Jersey Institute of TechnologyAffiliation:Newark, NJ, USAEmail:[mx6@njit\.edu](mailto:)
###### Abstract
When a large language model solves a mathematical problem, its reasoning is largely hierarchical, and the solution often branches at a few tokens where the next\-token entropy is high\. Such tree\-like structure embeds in hyperbolic space with far lower distortion than in Euclidean space\. Activation steering, however, usually edits the hidden states of a pretrained model by adding one fixed Euclidean vector at every token, even though most tokens of a solution are already determined by the context\. We proposeHyperbolicEntropySteering \(HEST\), which embeds the hidden states in the Poincaré ball with a lightweight probe whose only label is the model’s own next\-token entropy\. Where this entropy exceeds a threshold,HESTmoves the embedded state along the geodesic of steepest descent of a readout of the probe and maps the change back to the hidden state\. For the Busemann readout of a learned ideal point, we prove that a step of fixed length lowers it by the same amount at every state\. On three instruction\-tuned models from the Qwen2\.5\-Math and Llama\-3\.1 families,HESTwith the Busemann readout improves greedy accuracy on MATH\-500 and GSM8K in five of six settings, by up to 1\.8 points, whereas a contrastive steering vector added at every token lowers accuracy\. With a Euclidean probe trained in the same way, this gain disappears on Qwen2\.5\-Math\-1\.5B\-Instruct\. The gains are largest on problems where the model hesitates often, and accuracy on the remaining problems is almost unchanged\.
## 1Introduction
Figure 1:HESTsteers only the tokens at which the model hesitates, along a geodesic in the space of a hyperbolic probe\.\(a\) Fixed\-vector steering adds the same vectorαv\\alpha vat every token\. \(b, c\)HESTmaps hidden states into the Poincaré ball with a probe trained on next\-token entropy\. The readoutssis either the Busemann function of the ideal pointξ\\xior the distancerrto the originOO, and the arrows show the steering directionuu\. \(d\)HESTsteers only tokens whose entropyHiH\_\{i\}exceeds the thresholdθ\\theta\. \(e\) At such a token,hhis mapped tozz, moved by hyperbolic distancettalong the geodesic in the directionuu, and mapped back\. In \(e\),∥⋅∥\\lVert\\cdot\\rVertis the Euclidean norm\. The point clouds and step lengths in \(b, c\) are schematic, and Appendix[O](https://arxiv.org/html/2610.02391#A15)shows real embedded states\.Large language models solve mathematical problems through multi\-step reasoning, and such reasoning, like language itself, is largely hierarchical\. Hierarchical relations between concepts are reflected in the geometry of their representations\([Park et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib29)\)\. In mathematical reasoning, a chain of thought also tends to branch at a small number of high\-entropy tokens, which can determine the subsequent solution path\([Wang et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib42)\)\. Hyperbolic space suits such tree\-like structure, since its volume grows exponentially with the radius, and trees embed in it with far lower distortion than in a Euclidean space of the same dimension\([Nickel and Kiela, 2017](https://arxiv.org/html/2610.02391#bib.bib27);[Sala et al\., 2018](https://arxiv.org/html/2610.02391#bib.bib35)\)\. This property has been used to model hierarchies in neural networks, Transformers and language models\([Ganea et al\., 2018](https://arxiv.org/html/2610.02391#bib.bib7);[Yang et al\., 2024b](https://arxiv.org/html/2610.02391#bib.bib46);[Yang et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib47);[He et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib11);[Patil et al\., 2026a](https://arxiv.org/html/2610.02391#bib.bib30);[Patil et al\., 2026b](https://arxiv.org/html/2610.02391#bib.bib31)\), and it also makes hyperbolic space a natural candidate for representing the hidden states of Euclidean language models\.
Activation steering changes the behavior of a pretrained language model at inference time without updating its weights\. It shifts the hidden activations along a direction extracted from contrastive examples\([Turner et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib40);[Li et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib20);[Zou et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib50);[Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33)\)\. In activation\-addition methods, this direction is a Euclidean vector that is typically the same at every token position \(Figure[1](https://arxiv.org/html/2610.02391#S1.F1)a\)\. Contrastive Activation Addition\([Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33), CAA;\), for example, applies the same steering vector throughout generation\. In mathematical reasoning, such a vector perturbs every token of a solution, although most tokens are largely determined by the preceding context and the solution branches at only a few high\-entropy tokens\. Existing steering methods for mathematical reasoning mostly act at the step boundaries of long reasoning models\([Chen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib3);[Li et al\., 2026b](https://arxiv.org/html/2610.02391#bib.bib22)\)and classify the steps by keywords such as “Wait”, which instruction\-tuned models rarely produce\. We instead steer instruction\-tuned language models only at high\-entropy tokens, and we take the direction from a hyperbolic embedding of their hidden states that a lightweight probe learns from the same entropy\.
We start from an observation about the model’s own uncertainty\. The entropy of the next\-token distribution is low at most positions and exceeds a threshold at only a small fraction of them, which we call*hesitation points*\. The same entropy can also serve as a training signal, and we train a probe that maps hidden states into the Poincaré ball and orders them by entropy\. Without external labels, it learns a geometry in which confident states lie closer than hesitant ones to an ideal point on the boundary of the ball, or to its origin\. The Busemann function of this ideal point and the distance to the origin serve as readouts, and the negative Riemannian gradient of a readout gives a steering direction that depends on the state\. We build Hyperbolic Entropy Steering \(HEST\) on this observation \(Figure[1](https://arxiv.org/html/2610.02391#S1.F1)\)\. During greedy decoding,HESTembeds the hidden state at every hesitation point and moves the embedding along a geodesic that lowers the readout\. It then maps the change back to the hidden state, decodes the next token from the edited state and edits no other position\. We show that a step of hyperbolic lengthttlowers the Busemann readout by exactlyct\\sqrt\{c\}\\,t, where−c\-cis the curvature, and the radius readout by exactlyttwhenever the step does not pass the origin\. Hence, the step length has the same meaning at every state \(Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)\)\.
We evaluateHESTon Qwen2\.5\-Math\-1\.5B\-Instruct, our development model, Qwen2\.5\-Math\-7B\-Instruct and Llama\-3\.1\-8B\-Instruct, using MATH\-500, GSM8K and, for the development model, OlympiadBench\. With the Busemann readout,HESTimproves greedy accuracy in five of the six model and dataset pairs on MATH\-500 and GSM8K, by up to1\.81\.8points, while steering a median of0\.80\.8to3\.0%3\.0\\%of solution tokens\. A difference\-of\-means vector built from the same states and added at every token, as in CAA, lowers accuracy in all six pairs\. With a Euclidean probe trained in the same way and edits of the same size, the Busemann readout also lowers accuracy for the development model\. The gains are largest on the few problems on which the greedy solution hesitates often\. Most other greedy solutions are already correct, andHESTrarely changes their outcome\. Our contributions are:
- •We proposeHEST\(Section[3](https://arxiv.org/html/2610.02391#S3)\), an inference\-time steering method in which the model’s own next\-token entropy decides where to steer and also trains the hyperbolic probe that gives the direction\.HESTrequires no correctness labels, contrastive prompts, language model training or additional sampled solutions\.
- •We give a geometric account of the method \(Sections[3\.1](https://arxiv.org/html/2610.02391#S3.SS1)and[3\.2](https://arxiv.org/html/2610.02391#S3.SS2), Appendix[B](https://arxiv.org/html/2610.02391#A2)\)\. Both readouts have a direct interpretation, one geodesic step lowers a readout by an exact amount, and in the probe space the Busemann direction depends on the state, whereas its Euclidean counterpart does not\.
- •We evaluateHESTon three models against greedy decoding and CAA \(Section[4](https://arxiv.org/html/2610.02391#S4)\), ablate where to steer, in which direction and in which geometry \(Section[5](https://arxiv.org/html/2610.02391#S5)\), and analyze which problems benefit \(Section[4\.3](https://arxiv.org/html/2610.02391#S4.SS3)\)\.
## 2Related Work
#### Activation steering\.
Most activation steering methods add a fixed vector to the hidden states, at every generated token or at fixed positions of the prompt\. The vector is obtained by contrasting prompts or examples on behaviors such as truthfulness, refusal and sentiment\([Turner et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib40);[Li et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib20);[Zou et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib50);[Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33)\)\. ReFT and HyperSteer instead learn the intervention, as low\-rank edits at fixed prompt positions and as vectors that a hypernetwork generates from a steering prompt and the input\([Wu et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib44);[Sun et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib38)\)\. Several recent methods let the edit depend on the state\. AcT transports activations with coordinate\-wise affine maps from optimal transport\([Rodriguez et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib34)\), and ODESteer and FLAS move them along a vector field\([Zhao et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib49);[Jin et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib15)\)\. Curveball Steering works in the feature space of a kernel PCA\([Raval et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib32)\), and Angular and Spherical Steering rotate activations\([Vu and Nguyen, 2026](https://arxiv.org/html/2610.02391#bib.bib41);[You et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib48)\)\. K\-Steering, the closest to our mechanism, follows the gradient of a nonlinear classifier trained with attribute labels\([Oozeer et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib28)\)\. CLAS scales a direction at every layer by a coefficient linear in the activation, and it uses GSM8K to elicit step\-by\-step solutions\([Hsu et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib14)\)\. Other methods restrict where the edit is applied\. MAT\-Steer learns a gate that weights each steering vector at each prompt token\([Nguyen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib26)\), and Spherical Steering rotates only when a confidence score for the undesired direction exceeds a threshold\([You et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib48)\)\. FASB backtracks when a probe trained on labeled data detects a deviation\([Cheng et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib5)\)\. Helix lowers the largest logit where a score of the next\-token entropy and a Mahalanobis distance of the hidden state indicates divergence\([Atkinson, 2026](https://arxiv.org/html/2610.02391#bib.bib1)\)\.HESTcombines a state\-dependent direction with a sparse gate, and both come from the model’s own entropy rather than from labels, as the comparison in Appendix[A](https://arxiv.org/html/2610.02391#A1)details\.
#### Entropy and forking tokens\.
Token\-level entropy marks the positions where a reasoning chain often branches\. In reinforcement learning for reasoning, a minority of high\-entropy tokens determines the reasoning path, and restricting policy updates to these tokens matches or exceeds updating all tokens\([Wang et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib42)\)\. Hidden states also reflect uncertainty about which path the generation will take\([Zur et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib51)\)\. HARP treats tokens whose entropy exceeds a threshold as points of hesitation, applies dropout to the input embeddings and mixes the logits of a second forward pass\([Storaï and Hwang, 2025](https://arxiv.org/html/2610.02391#bib.bib37)\)\. Entropy\-gated branching instead selects among several continuations at such tokens with a process reward model\([Li et al\., 2026a](https://arxiv.org/html/2610.02391#bib.bib21)\)\. Some steering methods for mathematical reasoning target long reasoning models\. SEAL splits the reasoning into steps at double line breaks, classifies the steps by keywords and steers the representations at every step boundary\([Chen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib3)\)\. FlowSteer keeps this protocol but replaces the linear edit with flow matching\([Li et al\., 2026b](https://arxiv.org/html/2610.02391#bib.bib22)\)\.[Sun et al\. \(2026\)](https://arxiv.org/html/2610.02391#bib.bib39)study the reasoning trajectories of three Llama\-3\.1\-8B variants and also steer at step boundaries\. Their steering uses a predictor trained on correctness labels, or ideal trajectories from correct solutions\. Like HARP and entropy\-gated branching,HESTselects tokens by their entropy, but it edits their hidden states and uses neither keywords nor correctness labels\.
#### Latent\-space methods for test\-time scaling\.
Another line of work spends additional inference compute in latent space\. Fractional Reasoning scales a latent steering vector to control the depth of reasoning, and it reports its gains with majority voting or best\-of\-NNselection\([Liu et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib24)\)\. LatentSeek optimizes latent representations for each problem with a policy gradient at test time\([Li et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib19)\)\. SPREAD jointly optimizes steering vectors for several sampled solutions to diversify them, by Riemannian optimization over a product of spheres\([Khanh et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib16)\)\. These methods use several candidates or optimize per problem, whereasHESTedits a single greedy decode, so we do not use them as baselines\. Random embedding injection appends untrained random embeddings to the input to diversify reasoning, and it is complementary toHEST\([Kim et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib17)\)\.
#### Hyperbolic representation\.
[Ganea et al\. \(2018\)](https://arxiv.org/html/2610.02391#bib.bib7)define hyperbolic neural network layers through Möbius gyrovector operations in the Poincaré ball\.[Ghadimi Atigh et al\. \(2021\)](https://arxiv.org/html/2610.02391#bib.bib8)use Busemann functions to classify against ideal prototypes, and[Chen et al\. \(2026\)](https://arxiv.org/html/2610.02391#bib.bib4)build hyperbolic network layers from them\. HypFormer builds fully hyperbolic Transformers\([Yang et al\., 2024b](https://arxiv.org/html/2610.02391#bib.bib46)\)\. HypLoRA fine\-tunes large language models with low\-rank adapters in hyperbolic space\([Yang et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib47)\), and HELM builds hyperbolic large language models, including a variant whose experts have different curvatures\([He et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib11)\)\.[Patil et al\. \(2026a\)](https://arxiv.org/html/2610.02391#bib.bib30)survey hyperbolic language models\. HyperGuide projects hidden states into the Poincaré ball and fine\-tunes the model to use the resulting signal in multi\-step reasoning\([Liu et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib25)\)\. Our probe follows the ranking and triplet objective of HyperGuide, which trains its projection on distances to a solution in a search tree rather than on the model’s own entropy\. We use our readouts to steer a Euclidean model without fine\-tuning it\.
## 3Method
#### Setup and notation\.
LetMMbe a decoder\-only language model, decoded greedily, with residual streamh∈ℝdh\\in\\mathbb\{R\}^\{d\}at layerℓ\\ell\. After the model reads the token at positionii, it has a layer\-ℓ\\ellstatehih\_\{i\}and a distributionpip\_\{i\}over the next token, with entropy
Hi=−∑v∈𝒱pi\(v\)logpi\(v\)\.H\_\{i\}\\;=\\;\-\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{i\}\(v\)\\log p\_\{i\}\(v\)\.\(1\)It is computed from logits that the model produces during decoding, so it requires no annotation\. We call positioniia*hesitation point*whenHi\>θH\_\{i\}\>\\theta\. The entropy plays two roles inHEST: the probe learns to order states by it \(Section[3\.1](https://arxiv.org/html/2610.02391#S3.SS1)\), and it decides where to steer \(Section[3\.4](https://arxiv.org/html/2610.02391#S3.SS4)\)\.
### 3\.1A Hyperbolic Probe Trained on Entropy
We use the Poincaré ball𝔻cn=\{z∈ℝn:c∥z∥2<1\}\\mathbb\{D\}^\{n\}\_\{c\}=\\\{z\\in\\mathbb\{R\}^\{n\}:c\\lVert z\\rVert^\{2\}<1\\\}of curvature−c\-c, with metricgz=\(λzc\)2gEg\_\{z\}=\(\\lambda^\{c\}\_\{z\}\)^\{2\}g^\{E\}and conformal factorλzc=2/\(1−c∥z∥2\)\\lambda^\{c\}\_\{z\}=2/\(1\-c\\lVert z\\rVert^\{2\}\)\([Ganea et al\., 2018](https://arxiv.org/html/2610.02391#bib.bib7)\)\. The probeE:ℝd→𝔻cnE:\\mathbb\{R\}^\{d\}\\to\\mathbb\{D\}^\{n\}\_\{c\}is
E\(h\)=exp0c\(ϕ\(\(h−μ\)/σ\)ω\),exp0c\(v\)=tanh\(c∥v∥\)vc∥v∥,E\(h\)=\\exp^\{c\}\_\{0\}\\\!\\left\(\\frac\{\\phi\\big\(\(h\-\\mu\)/\\sigma\\big\)\}\{\\omega\}\\right\),\\qquad\\exp^\{c\}\_\{0\}\(v\)=\\tanh\\\!\\big\(\\sqrt\{c\}\\,\\lVert v\\rVert\\big\)\\,\\frac\{v\}\{\\sqrt\{c\}\\,\\lVert v\\rVert\},\(2\)whereμ\\muandσ\\sigmaare the coordinate\-wise mean and standard deviation of the training states, andϕ\\phiis a two\-layer perceptron with hidden width10241024and output dimensionn=128n=128\. The scalarω\\omegais a running root mean square of∥ϕ\(⋅\)∥\\lVert\\phi\(\\cdot\)\\rVertand fixes the scale of the embedding\.
#### Readouts\.
A readout is a scalar functionsson the ball, low at confident states and high at hesitant ones\. The first readout is built from the Busemann function of a learned ideal point on the boundary of the ball\([Ghadimi Atigh et al\., 2021](https://arxiv.org/html/2610.02391#bib.bib8)\)\. We write this point asξ/c\\xi/\\sqrt\{c\}with∥ξ∥=1\\lVert\\xi\\rVert=1and define
b\(z\)=Bξ\(cz\),Bξ\(y\)=log∥ξ−y∥21−∥y∥2\.b\(z\)=B\_\{\\xi\}\\big\(\\sqrt\{c\}\\,z\\big\),\\qquad B\_\{\\xi\}\(y\)=\\log\\frac\{\\lVert\\xi\-y\\rVert^\{2\}\}\{1\-\\lVert y\\rVert^\{2\}\}\.\(3\)HereBξB\_\{\\xi\}is the Busemann function of the unit ball, sobbisc\\sqrt\{c\}times that of𝔻cn\\mathbb\{D\}^\{n\}\_\{c\}\. Its level sets are horospheres, which are spheres tangent to the boundary at the ideal point, andbbdecreases without bound along every geodesic that ends atξ/c\\xi/\\sqrt\{c\}\. The second readout is the distance to the origin,
r\(z\)=dc\(0,z\)=2cartanh\(c∥z∥\),r\(z\)=d\_\{c\}\(0,z\)=\\frac\{2\}\{\\sqrt\{c\}\}\\operatorname\{artanh\}\\\!\\big\(\\sqrt\{c\}\\,\\lVert z\\rVert\\big\),\(4\)whose level sets are spheres centered at the origin\. Both readouts are interpretable, as confident states tend to lie on smaller horospheres than hesitant ones, or nearer to the origin \(Figure[1](https://arxiv.org/html/2610.02391#S1.F1)b,c\)\.
#### Training objective\.
Writezi=E\(hi\)z\_\{i\}=E\(h\_\{i\}\)\. We follow[Liu et al\. \(2026\)](https://arxiv.org/html/2610.02391#bib.bib25), who train a hyperbolic probe with a pairwise margin ranking loss\([Herbrich et al\., 2000](https://arxiv.org/html/2610.02391#bib.bib13)\)and a triplet loss\([Weinberger and Saul, 2009](https://arxiv.org/html/2610.02391#bib.bib43);[Schroff et al\., 2015](https://arxiv.org/html/2610.02391#bib.bib36)\)in the Poincaré ball\. The probe minimizes
ℒ=𝔼\(i,j\):Hi<Hj−δ\[γ\+s\(zi\)−s\(zj\)\]\+\+𝔼\(a,p,q\)\[γ\+dc\(za,zp\)−dc\(za,zq\)\]\+,\\mathcal\{L\}=\\mathop\{\\mathbb\{E\}\}\_\{\(i,j\):\\,H\_\{i\}<H\_\{j\}\-\\delta\}\\Big\[\\gamma\+s\(z\_\{i\}\)\-s\(z\_\{j\}\)\\Big\]\_\{\+\}\+\\mathop\{\\mathbb\{E\}\}\_\{\(a,p,q\)\}\\Big\[\\gamma\+d\_\{c\}\(z\_\{a\},z\_\{p\}\)\-d\_\{c\}\(z\_\{a\},z\_\{q\}\)\\Big\]\_\{\+\},\(5\)with\[x\]\+=max\(0,x\)\[x\]\_\{\+\}=\\max\(0,x\),δ=0\.3\\delta=0\.3nats andγ=0\.1\\gamma=0\.1\. The first term orders states by entropy, and the second keeps nearby states of one solution close: the positivepplies within five tokens of the anchoraa, and the negativeqqcomes from another problem\. The curvature and the ideal point are learned jointly withϕ\\phi\. The data are the states and entropies of greedy solutions to100100training problems, without any correctness label\. On held\-out problems, the Busemann readout orders8585to90%90\\%of the entropy pairs correctly in all settings, and the radius readout8282to89%89\\%\(Appendix[D](https://arxiv.org/html/2610.02391#A4)\)\.
### 3\.2Steering along a Geodesic
Lethhbe the state at a hesitation point andz=E\(h\)z=E\(h\)\. Because the metric is conformal, the Riemannian gradient∇ℛs\(z\)=\(λzc\)−2∇s\(z\)\\nabla^\{\\mathcal\{R\}\}s\(z\)=\(\\lambda^\{c\}\_\{z\}\)^\{\-2\}\\,\\nabla s\(z\)has the direction of the Euclidean gradient\. We set
u=−∇ℛs\(z\)∥∇ℛs\(z\)∥,z′=z⊕c\(tanh\(ct2\)uc\),u=\-\\frac\{\\nabla^\{\\mathcal\{R\}\}s\(z\)\}\{\\lVert\\nabla^\{\\mathcal\{R\}\}s\(z\)\\rVert\},\\qquad z^\{\\prime\}=z\\oplus\_\{c\}\\left\(\\tanh\\\!\\Big\(\\frac\{\\sqrt\{c\}\\,t\}\{2\}\\Big\)\\frac\{u\}\{\\sqrt\{c\}\}\\right\),\(6\)where∥⋅∥\\lVert\\cdot\\rVertis the Euclidean norm and⊕c\\oplus\_\{c\}is Möbius addition\([Ganea et al\., 2018](https://arxiv.org/html/2610.02391#bib.bib7)\)\. The pointz′z^\{\\prime\}equalsexpzc\(tv\)\\exp^\{c\}\_\{z\}\(t\\,v\)withv=u/λzcv=u/\\lambda^\{c\}\_\{z\}, the unit tangent vector in the directionuuunder the hyperbolic metric\.
###### Proposition 1\.
Letz′z^\{\\prime\}be given by Eq\.[6](https://arxiv.org/html/2610.02391#S3.E6)witht\>0t\>0and∇s\(z\)≠0\\nabla s\(z\)\\neq 0\. Then \(i\)dc\(z,z′\)=td\_\{c\}\(z,z^\{\\prime\}\)=t; \(ii\) for the radius readout, ifz≠0z\\neq 0andt≤r\(z\)t\\leq r\(z\), thenr\(z′\)=r\(z\)−tr\(z^\{\\prime\}\)=r\(z\)\-t; \(iii\) for the Busemann readout,b\(z′\)=b\(z\)−ctb\(z^\{\\prime\}\)=b\(z\)\-\\sqrt\{c\}\\,t\.
Part \(i\) follows from the left\-cancellation law of Möbius addition\. Parts \(ii\) and \(iii\) use two standard facts: distance and Busemann functions have unit gradient in the hyperbolic metric, and their descent curves are geodesics\([Bridson and Haefliger, 2013](https://arxiv.org/html/2610.02391#bib.bib2)\)\. The factorc\\sqrt\{c\}comes from the rescalingy=czy=\\sqrt\{c\}\\,z, and Appendix[B](https://arxiv.org/html/2610.02391#A2)gives the proof\. The step lengthttthus has the same meaning at every state, since it is the hyperbolic distance traveled and fixes the drop of the readout\. The conditiont≤r\(z\)t\\leq r\(z\)holds at every hesitation point of all greedy test solutions \(Appendix[B\.3](https://arxiv.org/html/2610.02391#A2.SS3.SSS0.Px1)\)\.
#### Role of the curvature\.
In the limitc→0c\\to 0, evaluated in Section[5](https://arxiv.org/html/2610.02391#S5), the probe becomes Euclidean, the geodesic step becomes a straight stepz′=z\+\(t/2\)uz^\{\\prime\}=z\+\(t/2\)\\,u, and the readouts become the norm and the linear function−⟨ξ,z⟩\-\\langle\\xi,z\\rangleup to scale \(Appendix[N](https://arxiv.org/html/2610.02391#A14)\)\. For the radius readout, the step points to the origin in both geometries, and the two differ only in how far a step of lengthttmoves\. The Busemann readout is where the geometries differ\. In the Poincaré ball, its descent curves are geodesics that converge on the ideal point, so the directionuudepends on the state\. The linear function, in contrast, has parallel hyperplanes as level sets and the same descent directionξ\\xieverywhere \(Appendix[B\.5](https://arxiv.org/html/2610.02391#A2.SS5)\)\. After the mapping back to the hidden state, any readout gives a state\-dependent direction inℝd\\mathbb\{R\}^\{d\}through the Jacobian of the probe \(Appendix[L](https://arxiv.org/html/2610.02391#A12)\), so the geometries differ in the probe space\.
### 3\.3Mapping the Step back to the Hidden State
We look for an editΔh\\Delta hwithE\(h\+Δh\)≈z′E\(h\+\\Delta h\)\\approx z^\{\\prime\}and solve this underdetermined problem by damped Gauss–Newton iterations fromΔh=0\\Delta h=0,
Δh←Δh\+J⊤\(JJ⊤\+ηI\)−1\(z′−E\(h\+Δh\)\),J=∂E∂h\|h\+Δh,\\Delta h\\;\\leftarrow\\;\\Delta h\+J^\{\\top\}\\big\(JJ^\{\\top\}\+\\eta I\\big\)^\{\-1\}\\big\(z^\{\\prime\}\-E\(h\+\\Delta h\)\\big\),\\qquad J=\\frac\{\\partial E\}\{\\partial h\}\\Big\|\_\{h\+\\Delta h\},\(7\)withη=10−3tr\(JJ⊤\)/n\\eta=10^\{\-3\}\\operatorname\{tr\}\(JJ^\{\\top\}\)/n\. We run at most four iterations and clip∥Δh∥\\lVert\\Delta h\\rVertto at most∥h∥\\lVert h\\rVert\. Each iteration solves ann×nn\\times nsystem, andJJhas a closed form\. The edited states realize the distancettand the dropct\\sqrt\{c\}\\,tof the Busemann readout in Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)to within10−410^\{\-4\}on average \(Appendix[C](https://arxiv.org/html/2610.02391#A3)\)\.
### 3\.4Decoding with the Entropy Gate
HESTruns inside greedy decoding, where the model producespip\_\{i\}after reading the tokenyiy\_\{i\}\. IfHi\>θH\_\{i\}\>\\theta, we remove the key and value entries of positioniifrom the cache\. We then recompute the forward pass ofyiy\_\{i\}withhi\+Δhih\_\{i\}\+\\Delta h\_\{i\}in place ofhih\_\{i\}at layerℓ\\elland choose the next token from the new distribution\. The edit thus acts on the state that produced the uncertain distribution, and one position later the model is usually confident again \(Appendix[E](https://arxiv.org/html/2610.02391#A5)\)\. The edited keys and values stay in the cache, so later tokens attend to the edited state\. No other position is edited \(Algorithm[1](https://arxiv.org/html/2610.02391#alg1)\)\. A steered decode that reaches the limit of30723072new tokens falls back to the greedy answer\.
### 3\.5Choice of Hyperparameters
The three hyperparameters ofHESTare the layerℓ\\ell, the thresholdθ\\thetaand the step lengthtt, and we take the layer from prior work\. For the Qwen models, we useℓ=19\\ell=19, the layer that SEAL uses for DeepSeek\-R1\-Distill\-Qwen\-1\.5B and 7B, which are fine\-tuned from Qwen2\.5\-Math\([Chen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib3)\)\. For Llama, we useℓ=13\\ell=13, the layer that CAA uses for Llama 2 7B Chat and ODESteer uses for Llama\-3\.1\-8B\([Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33);[Zhao et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib49)\), and Appendix[D](https://arxiv.org/html/2610.02391#A4)explains the layer indexing\. The threshold isθ=1\\theta=1for the development model, Qwen2\.5\-Math\-1\.5B\-Instruct\. Sinceexp\(Hi\)\\exp\(H\_\{i\}\)is the effective number of candidate tokens,Hi\>1H\_\{i\}\>1means more thane≈2\.7e\\approx 2\.7effective candidates\. Because entropy scales differ across models, we chooseθ\\thetafor each other model so thatHESTsteers about as many tokens per solution as on the development model\. We estimate this number before steering from the teacher\-forced entropies of the greedy solutions, which needs no labels \(Appendix[G](https://arxiv.org/html/2610.02391#A7)\)\. The step length is chosen on MATH\-500 and GSM8K for the development model \(Table[1](https://arxiv.org/html/2610.02391#S4.T1)\):t=0\.5t=0\.5for the Busemann readout andt=0\.25t=0\.25for the radius readout\. We keep these values for the other two models, so no label of their test sets enters their hyperparameters\.
## 4Empirical Results
### 4\.1Setup
Table 1:Accuracy \(%\) of Qwen2\.5\-Math\-1\.5B\-Instruct and change from greedy decoding \(points\) at four step lengthstt\. Greedy decoding:74\.2074\.20on MATH\-500,84\.8484\.84on GSM8K and41\.3941\.39on OlympiadBench\. All other models use the step lengthst⋆t^\{\\star\}\. Bold: best accuracy per benchmark\.#### Models and benchmarks\.
We use Qwen2\.5\-Math\-1\.5B\-Instruct as the development model, together with Qwen2\.5\-Math\-7B\-Instruct\([Yang et al\., 2024a](https://arxiv.org/html/2610.02391#bib.bib45)\)and Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib9)\)\. We evaluate on MATH\-500\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.02391#bib.bib12);[Lightman et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib23)\), the1,3191\{,\}319test problems of GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.02391#bib.bib6)\)and, for the development model, OlympiadBench\([He et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib10)\)\. From OlympiadBench, we take the675675English, text\-only mathematics problems with open\-ended answers, of which a fixed set of100100trains the probe and the other575575are used for evaluation\. OlympiadBench is harder, and the model hesitates on it about twice as densely as on MATH\-500, so we use it to testHESTunder dense hesitation \(Section[4\.3](https://arxiv.org/html/2610.02391#S4.SS3)\)\. For MATH\-500 and GSM8K, the probe is trained on100100problems from the training split of the same dataset\. All methods decode greedily from the same prompt with a limit of30723072new tokens\. All outputs are graded by one protocol that extracts the final answer and compares it with the reference by string, numeric and symbolic checks \(Appendix[F](https://arxiv.org/html/2610.02391#A6)\)\.
#### Baselines\.
We compare with greedy decoding and with CAA\([Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33)\)\. The CAA direction is the difference of meansv=𝔼\[h∣H≤θ\]−𝔼\[h∣H\>θ\]v=\\mathbb\{E\}\[h\\mid H\\leq\\theta\]\-\\mathbb\{E\}\[h\\mid H\>\\theta\]\. It uses the same probe\-training states, layer and threshold asHEST\. It is added at every generated token with multiplier11, as in most experiments of the original method\. CAA is thus the fixed\-direction, all\-token counterpart ofHEST, built from the same data\. We report it without the fallback of Section[3\.4](https://arxiv.org/html/2610.02391#S3.SS4), and with the fallback it still lowers accuracy \(Appendix[I](https://arxiv.org/html/2610.02391#A9)\)\. We do not include SEAL and FlowSteer, which rely on keyword\-labeled steps of long reasoning models\. We also do not include the method of[Sun et al\. \(2026\)](https://arxiv.org/html/2610.02391#bib.bib39), which needs correctness labels, or methods that modify the input or use several samples\. State\-dependent methods such as K\-Steering, ODESteer and CLAS learn from labels or contrastive examples \(Appendix[A](https://arxiv.org/html/2610.02391#A1)\), so Section[5](https://arxiv.org/html/2610.02391#S5)instead trains the probe ofHESTwithc→0c\\to 0\.
### 4\.2Main Results
For the development model \(Table[1](https://arxiv.org/html/2610.02391#S4.T1)\),HESTwith the Busemann readout improves MATH\-500 accuracy at every step length, by up to1\.801\.80points att=0\.5t=0\.5, andt=0\.5t=0\.5is also the best step length on GSM8K\. Averaged over the four step lengths, it still gains1\.101\.10points on MATH\-500 and0\.170\.17points on GSM8K\. The radius readout depends more on the step length on MATH\-500, where it gains1\.001\.00point att=0\.25t=0\.25and loses0\.800\.80points att=1t=1, and it does not improve GSM8K\. The Busemann readout does not improve OlympiadBench at any step length \(Section[4\.3](https://arxiv.org/html/2610.02391#S4.SS3)\)\.
Table[2](https://arxiv.org/html/2610.02391#S4.T2)applies the same configuration to the other models, with the threshold calibrated as in Section[3\.5](https://arxiv.org/html/2610.02391#S3.SS5)and the step lengths unchanged\. With the Busemann readout,HESTimproves greedy accuracy in five of the six model and dataset pairs, by0\.230\.23to1\.801\.80points, including three of the four pairs whose hyperparameters involve no test labels\. The exception is Llama\-3\.1\-8B\-Instruct on GSM8K, with1818problems fixed and2222broken\. Over all six pairs,HESTfixes8888problems and breaks6363\(Appendix[H](https://arxiv.org/html/2610.02391#A8)gives all counts and paired sign tests\)\. The radius readout gives smaller changes, between−0\.30\-0\.30and\+1\.00\+1\.00points\. CAA lowers accuracy in all six settings, by0\.380\.38to12\.6012\.60points, although it uses the same data\. It changes9797to100%100\\%of the solutions, fixing205205problems and breaking369369\(Appendix[I](https://arxiv.org/html/2610.02391#A9)\)\.HEST, in contrast, steers a median of0\.80\.8to3\.0%3\.0\\%of the tokens of a solution \(Appendix[K](https://arxiv.org/html/2610.02391#A11)\), and each steered token costs one extra single\-token forward pass of the model, besides the probe, its Jacobian and at most fourn×nn\\times nsolves\. Section[5](https://arxiv.org/html/2610.02391#S5)varies the positions and the direction separately at a matched edit size\.
Table 2:Accuracy \(%\) and change from greedy decoding \(points\)\.θ\\theta: gate threshold\. Steered: median percentage of tokens steered per solution \(Busemann\)\. Bold: best accuracy per row\.
### 4\.3Analysis of the Main Results
#### Hesitation groups\.
To see which problems benefit, Figure[2](https://arxiv.org/html/2610.02391#S4.F2)divides them by the number of hesitation points in their greedy solution, which is fixed before any intervention, into groups with00to1010,1111to2424and at least2525hesitation points\. Letggbe the greedy accuracy of a group,ffthe fraction of its greedy\-wrong problems thatHESTfixes, andbbthe fraction of its greedy\-right problems thatHESTbreaks\. The change in accuracy, in points, is thenΔ=100\[\(1−g\)f−gb\]\\Delta=100\\,\[\(1\-g\)\\,f\-g\\,b\]\. The figure showsggandΔ\\Deltafor every group, and Appendix[J](https://arxiv.org/html/2610.02391#A10)listsffandbb\.
Figure 2:Accuracy changes little on most problems, andHESTgains most where the greedy solution hesitates often\. Problems are grouped by the number of hesitation points in the greedy solution\. Left: greedy accuracy\. Right: change in accuracy withHEST, Busemann readout,t=0\.5t=0\.5\.First, most problems hesitate little and are already solved\. In all six settings, the00–1010group contains7171to81%81\\%of the problems, and in five of the six settings greedy decoding solves8989to98%98\\%of them\.HESTbreaks almost nothing in this group \(b≤0\.03b\\leq 0\.03\), and accuracy changes by−0\.28\-0\.28to\+0\.84\+0\.84points\. Second, the gain per problem tends to grow with hesitation\. In the group with at least2525hesitation points,HESTgains2\.942\.94to5\.065\.06points in five of the six settings, but this group is small\.
#### Dense hesitation\.
The two largest losses, on OlympiadBench and for Llama\-3\.1\-8B\-Instruct on GSM8K, occur in groups where greedy decoding solves only about a third of the problems\. On OlympiadBench,34%34\\%of the problems have at least2525hesitation points, against9%9\\%on MATH\-500, andHESTsteers2020tokens per solution on average, against99\(Appendix[K](https://arxiv.org/html/2610.02391#A11)\)\. More edits give more chances to change a correct derivation, and in the middle group the Busemann readout breaks22%22\\%of the solved problems\. The radius readout breaks8%8\\%and gains up to1\.221\.22points at larger steps \(Table[1](https://arxiv.org/html/2610.02391#S4.T1)\), so the setting chosen on MATH\-500 and GSM8K does not carry over to dense hesitation\.
Problem\(MATH\-500, geometry, level 3\)\. In△RST\\triangle RST, shown,sinR=25\\sin R=\\frac\{2\}\{5\}\. What issinT\\sin T?\[Diagram: right angle atSS;RT=5RT=5\.\] Common prefix\.… Given: …△RST\\triangle RSTis a right triangle with∠\\angle?first hesitation point,H=1\.19\>θH=1\.19\>\\theta
Greedy decoding∠\\angleTT=90∘\\,=90^\{\\circ\}… “since∠T=90∘\\angle T=90^\{\\circ\}, we know thatsin90∘=1\\sin 90^\{\\circ\}=1” …1\\boxed\{1\}✗
HEST: one geodesic step at this token∠\\angleRSTRST=90∘\\,=90^\{\\circ\}… “∠R\\angle Rand∠T\\angle Tare complementary” …sinT=cosR=215\\sin T=\\cos R=\\boxed\{\\tfrac\{\\sqrt\{21\}\}\{5\}\}✓
Figure 3:One steered token changes the course of the solution\. Top: next\-token entropy along the greedy solution, in which33of189189tokens exceedθ=1\\theta=1\. Bottom: the two continuations from the first of them\.HESTsteers1212of the583583tokens of its own solution\. Appendix[P](https://arxiv.org/html/2610.02391#A16)gives both solutions\.
#### An example\.
Figure[3](https://arxiv.org/html/2610.02391#S4.F3)follows the MATH\-500 problem with the shortest greedy solution among those thatHESTfixes for the development model\. The first of its three hesitation points is where the model states which angle is the right angle\. Greedy decoding writes∠T\\angle T, contrary to the diagram, and concludessinT=1\\sin T=1\. After one geodesic step at this token, the model writes∠RST\\angle RSTand eventually reaches21/5\\sqrt\{21\}/5\. In most other fixed problems, the first changed token is a choice of wording\.
## 5Ablation Study
On Qwen2\.5\-Math\-1\.5B\-Instruct, we ablate where to steer, in which direction and in which geometry\. Each variant ofHESTchanges a single component and keeps layer1919, the fallback and, unless stated otherwise, the Busemann readout \(Table[3](https://arxiv.org/html/2610.02391#S5.T3)\)\. Appendix[M](https://arxiv.org/html/2610.02391#A13)also varies the threshold and the layer, which were fixed before the ablation, and neither improves onθ=1\\theta=1and layer1919\.
Table 3:Ablations on Qwen2\.5\-Math\-1\.5B\-Instruct withθ=1\\theta=1and layer1919, in accuracy \(%\) and change from greedy decoding \(points\)\. \(a\) Where to steer, on MATH\-500 with the Busemann readout,t=0\.5t=0\.5\. Steered: median fraction of steered tokens\. Step: median∥Δh∥/∥h∥\\lVert\\Delta h\\rVert/\\lVert h\\rVert\. CAA has no fallback\. \(b\) Which direction, with the gate and step size ofHEST\. \(c\) Which geometry, where the Euclidean probe hasc→0c\\to 0and the same median edit size\. Bold marks the best of each comparison\.
#### Where to steer\.
Two variants apply a step of the same size along the probe direction at other positions\.*Every token*edits all generated positions, and*step boundaries*edits the tokens that contain a double line break, where the reasoning steps of SEAL begin\. Neither variant improves accuracy on MATH\-500 \(Table[3](https://arxiv.org/html/2610.02391#S5.T3)a\), and steering step boundaries lowers it by0\.400\.40points\. It edits about as many tokens asHEST, but only8\.2%8\.2\\%of the hesitation points fall at the start of a step\. CAA adds a vector about five times larger than theHESTstep at every token and lowers accuracy by4\.204\.20points\. A small step at every token thus stays close to neutral, and only edits at hesitation points yield a gain\.
#### Which direction\.
Two further variants keep the gate and the step size and change only the direction\.*Fixed direction*uses the unit CAA direction, scaled to the median size of theHESTedit, and*reverse direction*steps away from the ideal point \(Table[3](https://arxiv.org/html/2610.02391#S5.T3)b\)\. The reverse direction gains only0\.400\.40and0\.150\.15points over greedy decoding, and the fixed direction matchesHESTon MATH\-500 but gains only0\.080\.08points on GSM8K, against0\.530\.53\. With the gate and the small step, these three directions break at most4\.4%4\.4\\%of the changed solutions that were correct\. The direction decides how many wrong solutions are fixed, andHESTfixes the largest fraction of the changed wrong solutions on both benchmarks,15%15\\%on MATH\-500 and20%20\\%on GSM8K\. A fixed direction is thus competitive under the gate and the step size ofHEST, but not when CAA adds it at every token\.
#### Which geometry\.
The Euclidean probe, the limitc→0c\\to 0of the probe ofHEST, omits the exponential map and is trained in the same way\. It predicts entropy at least as well as the hyperbolic probe\. Its step length matches the median edit size ofHESTon training states \(Appendix[N](https://arxiv.org/html/2610.02391#A14)\)\. With the Busemann readout, the Euclidean probe lowers accuracy by0\.800\.80points on MATH\-500 and by0\.150\.15points on GSM8K, whereasHESTgains1\.801\.80and0\.530\.53points \(Table[3](https://arxiv.org/html/2610.02391#S5.T3)c\) and improves MATH\-500 at every step length of Table[1](https://arxiv.org/html/2610.02391#S4.T1)\. Over the two benchmarks, onlyHESTis right on4848problems, and only the Euclidean probe on2626\. It fixes a smaller fraction of the changed wrong solutions,11%11\\%against15%15\\%on MATH\-500 and14%14\\%against20%20\\%on GSM8K, and on MATH\-500 it breaks9%9\\%of the changed correct ones, against4%4\\%\. With the radius readout, whose step points to the origin in both geometries, the two probes differ by0\.800\.80points on MATH\-500 and match on GSM8K\. The curvature thus matters most for the Busemann readout, as Section[3\.2](https://arxiv.org/html/2610.02391#S3.SS2)predicts\.
## 6Limitations and Conclusion
#### Limitations\.
The gains ofHESTare bounded by how many problems can still be fixed, and near the ceiling of greedy decoding, as for Qwen2\.5\-Math\-7B\-Instruct on GSM8K, there is little room to improve\. When solutions hesitate densely, as on many OlympiadBench problems, correct answers are fragile andHESTcan break them\. Other decoding methods, models with more than88B parameters and tasks beyond mathematical reasoning remain to be tested\.
#### Conclusion\.
HESTsteers a language model with hyperbolic geometry, using the model’s own next\-token entropy as its only label\. The entropy selects the few tokens at which the model hesitates, and a probe trained on the same signal gives the direction through the Riemannian gradient of its readout\. Steering only these tokens improves greedy accuracy on mathematical reasoning in most settings, whereas a fixed vector added at every token lowers it\. On the development model, the gain disappears with a Euclidean probe that ranks entropy at least as well, which suggests that it depends on the hyperbolic geometry\. The code will be made publicly available\.
## Acknowledgments
This work was supported by the DOE SEA\-CROGS project \(DE\-SC0023191\) and the AFOSR project \(FA9550\-24\-1\-0231\)\.
## References
- Atkinson \(2026\)Craig Atkinson\.Tethered reasoning: Decoupling entropy from hallucination in quantized LLMs via manifold steering\.*arXiv preprint arXiv:2602\.17691*, 2026\.
- Bridson and Haefliger \(2013\)Martin R Bridson and André Haefliger\.*Metric spaces of non\-positive curvature*\.Springer Science & Business Media, 2013\.
- Chen et al\. \(2025\)Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang\.SEAL: Steerable reasoning calibration of large language models for free\.In*Second Conference on Language Modeling*, 2025\.URL[https://openreview\.net/forum?id=klPszYDIRT](https://openreview.net/forum?id=klPszYDIRT)\.
- Chen et al\. \(2026\)Ziheng Chen, Bernhard Schölkopf, and Nicu Sebe\.Hyperbolic Busemann neural networks\.In*CVPR*, 2026\.
- Cheng et al\. \(2026\)Zifeng Cheng, Jinwei Gan, Zhiwei Jiang, Cong Wang, Yafeng Yin, Xiang Luo, Yuchen Fu, and Qing Gu\.Steering when necessary: Flexible steering large language models with backtracking\.*Advances in Neural Information Processing Systems*, 38:8390–8413, 2026\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Ganea et al\. \(2018\)Octavian Ganea, Gary Bécigneul, and Thomas Hofmann\.Hyperbolic neural networks\.*Advances in Neural Information Processing Systems*, 31, 2018\.
- Ghadimi Atigh et al\. \(2021\)Mina Ghadimi Atigh, Martin Keller\-Ressel, and Pascal Mettes\.Hyperbolic Busemann learning with ideal prototypes\.*Advances in Neural Information Processing Systems*, 34:103–115, 2021\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- He et al\. \(2024\)Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al\.OlympiadBench: A challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3828–3850, 2024\.
- He et al\. \(2026\)Neil He, Rishabh Anand, Hiren Madhu, Ali Maatouk, Smita Krishnaswamy, Leandros Tassiulas, Menglin Yang, and Rex Ying\.HELM: Hyperbolic large language models via mixture\-of\-curvature experts\.*Advances in Neural Information Processing Systems*, 38:142604–142635, 2026\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\.Measuring mathematical problem solving with the MATH dataset\.*arXiv preprint arXiv:2103\.03874*, 2021\.
- Herbrich et al\. \(2000\)Ralf Herbrich, Thore Graepel, and Klaus Obermayer\.Large margin rank boundaries for ordinal regression\.In Alexander J\. Smola, Peter Bartlett, Bernhard Schölkopf, and Dale Schuurmans, editors,*Advances in Large Margin Classifiers*, pages 115–132\. MIT Press, 2000\.doi:10\.7551/mitpress/1113\.003\.0010\.
- Hsu et al\. \(2026\)Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan, and Mikhail Belkin\.Contextual linear activation steering of language models\.*arXiv preprint arXiv:2604\.24693*, 2026\.
- Jin et al\. \(2026\)Zehao Jin, Ruixuan Deng, Junran Wang, Xinjie Shen, and Chao Zhang\.Beyond steering vector: Flow\-based activation steering for inference\-time intervention\.In*Mechanistic Interpretability Workshop at ICML 2026*, 2026\.URL[https://openreview\.net/forum?id=t7M7vjIJbe](https://openreview.net/forum?id=t7M7vjIJbe)\.
- Khanh et al\. \(2026\)Ly Tran Ho Khanh, Dongxuan Zhu, Man\-Chung Yue, and Viet Anh Nguyen\.Test\-time diverse reasoning by Riemannian activation steering\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pages 31429–31437, 2026\.
- Kim et al\. \(2026\)Heejun Kim, Seungpil Lee, Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Taesup Kim, and Sundong Kim\.From noise to diversity: Random embedding injection in LLM reasoning\.*arXiv preprint arXiv:2605\.11936*, 2026\.
- Kydlíček \(2025\)Hynek Kydlíček\.Math\-Verify: Math verification library, 2025\.URL[https://github\.com/huggingface/math\-verify](https://github.com/huggingface/math-verify)\.
- Li et al\. \(2025\)Hengli Li, Chenxi Li, Tong Wu, Xuekai Zhu, Yuxuan Wang, Zhaoxin Yu, Eric Hanchen Jiang, Song\-Chun Zhu, Zixia Jia, Ying Nian Wu, et al\.Seek in the dark: Reasoning via test\-time instance\-level policy gradient in latent space\.*arXiv preprint arXiv:2505\.13308*, 2025\.
- Li et al\. \(2023\)Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg\.Inference\-time intervention: Eliciting truthful answers from a language model\.*Advances in Neural Information Processing Systems*, 36:41451–41530, 2023\.
- Li et al\. \(2026a\)Xianzhi Li, Ethan Callanan, Abdellah Ghassel, and Xiaodan Zhu\.Entropy\-gated branching for efficient test\-time reasoning\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5054–5069, 2026a\.
- Li et al\. \(2026b\)Yawei Li, Benjamin Bergner, Yinghan Zhao, Vihang Prakash Patil, Bei Chen, and Cheng Wang\.Steering large reasoning models towards concise reasoning via flow matching\.*Transactions on Machine Learning Research*, 2026b\.URL[https://openreview\.net/forum?id=qwcJMdGerK](https://openreview.net/forum?id=qwcJMdGerK)\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.In*International Conference on Learning Representations*, volume 2024, pages 39578–39601, 2024\.
- Liu et al\. \(2025\)Sheng Liu, Tianlang Chen, Pan Lu, Haotian Ye, Yizheng Chen, Lei Xing, and James Zou\.Fractional reasoning via latent steering vectors improves inference time compute\.*arXiv preprint arXiv:2506\.15882*, 2025\.
- Liu et al\. \(2026\)Yuyu Liu, Haotian Xu, Yanan He, Sarang Rajendra Patil, Mengjia Xu, and Tengfei Ma\.HyperGuide: Hyperbolic guidance for efficient multi\-step reasoning in large language models\.*arXiv preprint arXiv:2605\.24140*, 2026\.
- Nguyen et al\. \(2025\)Duy Nguyen, Archiki Prasad, Elias Stengel\-Eskin, and Mohit Bansal\.Multi\-attribute steering of language models via targeted intervention\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 20619–20634, 2025\.
- Nickel and Kiela \(2017\)Maximillian Nickel and Douwe Kiela\.Poincaré embeddings for learning hierarchical representations\.*Advances in Neural Information Processing Systems*, 30, 2017\.
- Oozeer et al\. \(2025\)Narmeen Fatimah Oozeer, Luke Marks, Fazl Barez, and Amir Abdullah\.Beyond linear steering: Unified multi\-attribute control for language models\.In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 23513–23557, Suzhou, China, 2025\. Association for Computational Linguistics\.doi:10\.18653/v1/2025\.findings\-emnlp\.1278\.URL[https://aclanthology\.org/2025\.findings\-emnlp\.1278/](https://aclanthology.org/2025.findings-emnlp.1278/)\.
- Park et al\. \(2025\)Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch\.The geometry of categorical and hierarchical concepts in large language models\.In*International Conference on Learning Representations*, volume 2025, pages 76441–76463, 2025\.
- Patil et al\. \(2026a\)Sarang Patil, Zeyong Zhang, Yiran Huang, Tengfei Ma, and Mengjia Xu\.Hyperbolic large language models\.*SIAM Review \(To appear\)*, 2026a\.
- Patil et al\. \(2026b\)Sarang Rajendra Patil, Ashish Parmanand Pandey, Ioannis Koutis, and Mengjia Xu\.Hierarchical Mamba meets hyperbolic geometry: A new paradigm for structured language embeddings\.*Transactions on Machine Learning Research*, 2026b\.URL[https://openreview\.net/forum?id=a3g13FKzct](https://openreview.net/forum?id=a3g13FKzct)\.
- Raval et al\. \(2026\)Shivam Raval, Hae Jin Song, Linlin Wu, Abir Harrasse, Jeff M Phillips, Fazl Barez, and Amirali Abdullah\.Curveball steering: The right direction to steer isn’t always linear\.*arXiv preprint arXiv:2603\.09313*, 2026\.
- Rimsky et al\. \(2024\)Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner\.Steering Llama 2 via contrastive activation addition\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15504–15522, 2024\.
- Rodriguez et al\. \(2025\)Pau Rodriguez, Arno Blaas, Michal Klein, Luca Zappella, Nicholas Apostoloff, Xavier Suau, et al\.Controlling language and diffusion models by transporting activations\.In*International Conference on Learning Representations*, volume 2025, pages 89812–89855, 2025\.
- Sala et al\. \(2018\)Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré\.Representation tradeoffs for hyperbolic embeddings\.In*International Conference on Machine Learning*, pages 4460–4469\. PMLR, 2018\.
- Schroff et al\. \(2015\)Florian Schroff, Dmitry Kalenichenko, and James Philbin\.FaceNet: A unified embedding for face recognition and clustering\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 815–823, 2015\.
- Storaï and Hwang \(2025\)Romain Storaï and Seung\-won Hwang\.HARP: Hesitation\-aware reframing in transformer inference pass\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 12305–12319, 2025\.
- Sun et al\. \(2025\)Jiuding Sun, Sidharth Baskaran, Zhengxuan Wu, Michael Sklar, Christopher Potts, and Atticus Geiger\.HyperSteer: Activation steering at scale with hypernetworks\.*arXiv preprint arXiv:2506\.03292*, 2025\.
- Sun et al\. \(2026\)Lihao Sun, Hang Dong, Bo Qiao, Qingwei Lin, Dongmei Zhang, and Saravan Rajmohan\.LLM reasoning as trajectories: Step\-specific representation geometry and correctness signals\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 26872–26887, 2026\.
- Turner et al\. \(2023\)Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid\.Steering language models with activation engineering\.*arXiv preprint arXiv:2308\.10248*, 2023\.
- Vu and Nguyen \(2026\)Minh Hieu Vu and Tan Nguyen\.Angular steering: Behavior control via rotation in activation space\.*Advances in Neural Information Processing Systems*, 38:121653–121690, 2026\.
- Wang et al\. \(2026\)Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong\-Hui Chen, Jianxin Yang, Zhenru Zhang, et al\.Beyond the 80/20 rule: High\-entropy minority tokens drive effective reinforcement learning for LLM reasoning\.*Advances in Neural Information Processing Systems*, 38:115452–115486, 2026\.
- Weinberger and Saul \(2009\)Kilian Q Weinberger and Lawrence K Saul\.Distance metric learning for large margin nearest neighbor classification\.*Journal of Machine Learning Research*, 10\(2\), 2009\.
- Wu et al\. \(2024\)Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts\.ReFT: Representation finetuning for language models\.*Advances in Neural Information Processing Systems*, 37:63908–63962, 2024\.
- Yang et al\. \(2024a\)An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al\.Qwen2\.5\-Math technical report: Toward mathematical expert model via self\-improvement\.*arXiv preprint arXiv:2409\.12122*, 2024a\.
- Yang et al\. \(2024b\)Menglin Yang, Harshit Verma, Delvin Ce Zhang, Jiahong Liu, Irwin King, and Rex Ying\.Hypformer: Exploring efficient transformer fully in hyperbolic space\.In*Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 3770–3781, 2024b\.
- Yang et al\. \(2026\)Menglin Yang, Aosong Feng, Bo Xiong, Jiahong Liu, Irwin King, Rex Ying, et al\.Hyperbolic fine\-tuning for large language models\.*Advances in Neural Information Processing Systems*, 38:29613–29648, 2026\.
- You et al\. \(2026\)Zejia You, Chunyuan Deng, and Hanjie Chen\.Spherical steering: Geometry\-aware activation rotation for language models\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=9PPeJsVLdB](https://openreview.net/forum?id=9PPeJsVLdB)\.
- Zhao et al\. \(2026\)Hongjue Zhao, Haosen Sun, Jiangtao Kong, Xiaochang Li, Qineng Wang, Liwei Jiang, Qi Zhu, Tarek F\. Abdelzaher, Yejin Choi, Manling Li, and Huajie Shao\.ODESteer: A unified ODE\-based steering framework for LLM alignment\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=CFewUmgIIL](https://openreview.net/forum?id=CFewUmgIIL)\.
- Zou et al\. \(2023\)Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann\-Kathrin Dombrowski, et al\.Representation engineering: A top\-down approach to AI transparency\.*arXiv preprint arXiv:2310\.01405*, 2023\.
- Zur et al\. \(2025\)Amir Zur, Atticus Geiger, Ekdeep Singh Lubana, and Eric Bigelow\.Are language models aware of the road not taken? Token\-level uncertainty and hidden state dynamics\.*arXiv preprint arXiv:2511\.04527*, 2025\.
Appendix
## Appendix AComparison with Related Steering Methods
Table[4](https://arxiv.org/html/2610.02391#A1.T4)comparesHESTwith the steering methods discussed in Section[2](https://arxiv.org/html/2610.02391#S2)along the design choices studied in this paper\. Most methods edit every token or fixed positions of the prompt, and MAT\-Steer weights its edits of the prompt tokens with a learned gate\. The few methods that restrict the positions during generation use a confidence score computed from a contrastive direction, a probe trained on labels, step boundaries, or a score that combines the next\-token entropy with a distance to a reference distribution\. Helix uses the last of these scores to lower the largest logit and does not edit the hidden state\.HESTrestricts the positions with the entropy alone, and the same entropy trains the probe that gives the direction\. Most methods also derive their direction from contrastive or labeled data\. Among the methods that report accuracy on mathematical benchmarks, SEAL and FlowSteer target long reasoning models, and the method of[Sun et al\. \(2026\)](https://arxiv.org/html/2610.02391#bib.bib39)needs correctness labels\. CLAS uses GSM8K to elicit step\-by\-step solutions, ReFT trains its interventions on the task, Helix evaluates a quantized model at high sampling temperatures, and Fractional Reasoning and SPREAD report accuracy over several samples\.
Table 4:Steering methods compared along the design choices ofHEST\.*Positions*: the tokens at which the edit is applied\.*Math*: whether accuracy on mathematical benchmarks is reported, and in which setting\. Each entry was checked against the original paper\.MethodDirectionPositionsTraining signalGeometryEvaluationMathCAA\([Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33)\)fixed, difference of meansall tokens after the promptcontrastive pairsEuclideanbehavioral questions, open\-ended generation, MMLU, TruthfulQA–ITI\([Li et al\., 2023](https://arxiv.org/html/2610.02391#bib.bib20)\)fixed, top attention headsevery generated tokenlabeled truthful and untruthful answersEuclideanTruthfulQA, NQ, TriviaQA, MMLU–CLAS\([Hsu et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib14)\)per\-layer direction, coefficient linear in the activationall tokensprobe direction, prompt and completion pairsEuclidean1111tasks including GSM8KGSM8K, to elicit step\-by\-step solutionsReFT\([Wu et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib44)\)learned low\-rank edit of a subspacefixed prefix and suffix positions of the prompttask supervision, frozen modelEuclideancommonsense and arithmetic reasoning, instruction following, GLUEGSM8K and three others, after task finetuningHyperSteer\([Sun et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib38)\)generated by a hypernetwork from the steering prompt and the inputall tokensAxBench steering prompts, language\-modeling lossEuclideanAxBench concepts–AcT\([Rodriguez et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib34)\)coordinate\-wise affine maps from optimal transportall tokenssource and target sentence setsEuclideantoxicity, concept induction, TruthfulQA, text\-to\-image models–ODESteer\([Zhao et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib49)\)classifier gradient, integrated as an ODEall generated tokenspositive and negative examplesEuclideanUltraFeedback, TruthfulQA, RealToxicityPrompts–FLAS\([Jin et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib15)\)concept\-conditioned velocity fieldall tokenspositive examples, language\-modeling lossEuclideanAxBench concepts–Curveball\([Raval et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib32)\)fixed direction in kernel PCA spaceall generated tokenscontrastive pairskernel feature spacebehavioral choices, tone traits–K\-Steering\([Oozeer et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib28)\)classifier gradientall tokensattribute labelsEuclideantone and debate style, MMLU as a capability check–Angular\([Vu and Nguyen, 2026](https://arxiv.org/html/2610.02391#bib.bib41)\)rotation in a fixed planeall tokens; adaptive variant where aligned with the featurecontrastive prompt setssphererefusal, emotiontinyGSM8K, as a capability checkSpherical\([You et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib48)\)geodesic rotation towards a fixed directioneach token, strength set by a confidence gatecontrastive answer pairssphereTruthfulQA and five multiple\-choice benchmarks–MAT\-Steer\([Nguyen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib26)\)fixed per attributelearned soft gate per prompt tokenlabeled positive and negative samplesEuclidean, norm preservingTruthfulQA, Toxigen, BBQ, HelpSteer–FASB\([Cheng et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib5)\)fixed per headprobe\-triggered, with backtrackinglabeled probeEuclideanTruthfulQA, six multiple\-choice tasks–Helix\([Atkinson, 2026](https://arxiv.org/html/2610.02391#bib.bib1)\)penalty on the largest logit, no edit of the hidden statetokens with low confidence from entropy and Mahalanobis distanceactivations of low\-temperature generations on factual promptsEuclidean, MahalanobisGSM8K, HumanEval, MMLU, creative ideationGSM8K, 4\-bit model, temperatures0\.50\.5to3\.03\.0SEAL\([Chen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib3)\)fixedstep boundariessteps classified by keywordsEuclideanMATH\-500, GSM8K, LiveCodeBenchlong reasoning modelsFlowSteer\([Li et al\., 2026b](https://arxiv.org/html/2610.02391#bib.bib22)\)flow matchingstep boundariesas SEALEuclideanMATH\-500, GSM8K, AIME, AMC, LiveCodeBenchlong reasoning models[Sun et al\. \(2026\)](https://arxiv.org/html/2610.02391#bib.bib39)fixed, or towards a centroidstep boundaries or before the answer, gatedcorrectness labelsEuclideanGSM8K, MATH\-500, MMLULlama\-3\.1\-8B\-InstructFractional Reasoning\([Liu et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib24)\)fixed, scaled, norm preservedall tokenscontrastive instruction promptsEuclideanGSM8K, MATH\-500, GPQAwith voting or best\-of\-NNSPREAD\([Khanh et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib16)\)optimized jointly forNNsamplestokens after each synchronization anchornoneEuclidean, fixed vector normsAIME, MATH\-500, OlympiadBenchPass@NNHEST\(ours\)Riemannian gradient of a probe readouthesitation pointsthe model’s own entropyhyperbolicMATH\-500, GSM8K, OlympiadBenchthree instruction\-tuned models#### State\-dependent methods as baselines\.
We do not run the state\-dependent methods of Table[4](https://arxiv.org/html/2610.02391#A1.T4)as baselines\. K\-Steering, ODESteer, FLAS and CLAS learn their edits from attribute labels, contrastive examples or positive examples of a target behavior, and hesitation has no such data apart from the entropy split\. Among them, only CLAS reports results on GSM8K, where it elicits step\-by\-step solutions from prompts that ask only for the answer\. Retraining each of them on the entropy split would change the methods themselves\. We therefore build the closest state\-dependent baseline from our own pipeline\. The Euclidean probe of Appendix[N](https://arxiv.org/html/2610.02391#A14)is trained on the same entropy and lowers a nonlinear readout of the hidden state, much as K\-Steering and ODESteer follow the gradient of a nonlinear classifier, and it differs fromHESTonly in the curvature\. CAA plays the same role for a fixed direction, since we build it from the same entropy split\.
## Appendix BHyperbolic Geometry and Proof of Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)
Throughout,c\>0c\>0is the curvature magnitude, so that the sectional curvature is−c\-c, andnnis the dimension of the ball\.
### B\.1The Poincaré Ball
The Poincaré ball model is
𝔻cn=\{z∈ℝn:c∥z∥2<1\},gz=\(λzc\)2gE,λzc=21−c∥z∥2,\\mathbb\{D\}^\{n\}\_\{c\}=\\big\\\{z\\in\\mathbb\{R\}^\{n\}:c\\lVert z\\rVert^\{2\}<1\\big\\\},\\qquad g\_\{z\}=\(\\lambda^\{c\}\_\{z\}\)^\{2\}\\,g^\{E\},\\qquad\\lambda^\{c\}\_\{z\}=\\frac\{2\}\{1\-c\\lVert z\\rVert^\{2\}\},\(8\)wheregEg^\{E\}is the Euclidean metric\. The metric is a positive multiple of the Euclidean one at every point, so the model is conformal: angles inggand ingEg^\{E\}agree\. The boundary sphere is at infinite hyperbolic distance from every interior point\. In the gyrovector formulation of[Ganea et al\. \(2018\)](https://arxiv.org/html/2610.02391#bib.bib7), the Möbius addition ofx,y∈𝔻cnx,y\\in\\mathbb\{D\}^\{n\}\_\{c\}is
x⊕cy=\(1\+2c⟨x,y⟩\+c∥y∥2\)x\+\(1−c∥x∥2\)y1\+2c⟨x,y⟩\+c2∥x∥2∥y∥2,x\\oplus\_\{c\}y=\\frac\{\\big\(1\+2c\\langle x,y\\rangle\+c\\lVert y\\rVert^\{2\}\\big\)\\,x\+\\big\(1\-c\\lVert x\\rVert^\{2\}\\big\)\\,y\}\{1\+2c\\langle x,y\\rangle\+c^\{2\}\\lVert x\\rVert^\{2\}\\lVert y\\rVert^\{2\}\},\(9\)and the distance is
dc\(x,y\)=2cartanh\(c∥\(−x\)⊕cy∥\)\.d\_\{c\}\(x,y\)=\\frac\{2\}\{\\sqrt\{c\}\}\\operatorname\{artanh\}\\\!\\big\(\\sqrt\{c\}\\,\\lVert\(\-x\)\\oplus\_\{c\}y\\rVert\\big\)\.\(10\)The exponential map atxxfollows the geodesic that leavesxxwith initial velocityv∈Tx𝔻cnv\\in T\_\{x\}\\mathbb\{D\}^\{n\}\_\{c\}for unit time\. It is given by
expxc\(v\)=x⊕c\(tanh\(cλxc∥v∥2\)vc∥v∥\)\.\\exp^\{c\}\_\{x\}\(v\)=x\\oplus\_\{c\}\\left\(\\tanh\\\!\\Big\(\\frac\{\\sqrt\{c\}\\,\\lambda^\{c\}\_\{x\}\\lVert v\\rVert\}\{2\}\\Big\)\\frac\{v\}\{\\sqrt\{c\}\\,\\lVert v\\rVert\}\\right\)\.\(11\)At the originλ0c=2\\lambda^\{c\}\_\{0\}=2, and Eq\.[11](https://arxiv.org/html/2610.02391#A2.E11)reduces to the map used in the probe, Eq\.[2](https://arxiv.org/html/2610.02391#S3.E2)\. For a smooth functionffon the ball, the Riemannian gradient is∇ℛf\(z\)=\(λzc\)−2∇f\(z\)\\nabla^\{\\mathcal\{R\}\}f\(z\)=\(\\lambda^\{c\}\_\{z\}\)^\{\-2\}\\nabla f\(z\), and its length in the hyperbolic metric is∥∇ℛf\(z\)∥g=\(λzc\)−1∥∇f\(z\)∥\\lVert\\nabla^\{\\mathcal\{R\}\}f\(z\)\\rVert\_\{g\}=\(\\lambda^\{c\}\_\{z\}\)^\{\-1\}\\lVert\\nabla f\(z\)\\rVert\.
### B\.2Proof of Part \(i\)
Letuuhave unit Euclidean norm andw=tanh\(ct/2\)u/cw=\\tanh\(\\sqrt\{c\}\\,t/2\)\\,u/\\sqrt\{c\}, so thatz′=z⊕cwz^\{\\prime\}=z\\oplus\_\{c\}wandc∥w∥=tanh\(ct/2\)\\sqrt\{c\}\\,\\lVert w\\rVert=\\tanh\(\\sqrt\{c\}\\,t/2\)\. Möbius addition satisfies the left\-cancellation law\(−z\)⊕c\(z⊕cw\)=w\(\-z\)\\oplus\_\{c\}\(z\\oplus\_\{c\}w\)=w\([Ganea et al\., 2018](https://arxiv.org/html/2610.02391#bib.bib7)\)\. By Eq\.[10](https://arxiv.org/html/2610.02391#A2.E10),
dc\(z,z′\)=2cartanh\(c∥w∥\)=2cartanh\(tanh\(ct2\)\)=t\.d\_\{c\}\(z,z^\{\\prime\}\)=\\frac\{2\}\{\\sqrt\{c\}\}\\operatorname\{artanh\}\\\!\\big\(\\sqrt\{c\}\\,\\lVert w\\rVert\\big\)=\\frac\{2\}\{\\sqrt\{c\}\}\\operatorname\{artanh\}\\\!\\Big\(\\tanh\\big\(\\tfrac\{\\sqrt\{c\}\\,t\}\{2\}\\big\)\\Big\)=t\.\(12\)Moreover,z′=expzc\(tv\)z^\{\\prime\}=\\exp^\{c\}\_\{z\}\(t\\,v\)withv=u/λzcv=u/\\lambda^\{c\}\_\{z\}\. This vector has unit length in the hyperbolic metric, since∥v∥g=λzc∥v∥=1\\lVert v\\rVert\_\{g\}=\\lambda^\{c\}\_\{z\}\\lVert v\\rVert=1, and substitutingtvt\\,vinto Eq\.[11](https://arxiv.org/html/2610.02391#A2.E11)givesww\.□\\square
### B\.3Proof of Part \(ii\): the Radius Readout
The gradient ofr\(z\)=2cartanh\(c∥z∥\)r\(z\)=\\frac\{2\}\{\\sqrt\{c\}\}\\operatorname\{artanh\}\(\\sqrt\{c\}\\lVert z\\rVert\)is∇r\(z\)=λzcz/∥z∥\\nabla r\(z\)=\\lambda^\{c\}\_\{z\}\\,z/\\lVert z\\rVertforz≠0z\\neq 0, so∥∇ℛr∥g=1\\lVert\\nabla^\{\\mathcal\{R\}\}r\\rVert\_\{g\}=1and the steering direction isu=−z/∥z∥u=\-z/\\lVert z\\rVert\. Thenwwis a negative multiple ofzz\. For collinear vectorsx=ρ1e^x=\\rho\_\{1\}\\hat\{e\}andy=ρ2e^y=\\rho\_\{2\}\\hat\{e\}with∥e^∥=1\\lVert\\hat\{e\}\\rVert=1, Eq\.[9](https://arxiv.org/html/2610.02391#A2.E9)simplifies: the numerator is\(ρ1\+ρ2\)\(1\+cρ1ρ2\)e^\(\\rho\_\{1\}\+\\rho\_\{2\}\)\(1\+c\\rho\_\{1\}\\rho\_\{2\}\)\\,\\hat\{e\}and the denominator is\(1\+cρ1ρ2\)2\(1\+c\\rho\_\{1\}\\rho\_\{2\}\)^\{2\}, sox⊕cy=ρ1\+ρ21\+cρ1ρ2e^x\\oplus\_\{c\}y=\\frac\{\\rho\_\{1\}\+\\rho\_\{2\}\}\{1\+c\\rho\_\{1\}\\rho\_\{2\}\}\\,\\hat\{e\}\. Writingcρ1=tanhα\\sqrt\{c\}\\,\\rho\_\{1\}=\\tanh\\alphaandcρ2=tanhβ\\sqrt\{c\}\\,\\rho\_\{2\}=\\tanh\\betagivescρ1\+ρ21\+cρ1ρ2=tanh\(α\+β\)\\sqrt\{c\}\\,\\frac\{\\rho\_\{1\}\+\\rho\_\{2\}\}\{1\+c\\rho\_\{1\}\\rho\_\{2\}\}=\\tanh\(\\alpha\+\\beta\)\. Withe^=z/∥z∥\\hat\{e\}=z/\\lVert z\\rVert,α=cr\(z\)/2\\alpha=\\sqrt\{c\}\\,r\(z\)/2andβ=−ct/2\\beta=\-\\sqrt\{c\}\\,t/2, the new point isz′=tanh\(c\(r\(z\)−t\)/2\)e^/cz^\{\\prime\}=\\tanh\\big\(\\sqrt\{c\}\\,\(r\(z\)\-t\)/2\\big\)\\,\\hat\{e\}/\\sqrt\{c\}\. For0<t≤r\(z\)0<t\\leq r\(z\), this givesr\(z′\)=r\(z\)−tr\(z^\{\\prime\}\)=r\(z\)\-t\.□\\square
#### The conditiont≤r\(z\)t\\leq r\(z\)in our experiments\.
Part \(ii\) requires the step to be no longer than the radius readout of the state\. We evaluater\(z\)r\(z\)at every hesitation point of the greedy test solutions, using the state thatHESTedits when its gate fires\. These states coincide with the edited states at the first hesitation point of a solution and approximate them afterwards\. Over the117,032117\{,\}032states of the seven radius probes, the smallest value ofr\(z\)r\(z\)for each probe lies between1\.131\.13and1\.551\.55for Qwen2\.5\-Math\-1\.5B\-Instruct, whose largest step length ist=1t=1\. It lies between0\.880\.88and1\.191\.19for the other two models, which uset=0\.25t=0\.25\. None of these states therefore violates the condition, and every radius step taken at them ends before the origin\.
### B\.4Proof of Part \(iii\): the Busemann Readout
A Busemann function is a renormalized distance to an ideal point on the boundary at infinity\. For a geodesic rayγ\\gammawith unit speed that ends at this point, it isB\(y\)=lims→∞\(d\(y,γ\(s\)\)−s\)B\(y\)=\\lim\_\{s\\to\\infty\}\\big\(d\(y,\\gamma\(s\)\)\-s\\big\)\([Bridson and Haefliger, 2013](https://arxiv.org/html/2610.02391#bib.bib2), Chapter II\.8\)\. First takec=1c=1\. The functionBξ\(y\)=log\(∥ξ−y∥2/\(1−∥y∥2\)\)B\_\{\\xi\}\(y\)=\\log\\big\(\\lVert\\xi\-y\\rVert^\{2\}/\(1\-\\lVert y\\rVert^\{2\}\)\\big\)is the Busemann function of the unit Poincaré ball associated with the ideal pointξ\\xi, normalized so thatBξ\(0\)=0B\_\{\\xi\}\(0\)=0\([Ghadimi Atigh et al\., 2021](https://arxiv.org/html/2610.02391#bib.bib8);[Bridson and Haefliger, 2013](https://arxiv.org/html/2610.02391#bib.bib2)\)\. Its level sets are horospheres, and a short calculation shows that the level set\{Bξ=logk\}\\\{B\_\{\\xi\}=\\log k\\\}is the Euclidean sphere with centerξ/\(1\+k\)\\xi/\(1\+k\)and radiusk/\(1\+k\)k/\(1\+k\), which is tangent to the boundary atξ\\xi\.
The descent behavior ofBξB\_\{\\xi\}is easiest to see in the upper half\-space modelℍn=\{x∈ℝn:xn\>0\}\\mathbb\{H\}^\{n\}=\\\{x\\in\\mathbb\{R\}^\{n\}:x\_\{n\}\>0\\\}with metric∥dx∥2/xn2\\lVert dx\\rVert^\{2\}/x\_\{n\}^\{2\}\. A Möbius transformation ofℝn∪\{∞\}\\mathbb\{R\}^\{n\}\\cup\\\{\\infty\\\}maps the unit ball isometrically ontoℍn\\mathbb\{H\}^\{n\}and can be chosen to sendξ\\xito the point at infinity\. Busemann functions are defined by distances alone, so they are preserved by isometries, and the Busemann function of the point at infinity isB∞\(x\)=−logxnB\_\{\\infty\}\(x\)=\-\\log x\_\{n\}up to an additive constant, which follows directly from the definition of Busemann functions\([Bridson and Haefliger, 2013](https://arxiv.org/html/2610.02391#bib.bib2), Chapter II\.8\)\. Its Euclidean gradient is−en/xn\-e\_\{n\}/x\_\{n\}, so its Riemannian gradient isxn2\(−en/xn\)=−xnenx\_\{n\}^\{2\}\(\-e\_\{n\}/x\_\{n\}\)=\-x\_\{n\}e\_\{n\}, which has length11in the metric\. The direction of steepest descent is therefore\+en\+e\_\{n\}at every point, and the geodesic that leaves a point in this direction is the vertical line through it\. Along this line, hyperbolic arc length isds=dxn/xnds=dx\_\{n\}/x\_\{n\}, so moving a hyperbolic distanceτ\\taumultipliesxnx\_\{n\}byeτe^\{\\tau\}and lowersB∞B\_\{\\infty\}by exactlyτ\\tau\. Transferring back to the ball, one step of hyperbolic lengthτ\\taualong the steepest\-descent geodesic ofBξB\_\{\\xi\}lowersBξB\_\{\\xi\}by exactlyτ\\tau\.
For generalcc, the dilationz↦czz\\mapsto\\sqrt\{c\}\\,zsends𝔻cn\\mathbb\{D\}^\{n\}\_\{c\}onto the unit ball\. Sinceλcz1=λzc\\lambda^\{1\}\_\{\\sqrt\{c\}z\}=\\lambda^\{c\}\_\{z\}, the metric of the unit ball pulls back tocctimes the metric of𝔻cn\\mathbb\{D\}^\{n\}\_\{c\}\. The dilation therefore multiplies all distances byc\\sqrt\{c\}and maps geodesics to geodesics\. For the readoutb\(z\)=Bξ\(cz\)b\(z\)=B\_\{\\xi\}\(\\sqrt\{c\}\\,z\), it sends the steepest\-descent geodesic ofbbatzzto that ofBξB\_\{\\xi\}atcz\\sqrt\{c\}\\,z\. A step of lengthttin𝔻cn\\mathbb\{D\}^\{n\}\_\{c\}becomes a step of lengthct\\sqrt\{c\}\\,tin the unit ball\. By the previous paragraph,b\(z′\)=b\(z\)−ctb\(z^\{\\prime\}\)=b\(z\)\-\\sqrt\{c\}\\,t\.□\\square
### B\.5Remarks on the Direction
#### The metric does not change the initial direction\.
Because the metric is conformal, the Riemannian gradient is a positive multiple of the Euclidean gradient, and after normalization both give the sameuu\. The geometry enters the method in two other places\. The step follows a geodesic of𝔻cn\\mathbb\{D\}^\{n\}\_\{c\}and covers a fixed hyperbolic distance \(Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)\)\. The embedding itself is also shaped by the geometry, because the probe is trained with the hyperbolic distance in Eq\.[5](https://arxiv.org/html/2610.02391#S3.E5)and with a readout defined on the ball\.
#### The Busemann direction depends on the state\.
The probe is defined in the coordinates of the ball, and the step is mapped back to the hidden state from these coordinates\. In the ball, the steepest\-descent geodesics ofBξB\_\{\\xi\}are the diameter throughξ\\xiand circular arcs that meet the boundary orthogonally atξ\\xi, so the directionuudiffers from state to state\. This dependence is not an artifact of the coordinates\. In the upper half\-space model, the descent directions look parallel, but the unit fieldV=xn∂nV=x\_\{n\}\\partial\_\{n\}is not parallel in the hyperbolic sense\. Forn=2n=2, the Christoffel symbols of the metric giveD∂1V=−∂1≠0D\_\{\\partial\_\{1\}\}V=\-\\partial\_\{1\}\\neq 0, whereDDis the Levi\-Civita connection\. Parallel transport along a horosphere therefore does not preserve the direction of steepest descent\. The Euclidean counterpart ofbb, a linear function, has a constant gradient, which is a parallel field\.
## Appendix CClosed\-Form Jacobian of the Probe
The update of Eq\.[7](https://arxiv.org/html/2610.02391#S3.E7)needsJ=∂E/∂hJ=\\partial E/\\partial hat every iteration\. BecauseEEis a fixed composition,JJhas a closed form\. Write the forward pass of Eq\.[2](https://arxiv.org/html/2610.02391#S3.E2)as
x=h−μσ,a=W1x\+b1,m=a⊙Φ\(a\),q=W2m\+b2ω,z=exp0c\(q\),x=\\frac\{h\-\\mu\}\{\\sigma\},\\quad a=W\_\{1\}x\+b\_\{1\},\\quad m=a\\odot\\Phi\(a\),\\quad q=\\frac\{W\_\{2\}m\+b\_\{2\}\}\{\\omega\},\\quad z=\\exp^\{c\}\_\{0\}\(q\),\(13\)whereΦ\\Phiis the standard normal distribution function, so thatm=GELU\(a\)m=\\mathrm\{GELU\}\(a\)in its exact form\. WithΦ′\\Phi^\{\\prime\}the standard normal density,∂mk/∂ak=Φ\(ak\)\+akΦ′\(ak\)\\partial m\_\{k\}/\\partial a\_\{k\}=\\Phi\(a\_\{k\}\)\+a\_\{k\}\\Phi^\{\\prime\}\(a\_\{k\}\)and
∂q∂h=ω−1W2diag\(Φ\(a\)\+a⊙Φ′\(a\)\)W1diag\(1/σ\)\.\\frac\{\\partial q\}\{\\partial h\}=\\omega^\{\-1\}\\,W\_\{2\}\\,\\mathrm\{diag\}\\big\(\\Phi\(a\)\+a\\odot\\Phi^\{\\prime\}\(a\)\\big\)\\,W\_\{1\}\\,\\mathrm\{diag\}\(1/\\sigma\)\.\(14\)Letν=∥q∥\\nu=\\lVert q\\rVertandf\(ν\)=tanh\(cν\)/\(cν\)f\(\\nu\)=\\tanh\(\\sqrt\{c\}\\,\\nu\)/\(\\sqrt\{c\}\\,\\nu\), so thatz=f\(ν\)qz=f\(\\nu\)\\,q\. Then
∂z∂q=f\(ν\)In\+f′\(ν\)νqq⊤,f′\(ν\)=sech2\(cν\)ν−tanh\(cν\)cν2,\\frac\{\\partial z\}\{\\partial q\}=f\(\\nu\)\\,I\_\{n\}\+\\frac\{f^\{\\prime\}\(\\nu\)\}\{\\nu\}\\,qq^\{\\top\},\\qquad f^\{\\prime\}\(\\nu\)=\\frac\{\\operatorname\{sech\}^\{2\}\(\\sqrt\{c\}\\,\\nu\)\}\{\\nu\}\-\\frac\{\\tanh\(\\sqrt\{c\}\\,\\nu\)\}\{\\sqrt\{c\}\\,\\nu^\{2\}\},\(15\)andJ=\(∂z/∂q\)\(∂q/∂h\)J=\(\\partial z/\\partial q\)\(\\partial q/\\partial h\), whereω\\omegais fixed after training\. The damping in Eq\.[7](https://arxiv.org/html/2610.02391#S3.E7)is relative to the scale of the problem,η=10−3tr\(JJ⊤\)/n\\eta=10^\{\-3\}\\operatorname\{tr\}\(JJ^\{\\top\}\)/n\. It keeps then×nn\\times nsolve stable whenJJ⊤JJ^\{\\top\}is ill\-conditioned, for example near the boundary of the ball, where the radial derivativesech2\(cν\)\\operatorname\{sech\}^\{2\}\(\\sqrt\{c\}\\,\\nu\)of the exponential map becomes much smaller than its tangential derivativef\(ν\)f\(\\nu\)\.
#### Accuracy of the mapping back\.
We reran the MATH\-500 decodes of the development model with the Busemann readout while logging every steered token, and the rerun reproduces all500500solutions\. Over the4,4264\{,\}426steered tokens in483483solutions, the mean drop of the readout per solution lies between0\.42930\.4293and0\.42950\.4295, againstct=0\.4294\\sqrt\{c\}\\,t=0\.4294in Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)\. The mean hyperbolic distance betweenzzandE\(h\+Δh\)E\(h\+\\Delta h\)lies between0\.50000\.5000and0\.50010\.5001, againstt=0\.5t=0\.5\. The edited states thus realize the distance and the drop of the readout that Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)prescribes to within10−410^\{\-4\}on average in every solution\. In the5656solutions with a single steered token, the drop lies between0\.42930\.4293and0\.42940\.4294and the distance is0\.50000\.5000, so the same holds for individual edits\. The Euclidean probes of Appendix[N](https://arxiv.org/html/2610.02391#A14)lower their readouts by the prescribed amounts equally closely\.
## Appendix DProbe Training Details
#### Layer indexing\.
We index layers by the output of a decoder block, starting from zero, so our layerℓ\\ellishidden\_states\[ℓ\+1\\ell\+1\]in the Hugging Face convention\. SEAL adds its vector to the input of decoder layer2020of DeepSeek\-R1\-Distill\-Qwen\-1\.5B and 7B, which is the output of our layer1919\([Chen et al\., 2025](https://arxiv.org/html/2610.02391#bib.bib3)\)\. CAA edits the output of the block with index1313in Llama 2 7B Chat\([Rimsky et al\., 2024](https://arxiv.org/html/2610.02391#bib.bib33)\)\. ODESteer reports layer1414of Llama\-3\.1\-8B, and its code, which counts layers from zero, uses index1313\([Zhao et al\., 2026](https://arxiv.org/html/2610.02391#bib.bib49)\)\. The CAA and ODESteer settings both correspond to our layer1313\.
#### Data\.
For each model and dataset, we decode100100training problems greedily once, with the prompt used at evaluation\. For every generated position, we record the layer\-ℓ\\ellstate and the entropy of the next\-token distribution\. The state at the last prompt position emits the first generated token, and we align states and entropies accordingly\. The number of states ranges from15,82115\{,\}821to87,81387\{,\}813per dataset \(Table[5](https://arxiv.org/html/2610.02391#A4.T5)\)\.
#### Length limit of the training decodes\.
This step only collects training states, so its length limit plays a different role from the limit of30723072new tokens used in all evaluations \(Section[4\.1](https://arxiv.org/html/2610.02391#S4.SS1)\)\. The limit is10241024new tokens for Qwen2\.5\-Math\-1\.5B\-Instruct on MATH and GSM8K,30723072on OlympiadBench and20482048for the other two models, which is at least twice the median length of the training solutions in every setting\. It cuts off99of the100100solutions for Qwen2\.5\-Math\-1\.5B\-Instruct on MATH,66on OlympiadBench and55for Llama\-3\.1\-8B\-Instruct on MATH, and no solution in the other settings\. Because the model is causal, a cut leaves every recorded state and entropy unchanged, and a longer limit would only add positions at the ends of these solutions\. These ends are nearly deterministic, since over the last200200tokens of the cut solutions the mean entropy is at most0\.190\.19nats and at most6\.5%6\.5\\%of the tokens exceed11nat\. A longer limit would therefore add few hesitation points and mostly confident states\.
#### Split, pairs and triplets\.
Twenty of the100100problems are held out for checkpoint selection\. The split is by problem, because states within one solution are strongly correlated\. Ranking pairs\(i,j\)\(i,j\)are sampled subject toHi<Hj−δH\_\{i\}<H\_\{j\}\-\\deltawithδ=0\.3\\delta=0\.3nats\. Triplet anchors and positives come from the same solution within five tokens of each other, and negatives come from a different problem\. Each optimization step uses40964096pairs and40964096triplets drawn from pools of400,000400\{,\}000pairs and200,000200\{,\}000triplets\.
#### Optimization\.
We use AdamW with learning rate10−310^\{\-3\}for the perceptron and10−210^\{\-2\}for the curvature and the ideal point, weight decay0\.010\.01,15001500steps and marginγ=0\.1\\gamma=0\.1\. The output dimension isn=128n=128and the hidden width is10241024\. We keep the checkpoint with the best pair accuracy on the held\-out problems\. Table[5](https://arxiv.org/html/2610.02391#A4.T5)reports the held\-out quality of all probes used in the main results\.
Table 5:Held\-out quality of the probes used in the main results\. States come from the greedy solutions of100100training problems, and2020of these problems are held out for checkpoint selection\. Pair accuracy is the fraction of held\-out pairs with an entropy gap above0\.30\.3nats that the readout orders correctly, and it also selects the checkpoint\.ρ\\rhois the Spearman correlation between the readout and the entropy on held\-out states, andccis the learned curvature magnitude\.
## Appendix EDecoding Procedure
Algorithm[1](https://arxiv.org/html/2610.02391#alg1)states the decoding loop\. The cache rollback makes the edit act on the state whose next\-token distribution is uncertain\. On the greedy solutions of Qwen2\.5\-Math\-1\.5B\-Instruct, the median entropy at hesitation points is1\.231\.23nats on MATH\-500 and1\.441\.44nats on GSM8K, while one position later it is0\.0750\.075and0\.0480\.048nats\. Without the rollback, the edit would land on a state at which the model is already confident and would leave the uncertain token unchanged\.
Algorithm 1HESTdecoding for one sequence1:Run the prompt through the model; obtain the cache
CCand the logits
oofor the first token
2:for
i=1,…,Tmaxi=1,\\dots,T\_\{\\max\}do
3:
yi←argmaxoy\_\{i\}\\leftarrow\\arg\\max o; append
yiy\_\{i\}to the output;if
yiy\_\{i\}is the end tokenthenstop
4:
\(o,C\)←M\(yi∣C\)\(o,C\)\\leftarrow M\(y\_\{i\}\\mid C\)\{ordinary forward pass\}
5:ifthe entropy of
softmax\(o\)\\mathrm\{softmax\}\(o\)exceeds
θ\\thetathen
6:Remove the last position from
CC
7:
h←h\\leftarrowlayer\-
ℓ\\ellstate of
yiy\_\{i\};
z←E\(h\)z\\leftarrow E\(h\);
z′←z^\{\\prime\}\\leftarrowEq\.[6](https://arxiv.org/html/2610.02391#S3.E6)
8:
Δh←\\Delta h\\leftarrowEq\.[7](https://arxiv.org/html/2610.02391#S3.E7), clipped to
∥Δh∥≤∥h∥\\lVert\\Delta h\\rVert\\leq\\lVert h\\rVert
9:
\(o,C\)←M\(yi∣C\)\(o,C\)\\leftarrow M\(y\_\{i\}\\mid C\)with
h←h\+Δhh\\leftarrow h\+\\Delta hat layer
ℓ\\ell
10:endif
11:endfor
12:ifthe output reached
Tmax=3072T\_\{\\max\}=3072tokens without an end token,returnthe answer of greedy decoding
The edit is applied once per hesitation point, but its effect persists\. The edited state is written into the keys and values of all later layers at that position, and every later token attends to it\. The procedure is causal and needs no information about the rest of the generation\. In a batch, the cache is rolled back for all sequences, and the edit is added only to the rows whose entropy exceeds the threshold\. For the other rows, the recomputed forward pass is identical to the original one, so a hesitation point in any row costs one extra forward pass of the whole batch\. The greedy answer for the fallback is needed only for the decodes that reach the limit\.
## Appendix FGrading Protocol
Grading free\-form mathematical answers needs two steps, extraction and equivalence\. We use one protocol for all models, because models differ in how they state their answers\.
#### Extraction\.
We try four sources in order, namely the last\\boxed\{\}expression, an explicit answer statement such as “the answer is”, a restatement of the final answer and the last number in the response\. The model produces the restatement greedily, without steering and with at most6464new tokens, when given its question, its reasoning and the prompt below\.
Generatethefinalanswerforthequery\{query\}basedonthereasoningprocess\{reasoning\}inthisformat:"Answer:\\\[\\boxed\{your\_answer\_here\}\\\]"\.Theentireanswershouldbecontainedcompletelywithinthe\\boxed\{\}command\.Donotincludeanyothertext\.
The Qwen2\.5\-Math models box their answers in9797to99%99\\%of the solutions\. For Llama\-3\.1\-8B\-Instruct, the four sources are used in31%31\\%,6%6\\%,59%59\\%and4%4\\%of the MATH\-500 solutions and in0%0\\%,8%8\\%,90%90\\%and2%2\\%of the GSM8K solutions\. A grader that reads only\\boxed\{\}would therefore underestimate this model substantially\.
#### Equivalence\.
The extracted answer is compared with the reference in both raw and normalized form, using a sequence of checks\. Containers such as tuples, intervals and sets are compared element by element\. Answers are then compared by string match and numerically, with integers compared exactly and decimals to a relative tolerance of10−410^\{\-4\}\. The remaining cases go to the symbolic checker Math\-Verify\([Kydlíček, 2025](https://arxiv.org/html/2610.02391#bib.bib18)\)and, for scalars, to the equivalence check of the Qwen2\.5\-Math evaluation toolkit\([Yang et al\., 2024a](https://arxiv.org/html/2610.02391#bib.bib45)\)\. Each check runs under a time limit of2020seconds\.
#### Grading errors and paired comparisons\.
All arms are graded by the same protocol, and the restatement is produced greedily by the unsteered model, so the grade of a response depends only on its text\. A solution whose text steering leaves unchanged therefore receives the same grade as under greedy decoding\. Grading errors can affect the paired comparisons only through the changed solutions, since only a changed solution can be counted as fixed or broken in Appendix[H](https://arxiv.org/html/2610.02391#A8)\.
## Appendix GCalibration of the Gate Threshold
The rule aims at the same number of steered tokens per solution as on the development model\. Its reference is the profile of Qwen2\.5\-Math\-1\.5B\-Instruct atθ=1\\theta=1, the percentage of its steered decodes with00to1010,1111to2424and at least2525steered tokens, which is72/18/1072/18/10on MATH\-500 and76/19/476/19/4on GSM8K\. A new model has no steered decodes beforeθ\\thetais chosen\. For a candidateθ\\theta, we therefore count the tokens withHi\>θH\_\{i\}\>\\thetain each greedy solution of the evaluation set, obtained by one teacher\-forced pass, and compute the same profile\. We take theθ\\thetawhose profile is closest to the reference inL1L\_\{1\}distance as the estimate \(Table[6](https://arxiv.org/html/2610.02391#A7.T6)\)\. On the development model, the greedy solutions give73/19/973/19/9and76/20/576/20/5, within anL1L\_\{1\}distance of33of the reference, so the greedy count closely estimates the number of steered tokens\. At the selected thresholds, the other models steer a median of44to66tokens per solution with the Busemann readout, against55for the development model\. The rule uses no labels\. It does read the unlabeled evaluation prompts and the model’s own greedy solutions to them, so the calibration adapts to the input distribution without seeing any answer\. It matches counts per solution rather than rates per token, because solution lengths differ across models and datasets\. For Llama\-3\.1\-8B\-Instruct, for example, the selected thresholds give1\.1%1\.1\\%of the tokens on MATH\-500 and3\.9%3\.9\\%on GSM8K, but a similar number of hesitation points per solution\.
Table 6:Gate\-threshold calibration\. For each candidateθ\\theta, the columns give the percentage of greedy tokens withH\>θH\>\\theta, the mean number of such tokens per solution, and the percentage of solutions with00–1010,1111–2424and at least2525of them\.L1L\_\{1\}is the distance to the reference profile of Qwen2\.5\-Math\-1\.5B\-Instruct atθ=1\\theta=1,72/18/1072/18/10on MATH\-500 and76/19/476/19/4on GSM8K, which counts its steered tokens per solution \(Appendix[G](https://arxiv.org/html/2610.02391#A7)\)\. The selected threshold \(bold\) minimizesL1L\_\{1\}\. Entropies come from one teacher\-forced pass over each model’s greedy solutions and use no labels\. For Qwen2\.5\-Math\-7B on GSM8K, the99greedy solutions that reached the length limit are excluded\.θ\\thetaTokens \(%\)Per solutionProfile \(%\)L1L\_\{1\}*Llama\-3\.1\-8B, MATH\-500*1\.06\.4548\.37/24/691301\.43\.3124\.833/29/39791\.52\.7920\.940/26/34641\.62\.3417\.547/25/28501\.71\.9514\.653/25/22381\.81\.6212\.160/24/16241\.91\.3310\.065/25/10142\.01\.088\.171/22/782\.20\.695\.283/14/322*Llama\-3\.1\-8B, GSM8K*1\.010\.0319\.025/54/211031\.35\.049\.571/24/6121\.43\.907\.481/15/491\.53\.005\.789/9/2251\.62\.324\.492/6/2311\.71\.723\.394/4/2351\.81\.252\.496/3/1392\.00\.651\.298/1/143
θ\\thetaTokens \(%\)Per solutionProfile \(%\)L1L\_\{1\}*Qwen2\.5\-Math\-7B, MATH\-500*0\.715\.56102\.430/41/30850\.814\.1293\.052/25/22390\.913\.2086\.964/17/19181\.012\.4682\.072/13/16111\.111\.8778\.278/8/14201\.211\.4175\.181/6/13241\.311\.0272\.683/5/12261\.410\.6970\.485/4/12291\.510\.4268\.686/3/11301\.610\.1967\.186/4/10281\.89\.8364\.787/5/8302\.09\.5462\.889/4/734*Qwen2\.5\-Math\-7B, GSM8K*0\.73\.9412\.059/35/6350\.82\.668\.181/16/390\.92\.006\.188/9/2241\.01\.574\.893/6/1331\.11\.253\.895/4/1371\.21\.013\.196/3/1391\.30\.842\.697/2/1411\.40\.722\.298/1/1431\.50\.611\.998/1/1431\.60\.541\.698/1/1431\.80\.431\.399/1/1442\.00\.371\.199/0/145
## Appendix HFull Results
Tables[7](https://arxiv.org/html/2610.02391#A8.T7)and[8](https://arxiv.org/html/2610.02391#A8.T8)give the number of problems fixed and broken relative to greedy decoding\. They also give the number of problems for which the steered decode reached the length limit, so that the fallback was used\. The fallback matters most for Llama\-3\.1\-8B\-Instruct on MATH\-500, where greedy decoding itself reaches the length limit on6565of the500500problems\. For the Busemann readout on this setting,44of the1717fixed problems are problems on which greedy decoding reaches the length limit while the steered decode finishes with a correct answer\. Without them,1313problems are fixed and1111broken\. For Qwen2\.5\-Math\-7B\-Instruct on MATH\-500,1616greedy solutions degenerate into repetition, and11of the88fixed problems is among them\.
#### Paired sign tests\.
Greedy decoding andHESTdecode the same problems, so we compare them with a two\-sided sign test on the problems whose outcome changes\. As a sparse and conservative intervention,HESTchanges the outcome of only99to4040problems per setting, or0\.70\.7to5\.6%5\.6\\%of a benchmark, which limits the power of the test in any single setting\. On MATH\-500, for example,2323outcomes change, andp<0\.05p<0\.05would require at least1717of them in one direction, whereasHESTfixes1616and breaks77\(p=0\.09p=0\.09\)\. In the order of Table[2](https://arxiv.org/html/2610.02391#S4.T2), the six settings with the Busemann readout givep=0\.09p=0\.09,0\.340\.34,0\.340\.34,0\.640\.64,0\.390\.39and0\.510\.51, and pooled over themHESTfixes8888problems and breaks6363\(p=0\.050p=0\.050\)\. This pooled test treats the six settings as independent, although the three models are evaluated on the same problems of each benchmark\. The test gives clearer results for the comparisons that motivate the design ofHEST\. CAA, which edits every token, fixes205205problems and breaks369369over the six settings \(p<10−11p<10^\{\-11\}\), so its losses are not due to chance\. Against the Euclidean probe of Section[5](https://arxiv.org/html/2610.02391#S5)with the Busemann readout,HESTalone is right on2323MATH\-500 problems and the probe alone on1010\(p=0\.035p=0\.035\)\. On GSM8K, the counts are2525and1616\(p=0\.21p=0\.21\), and pooled over the two benchmarks they are4848and2626\(p=0\.014p=0\.014\)\. These tests treat the probe as fixed, and since each setting uses a single trained probe, they do not measure the variation across training runs of the probe\. Probes trained with the same recipe at layers99and2525also improve MATH\-500 accuracy \(Appendix[M](https://arxiv.org/html/2610.02391#A13)\), and the Busemann readout improves it at every step length of Table[1](https://arxiv.org/html/2610.02391#S4.T1)\.
Table 7:Qwen2\.5\-Math\-1\.5B\-Instruct, full results\.Δ\\Delta: change from greedy decoding in points\. F/B: problems fixed \(greedy wrong,HESTright\) and broken \(greedy right,HESTwrong\)\. Cap: problems whose steered decode reached the 3072\-token limit and fell back to the greedy answer\. The largest change in each column is in bold\.Table 8:All models, full results for the twoHESTconfigurations of Table[2](https://arxiv.org/html/2610.02391#S4.T2)\. Columns as in Table[7](https://arxiv.org/html/2610.02391#A8.T7),ℓ\\ellis the steered layer, and the best accuracy in each row is in bold\.
## Appendix ICAA Baseline
The CAA directionvvis the raw difference of means of Section[4\.1](https://arxiv.org/html/2610.02391#S4.SS1)\. Relative to the median norm of the hidden states, its norm is0\.330\.33and0\.240\.24for Qwen2\.5\-Math\-1\.5B\-Instruct on MATH\-500 and GSM8K,0\.300\.30and0\.230\.23for Llama\-3\.1\-8B\-Instruct, and0\.180\.18and0\.220\.22for Qwen2\.5\-Math\-7B\-Instruct\. A singleHESTedit has a median relative norm of0\.060\.06on the development model \(Table[13](https://arxiv.org/html/2610.02391#A12.T13)\), and it is applied to a small fraction of the tokens\. Table[9](https://arxiv.org/html/2610.02391#A9.T9)gives further statistics\. Adding the fallback ofHESTto CAA does not change the conclusion: CAA remains below greedy decoding in all six settings\.
Table 9:CAA baseline in detail\. CAA adds the raw difference\-of\-means vector \(multiplier11\) at every generated token and is reported without the fallback\. Cap: decodes that reached the 3072\-token limit\. Length: mean solution length relative to greedy decoding\. Changed: percentage of solutions whose text differs from greedy decoding\. The last column applies the same fallback asHESTand is given only as a diagnostic\.
## Appendix JHesitation Groups and Hesitation Density
Table[10](https://arxiv.org/html/2610.02391#A10.T10)gives the change in accuracy for each group of hesitation points and for the threeHESTconfigurations\. It also lists the greedy accuracygg, the fixed rateffand the broken ratebbof each group\. The groups were fixed before the analysis\. The group with at least2525hesitation points could be split further\. However, the part with at least5050hesitation points holds only22,66and22problems for Qwen2\.5\-Math\-1\.5B\-Instruct on MATH\-500 and GSM8K and for Llama\-3\.1\-8B\-Instruct on MATH\-500\.
Table[11](https://arxiv.org/html/2610.02391#A10.T11)groups problems instead by hesitation density, which is the fraction of tokens in the greedy solution that exceed the threshold\. These groups were chosen after the results of Table[10](https://arxiv.org/html/2610.02391#A10.T10)were known\. With the Busemann readout att=0\.5t=0\.5, the densest group has the largest loss per problem for Qwen2\.5\-Math\-1\.5B\-Instruct on GSM8K and OlympiadBench and for Llama\-3\.1\-8B\-Instruct on both benchmarks\. This group holds only2020,1515,22and6060problems in these settings\. The median density of the greedy solutions of Qwen2\.5\-Math\-1\.5B\-Instruct is1\.07%1\.07\\%on MATH\-500,1\.62%1\.62\\%on GSM8K and2\.01%2\.01\\%on OlympiadBench\.
Table 10:Change from greedy decoding by the number of hesitation points in the greedy solution\. For each group:Δ\\Deltain points, the percentage of problems in the group \(%\), the greedy accuracygg, the fixed rateffamong greedy\-wrong problems and the broken ratebbamong greedy\-right problems, so thatΔ=100\[\(1−g\)f−gb\]\\Delta=100\\,\[\(1\-g\)f\-gb\]for the unrounded rates\.Table 11:Change from greedy decoding by hesitation density, the fraction of tokens in the greedy solution withH\>θH\>\\theta\. Columns as in Table[10](https://arxiv.org/html/2610.02391#A10.T10), except that the second column of each group is the number of problems\. These groups were defined after the results of Table[10](https://arxiv.org/html/2610.02391#A10.T10)were known\.
## Appendix KHow Many TokensHESTSteers
Table 12:How many tokensHESTsteers \(Busemann,t=0\.5t=0\.5\)\. Per problem, the steered fraction is the number of steered tokens divided by the length of the steered solution\. Pooled is the total number of steered tokens over the total number of generated tokens\. Steered per solution: mean number of steered tokens\. No steered token: percentage of solutions in which no token is steered\. Capped: steered decodes that reached the 3072\-token limit, and the percentage of all steered tokens that fall in them\. A few capped decodes repeat themselves and pass the threshold many times, which inflates the pooled fraction for Qwen2\.5\-Math\-7B\. The median is not affected\.Table[12](https://arxiv.org/html/2610.02391#A11.T12)describes how many tokensHESTsteers per solution\. The median fraction lies between0\.80\.8and3\.0%3\.0\\%in every setting\. Pooled over all tokens, the fraction is between1\.01\.0and4\.2%4\.2\\%except for Qwen2\.5\-Math\-7B\-Instruct, where a few steered decodes that reach the length limit repeat themselves and pass the threshold many times\. These decodes contain62%62\\%of all steered tokens on MATH\-500 and77%77\\%on GSM8K, and the fallback replaces their answers by those of greedy decoding\.
## Appendix LDiversity of the Steering Directions
We apply theHESTstep to400400states of Qwen2\.5\-Math\-1\.5B\-Instruct at layer1919whose entropy exceeds11\. The states come from5858MATH training problems, and we compare the resulting editsΔh\\Delta h\(Table[13](https://arxiv.org/html/2610.02391#A12.T13)\)\. A fixed steering vector would give pairwise cosines of11and a single principal component\. TheHESTedits have a median pairwise cosine of0\.210\.21for the Busemann readout, and3232principal components are needed to explain half of their variance\. Even the mean edit has a median cosine of only0\.470\.47with the individual edits\. The edits are more similar within a problem than across problems, with pairwise cosines of0\.340\.34within a problem and0\.210\.21across problems for the Busemann readout\.
Table 13:Diversity of theHESTedits on400400hesitation points of Qwen2\.5\-Math\-1\.5B\-Instruct\. The columns give the median pairwise cosine between edits, with the 10th and 90th percentiles in parentheses, the variance explained by the first principal component and by the first five, the number of components needed for half of the variance, the median relative size∥Δh∥/∥h∥\\lVert\\Delta h\\rVert/\\lVert h\\rVert, and the median cosine between each edit and the mean edit\.
## Appendix MAblation Details
Table[14](https://arxiv.org/html/2610.02391#A13.T14)gives the full statistics of the ablations of Section[5](https://arxiv.org/html/2610.02391#S5)\. All variants are graded with the protocol of Appendix[F](https://arxiv.org/html/2610.02391#A6)\. The fallback of Section[3\.4](https://arxiv.org/html/2610.02391#S3.SS4)changes the accuracy of every variant ofHESTby at most0\.400\.40points, so the conclusions do not depend on it\.
#### Threshold and layer\.
The thresholdθ=1\\theta=1was fixed before the ablation \(Section[3\.5](https://arxiv.org/html/2610.02391#S3.SS5)\)\. Lowering it toln2\\ln 2nearly triples the median fraction of steered tokens and lowers accuracy by1\.001\.00point, andθ=0\\theta=0is the every\-token variant\. Raising it toln4\\ln 4orln8\\ln 8steers at most0\.5%0\.5\\%of the tokens, and only1111and44problems change their outcome\. The probes for layers99and2525are trained with the same recipe on the same100100training problems\. They steer the same fraction of tokens as the probe for layer1919and give gains of0\.600\.60and0\.200\.20points, against1\.801\.80points at layer1919\.
#### Hesitation points and step boundaries\.
On the500500greedy MATH\-500 solutions, a solution has a median of66step boundaries and55hesitation points\. Only356356of the3,9743\{,\}974step starts are hesitation points \(9\.0%9\.0\\%\), and only356356of the4,3334\{,\}333hesitation points are step starts \(8\.2%8\.2\\%\)\. A hesitation point lies a median of2727tokens after the last step boundary\. The two variants thus edit about as many tokens, but at different positions\.
#### Fixed and broken solutions by direction\.
Table[15](https://arxiv.org/html/2610.02391#A13.T15)splits the changed solutions by their greedy outcome\. The break rates are low for all three directions\. The fix rates differ more, and the probe direction has the highest fix rate on both benchmarks\. Over the two benchmarks, the probe direction is correct and the fixed direction wrong on3939problems, and the opposite holds on3333problems\. Against the reverse direction, the counts are4545and3333\.
Table 14:Full statistics of the ablations on Qwen2\.5\-Math\-1\.5B\-Instruct, Busemann readout,t=0\.5t=0\.5\. Accuracy in % with the change from greedy decoding in points, with the fallback and without it\. F/B: problems fixed and broken\. Cap: steered decodes that reach the limit of30723072tokens\. Steered: median and pooled fraction of steered tokens in %\. Edits: median number of steered tokens per solution\. Step: median over problems of the mean∥Δh∥/∥h∥\\lVert\\Delta h\\rVert/\\lVert h\\rVertof a solution\. ForHESTon MATH\-500 it is measured at the hesitation points of the greedy solutions, and it was not logged forHESTon GSM8K\. Length: median number of tokens, against470470for greedy decoding on MATH\-500 and289289on GSM8K\.VariantAccuracyF/BCapNo fallbackSteeredEditsStepLengthPositionsMATH\-500Hesitation points \(HEST\)76\.00 \(\+1\.80\)16/7676\.001\.1 / 1\.650\.064472Every token74\.00 \(−\-0\.20\)20/21773\.80100 / 1004880\.067488Step boundaries73\.80 \(−\-0\.40\)5/7673\.801\.4 / 1\.460\.072480DirectionMATH\-500Fixed76\.00 \(\+1\.80\)13/4676\.001\.1 / 1\.650\.060458Reverse74\.60 \(\+0\.40\)9/7774\.201\.1 / 1\.550\.068468DirectionGSM8KProbe \(HEST\)85\.37 \(\+0\.53\)23/161485\.371\.6 / 2\.35–290Fixed84\.91 \(\+0\.08\)20/191184\.911\.6 / 2\.450\.060290Reverse84\.99 \(\+0\.15\)16/141584\.911\.6 / 2\.450\.066290ThresholdMATH\-500θ=ln2\\theta=\\ln 273\.20 \(−\-1\.00\)10/15573\.003\.1 / 3\.8150\.067476θ=1\\theta=1\(HEST\)76\.00 \(\+1\.80\)16/7676\.001\.1 / 1\.650\.064472θ=ln4\\theta=\\ln 474\.40 \(\+0\.20\)6/5774\.200\.2 / 0\.510\.063476θ=ln8\\theta=\\ln 874\.60 \(\+0\.40\)3/1674\.800\.0 / 0\.100\.060470LayerMATH\-500ℓ=9\\ell=974\.80 \(\+0\.60\)10/7374\.801\.1 / 1\.650\.052474ℓ=19\\ell=19\(HEST\)76\.00 \(\+1\.80\)16/7676\.001\.1 / 1\.650\.064472ℓ=25\\ell=2574\.40 \(\+0\.20\)12/11674\.001\.1 / 1\.650\.069470Table 15:Changed solutions by direction, with the gate and the step size ofHEST\. Fixed: fraction of the greedy\-wrong problems with a changed solution that the variant fixes\. Broken: fraction of the greedy\-right problems with a changed solution that it breaks\.
## Appendix NEuclidean Probes
The Euclidean probes of Section[5](https://arxiv.org/html/2610.02391#S5)take the limitc→0c\\to 0of the probe of Section[3](https://arxiv.org/html/2610.02391#S3)\. They keep the perceptron, the loss of Eq\.[5](https://arxiv.org/html/2610.02391#S3.E5), the training states, the held\-out problems, the optimizer and the number of steps of Appendix[D](https://arxiv.org/html/2610.02391#A4)\. Only the curvature and the operations that depend on it change\.
#### Formulas\.
Asc→0c\\to 0, the mapexp0c\\exp^\{c\}\_\{0\}tends to the identity, and the probe becomesE\(h\)=ϕ\(\(h−μ\)/σ\)/ωE\(h\)=\\phi\\big\(\(h\-\\mu\)/\\sigma\\big\)/\\omega\. The conformal factorλzc\\lambda^\{c\}\_\{z\}tends to22, so the metric tends to4gE4g^\{E\}and the distance tod0\(z,z′\)=2∥z−z′∥d\_\{0\}\(z,z^\{\\prime\}\)=2\\lVert z\-z^\{\\prime\}\\rVert\. The radius readout tends tor0\(z\)=2∥z∥r\_\{0\}\(z\)=2\\lVert z\\rVert, whose level sets are spheres around the origin\. The Busemann readout satisfiesb\(z\)/\(2c\)→−⟨ξ,z⟩b\(z\)/\(2\\sqrt\{c\}\)\\to\-\\langle\\xi,z\\rangle\. We therefore use the linear readoutb0\(z\)=−⟨ξ,z⟩b\_\{0\}\(z\)=\-\\langle\\xi,z\\ranglewith a learned unit vectorξ\\xi, which is half the Busemann function ofd0d\_\{0\}for the ideal point in the directionξ\\xi\. Its level sets are parallel hyperplanes orthogonal toξ\\xi, so its gradient is the same at every point\. In Eq\.[6](https://arxiv.org/html/2610.02391#S3.E6), Möbius addition tends to vector addition andtanh\(ct/2\)/c\\tanh\(\\sqrt\{c\}\\,t/2\)/\\sqrt\{c\}tends tot/2t/2, so the step becomes
z′=z\+t2u\.z^\{\\prime\}=z\+\\frac\{t\}\{2\}\\,u\.\(16\)This step moveszzbyd0\(z,z′\)=td\_\{0\}\(z,z^\{\\prime\}\)=t, lowersr0r\_\{0\}byttwhent≤r0\(z\)t\\leq r\_\{0\}\(z\)and lowersb0b\_\{0\}byt/2t/2, which is the limit of Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)\. For the linear readout,u=ξu=\\xiat every state, so the step in the probe space is the same vector everywhere, and only the Jacobian of the probe makes the edit depend on the state\. The Jacobian of Appendix[C](https://arxiv.org/html/2610.02391#A3)loses the factor ofexp0c\\exp^\{c\}\_\{0\}\. The mapping back to the hidden state, the clipping and the gate are those ofHEST\.
#### Step length\.
A step of lengthttdoes not change the hidden state by the same amount in the two geometries, since the probes differ in their metric and in their scale\. We therefore match the size of the edit instead of the step length\. For the MATH\-500 runs, we use the400400hesitation points of Appendix[L](https://arxiv.org/html/2610.02391#A12)\. For the GSM8K runs, we use all300300hesitation points of the probe\-training solutions, which come from8585problems\. On these states, we measure the median∥Δh∥/∥h∥\\lVert\\Delta h\\rVert/\\lVert h\\rVertofHESTat its step length and find by bisection the Euclideanttthat gives the same median, so no test label enters the choice \(Table[16](https://arxiv.org/html/2610.02391#A14.T16)\)\. On the MATH\-500 test decodes, the median edit is0\.0610\.061for the Euclidean Busemann probe and0\.0660\.066forHEST\.HESTwith half the step length still gains0\.800\.80points \(Table[1](https://arxiv.org/html/2610.02391#S4.T1)\), so this difference of8%8\\%does not explain the gap in accuracy\. On the GSM8K test decodes, the median edit of the Euclidean Busemann probe is0\.0580\.058, as on the training states\. The step length ofHESTwas chosen on the same test sets \(Section[3\.5](https://arxiv.org/html/2610.02391#S3.SS5)\), whereas the Euclidean step length only matches its edit size\. On MATH\-500, however,HESTgains accuracy at every step length of Table[1](https://arxiv.org/html/2610.02391#S4.T1), by0\.600\.60to1\.801\.80points, while the Euclidean Busemann probe loses0\.800\.80points at the matched edit size\.
#### Probe quality and results\.
The Euclidean probes order the held\-out entropy pairs at least as well as the hyperbolic probes\. Curvature thus brings no advantage in predicting the entropy, and the difference in steering lies in the direction that each geometry gives\. On MATH\-500, the gate steers a median of1\.1%1\.1\\%of the tokens in all four runs\. The Euclidean Busemann probe changes245245solutions andHEST270270\. Of the changed solutions that greedy decoding gets wrong, the Euclidean probe fixes10\.6%10\.6\\%andHEST15\.4%15\.4\\%\. Of those that greedy decoding gets right, it breaks9\.3%9\.3\\%andHEST4\.2%4\.2\\%\. The Euclidean Busemann edits are also more similar to each other\. On the400400hesitation points of Appendix[L](https://arxiv.org/html/2610.02391#A12), their median pairwise cosine is0\.300\.30, against0\.210\.21forHEST, and their remaining variation spreads over more principal components,4141against3232for half of the variance\. The radius edits of the two geometries are about equally diverse, with median cosines of0\.250\.25in both\. On GSM8K, the gate steers a median of1\.6%1\.6\\%of the tokens in all four runs\. The Euclidean Busemann probe changes498498solutions andHEST536536\. Of the changed solutions that greedy decoding gets wrong, the Euclidean probe fixes13\.8%13\.8\\%andHEST20\.4%20\.4\\%, and of those that it gets right, the two break4\.4%4\.4\\%and3\.8%3\.8\\%\. With the radius readout, the two geometries fix and break similar fractions and reach the same accuracy\.
Table 16:Euclidean probes against the probes ofHESTon Qwen2\.5\-Math\-1\.5B\-Instruct, withθ=1\\theta=1and layer1919\.cc: curvature magnitude, which is00for the Euclidean probes\. Pair acc\. andρ\\rho: held\-out quality as in Table[5](https://arxiv.org/html/2610.02391#A4.T5)\. Step: median∥Δh∥/∥h∥\\lVert\\Delta h\\rVert/\\lVert h\\rVerton the hesitation points of the probe\-training solutions, which the step lengthttof the Euclidean probe matches\. Accuracy in % with the change from greedy decoding in points, and the better accuracy of each pair is in bold\. F/B: problems fixed and broken\. Changed: solutions whose text differs from greedy decoding\. Fix rate: fraction of the changed greedy\-wrong solutions that become right\. Break rate: fraction of the changed greedy\-right solutions that become wrong\. Paired: problems on which onlyHESTis right and on which only the Euclidean probe is right\.ReadoutGeometryccPair acc\.ρ\\rhottStepAccuracyF/BChangedFix rateBreak ratePaired*MATH\-500*, greedy accuracy74\.2074\.20BusemannHyperbolic \(HEST\)0\.740\.8800\.5540\.50\.06076\.00\(\+1\.80\)16/72700\.1540\.042–Euclidean00\.8950\.5890\.6130\.06073\.40 \(−\-0\.80\)10/142450\.1060\.09323/10radiusHyperbolic \(HEST\)0\.530\.8420\.4720\.250\.01775\.20\(\+1\.00\)12/71810\.1560\.067–Euclidean00\.8490\.4930\.2520\.01774\.40 \(\+0\.20\)6/51850\.0770\.04712/8*GSM8K*, greedy accuracy84\.8484\.84BusemannHyperbolic \(HEST\)0\.710\.8550\.5010\.50\.05885\.37\(\+0\.53\)23/165360\.2040\.038–Euclidean00\.8800\.6330\.6720\.05884\.69 \(−\-0\.15\)15/174980\.1380\.04425/16radiusHyperbolic \(HEST\)0\.400\.8290\.4290\.250\.01884\.53\(−\-0\.30\)12/163630\.1540\.056–Euclidean00\.8320\.4700\.2410\.01884\.53\(−\-0\.30\)11/153540\.1430\.05410/10
## Appendix OProbe States on Real Data
Figure[4](https://arxiv.org/html/2610.02391#A15.F4)shows how the trained probes embed real hidden states of the development model on MATH\-500, and Figure[5](https://arxiv.org/html/2610.02391#A15.F5)shows all models and datasets\. The states come from the2020validation problems that were held out from probe training, and each panel shows3,0003\{,\}000of them sampled at random\.
#### How the states are drawn\.
The probe space hasn=128n=128dimensions, so any two\-dimensional picture must drop information\. We keep the information that the readouts use\. Lety=czy=\\sqrt\{c\}\\,z, which lies in the unit ball\. For the Busemann probe, we drawyyat distance∥y∥\\lVert y\\rVertfrom the center and at its angle toξ\\xi, which lies between0∘0^\{\\circ\}and180∘180^\{\\circ\}\. The Busemann readout depends only on these two quantities, because∥ξ−y∥2=1−2⟨ξ,y⟩\+∥y∥2\\lVert\\xi\-y\\rVert^\{2\}=1\-2\\langle\\xi,y\\rangle\+\\lVert y\\rVert^\{2\}\. Each state therefore keeps its exact readout, and the dashed level sets are exact horocycles\. Rotating a state about the axis through the origin andξ\\xichanges neither quantity, so all states related by such a rotation are drawn at the same point, and a half disk is enough\. A full disk would require a second direction to decide on which side of the axis each state falls, and this choice would be arbitrary\. For the radius probe, the readout depends only on∥y∥\\lVert y\\rVert, and we measure the angle from the first principal direction of the states\.
#### Comparison with the schematic\.
The real states follow the same ordering as the schematic in Figure[1](https://arxiv.org/html/2610.02391#S1.F1)b,c, and the pattern is similar across models and datasets\. They differ from the schematic in four ways\. The states do not fill the ball, since∥y∥\\lVert y\\rVertlies between0\.390\.39and0\.820\.82for90%90\\%of them, and almost no state lies at the origin or near the boundary\. With the Busemann probe, confident and hesitant states differ more in their angle toξ\\xithan in their norm\. States with entropy below0\.10\.1nats have a median angle of42∘42^\{\\circ\}to66∘66^\{\\circ\}, and states above the threshold have a median angle of66∘66^\{\\circ\}to87∘87^\{\\circ\}\. Confident states are thus closer toξ\\xiin direction rather than in norm\. With the radius probe, confident states lie at the inner edge of the occupied region rather than at the origin\. Their median norm is0\.500\.50to0\.620\.62, against0\.650\.65to0\.840\.84for states above the threshold\. Finally, the two groups overlap\. Across the1414probes, the Spearman correlation between the readout and the entropy of the plotted states ranges from0\.390\.39to0\.760\.76\.
These differences do not affect the method\.HESTuses only the gradient of the readout at the current state and a fixed step length\. Proposition[1](https://arxiv.org/html/2610.02391#Thmproposition1)holds wherever the state lies in the ball, so a step of lengthttlowers the readout by the same amount for every state\. The schematic is drawn to make the ordering of the readout easy to see, and Figure[1](https://arxiv.org/html/2610.02391#S1.F1)marks it as schematic\.
Figure 4:Held\-out states of Qwen2\.5\-Math\-1\.5B\-Instruct on MATH\-500, embedded by the Busemann probe \(left\) and the radius probe \(right\)\. Each statey=czy=\\sqrt\{c\}\\,zis drawn at its norm and at its angle to the ideal pointξ\\xi\(left\) or to the first principal direction \(right\), which keeps its readout exact\. Rotations about the horizontal axis leave the readout unchanged, so a half disk shows all states\. Dashed curves are level sets of the readout at its1010th,5050th and9090th percentiles\. Darker points have higher next\-token entropy\.Figure 5:Held\-out states of every model and dataset, drawn as in Figure[4](https://arxiv.org/html/2610.02391#A15.F4)\. Top: Busemann probes\. Bottom: radius probes\.
## Appendix PThe Example of Section[4\.3](https://arxiv.org/html/2610.02391#S4.SS3.SSS0.Px3)in Full
The problem is MATH\-500 test problem 389 \(geometry, level 3\)\. Its diagram places the right angle atSSand labels the sideRTRT, the hypotenuse, with length55\. The reference answer is21/5\\sqrt\{21\}/5\. We reproduce the problem and both solutions of Qwen2\.5\-Math\-1\.5B\-Instruct verbatim\. The steered solution sets the right angle correctly atSS, but it then callsRSRSthe hypotenuse andRTRTa leg\. The two labels are swapped consistently, so the final ratio, and hence the answer, is correct even though these intermediate statements are not\.
\\outputfile
grayProblemtables/case389\_problem\.txt\\outputfileforkorangeGreedy decoding:189189tokens, answer11, wrongtables/case389\_greedy\.txt\\outputfilehestblueHEST, Busemann readout,t=0\.5t=0\.5:583583tokens,1212steered, answer21/5\\sqrt\{21\}/5, correcttables/case389\_hest\.txt相似文章
HyperGuide:大型语言模型中高效多步推理的双曲引导方法
本文提出HyperGuide方法,将推理进展提炼为双曲几何信号,以指导LLMs的逐步生成,从而无需显式树搜索即可提高多步推理效率。
提示嵌入探针(PEP):从隐藏状态检测大语言模型幻觉
本文介绍了提示嵌入探针(PEP),一种参数高效的线性探针扩展方法,利用隐藏状态上的可学习提示嵌入来检测冻结的大语言模型中的幻觉。在 TriviaQA、GSM8K 和 MedQA 上使用 Qwen3 模型的评估显示,相比标准线性探针,该方法在预生成和跨模型设置中均有改进。
DenseSteer:引导小型语言模型走向密集数学推理
DenseSteer是一种无需训练的推理时框架,通过将小型语言模型的内部表示引导至密集推理模式,提升其数学推理能力,在不增加词元级负对数似然的情况下实现准确率提升。
潜在奖励引导:一种在推理大语言模型中隐式促进认知行为的自适应推理时框架
介绍了潜在奖励引导(LRS),一种自适应推理时框架,利用稀疏自编码器的潜在状态和学习的奖励模型,隐式促进推理大语言模型中的验证和回溯等认知行为,从而在多个模型和基准测试中提升性能。
转向几何:验证LLM转向空间中的人类价值几何结构
本文介绍了一个基准测试,以验证LLMs中激活转向的分布驱动方法能够恢复与Schwartz理论一致的人类价值拓扑结构,这些方法随模型规模提升而改善,但在指令调优后有所下降。