Representation Without Reward: A JEPA Audit for LLM Fine-Tuning

# A JEPA Audit for LLM Fine-Tuning
Source: [https://arxiv.org/html/2605.15394](https://arxiv.org/html/2605.15394)
## Representation Without Reward: A JEPA Audit for LLM Fine\-Tuning

###### Abstract

Joint\-embedding predictive architectures \(JEPAs\) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs\. For autoregressive language\-model fine\-tuning the principle entails a stricter requirement: the induced hidden\-state geometry must reach the language\-model head*and*improve the decoded task metric\. We test that requirement under a fixed Llama\-3\.2\-1B\-Instruct LoRA harness on natural\-language\-to\-regex generation, comparing twenty\-two training\-time auxiliaries across trajectory\-shape regularisation, distributional constraints, predictor/target asymmetry, Fisher\-metric Jacobi residuals, and a decoder\-visible JEPA objective constructed to lie in cross\-entropy’s positive cone\. The empirical answer is a structured null: several auxiliaries clear single\-cell pairedα=0\.10\\alpha=0\.10without correction \(T3\-Local atΔ=\+2\.53\\Delta=\+2\.53pp,p=0\.003p=0\.003being the strongest\), but none survives Bonferroni or Holm–Bonferroni at the relevant family\-wise threshold, even though many change curvature, anisotropy, variance, and gradient direction\. Decoder\-visible JEPA yields the first positive auxiliary–cross\-entropy gradient cosine in the study, yet exact match remains inside seed noise; a full\-fine\-tuning replication of the same auxiliary atn=5n=5seeds reproduces the null on both benchmarks \(TURK:Δ=\+0\.04\\Delta=\+0\.04pp,ppaired=0\.96p\_\{\\text\{paired\}\}=0\.96; SYNTH:Δ=\+0\.52\\Delta=\+0\.52pp,ppaired=0\.28p\_\{\\text\{paired\}\}=0\.28\), so the null is robust across LoRA and full fine\-tuning for the decoder\-visible construction\. Hidden\-state representation work and decoded\-task accuracy are therefore weakly coupled in this regime; we accordingly reframe LLM\-domain JEPA evaluation as a coupling problem, in which the operative question is under which metrics useful hidden geometry becomes decoder\-visible task signal\.

Disclaimer:This paper was prepared for informational purposes by the LLM Suite group of JP Morgan Chase and its affiliates \(‘JPMC’\) and is not a product of the Research Department of JP Morgan\. JP Morgan makes no representation, warranty or undertaking whatsoever and disclaims all liability for the completeness, accuracy or reliability of the information contained herein\. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful\.

## IIntroduction

Joint\-embedding predictive architectures propose a compelling learning principle: predict the representation of what is missing, not the pixels or tokens themselves\. Variants of this idea appear across several research programmes: self\-supervised predictive networks built around prediction error\[[22](https://arxiv.org/html/2605.15394#bib.bib30)\], the joint\-embedding predictive architecture \(JEPA\) framing that names the canonical form\[[19](https://arxiv.org/html/2605.15394#bib.bib32)\], and curiosity\-driven model\-building that extends the prediction\-error idea to exploration\[[23](https://arxiv.org/html/2605.15394#bib.bib31)\]\. The principle is currently exemplified in vision by I\-JEPA\[[1](https://arxiv.org/html/2605.15394#bib.bib1)\]and dense video extensions\[[21](https://arxiv.org/html/2605.15394#bib.bib5)\], which combine latent prediction with predictor/target asymmetry\. The scientific question for language\-model fine\-tuning is not whether such objectives can move hidden states\. They can\. The sharper question is whether the movement is coupled to the decoder and to the task metric\.

Autoregressive language models make this question unusually stringent\. The training signal is next\-token cross\-entropy, the downstream system is the language\-model head plus a decoding rule, and many symbolic tasks are scored by exact match\. A representation may become smoother, more isotropic, or more predictive under an auxiliary loss while leaving the selected token sequence unchanged\. We therefore separate two testable claims that the JEPA principle conflates in this setting:

> *H1 \(representation activity\)\.*Adding a JEPA\-style auxiliary to LoRA fine\-tuning induces a measurable change in hidden\-state geometry relative to the no\-auxiliary baseline — in anisotropy, trajectory curvature, seed\-to\-seed variance, or gradient direction\. *H2 \(decoder\-coupled payoff\)\.*That hidden\-state change translates, on the same checkpoints, into a measurable improvement of the decoded exact\-match accuracy\.

H1 is about whether the auxiliary*acts*; H2 is about whether the exact\-match metric*rewards*the action\. The two claims are not redundant: an auxiliary that satisfies H1 but not H2 is the empirical signature this paper sets out to characterise\.

Semantic Tube Prediction \(STP\)\[[16](https://arxiv.org/html/2605.15394#bib.bib2)\]provides a minimal language\-domain version of the JEPA idea\. It treats hidden states along an assistant response as a local trajectory on a semantic manifold and adds a cosine\-alignment loss between velocity vectors\. The reported16×16\\timesdata\-efficiency gain on NL\-RX\-SYNTH motivates a broader hypothesis: perhaps trajectory\- or representation\-predictive pressure can supply task\-relevant structure when only LoRA adapters are being trained\.

We test that hypothesis by fixing the model, LoRA setup, data format, splice point, decoder, and exact\-match evaluator, and varying only the auxiliary objective\. The study covers twenty\-two training\-time auxiliaries and one inference\-time projector for Llama\-3\.2\-1B\-Instruct on natural\-language\-to\-regex generation\. The objectives are organised not as an inventory of losses, but as a sequence of mechanistic questions: Do trajectory attractors help? Do contrastive or predictive pressures help? Do distributional constraints outside cross\-entropy’s implicit\-bias direction help? Is JEPA’s predictor/target asymmetry the missing ingredient? If hidden geometry matters but the Euclidean metric is wrong, does a Fisher\-metric Jacobi residual expose it? If gradient misalignment is the obstacle, can a decoder\-visible JEPA objective force the auxiliary into cross\-entropy’s positive cone?

The answer across these tests is a structured null\. Without statistical correction, several auxiliaries reach single\-cell pairedα=0\.10\\alpha=0\.10: prompt\-local JFR on TURK \(\+2\.53\+2\.53pp,p=0\.003p=0\.003\), T5 on TURK \(\+1\.53\+1\.53pp,p=0\.08p=0\.08\), and three Tier\-1 distributional cells on SYNTH \(L1, L3, L4 with pairedppbetween0\.060\.06and0\.100\.10\)\. With Bonferroni or Holm–Bonferroni correction at the relevant family\-wise threshold, none of these cells survives\. BYOL\-LLM, the direct test of predictor \+ EMA target \+ stop\-gradient asymmetry, remains inside seed noise on both benchmarks\. The Fisher\-metric replacements remove the geometric signature without improving exact match\. Decoder\-visible JEPA produces the first positive auxiliary–cross\-entropy gradient cosine, but its exact\-match gain also remains statistically unresolved both under LoRA and under a full\-fine\-tuning replication atn=5n=5seeds\.

The null is therefore not a story of inactive objectives\. Several losses reduce trajectory curvature, alter hidden\-state anisotropy, tighten seed\-to\-seed variance, or change gradient direction\. The paper’s central claim is instead a separation between representation work and decoded performance: in this harness, JEPA\-style auxiliaries can reshape hidden geometry without reliably changing the exact symbolic output selected by the language\-model head\.

#### Contributions\.

1. 1\.A decoder\-visible test of JEPA\-style representation shaping in LLM fine\-tuning\.We frame the central question as whether auxiliary representation objectives merely move hidden geometry or produce changes that survive the language\-model head and improve exact\-match decoding\.
2. 2\.A controlled hypothesis map\.We evaluate twenty\-two JEPA\-inspired training\-time auxiliaries and one inference\-time projector on Llama\-3\.2\-1B\-Instruct, keeping the model, LoRA setup, data, splice point, decoder, and evaluator fixed\. The menu covers trajectory\-shape attractors \(STP, T1–T6\), contrastive and predictive objectives \(T7, L12\), distributional constraints outside cross\-entropy’s implicit\-bias direction \(L1–L6, L9\), predictor/target asymmetry tests \(L13–L14\), Fisher\-metric Jacobi residuals \(Tier\-3\), and a decoder\-visible JEPA objective constructed to lie in cross\-entropy’s positive cone\.
3. 3\.A structured null for asymptotic exact\-match accuracy\.Without statistical correction, a handful of cells reach single\-cell pairedα=0\.10\\alpha=0\.10\(prompt\-local JFR and T5 on TURK; L1, L3, L4 on SYNTH\), but none survives Bonferroni or Holm–Bonferroni at the relevant family\-wise threshold\. This includes BYOL\-LLM \(predictor \+ EMA target \+ stop\-gradient asymmetry\) and decoder\-visible JEPA \(gradient inside the cross\-entropy positive cone\), both of which remain inside seed noise\.
4. 4\.Diagnostics separating performance inactivity from representational inactivity\.Several auxiliaries alter curvature, anisotropy, variance, and gradient direction even when exact\-match accuracy is unchanged\. The losses act; their action is weakly coupled to the decoder and metric in this harness\.

#### Paper organisation\.

Section[II](https://arxiv.org/html/2605.15394#S2)situates the paper against the JEPA, contrastive and distributional self\-supervision literature\. Section[III](https://arxiv.org/html/2605.15394#S3)introduces the unified splice point and gives formal definitions for every auxiliary in three structural classes\. Section[IV](https://arxiv.org/html/2605.15394#S4)describes the experimental harness and the statistical methodology, including the Welch’s\-test and family\-level corrections we use throughout\. Section[V](https://arxiv.org/html/2605.15394#S5)reports TURK and SYNTH results, the within\-family Jacobi\-field analysis, and the two failure modes \(T1 collapse, T9 fixed\-point decoding\)\. Section[VI](https://arxiv.org/html/2605.15394#S6)synthesises the diagnostics with the benchmark results into the decoder\-invisible representation\-work reading and discusses implications for future LLM\-domain JEPA work\.

## IIRelated Work

#### Joint\-embedding predictive architectures\.

JEPA methods learn by predicting representations rather than reconstructing inputs\. I\-JEPA\[[1](https://arxiv.org/html/2605.15394#bib.bib1)\]established this template for images with masked prediction, a learned predictor, and an EMA target encoder; V\-JEPA\-style extensions\[[21](https://arxiv.org/html/2605.15394#bib.bib5)\]extend the same idea to video and dense intermediate features\. These systems typically rely on asymmetry—for example target encoders, stop\-gradient operations, or predictor heads—to prevent collapse while preserving semantic information\. Our study intentionally removes most of this architecture in order to test a minimal question: how much can a single\-forward\-pass auxiliary loss attached to an LLM fine\-tuning loop accomplish by itself?

#### Contrastive, non\-contrastive, and distributional SSL\.

Contrastive objectives such as SimCLR\[[7](https://arxiv.org/html/2605.15394#bib.bib7)\]use negative pairs to shape representation geometry, while non\-contrastive methods such as BYOL\[[11](https://arxiv.org/html/2605.15394#bib.bib9)\], SimSiam\[[8](https://arxiv.org/html/2605.15394#bib.bib10)\], VICReg\[[5](https://arxiv.org/html/2605.15394#bib.bib11)\], and Barlow Twins\[[29](https://arxiv.org/html/2605.15394#bib.bib12)\]use predictor asymmetry, variance, or covariance constraints\. Recent theory connects these families through implicit variance regularisation and dualities between contrastive and non\-contrastive objectives\[[26](https://arxiv.org/html/2605.15394#bib.bib16),[10](https://arxiv.org/html/2605.15394#bib.bib17),[12](https://arxiv.org/html/2605.15394#bib.bib18)\]\. These results motivate the distributional losses in our study: if STP\-style trajectory alignment fails because it is too narrow, isotropy or covariance regularisation might still move the representation in a useful direction\.

#### JEPA theory and language\-domain variants\.

Recent analyses argue that JEPA\-style objectives can prefer semantically useful features under appropriate depth or architectural asymmetry\[[20](https://arxiv.org/html/2605.15394#bib.bib19)\], and that isotropic Gaussian embedding distributions can arise from minimax prediction\-risk formulations\[[4](https://arxiv.org/html/2605.15394#bib.bib20)\]\. Language\-domain variants include data2vec\[[3](https://arxiv.org/html/2605.15394#bib.bib26)\], LLM\-JEPA\[[15](https://arxiv.org/html/2605.15394#bib.bib3)\], and STP\[[16](https://arxiv.org/html/2605.15394#bib.bib2)\]\. LLM\-JEPA uses two views and a predictor/target\-style mechanism; STP strips the idea down to a single causal forward pass over the assistant trajectory\. Our paper is best read as a controlled study of that stripped\-down regime\. We do not test whether the full JEPA recipe with target encoders and predictor asymmetry can help LLMs; we test whether trajectory and distributional pressures alone improve exact match when added to LoRA fine\-tuning\.

#### Trajectory straightening as a regulariser\.

The straightening principle that motivates STP and our T1–T6 cells has recently been shown to lift task performance in a different downstream setting\.\[[27](https://arxiv.org/html/2605.15394#bib.bib4)\]train a world\-model encoder plus predictor with a curvature regulariser that is structurally identical to STP — a cosine penalty between consecutive latent velocities — and report a2020–60%60\\%improvement in goal\-reaching success under gradient\-based latent planning\. They additionally prove that under a linear\-dynamics assumption the straightening regulariser controls the condition number of the planning Hessian, providing a direct mechanism for the empirical gain\. Their result is not in tension with our null: their downstream task is gradient\-based planning over a learned latent dynamics model, where representation conditioning enters the optimisation Hessian directly; ours is autoregressive token generation under cross\-entropy followed by argmax decoding, where the representation enters only through the LM head’s softmax\. The contrast sharpens our structured\-null reading \(Section[VI\-D](https://arxiv.org/html/2605.15394#S6.SS4)\): the same geometric work that improves a gradient\-based downstream objective need not improve a metric whose dependence on the representation is mediated only by the final\-token argmax\. The decoder\-visible JEPA construction we pursue in Appendix[C](https://arxiv.org/html/2605.15394#A3)addresses this asymmetry directly by moving the auxiliary out ofhh\-space and into the post\-softmax distribution of the LM head\.

#### Positioning\.

The closest comparison is STP\[[16](https://arxiv.org/html/2605.15394#bib.bib2)\], which reports a16×16\\timesdata\-efficiency gain on NL\-RX\-SYNTH\. Our evaluation differs in scope: we measure final exact\-match accuracy after LoRA fine\-tuning, include NL\-RX\-TURK as a less saturated benchmark, and add diagnostics for curvature, anisotropy, and gradient alignment\. The two claims are therefore compatible: a geometric auxiliary may improve sample efficiency without improving the asymptotic exact\-match ceiling\. Our contribution is to show that, under this controlled exact\-match harness, the representational changes are real but not reliably task\-improving\.

## IIIMethods

We adopt a single notation throughout\. LetH∈ℝB×S×DH\\in\\mathbb\{R\}^\{B\\times S\\times D\}denote the final\-layer hidden states of a batch ofBBtraining examples, withSSthe padded sequence length andD=2048D=2048the hidden dimension of Llama\-3\.2\-1B\. For each examplebb, let\[lob,hib\)\[\\text\{lo\}\_\{b\},\\,\\text\{hi\}\_\{b\}\)be the half\-open range of token positions occupied by the assistant turn \(the regex output\), and letLb=hib−lobL\_\{b\}=\\text\{hi\}\_\{b\}\-\\text\{lo\}\_\{b\}be its length\.

The total training loss is

ℒtotal=ℒLM\+λ​\(t\)​ℒaux,\\mathcal\{L\}\_\{\\text\{total\}\}\\;=\\;\\mathcal\{L\}\_\{\\text\{LM\}\}\\;\+\\;\\lambda\(t\)\\,\\mathcal\{L\}\_\{\\text\{aux\}\},\(1\)whereℒLM\\mathcal\{L\}\_\{\\text\{LM\}\}is the standard token\-level cross\-entropy on the assistant span \(and the trailing end\-of\-turn token\),λ​\(t\)\\lambda\(t\)is the auxiliary weight at training steptt\(under our warmup\-decay schedule, see Section[IV](https://arxiv.org/html/2605.15394#S4)\), andℒaux\\mathcal\{L\}\_\{\\text\{aux\}\}is one of the auxiliaries defined below\. The single inference\-time intervention, T9 Tube\-Projected Decoding, is described separately in Section[III\-J](https://arxiv.org/html/2605.15394#S3.SS10); it does not enterℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}\.

The experimental template can be written as a single template:

ℒaux∈\{ℒtraj,ℒdist,ℒpred\},\\mathcal\{L\}\_\{\\text\{aux\}\}\\in\\left\\\{\\mathcal\{L\}\_\{\\text\{traj\}\},\\,\\mathcal\{L\}\_\{\\text\{dist\}\},\\,\\mathcal\{L\}\_\{\\text\{pred\}\}\\right\\\},\(2\)where

ℒtraj\\displaystyle\\mathcal\{L\}\_\{\\text\{traj\}\}=𝔼b,t​\[Dgeom​\(hb,t−1,hb,t,hb,t\+1\)\],\\displaystyle=\\mathbb\{E\}\_\{b,t\}\\\!\\left\[D\_\{\\text\{geom\}\}\(h\_\{b,t\-1\},h\_\{b,t\},h\_\{b,t\+1\}\)\\right\],\(3\)ℒdist\\displaystyle\\mathcal\{L\}\_\{\\text\{dist\}\}=D𝒫​\(P^ψ​\(Hspan\),P0\),\\displaystyle=D\_\{\\mathcal\{P\}\}\\\!\\left\(\\widehat\{P\}\_\{\\psi\(H\_\{\\text\{span\}\}\)\},P\_\{0\}\\right\),\(4\)ℒpred\\displaystyle\\mathcal\{L\}\_\{\\text\{pred\}\}=d​\(qϕ​\(zcontext\),sg⁡\(ztarget\)\)\.\\displaystyle=d\\\!\\left\(q\_\{\\phi\}\(z\_\{\\text\{context\}\}\),\\operatorname\{sg\}\(z\_\{\\text\{target\}\}\)\\right\)\.\(5\)The individual losses below are instantiations of these three forms\. This notation is useful because it separates what is being tested: trajectory shape, feature\-distribution shape, or context\-target prediction\.

TABLE I:Conceptual design axes\. The study is intended to falsify mechanisms, not merely to enumerate losses\.### III\-AWhere the Auxiliary Loss Attaches

Every auxiliary in this study shares a single splice point in the forward pass, illustrated in Fig\.[1](https://arxiv.org/html/2605.15394#S3.F1)\. The base model returns the final\-layer hidden statesh∈ℝB×T×Dh\\in\\mathbb\{R\}^\{B\\times T\\times D\}from a single forward pass; the auxiliary head and the language\-model head consume the same tensor in parallel, with no second backbone pass and no modification to the transformer architecture itself\.

![Refer to caption](https://arxiv.org/html/2605.15394v1/x1.png)Figure 1:The auxiliary loss splices into the standard LoRA fine\-tuning pipeline at the final\-layer hidden states, in parallel with the language\-model head\.The LM\-head branch is unchanged: it projects every position to vocabulary logits and computes the cross\-entropy on the assistant span plus the end\-of\-turn token\. The auxiliary branch first applies the EOS clip \(Section[IV](https://arxiv.org/html/2605.15394#S4)\) and then dispatches to one of three structural families — trajectory\-shape \(STP, T1, T2, T3, T3\-Local, T5, T6\), in\-batch contrastive \(T7\), and distributional \(L1–L4 projected through a sketcherψ:ℝD→ℝd′\\psi:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{d^\{\\prime\}\}\)\. The two scalar losses combine as in \([1](https://arxiv.org/html/2605.15394#S3.E1)\); the LoRA adapters and the auxiliary head’s parameters carry the gradient, and the base Llama weights stay frozen\. T5 \(Deep Semantic Tubes\) is the lone variant that additionally reads intermediate layers\{h\(4\),h\(8\),h\(12\),h\(16\)\}\\\{h^\{\(4\)\},h^\{\(8\)\},h^\{\(12\)\},h^\{\(16\)\}\\\}; every other variant operates only on the final\-layer state\.

Table[II](https://arxiv.org/html/2605.15394#S3.T2)lists the eighteen training\-time auxiliaries plus the single inference\-time intervention; the four falsification\-route additions \(Tier\-3 Fisher variants and decoder\-visible JEPA\) are documented in Appendices[B](https://arxiv.org/html/2605.15394#A2)and[C](https://arxiv.org/html/2605.15394#A3)\.

TABLE II:The hypothesis map: eighteen training\-time auxiliaries plus one inference\-time intervention\.\#NameClassEq\.STPSemantic Tube Prediction1st\-order attractor\([7](https://arxiv.org/html/2605.15394#S3.E7)\)T1Curvature\-Aware Tube2nd\-order attractor\([9](https://arxiv.org/html/2605.15394#S3.E9)\)T2Riemannian\-Metric Tubemetric\-cosine attractor\([10](https://arxiv.org/html/2605.15394#S3.E10)\)T3Jacobi\-Field Regulariser2nd\-order attractor\([11](https://arxiv.org/html/2605.15394#S3.E11)\)T3\-LocalPrompt\-Local JFRT3 with memory bank\([13](https://arxiv.org/html/2605.15394#S3.E13)\)T5Deep Semantic TubesT3 across layers\([14](https://arxiv.org/html/2605.15394#S3.E14)\)T6Multi\-Scale Tube BundleT3 across scales\([15](https://arxiv.org/html/2605.15394#S3.E15)\)T7Contrastive Tubein\-batch InfoNCE\([17](https://arxiv.org/html/2605.15394#S3.E17)\)T9Tube\-Projected Decodinginference\-time\([18](https://arxiv.org/html/2605.15394#S3.E18)\)L1SIGReg\-Statestate isotropy\([20](https://arxiv.org/html/2605.15394#S3.E20)\)L2SIGReg\-Tangenttangent isotropy\([21](https://arxiv.org/html/2605.15394#S3.E21)\)L3C\-Tube\-Sectionalcurvature\-variance\([22](https://arxiv.org/html/2605.15394#S3.E22)\)L4STP\-CMFhalf\-vs\-half CF\-MMD\([23](https://arxiv.org/html/2605.15394#S3.E23)\)L5VICReg\-VCvariance \+ covariance\([24](https://arxiv.org/html/2605.15394#S3.E24)\)L6SW\-Isosliced\-Wasserstein\([25](https://arxiv.org/html/2605.15394#S3.E25)\)L9Score\-MatchHyvärinen \+ N\(0,I\) score\([26](https://arxiv.org/html/2605.15394#S3.E26)\)L12CPCtemporal InfoNCE\([27](https://arxiv.org/html/2605.15394#S3.E27)\)L13BYOL\-LLMEMA target \+ predictor\([28](https://arxiv.org/html/2605.15394#S3.E28)\)L14I\-JEPA\-LLMmasked\-block predict\([29](https://arxiv.org/html/2605.15394#S3.E29)\)Three conceptual classes organise the discussion that follows\.*Trajectory\-shape auxiliaries*\(STP, T1–T6\) penalise directional misalignment, curvature, or higher\-order differences across the assistant span, drawing on the geodesic intuition behind STP\[[16](https://arxiv.org/html/2605.15394#bib.bib2)\]\.*Contrastive and predictive auxiliaries*\(T7\[[7](https://arxiv.org/html/2605.15394#bib.bib7)\], L12\) introduce discriminative structure, either across span halves or across positions\.*Distributional and predictor\-based auxiliaries*\(L1–L6, L9, L13, L14\) project hidden states or tangents through a sketcher and either enforce isotropy \(via Cramér–Wold CF distance, sliced\-Wasserstein, or score matching\) or learn a predictor against an EMA / fixed target that probes whether CE’s implicit\-bias direction\[[25](https://arxiv.org/html/2605.15394#bib.bib13)\]can be supplemented by an anti\-collapse prior\.

### III\-BSTP – First\-Order Direction Alignment

For each example we sample four indicesi1<i2<i3<i4i\_\{1\}<i\_\{2\}<i\_\{3\}<i\_\{4\}inside the assistant span and form

ub=hb,i2−hb,i1,vb=hb,i4−hb,i3\.u\_\{b\}=h\_\{b,i\_\{2\}\}\-h\_\{b,i\_\{1\}\},\\qquad v\_\{b\}=h\_\{b,i\_\{4\}\}\-h\_\{b,i\_\{3\}\}\.\(6\)The STP loss is the directional misalignment ofuuandvv:

ℒSTP=1B​∑b\[1−⟨ub,vb⟩‖ub‖​‖vb‖\]\.\\mathcal\{L\}\_\{\\text\{STP\}\}\\;=\\;\\frac\{1\}\{B\}\\sum\_\{b\}\\left\[1\-\\frac\{\\left\\langle u\_\{b\},\\,v\_\{b\}\\right\\rangle\}\{\\left\\lVert u\_\{b\}\\right\\rVert\\,\\left\\lVert v\_\{b\}\\right\\rVert\}\\right\]\.\(7\)Because both vectors are unit\-normalised,ℒSTP\\mathcal\{L\}\_\{\\text\{STP\}\}depends only on the angle between two velocity vectors of the trajectory; its gradient is perpendicular to each velocity and vanishes when the trajectory traces a fixed\-direction ray\.

### III\-CT1 – Curvature\-Aware Tube \(C\-Tube\)

Given four indicess<p<q<ts<p<q<t, define the chordτb=hb,t−hb,s\\tau\_\{b\}=h\_\{b,t\}\-h\_\{b,s\}and the average central second difference

db\(2\)=\\displaystyle d\_\{b\}^\{\(2\)\}\\;=12​\[\(hb,q−hb,p\)−\(hb,p−hb,s\)\]\\displaystyle\\tfrac\{1\}\{2\}\\bigl\[\(h\_\{b,q\}\-h\_\{b,p\}\)\-\(h\_\{b,p\}\-h\_\{b,s\}\)\\bigr\]\(8\)\+12​\[\(hb,t−hb,q\)−\(hb,q−hb,p\)\]\.\\displaystyle\{\}\+\\tfrac\{1\}\{2\}\\bigl\[\(h\_\{b,t\}\-h\_\{b,q\}\)\-\(h\_\{b,q\}\-h\_\{b,p\}\)\\bigr\]\.Letκb=db\(2\)−⟨db\(2\),τb⟩‖τb‖2​τb\\kappa\_\{b\}=d\_\{b\}^\{\(2\)\}\-\\frac\{\\left\\langle d\_\{b\}^\{\(2\)\},\\,\\tau\_\{b\}\\right\\rangle\}\{\\left\\lVert\\tau\_\{b\}\\right\\rVert^\{2\}\}\\tau\_\{b\}be its component orthogonal to the chord\. Then

ℒT1=1B​∑b‖κb‖22\.\\mathcal\{L\}\_\{\\text\{T1\}\}\\;=\\;\\frac\{1\}\{B\}\\sum\_\{b\}\\left\\lVert\\kappa\_\{b\}\\right\\rVert\_\{2\}^\{2\}\.\(9\)ℒT1=0\\mathcal\{L\}\_\{\\text\{T1\}\}=0exactly when the trajectory is a discrete geodesic \(zero geodesic curvature\)\. For aC2C^\{2\}trajectory sampled at spacingΔ\\Delta, the second\-difference estimatordb\(2\)d\_\{b\}^\{\(2\)\}is consistent for the geodesic curvature with relative bias𝒪​\(Δ2\)\\mathcal\{O\}\(\\Delta^\{2\}\), so the loss value‖κb‖2\\left\\lVert\\kappa\_\{b\}\\right\\rVert^\{2\}scales asΔ4​‖κgeo‖2\+𝒪​\(Δ6\)\\Delta^\{4\}\\left\\lVert\\kappa\_\{\\text\{geo\}\}\\right\\rVert^\{2\}\+\\mathcal\{O\}\(\\Delta^\{6\}\)\. The corresponding STP loss scales asΔ2​κgeo2\\Delta^\{2\}\\kappa\_\{\\text\{geo\}\}^\{2\}, so on a fixed\-curvature trajectoryℒT1/ℒSTP\\mathcal\{L\}\_\{\\text\{T1\}\}/\\mathcal\{L\}\_\{\\text\{STP\}\}vanishes as𝒪​\(Δ2\)\\mathcal\{O\}\(\\Delta^\{2\}\)asΔ→0\\Delta\\to 0\.

### III\-DT2 – Riemannian\-Metric Tube \(RIG\-Tube\)

We replace the Euclidean cosine in \([7](https://arxiv.org/html/2605.15394#S3.E7)\) with a cosine under a learned Riemannian metricgϕ​\(h\)g\_\{\\phi\}\(h\)\. Concretely,gϕ​\(h\)=I\+U​\(h\)​U​\(h\)⊤\+diag​\(exp⁡d​\(h\)\)g\_\{\\phi\}\(h\)=I\+U\(h\)U\(h\)^\{\\top\}\+\\mathrm\{diag\}\(\\exp d\(h\)\)is a low\-rank\-plus\-diagonal symmetric\-positive\-definite \(SPD\) perturbation of the identity, parameterised by a small multi\-layer perceptron \(MLP\) headϕ\\phithat maps a hidden statehhto a rank\-rrfactorU​\(h\)∈ℝD×rU\(h\)\\in\\mathbb\{R\}^\{D\\times r\}and a log\-diagonald​\(h\)∈ℝDd\(h\)\\in\\mathbb\{R\}^\{D\}\. The three additive terms each play a distinct role:*\(i\)*the ambient identityIIguarantees the metric is at least the Euclidean inner product, sogϕ​\(h\)⪰I≻0g\_\{\\phi\}\(h\)\\succeq I\\succ 0holds by construction with no extra constraint; it also recovers plain STP at initialisation whenU≡0U\\equiv 0andd≡−∞d\\equiv\-\\infty\.*\(ii\)*The low\-rank correctionU​\(h\)​U​\(h\)⊤U\(h\)U\(h\)^\{\\top\}concentrates curvature along a learnedrr\-dimensional subspace — the directions in which the model is allowed to lengthen or shorten its tangent inner products without the cost of a fullD×DD\\times Dparameterisation\.*\(iii\)*The positive diagonaldiag​\(exp⁡d​\(h\)\)\\mathrm\{diag\}\(\\exp d\(h\)\)supplies a per\-axis scale that does not have to fit inside the rank\-rrsubspace; the exponential parameterisation enforces strict positivity of every diagonal entry\. Their sum is positive definite as a sum of positive\-definite terms, which avoids any need for projection or clipping during training\. Witha=ht−hra=h\_\{t\}\-h\_\{r\},b=hr−hsb=h\_\{r\}\-h\_\{s\}\(three indicess<r<ts<r<t\), the loss is

ℒT2=𝔼​\[1−⟨a,b⟩gϕ​\(hr\)‖a‖gϕ​\(hr\)​‖b‖gϕ​\(hr\)\],\\mathcal\{L\}\_\{\\text\{T2\}\}\\;=\\;\\mathbb\{E\}\\\!\\left\[1\-\\frac\{\\left\\langle a,\\,b\\right\\rangle\_\{g\_\{\\phi\}\(h\_\{r\}\)\}\}\{\\left\\lVert a\\right\\rVert\_\{g\_\{\\phi\}\(h\_\{r\}\)\}\\,\\left\\lVert b\\right\\rVert\_\{g\_\{\\phi\}\(h\_\{r\}\)\}\}\\right\],\(10\)where⟨x,y⟩g=x⊤​g​y\\left\\langle x,\\,y\\right\\rangle\_\{g\}=x^\{\\top\}g\\,yand‖x‖g=⟨x,x⟩g\\left\\lVert x\\right\\rVert\_\{g\}=\\sqrt\{\\left\\langle x,\\,x\\right\\rangle\_\{g\}\}\. We initialisegϕ≡Ig\_\{\\phi\}\\equiv Iso thatℒT2\\mathcal\{L\}\_\{\\text\{T2\}\}is bit\-identical to plain STP at step zero \(curriculum guarantee: adding T2 cannot hurt before training\)\.

### III\-ET3 – Jacobi\-Field Regulariser \(JFR\)

For a batch sharing a common prefix \(or, in our NL\-RX setting, no shared prefix and the empty\-prefix case\) define the Jacobi\-field residual at positionttasJb​\(t\)=hb,t−h¯​\(t\)J\_\{b\}\(t\)=h\_\{b,t\}\-\\bar\{h\}\(t\)whereh¯​\(t\)=1B​∑bhb,t\\bar\{h\}\(t\)=\\tfrac\{1\}\{B\}\\sum\_\{b\}h\_\{b,t\}is the batch centroid trajectory\. The flat\-manifold limit of the Jacobi equation requiresJ¨b≡0\\ddot\{J\}\_\{b\}\\equiv 0along the trajectory, i\.e\. each residual trajectory is linear\. Discretising with a central 3\-point stencil,

ℒT3=1\|𝒱\|​∑\(b,t\)∈𝒱‖Jb​\(t\+1\)−2​Jb​\(t\)\+Jb​\(t−1\)‖22,\\mathcal\{L\}\_\{\\text\{T3\}\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{V\}\|\}\\\!\\\!\\sum\_\{\(b,t\)\\,\\in\\,\\mathcal\{V\}\}\\\!\\\!\\bigl\\lVert J\_\{b\}\(t\+1\)\-2J\_\{b\}\(t\)\+J\_\{b\}\(t\-1\)\\bigr\\rVert\_\{2\}^\{2\},\(11\)where𝒱=\{\(b,t\):1≤t≤Lb−2\}\\mathcal\{V\}=\\\{\(b,t\):1\\leq t\\leq L\_\{b\}\-2\\\}is the set of interior token positions across the \(variable\-length\) batch and\|𝒱\|=∑b\(Lb−2\)\|\\mathcal\{V\}\|=\\sum\_\{b\}\(L\_\{b\}\-2\)is its cardinality\. The implementation realises\|𝒱\|\|\\mathcal\{V\}\|exactly as the sum of a0/10/1pad mask, so the normalisation matches the equation bit\-for\-bit on variable\-length batches\.ℒT3=0\\mathcal\{L\}\_\{\\text\{T3\}\}=0iff every residual trajectory is exactly linear intt\.

### III\-FT3\-Local – Prompt\-Local JFR with Memory Bank

T3 \([11](https://arxiv.org/html/2605.15394#S3.E11)\) subtracts the batch centroid before applying the second\-difference penalty, but in NL\-RX where examples have unrelated prompts the batch centroid is a noisy tube centre\. T3\-Local replaces the batch mean by a retrieval\-weighted centroid drawn from a small memory bank of past trajectories\. Letℳ=\{\(a\(c\),H\(c\),L\(c\)\)\}c=1M\\mathcal\{M\}=\\\{\(a^\{\(c\)\},H^\{\(c\)\},L^\{\(c\)\}\)\\\}\_\{c=1\}^\{M\}store, for the most recentMMtraining examples, the prompt anchora\(c\)a^\{\(c\)\}\(the user\-message hidden\-state mean\), the assistant trajectoryH\(c\)H^\{\(c\)\}, and its lengthL\(c\)L^\{\(c\)\}\. For examplebbwith anchoraba\_\{b\}, take the top\-kkneighbours by cosine similarity and attention\-weight their trajectories,

h¯b​\(t\)=∑j=1kwb,j​H\(cj\)​\(t\),wb,j=softmaxj​\(⟨ab,a\(cj\)⟩τ​∥ab∥​∥a\(cj\)∥\),\\bar\{h\}\_\{b\}\(t\)=\\sum\_\{j=1\}^\{k\}w\_\{b,j\}\\,H^\{\(c\_\{j\}\)\}\(t\),\\;w\_\{b,j\}=\\mathrm\{softmax\}\_\{j\}\\\!\\Big\(\\tfrac\{\\langle a\_\{b\},\\,a^\{\(c\_\{j\}\)\}\\rangle\}\{\\tau\\,\\lVert a\_\{b\}\\rVert\\,\\lVert a^\{\(c\_\{j\}\)\}\\rVert\}\\Big\),\(12\)and apply the T3 stencil to the residualJb​\(t\)=hb,t−sg​\[h¯b​\(t\)\]J\_\{b\}\(t\)=h\_\{b,t\}\-\\mathrm\{sg\}\[\\bar\{h\}\_\{b\}\(t\)\]:

ℒT3\-Local=1\|𝒱\|​∑\(b,t\)∈𝒱‖Jb​\(t\+1\)−2​Jb​\(t\)\+Jb​\(t−1\)‖22\.\\mathcal\{L\}\_\{\\text\{T3\-Local\}\}=\\frac\{1\}\{\|\\mathcal\{V\}\|\}\\\!\\\!\\sum\_\{\(b,t\)\\in\\mathcal\{V\}\}\\\!\\\!\\bigl\\lVert J\_\{b\}\(t\+1\)\-2J\_\{b\}\(t\)\+J\_\{b\}\(t\-1\)\\bigr\\rVert\_\{2\}^\{2\}\.\(13\)The stop\-gradientsg​\[⋅\]\\mathrm\{sg\}\[\\cdot\]ensures the memory bank acts as a target encoder, not a second backward path: the auxiliary shapes the current trajectory toward the local centroid of stored neighbours without pulling the stored trajectories\. Defaults:k=8k=8,τ=0\.1\\tau=0\.1,M=512M=512\. Conceptually this is the closest variant we test to the EMA\-target asymmetry the JEPA literature identifies as essential \(Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)\)\.

### III\-GT5 – Deep Semantic Tubes \(DST\-JFR\)

Apply the JFR loss at a fixed subsetℒ\\mathcal\{L\}of transformer layers and average uniformly:

ℒT5=1\|ℒ\|​∑ℓ∈ℒℒT3​\(H\(ℓ\)\),\\mathcal\{L\}\_\{\\text\{T5\}\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{L\}\|\}\\sum\_\{\\ell\\in\\mathcal\{L\}\}\\mathcal\{L\}\_\{\\text\{T3\}\}\\\!\\bigl\(H^\{\(\\ell\)\}\\bigr\),\(14\)whereH\(ℓ\)H^\{\(\\ell\)\}is the hidden\-state output of layerℓ\\ell\. Defaultℒ=\{4,8,12,16\}\\mathcal\{L\}=\\\{4,8,12,16\\\}for Llama\-3\.2\-1B \(every fourth transformer layer plus the final\)\. The note’s formulation prescribes meta\-learned per\-layer weightsλℓ\\lambda\_\{\\ell\}; we use uniform weights and defer meta\-gradient to future work\.

### III\-HT6 – Multi\-Scale Tube Bundle \(MSTB\-JFR\)

Generalise the central second difference to a strided stencil at scaleΔ\\Delta,DΔ2​ht=ht\+Δ−2​ht\+ht−Δ,D^\{2\}\_\{\\Delta\}h\_\{t\}\\;=\\;h\_\{t\+\\Delta\}\-2h\_\{t\}\+h\_\{t\-\\Delta\},and average the*Δ\\Delta\-normalised*second differenceDΔ2​ht/Δ2D^\{2\}\_\{\\Delta\}h\_\{t\}/\\Delta^\{2\}across a scale set𝒮\\mathcal\{S\}:

ℒT6=1\|𝒮used\|​∑Δ∈𝒮used1\|𝒱Δ\|​∑\(b,t\)∈𝒱Δ‖Δ−2​DΔ2​hb,t‖22,\\mathcal\{L\}\_\{\\text\{T6\}\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{S\}\_\{\\text\{used\}\}\|\}\\\!\\\!\\sum\_\{\\Delta\\in\\mathcal\{S\}\_\{\\text\{used\}\}\}\\\!\\\!\\frac\{1\}\{\|\\mathcal\{V\}\_\{\\Delta\}\|\}\\\!\\\!\\sum\_\{\(b,t\)\\in\\mathcal\{V\}\_\{\\Delta\}\}\\\!\\\!\\bigl\\lVert\\Delta^\{\-2\}\\,D^\{2\}\_\{\\Delta\}h\_\{b,t\}\\bigr\\rVert\_\{2\}^\{2\},\(15\)where𝒱Δ=\{\(b,t\):Δ≤t≤Lb−1−Δ\}\\mathcal\{V\}\_\{\\Delta\}=\\\{\(b,t\):\\Delta\\leq t\\leq L\_\{b\}\-1\-\\Delta\\\}is the valid\-stencil set for scaleΔ\\Delta, and𝒮used\\mathcal\{S\}\_\{\\text\{used\}\}excludes anyΔ\\Deltafor which𝒱Δ\\mathcal\{V\}\_\{\\Delta\}is empty\. TheΔ−2\\Delta^\{\-2\}normalisation reflects the leading order of the strided second difference for a smooth trajectory,DΔ2​ht≈Δ2​h¨​\(t\)D^\{2\}\_\{\\Delta\}h\_\{t\}\\approx\\Delta^\{2\}\\,\\ddot\{h\}\(t\), so that all scales contribute on a comparable scale and𝒮=\{1\}\\mathcal\{S\}=\\\{1\\\}recovers T3 \([11](https://arxiv.org/html/2605.15394#S3.E11)\) bit\-for\-bit\. Default𝒮=\{1,2,3\}\\mathcal\{S\}=\\\{1,2,3\\\}\(NL\-RX assistant spans are typically5​–​305\\text\{\-\-\}30tokens, so larger scales rarely fit any example\)\.

### III\-IT7 – Contrastive Tube

T7 is the only loss in our menu that injects negative pressure\. For each example, split the assistant span into two halves atmidb=lob\+⌊Lb/2⌋\\text\{mid\}\_\{b\}=\\text\{lo\}\_\{b\}\+\\lfloor L\_\{b\}/2\\rfloorand mean\-pool the hidden states inside each half:

μbA=1\|halfA\|​∑t∈halfAhb,t,μbB=1\|halfB\|​∑t∈halfBhb,t\.\\mu^\{A\}\_\{b\}=\\tfrac\{1\}\{\|\\text\{half\}\_\{A\}\|\}\\\!\\\!\\sum\_\{t\\in\\text\{half\}\_\{A\}\}h\_\{b,t\},\\qquad\\mu^\{B\}\_\{b\}=\\tfrac\{1\}\{\|\\text\{half\}\_\{B\}\|\}\\\!\\\!\\sum\_\{t\\in\\text\{half\}\_\{B\}\}h\_\{b,t\}\.\(16\)A shared 2\-layer MLP projectorg:ℝD→ℝD→ℝPg:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{P\}withP=128P=128produceszbX=g​\(μbX\)z^\{X\}\_\{b\}=g\(\\mu^\{X\}\_\{b\}\), and weL2L\_\{2\}\-normalise toz^bX=zbX/‖zbX‖\\hat\{z\}^\{X\}\_\{b\}=z^\{X\}\_\{b\}/\\left\\lVert z^\{X\}\_\{b\}\\right\\rVert\. With temperatureτ=0\.07\\tau=0\.07the loss is symmetric InfoNCE with in\-batch negatives:

ℒT7\\displaystyle\\mathcal\{L\}\_\{\\text\{T7\}\}=12​B∑b\[CE\(z^bAZ^B⊤/τ,b\)\\displaystyle=\\frac\{1\}\{2B\}\\sum\_\{b\}\\biggl\[\\mathrm\{CE\}\\\!\\bigl\(\\hat\{z\}^\{A\}\_\{b\}\\,\\hat\{Z\}^\{B\\top\}/\\tau,\\,b\\bigr\)\(17\)\+CE\(z^bBZ^A⊤/τ,b\)\],\\displaystyle\\hskip 40\.00006pt\+\\;\\mathrm\{CE\}\\\!\\bigl\(\\hat\{z\}^\{B\}\_\{b\}\\,\\hat\{Z\}^\{A\\top\}/\\tau,\\,b\\bigr\)\\biggr\],whereZ^X=\[z^1X,…,z^BX\]⊤\\hat\{Z\}^\{X\}=\[\\hat\{z\}^\{X\}\_\{1\},\\ldots,\\hat\{z\}^\{X\}\_\{B\}\]^\{\\top\}andCE\\mathrm\{CE\}is the cross\-entropy with class labelbb\(the diagonal\)\.BBhere is the*effective*batch size after dropping examples whose assistant span is too short to produce two non\-empty halves \(median NL\-RX span is 12 tokens, so this dropping is rare in practice\)\. When fewer than two examples survive, the loss is set to zero with attached gradient\. Unlike \([7](https://arxiv.org/html/2605.15394#S3.E7)\)–\([11](https://arxiv.org/html/2605.15394#S3.E11)\), gradient does not vanish when the positive pair is well\-aligned: as long asz^bA\\hat\{z\}^\{A\}\_\{b\}is closer to some other example’sz^b′B\\hat\{z\}^\{B\}\_\{b^\{\\prime\}\}than to its ownz^bB\\hat\{z\}^\{B\}\_\{b\}, the loss is nonzero\.

### III\-JT9 – Tube\-Projected Decoding \(TPD\)

T9 is the single inference\-time intervention we test\. At each greedy decoding step, given the previous projected hidden stateht−1projh\_\{t\-1\}^\{\\,\\text\{proj\}\}and the current model outputhtraw=f​\(x≤t\)h\_\{t\}^\{\\,\\text\{raw\}\}=f\(x\_\{\\leq t\}\), the projector replaceshtrawh\_\{t\}^\{\\,\\text\{raw\}\}with

htproj=ht−1proj\+Rε​\(htraw−ht−1proj\),h\_\{t\}^\{\\,\\text\{proj\}\}\\;=\\;h\_\{t\-1\}^\{\\,\\text\{proj\}\}\\;\+\\;R\_\{\\varepsilon\}\\\!\\bigl\(h\_\{t\}^\{\\,\\text\{raw\}\}\-h\_\{t\-1\}^\{\\,\\text\{proj\}\}\\bigr\),\(18\)where the retractionRε:ℝD→ℝDR\_\{\\varepsilon\}:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\}shrinks the component of its argument orthogonal to the local tangent \(estimated from akk\-step history of projected states\) by a factorα​\(∥v⟂∥/ε\)\\alpha\(\\lVert v\_\{\\perp\}\\rVert/\\varepsilon\)and leaves the tangential component intact\. The tangent direction and tube radiusε\\varepsilonare taken from the trained checkpoint; T9 introduces no additional trainable parameters\. The projected hidden statehtprojh\_\{t\}^\{\\,\\text\{proj\}\}is then fed to the language\-model head for the next\-token argmax, while the key/value cache used by subsequent decoding steps is built from the un\-projected forward pass \(so the projector intervenes only on the LM\-head input, not on the attention state\)\.

#### Why these nine\.

STP and T2 differ in what inner product is used for the cosine\. T1 and T3 differ in which component of the second difference is penalised \(T1 only the orthogonal\-to\-chord part, T3 the full vector\)\. T5 and T6 are depth\- and scale\-extensions of T3 respectively; T3\-Local replaces the batch centroid with a retrieved one and is the closest variant to the EMA\-target asymmetry of the JEPA\-vision lineage\. T7 differs from all seven by being contrastive rather than reconstructive\. T9 differs from all eight training\-time losses by intervening only at decoding\. Together they span the attractor\-vs\-contrastive, first\-vs\-second\-order, depth, scale, retrieval\-augmented\- centroid, and inference\-vs\-training axes that the JEPA literature naturally suggests\.

### III\-KTier\-1: Distributional Auxiliaries Outside CE’s Implicit\-Bias Kernel

Sections[V\-B](https://arxiv.org/html/2605.15394#S5.SS2)and[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)together motivate a sharper question: do auxiliaries that are*provably*not in cross\-entropy’s implicit\-bias direction — because they constrain the*distribution*of hidden states or tangents rather than chord\-alignment — escape the present null? We add four such losses, all distributional rather than geometric, and test them under the same harness as the trajectory\-shape auxiliaries\. The closed\-form building blocks are:

#### Epps–Pulley empirical\-CF distance to𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\.

For a 1\-D sample\{ui\}i=1N\\\{u\_\{i\}\\\}\_\{i=1\}^\{N\}, the Plancherel\-closed\-form of the squared empirical\-characteristic\-function distance to the standard normal under Gaussian weighte−t2e^\{\-t^\{2\}\}is

TEP​\(\{ui\}\)=\\displaystyle T\_\{\\mathrm\{EP\}\}\(\\\{u\_\{i\}\\\}\)\\;=1N2​∑i,je−\(ui−uj\)2/4\\displaystyle\\frac\{1\}\{N^\{2\}\}\\sum\_\{i,j\}e^\{\-\(u\_\{i\}\-u\_\{j\}\)^\{2\}/4\}\(19\)−2​2/3N​∑ie−ui2/6\+12\.\\displaystyle\{\}\-\\;\\frac\{2\\sqrt\{2/3\}\}\{N\}\\sum\_\{i\}e^\{\-u\_\{i\}^\{2\}/6\}\\;\+\\;\\frac\{1\}\{\\sqrt\{2\}\}\.TEP→a\.s\.0T\_\{\\mathrm\{EP\}\}\\xrightarrow\{\\text\{a\.s\.\}\}0iff the sample distribution is𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\. The two non\-pair terms come from the closed\-form integralsEY∼𝒩​\(0,1\)​\[e−\(x−Y\)2/4\]=2/3​e−x2/6E\_\{Y\\sim\\mathcal\{N\}\(0,1\)\}\\\!\\bigl\[e^\{\-\(x\-Y\)^\{2\}/4\}\\bigr\]=\\sqrt\{2/3\}\\,e^\{\-x^\{2\}/6\}andEY,Y′​\[e−\(Y−Y′\)2/4\]=1/2E\_\{Y,Y^\{\\prime\}\}\\\!\\bigl\[e^\{\-\(Y\-Y^\{\\prime\}\)^\{2\}/4\}\\bigr\]=1/\\sqrt\{2\}\.

#### Sketcher\.

A linear projectorψ:ℝD→ℝd′\\psi:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{d^\{\\prime\}\}withd′=64d^\{\\prime\}=64, no bias, initialised either with a small \(∼10−2\\sim 10^\{\-2\}\) Gaussian \(L1, L2\) or with the Xavier scaleD−1/2D^\{\-1/2\}\(L4\)\. For L4 we additionally*freeze*ψ\\psi\(no gradient\), making the projection a random measurement map; an unfrozen L4 projector has a trivial zero\-loss fixed point atψ≡0\\psi\\equiv 0\(both half\-distributions collapse to a delta at the origin\)\.

#### L1 – SIGReg\-State\.

Stack the EOS\-clipped assistant\-span hidden states into the cloud\{hi\}i=1N\\\{h\_\{i\}\\\}\_\{i=1\}^\{N\}, project throughψ\\psi, and apply the Cramér–Wold mean of \([19](https://arxiv.org/html/2605.15394#S3.E19)\) overM=64M=64random unit directions:

ℒL1=1M​∑ℓTEP​\(\{aℓ⊤​ψ​\(hi\)\}i\)\.\\mathcal\{L\}\_\{\\text\{L1\}\}=\\frac\{1\}\{M\}\\sum\_\{\\ell\}T\_\{\\mathrm\{EP\}\}\\\!\\bigl\(\\\{a\_\{\\ell\}^\{\\top\}\\psi\(h\_\{i\}\)\\\}\_\{i\}\\bigr\)\.\(20\)By Cramér–Wold,ℒL1→0\\mathcal\{L\}\_\{\\text\{L1\}\}\\to 0iff the projected cloud is𝒩​\(0,Id′\)\\mathcal\{N\}\(0,I\_\{d^\{\\prime\}\}\)\. This is a*distributional*constraint on the marginal hidden\-state cloud, not on chord/cosine geometry of any single trajectory; by construction it is outside CE’sL2L\_\{2\}\-max\-margin implicit\-bias direction\[[25](https://arxiv.org/html/2605.15394#bib.bib13)\]\.

#### L2 – SIGReg\-Tangent\.

Apply L1 to the unit\-normalised step tangentsΔb,t=\(hb,t\+1−hb,t\)/‖hb,t\+1−hb,t‖\\Delta\_\{b,t\}=\(h\_\{b,t\+1\}\-h\_\{b,t\}\)/\\left\\lVert h\_\{b,t\+1\}\-h\_\{b,t\}\\right\\rVerton the assistant span:

ℒL2=1M​∑ℓTEP​\(\{aℓ⊤​ψ​\(Δi\)\}i\)\.\\mathcal\{L\}\_\{\\text\{L2\}\}=\\frac\{1\}\{M\}\\sum\_\{\\ell\}T\_\{\\mathrm\{EP\}\}\\\!\\bigl\(\\\{a\_\{\\ell\}^\{\\top\}\\psi\(\\Delta\_\{i\}\)\\\}\_\{i\}\\bigr\)\.\(21\)L2 forces the tangent distribution onto the maximum\-entropy distribution on𝕊D−1\\mathbb\{S\}^\{D\-1\}\(in projection\): where T1, T3 fix the trajectory’s*mean*velocity, L2 fixes its distributional*shape*\.

#### L3 – C\-Tube\-Sectional\.

For each example sampleK=4K=4random sorted triples\(sk,rk,tk\)\(s\_\{k\},r\_\{k\},t\_\{k\}\)inside the assistant span and define the discrete sectional curvatureκsec​\(hs,hr,ht\)=‖hr−12​\(hs\+ht\)‖2/14​‖ht−hs‖2\\kappa\_\{\\text\{sec\}\}\(h\_\{s\},h\_\{r\},h\_\{t\}\)=\\left\\lVert h\_\{r\}\-\\tfrac\{1\}\{2\}\(h\_\{s\}\+h\_\{t\}\)\\right\\rVert^\{2\}/\\tfrac\{1\}\{4\}\\left\\lVert h\_\{t\}\-h\_\{s\}\\right\\rVert^\{2\}; the loss is the across\-triple variance,

ℒL3=Var\(b,k\)​\[κsec\]\.\\mathcal\{L\}\_\{\\text\{L3\}\}\\;=\\;\\mathrm\{Var\}\_\{\(b,k\)\}\\\!\\bigl\[\\kappa\_\{\\text\{sec\}\}\\bigr\]\.\(22\)T1 \([9](https://arxiv.org/html/2605.15394#S3.E9)\) drivesκsec\\kappa\_\{\\text\{sec\}\}to zero \(discrete geodesic\); L3 drives its variance to zero \(constant\-curvature local model, in the Killing–Hopf sense\)\.

#### L4 – STP\-CMF\.

Split the unit\-normalised tangents into two halvesΔbA,ΔbB\\Delta^\{A\}\_\{b\},\\Delta^\{B\}\_\{b\}at the span midpoint, project through a*fixed*sketcherψc\\psi\_\{c\}\(trainableψc\\psi\_\{c\}has the trivialψc≡0\\psi\_\{c\}\\equiv 0escape\), and compare the two halves under the same Gaussian\-kernel CF distance as \([19](https://arxiv.org/html/2605.15394#S3.E19)\):

ℒL4=1B​M​∑b,ℓDCF​\(\{aℓ⊤​ψc​\(Δb,jA\)\},\{aℓ⊤​ψc​\(Δb,jB\)\}\),\\mathcal\{L\}\_\{\\text\{L4\}\}=\\frac\{1\}\{BM\}\\sum\_\{b,\\ell\}D\_\{\\text\{CF\}\}\\\!\\bigl\(\\\{a\_\{\\ell\}^\{\\top\}\\psi\_\{c\}\(\\Delta^\{A\}\_\{b,j\}\)\\\},\\\{a\_\{\\ell\}^\{\\top\}\\psi\_\{c\}\(\\Delta^\{B\}\_\{b,j\}\)\\\}\\bigr\),\(23\)withDCF​\(X,Y\)=⟨K⟩X​X−2​⟨K⟩X​Y\+⟨K⟩Y​YD\_\{\\text\{CF\}\}\(X,Y\)=\\langle K\\rangle\_\{XX\}\-2\\langle K\\rangle\_\{XY\}\+\\langle K\\rangle\_\{YY\}andK​\(z\)=e−z2/4K\(z\)=e^\{\-z^\{2\}/4\}\. L4 is a per\-example stationarity test on the tangent distribution: STP \([7](https://arxiv.org/html/2605.15394#S3.E7)\) asks for matched*point*chords; L4 asks for matched*distributions*\.

#### Why these four\.

Each provably cannot be satisfied by motion in CE’s implicit\-bias direction alone, sidestepping the orthogonality observation of Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)\. L1, L2 instantiate the LeJEPA prescription at two natural target distributions \(state, tangent\); L3 tests whether the geometry/distribution distinction matters at second order; L4 lifts STP from a per\-triple to a per\-trajectory distributional statement\.

### III\-LTier\-2: Predictor\-Based and Density\-Based Auxiliaries

The Tier\-1 question is whether*any*distributional constraint outside CE’s implicit\-bias direction lifts exact\-match accuracy\. To make that test less specific to a single distance choice \(Cramér–Wold via Epps–Pulley\) and to bring in the architectural component that Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)\-\(4\) names as the missing JEPA core — the predictor \+ EMA\-target \+ stop\-gradient triple — we add six further auxiliaries grouped into three substructural families:

*Distributional probes other than the empirical characteristic function\.*L5 \(VICReg variance–covariance\) and L6 \(sliced\-Wasserstein isotropy\) reuse L1’s projector geometry but swap the per\-direction Epps–Pulley CF distance for the two best\-known anti\-collapse alternatives\. L9 \(score matching\) uses Hyvärinen sliced score matching to estimate the score of the projected hidden\-state distribution and an additive deviation\-from\-𝒩​\(0,I\)\\mathcal\{N\}\(0,I\)score penalty\.

*Predictive contrastive structure\.*L12 \(CPC\) asks the auxiliary head to predictht\+kh\_\{t\+k\}fromhth\_\{t\}via a small linear predictor, with negatives drawn from the same position of every other batch example\.

*Predictor \+ target\-encoder asymmetry\.*L13 \(BYOL\-LLM\) and L14 \(I\-JEPA\-LLM\) carry the predictor \+ stop\-gradient \+ EMA\-target triple into our pipeline\. L13 uses mean\-pooled span halves as positive pairs with an EMA\-tracked target projector; L14 masks a contiguous block of assistant\-span positions and predicts the masked targets from a pooled summary of the visible positions\.

#### L5 – VICReg\-VC\.

Withz=ψ​\(Hflat\)z=\\psi\(H\_\{\\text\{flat\}\}\)from the L1 sketcher, combine a variance hinge with an off\-diagonal covariance penalty:

ℒL5=\\displaystyle\\mathcal\{L\}\_\{\\text\{L5\}\}=1d′​∑jmax⁡\(0,1−Var​\(z:,j\)\+ε\)\\displaystyle\\tfrac\{1\}\{d^\{\\prime\}\}\\sum\_\{j\}\\max\\\!\\bigl\(0,1\-\\sqrt\{\\mathrm\{Var\}\(z\_\{:,j\}\)\+\\varepsilon\}\\bigr\)\(24\)\+μ⋅1d′​\(d′−1\)​∑j≠kCj,k2,\\displaystyle\+\\mu\\cdot\\tfrac\{1\}\{d^\{\\prime\}\(d^\{\\prime\}\-1\)\}\\sum\_\{j\\neq k\}C\_\{j,k\}^\{2\},withCCthe empirical covariance ofzzandμ=1\\mu=1the standard VICReg weight\. Same projector geometry as L1; the distributional probe is direct second\-moment matching rather than a CF distance\.

#### L6 – Sliced\-Wasserstein Isotropy\.

ForMMrandom unit directionsaℓa\_\{\\ell\}, sort the projected samplesuℓ=sort​\(\{aℓ⊤​ψ​\(hi\)\}\)u\_\{\\ell\}=\\mathrm\{sort\}\(\\\{a\_\{\\ell\}^\{\\top\}\\psi\(h\_\{i\}\)\\\}\)and compare to the standard\-normal quantiles at midpointsqi=Φ−1​\(\(i−1/2\)/N\)q\_\{i\}=\\Phi^\{\-1\}\(\(i\-1/2\)/N\):

ℒL6=1M​∑ℓ1N​∑i\(uℓ\(i\)−qi\)2\.\\mathcal\{L\}\_\{\\text\{L6\}\}=\\tfrac\{1\}\{M\}\\sum\_\{\\ell\}\\tfrac\{1\}\{N\}\\sum\_\{i\}\(u\_\{\\ell\}^\{\(i\)\}\-q\_\{i\}\)^\{2\}\.\(25\)This is the midpoint\-quadrature estimator ofW22​\(Fn,𝒩​\(0,1\)\)W\_\{2\}^\{2\}\(F\_\{n\},\\mathcal\{N\}\(0,1\)\)\[[6](https://arxiv.org/html/2605.15394#bib.bib33),[18](https://arxiv.org/html/2605.15394#bib.bib34)\]; costO​\(M​N​log⁡N\)O\(MN\\log N\)vs\. L1’sO​\(M​N2\)O\(MN^\{2\}\)\. Same null as L1\.

#### L9 – Score Matching\.

Train a score netsθ:ℝD→ℝDs\_\{\\theta\}:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\}via Hyvärinen’s identity\[[17](https://arxiv.org/html/2605.15394#bib.bib35)\]and add a deviation\-from\-𝒩​\(0,I\)\\mathcal\{N\}\(0,I\)penalty:

ℒL9=\\displaystyle\\mathcal\{L\}\_\{\\text\{L9\}\}=𝔼z​\[12​‖sθ​\(z\)‖2\+tr​∇zsθ​\(z\)\]\\displaystyle\\mathbb\{E\}\_\{z\}\\\!\\bigl\[\\tfrac\{1\}\{2\}\\left\\lVert s\_\{\\theta\}\(z\)\\right\\rVert^\{2\}\+\\mathrm\{tr\}\\,\\nabla\_\{z\}s\_\{\\theta\}\(z\)\\bigr\]\(26\)\+λSM​𝔼z​\[‖sθ​\(z\)\+z‖2\],\\displaystyle\+\\lambda\_\{\\text\{SM\}\}\\,\\mathbb\{E\}\_\{z\}\\\!\\bigl\[\\left\\lVert s\_\{\\theta\}\(z\)\+z\\right\\rVert^\{2\}\\bigr\],withtr​∇zsθ\\mathrm\{tr\}\\,\\nabla\_\{z\}s\_\{\\theta\}estimated by Hutchinson with one Rademacher probe per step\[[24](https://arxiv.org/html/2605.15394#bib.bib36)\]\. The first term recovers∇zlog⁡p​\(z\)\\nabla\_\{z\}\\log p\(z\)at its minimum; the second is the only piece that pushes the backbone toward isotropy\.

#### L12 – Contrastive Predictive Coding\.

Withgk:ℝD→ℝPg\_\{k\}:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{P\}a learned linear predictor andttthe midpoint of each example’s valid range, train an in\-batch InfoNCE betweenhb,th\_\{b,t\}andhb,t\+kh\_\{b,t\+k\}with negatives drawn from the same position of every other batch example:

ℒL12=−1B′​∑blog⁡exp⁡\(gk​\(hb,t\)⊤​hb,t\+k/τ\)∑b′exp⁡\(gk​\(hb,t\)⊤​hb′,t\+k/τ\),\\mathcal\{L\}\_\{\\text\{L12\}\}=\-\\tfrac\{1\}\{B^\{\\prime\}\}\\sum\_\{b\}\\log\\\!\\frac\{\\exp\(g\_\{k\}\(h\_\{b,t\}\)^\{\\top\}h\_\{b,t\+k\}/\\tau\)\}\{\\sum\_\{b^\{\\prime\}\}\\exp\(g\_\{k\}\(h\_\{b,t\}\)^\{\\top\}h\_\{b^\{\\prime\},t\+k\}/\\tau\)\},\(27\)τ=0\.07\\tau=0\.07\. L12 differs from T7 \(Section[III\-I](https://arxiv.org/html/2605.15394#S3.SS9)\) on two axes: contrast is across*positions*not span halves, and the structure is predictive not reconstructive\.

#### L13 – BYOL\-LLM\.

Mean\-pool each example’s assistant span into halvesμbA,μbB\\mu^\{A\}\_\{b\},\\mu^\{B\}\_\{b\}\. With an online projectorfθf\_\{\\theta\}and predictorqϕq\_\{\\phi\}and a target projectorfθemaf\_\{\\theta\_\{\\mathrm\{ema\}\}\}updated by EMA \(τ=0\.996\\tau=0\.996\), the loss is symmetric BYOL:

ℒL13=\\displaystyle\\mathcal\{L\}\_\{\\text\{L13\}\}=12​B′∑b\[∥qϕ​\(fθ​\(μbA\)\)^−sgfθema​\(μbB\)^∥2\\displaystyle\\tfrac\{1\}\{2B^\{\\prime\}\}\\sum\_\{b\}\\\!\\bigl\[\\left\\lVert\\widehat\{q\_\{\\phi\}\(f\_\{\\theta\}\(\\mu^\{A\}\_\{b\}\)\)\}\-\\mathrm\{sg\}\\,\\widehat\{f\_\{\\theta\_\{\\mathrm\{ema\}\}\}\(\\mu^\{B\}\_\{b\}\)\}\\right\\rVert^\{2\}\(28\)\+\(A↔B\)\],\\displaystyle\\quad\+\(A\\leftrightarrow B\)\\bigr\],x^=x/‖x‖\\widehat\{x\}=x/\\left\\lVert x\\right\\rVert\. L13 is the only auxiliary we test containing the predictor \+ EMA\-target \+ stop\-gradient triple the JEPA literature identifies as the missing JEPA core\.

#### L14 – I\-JEPA\-LLM \(Single\-Pass Simplification\)\.

For each example, mask a contiguous blockℳb\\mathcal\{M\}\_\{b\}\(ρ=0\.25\\rho=0\.25of the assistant span\), pool the visible positions intoh¯bvis\\bar\{h\}^\{\\mathrm\{vis\}\}\_\{b\}, and train a predictorqϕq\_\{\\phi\}to predict frozen Xavier\-target\-projected masked states:

ℒL14=1\|ℳ\|​∑\(b,t\)∈ℳ‖qϕ​\(h¯bvis,posemb​\(t\)\)−sg​f⋆​\(hb,t\)‖2\.\\mathcal\{L\}\_\{\\text\{L14\}\}=\\tfrac\{1\}\{\|\\mathcal\{M\}\|\}\\\!\\\!\\sum\_\{\(b,t\)\\in\\mathcal\{M\}\}\\\!\\\!\\left\\lVert q\_\{\\phi\}\(\\bar\{h\}^\{\\mathrm\{vis\}\}\_\{b\},\\mathrm\{posemb\}\(t\)\)\-\\mathrm\{sg\}\\,f\_\{\\star\}\(h\_\{b,t\}\)\\right\\rVert^\{2\}\.\(29\)Faithful I\-JEPA would use an EMA backbone with a second forward pass; the single\-pass approximation freezesf⋆f\_\{\\star\}at random init for the same anti\-collapse reason as L4 \(a trainablef⋆f\_\{\\star\}has the trivialf⋆≡0f\_\{\\star\}\\equiv 0escape\)\.

#### Why these six\.

The Tier\-2 set tightens the parsimonious reading along the two axes Tier\-1 left underspecified: which distance one chooses for the distributional probe \(L5 vs L6 vs L1; L9 adds a score\-based probe\), and whether the missing predictor \+ target asymmetry is the active ingredient in the JEPA\-vision lift \(L12, L13, L14\)\. L13 in particular is the single experiment whose result is most directly informative about Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)\-\(4\): a clear lift would imply the predictor \+ EMA\-target asymmetry is what made LLM\-JEPA work\[[15](https://arxiv.org/html/2605.15394#bib.bib3)\], and a null tightens the reading further\. As reported in Section[V\-C](https://arxiv.org/html/2605.15394#S5.SS3), the L13 cell atn=3n=3on TURK lands atΔ=\+0\.60\\Delta=\+0\.60\(ppaired=0\.64p\_\{\\text\{paired\}\}=0\.64\); the asymmetry mechanism therefore does not, on its own, lift exact match in this single\-pass harness\.

## IVExperimental Setup

#### Datasets\.

We use NL\-RX\-TURK \(ntrain=8000n\_\{\\text\{train\}\}=8000,ntest=2000n\_\{\\text\{test\}\}=2000, evaluated on the first 500 test examples\) and NL\-RX\-SYNTH \(same sizes\), both following the chat\-message format used by\[[16](https://arxiv.org/html/2605.15394#bib.bib2)\]: a system prompt“Convert natural language to regular expression\.”, a user description, and an assistant regex\.

#### Model\.

Llama\-3\.2\-1B\-Instruct fine\-tuned with LoRA\[[14](https://arxiv.org/html/2605.15394#bib.bib6)\], rankr=32r=32,α=64\\alpha=64, dropout0, applied to all attention and multi\-layer\-perceptron projections \(`q,k,v,o,gate,up,down`\)\. The base model is frozen; only LoRA adapters and any auxiliary head are trained\. Mixed\-precision bf16 inference and training, with gradient checkpointing enabled\. Optimiser: AdamW, peak learning rate2×10−42\\times 10^\{\-4\}, weight decay0\.010\.01, cosine schedule\.

#### Training schedule\.

Batch size44,44epochs, max sequence length256256\. Each\(experiment,seed\)\(\\text\{experiment\},\\text\{seed\}\)cell trains in roughly50​–​6050\\text\{\-\-\}60minutes on a single workstation; the full sweep fits in approximately one machine\-day\.

#### λ\\lambdaschedule\.

Each auxiliary loss is weighted by the schedule

λ​\(t\)=\{λ0​tTwt<Twλ0Tw≤t<T−Tdλ0​\(1−\(1−ρ\)​t−\(T−Td\)Td\)t≥T−Td\\lambda\(t\)=\\begin\{cases\}\\lambda\_\{0\}\\,\\dfrac\{t\}\{T\_\{w\}\}&t<T\_\{w\}\\\\ \\lambda\_\{0\}&T\_\{w\}\\leq t<T\-T\_\{d\}\\\\ \\lambda\_\{0\}\\\!\\left\(1\-\(1\{\-\}\\rho\)\\dfrac\{t\-\(T\{\-\}T\_\{d\}\)\}\{T\_\{d\}\}\\right\)&t\\geq T\-T\_\{d\}\\end\{cases\}\(30\)with warmup fractionTw/T=0\.25T\_\{w\}/T=0\.25, decay fractionTd/T=0\.25T\_\{d\}/T=0\.25, and floor ratioρ=0\.1\\rho=0\.1\(so the final weight isλ0​ρ\\lambda\_\{0\}\\rho\)\. The warmup keeps the cross\-entropy clean early so the end\-of\-turn classifier locks in; the decay keeps the final logits dominated by the cross\-entropy loss\. Peak weights:λ0=1\.0\\lambda\_\{0\}=1\.0for STP, T1, T2, T5, T7;λ0∈\{3×10−4,10−3\}\\lambda\_\{0\}\\in\\\{3\\times 10^\{\-4\},\\,10^\{\-3\}\\\}for T3, whose squared\-second\-difference output is roughly103×10^\{3\}\\timeslarger than the cosine\-based losses \(the two values come from theλ\\lambdasweep in Section[V\-B](https://arxiv.org/html/2605.15394#S5.SS2)\)\.

#### Auxiliary span clip \(the “EOS clip”\)\.

Under causal\-LM conventionhLb−1h\_\{L\_\{b\}\-1\}predicts the end\-of\-turn marker, so any geometric auxiliary acting on it competes with the LM head’s EOS classifier on the same vector\. Without intervention this produced catastrophic EOS suppression empirically: every auxiliary variant fell to0%0\\%exact\-match accuracy while the baseline produced4\.24\.2–35\.8%35\.8\\%\. We therefore clip the right end of the assistant span bymargin=2\\mathrm\{margin\}=2tokens before passing it to any geometric auxiliary, removing bothhLb−1h\_\{L\_\{b\}\-1\}and the position immediately upstream of it in attention,hLb−2h\_\{L\_\{b\}\-2\}\.margin=1\\mathrm\{margin\}=1was tried first and proved insufficient;margin≥3\\mathrm\{margin\}\\geq 3is unnecessary on NL\-RX, whose targets do not end in a multi\-token sentinel\. The clip is applied identically to every variant under the label*EOS\-clipped span*\. The token\-margin hinge of decoder\-visible JEPA \(Appendix[C](https://arxiv.org/html/2605.15394#A3)\) is the single exception: its gradient lies inside cross\-entropy’s positive cone by construction, so it cooperates with the EOS classifier and is not clipped\. Figure[2](https://arxiv.org/html/2605.15394#S4.F2)illustrates the construction\.

![Refer to caption](https://arxiv.org/html/2605.15394v1/x2.png)Figure 2:Visibility of each loss inside the assistant span\.hL−1h\_\{L\-1\}predicts EOS directly;hL−2h\_\{L\-2\}feeds it via self\-attention\. Geometric auxiliaries see only the EOS\-clipped span\{h0,…,hL−3\}\\\{h\_\{0\},\\dots,h\_\{L\-3\}\\\}; cross\-entropy and the decoder\-visible margin hinge see the full span and supervise EOS\.
#### Metrics\.

*Exact\-match accuracy*is strict string equality between the greedy completion and the gold regex \(modulo`skip\_special\_tokens=True`during decoding\)\.*Prefix accuracy*is whether the gold regex is a prefix of the generated completion; reporting both decouples the “learned the regex” signal from the “learned to stop” signal\. Each\(experiment,seed\)\(\\text\{experiment\},\\,\\text\{seed\}\)cell is evaluated on the first500500test examples; we report mean and sample standard deviation over the seeds available for that cell\.

### IV\-AStatistical Methodology

For each\(variant,seed\)\(\\text\{variant\},\\,\\text\{seed\}\)cell we report two two\-sided test statistics against the matched no\-auxiliary baseline: the unpaired Welch’stt\-testt=\(x¯−y¯\)/sx2/nx\+sy2/nyt=\(\\bar\{x\}\-\\bar\{y\}\)/\\sqrt\{s\_\{x\}^\{2\}/n\_\{x\}\+s\_\{y\}^\{2\}/n\_\{y\}\}with Welch–Satterthwaite df, and the paired\-by\-seed Welch’stt\-testt=d¯/\(sd/n\)t=\\bar\{d\}/\(s\_\{d\}/\\sqrt\{n\}\)withn−1n\-1df on the per\-seed differencesdi=xi−yid\_\{i\}=x\_\{i\}\-y\_\{i\}\. Both are reported because they can disagree under lownnwhen seed\-level baseline noise dominates the auxiliary effect\.

We adoptα=0\.10\\alpha=0\.10as the single\-cell rejection threshold and apply Bonferroni / Holm–Bonferroni atα/k\\alpha/kwhenever a family of related cells is interpreted jointly \(e\.g\. the four\-cell Tier\-1 SYNTH family atα/4=0\.025\\alpha/4=0\.025\)\. Atn=3n=3the paired test hasν=2\\nu=2df: theα=0\.05\\alpha=0\.05criticalt=4\.30t=4\.30requires∼3\.7\\sim 3\.7pp at the observed within\-cell sd \(∼1\.5\\sim 1\.5pp\), whereasα=0\.10\\alpha=0\.10admits∼2\.5\\sim 2\.5pp and the largest mean lifts in the study \(e\.g\. T3\-Local at\+2\.53\+2\.53pp\) become detectable\. We treat single\-cellα=0\.10\\alpha=0\.10as a compute\-gating decision for follow\-up, not as a publication\-grade standard\.

Cells with only one completed seed are reported under the*escalation protocol*: a single seed is used to decide whether to expand a variant ton=3n=3, but nott\-test,pp\-value, or significance claim is made onn=1n=1alone\. These cells are flagged in every table and excluded from family\-level corrections\.

#### A note on significance testing as a methodological choice\.

Most evaluations of representation\-learning auxiliaries in the fine\-tuning literature report point estimates and seed standard deviations without seed\-level paired significance testing or family\-wise correction\. Under that convention several auxiliaries in this study would have positive face value: the largest single\-seed cells \(e\.g\.\+2\.73\+2\.73pp on L5 atn=1n=1, the\+2\.53\+2\.53pp seed\-1 lift on T3\-Local\) would be reported as headline gains, and seed\-1 leaders that collapse on expansion ton=3n=3\(Section[V\-A](https://arxiv.org/html/2605.15394#S5.SS1)\) would not be flagged\. The conclusions in this paper are therefore a function of the test budget — dataset, seed countn=3n=3,α=0\.10\\alpha=0\.10single\-cell threshold, and Bonferroni / Holm atα/k\\alpha/kfor joint families — and would shift under a different budget\. Readers should treat the auxiliaries listed here as*practical tools for fine\-tuning*, whose value is contingent on dataset and compute envelope, rather than as controlled, budget\-independent results\.

## VResults

Tables[III](https://arxiv.org/html/2605.15394#S5.T3)and[IV](https://arxiv.org/html/2605.15394#S5.T4)report exact\-match and prefix\-match accuracy on TURK and SYNTH respectively\. The TURK table is the primary result: its baseline is not saturated, and it includes the full set of twelve training\-time auxiliaries plus T9\. SYNTH is used as a secondary check because the no\-auxiliary baseline is already near its ceiling\.

TABLE III:Exact\-match and prefix accuracy on NL\-RX\-TURK atn=3n=3seeds, with unpaired and paired Welch’stt\-testpp\-values against the no\-auxiliary baseline\.TABLE IV:Exact\-match and prefix accuracy on NL\-RX\-SYNTH atn=3n=3seeds, with unpaired and paired Welch’stt\-testpp\-values against the no\-auxiliary baseline\.### V\-AMain TURK Results

Table[III](https://arxiv.org/html/2605.15394#S5.T3)reports the study’s primary empirical content\. The no\-auxiliary baseline reaches50\.67±1\.68%50\.67\\pm 1\.68\\%exact match\. The strongest mean training\-time cell atn=3n=3is T3\-Local at53\.20±1\.91%53\.20\\pm 1\.91\\%\(\+2\.53\+2\.53pp\), with the swept single\-layer T3 atλ0=3×10−4\\lambda\_\{0\}=3\\times 10^\{\-4\}next \(52\.93±0\.64%52\.93\\pm 0\.64\\%,\+2\.27\+2\.27pp\)\. Under our pre\-registeredα=0\.10\\alpha=0\.10rejection threshold \(Section[IV\-A](https://arxiv.org/html/2605.15394#S4.SS1)\), no completed training\-time auxiliary clears the unpaired Welch test on TURK\. The two pairedtt\-tests that do crossα=0\.10\\alpha=0\.10are T3\-Local \(ppaired=0\.003p\_\{\\text\{paired\}\}=0\.003\) and T5 DST\-JFR \(ppaired=0\.08p\_\{\\text\{paired\}\}=0\.08\); neither cell’s unpaired test rejects\.

This is not equivalent to “the losses do nothing”\. Most non\-collapsed auxiliaries reduce seed\-to\-seed variance relative to the baseline, with standard deviations falling well below the baseline’s1\.681\.68pp \(the L1 SIGReg\-State cell tightens to0\.530\.53pp, T3 atλ0=3×10−4\\lambda\_\{0\}=3\\times 10^\{\-4\}to0\.640\.64pp\)\. The most cautious reading is that the auxiliaries regularise the hypothesis space without reliably shifting its task\-metric mean\.

### V\-BWithin\-Family Analysis: Jacobi\-Field Variants

The Jacobi\-field family produces the largest mean lifts in Table[III](https://arxiv.org/html/2605.15394#S5.T3)and therefore admits the tightest within\-family comparison\. At the matchedλ0=10−3\\lambda\_\{0\}=10^\{\-3\}setting we observe the depth/scale ordering

\+1\.53⏟T5 \(depth\)<\+1\.86⏟T3 \(single\)<\+2\.13⏟T6 \(multi\-scale\)pp\.\\underbrace\{\+1\.53\}\_\{\\text\{T5 \(depth\)\}\}\\,<\\,\\underbrace\{\+1\.86\}\_\{\\text\{T3 \(single\)\}\}\\,<\\,\\underbrace\{\+2\.13\}\_\{\\text\{T6 \(multi\-scale\)\}\}\\quad\\text\{pp\.\}Lowering the single\-layer T3 weight to3×10−43\\times 10^\{\-4\}yields the best pure\-JFR cell \(\+2\.27\+2\.27pp\); replacing the batch centroid with a retrieval\-weighted prompt\-local centroid \(T3\-Local\) yields the largest mean overall \(\+2\.53\+2\.53pp\)\. None of these within\-family deltas crosses the unpairedα=0\.10\\alpha=0\.10threshold; the T3\-Local paired test reachesppaired=0\.003p\_\{\\text\{paired\}\}=0\.003, but with onlyn=3n=3this corresponds toν=2\\nu=2degrees of freedom and a single\-cell paired result that is not robust to the multiple\-comparison correction implied by the five JFR\-family rows in the table\.

Two structural conclusions follow, both qualified byn=3n=3\.*\(i\) Depth averaging does not automatically help\.*T5 underperforms matched single\-layer T3 at the sameλ0\\lambda\_\{0\}, consistent with the hypothesis that per\-layer auxiliary gradients are highly correlated under the cross\-entropy backbone, so that the depth average spends the same budget on redundant directions and attenuates the final\-layer signal\.*\(ii\) Scale mixing and prompt\-local retrieval add small but consistently positive mean shifts\.*T6 \(multi\-scale\) and T3\-Local are the two largest means among the JFR family, in line with the intuition that scale\-invariance and EMA/prompt\-local target asymmetry inject information the single\-scale, batch\-centroid T3 loss does not\. The most informative next step is not a sixth JFR variant but increased seed counts and data\-efficiency curves that can adjudicate whether these small mean shifts are stable\.

### V\-CContrastive and Distributional Auxiliaries

The contrastive variant T7 was meant to test whether negative pressure breaks out of the attractor\-loss regime\. It does not: T7 lands at\+0\.07\+0\.07pp on TURK withp≈0\.95p\\approx 0\.95\. At batch size44, this is not a decisive statement about contrastive learning in general, but it is a decisive statement about this small\-batch LoRA harness\.

The distributional auxiliaries L1–L4 were designed as a stronger escape test\. Unlike STP/T1/T3\-style trajectory losses, they constrain marginal or tangent distributions and were explicitly constructed to fall outside cross\-entropy’s implicit\-bias direction\. They also sit on baseline:Δexact=\+0\.20\\Delta\_\{\\text\{exact\}\}=\+0\.20to\+0\.73\+0\.73pp, with every paired and unpaired test atp≥0\.47p\\geq 0\.47\. This result is important because it rules out the simple explanation that the original geometric losses failed only because they were too aligned with cross\-entropy\. Even the out\-of\-kernel distributional losses do not move exact match\.

#### Tier\-2 results: the JEPA\-core test and the escalation protocol\.

All six Tier\-2 cells are now atn=3n=3on TURK, completing the training\-time grid at three seeds\. Two findings are worth flagging explicitly\.*First*, the largest single\-seed lift, L5 VICReg\-VC at\+2\.73\+2\.73pp on seed 0,*collapses*on expansion to ann=3n=3mean of51\.07±2\.40%51\.07\\pm 2\.40\\%\(Δ=\+0\.40\\Delta=\+0\.40,ppaired=0\.88p\_\{\\text\{paired\}\}=0\.88\)\. The seed\-0 mark was a single\-sample artefact\. We treat this as the clearest demonstration of why pre\-registered escalation matters in small\-nnexperiments: a static single\-seed table would have reportedΔexact=\+2\.73\\Delta\_\{\\text\{exact\}\}=\+2\.73as the apparent leader, and the protocol would have been wrong\.*Second*, L13 BYOL\-LLM — the only cell we test containing the predictor \+ EMA\-target \+ stop\-gradient triple that Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)\-\(4\) names as the JEPA\-core ingredient missing from STP and L1–L4 — lands at51\.27±0\.70%51\.27\\pm 0\.70\\%on TURK \(Δ=\+0\.60\\Delta=\+0\.60,ppaired=0\.64p\_\{\\text\{paired\}\}=0\.64\) and88\.93±0\.42%88\.93\\pm 0\.42\\%on SYNTH \(Δ=0\.00\\Delta=0\.00,ppaired=1\.00p\_\{\\text\{paired\}\}=1\.00\)\. The predictor \+ target asymmetry that lifted the saturated SYNTH baseline by∼6\\sim\\\!6pp in LLM\-JEPA\[[15](https://arxiv.org/html/2605.15394#bib.bib3)\]does*not*produce a detectable lift on either benchmark in our single\-pass harness\. L14 I\-JEPA\-LLM, the second predictor \+ frozen\-target cell, also sits inside seed noise on both benchmarks \(ΔTURK=−1\.13\\Delta\_\{\\text\{TURK\}\}=\-1\.13,ΔSYNTH=−0\.07\\Delta\_\{\\text\{SYNTH\}\}=\-0\.07\)\. The remaining three Tier\-2 cells spanΔ∈\[−1\.60,\+0\.60\]\\Delta\\in\[\-1\.60,\+0\.60\]on TURK with every paired and unpaired test atp≥0\.24p\\geq 0\.24: L6 SW\-Iso \(\+0\.60\+0\.60,ppaired=0\.48p\_\{\\text\{paired\}\}=0\.48\), L12 CPC \(\+0\.20\+0\.20,ppaired=0\.77p\_\{\\text\{paired\}\}=0\.77\), L9 Score\-Match \(−1\.60\-1\.60,ppaired=0\.35p\_\{\\text\{paired\}\}=0\.35\)\. No completed Tier\-2 cell on either benchmark clearsα=0\.10\\alpha=0\.10\.

### V\-DFailure Modes: T1 Collapse and T9 Decoding

T1 atλ0=1\.0\\lambda\_\{0\}=1\.0collapses training\-time performance on both benchmarks:2\.27±0\.50%2\.27\\pm 0\.50\\%exact match on TURK and9\.73±0\.76%9\.73\\pm 0\.76\\%on SYNTH\. The collapse is therefore not a dataset quirk\. It is an interaction failure between the curvature penalty and the language\-model fine\-tuning objective at that weight\.

T9 is worse as an inference intervention\. Activating tube\-projected decoding on the three T3 checkpoints yields0\.00%0\.00\\%exact and prefix accuracy\. Generated strings collapse into repetitive token loops, which suggests that the retraction creates an argmax fixed point under greedy decoding\. T9 is therefore a useful negative control: making the hidden state more tube\-consistent at inference time can destroy generation\.

### V\-ESYNTH Results and Family\-Level Analysis

On NL\-RX\-SYNTH, the no\-auxiliary baseline already reaches88\.93±0\.42%88\.93\\pm 0\.42\\%exact match\. Every completed non\-collapsed auxiliary falls within\+0\.87/−0\.13\+0\.87/\-0\.13pp of that baseline; SYNTH therefore cannot adjudicate small auxiliary effects through point estimates alone\. What it does add, once all four Tier\-1 distributional auxiliaries land atn=3n=3, is a coherent*family\-level*pattern: every L1–L4 cell sits above baseline, and three of the four clear the conventionalα=0\.10\\alpha=0\.10threshold under the paired Welch’stt\-test \(L4 STP\-CMF at𝐩paired=0\.06\\mathbf\{p\_\{\\text\{paired\}\}=0\.06\}, L3 C\-Tube\-Sectional at0\.090\.09, L1 SIGReg\-State at0\.100\.10; only L2 SIGReg\-Tangent at0\.500\.50stays clearly inside noise\)\. The unpaired versions land atp∈\[0\.09,0\.40\]p\\in\[0\.09,0\.40\]\. The strongest mean lift is L1 at\+0\.87\+0\.87pp, with L4 next at\+0\.80\+0\.80\.

We do not interpret the L1–L4 cluster as a positive finding\. Three considerations argue against doing so\.*First*, the absolute effect sizes \(≤1\\leq 1pp on a benchmark whose baseline is within∼11\\sim\\\!11points of the ceiling\) sit well below the resolution at which exact\-match is a reliable signal in this regime\.*Second*, the four Tier\-1 cells share the same three baseline seeds, so the paired tests are not statistically independent; the Bonferroni correction at the family\-wise error rateαFWE=0\.10\\alpha\_\{\\text\{FWE\}\}=0\.10reduces the per\-test threshold to0\.0250\.025, against which all four cells fail\. The Holm–Bonferroni step\-down procedure on the same four orderedpp\-values \(smallest first\) requires0\.057<0\.0250\.057<0\.025at the first step and likewise rejects every cell\.*Third*, expanding to the full ten\-cell distributional family on SYNTH \(L1–L6, L9, L12, L13, L14, all atn=3n=3\) leaves the qualitative picture intact: three Tier\-1 cells clear pairedα=0\.10\\alpha=0\.10uncorrected, every Tier\-2 distributional cell sits comfortably inside seed noise, and one Tier\-2 cell \(L9 Score\-Match\) clears pairedα=0\.10\\alpha=0\.10*with a negative delta*\(Δ=−2\.40\\Delta=\-2\.40,ppaired=0\.057p\_\{\\text\{paired\}\}=0\.057\)\. The only cell on SYNTH that crosses the uncorrected threshold under multiple\-cell coverage therefore points the wrong way\. Bonferroni at0\.10/10=0\.010\.10/10=0\.01rejects all ten cells; Holm–Bonferroni on the smallest pairedpp\-value0\.057≮0\.010\.057\\not<0\.01rejects every cell\. The best\-supported conclusion is that SYNTH shows a small Tier\-1 family\-coherent positive drift, an L9 cell that significantly*reduces*exact match under uncorrected paired testing, and otherwise no movement that survives correction\. “The auxiliaries do real geometric work” is consistent with the data; “the work moves exact match by a decision\-grade margin” is not\. T1 collapses on SYNTH at the same severity as on TURK \(9\.73±0\.76%9\.73\\pm 0\.76\\%,p<10−3p<10^\{\-3\}\), confirming that the collapse is dataset\-independent\.

### V\-FData\-Efficiency Curve

Section[VI\-E](https://arxiv.org/html/2605.15394#S6.SS5)explicitly named a data\-efficiency lift on T3\-Local as one of the conditions that would update the structured\-null reading: the original STP claim is framed as a sample\-efficiency gain rather than an asymptotic\-exact\-match gain, and a small\-data lift would be consistent with the auxiliaries doing real representational work that the saturated end of the curve does not reveal\. We therefore re\-train baseline, T3\-Local, T6 MSTB\-JFR and L4 STP\-CMF on TURK at five training\-data fractions \(5%, 10%, 25%, 50%, 100% ofntrain=8000n\_\{\\mathrm\{train\}\}=8000\), three seeds per cell, with all other harness conditions held fixed\.

![Refer to caption](https://arxiv.org/html/2605.15394v1/x3.png)Figure 3:Data\-efficiency curve on NL\-RX\-TURK\. Mean±\\pmone standard deviation across three seeds at each training\-data fraction; significance markers are paired Welch’stt\-tests of each auxiliary against the matched baseline cell \(∗\\ast:ppaired<0\.10p\_\{\\mathrm\{paired\}\}<0\.10,∗⁣∗\\ast\\ast:ppaired<0\.05p\_\{\\mathrm\{paired\}\}<0\.05\)\.Table[V](https://arxiv.org/html/2605.15394#S5.T5)reports per\-fraction means and pairedtt\-tests\. The data resolve the paper’s main question with a*regime\-conditional yes*\. Two distinct cells clear pairedα=0\.05\\alpha=0\.05, at different fractions and from different auxiliary families; two further cells sit just inside pairedα=0\.10\\alpha=0\.10at100%100\\%data:

*\(i\) T6 MSTB\-JFR is positive at both ends of the curve\.*At5%5\\%data \(400400examples\), T6 clearsppaired=0\.013p\_\{\\mathrm\{paired\}\}=0\.013withΔ=\+1\.00\\Delta=\+1\.00pp; the lift fades at10%10\\%\(Δ=\+0\.67\\Delta=\+0\.67,p=0\.50p=0\.50\) and reverses at25%25\\%\(Δ=−0\.27\\Delta=\-0\.27,p=0\.46p=0\.46\); but at100%100\\%data \(80008000examples\) T6 returns the largest mean lift in the cube \(Δ=\+2\.40\\Delta=\+2\.40pp,ppaired=0\.059p\_\{\\mathrm\{paired\}\}=0\.059\)\. The multi\-scale strided JFR is positive both when training data is scarce and at full\-data saturation, but null in between\.

*\(ii\) T3\-Local at the mid\-data point and at saturation\.*At50%50\\%data \(40004000examples\), T3\-Local clearsppaired=0\.029p\_\{\\mathrm\{paired\}\}=0\.029withΔ=\+1\.67\\Delta=\+1\.67pp; the lift persists at100%100\\%\(Δ=\+2\.00\\Delta=\+2\.00,p=0\.090p=0\.090, recovering the effect size of the original Table[III](https://arxiv.org/html/2605.15394#S5.T3)measurement\) but is absent in the small\-data regime \(Δ=\+0\.67\\Delta=\+0\.67,p=0\.13p=0\.13at5%5\\%;Δ=−0\.27\\Delta=\-0\.27,p=0\.79p=0\.79at10%10\\%\)\.

*\(iii\) L4 STP\-CMF shows no positive effect at any of the five fractions*\(Δ∈\[−1\.07,\+0\.73\]\\Delta\\in\[\-1\.07,\+0\.73\], everyppaired≥0\.37p\_\{\\mathrm\{paired\}\}\\geq 0\.37\)\. The completed50%50\\%\(Δ=\+0\.73\\Delta=\+0\.73,p=0\.37p=0\.37\) and100%100\\%\(Δ=\+0\.27\\Delta=\+0\.27,p=0\.83p=0\.83\) cells confirm the small\-data trend: L4’s distributional pressure does not, on its own, translate into exact\-match accuracy at any sample size we tested\.

The two paired\-α=0\.05\\alpha=0\.05cells clear on*disjoint*data fractions and come from*different auxiliary families*: T6 \(multi\-scale strided JFR\) at5%5\\%, T3\-Local \(prompt\-neighbour JFR\) at50%50\\%\. This is inconsistent with a single “small\-data sample\-efficiency lift” interpretation and consistent with a regime\-specific picture in which different auxiliaries help in different regimes\.

TABLE V:Data\-efficiency curve on NL\-RX\-TURK\. Exact\-match mean±\\pmsample standard deviation at each fraction \(three seeds\); paired Welch’stt\-test against the matched\-fraction baseline\.Bold: pairedp<0\.05p<0\.05\.These two paired\-α=0\.05\\alpha=0\.05cells are this study’s first positive results on a fixed exact\-match metric, but neither clears Bonferroni or Holm–Bonferroni atα/15≈0\.0033\\alpha/15\\approx 0\.0033over the cube; the cells live at*different fractions*and involve*different auxiliaries*, so they are not mutually corroborating; and L4 STP\-CMF, the third pre\-registered variant, shows no positive effect at any fraction \(Δ∈\[−1\.07,\+0\.73\]\\Delta\\in\[\-1\.07,\+0\.73\], everyp≥0\.37p\\geq 0\.37\), breaking the family\-coherence reading\. The conservative adoption: representational work*can*translate into exact\-match accuracy under specific conditions, but no cell survives multiple\-comparison correction\. The fourp<0\.10p<0\.10cells are candidate positives to confirm withn≥5n\\geq 5seeds, by adding SYNTH to the cube, and by reproducing on a second base model\.

### V\-GSummary of Empirical Findings

Six observations summarise the empirical content\.*\(i\)*No training\-time auxiliary clears the unpaired Welch test on TURK atα=0\.10\\alpha=0\.10; the strongest mean lift is\+2\.53\+2\.53pp \(T3\-Local\), inside seed noise\.*\(ii\)*On SYNTH the four Tier\-1 distributional cells form a small family\-coherent cluster \(everyΔ≥0\\Delta\\geq 0, three of four with pairedp<0\.10p<0\.10\) that does not survive Bonferroniα/4\\alpha/4or Holm–Bonferroni step\-down\.*\(iii\)*BYOL\-LLM, the only cell carrying the predictor \+ EMA\-target \+ stop\-gradient triple, lands inside seed noise on both benchmarks \(ΔTURK=\+0\.60\\Delta\_\{\\mathrm\{TURK\}\}=\+0\.60,ΔSYNTH=0\.00\\Delta\_\{\\mathrm\{SYNTH\}\}=0\.00\); the architectural asymmetry that lifts vision JEPA does not on its own move exact match in a single\-pass harness\.*\(iv\)*The TURK data\-efficiency cube \(Section[V\-F](https://arxiv.org/html/2605.15394#S5.SS6)\) reveals two paired\-α=0\.05\\alpha=0\.05cells on disjoint fractions and from different auxiliary families \(T6 at5%5\\%, T3\-Local at50%50\\%\) and two further cells just inside pairedα=0\.10\\alpha=0\.10\(T6 and T3\-Local at100%100\\%\); L4 shows no positive effect at any fraction\. No cell survives Bonferroni atα/15\\alpha/15, but four cells with pairedp<0\.10p<0\.10are this study’s first concrete instance of representational work translating into exact\-match accuracy\.*\(v\)*The two pre\-registered falsification routes both fail to break the null on TURK: Fisher cells lose every head\-to\-head against their Euclidean twins by1\.201\.20–2\.472\.47pp and erase the geometric signature observed in the Euclidean cells; decoder\-visible JEPA produces the study’s first positive gradient cosine but undershoots its pre\-registered threshold and leaves exact match inside seed noise\.*\(vi\)*T1 atλ0=1\\lambda\_\{0\}=1and T9 collapse on both benchmarks; L9 Score\-Match*reduces*SYNTH exact match by−2\.40\-2\.40pp \(ppaired=0\.057p\_\{\\text\{paired\}\}=0\.057\), the only Tier\-2 cell that crosses the uncorrected paired threshold\.

Within the harness studied here, adding richer single\-forward\-pass JEPA\-style geometric, distributional, decoder\-aligned, or positive\-cone regularisation is therefore not a reliable route to improved exact\-match accuracy\. The next informative directions \(Section[VI\-E](https://arxiv.org/html/2605.15394#S6.SS5)\) are seed expansion ton≥10n\\geq 10on the candidate cells, two\-view predictor architectures that go beyond a single causal forward pass, larger contrastive batches, and metrics that can resolve representation\-level gains directly\.

## VIDiscussion

The results leave two questions\. First, did the auxiliaries fail because they were absorbed by cross\-entropy or because they acted in directions that exact match does not reward? Second, which existing JEPA\-theory explanation survives contact with the diagnostics?

### VI\-ADiagnostic Findings

Appendix[A](https://arxiv.org/html/2605.15394#A1)reports three measurements on trained checkpoints: assistant\-span anisotropy, trajectory curvature, and the gradient cosine between auxiliary loss and language\-model cross\-entropy over LoRA\-trainable parameters\. Together they establish that the main null is*not*a no\-op null\.

First, JFR\-family auxiliaries reduce trajectory curvature from the baseline value of roughly2\.002\.00rad to the1\.211\.21–1\.451\.45rad range\. Second, several auxiliaries tighten seed\-to\-seed variance\. Third, and most importantly for the title, the measured gradient cosine is near zero or slightly negative rather than near one: across the JFR\-family TURK cells,ρ∈\[−0\.13,\+0\.01\]\\rho\\in\[\-0\.13,\+0\.01\]\. Thus the auxiliary gradients are not simply parallel to cross\-entropy’s update direction\. They are doing representational work that cross\-entropy is not already doing\.

We summarise this as an empirical diagnostic criterion\. An auxiliary is*representationally active but task\-metric inert*when

Δ​G\>0,\|ρaux,CE\|≪1,\|Δ​EM\|≤ϵ,\\Delta G\>0,\\qquad\\left\|\\rho\_\{\\text\{aux,CE\}\}\\right\|\\ll 1,\\qquad\\left\|\\Delta\\mathrm\{EM\}\\right\|\\leq\\epsilon,\(31\)whereΔ​G\\Delta Gis a measured geometry change, such as reduced curvature or changed anisotropy, and

ρaux,CE=⟨∇θLoRAℒaux,∇θLoRAℒLM⟩‖∇θLoRAℒaux‖​‖∇θLoRAℒLM‖\.\\rho\_\{\\text\{aux,CE\}\}=\\frac\{\\left\\langle\\nabla\_\{\\theta\_\{\\text\{LoRA\}\}\}\\mathcal\{L\}\_\{\\text\{aux\}\},\\,\\nabla\_\{\\theta\_\{\\text\{LoRA\}\}\}\\mathcal\{L\}\_\{\\text\{LM\}\}\\right\\rangle\}\{\\left\\lVert\\nabla\_\{\\theta\_\{\\text\{LoRA\}\}\}\\mathcal\{L\}\_\{\\text\{aux\}\}\\right\\rVert\\left\\lVert\\nabla\_\{\\theta\_\{\\text\{LoRA\}\}\}\\mathcal\{L\}\_\{\\text\{LM\}\}\\right\\rVert\}\.\(32\)The measured cells satisfy this pattern: they change geometry, they do not move in the CE direction, and they do not reliably move exact match\. The diagnostics therefore weaken three simple explanations of why exact match does not move —*CE absorption*\(the gradient\-cosine measurement rules this out\),*already\-straight trajectories*\(the curvature diagnostic rules this out\), and*missing anti\-collapse only*\(L1–L4 would have been the escape route, and were not\)\. The remaining explanation is more specific: the losses change geometry, but the exact\-match regex metric is insensitive to the changed geometry\. The auxiliaries act, but their action is decoder\-invisible\.

### VI\-BFailure\-mode mechanisms

Two specific failures admit short mechanistic readings\. T5 averages the same JFR pressure across four transformer layers; if the per\-layer gradients were independent this could increase signal, but the observed ordering suggests the opposite — the gradients are correlated under the CE backbone, so the depth average spends the same budget on redundant directions and dilutes the final\-layer signal\. T9 composes the autoregressive forward step with a retraction onto the learned tube; greedy decoding is already prone to local attractors, and a non\-expansive projection strengthens that contraction into a stable argmax fixed point, producing the repetitive\-token failure observed at0\.00%0\.00\\%exact and prefix accuracy\. A viable projector would need to preserve an entropy\-bearing direction or inject stochasticity\.

### VI\-CRelation to the Original STP Claim

Our experiments do not directly falsify STP’s data\-efficiency claim\[[16](https://arxiv.org/html/2605.15394#bib.bib2)\]: that claim is about a training\-data fraction curve on NL\-RX\-SYNTH, ours is about asymptotic exact\-match accuracy\. A loss can plausibly improve sample efficiency without moving the final exact\-match ceiling\. Section[V\-F](https://arxiv.org/html/2605.15394#S5.SS6)reports the TURK arm and finds two paired\-α=0\.05\\alpha=0\.05cells on disjoint fractions from different auxiliary families — partial corroboration; the SYNTH arm of that curve remains the empirical gap\. A second difference is scale: T7 uses in\-batch negatives but our setup fixes the batch size at44, so the T7 null is a small\-batch result, not a general statement about contrastive LLM auxiliaries\.

### VI\-DThe Structured\-Null Interpretation

The most compact interpretation is:

> The auxiliaries do representational work without decoder\-visible gain\. They reshape hidden\-state geometry in directions that are not cross\-entropy’s update direction, but those directions do not reliably affect exact\-match regex accuracy in this fine\-tuning regime\.

This reading explains the main observations at once: small and non\-significant mean gains, reduced variance, curvature reduction, gradient orthogonality, distributional\-loss nulls, and the contrast between training\-time regularisation and T9’s inference\-time fixed\-point collapse\. It also makes a clear prediction: metrics or regimes that reward representation geometry—data efficiency, calibration, out\-of\-distribution generalisation, semantic equivalence, or larger contrastive batches—are more likely to reveal value than another single\-forward\-pass trajectory penalty evaluated only by exact match\.

The reading is also corroborated by an independent positive result\. Wang et al\.\[[27](https://arxiv.org/html/2605.15394#bib.bib4)\]train a world\-model encoder with a curvature regulariser that, in our notation, is structurally identical to STP, and report2020–60%60\\%open\-loop and2020–30%30\\%MPC goal\-reaching gains relative to a non\-regularised baseline\. They additionally prove that, under a linear\-dynamics assumption, the same regulariser controls the condition number of the planning Hessian\. Their gain is precisely the kind that our structured\-null reading predicts*should*appear: their downstream metric is the optimisation cost of a gradient\-based planner whose Hessian explicitly depends on representation conditioning\. Our downstream metric is exact\-match decoded under argmax, which has no such dependence\. The two results together suggest that whether trajectory straightening helps a downstream task is governed less by whether the representation is straighter and more by whether the downstream objective’s optimisation landscape is shaped by representation geometry\.

We probe this reading directly with two pre\-registered falsification routes\. Appendix[B](https://arxiv.org/html/2605.15394#A2)replaces the Euclidean curvature norm with the LM head’s pulled\-back next\-token Fisher metric, additionally weights each per\-token contribution by a margin\-driven sigmoid, and projects the auxiliary gradient off any negative\-conflict component of CE\. If the null mistook “wrong direction in input space” for “no signal at the decoder,” any of these three interventions should have recovered some exact\-match accuracy\. The TURK cube \(Table[VII\(a\)](https://arxiv.org/html/2605.15394#A2.T7.st1)\) instead returns five head\-to\-head regressions of1\.201\.20–2\.472\.47pp against the Euclidean twins; on T3\-Local the regression deepens monotonically as components are added\. Variance contracts in every Fisher cell by1414–68%68\\%, matching the Euclidean cells: representation work is happening, the metric just does not expose it\. Appendix[C](https://arxiv.org/html/2605.15394#A3)moves the auxiliary out ofhh\-space entirely, scoring it on the post\-softmax distribution of the LM head; by construction its margin\-side gradient lies in cross\-entropy’s convex cone\. We pre\-registeredρaux,CE\>0\.20\\rho\_\{\\mathrm\{aux,CE\}\}\>0\.20on TURK atn=3n=3as the diagnostic\-level falsification threshold\. The result \(Table[VII\(b\)](https://arxiv.org/html/2605.15394#A2.T7.st2)\):ρaux,CE=\+0\.186\\rho\_\{\\mathrm\{aux,CE\}\}=\+0\.186, four standard deviations above the upper end of the existing band but undershooting the pre\-registered threshold; exact match remains inside seed noise and the auxiliary leaves anisotropy and curvature at baseline\. The directional prediction is correct, the magnitude prediction is overconfident, and the decoded\-task null persists — the structured\-null reading is robust against both falsification routes\.

### VI\-EFuture Directions and Decision Criteria

The most informative next experiments are ordered by how directly they test the structured\-null interpretation:

1. 1\.Data\-efficiency curves\.We have completed the TURK arm at4×5×34\\times 5\\times 3\(Section[V\-F](https://arxiv.org/html/2605.15394#S5.SS6)\); the SYNTH arm of the same five\-fraction sweep is the remaining benchmark gap\. Within the TURK cube, expanding the four cells with pairedp<0\.10p<0\.10ton≥5n\\geq 5seeds is the most informative seed\-expansion follow\-up\.
2. 2\.Seed expansion for plausible cells\.Expand T3\-Local, T3 at3×10−43\\times 10^\{\-4\}, T6, L1 and L4 ton≥10n\\geq 10before proposing new tube variants\.
3. 3\.Metric expansion\.Add regex semantic\-equivalence tests, calibration, and OOD splits to determine whether the geometry change is useful but invisible to exact string match\.
4. 4\.Scale the discriminative tests\.Revisit T7 and CPC only with substantially larger effective batches or memory\-bank negatives\.
5. 5\.Test full multi\-view JEPA\.Replace the single\-pass approximations with a genuine context\-target architecture with a target encoder and non\-trivial view construction\.
6. 6\.Replication under full fine\-tuning\.The closest prior comparators run their main experiments under*full*fine\-tuning: STP’s16×16\\timesdata\-efficiency claim and LLM\-JEPA’s headline accuracy gains are full\-parameter results, with LoRA reported only as a secondary appendix study \(LLM\-JEPA, Table 8\)\. The main grid in this paper is LoRA throughout, raising the question of whether the structured null is a property of the auxiliaries or of the regime\. We tested decoder\-visible JEPA, the most theoretically promising of the falsification routes, under full fine\-tuning atn=5n=5seeds on TURK with the same auxiliary hyperparameters, dropping the learning rate from2×10−42\{\\times\}10^\{\-4\}\(LoRA\) to2×10−52\{\\times\}10^\{\-5\}to match STP and LLM\-JEPA’s full\-FT recipe\. The result reproduces the LoRA null on both benchmarks\. On TURK the no\-auxiliary baseline and DV\-JEPA produced48\.72%±0\.6148\.72\\%\\pm 0\.61pp and48\.76%±1\.4248\.76\\%\\pm 1\.42pp, withΔ=\+0\.04\\Delta=\+0\.04pp,tpaired=\+0\.054t\_\{\\text\{paired\}\}=\+0\.054\(ν=4\\nu=4\),ppaired=0\.96p\_\{\\text\{paired\}\}=0\.96, and the unpaired test agreeing \(t=\+0\.058t=\+0\.058,p=0\.96p=0\.96\)\. On SYNTH the same comparison yielded88\.40%±0\.4788\.40\\%\\pm 0\.47pp \(regular\) versus88\.92%±0\.7388\.92\\%\\pm 0\.73pp \(DV\-JEPA\), withΔ=\+0\.52\\Delta=\+0\.52pp,tpaired=\+1\.260t\_\{\\text\{paired\}\}=\+1\.260\(ν=4\\nu=4\),ppaired=0\.28p\_\{\\text\{paired\}\}=0\.28, and the unpaired test in agreement \(t=\+1\.341t=\+1\.341,p=0\.22p=0\.22\)\. Bonferroni correction atα/2=0\.05\\alpha/2=0\.05across the two head\-to\-heads is therefore not approached, and neither cell clears the single\-cellα=0\.10\\alpha=0\.10threshold either\. The structured null is therefore robust across the LoRA and full\-FT regimes*and*across both benchmarks for the decoder\-visible construction, weakening the capacity\-ceiling reading and strengthening the broader conclusion that hidden\-geometry representation work and exact\-match decoded accuracy are weakly coupled in autoregressive next\-token training irrespective of which parameters carry gradient\. STP\-style straightening, which the comparators report as full\-FT\-positive, remains an open replication target; we did not run the trajectory\-shape arm under full FT in this paper\.

The conclusion would change if any one of these settings produced a stable, multiple\-comparison\-corrected gain while preserving the diagnostic evidence that the auxiliary acts outside the cross\-entropy direction\.

### VI\-FLimitations

n=3n=3seeds is small for detecting effects≤2\\leq 2pp at the variances observed here, so the present study is best read as a hypothesis test and prioritisation device, not a final power analysis\. Results are restricted to Llama\-3\.2\-1B\-Instruct with LoRA on the NL\-RX family; larger base models, full fine\-tuning, non\-symbolic tasks, or tasks with more semantic slack may behave differently\. T3 receives a two\-pointλ0\\lambda\_\{0\}sweep, but most auxiliaries are run at one weight, and the T1 collapse atλ0=1\.0\\lambda\_\{0\}=1\.0shows that weight matters — smaller T1 weights and broader sweeps remain useful checks\. The bf16 setup fixes batch size at44and rules out the large negative banks contrastive objectives typically rely on, which is the dominant limitation for interpreting T7\.

## VIIConclusion

This paper asked when JEPA\-style auxiliary objectives added to LoRA fine\-tuning create decoder\-visible task signal, and when they merely reshape hidden\-state geometry in directions that exact\-match decoding does not reward\. We studied twenty\-two training\-time auxiliaries and one inference\-time projector for Llama\-3\.2\-1B\-Instruct under fixed model, splice point, decoding, and evaluation conditions, and we constructed two pre\-registered falsification routes for the structured\-null reading itself: a Fisher\-metric Jacobi\-residual family \(decoder\-aligned curvature\) and decoder\-visible JEPA \(auxiliary gradient engineered to share its convex cone with cross\-entropy\)\.

The main result is a structured null robust against both falsification routes\. Without statistical correction, a handful of cells reach single\-cell pairedα=0\.10\\alpha=0\.10: prompt\-local JFR on TURK \(Δ=\+2\.53\\Delta=\+2\.53pp,ppaired=0\.003p\_\{\\text\{paired\}\}=0\.003\), T5 on TURK, three Tier\-1 distributional cells on SYNTH \(L1, L3, L4\), and four fraction\-specific cells in the data\-efficiency cube \(T6 at5%5\\%and100%100\\%, T3\-Local at50%50\\%and100%100\\%\)\. With Bonferroni or Holm–Bonferroni at the family\-wise threshold, none of these survives\. BYOL\-LLM, the direct test of JEPA\-core predictor \+ EMA\-target \+ stop\-gradient asymmetry, remains inside seed noise on both benchmarks even without correction\. The Fisher\-metric replacement loses every head\-to\-head against its Euclidean twin \(−1\.20\-1\.20to−2\.47\-2\.47pp on TURK\) and*erases*the study’s geometric signature: anisotropy and trajectory curvature snap back to baseline, indicating that confining the auxiliary to decoder\-perceptible directions removes its h\-space work without buying back exact\-match accuracy\. Decoder\-visible JEPA produces the study’s first cell with positive gradient cosine,ρaux,CE=\+0\.186±0\.050\\rho\_\{\\mathrm\{aux,CE\}\}=\+0\.186\\pm 0\.050, four standard deviations above the upper end of the\[−0\.69,\+0\.01\]\[\-0\.69,\+0\.01\]band every other cell occupies; the pre\-registeredρ\>0\.20\\rho\>0\.20threshold is just missed in the mean and exact match remains inside seed noise \(Δ=\+0\.93\\Delta=\+0\.93pp,ppaired=0\.48p\_\{\\mathrm\{paired\}\}=0\.48\)\. The directional prediction is correct, the magnitude prediction is overconfident, and the decoded\-task null persists\.

The null is not a representational null\. The diagnostics show that JFR\-family auxiliaries reduce trajectory curvature, that L1, L6, and L14 collapse anisotropy below baseline, and that several auxiliaries tighten seed\-to\-seed variance\. In this setting, JEPA\-style auxiliaries can therefore perform measurable representational work without producing reliable decoder\-visible gains\. Moving hidden geometry is not the same as improving the decoded task metric, and even an auxiliary explicitly engineered to share cross\-entropy’s gradient cone does not, at this seed count, translate into exact\-match accuracy\.

A full\-fine\-tuning replication of decoder\-visible JEPA on both benchmarks atn=5n=5seeds \(Section[VI\-E](https://arxiv.org/html/2605.15394#S6.SS5)\) reproduces this null\. With the LoRA adapters dropped and the LM body trained end\-to\-end atlr=2×10−5\\text\{lr\}=2\{\\times\}10^\{\-5\}, the no\-auxiliary baseline and DV\-JEPA reach48\.72%±0\.6148\.72\\%\\pm 0\.61pp and48\.76%±1\.4248\.76\\%\\pm 1\.42pp respectively on TURK \(Δ=\+0\.04\\Delta=\+0\.04pp,tpaired=\+0\.054t\_\{\\text\{paired\}\}=\+0\.054,ν=4\\nu=4,ppaired=0\.96p\_\{\\text\{paired\}\}=0\.96\), and88\.40%±0\.4788\.40\\%\\pm 0\.47pp and88\.92%±0\.7388\.92\\%\\pm 0\.73pp on SYNTH \(Δ=\+0\.52\\Delta=\+0\.52pp,tpaired=\+1\.260t\_\{\\text\{paired\}\}=\+1\.260,ν=4\\nu=4,ppaired=0\.28p\_\{\\text\{paired\}\}=0\.28\)\. Neither cell clears single\-cellα=0\.10\\alpha=0\.10, much less the Bonferroni gate atα/2=0\.05\\alpha/2=0\.05\. The structured null is therefore robust across LoRA and full fine\-tuning, and across both benchmarks, for the decoder\-visible construction\. This weakens the capacity\-ceiling reading and supports the broader claim that representation work and exact\-match accuracy are weakly coupled irrespective of which parameters carry gradient\.

This conclusion narrows the design space for future LLM\-domain JEPA work along two axes\. First, the seed budget: the full cube atn≥10n\\geq 10on the four candidate cells with pairedp<0\.10p<0\.10\(T6@5%5\\%, T6@100%100\\%, T3\-Local@50%50\\%, T3\-Local@100%100\\%\) and on decoder\-visible JEPA is the cheapest informative follow\-up\. Second, the metric: the structured\-null reading predicts that exact match is insensitive to the directions every existing auxiliary acts in; semantic\-equivalence regex evaluators, calibration tests, and OOD splits should expose representation\-level changes that exact match cannot\. Without one of these two extensions, additional single\-pass tube penalties evaluated only by exact match are unlikely to break the null this paper reports\.

## Appendix AEmpirical Diagnostics

A self\-contained Tier\-0 diagnostics module runs five post\-hoc measurements on the saved checkpoints without any additional training and without modifying any existing loss, trainer, or evaluator code: D1 last\-layer anisotropyA\(L\)A^\{\(L\)\}, D2 Hosseini\-Fedorenko trajectory curvatureCCon the assistant span, D3 gradient alignmentρ=cos⁡\(∇ℒaux,∇ℒLM\)\\rho=\\cos\(\\nabla\\mathcal\{L\}\_\{\\text\{aux\}\},\\nabla\\mathcal\{L\}\_\{\\text\{LM\}\}\)on training mini\-batches, D4 paired Welch’stt\-tests by seed \(already folded into Table[III](https://arxiv.org/html/2605.15394#S5.T3)’sppairedp\_\{\\text\{paired\}\}column\), and D5 the per\-position attribution of each variant’s auxiliary residual on the EOS\-clipped assistant span\.

### A\-ADiagnostic Equations

The five diagnostics are defined as follows; D3 is the gradient cosine of Eq\. \([32](https://arxiv.org/html/2605.15394#S6.E32)\) and is not repeated here\.

#### D1 — anisotropyA\(L\)A^\{\(L\)\}\.

Collect the multiset of final\-layer assistant\-span hidden states across the test forward,ℋ=\{hi∈ℝD:i=1,…,N\}\\mathcal\{H\}=\\\{h\_\{i\}\\in\\mathbb\{R\}^\{D\}:i=1,\\dots,N\\\}\. Sample\|𝒫\|=min⁡\(2000,N​\(N−1\)/2\)\|\\mathcal\{P\}\|=\\min\(2000,\\,N\(N\-1\)/2\)distinct unordered index pairs𝒫⊂\{\(i,j\):i<j\}\\mathcal\{P\}\\subset\\\{\(i,j\):i<j\\\}uniformly at random, and report the mean cosine

A\(L\)=1\|𝒫\|​∑\(i,j\)∈𝒫⟨hi,hj⟩‖hi‖​‖hj‖\.A^\{\(L\)\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(i,j\)\\in\\mathcal\{P\}\}\\frac\{\\langle h\_\{i\},\\,h\_\{j\}\\rangle\}\{\\left\\lVert h\_\{i\}\\right\\rVert\\,\\left\\lVert h\_\{j\}\\right\\rVert\}\.\(33\)A\(L\)→0A^\{\(L\)\}\\to 0is isotropic;A\(L\)→1A^\{\(L\)\}\\to 1is full cone collapse\[[9](https://arxiv.org/html/2605.15394#bib.bib14)\]\.

#### D2 — Hosseini\-Fedorenko curvatureCC\.

For each examplebbwith assistant\-span hidden trajectoryhb,1,…,hb,Lbh\_\{b,1\},\\dots,h\_\{b,L\_\{b\}\}\(Lb≥3L\_\{b\}\\geq 3\), define velocitiesvb,t=hb,t\+1−hb,tv\_\{b,t\}=h\_\{b,t\+1\}\-h\_\{b,t\}and average the angle between consecutive velocities,

Cb=1Lb−2​∑t=1Lb−2arccos⁡\(⟨vb,t,vb,t\+1⟩‖vb,t‖​‖vb,t\+1‖\)\.C\_\{b\}\\;=\\;\\frac\{1\}\{L\_\{b\}\-2\}\\sum\_\{t=1\}^\{L\_\{b\}\-2\}\\arccos\\\!\\left\(\\frac\{\\langle v\_\{b,t\},\\,v\_\{b,t\+1\}\\rangle\}\{\\left\\lVert v\_\{b,t\}\\right\\rVert\\,\\left\\lVert v\_\{b,t\+1\}\\right\\rVert\}\\right\)\.\(34\)The cell statistic is the mean across examples,C=\(1/B\)​∑bCbC=\(1/B\)\\sum\_\{b\}C\_\{b\}, reported in radians\[[13](https://arxiv.org/html/2605.15394#bib.bib22)\]\.

#### D4 — Welch–Satterthwaite degrees of freedom\.

Thettstatistics for the unpaired and paired tests are defined in Section[IV\-A](https://arxiv.org/html/2605.15394#S4.SS1)\. The unpaired Welch test uses the Welch–Satterthwaite approximation for the degrees of freedom,

d​funp=\(sx2/nx\+sy2/ny\)2\(sx2/nx\)2nx−1\+\(sy2/ny\)2ny−1;df\_\{\\mathrm\{unp\}\}\\;=\\;\\frac\{\\bigl\(s\_\{x\}^\{2\}/n\_\{x\}\+s\_\{y\}^\{2\}/n\_\{y\}\\bigr\)^\{2\}\}\{\\frac\{\(s\_\{x\}^\{2\}/n\_\{x\}\)^\{2\}\}\{n\_\{x\}\-1\}\+\\frac\{\(s\_\{y\}^\{2\}/n\_\{y\}\)^\{2\}\}\{n\_\{y\}\-1\}\};\(35\)the paired test usesd​fp=n−1df\_\{\\mathrm\{p\}\}=n\-1onnnpaired differences\. Bothpp\-values are two\-sided Student\-ttsurvival\-function probabilities at the correspondingd​fdf\.

#### D5 — per\-position auxiliary attribution\.

For JFR\-family cells, reconstruct the Jacobi residualJb,t=hb,t−h¯tJ\_\{b,t\}=h\_\{b,t\}\-\\bar\{h\}\_\{t\}used in training, whereh¯t\\bar\{h\}\_\{t\}is the relevant centroid \(mini\-batch mean forjfrandmstb\_jfr; per\-layer mean fordst\_jfr; cosine\-retrieved local centroid forlocal\_jfr\)\. For each scaleΔ∈𝒮\\Delta\\in\\mathcal\{S\}and centre positionttwith1≤t−Δ1\\leq t\-\\Deltaandt\+Δ<Lbt\+\\Delta<L\_\{b\}, compute the strided second difference

DΔ2​Jb,t=Jb,t\+Δ−2​Jb,t\+Jb,t−ΔΔ2\.D^\{2\}\_\{\\Delta\}J\_\{b,t\}\\;=\\;\\frac\{J\_\{b,\\,t\+\\Delta\}\-2\\,J\_\{b,t\}\+J\_\{b,\\,t\-\\Delta\}\}\{\\Delta^\{2\}\}\.Bucket each centre by relative position in the EOS\-clipped assistant span,

𝖻𝗄​\(t,Lb\)=\{frontif​\(t\+0\.5\)/Lb<1/3middleif​1/3≤\(t\+0\.5\)/Lb<2/3endif​2/3≤\(t\+0\.5\)/Lb\\mathsf\{bk\}\(t,\\,L\_\{b\}\)\\;=\\;\\begin\{cases\}\\text\{front\}&\\text\{if \}\(t\+0\.5\)/L\_\{b\}<1/3\\\\ \\text\{middle\}&\\text\{if \}1/3\\leq\(t\+0\.5\)/L\_\{b\}<2/3\\\\ \\text\{end\}&\\text\{if \}2/3\\leq\(t\+0\.5\)/L\_\{b\}\\end\{cases\}\(36\)and report bucket means

A¯\(𝖻𝗄\)=1N𝖻𝗄​∑\(b,t,Δ\):𝖻𝗄​\(t,Lb\)=𝖻𝗄‖DΔ2​Jb,t‖2,\\bar\{A\}^\{\(\\mathsf\{bk\}\)\}\\;=\\;\\frac\{1\}\{N\_\{\\mathsf\{bk\}\}\}\\sum\_\{\(b,t,\\Delta\):\\,\\mathsf\{bk\}\(t,L\_\{b\}\)=\\mathsf\{bk\}\}\\left\\lVert D^\{2\}\_\{\\Delta\}J\_\{b,t\}\\right\\rVert^\{2\},\(37\)whereN𝖻𝗄N\_\{\\mathsf\{bk\}\}is the count of\(b,t,Δ\)\(b,t,\\Delta\)tuples contributing to bucket𝖻𝗄\\mathsf\{bk\}\. A flat profile \(front≈\\approxmiddle≈\\approxend\) is the over\-smoothing critique of\[[2](https://arxiv.org/html/2605.15394#bib.bib25)\]; a tilt indicates the auxiliary fires preferentially at one end of the span\.

### A\-BEmpirical numbers

Table[VI](https://arxiv.org/html/2605.15394#A1.T6)reports the per\-experiment means acrossn=3n=3seeds for D1, D2, and D3\. These are the empirical numbers cited inline in Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1)\.

TABLE VI:Tier\-0 diagnostics across the full study atn=3n=3seeds: last\-layer anisotropyA\(L\)A^\{\(L\)\}, trajectory curvatureCC, and gradient alignmentρ\\rhobetween auxiliary and cross\-entropy gradients \(D3 reported only for variants whose loss can be reconstructed from the saved checkpoint without reloading a trained auxiliary head\)\. Values are mean±\\pmsample standard deviation across the three seeds\.CellA\(L\)A^\{\(L\)\}CC\(rad\)ρ\\rho*NL\-RX\-TURK*Regular \(baseline\)0\.122±0\.0050\.122\\pm 0\.0052\.002±0\.0012\.002\\pm 0\.001–STP0\.385±0\.0010\.385\\pm 0\.0011\.448±0\.0031\.448\\pm 0\.003–T2 \(RIG\-Tube\)0\.383±0\.0090\.383\\pm 0\.0091\.837±0\.0071\.837\\pm 0\.007–T3 \(JFR\),λ0=10−3\\lambda\_\{0\}=10^\{\-3\}0\.785±0\.0100\.785\\pm 0\.0101\.209±0\.0031\.209\\pm 0\.003−0\.099±0\.034\-0\.099\\pm 0\.034T3 \(JFR\),λ0=3⋅10−4\\lambda\_\{0\}=3\{\\cdot\}10^\{\-4\}0\.765±0\.0050\.765\\pm 0\.0051\.309±0\.0051\.309\\pm 0\.005−0\.049±0\.054\-0\.049\\pm 0\.054T3\-Local0\.762±0\.0030\.762\\pm 0\.0031\.272±0\.0041\.272\\pm 0\.004−0\.083±0\.045\-0\.083\\pm 0\.045T5 \(DST\-JFR\)0\.751±0\.0010\.751\\pm 0\.0011\.310±0\.0061\.310\\pm 0\.006−0\.088±0\.025\-0\.088\\pm 0\.025T6 \(MSTB\-JFR\)0\.794±0\.0100\.794\\pm 0\.0101\.397±0\.0081\.397\\pm 0\.008−0\.106±0\.070\-0\.106\\pm 0\.070T7 \(Contrastive Tube\)0\.140±0\.0040\.140\\pm 0\.0042\.028±0\.0032\.028\\pm 0\.003–L1 \(SIGReg\-State\)0\.018±0\.001\\mathbf\{0\.018\}\\pm 0\.0011\.998±0\.0041\.998\\pm 0\.004–L2 \(SIGReg\-Tangent\)0\.151±0\.0090\.151\\pm 0\.0092\.032±0\.0022\.032\\pm 0\.002–L3 \(C\-Tube\-Sectional\)0\.234±0\.0060\.234\\pm 0\.0061\.669±0\.002\\mathbf\{1\.669\}\\pm 0\.002–L4 \(STP\-CMF\)0\.124±0\.0060\.124\\pm 0\.0062\.005±0\.0022\.005\\pm 0\.002–L5 \(VICReg\-VC\)0\.479±0\.0030\.479\\pm 0\.0032\.017±0\.0022\.017\\pm 0\.002–L6 \(SW\-Iso\)0\.021±0\.003\\mathbf\{0\.021\}\\pm 0\.0032\.023±0\.0072\.023\\pm 0\.007–L9 \(Score\-Match\)0\.185±0\.0040\.185\\pm 0\.0042\.039±0\.0082\.039\\pm 0\.008–L12 \(CPC\)0\.111±0\.0060\.111\\pm 0\.0061\.955±0\.0021\.955\\pm 0\.002–L13 \(BYOL\-LLM\)0\.107±0\.0030\.107\\pm 0\.0031\.988±0\.0051\.988\\pm 0\.005–L14 \(I\-JEPA\-LLM\)0\.029±0\.001\\mathbf\{0\.029\}\\pm 0\.0011\.991±0\.0061\.991\\pm 0\.006–T1 \(collapsed,λ0=1\\lambda\_\{0\}=1\)0\.684±0\.0300\.684\\pm 0\.0301\.888±0\.0571\.888\\pm 0\.057−0\.134±0\.024\-0\.134\\pm 0\.024*Tier\-3 \(Fisher metric\)*Fisher\-JFR \(C1\)0\.134±0\.0070\.134\\pm 0\.0071\.978±0\.0051\.978\\pm 0\.005\+0\.024±0\.021\+0\.024\\pm 0\.021Fisher\-MSTB \(C1\)0\.130±0\.0040\.130\\pm 0\.0041\.994±0\.0011\.994\\pm 0\.001\+0\.104±0\.066\+0\.104\\pm 0\.066Fisher\-Local\-JFR \(C1\)0\.129±0\.0020\.129\\pm 0\.0021\.995±0\.0061\.995\\pm 0\.006\+0\.018±0\.233\+0\.018\\pm 0\.233\+ margin \(C1\+C2\)0\.127±0\.0060\.127\\pm 0\.0061\.993±0\.0041\.993\\pm 0\.004\+0\.003±0\.065\+0\.003\\pm 0\.065\+ margin \+ PCGrad \(C1\+C2\+C3\)0\.126±0\.0070\.126\\pm 0\.0071\.993±0\.0041\.993\\pm 0\.004\+0\.007±0\.166\+0\.007\\pm 0\.166*Decoder\-visible JEPA*DV\-JEPA0\.131±0\.0090\.131\\pm 0\.0092\.003±0\.0122\.003\\pm 0\.012\+0\.186±0\.050\\mathbf\{\+0\.186\}\\pm 0\.050*NL\-RX\-SYNTH*Regular \(baseline\)0\.101±0\.0050\.101\\pm 0\.0051\.991±0\.0071\.991\\pm 0\.007–STP0\.345±0\.0070\.345\\pm 0\.0071\.339±0\.0081\.339\\pm 0\.008–T2 \(RIG\-Tube\)0\.403±0\.0080\.403\\pm 0\.0081\.803±0\.0081\.803\\pm 0\.008–T3 \(JFR\)0\.715±0\.0060\.715\\pm 0\.0061\.097±0\.0051\.097\\pm 0\.005−0\.685±0\.113\\mathbf\{\-0\.685\}\\pm 0\.113T3\-Local0\.694±0\.0170\.694\\pm 0\.0171\.147±0\.0011\.147\\pm 0\.001−0\.541±0\.051\\mathbf\{\-0\.541\}\\pm 0\.051T5 \(DST\-JFR\)0\.697±0\.0150\.697\\pm 0\.0151\.186±0\.0031\.186\\pm 0\.003−0\.492±0\.028\\mathbf\{\-0\.492\}\\pm 0\.028T6 \(MSTB\-JFR\)0\.763±0\.0210\.763\\pm 0\.0211\.303±0\.0051\.303\\pm 0\.005−0\.496±0\.045\\mathbf\{\-0\.496\}\\pm 0\.045T7 \(Contrastive Tube\)0\.126±0\.0030\.126\\pm 0\.0032\.017±0\.0052\.017\\pm 0\.005–L1 \(SIGReg\-State\)0\.012±0\.001\\mathbf\{0\.012\}\\pm 0\.0012\.022±0\.0042\.022\\pm 0\.004–L2 \(SIGReg\-Tangent\)0\.137±0\.0110\.137\\pm 0\.0112\.034±0\.0042\.034\\pm 0\.004–L3 \(C\-Tube\-Sectional\)0\.058±0\.0010\.058\\pm 0\.0012\.076±0\.0022\.076\\pm 0\.002–L4 \(STP\-CMF\)0\.097±0\.0040\.097\\pm 0\.0041\.999±0\.0011\.999\\pm 0\.001–L5 \(VICReg\-VC\)0\.536±0\.0020\.536\\pm 0\.0022\.030±0\.0062\.030\\pm 0\.006–L6 \(SW\-Iso\)0\.016±0\.002\\mathbf\{0\.016\}\\pm 0\.0022\.044±0\.0012\.044\\pm 0\.001–L9 \(Score\-Match\)0\.160±0\.0020\.160\\pm 0\.0022\.039±0\.0072\.039\\pm 0\.007–L12 \(CPC\)0\.092±0\.0030\.092\\pm 0\.0031\.932±0\.0041\.932\\pm 0\.004–L13 \(BYOL\-LLM\)0\.112±0\.0140\.112\\pm 0\.0141\.965±0\.0071\.965\\pm 0\.007–L14 \(I\-JEPA\-LLM\)0\.026±0\.002\\mathbf\{0\.026\}\\pm 0\.0021\.977±0\.0031\.977\\pm 0\.003–T1 \(collapsed,λ0=1\\lambda\_\{0\}=1\)0\.599±0\.0140\.599\\pm 0\.0141\.806±0\.0131\.806\\pm 0\.013−0\.095±0\.028\-0\.095\\pm 0\.028Three observations follow directly from Table[VI](https://arxiv.org/html/2605.15394#A1.T6)and are folded into Section[VI\-A](https://arxiv.org/html/2605.15394#S6.SS1):*\(i\)*every TURK cell with a measurableρ\\rhois essentially orthogonal or slightly anti\-aligned to the cross\-entropy gradient – the predicted “in CE’s implicit\-bias kernel” regime is not attained on this setup\. The TURK gradient cosines lie in\[−0\.13,−0\.05\]\[\-0\.13,\-0\.05\]\(mildly anti\-aligned\), but the SYNTH cosines are dramatically more anti\-aligned,ρ∈\[−0\.69,−0\.49\]\\rho\\in\[\-0\.69,\-0\.49\]across the JFR family\. The dataset asymmetry indicates that on the saturated SYNTH benchmark the auxiliary’s gradient direction actively opposes cross\-entropy’s, which is consistent with the small SYNTH effect sizes: the auxiliary is fighting CE rather than cooperating with it\.

*\(ii\)*The no\-auxiliary baseline trajectory\-curvature on the assistant span isC≈2\.00C\\approx 2\.00rad on both benchmarks, and the JFR family \(T​3T3,T​3T3\-Local,T​5T5,T​6T6, theT​3T3λ\\lambda\-sweep\) reduces it to1\.101\.10–1\.401\.40rad, while every distributional and predictor\-based auxiliary \(L1–L6, L9, L12–L14\) leaves curvature at baseline values \(≥1\.93\\geq 1\.93\)\. The exception is L3 C\-Tube\-Sectional, which explicitly penalises sectional\-curvature variance and reduces TURKCCto1\.671\.67\. Curvature reduction is therefore a JFR\-family signature, not a generic auxiliary effect, and the distributional family does its representational work along orthogonal axes\.

*\(iii\)*The eighteen original variants partition cleanly into three anisotropy regimes\. The JFR family*concentrates*the assistant\-span cone, raisingA\(L\)A^\{\(L\)\}from baseline∼0\.10\\sim\\\!0\.10–0\.120\.12to0\.690\.69–0\.800\.80\. The distributional and predictor families largely leave anisotropy near baseline \(L4, L12, L13:0\.090\.09–0\.130\.13\)\. Three cells*collapse*anisotropy below baseline toA\(L\)<0\.03A^\{\(L\)\}<0\.03: L1 SIGReg\-State \(0\.0120\.012on SYNTH,0\.0180\.018on TURK\), L6 SW\-Iso \(0\.0160\.016,0\.0210\.021\), and L14 I\-JEPA\-LLM \(0\.0260\.026,0\.0290\.029\)\. All three have explicit isotropy mechanisms in their loss construction — the empirical\-CF and sliced\-Wasserstein probes in L1/L6 and the random target\-projector of L14 — and the diagnostics confirm those mechanisms are active\. The metric does not reward this isotropy: L1 has the highest paired\-test signal on SYNTH \(ppaired=0\.10p\_\{\\text\{paired\}\}=0\.10, near\-significant\), but L6 and L14 both sit on baseline\. Neither pushing the cone tighter nor scattering it toward isotropic is a reliable route to exact\-match accuracy in this harness\.

*\(iv\) Two pre\-registered falsification routes for the structured\-null reading have completed and both reinforce it\.*The Tier\-3 Fisher\-metric variants \(Appendix[B](https://arxiv.org/html/2605.15394#A2)\) preserve baseline anisotropy \(A\(L\)=0\.13A^\{\(L\)\}=0\.13\) and baseline curvature \(C=1\.99C=1\.99\) at every component setting tested — the Fisher metric does*not*merely reroute the representational work onto a decoder\-aligned subspace, it*erases*the study’s geometric signature entirely\. This is a strong negative finding: the Euclidean JFR family’s signature \(A\(L\)=0\.78A^\{\(L\)\}=0\.78,C=1\.21C=1\.21rad\) lived in directions that the Fisher pull\-back metric of the LM head treats as zero curvature, so confining the auxiliary to decoder\-perceptible directions removes its h\-space signature without buying back exact\-match accuracy\. Decoder\-visible JEPA \(Appendix[C](https://arxiv.org/html/2605.15394#A3)\) goes the other way and engineers the auxiliary’s gradient to share its convex cone with the cross\-entropy gradient by construction; we observe the study’s first cell with positive mean gradient cosine,ρaux,CE=\+0\.186±0\.050\\rho\_\{\\mathrm\{aux,CE\}\}=\+0\.186\\pm 0\.050on TURK, against the\[−0\.69,\+0\.01\]\[\-0\.69,\+0\.01\]band every other cell occupies\. The pre\-registeredρ\>0\.20\\rho\>0\.20threshold is just missed in the mean \(one seed clears at\+0\.231\+0\.231, two do not\), and exact match remains inside seed noise \(Δ=\+0\.93\\Delta=\+0\.93pp,ppaired=0\.48p\_\{\\mathrm\{paired\}\}=0\.48; Appendix[C](https://arxiv.org/html/2605.15394#A3)\)\. Even a positive\-cone auxiliary does not, in this harness, translate to decoded\-task accuracy atn=3n=3seeds\.

D5 per\-position attribution measurements are written to the per\-cell diagnostic JSON files; the headline finding is that T3 vanilla, T3 swept, and T5 distribute auxiliary mass roughly uniformly across the front/middle/end thirds of the assistant span \(consistent with the Bachmann\-Nagarajan over\-smoothing critique\[[2](https://arxiv.org/html/2605.15394#bib.bib25)\]\), T6 multi\-scale shows a mild U\-shape, and T3\-Local concentrates auxiliary mass disproportionately at the end of the assistant span \(∼6×102\\sim 6\\times 10^\{2\}vs\.∼3×102\\sim 3\\times 10^\{2\}on the front\)\.

## Appendix BFisher\-Metric Auxiliaries: A Decoder\-Aligned Curvature Test

The structured\-null reading \(Section[VI\-D](https://arxiv.org/html/2605.15394#S6.SS4)\) makes a sharp prediction: an auxiliary that does its representational work in directions*the decoder is sensitive to*should, if the structured\-null reading is causal, recover some of the lost exact\-match signal\. The next\-token Fisher information of the LM head’s softmax, pulled back to hidden\-state space, is the canonical metric on those directions: it equals the local KL divergence of the decoded next\-token distribution to second order\. This appendix documents a Fisher\-metric variant of the JFR family, two auxiliary\-gradient interventions designed to make the geometry pressure visible to cross\-entropy, and a three\-cell empirical test of whether either change moves the structured null on TURK\.

### B\-APulled\-back next\-token Fisher metric

LetW∈ℝV×DW\\in\\mathbb\{R\}^\{V\\times D\}be the LM\-head weight matrix andh∈ℝDh\\in\\mathbb\{R\}^\{D\}a hidden state\. Writep=softmax​\(W​h\)p=\\mathrm\{softmax\}\(Wh\)for the next\-token categorical distribution\. The Fisher information of the softmax in logit space isDiag​\(p\)−p​p⊤\\mathrm\{Diag\}\(p\)\-pp^\{\\top\}; pulling it back throughWWgives the metric on hidden\-state perturbations

G​\(h\)=W⊤​\(Diag​\(p\)−p​p⊤\)​W∈ℝD×D\.G\(h\)\\;=\\;W^\{\\top\}\\bigl\(\\mathrm\{Diag\}\(p\)\-pp^\{\\top\}\\bigr\)W\\;\\in\\;\\mathbb\{R\}^\{D\\times D\}\.\(38\)The squaredGG\-norm of a perturbationv∈ℝDv\\in\\mathbb\{R\}^\{D\}admits a closed form which never materialisesGG:

‖v‖G2=𝔼y∼p​\[\(W​v\)y2\]−\(𝔼y∼p​\[\(W​v\)y\]\)2=Vary∼p​\[\(W​v\)y\]\.\\left\\lVert v\\right\\rVert\_\{G\}^\{2\}\\;=\\;\\mathbb\{E\}\_\{y\\sim p\}\\\!\\bigl\[\(Wv\)\_\{y\}^\{\\,2\}\\bigr\]\-\\bigl\(\\mathbb\{E\}\_\{y\\sim p\}\\\!\\bigl\[\(Wv\)\_\{y\}\\bigr\]\\bigr\)^\{\\\!2\}\\;=\\;\\mathrm\{Var\}\_\{y\\sim p\}\\\!\\bigl\[\(Wv\)\_\{y\}\\bigr\]\.\(39\)Identity \([39](https://arxiv.org/html/2605.15394#A2.E39)\) costs one LM\-head linear per position; at Llama\-3\.2\-1B scale \(V≈128​kV\\approx 128\\text\{k\},D≈2048D\\approx 2048\) this dominates the auxiliary’s compute and is the only added cost over the Euclidean JFR family\. The metric is calibrated against the decoder via the second\-order identity

‖v‖G2=2​KL​\(ph∥ph\+v\)\+O​\(‖v‖3\),\\left\\lVert v\\right\\rVert\_\{G\}^\{2\}\\;=\\;2\\,\\mathrm\{KL\}\\\!\\bigl\(p\_\{h\}\\,\\\|\\,p\_\{h\+v\}\\bigr\)\\;\+\\;O\(\\left\\lVert v\\right\\rVert^\{3\}\),\(40\)so penalising‖v‖G2\\left\\lVert v\\right\\rVert\_\{G\}^\{2\}penalises the change in next\-token distribution to leading order, not motion in directions the decoder ignores\.

We always evaluateppin stop\-gradient: letting the optimiser propagate gradient throughGGopens a trivial escape in which the auxiliary is shrunk by sharpening the next\-token distribution rather than by reducing curvature\. TheWWin Eq\. \([38](https://arxiv.org/html/2605.15394#A2.E38)\) is the LM head, frozen under our LoRA setup; if it were trainable, an analogous stop\-gradient onWWinsideGGwould be appropriate\.

### B\-BFisher\-metric Jacobi\-field family

Replacing the Euclidean residual norm in Eqs\. \([11](https://arxiv.org/html/2605.15394#S3.E11)\), \([15](https://arxiv.org/html/2605.15394#S3.E15)\), \([13](https://arxiv.org/html/2605.15394#S3.E13)\) with the squared Fisher norm of Eq\. \([39](https://arxiv.org/html/2605.15394#A2.E39)\) yields three auxiliaries\. LetJb,t=hb,t−h¯tJ\_\{b,t\}=h\_\{b,t\}\-\\bar\{h\}\_\{t\}be the Jacobi residual against the mini\-batch mean \(or, for Fisher\-Local\-JFR, the prompt\-neighbour centroid of Section[III\-F](https://arxiv.org/html/2605.15394#S3.SS6)\)\. Then

ℒF​\-​JFR\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{F\\text\{\-\}JFR\}\}=1\|V\|​∑\(b,t\)∈V‖Jb,t−1−2​Jb,t\+Jb,t\+1‖G​\(hb,t\)2,\\displaystyle=\\frac\{1\}\{\|V\|\}\\sum\_\{\(b,t\)\\in V\}\\bigl\\\|J\_\{b,t\-1\}\-2J\_\{b,t\}\+J\_\{b,t\+1\}\\bigr\\\|\_\{G\(h\_\{b,t\}\)\}^\{2\},\(41\)ℒF​\-​MSTB\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{F\\text\{\-\}MSTB\}\}=1\|𝒮\|​∑Δ∈𝒮1\|VΔ\|​∑\(b,t\)∈VΔ‖DΔ2​Jb,t‖G​\(hb,t\)2Δ4,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{S\}\|\}\\sum\_\{\\Delta\\in\\mathcal\{S\}\}\\frac\{1\}\{\|V\_\{\\Delta\}\|\}\\sum\_\{\(b,t\)\\in V\_\{\\Delta\}\}\\frac\{\\bigl\\\|D^\{2\}\_\{\\Delta\}J\_\{b,t\}\\bigr\\\|\_\{G\(h\_\{b,t\}\)\}^\{2\}\}\{\\Delta^\{4\}\},\(42\)ℒF​\-​LJFR\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{F\\text\{\-\}LJFR\}\}=1\|V\|​∑\(b,t\)∈V‖J~b,t−1−2​J~b,t\+J~b,t\+1‖G​\(hb,t\)2,\\displaystyle=\\frac\{1\}\{\|V\|\}\\sum\_\{\(b,t\)\\in V\}\\bigl\\\|\\tilde\{J\}\_\{b,t\-1\}\-2\\tilde\{J\}\_\{b,t\}\+\\tilde\{J\}\_\{b,t\+1\}\\bigr\\\|\_\{G\(h\_\{b,t\}\)\}^\{2\},\(43\)whereVVis the set of stenciled interior positions over the EOS\-clipped assistant span,𝒮=\{1,2,3\}\\mathcal\{S\}=\\\{1,2,3\\\}as in Eq\. \([15](https://arxiv.org/html/2605.15394#S3.E15)\),DΔ2​Jb,t=Jb,t\+Δ−2​Jb,t\+Jb,t−ΔD^\{2\}\_\{\\Delta\}J\_\{b,t\}=J\_\{b,t\+\\Delta\}\-2J\_\{b,t\}\+J\_\{b,t\-\\Delta\}, andJ~b,t=hb,t−sg​\(h¯b,tlocal\)\\tilde\{J\}\_\{b,t\}=h\_\{b,t\}\-\\mathrm\{sg\}\\\!\\bigl\(\\bar\{h\}^\{\\mathrm\{local\}\}\_\{b,t\}\\bigr\)uses the detached local centroid\. Fisher\-Local\-JFR additionally masks the interior by the conjunctionvalidb,t−1∧validb,t∧validb,t\+1\\mathrm\{valid\}\_\{b,t\-1\}\\wedge\\mathrm\{valid\}\_\{b,t\}\\wedge\\mathrm\{valid\}\_\{b,t\+1\}over the centroid validity mask, matching the Euclidean local\-JFR mask so that ”fake zero” centroids at non\-populated positions never leak into the second difference\.

#### Token\-margin weighting \(C2\)\.

Each per\-token Fisher contribution is multiplied by

wb,t=σ​\(γ​\(τ−mb,t\)\),mb,t=ℓb,t,y⋆−maxv≠y⋆⁡ℓb,t,v,w\_\{b,t\}\\;=\\;\\sigma\\\!\\bigl\(\\gamma\\,\(\\tau\-m\_\{b,t\}\)\\bigr\),\\quad m\_\{b,t\}=\\ell\_\{b,t,y^\{\\star\}\}\-\\max\_\{v\\neq y^\{\\star\}\}\\ell\_\{b,t,v\},\(44\)withy⋆y^\{\\star\}the gold next token under the HF causal\-LM shift; the weight concentrates on uncertain positions \(mb,t<τm\_\{b,t\}<\\tau\)\.τ\\tauis set adaptively per batch from theqq\-th quantile of finite margins \(q=0\.5q=0\.5default\)\.

#### PCGrad gradient surgery \(C3\)\.

The training\-time gradient cosine is mildly negative on TURK \(ρaux,CE≈−0\.05\\rho\_\{\\mathrm\{aux,CE\}\}\\approx\-0\.05\) but strongly negative on SYNTH JFR cells \(∈\[−0\.69,−0\.49\]\\in\[\-0\.69,\-0\.49\]\)\. We test asymmetric PCGrad\[[28](https://arxiv.org/html/2605.15394#bib.bib29)\]:

g~aux=gaux−min⁡\(0,c\)​gCE,c=⟨gaux,gCE⟩‖gCE‖2\+ε,\\tilde\{g\}\_\{\\mathrm\{aux\}\}=g\_\{\\mathrm\{aux\}\}\-\\min\(0,\\,c\)\\,g\_\{\\mathrm\{CE\}\},\\quad c=\\frac\{\\left\\langle g\_\{\\mathrm\{aux\}\},\\,g\_\{\\mathrm\{CE\}\}\\right\\rangle\}\{\\left\\lVert g\_\{\\mathrm\{CE\}\}\\right\\rVert^\{2\}\+\\varepsilon\},\(45\)stepping ongCE\+λ​g~auxg\_\{\\mathrm\{CE\}\}\+\\lambda\\,\\tilde\{g\}\_\{\\mathrm\{aux\}\}\. Inner products are taken over the flattened LoRA\-trainable parameters \(matching howρaux,CE\\rho\_\{\\mathrm\{aux,CE\}\}is measured\)\. The implementation runs two backward passes per step, adding roughly2×2\\timeswall\-clock and no extra hyperparameters\.

#### Results on TURK\.

TABLE VII:Pre\-registered falsification routes on NL\-RX\-TURK \(n=3n=3seeds\)\. \(a\) Tier\-3 Fisher\-metric cells; component flags*C1*Fisher\-metric Jacobi residual \(Eq\. \([39](https://arxiv.org/html/2605.15394#A2.E39)\)\),*C2*margin weighting \(Eq\. \([44](https://arxiv.org/html/2605.15394#A2.E44)\)\),*C3*PCGrad surgery \(Eq\. \([45](https://arxiv.org/html/2605.15394#A2.E45)\)\); “Twin” is the matched Euclidean variant from Table[III](https://arxiv.org/html/2605.15394#S5.T3); baselineregular=50\.67±1\.6850\.67\\pm 1\.68\. \(b\) Decoder\-visible JEPA;ρaux,CE\\rho\_\{\\mathrm\{aux,CE\}\}is Eq\. \([32](https://arxiv.org/html/2605.15394#S6.E32)\),A\(L\)A^\{\(L\)\}andCCare Eqs\. \([33](https://arxiv.org/html/2605.15394#A1.E33)\) and \([34](https://arxiv.org/html/2605.15394#A1.E34)\)\.\(a\)Tier\-3 Fisher\-metric cells\.
\(b\)Decoder\-visible JEPA\.

We tested all five Tier\-3 cell groups on TURK \(Table[VII\(a\)](https://arxiv.org/html/2605.15394#A2.T7.st1),5×3=155\\times 3=15runs\)\. C1 denotes Fisher\-metric replacement, C2 the margin weighting of Eq\. \([44](https://arxiv.org/html/2605.15394#A2.E44)\), C3 the PCGrad surgery of Eq\. \([45](https://arxiv.org/html/2605.15394#A2.E45)\)\. Four observations\.

*\(i\) Decoder\-aligned curvature does not rescue the training\-time null*\. No Fisher cell clears uncorrected pairedα=0\.10\\alpha=0\.10against the no\-auxiliary baseline\. The largest mean lift is C1 alone on T3\-Local at\+1\.33\+1\.33pp \(ppaired=0\.18p\_\{\\mathrm\{paired\}\}=0\.18\); the smallest is C1\+C2\+C3 on T3\-Local at\+0\.07\+0\.07pp \(ppaired=0\.96p\_\{\\mathrm\{paired\}\}=0\.96\)\.

*\(ii\) Five head\-to\-head regressions in the same direction*\. Every Fisher cell underperforms its matched Euclidean twin: JFR \(52\.53→50\.5352\.53\\to 50\.53,−2\.00\-2\.00pp\), MSTB\-JFR \(52\.80→51\.4052\.80\\to 51\.40,−1\.40\-1\.40pp\), Local\-JFR \(53\.20→52\.0053\.20\\to 52\.00,−1\.20\-1\.20pp\), Local\-JFR \+ margin \(53\.20→51\.5353\.20\\to 51\.53,−1\.67\-1\.67pp\), and Local\-JFR \+ margin \+ PCGrad \(53\.20→50\.7353\.20\\to 50\.73,−2\.47\-2\.47pp\)\. In particular, the strongest single cell in this study — T3\-Local Euclidean,ppaired=0\.003p\_\{\\mathrm\{paired\}\}=0\.003unpaired — falls below the pairedα=0\.10\\alpha=0\.10threshold once its Euclidean curvature is replaced by the Fisher\-metric counterpart\.

*\(iii\) Components stack monotonically in the wrong direction on T3\-Local*\. Holding the auxiliary base fixed at T3\-Local and adding components one at a time produces a monotone decrease in mean accuracy:

52\.00⏟C1 alone→51\.53⏟C1\+C2→50\.73⏟C1\+C2\+C3\.\\underbrace\{52\.00\}\_\{\\text\{C1 alone\}\}\\;\\to\\;\\underbrace\{51\.53\}\_\{\\text\{C1\+C2\}\}\\;\\to\\;\\underbrace\{50\.73\}\_\{\\text\{C1\+C2\+C3\}\}\.Each additional intervention \(margin weighting, then PCGrad surgery\) costs roughly0\.50\.5–0\.80\.8pp on top of the previous, giving a total drop of−1\.27\-1\.27pp from C1 alone to the full stack\. The PCGrad surgery in particular was predicted by the gradient\-cosine diagnostic to help whenρaux,CE\\rho\_\{\\mathrm\{aux,CE\}\}is anti\-aligned; on TURK the measuredρ≈−0\.05\\rho\\approx\-0\.05is only mildly anti\-aligned, so PCGrad should mostly be a no\-op there\. The result is consistent with the prediction in sign but somewhat negative in magnitude: removing the small CE\-conflicting component of an already\-weak auxiliary appears to remove signal CE was actually using\.

*\(iv\) Variance contracts in every Fisher cell*\. Sample standard deviation drops in every head\-to\-head: JFR0\.90→0\.640\.90\\to 0\.64, MSTB1\.40→1\.201\.40\\to 1\.20, Local\-JFR1\.91→1\.251\.91\\to 1\.25, Local\-JFR \+ margin1\.91→0\.611\.91\\to 0\.61, Local\-JFR \+ margin \+ PCGrad1\.91→0\.641\.91\\to 0\.64\. The auxiliary is doing work—the Fisher\-metric mean is just lower, and tighter, than the Euclidean mean, and tightening correlates with loss of mean accuracy rather than gain\.

#### Reading\.

The five Fisher cells return five regressions, deepening monotonically as each component is layered on T3\-Local\. Decoder alignment in the Fisher sense is therefore necessary but not sufficient for decoder\-visible signal: the Fisher norm restricts motion to directions that change next\-token probabilities, but motion in those directions still leaves exact match unmoved\. PCGrad’s additional−0\.80\-0\.80pp drop on TURK is consistent with the projection mechanism — the measured TURK anti\-alignment \(ρ≈−0\.05\\rho\\approx\-0\.05\) is small, so the projection coefficient mostly evaluates near zero, and the small negative correction removes signal CE was implicitly using\. The cleaner test is on the SYNTH JFR cells whereρ∈\[−0\.69,−0\.49\]\\rho\\in\[\-0\.69,\-0\.49\]leaves real room for the projection to act; that sweep is the most informative remaining experiment in the Tier\-3 family\.

## Appendix CDecoder\-Visible JEPA: A Pre\-Registered Falsification Test

The Tier\-3 Fisher experiment \(Appendix[B](https://arxiv.org/html/2605.15394#A2)\) tested decoder\-aligned*curvature*and lost five head\-to\-head comparisons\. The complementary falsification moves the auxiliary out of hidden\-state space entirely — scoring it on the decoded next\-token distribution rather than on the hidden state that produces it\.

### C\-AConstruction

![Refer to caption](https://arxiv.org/html/2605.15394v1/x4.png)Figure 4:Decoder\-visible JEPA architecture\. Interior\(t,t\+k\)\(t,t\+k\)pairs fork∈\{2,…,K\}k\\in\\\{2,\\ldots,K\\\}on the EOS\-clipped span \(k=1k=1omitted because it duplicates CE\) feed a residual MLP predictorqϕ​\(ht,k\)q\_\{\\phi\}\(h\_\{t\},k\), with stop\-gradient on the targetht\+kh\_\{t\+k\}; both are projected through the shared frozen LM headWWand the KL is computed in distribution space\. The margin hinge consumesW​htWh\_\{t\}directly at supervised positions\. No gradient flows throughWWor the target side\.The auxiliary head shares the LM headWWwith the cross\-entropy objective and has two terms\. Letστ\(⋅\)=softmax\(⋅/τ\)\\sigma\_\{\\tau\}\(\\cdot\)=\\mathrm\{softmax\}\(\\cdot/\\tau\)denote temperature\-τ\\tausoftmax\.

#### Multi\-horizon DV\-JEPA\-KL\.

For every interior anchor positionttin the EOS\-clipped assistant span and every horizonkkin a configurable set𝒦⊆\{2,3,…,K\}\\mathcal\{K\}\\subseteq\\\{2,3,\\ldots,K\\\}, define the per\-pair KL

KLb,t\(k\)=KL​\(sg​\[στ​\(W​hb,t\+k\)\]∥στ​\(W​qϕ​\(hb,t,k\)\)\),\\mathrm\{KL\}\_\{b,t\}^\{\(k\)\}\\;=\\;\\mathrm\{KL\}\\\!\\Big\(\\mathrm\{sg\}\\big\[\\sigma\_\{\\tau\}\(Wh\_\{b,t\+k\}\)\\big\]\\,\\Big\\\|\\,\\sigma\_\{\\tau\}\\big\(Wq\_\{\\phi\}\(h\_\{b,t\},k\)\\big\)\\Big\),\(46\)and aggregate it across pairs and horizons,

ℒDV​\-​JEPA=1\|𝒦\|​∑k∈𝒦1\|Vk\|​∑\(b,t\)∈VkKLb,t\(k\),\\mathcal\{L\}\_\{\\mathrm\{DV\\text\{\-\}JEPA\}\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{k\\in\\mathcal\{K\}\}\\frac\{1\}\{\|V\_\{k\}\|\}\\sum\_\{\(b,t\)\\in V\_\{k\}\}\\mathrm\{KL\}\_\{b,t\}^\{\(k\)\},\(47\)wheresg​\[⋅\]\\mathrm\{sg\}\[\\cdot\]is stop\-gradient andVkV\_\{k\}is the set of EOS\-clipped interior positions for which bothttandt\+kt\+klie inside the per\-row assistant length\. Thek=1k=1horizon is omitted because it duplicates CE; horizonsk≥2k\\geq 2supply information CE alone never asks for and a plausible source of the multi\-step coherence regex generation demands\.

#### Token\-margin hinge\.

At every supervised assistant position\(b,t\)\(b,t\)with HF\-shifted gold targety⋆=𝚕𝚊𝚋𝚎𝚕𝚜b,t\+1y^\{\\star\}=\\mathtt\{labels\}\_\{b,t\+1\}and LM\-head logitsz=W​hb,tz=Wh\_\{b,t\},

ℒmargin=1\|Y\|​∑\(b,t\)∈Ymax⁡\(0,m−zy⋆\+maxj≠y⋆⁡zj\),\\mathcal\{L\}\_\{\\mathrm\{margin\}\}\\;=\\;\\frac\{1\}\{\|Y\|\}\\sum\_\{\(b,t\)\\in Y\}\\max\\\!\\Big\(0,\\;m\-z\_\{y^\{\\star\}\}\+\\max\_\{j\\neq y^\{\\star\}\}z\_\{j\}\\Big\),\(48\)with\|Y\|\|Y\|the number of supervised positions\. Unlike the geometric auxiliaries this loss is not EOS\-clipped: by construction its gradient lies in cross\-entropy’s positive cone \(Sec\.[C\-B](https://arxiv.org/html/2605.15394#A3.SS2)\)\. The combined auxiliaryℒaux=ℒDV​\-​JEPA\+β​ℒmargin\\mathcal\{L\}\_\{\\mathrm\{aux\}\}=\\mathcal\{L\}\_\{\\mathrm\{DV\\text\{\-\}JEPA\}\}\+\\beta\\,\\mathcal\{L\}\_\{\\mathrm\{margin\}\}withβ=1\\beta=1is added to CE under the same warm\-up–decay schedule as every other variant\. The horizon predictorqϕq\_\{\\phi\}is a residual GELU MLP with a per\-horizon embedding,

qϕ​\(ht,k\)=ht\+MLPϕ​\(ht\+ek\),q\_\{\\phi\}\(h\_\{t\},k\)=h\_\{t\}\+\\mathrm\{MLP\}\_\{\\phi\}\(h\_\{t\}\+e\_\{k\}\),\(49\)zero\-initialised at the output soqϕ=idq\_\{\\phi\}=\\mathrm\{id\}at step zero; with hidden width512512it adds≈2\.1\\approx 2\.1M parameters, on the same order as the LoRA adapters themselves\.

### C\-BWhy this is decoder\-visible

The decoder is the argmax ofW​hWh, so two perturbationsh,h\+vh,h\+vare decoder\-equivalent whenargmaxy\(Wh\)y=argmaxy\(W\(h\+v\)\)y\\arg\\max\_\{y\}\(Wh\)\_\{y\}=\\arg\\max\_\{y\}\(W\(h\+v\)\)\_\{y\}pointwise\. Anhh\-space penalty such as‖v‖2\\left\\lVert v\\right\\rVert^\{2\}or‖v‖G​\(h\)2\\left\\lVert v\\right\\rVert^\{2\}\_\{G\(h\)\}\(Appendix[B](https://arxiv.org/html/2605.15394#A2)\) need not vanish on such avv, so the auxiliary gradient can have non\-zero norm while the task gradient is zero — the structured\-null pattern of Sec\.[V](https://arxiv.org/html/2605.15394#S5)\. The KL distance betweenστ​\(W​ht\+k\)\\sigma\_\{\\tau\}\(Wh\_\{t\+k\}\)andστ​\(W​qϕ​\(ht,k\)\)\\sigma\_\{\\tau\}\(Wq\_\{\\phi\}\(h\_\{t\},k\)\)in Eq\. \([47](https://arxiv.org/html/2605.15394#A3.E47)\) is, by contrast, invariant to any transformation that preserves the post\-softmax distribution: the auxiliary cannot fire in directions flat at the decoder\.

The margin hingemax⁡\(0,m−zy⋆\+zj′\)\\max\(0,\\,m\-z\_\{y^\{\\star\}\}\+z\_\{j^\{\\prime\}\}\)has subgradient∂ℒmargin/∂hb,t=𝟙​\[unsatisfied\]​\(Wj′−Wy⋆\)\\partial\\mathcal\{L\}\_\{\\mathrm\{margin\}\}/\\partial h\_\{b,t\}=\\mathbb\{1\}\[\\text\{unsatisfied\}\]\(W\_\{j^\{\\prime\}\}\-W\_\{y^\{\\star\}\}\), which lies in the convex cone\{Wj−Wy⋆\}j≠y⋆\\\{W\_\{j\}\-W\_\{y^\{\\star\}\}\\\}\_\{j\\neq y^\{\\star\}\}; cross\-entropy’s gradient∑j≠y⋆pt,j​\(Wj−Wy⋆\)\\sum\_\{j\\neq y^\{\\star\}\}p\_\{t,j\}\(W\_\{j\}\-W\_\{y^\{\\star\}\}\)lies in the same cone\. Whenever the margin is unsatisfied,ρaux,CE\\rho\_\{\\mathrm\{aux,CE\}\}is therefore bounded below by a strictly positive constant that depends only on the next\-token distribution’s effective support — the structural reason the decoder\-visible auxiliary should land with positive gradient cosine\.

### C\-CEmpirical results

The gradient\-cosine diagnostic \(Eq\. \([32](https://arxiv.org/html/2605.15394#S6.E32)\), Appendix[A](https://arxiv.org/html/2605.15394#A1)\) was the metric pre\-registered to adjudicate the prediction*before*exact match\. We pre\-registered the rule:*ρaux,CE\>0\.20\\rho\_\{\\mathrm\{aux,CE\}\}\>0\.20on TURK atn=3n=3seeds breaks the structured\-null reading at the diagnostic level*;ρ\\rhoinside the existing\[−0\.13,\+0\.01\]\[\-0\.13,\+0\.01\]band reinforces it\. Both metrics are now in \(Table[VII\(b\)](https://arxiv.org/html/2605.15394#A2.T7.st2)\)\.

The directional prediction holds: this is the first cell in the study with positive meanρ\\rho, four standard deviations above the upper end of the existing band\. The thresholdρ\>0\.20\\rho\>0\.20is just missed in the mean \(one of three seeds clears, two do not\), and exact match remains inside seed noise\.A\(L\)A^\{\(L\)\}andCCstay at baseline values: the gradient direction was rotated into CE’s positive cone without inducing the cone\-collapse or trajectory\-straightening signature the JFR family produced\. The pre\-registration therefore returns the third option neither side anticipated — positive but sub\-thresholdρ\\rho, no exact\-match follow\-through — which sharpens the structured\-null reading: atn=3n=3seeds, even an auxiliary engineered to share CE’s gradient cone does not translate into decoded accuracy\. Wider seed budget \(n≥10n\\geq 10\) and completion of the SYNTH arm are the cheapest follow\-ups\.

## References

- \[1\]M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. Ballas\(2023\)Self\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Note:arXiv:2301\.08243v3Cited by:[§I](https://arxiv.org/html/2605.15394#S1.p1.1),[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]G\. Bachmann and V\. Nagarajan\(2024\)The pitfalls of next\-token prediction\.InProceedings of the International Conference on Machine Learning \(ICML\),Note:arXiv:2403\.06963Cited by:[§A\-A](https://arxiv.org/html/2605.15394#A1.SS1.SSS0.Px4.p1.11),[§A\-B](https://arxiv.org/html/2605.15394#A1.SS2.p6.2)\.
- \[3\]A\. Baevski, W\. Hsu, Q\. Xu, A\. Babu, J\. Gu, and M\. Auli\(2022\)Data2vec: a general framework for self\-supervised learning in speech, vision and language\.Proceedings of the International Conference on Machine Learning \(ICML\)\.Note:arXiv:2202\.03555Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px3.p1.1)\.
- \[4\]R\. Balestriero and Y\. LeCun\(2025\)LeJEPA: provable and scalable self\-supervised learning without the heuristics\.arXiv preprint\.Note:arXiv:2511\.08544Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px3.p1.1)\.
- \[5\]A\. Bardes, J\. Ponce, and Y\. LeCun\(2022\)VICReg: variance\-invariance\-covariance regularization for self\-supervised learning\.InProceedings of the International Conference on Learning Representations \(ICLR\),Note:arXiv:2105\.04906Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]N\. Bonneel, J\. Rabin, G\. Peyré, and H\. Pfister\(2015\)Sliced and radon wasserstein barycenters of measures\.Journal of Mathematical Imaging and Vision51\(1\),pp\. 22–45\.Cited by:[§III\-L](https://arxiv.org/html/2605.15394#S3.SS12.SSS0.Px2.p1.7)\.
- \[7\]T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton\(2020\)A simple framework for contrastive learning of visual representations\.InProceedings of the International Conference on Machine Learning \(ICML\),Note:arXiv:2002\.05709Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1),[§III\-A](https://arxiv.org/html/2605.15394#S3.SS1.p4.1)\.
- \[8\]X\. Chen and K\. He\(2021\)Exploring simple siamese representation learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Note:arXiv:2011\.10566Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]K\. Ethayarajh\(2019\)How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt\-2 embeddings\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:1909\.00512Cited by:[§A\-A](https://arxiv.org/html/2605.15394#A1.SS1.SSS0.Px1.p1.5)\.
- \[10\]Q\. Garrido, Y\. Chen, A\. Bardes, L\. Najman, and Y\. LeCun\(2024\)On the duality between contrastive and non\-contrastive self\-supervised learning\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Note:arXiv:2206\.02574, oral presentationCited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.
- \[11\]J\. Grill, F\. Strub, F\. Altché, C\. Tallec, P\. H\. Richemond, E\. Buchatskaya, C\. Doersch, B\. A\. Pires, Z\. D\. Guo, M\. G\. Azar, B\. Piot, R\. Mvtchell, A\. Ahuja, E\. Agapow, and C\. Beurie\(2020\)Bootstrap your own latent: a new approach to self\-supervised learning\.InProceedings of the Conference on Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2006\.07733Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.
- \[12\]M\. S\. Halvagal, A\. Laborieux, and F\. Zenke\(2023\)Implicit variance regularization in non\-contrastive ssl\.Proceedings of the Conference on Neural Information Processing Systems \(NeurIPS\)\.Note:arXiv:2212\.04858Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]E\. A\. Hosseini and E\. Fedorenko\(2023\)Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language\.InProceedings of the Conference on Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2311\.04930Cited by:[§A\-A](https://arxiv.org/html/2605.15394#A1.SS1.SSS0.Px2.p1.5)\.
- \[14\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InProceedings of the International Conference on Learning Representations \(ICLR\),Note:arXiv:2106\.09685Cited by:[§IV](https://arxiv.org/html/2605.15394#S4.SS0.SSS0.Px2.p1.5)\.
- \[15\]H\. Huang, Y\. LeCun, and R\. Balestriero\(2025\-10\)LLM\-JEPA: large language models meet joint embedding predictive architectures\.Note:arXiv:2509\.14252v2, preprint, October 2025Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px3.p1.1),[§III\-L](https://arxiv.org/html/2605.15394#S3.SS12.SSS0.Px7.p1.3),[§V\-C](https://arxiv.org/html/2605.15394#S5.SS3.SSS0.Px1.p1.26)\.
- \[16\]H\. Huang, Y\. LeCun, and R\. Balestriero\(2026\-02\)Semantic tube prediction: beating llm data efficiency with jepa\.Note:arXiv:2602\.22617v1, preprint, February 2026Cited by:[§I](https://arxiv.org/html/2605.15394#S1.p3.1),[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px3.p1.1),[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px5.p1.1),[§III\-A](https://arxiv.org/html/2605.15394#S3.SS1.p4.1),[§IV](https://arxiv.org/html/2605.15394#S4.SS0.SSS0.Px1.p1.2),[§VI\-C](https://arxiv.org/html/2605.15394#S6.SS3.p1.2)\.
- \[17\]A\. Hyvärinen\(2005\)Estimation of non\-normalized statistical models by score matching\.Journal of Machine Learning Research6,pp\. 695–709\.Cited by:[§III\-L](https://arxiv.org/html/2605.15394#S3.SS12.SSS0.Px3.p1.2)\.
- \[18\]S\. Kolouri, K\. Nadjahi, U\. Şimşekli, R\. Badeau, and G\. Rohde\(2019\)Generalized sliced Wasserstein distances\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:1902\.00434Cited by:[§III\-L](https://arxiv.org/html/2605.15394#S3.SS12.SSS0.Px2.p1.7)\.
- \[19\]Y\. LeCun\(2022\)A path towards autonomous machine intelligence, version 0\.9\.2\.Note:OpenReviewPosition paper introducing the joint\-embedding predictive architecture \(JEPA\)Cited by:[§I](https://arxiv.org/html/2605.15394#S1.p1.1)\.
- \[20\]E\. Littwin, O\. Saremi, M\. Advani, V\. Thilak, P\. Nakkiran, C\. Huang, and J\. Susskind\(2024\)How jepa avoids noisy features: the implicit bias of deep linear self distillation networks\.InProceedings of the Conference on Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2407\.03475Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px3.p1.1)\.
- \[21\]L\. Mur\-Labadia, M\. Muckley, A\. Bar, M\. Assran, K\. Sinha, M\. Rabbat, Y\. LeCun, N\. Ballas, and A\. Bardes\(2026\-03\)V\-jepa 2\.1: unlocking dense features in video self\-supervised learning\.Note:arXiv:2603\.14482v2, preprint, March 2026Cited by:[§I](https://arxiv.org/html/2605.15394#S1.p1.1),[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]J\. Schmidhuber\(1990\)Making the world differentiable: on using self\-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non\-stationary environments\.Technical reportTechnical ReportFKI\-126\-90,Institut für Informatik, Technische Universität München\.Cited by:[§I](https://arxiv.org/html/2605.15394#S1.p1.1)\.
- \[23\]J\. Schmidhuber\(1991\)Curious model\-building control systems\.Proceedings of the International Joint Conference on Neural Networks \(IJCNN\),pp\. 1458–1463\.Cited by:[§I](https://arxiv.org/html/2605.15394#S1.p1.1)\.
- \[24\]Y\. Song, S\. Garg, J\. Shi, and S\. Ermon\(2020\)Sliced score matching: a scalable approach to density and score estimation\.InProceedings of the Conference on Uncertainty in Artificial Intelligence \(UAI\),Note:arXiv:1905\.07088Cited by:[§III\-L](https://arxiv.org/html/2605.15394#S3.SS12.SSS0.Px3.p1.4)\.
- \[25\]D\. Soudry, E\. Hoffer, M\. S\. Nacson, S\. Gunasekar, and N\. Srebro\(2018\)The implicit bias of gradient descent on separable data\.Journal of Machine Learning Research19,pp\. 1–57\.Note:arXiv:1710\.10345Cited by:[§III\-A](https://arxiv.org/html/2605.15394#S3.SS1.p4.1),[§III\-K](https://arxiv.org/html/2605.15394#S3.SS11.SSS0.Px3.p1.6)\.
- \[26\]Y\. Tian, X\. Chen, and S\. Ganguli\(2021\)Understanding self\-supervised learning dynamics without contrastive pairs\.InProceedings of the International Conference on Machine Learning \(ICML\),Note:arXiv:2102\.06810Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]Y\. Wang, O\. Bounou, G\. Zhou, R\. Balestriero, T\. G\. J\. Rudner, Y\. LeCun, and M\. Ren\(2026\-03\)Temporal straightening for latent planning\.Note:arXiv:2603\.12231v1, preprint, March 2026Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px4.p1.3),[§VI\-D](https://arxiv.org/html/2605.15394#S6.SS4.p3.4)\.
- \[28\]T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. Finn\(2020\)Gradient surgery for multi\-task learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2001\.06782Cited by:[§B\-B](https://arxiv.org/html/2605.15394#A2.SS2.SSS0.Px2.p1.2)\.
- \[29\]J\. Zbontar, L\. Jing, I\. Misra, Y\. LeCun, and S\. Deny\(2021\)Barlow twins: self\-supervised learning via redundancy reduction\.InProceedings of the International Conference on Machine Learning \(ICML\),Note:arXiv:2103\.03230Cited by:[§II](https://arxiv.org/html/2605.15394#S2.SS0.SSS0.Px2.p1.1)\.