The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA
Summary
The study finds that rank truncation in matrix-CODI models does not impact accuracy on reasoning tasks, indicating that rank may not correlate with parallel reasoning paths in latent representations.
View Cached Full Text
Cached at: 09/04/26, 06:20 AM
# The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA
Source: [https://arxiv.org/abs/2609.03090](https://arxiv.org/abs/2609.03090)
[View PDF](https://arxiv.org/pdf/2609.03090)
> Abstract:Continuous chain\-of\-thought models compress reasoning into latent tokens\. Matrix\-valued variants, which route each latent token through a d x d matrix bottleneck, introduce rank as a single\-sample structural observable on the latent matrix Z\. If matrix latents carry parallel reasoning paths via superposition, rank should track them, and truncating Z to low rank should hurt accuracy on tasks whose solutions plausibly require multiple components\. Across four training regimes of a matrix\-CODI model \(three on ProsQA, one on GSM8K\-Aug below the learning threshold\), the rank\-k projection ablation curve is flat to within 0\.6 percentage points\. A three\-seed replication yields 81\.0 \+/\- 2\.0 percentage points accuracy while the final effective rank of Z spans \{4, 12, 13\}; the loss does not reward any particular rank\. To test whether rank\-blindness arises from the flatten\-then\-project readout alone, we trained four readouts: a bilinear reparametrization, a bilinear\-plus\-GELU readout nonlinear in Z, an SVD\-augmented readout feeding singular values through an MLP, and a quadratic readout in Z Z^T\. All four rank\-k curves remain flat \(Spearman p\-values 0\.63, 0\.14, 0\.82, 0\.46\)\. The flat curves persist for readouts nonlinear in Z\. A linear probe on Z underperforms a raw pretrained hidden state at target prediction \(AUC 0\.673 vs\. 0\.846\)\. A negative control on vanilla GPT\-2 SFT \(no matrix bottleneck, no Z, three seeds, n=500\) reproduces a flat rank\-k curve under the same intervention paradigm with pooled\-mean range 0\.20pp, and a random\-h sensitivity floor lands at the same accuracy: the rank\-k ablation alone conflates rank\-blindness with position\-irrelevance\.
## Submission history
From: Samuel Larson \[[view email](https://arxiv.org/show-email/cae88526/2609.03090)\] **\[v1\]**Wed, 2 Sep 2026 19:03:24 UTC \(118 KB\)Similar Articles
Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study
This study empirically demonstrates that gradient-based LoRA rank allocation, effective in supervised fine-tuning, degrades performance in GRPO-based reinforcement learning due to flatter gradient landscapes and a gradient amplification effect.
The Loss Does Not See the Basis, but Adam Does
This paper investigates why Adam does not exhibit gradient descent's implicit low-rank bias in factored models, showing that coordinate-wise preconditioning breaks the relevant symmetry, while shared-scalar methods like Muon and Shampoo preserve it.
GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training
GRACE proposes a gradient-aligned method that scores individual reasoning steps to select the most valuable data for post-training, achieving 108.8% of full-data performance with only 20% of the data.
RankShift: In-Database Detection and Explanation of Categorical Shifts
RankShift is a novel in-database method for detecting and explaining categorical shifts in data streams, using Pearson scores to identify responsible categories, and it shows competitive performance against autoencoders on log datasets.
More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models
This research paper investigates position bias in reasoning models, finding that bias scales with the length of the reasoning trajectory rather than being eliminated by 'more thinking.' The study provides causal evidence and a diagnostic toolkit for auditing this length-driven bias in multiple-choice QA evaluations.