Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Summary
This paper demonstrates that Transformers in LLMs exhibit linear superposition when combining inputs, supporting the Superposition Linearity Hypothesis, and introduces a guided decoding method to disentangle superposed outputs.
View Cached Full Text
Cached at: 09/25/26, 03:46 PM
Paper page - Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Source: https://huggingface.co/papers/2609.29845
Abstract
WhileLargeLanguageModels(LLMs)relyonhighlynon-linearcomponents,inthisworkwedemonstratethattheyexhibitfundamentallinearity:wheninputsfromdistincttextstreamsarelinearlycombined,themodeloutputsasuperpositionoftheindividualnext-tokendistributions.WetermthistheSuperpositionLinearityHypothesis.WeprovideevidencethatsuperpositionisanintrinsicpropertyoftheTransformerarchitectureratherthananemergentconsequenceoftraining;infact,weobservethatittendstodiminishaspretrainingprogresses.However,wedemonstratethatlinearitycanbesubstantiallyrestoredthroughlightweightfine-tuning,significantlyreducingthedivergencebetweenthepredictednext-tokendistributionandtheaverageoftheindividualnext-tokendistributions.Finally,weintroduceaguideddecodingprocedurethatdisentanglessuperposedoutputs,enablingthesimultaneousgenerationoftwocoherentcontinuationsfromasingleforwardpass.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.29845
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.29845 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.29845 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.29845 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
How LLMs Actually Work (26 minute read)
A detailed walkthrough of how transformer-based LLMs work, covering tokenization, embeddings, attention, and next-token prediction without heavy math.
@v0xium: I strongly recommend reading the Transformer chapter from Speech and Language Processing by Dan Jurafsky and James H. M…
Recommends the Transformer chapter from the freely available book Speech and Language Processing by Dan Jurafsky and James H. Martin, a top resource for understanding the mathematics behind LLMs, and includes a quoted explanation of LLM pre-training.
@ickma2311: Efficient AI Lecture 12: Transformer and LLM This lecture is not only about how LLMs work. It also explains the buildin…
Lecture notes from an Efficient AI course covering Transformer and LLM fundamentals, including multi-head attention, positional encoding, KV cache, and the connection between model architecture and inference efficiency. The content explains how design choices in transformers affect memory, latency, and hardware efficiency.
How do LLMs store so much knowledge? A look at feature superposition
Explores how large language models compress vast knowledge into finite space using feature superposition, explaining the distinction between dimensions and features with biological analogies.
Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
This paper proposes Multi-Stream LLMs, which transition from sequential message-based instruction tuning to parallel stream processing. This approach allows language models to simultaneously read, think, and generate across multiple concurrent data flows, addressing bottlenecks in autonomous agent applications.