ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization

# ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization
Source: [https://arxiv.org/html/2608.12756](https://arxiv.org/html/2608.12756)
\\workshoptitleLinguistic Principles for Foundation Models

Lixing Li

###### Abstract

Adaptive latent tokenization maps a fine\-grained input to a shorter sequence of continuous representations associated with input\-dependent spans\. We introduce*ReconSpan*, which divides text into chunks that a backward decoder can reconstruct from a single contextual prefix code and retains one such code as the latent token for each chunk\. The reconstruction criterion is applied when chunks are formed, allowing one trained autoencoder to produce average chunk lengths from 6\.5 to 12\.2\. At matched average length, reconstruction\-guided boundaries preserve more text than random boundaries\. Readers of the resulting latent sequence recover topic information reliably but struggle to extract exact details\.

## 1Introduction

Language models do not operate directly on raw text: tokenization determines the sequence positions to which representation and computation are assigned\. Conventional subword tokenizers choose these units before the model runs, largely from corpus\-level frequency statistics\([25](https://arxiv.org/html/2608.12756#bib.bib1)\)\. Their granularity is therefore fixed rather than conditioned on the actual input\.

Byte\- and character\-level inputs avoid a fixed subword vocabulary and retain fine\-grained information, but they also produce substantially longer sequences\([27](https://arxiv.org/html/2608.12756#bib.bib2);[20](https://arxiv.org/html/2608.12756#bib.bib6)\)\. Adaptive latent tokenization instead groups neighboring units into input\-dependent spans and represents each span with one continuous latent token\. Its chunking rule is central: boundaries determine when new latent positions are created and thus where representation and computation are allocated\. Here, a latent token is a contextual representation assigned to a span boundary; unlike a conventional token embedding, it may encode preceding context as well as the associated span\. This makes unit formation a model\-dependent allocation decision rather than a fixed preprocessing choice\.

We propose*ReconSpan*, an adaptive latent\-tokenization method that divides text into chunks a decoder can reconstruct from a single prefix code and retains one such code as the latent token for each chunk\. A causal encoder produces a code at every position; starting from the final code, a backward decoder reconstructs until an error criterion is exceeded, accepts the resulting suffix as a chunk, and repeats from the first unreconstructed position\. Differences in reconstruction reach give shorter chunks to difficult spans and longer chunks to easier ones\. Because the criterion is applied only during chunking, it can also be relaxed to increase average chunk length without retraining, giving ReconSpan input\-dependent allocation and post\-training control of granularity\.

We evaluate both the induced tokenization and the information accessible from its latent tokens\. ReconSpan produces average chunk lengths from 6\.5 to 12\.2 tokens, and at matched average length its reconstruction\-guided boundaries preserve more text than random boundaries\. Native reconstruction tests the selected spans through the autoencoding route, while a separately trained*reader*consumes the contextual latent tokens directly and predicts text or task outputs\. These readers recover topic information reliably but struggle to extract exact details, exposing a gap between information retained by the autoencoder and information accessible to another model\.

Our contributions are:

- •We introduce reconstruction fidelity as a chunk\-allocation criterion for adaptive latent tokenization over an existing subword sequence\.
- •We show that one autoencoder supports multiple post\-training granularities and that its variable\-span boundaries outperform length\-matched random boundaries in native reconstruction\.
- •We characterize the resulting contextual latent tokens through direct readout, separating information retained by the autoencoder from information accessible to readers of different scales and levels of task adaptation\.

## 2Related Work

Learned tokenizers and hierarchical sequence models differ in the signal that determines their source\-text chunks\. We organize the closest work by this allocation rule\.

#### Chunking by fixed position\.

MEGABYTE divides byte sequences into fixed\-size patches and applies separate local and global models within and across patches\([30](https://arxiv.org/html/2608.12756#bib.bib3)\)\. Extensible Tokenization instead contextualizes existing subword embeddings and retains representations at a regular stride\([26](https://arxiv.org/html/2608.12756#bib.bib4)\)\. Its stride can be selected at inference, but equal\-size allocation does not adapt boundaries to the input\. It is especially close to ReconSpan in accepting subword inputs, emitting continuous contextual representations, and allowing granularity to change after training\.

#### Chunking by predictive entropy\.

The Byte Latent Transformer \(BLT\) creates variable byte patches at spikes in next\-byte entropy, assigning shorter patches where the sequence is less predictable\([20](https://arxiv.org/html/2608.12756#bib.bib6)\)\. Dynamic Token Pooling also studies an entropy\-supervised boundary predictor alongside its other variants\([18](https://arxiv.org/html/2608.12756#bib.bib5)\)\. These methods use forward predictive uncertainty; ReconSpan instead measures backward reconstruction fidelity\.

#### Chunking by semantic similarity\.

SemToken embeds existing tokens contextually, merges adjacent semantically similar spans, and varies granularity with local semantic density\([13](https://arxiv.org/html/2608.12756#bib.bib9)\)\. This directly targets semantic allocation\. ReconSpan does not optimize similarity or claim that its boundaries are semantic; whether reconstruction difficulty aligns with linguistic structure remains an empirical question\.

#### Chunking by learned boundary scores\.

Dynamic Token Pooling predicts variable character\-level segments using end\-to\-end, tokenizer\-supervised, entropy\-supervised, or linguistic objectives\([18](https://arxiv.org/html/2608.12756#bib.bib5)\)\. H\-Net learns content\- and context\-dependent routing jointly with a hierarchical byte\-level language model\([9](https://arxiv.org/html/2608.12756#bib.bib7)\)\. Charformer is an earlier soft\-block precursor: it scores candidate byte blocks from the end\-task loss, although its final downsampling is fixed\([27](https://arxiv.org/html/2608.12756#bib.bib2)\)\. FLEXITOKENS likewise learns variable byte boundaries while relaxing the fixed target\-rate objective used by related models\([19](https://arxiv.org/html/2608.12756#bib.bib8)\)\. In these systems, allocation is trained as part of language modeling\.

#### Chunking by reconstruction fidelity\.

ReconSpan places a boundary according to how far a backward decoder can successfully reconstruct from each contextual encoder state\. The signal is measured autoencoder reconstruction rather than a fixed position, learned router, forward entropy, or semantic density\. Changing the accepted reconstruction criterion adjusts average span length after training\. Our contribution is this allocation criterion and a characterization of the resulting latent representations, not continuous tokens or adaptive segmentation by themselves\.

## 3Method

ReconSpan requires a generic autoencoding model and an inference\-time chunking algorithm\.

### 3\.1Autoencoding model

Letx1:nx\_\{1:n\}be a token sequence\. ReconSpan requires two learned components\.

- •A*prefix encoder*maps any token sequence to one code, E:𝒱∗→ℝd,ct=E\(x1:t\)\.E:\\mathcal\{V\}^\{\*\}\\rightarrow\\mathbb\{R\}^\{d\},\\qquad c\_\{t\}=E\(x\_\{1:t\}\)\.\(1\)A causal model such as a Transformer or a Mamba can generatec1,…,cnc\_\{1\},\\ldots,c\_\{n\}in one pass\.
- •A*backward decoder*mapsctc\_\{t\}to the encoded tokens in reverse order,xt,xt−1,…x\_\{t\},x\_\{t\-1\},\\ldots\. Backward decoding is the point of the design: decoding newest\-first, the position where the decoder first fails is a direct measurement of how far back that one code reconstructs\. A forward decoder would instead have to be told where to start — the very quantity we want to measure\. ReconSpan’s reach\-based criterion therefore relies on backward decoding\.

The autoencoding path is therefore

\(x1,x2,…,xt\)→𝐸ct→𝐷\(xt,xt−1,…,x1\)\.\(x\_\{1\},x\_\{2\},\\ldots,x\_\{t\}\)\\xrightarrow\{E\}c\_\{t\}\\xrightarrow\{D\}\(x\_\{t\},x\_\{t\-1\},\\ldots,x\_\{1\}\)\.\(2\)

### 3\.2Chunking algorithm

The decoder will not reconstruct arbitrarily long prefixes, so we only ask it to decode within its own capacity; the point at which it fails sets each chunk boundary, giving adaptive\-length chunks and their latent tokens\. Algorithm[1](https://arxiv.org/html/2608.12756#algorithm1)states the procedure\. Because the text being tokenized is already known, the decoder is*teacher\-forced*against it: at each reverse step it is fed the true previous tokens and we record only whether its own greedy \(argmax\) prediction matches — nothing is sampled or generated\.

Write the resulting chunk endpoints in chronological order as0=b0<b1<⋯<bm=n0=b\_\{0\}<b\_\{1\}<\\cdots<b\_\{m\}=n, so chunkiiisxbi−1\+1:bix\_\{b\_\{i\-1\}\+1:b\_\{i\}\}and its contextual latent token iscbi=E\(x1:bi\)c\_\{b\_\{i\}\}=E\(x\_\{1:b\_\{i\}\}\)\.

Algorithm 1ReconSpan chunking\.Input:tokensx1:nx\_\{1:n\}, encoderEE, backward decoderDD, stopping rule\.Output:contextual latent\-token sequence𝒞\\mathcal\{C\}\.

1. 1\.Compute all prefix codes\(c1,…,cn\)←E\(x1:n\)\(c\_\{1\},\\ldots,c\_\{n\}\)\\leftarrow E\(x\_\{1:n\}\)in one causal pass; set𝒞←⟨⟩\\mathcal\{C\}\\leftarrow\\langle\\,\\rangleandt←nt\\leftarrow n\.
2. 2\.whilet≥1t\\geq 1: 1. \(a\)Teacher\-forceDDfromctc\_\{t\}againstxt,xt−1,…x\_\{t\},x\_\{t\-1\},\\ldotsuntil the stopping rule fires atxbx\_\{b\}, so thatxb\+1,…,xtx\_\{b\+1\},\\ldots,x\_\{t\}are accepted; setb←0b\\leftarrow 0if it reaches the beginning without firing\. If it fires on the first step, setb←t−1b\\leftarrow t\-1\(a one\-token chunk, guaranteeing progress\)\. 2. \(b\)Appendctc\_\{t\}to𝒞\\mathcal\{C\}as the code for chunkxb\+1,…,xtx\_\{b\+1\},\\ldots,x\_\{t\}; sett←bt\\leftarrow bto resume from the first unreconstructed endpoint\.
3. 3\.Reverse the collected latent tokens into chronological order andreturn𝒞=\(cb1,…,cbm\)\\mathcal\{C\}=\(c\_\{b\_\{1\}\},\\ldots,c\_\{b\_\{m\}\}\)— only these chunk\-boundary prefix codes are retained\.

We use two stopping families\.*Failure\(mm\)*stops at themmth incorrectly reconstructed token\.*Failure\(1\)*ends a chunk at the first mistake and largermmtolerates errors\.*Logit\-gap\(τ\\tau\)*is the continuous version\. Letyjy\_\{j\}be the actual token at reverse stepjj; the decoder stops at the firstkkfor which

Gk=∑j=1k\(maxv∈𝒱⁡zv\(j\)−zyj\(j\)\)\>τ\.G\_\{k\}=\\sum\_\{j=1\}^\{k\}\\left\(\\max\_\{v\\in\\mathcal\{V\}\}z\_\{v\}^\{\(j\)\}\-z\_\{y\_\{j\}\}^\{\(j\)\}\\right\)\>\\tau\.\(3\)Each summand is zero when the model predicts the correct token and otherwise measures by how much it was missed, soGkG\_\{k\}accumulates near\-misses instead of counting outright errors\. Raisingmmorτ\\taulengthens chunks on a fixed trained model and thus increases the average number of input tokens represented by each latent token\.

The number of sequential model invocations, which is architecture\-invariant, isO⁡\(1\+n\)O\(1\+n\)\. Producing all prefix codesc1,…,cnc\_\{1\},\\ldots,c\_\{n\}takes a single encoder forward pass, whatever the encoder architecture\. The backward reconstruction is autoregressive, so in the worst case — a failure at every step, making every chunk one token — it needsO⁡\(n\)O\(n\)sequential decoder calls\.

In practice, the decoder reads a fixed block ofWWpositions per endpoint for efficient batching, so decoding isO⁡\(W​n\)O\(Wn\)work in the worst case with our specific Mamba decoder\. Appendix[D](https://arxiv.org/html/2608.12756#A4)describes a variant that keeps the sameO⁡\(W​n\)O\(Wn\)total work but cuts the sequential decode calls toO⁡\(W\)O\(W\), independent ofnn, by materializing all endpoint decodes at once, at the cost of more physical computation\.

### 3\.3Training

The only training objective is autoencoder reconstruction; the span allocation emerges from the decoder’s capacity rather than from an explicit length target\. A training step operates on one windowx1:Lx\_\{1:L\}\. The encoder produces its final codecL=E\(x1:L\)c\_\{L\}=E\(x\_\{1:L\}\), a learned projection maps it to the decoder’s initial recurrent state, and the decoder is teacher\-forced to reproduce the window in reverse,y=\(xL,…,x1,EOS\)y=\(x\_\{L\},\\ldots,x\_\{1\},\\mathrm\{EOS\}\)\. Each step reconstructs the firstk≤L\+1k\\leq L\+1reversed targets, and the loss is the token cross\-entropy over them,

ℒAE=−∑j=1klogpD\(yj∣y<j,cL\)\.\\mathcal\{L\}\_\{\\mathrm\{AE\}\}=\-\\sum\_\{j=1\}^\{k\}\\log p\_\{D\}\(y\_\{j\}\\mid y\_\{<j\},c\_\{L\}\)\.\(4\)The projection is differentiable and no stop\-gradient is placed on the code, so this loss reaches the encoder as well as the decoder\. Encoder and decoder can therefore be trained jointly\.

Capping the decode length atk≤L\+1k\\leq L\+1makes most steps reconstruct only a*suffix*of the window rather than the whole text\. This is a deliberate match to how ReconSpan uses the decoder: the design is intended to prioritize recent tokens and thereby extend the successful suffix that determines a chunk boundary\.

### 3\.4Implementation overview

Our encoder is a Pythia\-410M Transformer\([1](https://arxiv.org/html/2608.12756#bib.bib14)\)whose final 1024\-dimensional hidden state is the code; the decoder is a Mamba2\-130M backward model\([5](https://arxiv.org/html/2608.12756#bib.bib13)\)\. The code is up\-projected to the decoder’s per\-layer initial SSM states, and a BOS token begins the rollout\. Pythia shares the GPT\-NeoX tokenizer with the Mamba2 decoder, so the two never disagree on vocabulary\. We also evaluate two auxiliary Mamba2 encoders: a 4096\-dimensional projected SSM state and a 1024\-dimensional hidden state\. The former requires four times the code width to approach the Transformer’s short\-span reconstruction, while the latter is weaker at equal width\. Appendix[A](https://arxiv.org/html/2608.12756#A1)defines these variants and reports their reconstruction comparison\.

Each step samples a short\-biased encode lengthLLand an independent decode lengthk≤L\+1k\\leq L\+1; Section[3\.3](https://arxiv.org/html/2608.12756#S3.SS3)explains why a partial suffix, not the full window, is the right target\. Training runs in two stages on FineWeb\([23](https://arxiv.org/html/2608.12756#bib.bib17)\): a 7B\-token backward\-decoder pretrain with the encoder frozen, then 3B tokens of joint encoder–decoder training\. Appendix[A](https://arxiv.org/html/2608.12756#A1)gives the data pipeline, sampling ranges, schedule, and optimizer\.

## 4Tokenizer properties

We characterize the tokenizer through four measurements: single\-code autoencoder reconstruction, semantic structure in the raw code geometry, the span lengths induced by different stopping rules, and how much text survives a native autoencoder round trip under the selected boundaries\.

### 4\.1Autoencoder reconstruction quality

Table 1:Single\-code reconstruction on 500 held\-out Wikipedia windows, Transformer encoder \(left\) versus SONAR \(right\); both use 1024\-dimensional codes\. Exact is full\-window exact match; RL is ROUGE\-L; suffix is the mean length of the longest exactly reconstructed suffix — the contiguous run of most\-recent tokens the backward decoder emits without an error\.ReconSpan Transformer

SONAR

We measure how much a single code can give back, comparing our Transformer encoder against SONAR\([6](https://arxiv.org/html/2608.12756#bib.bib11)\)\. For each lengthℓ\\ellwe sample oneℓ\\ell\-token window from each of 500 held\-out Wikipedia documents, encode the whole window into one code, decode it autoregressively in reverse, and score the result against the original using BLEU\([22](https://arxiv.org/html/2608.12756#bib.bib21)\)and ROUGE\-L\([12](https://arxiv.org/html/2608.12756#bib.bib22)\), alongside exact\-match measures\. SONAR is a multilingual text autoencoder that maps a sentence to one 1024\-dimensional embedding and decodes it back, so its code width matches ours; as a strong open\-source autoencoder it gives an external reference point for our reconstruction quality\.

Table[1](https://arxiv.org/html/2608.12756#S4.T1)reports the comparison\. The key column is*suffix*, the quantity most closely related to the reconstruction reach used to set chunk length\. The Transformer recovers an average exact suffix of roughly88–1111tokens fromℓ=8\\ell=8toℓ=256\\ell=256, whereas SONAR’s suffix falls sharply beyondℓ=32\\ell=32even when its aggregate overlap remains competitive\. This stable short\-range reconstruction is the capacity needed by the chunking experiments that follow\.

### 4\.2Semantic geometry

Reconstruction asks a code to retain the tokens of its window, but does not directly place semantically similar inputs nearby\. We probe this untrained property with four tasks from MTEB\([17](https://arxiv.org/html/2608.12756#bib.bib10)\): STSBenchmark and STS17\([3](https://arxiv.org/html/2608.12756#bib.bib30)\), SICK\-R\([15](https://arxiv.org/html/2608.12756#bib.bib31)\), and NFCorpus\([2](https://arxiv.org/html/2608.12756#bib.bib32)\)\. Table[2](https://arxiv.org/html/2608.12756#S4.T2)shows that SONAR, which is trained with a similarity objective, leads on every task, often by a wide margin\. Thus the raw code geometry is not strongly organized by semantic similarity\. This protocol evaluates one full\-input encoder code rather than a sequence of boundary tokens, so it characterizes the code generator rather than the complete tokenizer\.

Table 2:MTEB is a standard benchmark that measures how well fixed\-size text embeddings capture meaning, through semantic\-similarity and retrieval tasks; here we apply it to the raw codes\. STSBenchmark and STS17 \(English\) score how well code cosine similarity tracks human sentence\-pair ratings, and SICK\-R does the same on sentence pairs probing compositional meaning\. NFCorpus is a biomedical document\-retrieval task\. Higher is better on all four\.
### 4\.3Adaptive span allocation

![Refer to caption](https://arxiv.org/html/2608.12756v1/figures/chunk_length_distributions.png)Figure 1:Chunk\-length distributions \(raw counts\) for four stopping rules on the same 500 Wikipedia documents, each capped at 1000 tokens; vertical lines mark the mean\. Looser rules \(left to right, top to bottom\) shift mass toward longer chunks\. Teacher\-forced decoding evaluatesW=32W=32positions per endpoint, so theW=32W\{=\}32cap truncates the few windows the decoder could carry further, producing the small pile\-up in the length\-32 bin \(<1%1\\%of chunks\)\.We study how the stopping criterion controls span allocation and whether the resulting chunk lengths vary across inputs\. Because every chunk contributes one latent position, mean chunk length directly measures the tokenizer’s average granularity\. A looser rule — one that permits the decoder to continue further before stopping — increases this mean\. Figure[1](https://arxiv.org/html/2608.12756#S4.F1)shows this across four rules applied post hoc to one trained model on 500 Wikipedia documents, spanning mean lengths of 6\.50 to 12\.17 input tokens\. The behavior carries to larger scale: over 2 million FineWeb documents,*Failure\(1\)*gives a mean chunk length of 9\.56 tokens\. Realized averages also differ across Wikipedia, FineWeb, and the downstream datasets in Section[5\.2](https://arxiv.org/html/2608.12756#S5.SS2), reflecting input\-dependent variation in reconstruction reach\.

The one caveat is the spike at length one, which has a mechanical cause\. The backward decoder predicts the newest token first, with no reconstructed context to condition on, so that token is the hardest to get right\. Two outcomes then both produce a length\-one chunk: the decoder misses this first token, and the forced\-progress rule of Section[3\.2](https://arxiv.org/html/2608.12756#S3.SS2)retains it anyway; or it reconstructs the first token but misses the second\.*Failure\(1\)*thus piles both the stop\-at\-one and stop\-at\-two cases onto length one\. These single\-token chunks are common but carry only 4\.3% of the text, so they cost latent positions without greatly changing the mean; a looser rule or an explicit minimum chunk length removes most of them\.

### 4\.4Native reconstruction under selected boundaries

We test whether capacity\-based boundaries identify spans that survive a native autoencoder round trip\. After ReconSpan selects the boundaries, each resulting chunk is encoded in isolation and decoded autoregressively; Appendix[C](https://arxiv.org/html/2608.12756#A3)defines this protocol formally and distinguishes its chunk codes from the prefix codes consumed directly by the reader\.

Two conclusions follow\. First, at an identical latent\-token count and mean length, ReconSpan boundaries recover more of each document than the length\-matched random control, so where the boundaries fall, not merely how many there are, decides how much text survives\. Second, the stopping rule acts as a granularity dial\. The results suggest that mean chunk length largely predicts quality across the two rule families: failure and logit\-gap rules at similar mean lengths reach similar reconstruction quality\. Thus the same trained model exposes a controllable quality–granularity tradeoff\.

Table 3:Native reconstruction under ReconSpan boundaries on held\-out WikiText\([16](https://arxiv.org/html/2608.12756#bib.bib27)\)\.*Mean chunk len\.*is the mean chunk length in tokens\.*Exact chunk*is the fraction of chunks reconstructed perfectly;*exact token*is the fraction of document tokens that lie inside such perfect chunks;*suffix*is the fraction of tokens inside each chunk’s correctly reconstructed newest\-first run, crediting chunks that are only partly right\. PPL is conditional Qwen2\.5\-1\.5B perplexity\([24](https://arxiv.org/html/2608.12756#bib.bib19)\)\. The*random*row places boundaries uniformly while matching*Failure\(1\)*’s per\-document chunk count; despite similar exact\-chunk accuracy, it trails ReconSpan on all token\-level measures\.

## 5Reading the latent tokens

Can downstream language models be trained to operate directly on ReconSpan’s variable\-length latent\-token sequence, and what information can they recover? We first define and train readers that predict text directly from the latent tokens, then evaluate their access to semantic, lexical, and retrieval information\.

### 5\.1Reader model and training

We train a separate language model, called a*reader*, to predict text directly from the latent\-token sequence rather than from the source text\. For the boundaries0=b0<⋯<bm=n0=b\_\{0\}<\\cdots<b\_\{m\}=nfrom Section[3\.2](https://arxiv.org/html/2608.12756#S3.SS2), the latent\-token prefix at boundarybib\_\{i\}is

𝒞≤i:=\(cb1,…,cbi\),cbj=E\(x1:bj\)\.\\mathcal\{C\}\_\{\\leq i\}:=\(c\_\{b\_\{1\}\},\\ldots,c\_\{b\_\{i\}\}\),\\qquad c\_\{b\_\{j\}\}=E\(x\_\{1:b\_\{j\}\}\)\.\(5\)The readerRRmaps this variable\-length sequence of continuous codes to a distribution over output token sequences,

R:\(ℝd\)∗→Δ⁡\(𝒱∗\),pR​\(y∣𝒞≤i\)=∏j=1\|y\|pR​\(yj∣𝒞≤i,y<j\)\.R:\(\\mathbb\{R\}^\{d\}\)^\{\*\}\\rightarrow\\Delta\(\\mathcal\{V\}^\{\*\}\),\\qquad p\_\{R\}\(y\\mid\\mathcal\{C\}\_\{\\leq i\}\)=\\prod\_\{j=1\}^\{\|y\|\}p\_\{R\}\(y\_\{j\}\\mid\\mathcal\{C\}\_\{\\leq i\},y\_\{<j\}\)\.\(6\)Thus the reader predicts a continuation or task output directly from contextual prefix codes, without first decoding them back to text\.

Each code is standardized coordinate\-wise using corpus\-level mean and standard deviation, then mapped to the reader’s embedding dimension by a learned linear adapter and RMSNorm\([31](https://arxiv.org/html/2608.12756#bib.bib23)\)\. The adapted codes are followed by a separator and the text output\. During generic training, the reader receives a randomly truncated latent\-token prefix formed with*Failure\(1\)*; next\-token loss is masked on the codes and separator and applied only to the continuation\. We fully fine\-tune Pythia\-410M, whereas Llama\-3\-8B\([7](https://arxiv.org/html/2608.12756#bib.bib29)\)updates LoRA weights\([8](https://arxiv.org/html/2608.12756#bib.bib24)\), the adapter, and RMSNorm\. Training uses generic FineWeb continuations\.

Table[4](https://arxiv.org/html/2608.12756#S5.T4)shows that the latent tokens support fluent generation\. Under an external perplexity scorer, continuations from the larger reader are comparable to human text\. Fluency does not establish faithfulness, however: qualitative samples preserve topic, register, and local syntax while exact content remains difficult to access\. Appendix[E](https://arxiv.org/html/2608.12756#A5)gives further implementation details and reports unsuccessful code\-to\-code reader variants\.

Table 4:Conditional perplexity of generated continuations on 150 held\-out WikiText windows under Qwen2\.5\-1\.5B\([24](https://arxiv.org/html/2608.12756#bib.bib19)\); lower is better\. The Llama reader and human continuations are similar, while the smaller reader and native decode are moderately higher\. Learned readers use top\-kksampling; repetitive greedy decoding is omitted\.
### 5\.2Information accessible to downstream readers

We test what the reader can recover along a spectrum of information specificity\. AG News\([32](https://arxiv.org/html/2608.12756#bib.bib16)\)tests coarse semantic information through four\-way topic classification\. LAMBADA\([21](https://arxiv.org/html/2608.12756#bib.bib15)\)requires exact lexical information to predict a passage’s final word\. HotpotQA\([29](https://arxiv.org/html/2608.12756#bib.bib18)\)requires multi\-hop retrieval from ten documents followed by open\-answer generation\. Across these tasks, we vary reader scale, task\-specific adaptation, and chunking granularity\.

Table[5](https://arxiv.org/html/2608.12756#S5.T5)measures direct access against two references\. The shuffled control replaces each example’s codes with codes from another example, so improvement over it shows that predictions use example\-specific information\. The round trip instead decodes isolated chunk codes back to text before prediction; it measures information retained by the native autoencoder route and provides a route\-specific reference\. Together, these comparisons separate information retention from reader access\.

Across the three tasks, readers access coarse semantics more readily than exact details\. The Llama reader reaches raw\-text\-level topic classification on AG News, whereas LAMBADA remains far below the round\-trip reference\. Generic readout is also weak on HotpotQA and does not improve over the shuffled control\. Thus the codes support semantic recognition, but exact lexical access and evidence retrieval remain difficult for a generic reader\.

Task adaptation provides a second, supervised reference for direct readout under the training used here\. It improves all three tasks and moves HotpotQA toward the round\-trip reference, although its advantage over the shuffled control does not reach the marked confidence threshold\. These gains show that targeted training can extract task\-relevant information left unused by generic FineWeb continuation training\.

Although the readers are trained only with*Failure\(1\)*codes, they do not collapse when evaluated with the coarser*Failure\(2\)*policy\. Topic classification remains stable, LAMBADA remains numerically above its shuffled control, and HotpotQA is comparable to its generic*Failure\(1\)*result\. The learned interface therefore tolerates a substantial change in chunk granularity without retraining\.

Overall, the native route retains information that current readers do not fully access\. Topic information is readily recoverable by a downstream language model, task supervision narrows the access gap, and exact\-detail readout remains the central limitation\.

Table 5:Downstream readout from ReconSpan latent tokens \(%\)\.*Control*substitutes another example’s codes\.*Text*gives raw context to the untouched base model, and*roundtrip*gives it autoencoder\-reconstructed text\. Metrics are exact next\-word accuracy for LAMBADA, four\-way closed\-set accuracy for AG News, and answer exact match for HotpotQA\. A superscript∗\*marks readout above the shuffled control at 95% confidence, using a normal approximation to the difference of binomial proportions with standard errors computed from the evaluation\-set size\. Task FT is a supervised direct\-readout reference, not a matched\-training baseline\. Mean len\. is the average number of input tokens per latent token; unmarked rows use*Failure\(1\)*\.

## 6Limitations and future directions

The experiments expose several limitations of the current tokenizer, autoencoder, and reader\.

### 6\.1Limitations

#### Reader access\.

Direct\-code readers remain weak on exact\-content and retrieval tasks, even when scaling the reader or adapting it to the target task improves performance\. By contrast, the native round trip answers the same tasks much better, showing that isolated codes for the selected chunks retain information the current reader cannot yet access\. It is a reference for the native reconstruction route, while task\-adapted direct readout provides a separate, supervised reference for reader access\. One hypothesis is that code geometry makes this access difficult: the autoencoder is trained for reconstruction rather than to place semantically similar inputs nearby, and its weak MTEB results \(Section[4\.2](https://arxiv.org/html/2608.12756#S4.SS2)\), especially on retrieval, are consistent with this explanation but do not establish it as a cause\.

#### Short chunks\.

Too much of the latent sequence is spent on very short chunks\. Under*Failure\(1\)*, 28\.1% of chunks cover a single token\. These chunks carry only 4\.3% of the text while consuming 28\.1% of the codes, so a disproportionate share of the latent sequence represents very little source text\. Relaxing the rule to*Failure\(2\)*reduces them to 4\.1%, which is part of why that setting nearly doubles the mean chunk length at little measured cost on topic\-level tasks; a stopping rule with an explicit minimum chunk length is an obvious remaining improvement\.

#### Backward\-decoder pretraining\.

ReconSpan requires a model that decodes text backward from a code\. Such pretrained decoders are not readily available, so our decoder must be trained from scratch before the autoencoder can be jointly optimized\. This raises the entry cost relative to methods built entirely from existing pretrained language models\.

#### Boundary\-selection cost\.

The reconstruction scan is computationally expensive at corpus scale\. Although it isO⁡\(n\)O\(n\)for fixedWW, constructing the reader corpus took about 12 hours for 2M documents, roughly two to three times the 4–5 hours of reader training it fed\. Fewer latent positions also do not by themselves establish lower end\-to\-end compute: the Transformer encoder has quadratic work in source length, and a one\-use setting must pay the boundary\-selection cost before any downstream benefit\.

#### Scope of the tokenizer\.

ReconSpan operates over an existing subword sequence and is therefore a higher\-level latent tokenizer, not a replacement for byte\-to\-text tokenization\. Its native reconstruction and direct\-reader routes also use different code constructions \(Appendix[C](https://arxiv.org/html/2608.12756#A3)\)\. Finally, we do not compare downstream language\-model quality or efficiency directly with end\-to\-end byte tokenizers such as BLT or H\-Net\.

### 6\.2Future directions

Readers trained at larger scale, on better\-matched data, or with objectives aimed at exact retrieval may recover more of the information demonstrated by the native round trip\.

Training the autoencoder with an additional semantic\-clustering objective could make related inputs easier for a reader to recognize\. This would also directly test the hypothesis, motivated by Section[4\.2](https://arxiv.org/html/2608.12756#S4.SS2), that unstructured code geometry contributes to weak readout\.

A different native decoder could replace the backward language model\. For example, a Transformer decoder could receive the code through a soft prompt\([11](https://arxiv.org/html/2608.12756#bib.bib25)\)or cross\-attention\([28](https://arxiv.org/html/2608.12756#bib.bib20)\)\.

The linguistic structure of the boundaries placed by ReconSpan remains unexplored\. Testing whether they align with syntax, discourse, or information density could explain what the reconstruction criterion treats as difficult and guide better allocation rules\. Appendix[F](https://arxiv.org/html/2608.12756#A6)visualizes how changes in reconstruction reach produce the boundaries for one example\.

A further direction is to evaluate ReconSpan as a context\-compression interface, measuring prefill latency, KV\-cache memory, boundary\-selection cost, and amortization across repeated reads rather than inferring efficiency from sequence reduction alone\.

## 7Conclusion

ReconSpan forms adaptive latent tokens by retaining contextual encoder states at boundaries set by backward reconstruction reach\. One autoencoder exposes mean chunk lengths from 6\.50 to 12\.17 after training, and its boundaries reconstruct better than length\-matched random ones\. Downstream language models operate directly on these sequences, recovering topics more readily than exact lexical details; task adaptation narrows the native\-reconstruction gap\. These results establish reconstruction fidelity as a viable allocation criterion and motivate stronger readers and linguistic boundary analysis\.

## References

- Bidermanet al\.\(2023\)S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. Van Der WalPythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 2397–2430\.External Links:[Link](https://proceedings.mlr.press/v202/biderman23a.html)Cited by:[§3\.4](https://arxiv.org/html/2608.12756#S3.SS4.p1.1)\.
- Botevaet al\.\(2016\)V\. Boteva, D\. Gholipour Ghalandari, A\. Sokolov, and S\. RiezlerA full\-text learning to rank dataset for medical information retrieval\.InAdvances in Information Retrieval: 38th European Conference on IR Research,pp\. 716–722\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-30671-1%5F58)Cited by:[§4\.2](https://arxiv.org/html/2608.12756#S4.SS2.p1.1)\.
- Ceret al\.\(2017\)D\. Cer, M\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. SpeciaSemEval\-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation\.InProceedings of the 11th International Workshop on Semantic Evaluation \(SemEval\-2017\),pp\. 1–14\.External Links:[Document](https://dx.doi.org/10.18653/v1/S17-2001),[Link](https://aclanthology.org/S17-2001/)Cited by:[§4\.2](https://arxiv.org/html/2608.12756#S4.SS2.p1.1)\.
- Chenet al\.\(2016\)T\. Chen, B\. Xu, C\. Zhang, and C\. GuestrinTraining deep nets with sublinear memory cost\.arXiv preprint arXiv:1604\.06174\.Cited by:[Appendix A](https://arxiv.org/html/2608.12756#A1.SS0.SSS0.Px8.p1.1)\.
- Dao and Gu \(2024\)T\. Dao and A\. GuTransformers are SSMs: generalized models and efficient algorithms through structured state space duality\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 10041–10071\.External Links:[Link](https://proceedings.mlr.press/v235/dao24a.html)Cited by:[§3\.4](https://arxiv.org/html/2608.12756#S3.SS4.p1.1)\.
- Duquenneet al\.\(2023\)P\. Duquenne, H\. Schwenk, and B\. SagotSONAR: sentence\-level multimodal and language\-agnostic representations\.arXiv preprint arXiv:2308\.11466\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.11466),[Link](https://arxiv.org/abs/2308.11466)Cited by:[§4\.1](https://arxiv.org/html/2608.12756#S4.SS1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1](https://arxiv.org/html/2608.12756#S5.SS1.p2.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§5\.1](https://arxiv.org/html/2608.12756#S5.SS1.p2.1)\.
- Hwanget al\.\(2026\)S\. Hwang, B\. Wang, and A\. GuDynamic chunking for end\-to\-end hierarchical sequence modeling\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZbfLR9NbNF)Cited by:[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px4.p1.1)\.
- LCM Teamet al\.\(2024\)LCM Team, L\. Barrault, P\. Duquenne, M\. Elbayad, A\. Kozhevnikov, B\. Alastruey, P\. Andrews, M\. Coria, G\. Couairon, M\. R\. Costa\-jussà, D\. Dale, H\. Elsahar, K\. Heffernan, J\. M\. Janeiro, T\. Tran, C\. Ropers, E\. Sánchez, R\. San Roman, A\. Mourachko, S\. Saleem, and H\. SchwenkLarge concept models: language modeling in a sentence representation space\.arXiv preprint arXiv:2412\.08821\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.08821),[Link](https://arxiv.org/abs/2412.08821)Cited by:[Appendix E](https://arxiv.org/html/2608.12756#A5.p1.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.243),[Link](https://aclanthology.org/2021.emnlp-main.243/)Cited by:[§6\.2](https://arxiv.org/html/2608.12756#S6.SS2.p3.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§4\.1](https://arxiv.org/html/2608.12756#S4.SS1.p1.1)\.
- Liu and Yu \(2026\)D\. Liu and Y\. YuSemToken: semantic\-aware tokenization for efficient long\-context language models\.InProceedings of the 15th Joint Conference on Lexical and Computational Semantics \(\*SEM 2026\),pp\. 1–12\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.starsem-conference.1),[Link](https://aclanthology.org/2026.starsem-conference.1/)Cited by:[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px3.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InThe Seventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[Appendix A](https://arxiv.org/html/2608.12756#A1.SS0.SSS0.Px7.p1.1)\.
- Marelliet al\.\(2014\)M\. Marelli, S\. Menini, M\. Baroni, L\. Bentivogli, R\. Bernardi, and R\. ZamparelliA SICK cure for the evaluation of compositional distributional semantic models\.InProceedings of the Ninth International Conference on Language Resources and Evaluation \(LREC’14\),pp\. 216–223\.External Links:[Link](https://aclanthology.org/L14-1314/)Cited by:[§4\.2](https://arxiv.org/html/2608.12756#S4.SS2.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.InThe Fifth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[Table 3](https://arxiv.org/html/2608.12756#S4.T3)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, N\. Tazi, L\. Magne, and N\. ReimersMTEB: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.148),[Link](https://aclanthology.org/2023.eacl-main.148/)Cited by:[§4\.2](https://arxiv.org/html/2608.12756#S4.SS2.p1.1)\.
- Nawrotet al\.\(2023\)P\. Nawrot, J\. Chorowski, A\. Lancucki, and E\. M\. PontiEfficient transformers with dynamic token pooling\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6403–6417\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.353),[Link](https://aclanthology.org/2023.acl-long.353/)Cited by:[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px4.p1.1)\.
- Owodunniet al\.\(2026\)A\. T\. Owodunni, O\. Ahia, and S\. KumarFLEXITOKENS: flexible tokenization for evolving language models\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 17170–17190\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.848),[Link](https://aclanthology.org/2026.findings-acl.848/)Cited by:[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px4.p1.1)\.
- Pagnoniet al\.\(2025\)A\. Pagnoni, R\. Pasunuru, P\. Rodriguez, J\. Nguyen, B\. Muller, M\. Li, C\. Zhou, L\. Yu, J\. E\. Weston, L\. Zettlemoyer, G\. Ghosh, M\. Lewis, A\. Holtzman, and S\. IyerByte latent transformer: patches scale better than tokens\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9238–9258\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.453),[Link](https://aclanthology.org/2025.acl-long.453/)Cited by:[§1](https://arxiv.org/html/2608.12756#S1.p2.1),[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px2.p1.1)\.
- Papernoet al\.\(2016\)D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. FernándezThe LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1525–1534\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1144),[Link](https://aclanthology.org/P16-1144/)Cited by:[§5\.2](https://arxiv.org/html/2608.12756#S5.SS2.p1.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.External Links:[Document](https://dx.doi.org/10.3115/1073083.1073135),[Link](https://aclanthology.org/P02-1040/)Cited by:[§4\.1](https://arxiv.org/html/2608.12756#S4.SS1.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§3\.4](https://arxiv.org/html/2608.12756#S3.SS4.p2.1)\.
- Qwen Team \(2024\)Qwen TeamQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[Table 3](https://arxiv.org/html/2608.12756#S4.T3),[Table 4](https://arxiv.org/html/2608.12756#S5.T4)\.
- Sennrichet al\.\(2016\)R\. Sennrich, B\. Haddow, and A\. BirchNeural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1715–1725\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1162),[Link](https://aclanthology.org/P16-1162/)Cited by:[§1](https://arxiv.org/html/2608.12756#S1.p1.1)\.
- Shaoet al\.\(2024\)N\. Shao, S\. Xiao, Z\. Liu, and P\. ZhangFlexibly scaling large language models contexts through extensible tokenization\.arXiv preprint arXiv:2401\.07793\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2401.07793),[Link](https://arxiv.org/abs/2401.07793)Cited by:[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px1.p1.1)\.
- Tayet al\.\(2022\)Y\. Tay, V\. Q\. Tran, S\. Ruder, J\. P\. Gupta, H\. W\. Chung, D\. Bahri, Z\. Qin, S\. Baumgartner, C\. Yu, and D\. MetzlerCharformer: fast character transformers via gradient\-based subword tokenization\.InThe Tenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=JtBRnrlOEFN)Cited by:[§1](https://arxiv.org/html/2608.12756#S1.p2.1),[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px4.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§6\.2](https://arxiv.org/html/2608.12756#S6.SS2.p3.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259),[Link](https://aclanthology.org/D18-1259/)Cited by:[§5\.2](https://arxiv.org/html/2608.12756#S5.SS2.p1.1)\.
- Yuet al\.\(2023\)L\. Yu, D\. Simig, C\. Flaherty, A\. Aghajanyan, L\. Zettlemoyer, and M\. LewisMEGABYTE: predicting million\-byte sequences with multiscale transformers\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/f8f78f8043f35890181a824e53a57134-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.12756#S2.SS0.SSS0.Px1.p1.1)\.
- Zhang and Sennrich \(2019\)B\. Zhang and R\. SennrichRoot mean square layer normalization\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/1e8a19426224ca89e83cef47f1e7f53b-Abstract.html)Cited by:[§5\.1](https://arxiv.org/html/2608.12756#S5.SS1.p2.1)\.
- Zhanget al\.\(2015\)X\. Zhang, J\. Zhao, and Y\. LeCunCharacter\-level convolutional networks for text classification\.InAdvances in Neural Information Processing Systems,Vol\.28,pp\. 649–657\.External Links:[Link](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html)Cited by:[§5\.2](https://arxiv.org/html/2608.12756#S5.SS2.p1.1)\.

## Appendix AImplementation and training details

#### Models and tokenizer\.

All variants share a Mamba2\-130M backward decoder \(24 layers\) and the GPT\-NeoX tokenizer\. We evaluate three encoders: a*projected SSM state*\(Mamba2\-130M, whose SSM state is compressed to a 4096\-dimensional code\), and two*hidden\-state*encoders whose final 1024\-dimensional hidden state is the code directly — Pythia\-410M \(Transformer\) and Mamba2\-370M\. Pythia shares the GPT\-NeoX tokenizer with the Mamba2 checkpoints, so encoder comparisons introduce no vocabulary change\.

#### Projections\.

The two families project differently\. For the hidden\-state encoders the code*is*the encoder’s final 1024\-dimensional hidden state, so there is no down\-projection; a single learned linear maps the code to annlayer×kc×knn\_\{\\text\{layer\}\}\\times k\_\{c\}\\times k\_\{n\}tensor \(kc=128k\_\{c\}\{=\}128channels,kn=8k\_\{n\}\{=\}8per layer\), which is expanded to each decoder layer’s full SSM initial state\(nheads,headdim,N\)=\(24,64,128\)\(n\_\{\\text\{heads\}\},\\text\{headdim\},N\)=\(24,64,128\)by parameter\-freerepeat\_interleavealong the channel \(×12\\times 12\) and state \(×16\\times 16\) axes; convolution states start cold\. For the projected\-SSM\-state encoder the code is compressed*from*the encoder’s per\-layer states by a non\-trivial map: each layer’s state \(D=1536D\{=\}1536channels×N=128\\times\\,N\{=\}128state\) is \(i\) channel\-compressed by a learned linearD→kcD\\\!\\to\\\!k\_\{c\}, \(ii\) mean\-pooled along the state dimensionN→knN\\\!\\to\\\!k\_\{n\}\(16×16\\times\), and \(iii\) averaged over layers within four groups\[\(0,12\),\(12,18\),\(18,21\),\(21,24\)\]\[\(0,12\),\(12,18\),\(18,21\),\(21,24\)\]; the four group codes are concatenated into the4​kc​kn=40964\\,k\_\{c\}k\_\{n\}=4096\-dimensional code\. The lossy, parameter\-free pooling is placed deliberately on the state and layer axes, while the information\-bearing channel dimension keeps a learned map\. The up\-projection mirrors these steps, and its weights are initialized as the pseudo\-inverse of the down\-projection so the round trip is near\-lossless before training\.

#### Why the widths differ\.

A hidden state is 1024\-dimensional, matched to SONAR’s width for the single\-code comparison in Section[4\.1](https://arxiv.org/html/2608.12756#S4.SS1)\. The full Mamba2\-130M SSM state is far larger — 24 layers of1536×1281536\\times 128values, about 4\.7M in total — and already sparse, so compressing it to 1024 would discard too much; we keep 4096\. Even at four times the width, the SSM\-state code reconstructs less well than the 1024\-dimensional Transformer code \(Figure[2](https://arxiv.org/html/2608.12756#A1.F2)\)\.

#### Encoder choice\.

We evaluated all three encoders on the single\-code protocol of Section[4\.1](https://arxiv.org/html/2608.12756#S4.SS1)before committing to the Transformer\. Figure[2](https://arxiv.org/html/2608.12756#A1.F2)plots ROUGE\-L against encode length for the three encoders and SONAR\. The Transformer hidden\-state encoder is strongest at the short lengths \(ℓ≤32\\ell\\leq 32\) that set almost every chunk boundary; the projected SSM state matches it only by spending the4×4\\timeswider 4096\-dimensional code, and the Mamba hidden state at equal width falls off fastest\. This is why the main paper uses the Transformer encoder throughout and reports the other two only here\.

![Refer to caption](https://arxiv.org/html/2608.12756v1/figures/reconstruction_by_length.png)Figure 2:Single\-code reconstruction ROUGE\-L versus encode length, on 500 held\-out Wikipedia windows \(protocol in Section[4\.1](https://arxiv.org/html/2608.12756#S4.SS1)\)\. Among the three encoders the Transformer hidden\-state variant is strongest at the short lengths that set chunk boundaries; the projected SSM state matches it only at4×4\\timesthe code width, and the Mamba hidden state is weakest\. SONAR is shown for reference; its drop below 16 tokens reflects training on full sentences rather than short fragments\.
#### Data and batching\.

Training text is FineWeb, tokenized into a flat queue and concatenated end\-to\-end with no separator, so a sampled window may cross a document boundary \(GPT\-style\)\. Each step samples one encode lengthLLfrom a mixture — 25% uniform on\[1,20\]\[1,20\]and the remainder log\-uniform on\[20,4096\]\[20,4096\]— and an independent decode lengthkklog\-uniform on\[1,4096\]\[1,4096\]clipped tok≤L\+1k\\leq L\+1\. All samples in a step share the sameLL, so each batch is a dense\(B,L\)\(B,L\)block that needs no padding;BBis set per step from a throughput target and a length\-aware memory cap\.

#### Two\-stage training\.

Stage 1 freezes the encoder and trains the backward decoder and projection for approximately 7B tokens\. Stage 2 initializes each encoder variant from that shared decoder, attaches a fresh projection, and jointly trains encoder, decoder, and projection for a further 3B tokens\.

#### Optimizer\.

AdamW\([14](https://arxiv.org/html/2608.12756#bib.bib26)\)with encoder/decoder/projector learning rates×10−55\\\!\\times\\\!10^\{\-5\},10−410^\{\-4\}, and10−310^\{\-3\}, weight decay 0\.1,β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), 1000 warmup steps, cosine decay to 10% of peak, and gradient clipping at 1\.0\. The random seed is 42; checkpoints record model step and total tokens processed\.

#### Compute\.

All runs use a single NVIDIA A100 80GB PCIe with gradient checkpointing\([4](https://arxiv.org/html/2608.12756#bib.bib28)\)enabled\. Stage 1 \(the shared 7B\-token decoder pretrain\) takes about 28 hours; stage 2 \(the 3B\-token joint training\) takes about 15 hours per encoder variant\.

## Appendix BReproducibility and compute accounting

The implementation uses Linux, Python 3\.10, CUDA 12\.8, PyTorch 2\.9\.1, and Transformers 4\.51\.3\. Random seeds are 42 for autoencoder training and data sampling, 0 for reader validation splitting and task adaptation, and 0 for the rate\-matched random\-boundary control\. Source datasets and third\-party model weights are obtained from the repositories identified in Appendix[G](https://arxiv.org/html/2608.12756#A7)\. Code and checkpoints are not publicly released\.

The downstream evaluations use 512 LAMBADA passages, 995 AG News articles, and the first 100 examples of the HotpotQA validation split\. The confidence markers in Table[5](https://arxiv.org/html/2608.12756#S5.T5)use these sample counts in the binomial standard errors stated in its caption; they do not measure variation across training runs\.

Table[6](https://arxiv.org/html/2608.12756#A2.T6)accounts for the result\-producing runs\. Reader\-corpus construction is listed separately because it is a material boundary\-selection cost rather than reader optimization\. The short task\-adaptation runs took 12 minutes for LAMBADA \(including 3\.5 minutes of chunking\), 5\.3 minutes for AG News optimization, and 2\.9 minutes for HotpotQA optimization plus about 8 minutes of cached encoding\. Other evaluation and plotting jobs range from minutes to roughly two hours each\. Including a conservative six GPU\-hour allowance for those jobs, the paper pipeline used approximately 94–102 A100 GPU\-hours\. Retained job logs indicate that preliminary, ablation, debugging, and failed runs kept the full research project below approximately 200 A100\-equivalent GPU\-hours\. CPU\-only dataset loading and reporting were negligible relative to GPU work; the exact host CPU model was not recorded\. Peak local storage was about 140GB, dominated by cached reader data and evaluation artifacts\.

Table 6:Compute for the principal paper pipeline on one NVIDIA A100 80GB PCIe\. Ranges reflect measured variation between reader variants; task adaptation is itemized in the text\.
## Appendix CNative autoencoder round trip

The direct reader operates on cumulative prefix codes𝒞≤i\\mathcal\{C\}\_\{\\leq i\}as defined in Section[5\.1](https://arxiv.org/html/2608.12756#S5.SS1)\. The reconstruction experiment of Section[4\.4](https://arxiv.org/html/2608.12756#S4.SS4)instead uses the same ReconSpan boundaries to test each selected chunk through the native autoencoder\. Given0=b0<b1<⋯<bm=n0=b\_\{0\}<b\_\{1\}<\\cdots<b\_\{m\}=n, encode every chunk in isolation,

ki=E\(xbi−1\+1:bi\),𝒦=\(k1,…,km\)\.k\_\{i\}=E\(x\_\{b\_\{i\-1\}\+1:b\_\{i\}\}\),\\qquad\\mathcal\{K\}=\(k\_\{1\},\\ldots,k\_\{m\}\)\.\(7\)LetD∗​\(ki\)D^\{\*\}\(k\_\{i\}\)denote the tokens produced autoregressively by the backward decoder before EOS\. Because the decoder generates newest\-first, the reconstructed chunk and document are

x^\(i\)=reverse⁡\(D∗​\(ki\)\),x^=x^\(1\)​‖⋯‖​x^\(m\),\\hat\{x\}^\{\(i\)\}=\\operatorname\{reverse\}\(D^\{\*\}\(k\_\{i\}\)\),\\qquad\\hat\{x\}=\\hat\{x\}^\{\(1\)\}\\\|\\cdots\\\|\\hat\{x\}^\{\(m\)\},\(8\)where∥\\\|denotes concatenation\. Boundary selection is teacher\-forced, but this round\-trip decode conditions on its own previous outputs\. Re\-encoding isolated chunks is used only for this native\-decoding diagnostic and for decoded\-text controls; the direct reader path is simply𝒞≤i↦y\\mathcal\{C\}\_\{\\leq i\}\\mapsto yand does not perform this extra work\.

## Appendix DParallel boundary selection

Letrt≤Wr\_\{t\}\\leq Wbe the backward reach obtained from the code ending at positiontt, whereWWis the decode block size of Section[3\.2](https://arxiv.org/html/2608.12756#S3.SS2)\. Greedy ReconSpan retains the current endpoint, moves tot−rtt\-r\_\{t\}, and repeats\. Its work is linear for fixedWW, but the endpoint chain is serial and can containnnone\-token chunks in the worst case\.

The dynamic\-programming variant evaluates theWWreverse positions for every endpoint in parallel, producing all reachesr1,…,rnr\_\{1\},\\ldots,r\_\{n\}\. It then solves a minimum\-cover recurrence over candidate intervals\[t−rt\+1,t\]\[t\-r\_\{t\}\+1,t\]\. Grouping transitions by reverse offset requiresWWparallel rounds after the prefix codes are available\. The total work remainsO⁡\(W​n\)O\(Wn\)and the sequential decode calls becomeO⁡\(1\+W\)O\(1\+W\), but allnnendpoint decodes are materialized rather than only the greedy boundary chain\. This is faster in latency but uses more physical compute and memory\.

## Appendix EReader diagnostics and code\-to\-code reader details

The code\-to\-code reader predicts the next*chunk code*from the latent\-token prefix, following the Large Concept Model formulation\([10](https://arxiv.org/html/2608.12756#bib.bib12)\)\. Using the boundariesb1<⋯<bmb\_\{1\}<\\cdots<b\_\{m\}found by ReconSpan, define the prefix codepi=E\(x1:bi\)p\_\{i\}=E\(x\_\{1:b\_\{i\}\}\)and the independently re\-encoded chunk codeki=E\(xbi−1\+1:bi\)k\_\{i\}=E\(x\_\{b\_\{i\-1\}\+1:b\_\{i\}\}\)\. The model is trained on sequences of these codes to predictki\+1k\_\{i\+1\}given prefix information throughpip\_\{i\}\. Generation is a loop through the native decoder: predict a chunk code, decode it to text, re\-encode the generated text to update the prefix representation, and predict the next code\. The reader therefore depends on the code\-to\-text bridge as well as on its own predictions\.

Two variants were trained\. A continuous diffusion head, which should model a multimodal next\-code distribution, collapsed toward the conditional mean and produced codes that did not decode to usable text\. Quant\-LCM replaces the continuous target with residual vector quantization — the code is mapped to per\-level codebook indices and predicted by per\-level softmax classification — which trains stably and is the version reported here\. It uses an eight\-layer, 1024\-wide causal context tower over the chunk sequence\. Its conditional perplexity is 290\.43 on the protocol of Table[4](https://arxiv.org/html/2608.12756#S5.T4), roughly 25 times the code\-to\-token reader’s, and its generations are locally broken and repetitive\. We therefore report it as a negative result and use the code\-to\-token reader in the main body\. The failure is a prediction failure rather than a representation failure: the same codes support fluent generation when the model is asked for tokens instead of the next code\.

For reference, the Pythia code\-to\-token reader fine\-tunes all 406M base\-model parameters and a 1024\-to\-1024 adapter\. Code positions are standardized and RMS\-normalized; loss is applied only to continuation tokens\.

## Appendix FBoundary observations

Figure[3](https://arxiv.org/html/2608.12756#A6.F3)shows how per\-position reconstruction reach induces unequal chunks in one document\. We decode from every prefix code and record how many consecutive tokens are reconstructed before the first error\. The gradual increases partly follow mechanically from advancing the endpoint; the abrupt drops show positions from which the decoder fails much earlier, forcing the greedy cover to allocate a new code\. Whether such drops align systematically with linguistic structure remains an open question\.

![Refer to caption](https://arxiv.org/html/2608.12756v1/figures/boundary_example_short.png)Figure 3:Per\-position backward decode reach for one 63\-token passage under*Failure\(1\)*\. Red dashed lines mark the eight interior boundaries of the nine chunks selected by ReconSpan, and the gray dotted line marks the 32\-token decoding window\. Reach changes sharply across positions, producing unequal chunk lengths and concentrating boundaries near positions where reconstruction fails early\. This example motivates studying whether the boundaries align with syntax, discourse, or information density; it does not establish such alignment\.The corresponding text is shown below, with vertical bars at the exact token boundaries used in Figure[3](https://arxiv.org/html/2608.12756#A6.F3)\.

> Water boils at 100 degrees Celsius, and Genghis Khan founded the Mongol∣\\midEmpire in 1206\. The clarinet has a single\-reed∣\\midmouthpiece, while photos∣\\midynthesis∣\\midconverts sunlight into glucose∣\\mid\. Napoleon was exiled to Elba in 1814∣\\mid, but the liver performs∣\\midover 500∣\\midmetabolic functions\.

The within\-word and punctuation\-adjacent splits make clear that this example does not by itself imply that the learned boundaries coincide with linguistic units\.

## Appendix GExisting assets, licenses, and terms

Tables[7](https://arxiv.org/html/2608.12756#A7.T7)and[8](https://arxiv.org/html/2608.12756#A7.T8)report the repository identifiers and license information verified from the official cards and repositories in August 2026\. The experiment records pin the cached revisions used\. Assets with non\-commercial or unclear terms were used only for academic research and evaluation and are not redistributed\.

Table 7:Datasets used in the reported experiments\. “Not specified” means that the official repository does not publish a named license; it is not an inferred license\.Table 8:External model and baseline assets\. License restrictions apply to the original assets; no third\-party weights are redistributed with this preprint\.#### Language\-model use\.

Language models are core experimental components: Pythia and Mamba form the autoencoder, Pythia and Llama are readers, and Qwen is an external perplexity scorer\. Their roles and training are described in Sections[3](https://arxiv.org/html/2608.12756#S3)and[5](https://arxiv.org/html/2608.12756#S5)\. An LLM was also used for writing, editing, and formatting assistance; it did not generate measurements or determine the experimental conclusions\.