Content is What Remains: Invariant Speech Tokenization from Parallel Utterances

# Content is What Remains: Invariant Speech Tokenization from Parallel Utterances
Source: [https://arxiv.org/html/2607.19033](https://arxiv.org/html/2607.19033)
Wagner Thallinger Stankovic Zusag

###### Abstract

Discrete speech tokenizers aim to disentangle semantic from acoustic information, yet targets from self\-supervised learning \(SSL\) models like HuBERT retain non\-linguistic variation: speaker identity, prosody, and channel conditions leak into tokens, inflating entropy\. Our insight: when enough speakers utter the same words under varying conditions, linguistic content is the only shared factor\. We proposePINT\(Parallel INvariant Tokenization\), fine\-tuning an SSL encoder with alignment losses across parallel utterances and augmentations to distill this shared residual\.PINTcollapses identical words onto consistent token sequences, drastically reducing conditional entropy\. Unlike ASR text,PINTtokens preserve frame\-level temporal grounding and serve as drop\-in semantic targets for audio codecs\. Experiments show a98\.7%98\.7\\%relative reduction in speaker probe accuracy \(93\.1%→1\.2%93\.1\\%\\\!\\to\\\!1\.2\\%\),42%42\\%lower ABX error rate, and2727–30%30\\%lower LM perplexity versus baselines, confirming that the right invariance is key to efficient learning\.111Models:[https://github\.com/nyrahealth/PINT](https://github.com/nyrahealth/PINT)

###### keywords:

speech coding, speech tokenization, discrete representations, speech compression, robust content representations

## 1Introduction

Subword tokenization in NLP succeeds partly because discretization that respects a signal’s latent structure yields sequences that are highly compressible and predictable for autoregressive models\. Discrete speech tokenization extends this premise to audio, supporting LM\-based speech generation, dialogue, and cross\-modal reasoning\[borsos2023audiolm,zhang2023speechgpt,defossez2024moshi,wang2023viola\]\. In contemporary speech\-codecs and speech generation systems, tokens often appear in a factorized representation \(per timestep/frame\):*semantic*tokens should capture linguistic content, while*acoustic*residual layers model speaker, prosody, and other signal details\[zhang2024speechtokenizer,wu2023audiodec,defossez2024moshi\]\.

The nuisance\-leakage problem\.Semantic tokens in tokenizers commonly discretize intermediate SSL features, typically HuBERT\[hsu2021hubert\]or WavLM\[chen2022wavlm\], via distillation\[zhang2024speechtokenizer,defossez2024moshi\]into codebooks or dual encoders where one encodes semantics and the other fills in the gaps\[xy\-tokenizer\]\. However, these encoders leak non\-linguistic variation: speaker identity remains recoverable\[qian2022contentvec,yeh2024completeness\], and token sequences can change by\>40%\>\{\}40\\%under acoustic perturbations alone\[gat2023augmentation\], a brittleness that has been found to correlate with downstream degradation\[vashishth2024stab,chang2025dcspin,zhong2025accent\]\. Formally, letccdenote linguistic content and𝐳\\mathbf\{z\}the token sequence\. When different realizations of the sameccyield different𝐳\\mathbf\{z\}, the conditional entropyH​\(𝐳∣c\)H\(\\mathbf\{z\}\\mid c\)is bounded away from zero by non\-linguistic variation\. This leakage hinders both sequence compressibility and applications that benefit from disentangled content and style representations such as accent conversion\[zhong2025accent\], voice conversion\[qian2022contentvec\], and expressive synthesis\[zhang2025vevo\], where downstream models are forced to undo entanglement that a cleaner representation would have prevented\. A cleanly disentangled representation would instead allow models to manipulate style independently of content\.

Related work\.ContentVec\[qian2022contentvec\]and DC\-Spin\[chang2025dcspin\]pursue speaker invariance via contrastive or clustering objectives; Gat et al\.\[gat2023augmentation\]and NAST\[messica2024nast\]target augmentation robustness; StableToken\[song2025stabletoken\]achieves noise invariance through multi\-branch voting but does not address speaker or prosodic variation\. On the codec side, SpeechTokenizer\[zhang2024speechtokenizer\], Moshi/Mimi\[defossez2024moshi\], DualCodec\[dualcodec\], XY\-Tokenizer\[xy\], and SAC\[sac\]each improve semantic–acoustic disentanglement through distillation, dual encoders, or split\-stream quantization\. Yet all inherit features \(HuBERT, WavLM, Whisper\[whisper\]or w2v\-BERT\[w2vBert\]\) as semantic targets without addressing the nuisance leakage already present in those representations\. PINT is orthogonal: rather than redesigning codec architectures, it cleans the encoder upstream, producing invariant representations that serve as drop\-in targets for any of these frameworks\.

PINT: formulation and approach\.We argue that a token is only truly semantic if it satisfies a dual criterion:*content capture*: it carries sufficient phonetic information and*nuisance invariance:*it discards everything else\. nuisance invariance drivesH​\(𝐳∣c\)→0H\(\\mathbf\{z\}\\mid c\)\\\!\\to\\\!0, ensuring identical content maps to identical token sequences\. This allows run\-length encoding \(RLE\) and byte pair encoding \(BPE\) to collapse redundant frames, drastically improving bit rate efficiency and predictability for autoregressive modeling, since the remaining entropy reflects genuine linguistic structure discarding other factors\.

We introduce PINT to operationalize this through parallel data: when multiple speakers utter the same sentence under different conditions, linguistic content is the only shared factor\. By fine\-tuning a HuBERT encoder with parallel\-data alignment losses and aggressive augmentations, PINT distills this shared residual, leaving content as the only consistent signal\. Unlike ASR\-derived text tokens, PINT preserves frame\-level temporal grounding facilitating integration into modern audio codec architectures\.

The payoff is measurable and model\-agnostic: an identical 85 M\-parameter decoder\-only transformer achieves27–30 % lower perplexityon PINT tokens than on HuBERT or WavLM tokens, reaching WavLM's final perplexity in 23×\\timesfewer steps \- evidence that invariance, not architecture, is the binding constraint on token quality\.

Contributions\.

\(1\) We formalize semantic speech tokenization through a dual criterion of content capture and nuisance invariance, showing that nuisance leakage directly inflates conditional entropy and hurts compressibility\. \(2\) We propose PINT, a training framework leveraging parallel data and augmentation to jointly optimize continuous and discrete representations, satisfying both criteria simultaneously\. \(3\) We show that PINT yields highly compressible content tokens, reducing speaker probe accuracy by 98\.7 % \(93\.1 %→\\to1\.2 %\), improving ABX discriminability by 42 %, and lowering LM perplexity by 27–30 % vs\. HuBERT/WavLM \(test perplexity 1\.95 vs\. 2\.78/2\.67\)\.

## 2Method

### 2\.1Data and Parallel Supervision

Training and evaluation draw on four categories of English speech, summarized in Table[1](https://arxiv.org/html/2607.19033#S2.T1)\.True\-parallel corpora\(ARCTIC, CHAINS, CSTR\-VCTK, EnDialects, ESD, TIMIT\) provide multiple recordings of the same transcripts across speakers, accents, speaking styles, and emotional conditions, yielding naturally aligned utterance groups that span a wide range of variability\.Non\-parallel corpora\(LibriSpeech 960 h, Tedlium\-3 374 h\) extend linguistic and acoustic coverage; since they lack shared transcripts, pseudo\-parallel pairs are formed on\-the\-fly via stochastic augmentations like additive noise \(white, brown, pink\), reverberation, speed/pitch perturbation, and channel distortion applied to the same utterance within a batch\. Further we synthesize multiple parallel samples for these datasets using Kokoro\[kokoro\]Evaluation\-only: RAVDESS\[ravdess\]\(emotion probes\) and 6k hours clean subset of Librilight for LM training\.Noise: We treat noise\[noise\]as parallel data with empty transcripts during training\. All corpora are force\-aligned with the Montreal Forced Aligner \(MFA\)\[mfa\]for word\-level timestamps\. Evaluations use data strictly excluded from training: CSTR\-VCTK holds out 10 % of speakers; TIMIT and LibriSpeech use standard non\-overlapping partitions; for corpora without official splits, all utterances sharing a normalized transcript are assigned to the same split to prevent transcript leakage\.222The Use column in Table[1](https://arxiv.org/html/2607.19033#S2.T1)codes each corpus’s role: T=Stage A training, KM=kk\-means fitting, I=invariance evaluation, LM=language model\.

Table 1:Dataset statistics and parallel properties\. Hours: total audio length used; \#S: parallel samples; \#G: parallel groups\. Use codes: T=Stage A; KM=kk\-means; I=invariance evaluation; LM=language model\.DatasetHours\#S\#GParallel propertiesUseARCTIC\[arctic\]13\.71515k1134accents/ dialectsTCHAINS\[chains\]9\.666k56accents, speed, styleTCSTR\-VCTK\[cstr\]82\.58888k13615accents/ dialectsT IEnDialects\[english\-dialects\]30\.41717k2273accents/ dialectsTESD \(EN\)\[esd\]13\.31717k383emotionsTRAVDESS\[ravdess\]1\.47202emotionsISynSpeech\[synspeech\]196\.6109109k109TTIMIT\[timit\]3\.74\.44\.4k452T KMNoise\[noise\]30\.01010k1T ILibriSpeech\[librispeech\]120\.47373k——T ITedlium\-3\[tedlium3\]374\.8244244k——TLibriLight\[librilight\]6000——LM
### 2\.2Model Overview

PINT starts from HuBERT\-base\[hsu2021hubert\]\. Given a waveform𝐱\\mathbf\{x\}, the encoder produces continuous frame\-level features that serve as input to a decoder branch predicting phoneme sequences via cross\-entropy \(CE\)\. In Stage B, an additional id\-sequence head outputs logits overK=200K\{=\}200discrete codes plus a blank symbol for connectionist temporal classification \(CTC\) training\[graves2006ctc\]\.

### 2\.3Stage A: Parallel Invariance Training

Mini\-batches contain utterances indexed byb∈\{1,…,B\}b\\in\\\{1,\\dots,B\\\}, wheregbg\_\{b\}denotes the transcript\-group index\.

\(1\) Sequence\-level soft dynamic time warping \(DTW\) loss\.Let𝐇b=\[𝐡b,1,…,𝐡b,Tb\]∈ℝTb×D\\mathbf\{H\}\_\{b\}=\[\\mathbf\{h\}\_\{b,1\},\\dots,\\mathbf\{h\}\_\{b,T\_\{b\}\}\]\\in\\mathbb\{R\}^\{T\_\{b\}\\times D\}denote encoder frame features for utterancebb, whereTbT\_\{b\}is the number of frames andDDis the feature dimension\. For each positive parallel pair\(i,j\)\(i,j\)withgi=gjg\_\{i\}\{=\}g\_\{j\}, let𝒫\\mathcal\{P\}be the set of all such pairs in the mini\-batch\. Soft\-DTW\[cuturi2017softdtw\]computes a distancedi​jsdtwd^\{\\mathrm\{sdtw\}\}\_\{ij\}between𝐇i\\mathbf\{H\}\_\{i\}and𝐇j\\mathbf\{H\}\_\{j\}, normalized by the maximum sequence length:

ℒsdtw=1\|𝒫\|​∑\(i,j\)∈𝒫di​jsdtwmax⁡\(Ti,Tj\)\.\\mathcal\{L\}\_\{\\mathrm\{sdtw\}\}=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\(i,j\)\\in\\mathcal\{P\}\}\\frac\{d^\{\\mathrm\{sdtw\}\}\_\{ij\}\}\{\\max\(T\_\{i\},T\_\{j\}\)\}\.\(1\)Since parallel pairs share only linguistic content while differing in speaker, style, recording conditions etc\. minimizing this loss forces the encoder to output consistent representations for content and suppress nuisance variation\.

\(2\) Word\-level contrastive loss\.Using word boundaries from forced alignment, we compute word representations by averaging encoder frames over each word span:𝐯b,m=1\|𝒯b,m\|​∑t∈𝒯b,m𝐡b,t\\mathbf\{v\}\_\{b,m\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{b,m\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{b,m\}\}\\mathbf\{h\}\_\{b,t\}\. Positive pairs𝒱\+\\mathcal\{V\}^\{\+\}consist of the same word position across parallel utterances; negative pairs𝒱−\\mathcal\{V\}^\{\-\}consist of words whose phoneme sets have Jaccard similarity below a thresholdτneg\\tau\_\{\\mathrm\{neg\}\}\.333Jaccard\-based gating avoids contradictory gradients from phonetically similar negatives\.The loss attracts same\-word representations and repels different ones:

ℒword=1−cos¯​\(𝒱\+\)\+λneg​cos¯​\(𝒱−\),\\mathcal\{L\}\_\{\\mathrm\{word\}\}=1\-\\overline\{\\cos\}\(\\mathcal\{V\}^\{\+\}\)\\;\+\\;\\lambda\_\{\\mathrm\{neg\}\}\\,\\overline\{\\cos\}\(\\mathcal\{V\}^\{\-\}\),\(2\)wherecos¯​\(⋅\)\\overline\{\\cos\}\(\\cdot\)is the duration\-weighted mean cosine similarity\. This attraction\-repulsion prevents the trivial collapse that the sDTW loss alone would suffer from and further encourages representations that are aligned on the word level\.

\(3\) Decoder CE loss\.A two\-layer Transformer decoder trained with teacher\-forced CE over phoneme targets from transcript phonemization, attending to the encoder output via cross\-attention ensures*content capture*\. The Stage A objective combines all three losses \(allλ∗\\lambda\_\{\*\}are scalar weights\):

ℒA=λsdtw​ℒsdtw\+λword​ℒword\+λce​ℒce\.\\mathcal\{L\}\_\{\\mathrm\{A\}\}=\\lambda\_\{\\mathrm\{sdtw\}\}\\mathcal\{L\}\_\{\\mathrm\{sdtw\}\}\+\\lambda\_\{\\mathrm\{word\}\}\\mathcal\{L\}\_\{\\mathrm\{word\}\}\+\\lambda\_\{\\mathrm\{ce\}\}\\mathcal\{L\}\_\{\\mathrm\{ce\}\}\.\(3\)
We setλsdtw=\.5\\lambda\_\{\\mathrm\{sdtw\}\}=\.5,λword=2\\lambda\_\{\\mathrm\{word\}\}=2,λneg=1\\lambda\_\{\\mathrm\{neg\}\}=1, andλce=10\\lambda\_\{\\mathrm\{ce\}\}=10\.

### 2\.4Stage B: Discrete Token Training

Stage A captures content with invariant continuous embeddings, yet converting them to tokens via a post\-hockk\-means codebook does not guarantee sequence\-level optimality since the clustering objective is agnostic to sequential consistency across parallel utterances\. Stage B addresses this via end\-to\-end optimization that directly targets discrete sequence consistency while jointly fine\-tuning all components, enforcing invariance in both continuous and discrete form: Following Lee et al\.\[lee2022textless\], we attach a linear layer that maps encoder frames to logits over a discrete unit vocabulary\. For each transcript group, the teacher \(an EMA copy of the student\) processes an anchor utterance \(synthetic for synth\-aug\), obtains token IDs by framewise argmax decoding of these logits, and then applies deduplication to form a discrete target sequence\. This yields a common reference sequence per group\. The student is trained with CTC to align every utterance in the group to this shared target\. Because CTC permits many\-to\-one alignments, variation in duration and speaking rate is absorbed without penalty, pushing all speakers toward the same discrete output\. Unlike Lee et al\., who freeze the encoder, we train end\-to\-end via a student\-teacher framework\[tarvainen2017meanteacher,data2vec\]\. Joint updates allow the semantic encoder to co\-adapt with the discrete partition; \(blanks are disallowed, as deduplication makes them unnecessary\)\.ℒmarg\\mathcal\{L\}\_\{\\mathrm\{marg\}\}promotes a uniform argmax distribution on the batch level to avoid collapse, whileℒorth\\mathcal\{L\}\_\{\\mathrm\{orth\}\}discourages correlations among its prototype vectors to improve code separability\. We downweight the discrete losses relative to the Stage A terms \(λid=0\.6\\lambda\_\{\\mathrm\{id\}\}\{=\}0\.6,λm,o=0\.3\\lambda\_\{\\mathrm\{m\},\\mathrm\{o\}\}\{=\}0\.3\) so that continuous invariance training remains the dominant learning signal\.

## 3Experiments

We evaluate PINT along three axes that directly test the dual criterion: content capture \(§[3\.1](https://arxiv.org/html/2607.19033#S3.SS1)\) and invariance and robustness \(§[3\.2](https://arxiv.org/html/2607.19033#S3.SS2)\)\. Further we evaluate the downstream compression benefit unlocked by the above\. Baselines are HuBERT\-base layer 9 and WavLM\-base layer 12 since these layers have been found to correspond most closly to phonemes\[hsu2021hubert,chen2022wavlm\]\. Unless noted, discrete tokens usekk\-means \(k=200k\{=\}200\) fit on TIMIT for baselines\. PINT remains strong at far smaller vocabularies, while the baselines degrade more quickly and saturate around 200; we therefore use K=200 for all systems for direct comparability\. Ablations:*dec\-AR*uses exclusively auto\-regressive phoneme decoding as loss;*dec\-CTC*uses exclusively a CTC loss for training;*synth\-only*uses only synthetic data from Kokoro\[synspeech\]and all losses;*synth\-aug*uses all losses and real data, but all parallel data for the real data is synthetically generated meaning real samples only interact with synthetic ones in DTW and Word level losses but never with other real ones;*w/o noise*drops noise augmentation and noise as parallel data\.

### 3\.1Content Capture

We measure content capture via ASR performance, phoneme discriminability \(ABX\), and phone\-normalised mutual information \(PNMI\)\.Word error rate \(WER\) / character error rate \(CER\): following\[zhang2024speechtokenizer\]we train a 2\-layer bidirectional long short\-term memory \(BLSTM\) network \(hidden dimension 1024\) with CTC on LibriSpeech train\-clean\-360, evaluated on test\-clean; in continuous mode the BLSTM receives frame features directly, in discrete mode a k means model maps to ids or in PINT's case the argmax logit output of the last linear layer\.ABX\[schatz2013abx\]: minimal\-pair phonetic discriminability on LibriSpeech dev\-clean via thezrc\_abx2toolkit \(20 ms frame rate\); we report within\-speaker and across\-speaker error \(lower is better\)\.PNMIas defined in\[hsu2021hubert\]\. Results are shown in Table[2](https://arxiv.org/html/2607.19033#S3.T2)\.

Table 2:Content capture on LibriSpeech\. Each cell showscts / dis: continuous vs\.kk\-means discrete \(k=200k\{=\}200\)\.↓\\downarrowlower is better;↑\\uparrowhigher is better\.Bold: best per sub\-column\.cts / disABX↓\\downarrowModelCERWERPNMI↑\\uparrowwit\.acr\.BaselinesHuBERT4\.33 / 7\.5510\.99 / 21\.370\.790\.0550\.066WavLM4\.03 / 6\.4011\.53 / 18\.420\.810\.0470\.059PINT\(ours\)3\.84 / 4\.659\.79 / 12\.130\.780\.0400\.042Ablationsdec\-AR3\.90 / 6\.0010\.08 / 17\.930\.780\.0760\.086synth\-only5\.87 / 7\.3714\.95 / 18\.550\.750\.0550\.062w/o noise3\.80 / 4\.759\.71 / 12\.520\.780\.0430\.050dec\-CTC4\.20 / 4\.73/ 10\.77 / 12\.010\.770\.0730\.086synth\-aug3\.79 / 4\.679\.91 / 12\.410\.780\.0420\.044PINT outperforms both baselines on CER and WER in continuous and discrete mode while achieving substantially better ABX discriminability \(0\.040 vs\. 0\.042 across\-speaker\), demonstrating that our invariance training does not sacrifice content\. Among the ablations,*synth\-only*degrades both ASR and ABX, showing that real data is essential for real world performance;*dec\-CTC*and*dec\-AR*underperform on ABX, confirming that the full multi\-loss training is beneficial for phoneme discriminability\. Most importantly*synth\-aug*performs very well\. Since the required parallelism can be generated synthetically, extending PINT to languages without parallel human corpora appears feasible\.

### 3\.2Invariance and Robustness

Table 3:Invariance and noise robustness\. Probes: lower↓\\downarrow= more invariant\. Invar\.: lower↓\\downarrow= more consistent\. Noise: concentration higher↑\\uparrow,Noise: normalised token\-distribution entropy / ids used \(Ent/Ids, lower↓\\downarrowis better\) RMS SD lower↓\\downarrow\.Bold: best per column\.Probes \(%\)↓\\downarrowInvar\.↓\\downarrowNoiseModelSpkEmoDTWEditEnt/Ids↓\\downarrowRMS SD↓\\downarrowBaselinesHuBERT93\.155\.40\.16090\.22750\.499/1390\.9051WavLM78\.954\.60\.12180\.20960\.642/1911\.0010PINT \(ours\)1\.232\.10\.00920\.06590\.000/10\.0049Ablationsdec\-AR20\.144\.20\.02690\.16540\.470/1530\.7092synth\-only4\.132\.90\.00890\.24850\.488/1750\.6055w/o noise1\.227\.50\.03700\.14120\.566/1980\.7527dec\-CTC43\.043\.10\.04850\.31760\.329/650\.4989synth\-aug2\.332\.30\.00750\.08690\.000/10\.0053We evaluate invariance through probes, parallel\-utterance consistency, and noise robustness\.Speaker/emotion probes: x\-vector time\-delay neural network \(TDNN\) classifiers\[snyder2018xvector\]on frozen representations; speaker identity on CSTR\-VCTK held\-out speakers, emotion\+intensity on RAVDESS; lower accuracy indicates greater invariance\.Parallel invariance\(CSTR\-VCTK test set\): DTW cosine ratio \(mean intra\-group DTW\-aligned cosine distance normalized by inter\-group distance\) and average normalized edit distance between deduplicated token sequences of parallel utterances normalized by the longer sequences length\.Noise robustness:*Entropy*for discrete: Entropy of the distribution of discrete id's when feeding 2 noise classes\[noise\]not seen in training through the encoders, this is paired with the number of unique ids used atleast once; For continuous:*RMS standard deviation ratio \(RMS SD\)*root\-mean\-square of per\-dimension standard deviations for noise vs\. speech embeddings \(near zero = negligible noise variation\)\. Results are shown in Table[3](https://arxiv.org/html/2607.19033#S3.T3)\.

PINT reduces speaker probe accuracy from 93 % \(HuBERT\) to 1\.2 %, demonstrating near\-complete speaker invariance\. Emotion \(7 emotions \+ neutral\) accuracy also drops substantially \(55 %→\\to32–33 %\), though some signal persists\. Parallel\-corpus DTW distances confirm that continuous representations collapse intra/inter group distance by∼20×\{\\sim\}20\\timesrelative to HuBERT with a much lower sequence variablity also at the discrete level\. Noise robustness is essentially perfect for PINT \(entropy=0=0, RMS SD<0\.02<0\.02\), whereas baselines exhibit substantial drift\. The ablations reveal that each component and the data contribute distinctly\. Using sDTW without word level loss leads to collapse\. The*dec\-AR*variant retains 20\.1 % speaker accuracy which is∼16\.8×\{\\sim\}16\.8\\timesworse than full PINT\.*synth\-only*achieves low speaker leakage but high edit distance \(0\.25 vs\. 0\.06\), indicating that while synthetic variation removes speaker signal, atleast some real data is needed for discrete sequence consistency\.*w/o noise*achieves the lowest emotion leakage but poor noise concentration and high DTW distance, confirming that noise augmentation is important for robustness without sacrificing invariance\.

### 3\.3Compression Efficiency

When identical content consistently maps to identical \(similar\) tokens, sequences contain clear patterns that compression algorithms can exploit\. We test this prediction with progressive compression schemes \(Table[4](https://arxiv.org/html/2607.19033#S3.T4)\), reporting tokens per second and bits per second \(⌈log2V⌉×\\lceil\\log\_\{2\}V\\rceil\\timestok/s\)\. TheTxcolumn applies the same formula to characters and character level BPE on transcripts, providing a text\-grounded compressibility reference\.

Table 4:Encoding compressibility:tok/s / b/sper cell \(lower↓\\downarrow= more compact\)\.Hu: HuBERT\-L9;WL: WavLM\-L12;PI: PINT\-L12\.Tx: character\-level BPE on transcripts\.tok/s / b/sEnc\.𝑽\\bm\{V\}HuWLPITxRaw50/40050/40050/40014\.6/116Dedup24\.6/19724\.8/19812\.6/101—RLE24\.6/24624\.8/27312\.6/152—BPE\-orig4k15\.0/18015\.3/18310\.0/1204\.1/50BPE\-orig8k13\.2/17213\.4/1758\.8/1143\.7/48BPE\-dedup4k9\.3/1129\.4/1134\.9/59—BPE\-dedup8k8\.2/1078\.3/1084\.3/56—PINT RLE achieves152,bits/s—a2\.6×2\.6\\timesreduction from 400,bits/s—just 31,% above raw text \(116,bits/s\), while retaining frame\-level time alignment\. HuBERT and WavLM reach only 246–273,bits/s \(∼1\.6×\{\\sim\}1\.6\\timescompression\), as speaker and channel variability disrupt token runs with surface\-level changes rather than true content shifts\.

The deduplication rate \(74\.8 % for PINT vs\. 50\.7 % for HuBERT\) is the most direct fingerprint of nuisance invariance: it teases out precisely the consistent structure that both RLE and BPE can exploit\. After deduplication and BPE atV=8,192V\{=\}8\{,\}192, PINT reaches56 bits/sapproaching text BPE \(48 bits/s\) while baselines stall at 107\-108 bits/s\. BPE merges tokennn\-grams that recur consistently across the corpus; because PINT minimizes dependence on speaker and acoustic conditions, the same content more reliably maps to the same token sequence, and BPE discovers stable patterns just as it finds recurring character sequences in text\. In HuBERT and WavLM, the same content uttered under different conditions produces different token sequences, fragmenting these patterns and blocking the merges\.

Autoregressive LM perplexity\.The ultimate test of token quality is whether an autoregressive model finds them easy to predict\. We train identical 85 M\-parameter decoder\-only Transformers \(12 layers, hidden dimension 1024, 16 heads, context 3072 tokens, weight\-tied embeddings\) on raw token id streams from each tokenizer using LibriLight\. Figure[1](https://arxiv.org/html/2607.19033#S3.F1)shows the training curves\. PINT achieves test perplexity1\.95, versus 2\.78 for HuBERT and 2\.67 for WavLM, a27–30 %reduction under identical architecture and data\. Strikingly, PINT already matches WavLM's perplexity after only∼1,400\{\\sim\}1\{,\}400steps, roughly23×\\timesfewer iterationsand continues to improve far below the baseline ceiling\. PINTs supperior token consistency directly reduces the burden that nuisance leakage forces on the LM\.

![Refer to caption](https://arxiv.org/html/2607.19033v1/x1.png)Figure 1:Training perplexity on LibriLight for an identical 85 M\-parameter transformer LM on HuBERT L9, WavLM L12, and PINT L12 \(k=200k\{=\}200\) tokens\. Dashed lines: final test perplexity\. PINT converges to 27–30 % lower perplexity, consistent with its superior sequence compressibility \(Table[4](https://arxiv.org/html/2607.19033#S3.T4)\)\.

## 4Conclusion

PINT shows that enforcing invariance yields audio content compressibility close to text, providing a cleaner interface to NLP techniques\. By treating parallel utterances as natural supervision and jointly satisfying invariance and discriminability, the resulting tokens are speech\-grounded and speaker\-agnostic, with a 27–30 % perplexity reduction confirming that nuisance leakage bottlenecks learning efficiency\. As invariant, frame\-aligned representations, PINT tokens serve as drop\-in targets for codec architectures and disentanglement\-sensitive tasks\. Future work includes multilingual extension via synthetic data, codec integration, and duration\-factorized generation schemes that exploit PINT's RLE compressibility\.

## 5Generative AI Use Disclosure

Some code used in the experiments was written with help from a coding assistant \(Claude by Anthropic\)\. The experiments were run manually and results were manually verified\. Generative AI was also used in the formatting of tables and plots\. The paper was manually written\. The authors assume full responsibility and accountability for the content of this submission\.

## References