Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
Summary
This paper introduces hierarchical rotary positional encoding (hRoPE) to represent text hierarchy and uses compression depth as a signature of paragraph structure, showing it correlates with genuine hierarchical organization better than other metrics.
View Cached Full Text
Cached at: 09/28/26, 12:04 PM
Paper page - Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
Source: https://huggingface.co/papers/2609.23551
Abstract
Standardpositionalencodingsrepresentpositionasaone-dimensionalreading-ordercoordinate,butreadingorderalonedoesnotdeterminehierarchicaltextualstructure.Weuseahierarchicalrotarypositionalencoding(hRoPE)thatrepresentsparagraph,sentence,andtokenindicesasseparatechannels,holdthetokensequencefixed,interveneontheparagraphcoordinatep1,andmeasurecross-paragraphattentionwithatoken-distance-exactestimator.Attentioniscompressedrelativetoatoken-distance-matchedbaselineineverycorpus,butcompressionaloneisnotdiagnosticoftruestructure:anarchitecturallyidenticalchannelwithdensity-matchedrandomlabelsiscompressedtoo,moreshallowly.Whatdistinguishesrealstructureisthedepthofcompression,whichisgreaterandcorpus-dependentwhilethecontrol’sisnot.Comparingeightcorpus-onlyquantitiesacrossthreeconstructs(lexicalpersistence,paragraphlength,embedding-basedcoherence),nonefullyreproducesthecross-corpusorderingofdepth,thoughembedding-basedcoherencecomesclosest.Compressiondepth,notitslocation,isthereproduciblesignatureofgenuineparagraphstructureinoursetting.
View arXiv pageView PDFGitHub0Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.23551 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.23551 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.23551 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Text Distance from Nested and Hierarchical Repetitions: A Compression-Based Perspective
This paper presents a new method for structural sequence analysis using the Ladderpath approach to extract nested and hierarchical repetitions, defining three distance measures that outperform gzip-based NCD and BERT in out-of-distribution and few-shot text classification tasks, offering a lightweight and interpretable alternative.
DepthBench: Measuring How Residual Connections Enable More Computational Depth
This paper introduces DepthBench, a controlled benchmark for studying computational depth in Transformers by varying width-depth ratios. It identifies that residual connection designs like HC and FullAttnRes are key to translating architectural depth into effective computation.
When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning
This paper analyzes why compression statistics for LLM pruning can be reproducible yet select suboptimal endpoints, introducing information boundaries and observation fibers to model the gap. It proposes group-resolved and model-specific mask selection methods that improve worst-group perplexity across dense LLMs and OLMoE.
High Dimensional, Dynamic Rotary Positional Embedding [P]
Introduces HDD-RoPE, an extension of rotary positional embeddings that uses high-dimensional chunks and data-dependent rotation rates, showing faster convergence on TinyStories compared to xPos.
Context Compression Is Not One Thing: Readable Symbolic Re-expression vs. Coherent Summary at Matched Budget
This paper proposes Telegraph English, a readable symbolic format for context compression that outperforms matched-budget baselines on multi-hop QA datasets, preserving entity content more densely.