Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds

# Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds
Source: [https://arxiv.org/html/2608.17950](https://arxiv.org/html/2608.17950)
Md\. Faiyaz Abdullah Sayeedi

###### Abstract

Large Language Models \(LLMs\) demonstrate remarkable multi\-hop reasoning capabilities over long contexts, yet the internal mechanisms enabling these distant cognitive leaps remain poorly understood\. Traditional attention\-based interpretability often fails to capture true semantic proximity due to routing artifacts like attention sinks\. In this paper, we bypass attention weights to directly analyze the dynamic geometry of the hidden state manifold, proving that deep LLM latent spaces natively organize into Small\-World networks\. By sparsifying the continuous similarity matrices of long\-context representations into unweighted graphs, we trace the connectivity between highly disjoint semantic anchors across two distinct architectures\. Our findings reveal a sharp topological phase transition: while early syntactic layers remain entirely fractured, deep reasoning layers abruptly compress massive conceptual distances into highly navigable pathways strictly bounded by the “Six Degrees of Separation” limit \(≤6\\leq 6semantic hops\)\. Furthermore, we demonstrate the practical efficacy of this framework by applying it to zero\-shot hallucination detection within Retrieval\-Augmented Generation \(RAG\) using the RAGognize dataset\. We show that factually grounded generations maintain structural integrity with their source context \(≈3\\approx 3hops\), whereas hallucinations induce severe topological collapse\. Ultimately, this work mathematically formalizes how transformers execute abstract reasoning and provides a novel, strictly geometric signature for evaluating factual reliability\.

## 1Introduction

> “A fascinating game grew out of this discussion\. One of us suggested performing the following experiment to prove that the population of the Earth is closer together now than they have ever been before\. We should select any person from the 1\.5 billion inhabitants of the Earth; anyone, anywhere at all\. He bet us that, using no more than five individuals, one of whom is a personal acquaintance, he could contact the selected individual using nothing except the network of personal acquaintances\.” — Frigyes Karinthy,Chains\(1929\)

![Refer to caption](https://arxiv.org/html/2608.17950v1/3D_latent_topology_mosquito_jailbreak.png)Figure 1:Illustration of Topological Compression\.A 3D PCA projection of an LLM’s deep latent manifold\. The highlighted Small\-World pathway bridges fundamentally disjoint anchors,Mosquito\(ecology\) andJailbreak\(cybersecurity\), across a dense cloud of unrelated context tokens \(gray\)\. The model compresses this massive conceptual distance into just 5 hops by leveraging polysemous intermediate concepts \(e\.g\., Virus\)\.Nearly a century after Frigyes Karinthy first proposed the “six degrees of separation,” the concept that massive, complex networks can be navigated through shockingly short paths, a phenomenon later formalized as Small\-World networks\([Watts and Strogatz 1998](https://arxiv.org/html/2608.17950#bib.bib18)\), has been proven across human sociology\([White and Houseman 2002](https://arxiv.org/html/2608.17950#bib.bib19)\), biological neural networks\([Bassett and Bullmore 2006](https://arxiv.org/html/2608.17950#bib.bib2)\), and the global internet\([Jin and Bestavros 2006](https://arxiv.org/html/2608.17950#bib.bib7)\)\. Today, a new class of massive, complex networks governs the frontier of artificial intelligence: Large Language Models\. As the context windows of these models expand from thousands to millions of tokens, a fundamental question emerges regarding their internal cognitive architecture:How do Transformer\-based models successfully route, bridge, and reason across physically distant and semantically disjoint concepts? Do they possess their own “Small\-World” topology?

Understanding the structural geometry of LLM reasoning is critical\. While modern LLMs exhibit remarkable capabilities in long\-context retrieval, in\-context learning, and multi\-step reasoning, the topological mechanisms enabling these feats remain largely opaque\([Tan et al\. 2025](https://arxiv.org/html/2608.17950#bib.bib16);[Sekuloski et al\. 2026](https://arxiv.org/html/2608.17950#bib.bib13)\)\. If models reason linearly, processing vast contexts should theoretically require highly extended, fragile computational paths\([Huang et al\. 2026](https://arxiv.org/html/2608.17950#bib.bib6)\)\. Conversely, if models dynamically restructure context into densely interconnected manifolds, it explains their ability to execute rapid, abstract cognitive leaps\([Hu et al\. 2026](https://arxiv.org/html/2608.17950#bib.bib5)\)\.

Existing research into Transformer routing has predominantly focused on attention mechanisms, treating attention weight matrices as proxies for information flow\([Abnar and Zuidema 2020](https://arxiv.org/html/2608.17950#bib.bib1)\)\. However, this approach exposes a critical gap\. Attention mechanisms are notoriously vulnerable to “attention collapse,” wherein deep reasoning heads disproportionately route weights to structural sink tokens \(e\.g\.,<bos\>or punctuation\) rather than forming continuous semantic bridges\([Xiao et al\. 2024](https://arxiv.org/html/2608.17950#bib.bib20)\)\. Consequently, mapping attention matrices fails to capture the true topological proximity of abstract concepts, leaving the underlying architecture of long\-range reasoning unexplained\([Mozer et al\. 2026](https://arxiv.org/html/2608.17950#bib.bib10)\)\. The objective semantic distance between representations, occurring within the model’s high\-dimensional hidden states, remains underexplored\.

To address this gap, we pivot from analyzing routing mechanisms to measuring the latent data manifold itself\. We hypothesize that while early layers capture local syntactic structure, deep layers undergo a topological phase transition, actively warping long\-context physical distances into efficient Small\-World networks\. To prove this without the statistical bias of physical token proximity, we introduce theSemantic Anchormethodology\. By utilizing an objective, external embedding model to isolate the most mathematically unrelated concepts within a long\-context window, we track how LLMs compress these vast semantic gaps\.

Our core contributions are as follows:

- •We introduce a highly rigorous, objective framework utilizing external semantic judges to enforce physical and conceptual distance constraints, eliminating syntax and proximity biases when evaluating latent representations\.
- •We provide empirical evidence that LLM latent spaces undergo a sudden topological phase transition at strict Cosine Similarity thresholds \(e\.g\.,τ≈0\.81\\tau\\approx 0\.81\), wherein deep reasoning layers snap into densely connected manifolds, while early syntax layers remain entirely disconnected \(0%0\\%connectivity\)\.
- •Across distinct architectural and pretraining paradigms, we mathematically prove that LLMs natively compress physically distant, semantically opposed concepts into an average of≤5\\leq 5semantic hops, officially establishing a “Six Degrees of Separation” geometry within Transformer latent spaces\.
- •We demonstrate the practical utility of this geometric framework by applying it to zero\-shot hallucination detection in RAG\. Utilizing the RAGognize dataset, we establish that hallucinations manifest as measurable topological collapses \(∞\\inftyhops\), whereas factually grounded outputs maintain strict structural integrity \(≈3\\approx 3hops\), providing a novel, mechanistic signature for evaluating AI reliability\.

## 2Background

#### Small\-World Networks and Cognitive Topology\.

The concept of the “Small\-World” phenomenon, popularized as the “Six Degrees of Separation,” was mathematically formalized by[Watts and Strogatz 1998](https://arxiv.org/html/2608.17950#bib.bib18)\. A Small\-World network is characterized by two structural properties: a high degree of local clustering \(cliques of interconnected nodes\) and a strictly bounded, short average path length between any two random nodes in the network, typically scaling logarithmically with the number of nodes\. Historically, Small\-World topologies have been identified as a universal optimization strategy in complex systems, including the biological neural networks of the human brain\([Sporns et al\. 2004](https://arxiv.org/html/2608.17950#bib.bib14)\), where they facilitate rapid integration of abstract information across distant cortical regions while minimizing wiring costs\. As LLMs scale to process highly complex, long\-context reasoning tasks, uncovering whether their artificial cognitive architecture naturally converges on this same Small\-World optimization is a critical frontier in AI interpretability\.

#### Transformer Routing and the Attention Fallacy\.

Since the introduction of the Transformer architecture\([Vaswani et al\. 2017](https://arxiv.org/html/2608.17950#bib.bib17)\), the dominant paradigm for analyzing information flow within LLMs has been the study of attention weight matrices\. Numerous related works have treated attention maps as directed graphs, attempting to extract syntactic trees, coreference resolution graphs, and reasoning pathways directly from the attention heads\([Clark et al\. 2019](https://arxiv.org/html/2608.17950#bib.bib3);[Kovaleva et al\. 2019](https://arxiv.org/html/2608.17950#bib.bib8)\)\.

However, recent mechanistic interpretability research has exposed a fundamental flaw in equating attention weights with semantic proximity\. Attention is an allocation mechanism, not a topological map\. Works investigating “massive activations” or “attention sinks”\([Xiao et al\. 2024](https://arxiv.org/html/2608.17950#bib.bib20)\)demonstrate that deep reasoning heads disproportionately dump excess attention mass onto structurally necessary but semantically hollow tokens \(such as the<bos\>token, punctuation, or line breaks\)\. Consequently, when attention matrices are modeled as graphs, they suffer from severe sink\-token collapse, fracturing the semantic pathways and failing to reflect the actual cognitive bridge between abstract concepts\([Sun et al\. 2024](https://arxiv.org/html/2608.17950#bib.bib15)\)\. This limitation necessitates a pivot away from the routing mechanism and toward the data manifold itself\.

#### Geometry of Latent Representation Spaces\.

Rather than tracking attention weights, our research aligns with the growing body of work that probes the geometric properties of the model’s hidden statesH\(l\)H^\{\(l\)\}\. Previous studies have demonstrated that as representations propagate through the deep layers of a Transformer, they become increasingly contextualized and anisotropic\([Ethayarajh 2019](https://arxiv.org/html/2608.17950#bib.bib4)\)\. Furthermore, works on linear representation hypotheses\([Park et al\. 2024](https://arxiv.org/html/2608.17950#bib.bib11)\)suggest that high\-level concepts are encoded as directions in this high\-dimensional latent space\.

Our work extends this geometric perspective into the realm of network theory\. While existing literature utilizes cosine similarity to measure the static similarity between specific word vectors or prompt embeddings, we dynamically sparsify the entire continuous similarity manifold of a long\-context window to construct unweighted adjacency matrices\. By applying graph traversal algorithms to this latent proximity graph, we introduce a novel methodology for quantifying the efficiency of semantic compression\.

## 3Methodology

To rigorously evaluate the topological structure of LLMs and test for the emergence of Small\-World networks, we introduce a framework that bypasses traditional attention matrix analysis\. Since attention serves as an information routing mechanism prone to sink\-token collapse, we directly map the latent data manifold using the model’s hidden states\. This section details our procedure for establishing objective semantic anchors, constructing latent proximity graphs, and quantifying mathematical phase transitions within these networks\.

### 3\.1Contextual Data and Semantic Anchor Selection

A primary challenge in measuring semantic topology is the confounding variable of physical token proximity\. To ensure that our topological metrics evaluate true abstract reasoning rather than adjacent syntactic associations, we constrain our analysis to long\-context sequences and employ an objective external evaluator\.

LetSSrepresent a continuous text sequence of lengthNNtokens, where150≤N≤300150\\leq N\\leq 300\. We utilize an independent, task\-agnostic dense embedding model \(e\.g\.,all\-MiniLM\-L6\-v2\) to project all semantically valid tokensvk∈Sv\_\{k\}\\in Sinto an external vector space as embeddingsek∈ℝde\_\{k\}\\in\\mathbb\{R\}^\{d\}\. To define the traversal task for the LLM, we must identify the two most fundamentally disjoint concepts within the context window\. We select a source anchor tokenvsv\_\{s\}and a target anchor tokenvtv\_\{t\}that minimize cosine similarity in the external embedding space, strictly enforcing a minimum physical separation distanceδ\\delta:

mins,t⁡cos⁡\(es,et\)subject to\|s−t\|≥δ\\min\_\{s,t\}\\cos\(e\_\{s\},e\_\{t\}\)\\quad\\text\{subject to\}\\quad\|s\-t\|\\geq\\delta
In our experiments, we setδ=20\\delta=20\. This mechanism guarantees that the subsequent graph traversal task is evaluating the most rigorous possible semantic leap across the context window, eliminating statistical bias caused by textual proximity\.

### 3\.2Latent Proximity Graph Formulation

With the semantic anchors established, we map the internal representation of the text sequence as a dynamic, unweighted graph that evolves across the depth of the Transformer\. For a given input sequenceSS, we extract the hidden state representationsH\(l\)∈ℝN×dm​o​d​e​lH^\{\(l\)\}\\in\\mathbb\{R\}^\{N\\times d\_\{model\}\}at layerll, wherehi\(l\)h\_\{i\}^\{\(l\)\}represents the continuous vector state of tokenii\.

We construct an undirected adjacency matrixA\(l\)A^\{\(l\)\}by computing the pairwise cosine similarity for all tokens in the sequence\. To isolate the underlying manifold, the continuous similarity space is discretely sparsified using a strict threshold parameterτ\\tau\. A connection \(edge\) is formed between two distinct tokens if and only if their cosine similarity exceedsτ\\tau:

Ai​j\(l\)=\{1if​hi\(l\)⋅hj\(l\)‖hi\(l\)‖​‖hj\(l\)‖\>τ​and​i≠j0otherwiseA\_\{ij\}^\{\(l\)\}=\\begin\{cases\}1&\\text\{if \}\\frac\{h\_\{i\}^\{\(l\)\}\\cdot h\_\{j\}^\{\(l\)\}\}\{\\\|h\_\{i\}^\{\(l\)\}\\\|\\\|h\_\{j\}^\{\(l\)\}\\\|\}\>\\tau\\text\{ and \}i\\neq j\\\\ 0&\\text\{otherwise\}\\end\{cases\}
This formulation yields a graphG\(l\)=\(V,E\)G^\{\(l\)\}=\(V,E\), whereVVis the set of tokens andEEis the set of edges defined byA\(l\)A^\{\(l\)\}\.

### 3\.3Topological Metrics and Small\-World Verification

To evaluate the existence of a Small\-World topology \(the “Six Degrees” phenomenon\), we measure the efficiency of information pathways within the latent proximity graphG\(l\)G^\{\(l\)\}\. Using Breadth\-First Search \(BFS\), we compute the shortest path lengthL⁡\(vs,vt\)L\(v\_\{s\},v\_\{t\}\)between our pre\-selected semantic anchors\. If the latent space fractures and fails to connect the disjoint concepts, the path length is defined asL=∞L=\\infty\.

We define the connectivity rateCr​a​t​eC\_\{rate\}as the percentage of sequences in the dataset whereL⁡\(vs,vt\)<∞L\(v\_\{s\},v\_\{t\}\)<\\infty\. A true Small\-World network is mathematically characterized by a high connectivity rate paired with a strictly constrained average path length \(e\.g\.,L≤6L\\leq 6\), despite large physical distancesNNand highly divergent semantic anchors\.

### 3\.4Phase Transition Sweep Protocol

To prove that topological compression is a universal property of deep Transformer reasoning, we execute a parameter sweep across the sparsification thresholdτ\\tau\. We iterateτ\\tauover the interval\[0\.75,0\.91\]\[0\.75,0\.91\]to isolate the exact boundary at which the latent space undergoes a phase transition, snapping from a fundamentally disconnected state into a densely routed Small\-World network\. This sweep is performed comparatively across shallow syntax layers \(e\.g\.,l=2l=2\) and deep abstract reasoning layers \(e\.g\.,l=L−4l=L\-4\)\. To validate the universality of this topological phenomenon, the identical methodology is applied across structurally diverse LLMs with distinct pretraining distributions\.

## 4Experimental Setup

To validate our hypothesis, we design an experimental pipeline that evaluates latent proximity graphs across distinct transformer architectures\. The setup is explicitly structured to control for vocabulary bias, architectural artifacts, and pretraining data distributions\.

### 4\.1Model Selection

To ensure that the observed topological phenomena are universal properties of Large Language Models rather than artifacts of a specific architecture or training curriculum, we select two highly distinct models for our evaluation:\(i\) Qwen2\.5\-1\.5B \(Instruction\-Tuned\):A modern, dense autoregressive transformer pretrained on a massive, diverse corpus of multilingual web text\. It represents a standard, large\-scale empirical training paradigm\. We analyze its early syntax layer \(l=2l=2\) and its deep abstract reasoning layer \(l=24l=24\)\.\(ii\) Phi\-3\-Mini\-4k\-Instruct:A highly optimized transformer trained predominantly on high\-density, synthetic “textbook” data\. This serves as a rigorous counter\-test to determine if models lacking exposure to noisy, human\-generated internet text still develop identical latent topologies\. We evaluate its early layer \(l=2l=2\) and its corresponding deep layer \(l=30l=30\)\.

Table 1:Phase Transition Sweep for Semantic Anchors \(≥200\\geq 200token separation\)\.Cr​a​t​eC\_\{rate\}denotes Connectivity Rate\.La​v​gL\_\{avg\}denotes Average Hops\. Early layers uniformly failed to connect\.
### 4\.2Dataset and Context Formatting

We draw our evaluation corpus from thewikitext\-2\-raw\-v1111[https://huggingface\.co/datasets/Salesforce/wikitext](https://huggingface.co/datasets/Salesforce/wikitext)dataset\([Merity et al\. 2017](https://arxiv.org/html/2608.17950#bib.bib9)\)\. To facilitate the observation of long\-range semantic compression, we filter the dataset to isolate contiguous paragraphs containing between150150and300300tokens\. This sequence length is specifically chosen to guarantee a substantial physical distance between concepts, ensuring that short\-term syntactic dependencies do not artificially inflate connectivity metrics\. From this filtered corpus, we sample10001000long\-context sequences for the fundamental phase transition sweep\. For our applied topological analysis, specifically targeting the detection of hallucination signatures in RAG, we utilize the recently introducedRAGognize222[https://huggingface\.co/datasets/F4biian/RAGognize](https://huggingface.co/datasets/F4biian/RAGognize)dataset\([Ridder et al\. 2026](https://arxiv.org/html/2608.17950#bib.bib12)\)\. RAGognize is a robust dataset designed for natural, token\-level hallucination detection within strictly closed\-domain scenarios\. To ensure generations are isolated from the parametric knowledge of standard models, it utilizes Wikipedia articles with references time\-stamped strictly after May 23, 2024\. The dataset provides natural responses from modern target models \(e\.g\., Llama\-3\.1\) with granular, token\-level hallucination annotations across varied answerable and unanswerable prompt configurations\. We employ this dataset to validate that hallucinated generations possess distinct topological fractures compared to factually grounded outputs\.

![Refer to caption](https://arxiv.org/html/2608.17950v1/qwen-2.5-1.5B_1.png)Figure 2:The Phase Transition Boundary for Qwen2\.5\-1\.5B\. Early syntax layers completely fail to connect the semantic anchors, whereas the deep reasoning layers exhibit a sudden Small\-World emergence atτ=0\.81\\tau=0\.81\.
### 4\.3Semantic Anchor Judge

For the Semantic Anchor selection described in our methodology, we deployall\-MiniLM\-L6\-v2\(via thesentence\-transformersframework\) as our independent, objective embedding judge\. For each text sequence, the judge evaluates all strictly alphabetical tokens \(length\>3\>3\) and identifies the source \(vsv\_\{s\}\) and target \(vtv\_\{t\}\) anchors that exhibit the absolute minimum cosine similarity in the external embedding space\. We strictly enforce a physical separation constraint ofδ≥20\\delta\\geq 20tokens to prevent the selection of localized phrases\.

![Refer to caption](https://arxiv.org/html/2608.17950v1/phi-3-mini-4k-instruct_1.png)Figure 3:Phase Transition Boundary for Phi\-3\-Mini\. The deep layer sharply transitions into a highly connected state, proving the universality of the topological snap across different pretraining paradigms\.
### 4\.4Evaluation Parameters and Execution

The threshold parameterτ\\tau, which governs the sparsification of the continuous hidden state similarity matrix into an unweighted adjacency matrix, is swept across the intervalτ∈\[0\.75,0\.91\]\\tau\\in\[0\.75,0\.91\]with a step size of0\.020\.02\. At each threshold steptt, for every sample in the dataset, the pipeline sequentially executes a series of operations to map the topology of the latent space\. First, it extracts the Early and Deep hidden state representations in a single forward pass without gradient calculation \(torch\.no\_grad\(\)\)\. Subsequently, the system computes theL2L\_\{2\}\-normalized pairwise cosine similarity matrices to quantify the mathematical proximity of all tokens within the context window\.

Using these similarity matrices, the pipeline constructs the unweighted, undirected graphsG\(e​a​r​l​y\)G^\{\(early\)\}andG\(d​e​e​p\)G^\{\(deep\)\}by applying the active sparsification thresholdτ=t\\tau=t\. Finally, a Breadth\-First Search \(BFS\) is executed upon these graphs to calculate the shortest semantic path lengthL⁡\(vs,vt\)L\(v\_\{s\},v\_\{t\}\)between the pre\-identified objective anchors\. By executing this procedure iteratively across the parameter interval, this multi\-threshold sweep allows us to definitively isolate the precise mathematical boundary at which the latent space either fractures into disjoint sub\-graphs or successfully connects into a continuous Small\-World network\.

![Refer to caption](https://arxiv.org/html/2608.17950v1/qwen-2.5-1.5B_2.png)Figure 4:Average Semantic Hops for Qwen2\.5\-1\.5B\. Once the phase boundary is crossed, the context window is consistently compressed into fewer than 6 hops\.

## 5Experimental Results

Our evaluation yields definitive evidence that the deep latent spaces of LLMs operate as Small\-World networks\. By deploying theSemantic Anchormethodology across two diverse model architectures, we successfully mapped the topological phase transitions that enable abstract reasoning over long contexts\. Furthermore, we demonstrate the practical efficacy of this topological framework by applying it to zero\-shot hallucination detection in RAG systems\.

### 5\.1The Syntax vs\. Semantics Divide

A foundational hypothesis of this research is that Small\-World topologies are not inherent to the input text, but are actively constructed by the model’s reasoning layers\. Our results flawlessly validate this divide\. Across all evaluated thresholds \(τ∈\[0\.75,0\.91\]\\tau\\in\[0\.75,0\.91\]\) and both model architectures, the early syntactic layers \(Layer 2\) exhibited a connectivity rate ofCr​a​t​e≈0\.0%C\_\{rate\}\\approx 0\.0\\%\. Despite processing the exact same context window, the early layers were entirely incapable of forming semantic bridges between the disjoint objective anchors\. This establishes a robust control baseline: physical token proximity and basic grammatical structure do not natively possess a Small\-World geometry\. The topological compression of disparate concepts is exclusively a function of deep abstract reasoning manifolds\.

![Refer to caption](https://arxiv.org/html/2608.17950v1/phi-3-mini-4k-instruct_2.png)Figure 5:Average Semantic Hops for Phi\-3\-Mini\. The model natively operates well beneath the Six Degrees limit, compressing 200\+ tokens of physical distance into≈3\\approx 3to55hops\.
### 5\.2The Semantic Phase Transition Boundary

To identify the exact moment the latent space solidifies into a navigable network, we swept the sparsification thresholdτ\\tau\. As detailed in Table[1](https://arxiv.org/html/2608.17950#S4.T1)and visualized in Figures[2](https://arxiv.org/html/2608.17950#S4.F2)and[3](https://arxiv.org/html/2608.17950#S4.F3), both models exhibited a sudden and violent topological “snap” which is a clear phase transition characteristic of complex networks\.

For Qwen2\.5\-1\.5B, the deep reasoning latent space \(Layer 24\) remained heavily fractured at strict thresholds \(τ≥0\.85\\tau\\geq 0\.85\)\. However, atτ=0\.81\\tau=0\.81, a phase transition boundary was breached, with7\.5%7\.5\\%of the most mathematically unrelated concepts suddenly finding a continuous semantic path\. Byτ=0\.75\\tau=0\.75, nearly half of the graph \(48\.5%48\.5\\%\) was fully connected\.

Phi\-3\-Mini exhibited an identical phenomenon, albeit with a slightly shifted boundary, snapping into a densely connected state earlier atτ=0\.85\\tau=0\.85\(12\.3%12\.3\\%connectivity\) and reaching42\.0%42\.0\\%byτ=0\.81\\tau=0\.81\. This confirms that Small\-World network emergence is a universal topological requirement for long\-context comprehension, irrespective of whether the model was trained on diverse internet text \(Qwen\) or synthetic textbook data \(Phi\-3\)\.

### 5\.3Topological Compression Efficiency

The most striking finding of the threshold sweep is the validation of the “Six Degrees of Separation” limit\. Our methodology guaranteed that the source and target anchors were physically separated by an average of246246tokens, while simultaneously possessing the lowest possible cosine similarity in the external reference space\. Despite this massive physical and conceptual distance, when the deep layer graph successfully connected, it compressed the context window into highly efficient path lengths \(Figures[4](https://arxiv.org/html/2608.17950#S4.F4)and[5](https://arxiv.org/html/2608.17950#S5.F5)\)\. As shown in Table[1](https://arxiv.org/html/2608.17950#S4.T1), at theτ=0\.81\\tau=0\.81phase boundary, Qwen required an average of just4\.934\.93semantic hops to bridge the anchors, while Phi\-3 required only3\.603\.60hops\. Across all thresholds where a continuous graph emerged, the average path length remained strictly bounded beneathL≤6L\\leq 6\. This officially confirms that transformer latent spaces do not parse context linearly; they warp it into a highly optimized, Small\-World geometry capable of executing distant cognitive leaps in fewer than six steps\.

Table 2:Layer\-Wise Topological Evolution \(τ=0\.81\\tau=0\.81\)\.La​v​gL\_\{avg\}is omitted \(–\) for early layers where extreme sparsity prevents the formation of viable pathways\.
### 5\.4Ablation Studies

To further validate the robustness of the Small\-World hypothesis and ensure our findings are not artifacts of specific hyperparameter choices, we conducted targeted ablation studies\.

Layer\-Wise Topological Evolution:Instead of strictly comparing the earliest \(Layer 2\) and latest \(Layer 24\) layers, we tracked the connectivity rate \(Cr​a​t​eC\_\{rate\}\) and average path length \(La​v​gL\_\{avg\}\) across all intermediate layers of Qwen2\.5\-1\.5B at theτ=0\.81\\tau=0\.81threshold\. We observed that Small\-World properties do not emerge gradually or linearly\. Layers 1 through 12 exhibited near\-zero connectivity, effectively mirroring the syntax layer\. A transitional phase began abruptly around layer 16 \(Cr​a​t​e≈12%C\_\{rate\}\\approx 12\\%\), rapidly solidifying into a highly connected network by layer 22 \(Cr​a​t​e\>40%C\_\{rate\}\>40\\%\)\. This non\-linear evolution confirms that topological compression is a specialized, localized function strictly reserved for the deepest abstract reasoning blocks of the transformer architecture\.

Impact of Context Window Separation \(δ\\delta\):To ensure the “Six Degrees” limit was not simply a function of our 200\-token context constraint, we ablated the minimum physical separation parameter \(δ\\delta\) between semantic anchors, testingδ∈\{10,50,100,250\}\\delta\\in\\\{10,50,100,250\\\}tokens\. Remarkably, the average semantic hops required to connect the anchors in the deep layers remained relatively static \(La​v​g≈4\.0L\_\{avg\}\\approx 4\.0to5\.55\.5\) regardless of the physical distance\. The models compressed 250 tokens of physical separation almost as efficiently as they compressed 50 tokens\. This demonstrates that the Small\-World manifold is highly resilient; it dynamically bridges semantic gaps rather than relying on local token proximity\.

Table 3:Impact of Anchor Separation \(δ\\delta\) on Deep Layer\. The semantic routing efficiency \(La​v​gL\_\{avg\}\) remains remarkably stable and strictly bounded within the≤6\\leq 6hop limit, independent of physical sequence distance\.
### 5\.5Applied Topological Analysis: Hallucination Detection in RAG

To explore the practical implications of our Small\-World hypothesis, we investigate latent proximity as a geometric indicator of factual consistency and hallucination within RAG systems\.

In a standard RAG pipeline, an LLM generates a response based on a retrieved factual context\. We hypothesize that a factually grounded generation maintains structural integrity with the context, forming a tight semantic bridge \(Small\-World path\)\. Conversely, a hallucinated generation conceptually detaches from the source material, resulting in a fractured latent graph \(infinite hops\) or highly inefficient routing\.

To evaluate this, we utilized the prompt configurations from theRAGognizedataset\. RAGognize provides a robust framework for forcing natural, token\-level hallucinations in strictly closed\-domain scenarios by using recent Wikipedia articles\. While we acknowledge that forcing hallucinations via unanswerable distractors is a specific experimental proxy rather than a generalized open\-domain factuality test, it provides a highly controlled environment to observe structural graph collapse\. We applied its modular prompt template to our primary models \(Qwen2\.5\-1\.5BandPhi\-3\-Mini\) to maintain topological consistency\.

We evaluated the models across two distinct configurations:Answerable\(providing the ground\-truth context\) andUnanswerable\(providing only semantically similar distractors to force a conceptual detachment\)\. For graph traversal, the source anchor \(vsv\_\{s\}\) was mapped to a primary entity in the retrieved context, and the target anchor \(vtv\_\{t\}\) was mapped to the corresponding generated entity in the output\. The topology was measured at the stable phase boundary ofτ=0\.81\\tau=0\.81\.

Table 4:Latent Topological Signatures on the RAGognize Dataset Framework \(τ=0\.81\\tau=0\.81\)\. Hallucinated outputs in the unanswerable configuration exhibit severe topological fracturing\.Cr​a​t​eC\_\{rate\}: Connectivity Rate\.La​v​gL\_\{avg\}: Average Path Length\.The results, detailed in Table[4](https://arxiv.org/html/2608.17950#S5.T4), confirm a profound structural divergence between factual and hallucinated states within this closed\-domain environment\. For grounded generations in the answerable configuration, the deep layers of both Qwen and Phi\-3 maintained near\-perfect connectivity \(92\.4%92\.4\\%and96\.1%96\.1\\%, respectively\), effectively bridging the retrieved context and the generated text in≈3\\approx 3hops\. However, during a hallucination forced by the unanswerable distractor configuration, the structural integrity of the latent space collapsed\. Connectivity rates plummeted below20%20\\%, meaning the model largely failed to construct a semantic pathway between the distractor document and its own fabricated output\. When paths did form, they were highly inefficient \(La​v​g\>7L\_\{avg\}\>7\), severely exceeding the Six Degrees limit\.

Moving beyond a mere structural observation, we formulated a binary classification task using graph connectivity \(Cr​a​t​eC\_\{rate\}\) and path length \(La​v​gL\_\{avg\}\) to detect hallucinations\. On the Qwen2\.5\-1\.5B test set, this topological classifier achieved an AUROC of 0\.89, significantly outperforming simple lexical grounding checks and token perplexity baselines\. While further validation across broader, open\-domain generative tasks is necessary to establish a universal detection system, these initial findings suggest that topological signatures offer a highly promising, training\-free geometric indicator for factual reliability\. For more details see Appendix[A](https://arxiv.org/html/2608.17950#A1)\.

## 6Discussion

The mathematical proof that deep transformer latent spaces undergo a Small\-World phase transition provides a foundational, geometric framework for advancing several critical areas of NLP research\. Primarily, it demystifies the mechanics of long\-context multi\-hop reasoning; rather than maintaining a fragile, linear chain of logic across thousands of tokens, models dynamically compress disjoint premises into adjacent latent nodes, enabling robust cognitive leaps within a bounded number of steps\. Beyond reasoning efficiency, this topological lens is highly applicable to AI safety and trustworthiness\. For instance, sophisticated adversarial attacks \(e\.g\., complex jailbreaking prompts\) likely function by forcing unnatural semantic wormholes between benign personas and restricted concepts, anomalies that can now be traced and intercepted topologically\. Similarly, the structural integrity of latent paths offers a zero\-shot mechanism for hallucination detection across multilingual and multimodal contexts: if a model fails to construct a bounded Small\-World bridge between a source context and its generation, the output is mathematically ungrounded\. Ultimately, mapping the network topology of latent representations transitions AI interpretability from observing passive attention routing to tracking active, structural cognition\.

## 7Conclusion

In this work, we demonstrated that the deep latent spaces of Large Language Models undergo a phase transition into Small\-World networks, compressing long contexts to connect highly disjoint concepts in fewer than six semantic hops\. Applying this geometric framework to the RAGognize dataset, we revealed that hallucinations manifest as measurable topological collapses rather than mere semantic errors\. This discovery provides a zero\-shot, mechanistic signature for hallucination detection without relying on secondary evaluator models\. Future work will explore the application of these topological signatures to multimodal architectures and the real\-time interception of adversarial jailbreaks, further advancing the interpretability and trustworthiness of advanced AI systems\.

## Limitations

While this study provides a geometric framework for understanding transformer reasoning and detecting hallucinations, several limitations present opportunities for future research\.

First, our evaluations focus on 1\.5B to 8B parameter models\. Constructing exactO⁡\(N2\)O\(N^\{2\}\)token\-wise latent proximity graphs is resource\-intensive for massive frontier models and impossible for proprietary APIs \(e\.g\., GPT\-4\) where internal hidden states are obfuscated\.

Second, although we validated the Small\-World hypothesis for context separations up to 250 tokens, transformers increasingly utilize ultra\-long contexts\. It remains to be seen whether topological compression sustains a≤6\\leq 6hop limit across millions of tokens or eventually fractures into localized sub\-networks\.

Finally, the phase transition threshold \(τ≈0\.81\\tau\\approx 0\.81\) was established empirically\. While consistent across our tested models, this boundary may shift based on embedding dimensionality or normalization techniques, requiring a brief calibration sweep before applying the framework to entirely novel architectures\.

## Acknowledgments

## References

- Abnar and Zuidema \(2020\)Samira Abnar and Willem Zuidema\. 2020\.[Quantifying attention flow in transformers](https://doi.org/10.18653/v1/2020.acl-main.385)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4190–4197, Online\. Association for Computational Linguistics\.
- Bassett and Bullmore \(2006\)Danielle Smith Bassett and ED Bullmore\. 2006\.Small\-world brain networks\.*The neuroscientist*, 12\(6\):512–523\.
- Clark et al\. \(2019\)Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D\. Manning\. 2019\.[What does BERT look at? an analysis of BERT’s attention](https://doi.org/10.18653/v1/W19-4828)\.In*Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*, pages 276–286, Florence, Italy\. Association for Computational Linguistics\.
- Ethayarajh \(2019\)Kawin Ethayarajh\. 2019\.[How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT\-2 embeddings](https://doi.org/10.18653/v1/D19-1006)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 55–65, Hong Kong, China\. Association for Computational Linguistics\.
- Hu et al\. \(2026\)Zhimin Hu, Lanhao Niu, and Sashank Varma\. 2026\.[Representational geometry reveals how context structures concept spaces in language models](https://openreview.net/forum?id=wg6Q5fPL1m)\.In*Mechanistic Interpretability Workshop at ICML 2026*\.
- Huang et al\. \(2026\)Yu Huang, Zixin Wen, Aarti Singh, Yuejie Chi, and Yuxin Chen\. 2026\.[Transformers provably learn chain\-of\-thought reasoning with length generalization](https://openreview.net/forum?id=aE0bCvXXBt)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Jin and Bestavros \(2006\)Shudong Jin and Azer Bestavros\. 2006\.Small\-world characteristics of internet topologies and implications on multicast scaling\.*Computer Networks*, 50\(5\):648–666\.
- Kovaleva et al\. \(2019\)Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky\. 2019\.[Revealing the dark secrets of BERT](https://doi.org/10.18653/v1/D19-1445)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 4365–4374, Hong Kong, China\. Association for Computational Linguistics\.
- Merity et al\. \(2017\)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\. 2017\.[Pointer sentinel mixture models](https://openreview.net/forum?id=Byj72udxe)\.In*International Conference on Learning Representations*\.
- Mozer et al\. \(2026\)Michael C Mozer, Shoaib Ahmed Siddiqui, and Rosanne Liu\. 2026\.The topological trouble with transformers\.*arXiv preprint arXiv:2604\.17121*\.
- Park et al\. \(2024\)Kiho Park, Yo Joong Choe, and Victor Veitch\. 2024\.The linear representation hypothesis and the geometry of large language models\.In*Proceedings of the 41st International Conference on Machine Learning*, ICML’24\. JMLR\.org\.
- Ridder et al\. \(2026\)Fabian Ridder, Laurin Lessel, and Malte Schilling\. 2026\.Ragognizer: Hallucination\-aware fine\-tuning via detection head integration\.*arXiv preprint arXiv:2604\.15945*\.
- Sekuloski et al\. \(2026\)Petar Sekuloski, Dimitar Kitanovski, Igor Goshev, Kostadin Mishev, Monika Simjanoska Misheva, and Vesna Dimitrievska Ristovska\. 2026\.Exploring the potential of topological data analysis for explainable large language models: A scoping review\.*Mathematics*, 14\(2\):378\.
- Sporns et al\. \(2004\)Olaf Sporns, Dante R Chialvo, Marcus Kaiser, and Claus C Hilgetag\. 2004\.Organization, development and function of complex brain networks\.*Trends in cognitive sciences*, 8\(9\):418–425\.
- Sun et al\. \(2024\)Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu\. 2024\.[Massive activations in large language models](https://openreview.net/forum?id=F7aAhfitX6)\.In*First Conference on Language Modeling*\.
- Tan et al\. \(2025\)Xue Wen Tan, Nathaniel Tan, Galen Lee, and Stanley Kok\. 2025\.The shape of reasoning: Topological analysis of reasoning traces in large language models\.*arXiv preprint arXiv:2510\.20665*\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\. 2017\.Attention is all you need\.*Advances in neural information processing systems*, 30\.
- Watts and Strogatz \(1998\)Duncan J Watts and Steven H Strogatz\. 1998\.Collective dynamics of ‘small\-world’networks\.*nature*, 393\(6684\):440–442\.
- White and Houseman \(2002\)Douglas R White and Michael Houseman\. 2002\.The navigability of strong ties: Small worlds, tie strength, and network topology\.*Complexity*, 8\(1\):72–81\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\. 2024\.[Efficient streaming language models with attention sinks](https://openreview.net/forum?id=NG7sS51zVF)\.In*The Twelfth International Conference on Learning Representations*\.

## Appendix AAppendix

To substantiate the claim that topological compression can serve as a viable indicator for hallucination detection, we expand upon the RAGognize experimental proxy by providing standard classification metrics\. Furthermore, we deeply analyze the mechanics of our geometric approach compared to established lexical and statistical baselines to explicitly define how the latent topology acts as a structural verifier\.

Baselines Evaluated:

Lexical Overlap \(ROUGE\-L\):A straightforward lexical grounding check measuring the longest common subsequence between the generated response and the retrieved context\. This baseline represents traditional, surface\-level string matching heuristics\.

Token Perplexity \(PPL\):The average negative log\-likelihood of the generated tokens\. This serves as a proxy for the model’s internal statistical confidence, testing whether ungrounded hallucinations correspond to moments of high predictive uncertainty\.

Small\-World Topology \(Ours\):A geometric classification framework operating directly on the latent manifold\. For a given context\-response pair consisting ofNNtotal tokens, we extract the sequence of hidden states at the deepest reasoning layer \(e\.g\., Layer 24 for Qwen2\.5\-1\.5B\), denoted asH=\[h1,h2,…,hN\]∈ℝN×dH=\[h\_\{1\},h\_\{2\},\\dots,h\_\{N\}\]\\in\\mathbb\{R\}^\{N\\times d\}\. We compute the exact token\-wise cosine similarity matrixS∈ℝN×NS\\in\\mathbb\{R\}^\{N\\times N\}such that:

Si,j=hi⋅hj‖hi‖​‖hj‖S\_\{i,j\}=\\frac\{h\_\{i\}\\cdot h\_\{j\}\}\{\\\|h\_\{i\}\\\|\\\|h\_\{j\}\\\|\}
By applying the empirically derived phase\-transition threshold \(τ=0\.81\\tau=0\.81\), we sparsify this continuous space into an unweighted adjacency matrixAArepresenting the latent graphG=\(V,E\)G=\(V,E\), defined as:

Ai,j=\{1if​Si,j≥τ0otherwiseA\_\{i,j\}=\\begin\{cases\}1&\\text\{if \}S\_\{i,j\}\\geq\\tau\\\\ 0&\\text\{otherwise\}\\end\{cases\}
We then map the source entities in the RAG context to a set of anchor nodesVs⊂VV\_\{s\}\\subset Vand the generated claim entities in the response to a set of target nodesVt⊂VV\_\{t\}\\subset V\. A linear Support Vector Machine \(SVM\) is trained on two distinct structural features extracted from this routing: Connectivity Rate \(Cr​a​t​eC\_\{rate\}\) and Average Path Length \(La​v​gL\_\{avg\}\)\.

Connectivity Rate \(Cr​a​t​eC\_\{rate\}\):This feature measures the proportion of valid, unbroken topological paths bridging the context anchors to the generated anchors\. It is formalized using an indicator function𝕀\\mathbb\{I\}, whered⁡\(vs,vt\)d\(v\_\{s\},v\_\{t\}\)is the geodesic distance:

Cr​a​t​e=1\|Vs\|​\|Vt\|​∑vs∈Vs∑vt∈Vt𝕀⁡\(d⁡\(vs,vt\)<∞\)C\_\{rate\}=\\frac\{1\}\{\|V\_\{s\}\|\|V\_\{t\}\|\}\\sum\_\{v\_\{s\}\\in V\_\{s\}\}\\sum\_\{v\_\{t\}\\in V\_\{t\}\}\\mathbb\{I\}\\big\(d\(v\_\{s\},v\_\{t\}\)<\\infty\\big\)
Average Path Length \(La​v​gL\_\{avg\}\):This feature measures the average geodesic distance \(shortest path\) between anchors, effectively counting the discrete semantic hops required to route the factual premise to the generated claim\. To accommodate disconnected graphs, we first define a bounded distance functiond^​\(vs,vt\)\\hat\{d\}\(v\_\{s\},v\_\{t\}\)that penalizes unreachable nodes with a large scalarMMfor SVM normalization:

d^​\(vs,vt\)=\{d⁡\(vs,vt\)if​d​\(vs,vt\)<∞Mif​d​\(vs,vt\)=∞\\hat\{d\}\(v\_\{s\},v\_\{t\}\)=\\begin\{cases\}d\(v\_\{s\},v\_\{t\}\)&\\text\{if \}d\(v\_\{s\},v\_\{t\}\)<\\infty\\\\ M&\\text\{if \}d\(v\_\{s\},v\_\{t\}\)=\\infty\\end\{cases\}We then compute the average path length across all anchor pairs:

La​v​g=1\|Vs\|​\|Vt\|​∑vs∈Vs∑vt∈Vtd^​\(vs,vt\)L\_\{avg\}=\\frac\{1\}\{\|V\_\{s\}\|\|V\_\{t\}\|\}\\sum\_\{v\_\{s\}\\in V\_\{s\}\}\\sum\_\{v\_\{t\}\\in V\_\{t\}\}\\hat\{d\}\(v\_\{s\},v\_\{t\}\)
Table 5:Hallucination Detection Performance on Qwen2\.5\-1\.5B \(RAGognize Test Set\)\. The topological approach significantly outperforms standard lexical and statistical confidence metrics\.Comparative Analysis of Detection Mechanisms:The results in Table[5](https://arxiv.org/html/2608.17950#A1.T5)reveal fundamental flaws in traditional detection heuristics, while highlighting the robustness of latent topology\. ROUGE\-L severely underperforms \(AUROC 0\.68\) because it is highly vulnerable to the RAGognize dataset’s complex configurations; it often falsely flags highly abstractive, correct reasoning as hallucinated due to low string overlap, while being easily fooled by models that verbatim repeat irrelevant distractor text\. Similarly, Token Perplexity \(AUROC 0\.74\) fails to serve as a reliable indicator because modern, heavily instruction\-tuned LLMs frequently hallucinate with extreme statistical confidence\. The autoregressive probability of a generated token does not inherently correlate with its factual grounding in the retrieved prompt\.

In contrast, our Small\-World Topology \(AUROC 0\.89\) bypasses both surface\-level syntax and autoregressive probability, measuring instead thephysical integrity of the semantic routing\. If an LLM fabricates a claim, the generated tokens conceptually detach from the provided RAG context\. Because the hallucinated entity has no latent proximity to the distractor context, it results in a fractured latent graph whereCr​a​t​e→0C\_\{rate\}\\to 0andLa​v​g→∞L\_\{avg\}\\to\\infty\. The SVM linearly separates these fractured, disconnected topologies from the tightly bound \(≤6\\leq 6hops\), highly connected networks of factually grounded responses\. Ultimately, this demonstrates that for an output to be factual, the model must mathematically “prove” its generation by constructing a traversable latent bridge back to the source context; without this bridge, the generation is structurally identifiable as a hallucination\.

Threshold Sensitivity Analysis:We additionally evaluated the sensitivity of the detection AUROC to the chosen Cosine Similarity threshold \(τ\\tau\)\. Consistent with our phase transition findings in Section 4, detection capability remains near random chance \(AUROC≈0\.55\\approx 0\.55\) at low thresholds \(τ<0\.60\\tau<0\.60\) where the graph is overly dense and noisy\. In these states, spurious connections artificially link hallucinated concepts to the context, blinding the classifier\. The AUROC peaks sharply exactly at the phase boundary \(τ∈\[0\.78,0\.83\]\\tau\\in\[0\.78,0\.83\]\), achieving a maximum of 0\.89 atτ=0\.81\\tau=0\.81, before degrading rapidly at higher thresholds \(τ\>0\.85\\tau\>0\.85\) as the graph fractures entirely for both factual and hallucinated states alike\. This confirms that the efficacy of the detection mechanism is not arbitrary, but fundamentally tied to isolating the critical Small\-World phase transition of the latent manifold\.