Cell-Based Representation of Relational Binding in Language Models
Summary
Study reveals that LLMs encode discourse-level relational binding through Cell-based Binding Representation (CBR), a low-dimensional linear subspace where each cell maps to entity-relation pairs, providing causal evidence for how models track entities and relations.
View Cached Full Text
Cached at: 04/22/26, 08:30 AM
# Cell-Based Representation of Relational Binding in Language Models Source: [https://arxiv.org/abs/2604.19052](https://arxiv.org/abs/2604.19052) [View PDF](https://arxiv.org/pdf/2604.19052) > Abstract:Understanding a discourse requires tracking entities and the relations that hold between them\. While Large Language Models \(LLMs\) perform well on relational reasoning, the mechanism by which they bind entities, relations, and attributes remains unclear\. We study discourse\-level relational binding and show that LLMs encode it via a Cell\-based Binding Representation \(CBR\): a low\-dimensional linear subspace in which each \`\`cell'' corresponds to an entity\-\-relation index pair, and bound attributes are retrieved from the corresponding cell during inference\. Using controlled multi\-sentence data annotated with entity and relation indices, we identify the CBR subspace by decoding these indices from attribute\-token activations with Partial Least Squares regression\. Across domains and two model families, the indices are linearly decodable and form a grid\-like geometry in the projected space\. We further find that context\-specific CBR representations are related by translation vectors in activation space, enabling cross\-context transfer\. Finally, activation patching shows that manipulating this subspace systematically changes relational predictions and that perturbing it disrupts performance, providing causal evidence that LLMs rely on CBR for relational binding\. ## Submission history From: Qin Dai \[[view email](https://arxiv.org/show-email/9cae4248/2604.19052)\] **\[v1\]**Tue, 21 Apr 2026 03:58:47 UTC \(16,089 KB\)
Similar Articles
Tracing Relational Knowledge Recall in Large Language Models
Researchers trace how LLMs recall relational facts by probing per-head attention contributions, showing these are strong linear features whose fidelity correlates with relation specificity and entity connectedness.
How can embedding models bind concepts?
This paper investigates why CLIP struggles with concept binding, showing that while CLIP's binding function is high-complexity, controlled transformer models can learn low-complexity binding functions through multiplicative interactions that generalize better.
Causal Probing for Internal Visual Representations in Multimodal Large Language Models
This paper proposes a causal framework for probing internal visual representations in Multimodal Large Language Models, revealing differences in how entities and abstract concepts are encoded. The study highlights that increasing model depth is crucial for encoding abstract concepts and uncovers a disconnect between perception and reasoning in current MLLMs.
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs
This paper introduces Layer-wise Representation Dynamics (LRD), a framework with three measurement families to analyze how hidden states change across layers in language models. Applied to 31 models on 30 MTEB tasks, LRD reveals architectural differences and enables label-free model selection and inference-time layer pruning.
When Should Models Change Their Minds? Contextual Belief Management in Large Language Models
This paper introduces Contextual Belief Management (CBM) for LLMs to handle long-term information, proposes the BeliefTrack benchmark for evaluation, and demonstrates that reinforcement learning and representation-level steering significantly reduce belief management failures.