MiA-Signature: Approximating Global Activation for Long-Context Understanding
Summary
This paper introduces MiA-Signature, a compressed representation of global activation patterns in LLMs designed to improve long-context understanding. It proposes using submodular-based selection to approximate full activation states, yielding performance gains in RAG and agentic systems.
View Cached Full Text
Cached at: 05/08/26, 07:17 AM
Paper page - MiA-Signature: Approximating Global Activation for Long-Context Understanding
Source: https://huggingface.co/papers/2605.06416
Abstract
Researchers propose a compressed representation method for global activation patterns in large language models that approximates full activation states while maintaining computational efficiency and improving performance in long-context tasks.
A growing body of work in cognitive science suggests that reportable conscious access is associated with global ignition over distributed memory systems, while such activation is only partially accessible as individuals cannot directly access or enumerate all activated contents. This tension suggests a plausible mechanism that cognition may rely on a compact representation that approximates the global influence of activation on downstream processing. Inspired by this idea, we introduce the concept ofMindscape Activation Signature(MiA-Signature), a compressed representation of the global activation pattern induced by a query. In LLM systems, this is instantiated viasubmodular-based selectionofhigh-level conceptsthat cover the activated context space, optionally refined through lightweight iterative updates usingworking memory. The resulting MiA-Signature serves as a conditioning signal that approximates the effect of the full activation state while remaining computationally tractable. Integrating MiA-Signatures into bothRAGandagentic systemsyields consistent performance gains across multiplelong-context understandingtasks.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.06416
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.06416 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.06416 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.06416 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
The paper introduces MISA, a method that applies a mixture-of-experts approach to the indexer heads in sparse attention mechanisms, significantly reducing computational costs for long-context LLM inference while maintaining performance.
LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
LongAct proposes a saliency-guided sparse update strategy for improving long-context reasoning in LLMs by selectively updating weights associated with high-magnitude activations in query and key vectors, achieving ~8% improvement on LongBench v2.
@rwayne: Yesterday an interesting paper dropped on arXiv that directly translates the 'consciousness' mechanism from cognitive science into long-context engineering.
Researchers propose applying the "global ignition" consciousness mechanism from cognitive science to long-context engineering, introducing the MiA-Signature method that uses submodular selection of high-level concepts to cover the activation space. Applied to RAG and agentic systems, it delivers consistent performance improvements across multiple long-context tasks.
Unsupervised Features Mining via Activation Geometry
This paper introduces Mining via Activation Geometry (MAG), an unsupervised framework that extracts reasoning features from LLM activations using natural-language instructions, enabling activation steering and effective training data selection for classifier probes.
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention [P]
Sebastian Raschka reviews recent innovations in LLM architectures focused on long-context efficiency, including KV sharing, compressed convolutional attention, and layer-wise attention budgeting from models like Gemma 4, ZAYA1, Laguna XS.2, and DeepSeek V4.