Tag
The paper investigates why Graph Transformers rarely form attention sinks compared to LLMs, showing that the attention scoring function — not graph structure — is the main cause: replacing GRIT's gated logit with dot-product attention raises its sink rate from 0.09 to 0.68.
This paper investigates the life cycle of massive activations in transformers, showing that weight decay causally regulates their scale — with an AdamW-preconditioned growth vs. decay balance model predicting peak magnitudes scaling as λ^{-1/2}.
The paper introduces StreamingLLM, a framework that uses attention sinks to enable large language models to handle infinite sequence lengths without fine-tuning, improving efficiency in streaming applications.
This paper demonstrates that attention sinks, representation collapse, and norm stratification are not unique to attention mechanisms but are general consequences of content-based routing under a norm-blind similarity metric, as shown across multiple architectures including transformers, graph attention, state-space models, and recurrent mixers.
This paper analyzes how large language models internally process graph tokens in Graph Language Models (GLMs), finding a decoupling between activation-level saliency and graph-semantic utility. Graph sink tokens emerge as activation outliers but are not the primary carriers of graph structure, revealing limitations in current graph-token construction and alignment mechanisms.
This paper identifies the 'Massive Emergence Layer' where extreme activations in LLMs originate and propagate, proposing a method to mitigate their rigidity and improve model performance on tasks like math reasoning and instruction following.