full-attention

Alibaba's Tongyi Lab introduces HydraHead, a research paper proposing a new attention hybridization architecture that fuses Full Attention and Linear Attention at the head level for more efficient long-context models.

HydraHead is a novel attention hybridization architecture that combines Full and Linear Attention at the head level, achieving superior long-context performance with reduced training overhead via interpretability-driven selection and scale-normalized fusion.