attention-metrics

This paper critiques standard attention metric methods in transformers, revealing that choices like keeping or dropping sink tokens can reverse conclusions, and proposes using compositional data analysis to separate sink and content attention components for more accurate interpretation.