depth-wise-routing

This paper introduces Low-Rank Attention Residuals (LR-AttnRes) for LLMs, which decouple routing from representation by using low-dimensional keys for depth-wise attention, improving performance while reducing FLOPs.