I built ALHR: A tree based sparse attention system that achieves sub-quadratic inference while retaining accuracy. [P]

Reddit r/MachineLearning Papers
sparse-attention efficient-inference transformers kv-cache-compression hierarchical-routing memory-efficiency

Summary

作者介绍 ALHR（Adaptive Learnable Hierarchical Routing），一种基于静态二叉树与可学习函数的稀疏注意力系统，在 MQAR 测试中每查询仅读取约 30 个键（相比稠密注意力的 512 个），实现约 35.3 倍 KV 压缩，同时保持接近的准确率，并将推理复杂度降至 NlogN、显存线性扩展。

ALHR - Adaptive Learnable Hierarchical Routing, uses static binary trees and learnable functions to minimize the amount of keys to be read. It does use a dense teacher while phase 1 of training however. MQAR TEST AT 1024 TOKENS - Average keys read per query by dense - 512 Keys Average keys read per query by ALHR - 30 keys Top - 1 accuracy of dense - 94.9% Top - 1 accuracy of ALHR - 92.1% KV Compression of dense - 1x(100% read) KV Compression of ALHR - 35.3x(2.83% read) Peak VRAM of dense - 57 MB (Scales quadratically) Peak VRAM of ALHR - 422 MB (scales linearly) Cache compression of ALHR - 100% The true log and Kaggle cell used to run it are in the logs folder in the repo limitations: Full scale tests are still not completed, The training of this model would still be quadratic but the inference would be NlogN (as indicated in the logs in the repo) Would love your opinions ALHR Repository: https://github.com/vdev-ctrl/Adaptive-Learnable-Hierarchical-Routing-
Original Article

Similar Articles

Subquadratic AI introduces SubQ-1.1-Small, a new model using Smart Sparse Attention

Reddit r/singularity

Subquadratic AI introduces SubQ-1.1-Small, a model leveraging Smart Sparse Attention to achieve near-perfect long-context retrieval up to 12M tokens with up to 1,000x attention compute reduction. It balances long-context optimization with strong general reasoning, outperforming baselines on benchmarks like NIAH and RULER.