@lateinteraction: guess what NVIDIA used here for an "attention-based encoder-decoder to retrieve directly from its own internal represen…
Summary
NVIDIA utilized late interaction, a form of sparse attention, for an attention-based encoder-decoder to retrieve directly from internal representations.
View Cached Full Text
Cached at: 05/08/26, 09:56 AM
guess what NVIDIA used here for an “attention-based encoder-decoder to retrieve directly from its own internal representations”?
late interaction is sparse attention https://t.co/oRar8dRIqb
Similar Articles
@lateinteraction: Late-interaction sparse retrieval? With neuron-level inverted indexing, on top of unsupervised sparse autoencoders. Wor…
This paper presents a single-stage sparse coding method using unsupervised sparse autoencoders and natural inverted indexing to accelerate multi-vector retrieval, outperforming traditional k-means based approaches.
@h100envy: CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $12…
A CMU PhD who developed the kernels now used by NVIDIA in TensorRT-LLM explains fast attention, covering fused CUDA kernels, FlashInfer, Triton, and paged-KV attention, enabling more tokens per second on the same GPU.
@yifeiwang77: Thanks for sharing our work @lateinteraction @sum! The idea is extremely simple: - multi-vector retrieval is so costly …
The author shares their work on reducing the cost of multi-vector retrieval by using k-means as top-1 sparse coding. Omar Khattab adds that late-interaction sparse retrieval with neuron-level inverted indexing on unsupervised sparse autoencoders works well.
@ekzhang1: me looking at people like this guy who write real gpu kernels :)
AI model Claude was used to write a FlashAttention forward kernel using the pyptx DSL, achieving near-parity performance with hand-tuned FlashAttention-4 on NVIDIA B200 hardware.
@_avichawla: NVIDIA researchers built a new transformer variant. One small change to the layers made: - decoding 1.7x faster - long-…
NVIDIA researchers introduced SparDA, a transformer variant that adds a fourth projection (Forecast) to predict next-layer KV blocks, enabling prefetching from CPU memory and reducing selection cost, achieving 1.7x faster decoding and 6.5 point accuracy gain on long reasoning.