Tag
An overview comparing the architectures of major American open-weight models, highlighting shared design patterns like sliding windows, latent MoE variants, and abundant normalization layers, with a focus on Muse Spark Open's distinctive approach. The analysis notes how differing constraints between American and Chinese open-weight labs lead to divergent architecture and scaling choices.
This paper analyzes the mechanics of long-context hybrid LLMs combining full attention with sliding-window (SWA) or gated linear attention (GLA/GDN), identifying a Seesaw Effect in context extension and proposing Sliding-Window Linear Attention that achieves 16x training-free length extrapolation with 100% NIAH-SK1 accuracy at 64k context.
A blog post explaining the simple two-stack algorithm for maintaining sliding-window aggregations (min, quantiles, HyperLogLog, etc.) with amortized O(1) operations, which works for associative aggregations without inverses, and tracing the algorithm's folklore origins back through Tangwongsan et al. and D. Sleator's lecture notes.
Prefix Sliding reduces memory costs during long reasoning by discarding unimportant intermediate tokens, enabling efficient test-time scaling without retraining, achieving up to 3x speedup in existing models.
GLIDE introduces a layer-wise adaptive mechanism that strategically integrates sliding-window softmax attention with linear recurrent aggregation for efficient LLM inference, reducing KV cache I/O and latency for long contexts without compromising quality.
This paper proposes a sliding-window-based reinforcement learning framework (SWRL) for end-to-end online scheduling in dynamic assembly flow shop scheduling with complex kitting constraints, demonstrating consistent tardiness reductions over classical dispatching rules and existing deep reinforcement learning methods on real-world instances.
This paper proposes Kara, a sliding-window KV cache compression method for efficient serving of reasoning LLMs, addressing limitations in existing compression techniques by using bidirectional attention and a Token2Chunk module. The method is integrated into the KvLLM inference framework built upon vLLM, improving output throughput while maintaining performance.
Tensor Cache introduces a two-level caching mechanism that compresses evicted key-value pairs from sliding-window attention into a fixed-size associative memory, improving long-context language modeling without unbounded memory growth.