context-parallelism

该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈，提出按并行 rank 分片 MLA KV 存储、异步加载等优化，将 p99 延迟从 83.32 秒降至 0.67 秒，并显著提升预填充吞吐。

FlashMask-3 introduces a Hopper-optimized mask-aware sparse attention kernel and zero-communication distributed mask localization for long-context training, achieving major TFLOPs and throughput gains over FlashMask, FlexAttention, MagiAttention, and Megatron-Core baselines, including 7.3-11.0% end-to-end speedup on a 118B MoE trained on 128 H100 GPUs at 128K context.

Design-CP introduces context-parallel inference strategies for RFdiffusion 3 that enable the all-atom design of large multimeric protein nanoparticles by distributing quadratic activations across multiple GPUs, making large-assembly protein design feasible on smaller GPU clusters.