rdma

AWS introduces a push-based NIXL connector in vLLM that moves KV cache transfer coordination off the critical path for disaggregated prefill-decode inference, delivering up to 3x better TTFT and 30% better TPOT on EC2 P5en instances with EFA.

苹果发布新版 MLX 框架，展示如何在单台 Mac Studio 上运行 Muse Spark Open（近万亿参数）等前沿模型，并通过 RDMA over Thunderbolt 驱动在多台 Mac 间实现分布式张量/流水线并行推理，在四台 M5 Ultra 集群上达到每秒 36 tokens 以上的解码速度。

vLLM 推出基于推送（push-based）的 NIXL KV 缓存传输机制，将传输协调与准备工作移出关键路径，在分离式预填充/解码推理中最高可将 TTFT 改善 3 倍、TPOT 改善 30%。该功能在 AWS Trainium/EFA 集群上开发验证后，作为加速器无关特性贡献给 vLLM。

ByteDance and Fudan University researchers present ByteExpress, a software communication layer over RDMA for large-scale parameter-server inference and training, reporting up to 74% lower P99 push latency and significant CPU savings versus TCP in production deployments.

在 PyTorch Conference North America 2026 上，AMD 团队将展示 MoRI 通信栈与 vLLM 结合，实现跨 pod 的宽专家并行（Wide Expert Parallelism）以及基于 RDMA 的 KV 缓存传输，用于 AMD Instinct GPU 上的分离式 MoE 服务。

The authors have created a lightweight, performant asynchronous reinforcement learning stack in pure JAX, sharing a work log with insights on inference, RDMA weight transfer, memory optimizations, and sharding for scaling RL systems.

The article reviews the MikroTik CRS804-4DDQ-HRM, a cost-effective desktop 400GbE switch with four QSFP56-DD ports and 1.6Tbps switching capacity, emphasizing its value for high-performance networking environments.

Exo labs claims to achieve 4.8 tb/s memory bandwidth through clustering Mac Studios using their RDMA solution, which could significantly boost AI inference performance. This Reddit discussion explores the implications for users considering hardware setups for AI workloads.

NVIDIA introduces ModelExpress, a tool that accelerates distribution of model artifacts across GPU clusters by using GPU-to-GPU RDMA transfers and optimized streaming from storage, reducing model startup times from 8 minutes to under 2 minutes for large models like DeepSeek-V4 Pro.

A setup guide for using a custom Docker/Podman toolbox with ROCm/RCCL RDMA support to cluster two AMD Strix Halo nodes, enabling vLLM with tensor parallelism across 256GB unified memory.

Experimental implementation of RDMA over USB4 demonstrated using Thunderbolt and Strix Halo, potentially enabling high-speed data transfer with any USB4 host.

DualPath is a system that breaks the storage bandwidth bottleneck in agentic LLM inference by introducing a dual-path KV-cache loading mechanism, improving throughput by up to 1.87x offline and 1.96x online.

The author shares a technical deep dive into saturating NIC and disk bandwidth using RDMA and io_uring, highlighting how an AI agent achieved the optimization with a flawed explanation, then details the actual debugging process.

A user proposes building a heterogeneous AI cluster using Blackwell GPUs and high-memory servers connected via RDMA, seeking collaboration on Tinygrad driver development.

NVIDIA's Spectrum-X Ethernet fabric with Multipath Reliable Connection (MRC) sets a new standard for gigascale AI networking, improving throughput, load balancing, and resilience for large-scale AI training, as demonstrated by OpenAI, Microsoft, and Oracle.

MLX's RDMA-over-Thunderbolt implementation for macOS is now available as a standalone library, enabling high-speed Mac clusters for local AI workloads.

A demo shows Qwen3.6 35B vision model running across two M5 Max MacBook Pros connected via RDMA over Thunderbolt 5, achieving near-instant responses with prefix caching. The model correctly identifies Apple Park but misidentifies a person in the image.