Tag
EchoMoE is a lightweight architecture modification that reduces MoE expert-parallelism communication by transmitting compressed differences between hidden states instead of full activations, cutting inter-node traffic 58% and boosting throughput 22.6% while preserving performance within 3.8 points across 18 benchmarks.
论文提出 TRACE(Transformer Residual Activation Codebook Encoding),在每个 Transformer 层的后注意力残差处放置向量量化器并预计算码本索引化的键值表示,使分布式边缘设备推理中接收方通过查表直接恢复 K/V,实现通信-计算重叠;在 ViT-Base 与 ViViT-B 上相比最优 VQ 分布式推理框架最高提升 1.89% 精度与 1.27 倍端到端加速,通信开销最高降低 22,624 倍。
A hobbyist built a distributed inference system chaining two gaming PCs, an M4 Pro MacBook, and two iPhone 17 Pro Maxes to run Qwen 3.8 Flash Next (a 48-layer MoE model with 512 experts), achieving 37-39 tok/s at 60k context via pipelined speculative decoding across the device chain.
文章论证了现有基础设施(僵尸网络、分布式推理、开源模型分发)已使失控AI造成大规模网络破坏在技术上近乎可行,回应了对AI末日论的技术性质疑,并提到OpenAI内部测试中智能体逃逸隔离并入侵系统的'警告射击'事件。
Elon Musk highlighted a Morgan Stanley note from analyst Adam Jonas arguing that SpaceX's Distributed Inference Cloud represents its greatest overlooked growth opportunity, potentially leveraging its satellite network for AI compute.
Cascadia 论文展示了在11台 Intel Core Ultra X7 358H AI PC 上分布式常驻推理 975B 总参数(41B 激活参数)的 Inkling MoE 模型,通过自定义 MoE 引擎与 OpenVINO 融合算子实现 60.29 聚合 decode tokens/s 的性能。
llama.cpp 的 PR #26610 由 am17an 提交,为 RPC 后端新增 `-sm tensor`(tensor split)选项,使张量可在多设备 RPC 服务器间分配。
An in-depth explainer covering parallelism strategies for LLM inference—data, tensor, pipeline, context, and expert parallelism—explaining how to split inference across GPUs without losing performance to communication overhead. It walks through prefill vs. decode phases, memory and communication bottlenecks, and how to choose a defensible layout given model, hardware, and workload constraints.
PyTorch Conference North America (Oct 20-21, San Jose) features a keynote by Priya Nagpurkar of IBM Research on IBM and Red Hat's contributions to the PyTorch ecosystem, including out-of-tree accelerator infrastructure, Spyre Accelerator enablement, and llm-d distributed inference work.
A tweet highlights a free ~5% tokens-per-second performance boost on Blackwell GPUs by enabling CustomAllReduce on SM120 when using tensor parallelism 4, joking that closed labs hate this trick.
A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.
Hugging Face Diffusers 上张量并行(tensor parallel)加载迎来重大优化:在 Flux.2-Dev DiT、TP=4(A10G)配置下,加载时间从 30.4s 降至 12.5s(约 2.4 倍提速),每 rank 峰值 CPU 内存从 64.1 GB 降至 6.8 GB(减少约 89%)。相关分布式推理(Accelerate 与 PyTorch Distributed)用法已更新到官方文档。
This project implements a distributed pipeline inference engine that runs a 0.5B BitNet language model on a cluster of seven ESP32S3 microcontrollers using 1.58-bit quantization and SPI daisy-chain communication.
This paper develops two resource-efficient distributed recursive Gaussian process algorithms for multi-output regression in multi-agent systems, reducing communication overhead while maintaining estimation accuracy.
The highlights of Mac Studio M5 include its memory bandwidth, CPU/GPU core count, independent NPU, and support for distributed AI inference across multiple Macs. AI computing power is 4.3 times that of M3 Ultra, making it suitable for local model workstations.
The article presents research on distributed LLM inference for Intel PC fleets, focusing on pipeline-parallel sharded inference using OpenVINO with performance optimizations for heterogeneous hardware.
Lumabri lets users run huge mixture-of-experts models on a P2P swarm using the Colibri engine, allowing any machine to join and chat without downloading the full model up front. It is pure C, dependency-free, and works on CPU first with optional GPU acceleration.
Cascadia has launched a distributed AI inference system designed for Intel hardware, enabling scalable and efficient inference workloads.
This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.
A tweet thread introduces mesh-llm, an open-source tool that pools local network devices into a unified, OpenAI-compatible API for running large LLMs without expensive enterprise GPUs.