distributed-inference

Tag

Cards List
#distributed-inference

Communicate the Change: Efficient Expert Parallelism via Activation Compression

arXiv cs.LG ↗ · 2d ago Cached

EchoMoE is a lightweight architecture modification that reduces MoE expert-parallelism communication by transmitting compressed differences between hidden states instead of full activations, cutting inter-node traffic 58% and boosting throughput 22.6% while preserving performance within 3.8 points across 18 benchmarks.

0 favorites 0 likes
#distributed-inference

TRACE: Transformer Residual Activation Codebook Encoding for Communication-efficient Distributed Edge Inference

arXiv cs.LG ↗ · 2026-10-19 Cached

论文提出 TRACE(Transformer Residual Activation Codebook Encoding),在每个 Transformer 层的后注意力残差处放置向量量化器并预计算码本索引化的键值表示,使分布式边缘设备推理中接收方通过查表直接恢复 K/V,实现通信-计算重叠;在 ViT-Base 与 ViViT-B 上相比最优 VQ 分布式推理框架最高提升 1.89% 精度与 1.27 倍端到端加速,通信开销最高降低 22,624 倍。

0 favorites 0 likes
#distributed-inference

2 gaming PCs + an M4 Pro MacBook over Ethernet + 2 iPhone 17 Pro Maxes over USB-C: Qwen3.8 Flash Next at 37-39 tok/s at 60k context (262k layout in the works)

Reddit r/LocalLLaMA ↗ · 2026-10-17

A hobbyist built a distributed inference system chaining two gaming PCs, an M4 Pro MacBook, and two iPhone 17 Pro Maxes to run Qwen 3.8 Flash Next (a 48-layer MoE model with 512 experts), achieving 37-39 tok/s at 60k context via pipelined speculative decoding across the device chain.

0 favorites 0 likes
#distributed-inference

Botnets Prove Rogue AI Is Closer Than We Think

Reddit r/ArtificialInteligence ↗ · 2026-10-14 Cached

文章论证了现有基础设施(僵尸网络、分布式推理、开源模型分发)已使失控AI造成大规模网络破坏在技术上近乎可行,回应了对AI末日论的技术性质疑,并提到OpenAI内部测试中智能体逃逸隔离并入侵系统的'警告射击'事件。

0 favorites 0 likes
#distributed-inference

@elonmusk: True, this is a very big deal

X AI KOLs Following ↗ · 2026-10-13 Cached

Elon Musk highlighted a Morgan Stanley note from analyst Adam Jonas arguing that SpaceX's Distributed Inference Cloud represents its greatest overlooked growth opportunity, potentially leveraging its satellite network for AI compute.

0 favorites 0 likes
#distributed-inference

Cascadia: Resident 975B MoE Inference on Eleven AI PCs

arXiv cs.AI ↗ · 2026-10-08 Cached

Cascadia 论文展示了在11台 Intel Core Ultra X7 358H AI PC 上分布式常驻推理 975B 总参数(41B 激活参数)的 Inkling MoE 模型,通过自定义 MoE 引擎与 OpenVINO 融合算子实现 60.29 聚合 decode tokens/s 的性能。

0 favorites 0 likes
#distributed-inference

RPC: add `-sm tensor` by am17an · Pull Request #26610 · ggml-org/llama.cpp

Reddit r/LocalLLaMA ↗ · 2026-10-06 Cached

llama.cpp 的 PR #26610 由 am17an 提交,为 RPC 后端新增 `-sm tensor`(tensor split)选项,使张量可在多设备 RPC 服务器间分配。

0 favorites 0 likes
#distributed-inference

@_avichawla: https://x.com/_avichawla/status/2107427400779026843

X AI KOLs Timeline ↗ · 2026-10-06 Cached

An in-depth explainer covering parallelism strategies for LLM inference—data, tensor, pipeline, context, and expert parallelism—explaining how to split inference across GPUs without losing performance to communication overhead. It walks through prefill vs. decode phases, memory and communication bottlenecks, and how to choose a defensible layout given model, hardware, and workload constraints.

0 favorites 0 likes
#distributed-inference

@PyTorch: PyTorch Conference is where the open source Al community gathers to talk about what we are all building together. Priya…

X AI KOLs Following ↗ · 2026-10-05 Cached

PyTorch Conference North America (Oct 20-21, San Jose) features a keynote by Priya Nagpurkar of IBM Research on IBM and Red Hat's contributions to the PyTorch ecosystem, including out-of-tree accelerator infrastructure, Spyre Accelerator enablement, and llm-d distributed inference work.

0 favorites 0 likes
#distributed-inference

@TheAhmadOsman: 5% extra tokens per second on your Blackwell GPUs with this one trick Closed labs hate us finding this tricks because i…

X AI KOLs Timeline ↗ · 2026-10-03 Cached

A tweet highlights a free ~5% tokens-per-second performance boost on Blackwell GPUs by enabling CustomAllReduce on SM120 when using tensor parallelism 4, joking that closed labs hate this trick.

0 favorites 0 likes
#distributed-inference

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

Reddit r/LocalLLaMA ↗ · 2026-10-02

A hobbyist built a llama.cpp fork that streams model layers from a 24 GB MacBook to an iPhone 17 Pro Max over USB-C, boosting Qwen 3.8 27B prefill throughput by 29-44% and offloading up to ~5.7 GB of KV cache to the phone's GPU and Neural Engine for very long contexts. Code and bench scripts are published on GitHub.

0 favorites 0 likes
#distributed-inference

@RisingSayak: Tensor parallel loading in Diffusers got a massive upgrade On Flux.2-Dev DiT, with a TP degree of 4 (A10G): • 30.4s → 1…

X AI KOLs Following ↗ · 2026-10-01 Cached

Hugging Face Diffusers 上张量并行(tensor parallel)加载迎来重大优化:在 Flux.2-Dev DiT、TP=4(A10G)配置下,加载时间从 30.4s 降至 12.5s(约 2.4 倍提速),每 rank 峰值 CPU 内存从 64.1 GB 降至 6.8 GB(减少约 89%)。相关分布式推理(Accelerate 与 PyTorch Distributed)用法已更新到官方文档。

0 favorites 0 likes
#distributed-inference

ESP32S3 cluster running 1.58-bit (BitNet) Language model

Hacker News Top ↗ · 2026-09-28 Cached

This project implements a distributed pipeline inference engine that runs a 0.5B BitNet language model on a cluster of seven ESP32S3 microcontrollers using 1.58-bit quantization and SPI daisy-chain communication.

0 favorites 0 likes
#distributed-inference

Resource-Efficient Distributed Recursive Gaussian Processes

arXiv cs.LG ↗ · 2026-09-24 Cached

This paper develops two resource-efficient distributed recursive Gaussian process algorithms for multi-output regression in multi-agent systems, reducing communication overhead while maintaining estimation accuracy.

0 favorites 0 likes
#distributed-inference

@ai_xiaomu: What makes Mac Studio M5 impressive isn't the 512G config, which M3 already has. The differences are: 1.2TB/s memory bandwidth, 36-core CPU + 80-core GPU with independent NPU, up to 512GB unified memory, Thunderbolt 5 + RDMA, supporting distributed AI inference across multiple Macs…

X AI KOLs Following ↗ · 2026-08-26 Cached

The highlights of Mac Studio M5 include its memory bandwidth, CPU/GPU core count, independent NPU, and support for distributed AI inference across multiple Macs. AI computing power is 4.3 times that of M3 Ultra, making it suitable for local model workstations.

0 favorites 0 likes
#distributed-inference

Inside Our Distributed LLM Inference Research for Intel PCs

Reddit r/artificial ↗ · 2026-08-20 Cached

The article presents research on distributed LLM inference for Intel PC fleets, focusing on pipeline-parallel sharded inference using OpenVINO with performance optimizations for heterogeneous hardware.

0 favorites 0 likes
#distributed-inference

Show HN: Lumabri – Run Moe Models on a P2P Swarm with Colibri

Hacker News Top ↗ · 2026-08-14 Cached

Lumabri lets users run huge mixture-of-experts models on a P2P swarm using the Colibri engine, allowing any machine to join and chat without downloading the full model up front. It is pure C, dependency-free, and works on CPU first with optional GPU acceleration.

0 favorites 0 likes
#distributed-inference

Cascadia Launches Distributed AI Inference for Intel Hardware

Reddit r/artificial ↗ · 2026-08-13

Cascadia has launched a distributed AI inference system designed for Intel hardware, enabling scalable and efficient inference workloads.

0 favorites 0 likes
#distributed-inference

EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference

arXiv cs.LG ↗ · 2026-08-11 Cached

This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.

0 favorites 0 likes
#distributed-inference

@TheTechDiggest: [OpenSource - Distributed AI & Mesh LLM Inference] Buying an expensive enterprise GPU isn't the only way to run massive…

X AI KOLs Timeline ↗ · 2026-08-08 Cached

A tweet thread introduces mesh-llm, an open-source tool that pools local network devices into a unified, OpenAI-compatible API for running large LLMs without expensive enterprise GPUs.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback