Tag
This project implements a distributed pipeline inference engine that runs a 0.5B BitNet language model on a cluster of seven ESP32S3 microcontrollers using 1.58-bit quantization and SPI daisy-chain communication.
This paper develops two resource-efficient distributed recursive Gaussian process algorithms for multi-output regression in multi-agent systems, reducing communication overhead while maintaining estimation accuracy.
The highlights of Mac Studio M5 include its memory bandwidth, CPU/GPU core count, independent NPU, and support for distributed AI inference across multiple Macs. AI computing power is 4.3 times that of M3 Ultra, making it suitable for local model workstations.
The article presents research on distributed LLM inference for Intel PC fleets, focusing on pipeline-parallel sharded inference using OpenVINO with performance optimizations for heterogeneous hardware.
Lumabri lets users run huge mixture-of-experts models on a P2P swarm using the Colibri engine, allowing any machine to join and chat without downloading the full model up front. It is pure C, dependency-free, and works on CPU first with optional GPU acceleration.
Cascadia has launched a distributed AI inference system designed for Intel hardware, enabling scalable and efficient inference workloads.
This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.
A tweet thread introduces mesh-llm, an open-source tool that pools local network devices into a unified, OpenAI-compatible API for running large LLMs without expensive enterprise GPUs.
Chinese researchers have developed an all-optical interconnect system that links standard electronic chips, boosting AI distributed inference speeds by over 100 times while using just one-ninth of typical computational resources. The breakthrough, published in National Science Review, uses silicon photonic transceiver chips and FPGAs to achieve dramatic efficiency gains.
Antirez announces high probability of merging a branch implementing GLM 5.2 in DwarfStar, which could become the best model for 512GB Mac Studio and potentially run on distributed 128GB MacBooks with 2-bit quantization.
A researcher debuted Shard, achieving 30 tok/s inference on a 744B parameter model distributed across 6 consumer GPUs over the open internet, a 15-20x improvement over previous methods.
A pull request to vLLM adds support for tensor parallelism degree 3 for MiniMax M3 with its NVFP4 quantization, enabling the model to run on 3x DGX Sparks with 87GB memory each.
vLLM integrates Mooncake Store for distributed KV cache reuse, enabling cross-node prefix caching to efficiently serve agentic workloads with high token reuse.
A tweet recommending a paper that is described as the bible of distributed inference.
A blog post guides readers through setting up a Raspberry Pi cluster for distributed training and inference, part of a series aimed at making distributed AI accessible using affordable hardware.
antirez announces receiving an M5 Max 128GB MacBook Pro from audreyt to develop DwarfStar4 and experiment with distributed inference across M3 Max and M5 Max hardware.
Federation of Experts (FoE) restructures mixture-of-experts blocks into clusters that process KV heads independently, eliminating inter-node communication bottlenecks and improving inference throughput and latency by up to 5.2x while maintaining generation quality.
A user shares their $25k hardware setup of two 512GB RAM M3 Ultra Mac Studios for running large language models locally, having tested DeepSeek V3 Q8 and GLM 5.1 Q4 via the exo distributed inference backend, while awaiting Kimi 2.6 MLX optimization.