Tag
This paper proposes a manifold-aware encoding strategy for general coded computing that exploits the intrinsic low-dimensional geometry of high-dimensional data to improve straggler resilience in distributed systems. Experiments show reduced mean squared recovery error in neural network inference and polynomial evaluation tasks.
PyTorch 2.14 introduces updates to Inductor, PyTorch Distributed, dynamic shapes, Apple Silicon support, and more, with a live Q&A and PyTorchCon announcement.
This paper introduces General Coded Computing (GCC), a learning-theoretic framework for mitigating stragglers in distributed computing systems, providing theoretical performance guarantees and experimental validation on deep neural networks.
An interactive, explorable explanation of various parallelization schemes for training transformers, adapted from academic content on scaling models.
This article explores the limitations of Pubsub systems, focusing on challenges in distributed computing environments.
apra-fleet is a tool that lets you run a fleet of AI agents across your machines.
Running coding agents across multiple machines reveals that managing the control plane is more challenging than the agents themselves, offering insights into distributed agent orchestration.
A technical note on performing RL post-training across 14 Macs distributed in 4 countries, highlighting distributed compute for reinforcement learning.
Prime Intellect engineers demonstrated a method to train reasoning models in 30 minutes using distributed RL over the open internet, utilizing Prime-RL, LLM judges, and multi-cloud GPUs, enabling open models to compete with closed labs without owning data centers.
Mesh LLM is a distributed AI computing platform that pools idle GPUs across multiple machines to run large language models, exposing a single OpenAI-compatible API. It leverages iroh's peer-to-peer networking to enable private, decentralized inference without a central server.
The article revisits the SETI@home distributed computing model as a potential solution to alleviate the demand for new database centers by utilizing idle office and home PCs for processing tasks.
This paper investigates distributed sketching for OLS regression, where sketches are built from partitioned subsets rather than the whole dataset, reducing computational cost. The authors characterize the exact excess loss of the averaged estimator and show it matches that of whole-data sketching when subset covariance divergence is small.
Ray 2.56 has been released with improvements to Ray Data, Ray Serve for LLMs, GPU-domain-aware placement groups, and Kubernetes integration.
This paper presents a programmable probabilistic computer with one million p-bits by networking FPGAs, achieving Gibbs sampling at over a trillion flips per second for Ising models while introducing a design rule for scaling beyond single-chip limits.
A personal account of hunting for million-digit prime numbers using a home computer setup.
A discussion explores whether AI training could be decentralized like Bitcoin mining, with participants contributing GPU resources to train open-source models in exchange for tokens, raising questions about verification, fake gradients, and efficiency.
A retrospective on the eight fallacies of distributed computing, originally formulated by Sun Microsystems engineers, examining their continued relevance 21 years later for network operators and developers.
Microsoft AI announces MAI-Thinking-1, a 35B active/1T total MoE reasoning model competitive on STEM and coding tasks, developed using Ray for distributed training and orchestration.
A blog post summarizing a panel at the Dutch-Belgian DataBase Day, which discussed practical data systems problems often ignored by academic research, such as variable-length string handling, unrealistic benchmarks, and networking challenges.
Snowflake now supports job-based batch inference powered by Ray, enabling distributed GPU execution for scaling model inference over millions of unstructured datapoints with a single API call.