parallelism

A new runtime for the K and Q programming languages that reimagines execution using SIMD, parallelism, fusion, and compression to better leverage modern hardware.

An article arguing that multi-agent setups are often overused, providing guidelines on when to use single vs multi-agent architectures based on task parallelism and coordination needs.

Recommending Harrison Chase's blog post on multi-agent system design. The key insight is that multi-agent systems split into 'reading' and 'writing' tasks; reading is easy to parallelize while writing leads to conflicts. It also summarizes suitable and unsuitable scenarios and engineering prerequisites.

This article explains how batching and parallel operations improve latency and efficiency in AI computer use systems, highlighting open-source implementations like pi-computer-use and cua-driver that achieved significant performance gains before similar features appeared in Codex.

Rob Pike's 2012 talk explains the distinction between concurrency and parallelism, emphasizing that concurrency is about composition of independently executing tasks, while parallelism is about simultaneous execution. The talk is a classic in computer science education.

The author discusses the architecture for scaling AI memory ingestion from thousands to millions of documents, emphasizing orchestration and parallelism over raw compute power, and using Prefect for workflow management.

The author shares that upgrading from single 24GB GPU to dual 24GB GPUs improved LLM performance not through larger models but through parallelism, using a main orchestrator agent with smaller subagents, increasing overall throughput for coding tasks.

A tweet listing 16 inference optimization techniques for achieving sub-second LLM responses, including KV-caching, speculative decoding, FlashAttention, and various parallelism methods.

A detailed roadmap for transitioning from a single AI agent to a coordinated swarm of agents, covering when to split, how to run parallel subagents without conflicts, and how to maintain sanity at scale using Claude Code primitives.

This paper introduces DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that uses generation-axis pipeline parallelism and trainer-assisted generation to improve throughput by 1.56-2.10x over existing systems.

This blog post analyzes the PivCo-Huffman paper, which introduces 'merge' operations for parallel Huffman decoding, enabling efficient vectorized and GPU-friendly decoding without interleaving overhead.

Discussion about rewriting parallelism to improve kernel performance using CuTe DSL and tile programming models for the FA4 (FlashAttention 4) kernel.

Explains that inference kernels differ from training, with Flash Attention 4 focusing on changing parallelism across KV and supporting small irregular loads.

This blog from NVIDIA Research discusses how sequence parallelism can scale long-video training systems for both understanding and generation, addressing the challenge of fitting very long video sequences across multiple GPUs.

The article shares practical lessons for building low-latency, high-throughput AI agents, including workload estimation, token reduction, parallelism, microservices, and handling LLM failures.

Kakuna is a skill that hardens codebases by automating boring tasks, making production-ready commits with audit trails, and encoding opinions on designing apps for both human and agent access, focusing on subagent parallelism and the 'mullet factory' approach.

A learner shares enthusiasm for Stanford CS336 lecture 7 on GPU parallelism, which covers fundamental operations and connects them to multi-GPU setups and parallelism techniques like tensor, data, and pipeline parallelism.

DynaTrain is a distributed training system enabling sub-second online reconfiguration of parallelism for large language models, using a Virtual Parameter Space abstraction to achieve up to three orders of magnitude faster transitions than existing methods.

This paper introduces SNLP, a framework that enables layer-parallel inference for transformers by replacing exact Newton corrections with structured approximations, achieving up to 2.3x speedup on a 0.5B model while improving perplexity.

A technical deep-dive into common causes of failed pretraining runs in large language models, including causality-breaking issues in expert routing and numerical precision bugs, with examples from Llama 4, Gemini 2 Pro, and GPT-4.