Tag
OpenAI researcher Noam Brown discusses how multi-agent systems scale test-time compute through parallelization, highlights an experiment with 10,000 agents, and stresses the critical role of the underlying general model in driving AI progress.
This article explains concurrency and threads in inference engineering, covering multi-threading, parallelization, and race conditions with visual examples.
ConvMem is a training-free, parallelizable framework that reformulates long-context reasoning in large language models as hierarchical convolution to improve efficiency, avoid overfitting, and outperform baseline methods.
An interactive, explorable explanation of various parallelization schemes for training transformers, adapted from academic content on scaling models.
Lucen is a source-to-source Python compiler that parallelizes for-loops via comment pragmas, guaranteeing bit-identical results and safe fallbacks to sequential execution when parallelism cannot be proven safe.
Anthropic's Ultracode is a highly efficient token-burning coding agent that excels at parallelized subagent-based dynamic workflows, though it requires proper repo setup to fully leverage its capabilities.
Summary of Lecture 19 on efficient AI distributed training, covering data, pipeline, tensor, and sequence parallelism methods with notes on memory and communication bottlenecks.
This paper introduces a parallelization strategy and adaptive steering mechanism for the Baymex algorithm to efficiently learn discretized Bayesian network classifiers for clinical data, achieving speedups over 54x on a 16-core CPU and comparable or better predictive performance than traditional models while maintaining explainability.
The article describes five key workflow patterns for building agentic AI systems in enterprise settings, as summarized by Anthropic: prompt chaining, routing, parallelization, orchestrator, and evaluator-optimizer, with tips to prefer simpler workflows before using full agents.
This paper presents a parallelization framework for CFR algorithms using linear algebra operations, achieving up to four orders of magnitude speedup on GPU compared to CPU implementations.
OpenAI researchers discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training across a wide range of tasks. They found that more complex tasks and more powerful models tolerate larger batch sizes, suggesting future AI systems can scale further through increased parallelization.