scalability

OpenAI built Pando, a new horizontally scalable Git backend for its monorepo to handle agent-driven development at scale, cutting median push latency by nearly 5x and serving nearly a petabyte of Git data per day while staying compatible with the Git protocol.

A practitioner asks the community how to handle AI image and video content moderation at scale (1M+ daily uploads), seeking recommendations on open-source models, commercial APIs like Sightengine and AWS Rekognition, hybrid architectures, cost optimization, and false positive/negative tradeoffs.

该论文提出 Transformer Energy-based Model (TEM)，通过在潜空间中构建可扩展的 Transformer 能量函数并多层能量聚合，实现了无需 classifier-free guidance 的高质量图像生成（ImageNet 256×256 FID-50K 达 3.82），验证了 EBM 的组合生成、零样本修复与分布外检测能力。

OpenAI 发布 Habitat 在线存储平台扩展系列文章第二部分，介绍如何通过与 Azure 团队合作、采用协同管理容器的多租户方案，将底层 Azure Cosmos DB 扩展至服务超过 10 亿 ChatGPT 用户、500+ PB 数据的全球分布式存储。

该论文提出垂直切片Wasserstein距离(VSW)，一种用于比较meta-measures（分布的分布）的可扩展度量，证明其为有效度量并具有维度无关的统计收敛保证，在效率和精度上优于现有方法。

一篇技术博客深入分析了把 Git 仓库放到对象存储(S3)上的常见误区，指出当前 S3-backed WAL 方案在写密集场景下的不足，并重新剖析了 Git 作为内容寻址存储加图数据库的数据模型对存储层的真实需求。

GitHub reports that Git activity more than doubled year over year to 473.3B events per month and is rebuilding its Git infrastructure to support agentic, agent-scale software development. The post details architectural challenges such as commit latency bottlenecks, surging write throughput, ref contention, and read fan-out from CI and code scanning.

A new moving-horizon approximate branch-and-reduce method for training near-optimal deep classification trees on large-scale datasets with continuous features, achieving better accuracy than heuristic baselines and far greater scalability than global optimal solvers.

A technical deep-dive explaining how traditional lock coupling in concurrent binary trees suffers from physical lock contention at the root node, limiting scalability as CPU core counts grow, with benchmark results on a 16-core system.

Using the Devin AI agent to automate the creation of pixel-perfect product tutorials at scale, achieving results superior to human-made tutorials with improved consistency and parallel processing capabilities.

Boris Cherny, creator of Claude Code, emphasizes that AI tools work better with freedom and tools rather than rigid workflows, as general learning systems scale more effectively. This insight is discussed in the context of agent-led development.

Microsoft Research introduces Agensh, a self-organized multi-agent harness that scales to over 1,000 agents, demonstrating improved test-pass rates on coding tasks without a central orchestrator.

The article explains that PostgreSQL's SELECT DISTINCT clause does not scale efficiently, as it always scans all matching rows, leading to performance problems in certain workloads, and provides insights and workarounds.

Marin has built scalable infrastructure to process 25T tokens from open datasets on Hugging Face for training their 535B language model, emphasizing the power of the open community.

The blog post discusses a caching solution for routing engine estimates in a gig economy delivery app using Redis and H3 hexagonal coordinates, and highlights challenges with Redis cluster key distribution and multi-key commands.

This paper introduces methods to bound four key memory peaks in large Mixture-of-Experts training, enabling training at 1M context length with fixed GPU memory and up to 10.4× throughput improvement over baselines.

Toollery is a training-free candidate-compression framework that improves scalability and efficiency for LLM agent tool and skill selection through retrieval-based methods.

Agensh is a scalable self-organized multi-agent system without a central orchestrator that improves performance on complex tasks by scaling the number of agents, showing significant test-pass rate increases on benchmarks like ProgramBench and pandoc.

The author shares a novel use case for Jev in building custom verifiers for AI agent harnesses, enabling scalable test-time compute by combining System One and System Two models.

This paper introduces Compressed Active Subspaces (CAS), a method to enable scalable Bayesian inference in large neural networks by compressing parameter spaces and maintaining predictive performance.