Tag
In a two-week sprint, the team used an internal Claude model to achieve a 3x speed improvement for key user journeys on claude.ai and the desktop app, significantly reducing wait times without any customer-facing incidents.
Adaptive Lossless floating-Point (ALP) Encoding is a new lightweight encoding for floating-point data in Apache Parquet, offering compression ratios similar to zstd with faster decompression, random-access support, and GPU/SIMD-friendly decoding.
Modal details how they optimized inference performance for trillion-parameter coding agents, achieving significant improvements in throughput and interactivity for their service.
The paper introduces a method to prune encoder layers in the Whisper ASR model, reducing encoder size by 18.5% and recovering performance through unlabeled data distillation, with code and a pre-trained model released for adoption.
Halogen version 0.12.0 fixes performance degradation at high context depths, showing improved decode and prefill speeds for Qwen3.8-Flash-Next at 1 million tokens of context on AMD Ryzen AI Max+ hardware.
This paper proposes Cache-to-Cache (C2C), a novel paradigm for direct semantic communication between large language models using KV-Cache, which enhances response quality and reduces latency compared to text-based methods.
This article explains how GPU memory is utilized during large language model (LLM) inference, breaking it down into four key components: model weights, KV cache, activations/workspace, and runtime overhead. It highlights the importance of quantization in optimizing memory usage for better performance.
Go 1.27 introduces size-specialized memory allocation for allocations of 80 bytes or fewer, improving allocation speeds by 20-30% and overall program performance by up to 1% for allocation-heavy code.
A technique to offload the KV cache of Qwen3.8-Flash-Next to system RAM is demonstrated, allowing long-context inference with minimal decode slowdown by leveraging the model's efficient architecture.
Polymarket has built in-house indexing systems powered by rindexer in Rust, achieving near-instant updates for trades, positions, and balances with onchain data events processed up to 28 seconds faster.
The co-inventor of ChatGPT announces the release of a new AI model named Jev, trained with RLCD, claiming it is 20-200x faster, 40-400x cheaper, and optimized for composable intelligence as a path to AGI.
An implementation of expert lookahead achieves over 10% performance improvement for MoE models running on low-memory devices using slotstream, with additional gains from a correction model.
This pull request adds missing AMD GCN MMQ configuration to ggml-cuda for HIP, enhancing prefill performance for RDNA2 GPUs such as MI50 and MI60 in the llama.cpp inference library.
The DSPy 3.4 release candidate is announced, moving away from litellm to improve import speed and reduce dependencies. Community members like Drew Breunig have praised the improvements.
A performance throttle in the Apple M3 Neural Engine was discovered when weight sizes are multiples of 1 MiB, causing throughput drops. By avoiding the problematic DMA path, token throughput for models like Llama 3.2 and Qwen3-8B was significantly improved.
An author improved MLX performance on an M3 Ultra Studio to achieve 73 tok/s for a 4-bit Qwen3.8-Flash-Next model, which shows intelligence scores matching GPT-5.6 Sol, highlighting the growing potential of local AI.
Swift-Qwen3.8-27B is UkisAI's reasoning-efficient derivative of Qwen3.8-27B, reducing thinking tokens by 58.3% while maintaining near-identical performance.
The authors have created a lightweight, performant asynchronous reinforcement learning stack in pure JAX, sharing a work log with insights on inference, RDMA weight transfer, memory optimizations, and sharding for scaling RL systems.
This article provides a comprehensive guide on Jump Labels in the Linux kernel, explaining their hardware background, usage, and implementation details for kernel programmers.
Michael Stapelberg details how he removed Debian Code Search's last cgo dependency by reimplementing the TurboPFor integer codec in Go using newly available SIMD support, leveraging AVX512 to exceed the reference implementation's performance.