Tag
yibie announces strata-mlx, an unofficial MLX-based inference engine for Mac that runs the 125B MoE model Qwen3.8-Flash-Next from Strata's GGUF files, achieving 1.7–2.9x faster generation than llama.cpp and enabling lower-RAM Macs to offload experts to SSD.
ROCmFPX 解决了预填充阶段 MMQ 阻塞问题，通过寄存器路径将 Q4_0_ROCMI4 权重反量化为 BF16 WMMA GEMM，在 Qwen3.8 Flash Next HC-Q8 上实现 ub 512 约 13%、ub 1024 约 23% 的吞吐提升，稀疏专家处理时间从约 2.3 秒降至 1.8 秒。
Jane Street 分享了其内部消息框架 Aria 的扩展实践：通过为 topic partition 引入索引与树拆分策略优化 tip recovery 过滤流程，在生产负载上实现 30% 的 CPU 使用率下降。
audio.cpp 带来多项运行时性能优化：Higgs Audio TTS 显存占用减少 48%（低于 6GB），HTDemucs 提速 2.2 倍，PocketTTS 在 CPU 上提速 2.2 倍，同时 WebUI 新增实验性的生成历史功能，目前支持 110+ 音频模型家族。
Georgi Gerganov recommends upgrading to DFlash speculative decoding for faster Qwen3.8-27B inference in llama.cpp, requiring the latest llama.cpp v0.6.0 release with the ggml-org/Qwen3.8-27B-GGUF model.
This paper presents Dostoevsky, a system that improves the space-time trade-offs of LSM-tree based key-value stores by adaptively removing superfluous merging during compaction, yielding better performance than existing compaction strategies.
Developer forked the Strata LLM inference engine and used Claude Code to heavily modify it for an IBM AC922 POWER9 + Tesla V100 system, boosting Qwen3.8-FN UD-Q4_K_XL performance from 15 tk/s decode to ~113 tk/s decode and 7,357 tk/s prefill via memory management, NVLink optimization, and expert caching improvements.
Gergely Orosz highlights turbopuffer openly sharing their v3 architecture migration, where the new design was initially up to 128x slower than the old one, with each optimization (like batched reads yielding a ~9x improvement) publicly documented.
A hand-written GEMM kernel for NVIDIA Blackwell (B200), built with CUTLASS's CuTeDSL from raw primitives (tcgen05, TMA, mbarriers, TMEM), goes from 164 to 1401 TFLOP/s — about 97% of cuBLAS — with one key change doing most of the work and prefill vs decode workloads yielding different optimizations.
A practical guide covering common inference bottlenecks — including runtime selection (Ollama, llama.cpp, vLLM, SGLang), hardware-specific compilation flags, quantization choices (IQ vs Q_K vs NL), and model selection by task — with best practices for optimizing LLM inference within hardware constraints.
PyTorch/Meta blog reveals Jagged Flash Attention built with TLX (Triton Low-level Extensions) on NVIDIA Blackwell B200, using ~3.2K lines of Triton code—3x less than FlashAttention-4—while outperforming FA4 by ~13% forward and ~50% backward on jagged attention shapes used in Meta's Generative Ads Model.
ParadeDB responds to PlanetScale's new TIN Postgres text-search extension by closing its 8x BM25 performance gap through optimization passes rather than architectural changes, benchmarking both on the same StackExchange dataset.
Suvaditya Mukherjee of Hugging Face will present on using the PyTorch Profiler to identify performance bottlenecks and optimize model execution at PyTorch Conference North America 2026 in San Jose, October 20-21.
A blog post detailing performance optimizations in llama.cpp that make prompt lookup drafting up to 42x faster and reduce memory usage by 2.6x, based on techniques from Daniel Lemire and Martin Ankerl.
The article provides a step-by-step guide to tuning a server for benchmarking to reduce run-to-run noise and ensure measurements are repeatable, with techniques like hardware inspection and core pinning.
This article details the porting of the Kev-0.8B model, a Qwen3.5 variant, to Core ML, demonstrating substantial improvements in inference speed and memory efficiency on Apple Silicon devices through benchmarks.
This article describes how Anthropic tripled the speed of Claude.ai in two weeks, along with the author's experience applying this method to optimize their own website, and the release of the '闪电.skill' tool for public use.
This article discusses techniques for writing efficient C++ code, emphasizing data-oriented design and performance optimization for applications like games and real-time processing.
PSA advising to increase the -cram parameter in llama.cpp for better performance in agentic workflows with long contexts, based on personal experience with Qwen 27B 3.8.
The article presents an open-source study on optimizing latency for local vision-language models through benchmarking and techniques like native batching and MLX quantization, achieving significant speedups while maintaining decision accuracy on Apple hardware.