Tag
作者认为64GB内存不再是本地LLM的'死区'——在M4 Max上运行Qwen3.8 27B进行agentic coding，KV cache和prompt cache使64GB恰好成为sweet spot，打破了此前认为该容量尴尬的共识。
LRMoE is a memory-aware MoE training system that jointly recalibrates backward demand and balances retained workload on memory-constrained GPU nodes, reducing cumulative MoE-layer training time by 21.2-44.8% on Qwen3-30B-A3B and GLM4.5-106B-A12B compared to SYMI.
The article benchmarks Go 1.28's new HTTP/2 'parking' mechanism, which lets idle connection-serving goroutines exit and be recreated on demand, showing 84.7–92.6 MiB (about 39–41%) RSS savings at 5,000 idle connections compared to the same binary with parking disabled.
A detailed write-up demonstrating how NixOS 26.05 can be tuned to idle at just 96MB RAM and complete full rebuilds within 512MB, using kernel cmdline flags, module blacklisting, and disabling unused hardware support for KVM guests.
MemSteer, developed at Peking University and Microsoft Research Asia, converts user histories into automatically generated supervision tests for agent memory, and MemSteer-Opt uses the resulting evidence to optimize storage, retrieval, and presentation without hand-labeled data or fine-tuning, improving BM25 baselines on ATM-Bench and RHELM.
A developer shares ApexW, an experimental PyTorch optimizer using compact optimizer states and a Triton-based execution path, claiming ~1.5x faster optimizer steps and halved optimizer-state memory on large-tensor workloads — while openly noting regressions on many-small-tensor models and minimal end-to-end gains.
A Corsix.org blog post explores reducing binary size for enum-to-string conversion functions by replacing pointer tables with compact offset tables, and further compressing those offsets using a linear approximation corrected by a small lookup table of residuals.
PACE 是一种免微调的后训练四比特量化器，用于线性注意力循环状态，通过降低量化频率（每隔几个 token 才舍入一次）并采用列对齐的量化尺度来减少误差，从而在 4B-35B 参数混合模型上以约三分之二的内存达到与八比特 DAMP 相当的性能。
斯坦福 CS336（2026 春）第 6 讲的笔记，讲解如何通过减少 GPU 内存往返来优化内核：对比 softmax 多次遍历与融合行内核的内存流量，介绍 Triton 的块级编程模型，并强调先用 CUDA event 测量基线、再做内核融合与分块优化。
The article introduces dp_knapsack_sliding_hirschberg, a new activation memory budget solver for PyTorch that reduces peak RAM usage during memory planning by up to 20x compared to the default dp_knapsack solver, explaining how 0/1 knapsack dynamic programming is used to choose which operations to save or recompute.
PR #29887 adds a GPU cache for MoE experts kept in host memory in llama.cpp, improving performance when expert weights overflow to CPU RAM.
Alex Alejandre explores compiling the Janet language for the Linux x32 ABI to get 32-bit pointers with 64-bit speed, reporting up to ~20-25% memory savings on Arch but a 50% slowdown for that workload — highlighting that x32 offers dramatic potential for pointer-heavy programs but lacks packaging and ecosystem support.
WakeKV is a reactive, reversible KV-cache residency policy that dynamically demotes cooling attention heads to a recoverable CPU reservoir, outperforming frozen classification and destructive eviction baselines (SnapKV, R-KV, ReasonAlloc) on long-context serving, with a FlexiCache/vLLM implementation confirming throughput gains on Mistral-7B.
Hugging Face Diffusers 上张量并行（tensor parallel）加载迎来重大优化：在 Flux.2-Dev DiT、TP=4（A10G）配置下，加载时间从 30.4s 降至 12.5s（约 2.4 倍提速），每 rank 峰值 CPU 内存从 64.1 GB 降至 6.8 GB（减少约 89%）。相关分布式推理（Accelerate 与 PyTorch Distributed）用法已更新到官方文档。
PReCache is a training-free KV cache sharing framework that uses low-rank precomputation and neutral reconstruction to reduce memory and computation redundancy in multi-LoRA agent systems, achieving speedups and maintaining accuracy.
WonderBox is a native macOS system toolbox that provides system monitoring, memory optimization, cache cleaning, and other utilities to help users manage their Mac efficiently.
A user shares their experience switching to the QWEN3.6-27B-MLX-8bit model for local AI tasks, finding it performs comparably to larger models while saving significant RAM on a Mac Studio, improving workflow stability.
This paper introduces methods to bound four key memory peaks in large Mixture-of-Experts training, enabling training at 1M context length with fixed GPU memory and up to 10.4× throughput improvement over baselines.
The paper introduces PAGE, a partition-aware gated KV-cache eviction method that uses a scalar metric to predict input classes and apply eviction only when safe, reducing accuracy degradation in large language models.
TierKV proposes a predictive multi-tier KV caching framework to optimize memory usage and throughput for long-context LLMs on mobile devices, achieving significant performance improvements with minimal accuracy degradation.