Tag
Cloudflare optimized the memory usage of their Pingora-based load-balancing service by refining the pingora-ketama consistent hashing library in Rust, reclaiming over 100TB of RAM globally.
Memorable introduces a method to optimize memory in AI agents using embeddings instead of tokens, enabling procedural memory that persists across multiple runs.
Edge0 is a streaming MoE inference engine that uses trained routing prediction to serve 35B Mixture-of-Experts models from SSD, achieving near-fp16 performance on consumer hardware with low memory usage.
This paper introduces techniques to manage memory peaks in training large Mixture-of-Experts models with long context lengths, including Pipelined LLEP, Ring-DTP, SCO, and OffloadStreamAdamW, which enable fixed GPU working sets and improve throughput up to 10.4x.
BeaconKV is a training-free method that uses beacon queries to compress key-value cache for efficient inference in Large Reasoning Models, reducing memory usage and improving throughput without sacrificing accuracy.
HeadWiseKV is a training-free framework that compresses KV caches in hybrid long-context language models, reducing GPU memory usage and extending context lengths while maintaining quality.
The paper proposes Random Attention, a KV cache eviction method that uses random selection instead of scoring, matching selective methods while improving throughput in reasoning tasks.
A video explaining memory optimization techniques used in the Superlogical server, including in-memory compression, dynamic thread management, and fast binary snapshots.
slotstream is an open-source tool that enables running large AI models like Qwen3.8-Flash-Next on Macs with limited memory by streaming model weights from SSD, achieving approximately 12 tokens per second on a 48GB Mac.
FlashAccel is a co-designed system that integrates High-Bandwidth Flash into GPUs to enhance LLM inference, achieving improved throughput and energy efficiency by mitigating access latency and optimizing bandwidth utilization.
A new checkpoint for Qwen3.8 Flash-Next offloads the ngram lookup table to SSD, reducing RAM usage by 48 GB while maintaining inference speed in SGLang.
Zod 4.5 implements method memoization to drastically reduce memory usage, achieving up to 9.8x less heap retention than previous versions.
Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without retraining, offering a cheaper and more reliable inference solution for LLMs.
Cloudflare optimized their 1.1.1.1 DNS cache to reduce memory usage by over 50%, saving 100 terabytes of memory while improving performance through refined Rust data structures.
This paper analyzes KV-cache eviction strategies in sparse-attention models, showing that selecting largest weights is near-optimal and that published margins come from memory and query information, with ContourKV achieving strong performance.
Mitchell Hashimoto announces significant memory usage improvements in the upcoming 1.4 release of the Ghostty terminal emulator, with up to 450x lower memory for non-visible windows.
This paper presents a feature-major codebook layout for memory-efficient sparse-binary self-organizing maps, enabling the training of large-scale maps up to 1.05 million neurons on a single GPU with significant speed improvements for MEDLINE data.
User feels Codex has quietly updated, significantly optimizing session loading speed and memory usage, enhancing smoothness for long development projects.
The blog post details techniques to optimize memory usage in a C++ markdown parser, significantly reducing AST node size from 232 bytes to 16 bytes through arena allocation and pointer compression, while maintaining parsing speed.
Unsloth Desktop is a free, open-source desktop application that integrates training, inference, and Agent toolchain through extreme optimization of VRAM and speed, significantly improving the user experience of AI models like Qwen3.8 27B.