Tag
Introducing rust-glancer, an alternative Language Server Protocol implementation for Rust focused on low memory usage, featuring immediate indexing after restart and better performance on older hardware.
WhiteMatter introduces all-to-all cross-layer connections in Transformers via KV mixing, reducing memory footprint and improving performance over standard architectures in pretraining experiments.
Shoehorn is a developer tool that quantizes language models to perfectly fit within a machine's available memory, achieving up to 99.99% memory usage efficiency for local inference.
An individual trained a diffusion model to generate 32x32 pixel images on a Shrike lite microcontroller with only 264KB RAM, experimenting with FPGA acceleration that hit memory bottlenecks, resulting in noisy but sometimes interesting outputs.
The paper proposes Attention-Aware Transform Coding (AATC) for compressing KV caches in large language models, achieving near-lossless accuracy at around 5.8x compression by minimizing distortion through attention mechanisms.
This paper addresses deployment bugs in the Nanbeige4.2-3B Looped Transformer model on Apple Silicon and introduces a memory-efficient chunked-prefill strategy to enable agentic tasks.
This paper proposes a joint optimization of memory and computing frequency for energy-efficient DNN inference on mobile devices, achieving up to 10.4% energy reduction compared to other methods.
This paper introduces Thought-Aware Attention Matching (TAM), a method for compressing KV caches during long chain-of-thought reasoning by segmenting reasoning traces, adaptively allocating compression budgets, and protecting pivotal tokens. Experiments on AIME 2024 and MATH-500 with Qwen3-4B show significant memory reduction while maintaining accuracy.
LazyTrain is a research system that optimizes LLM training on limited hardware by formulating checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, achieving ~1.24x TFLOPS improvement over matched baselines.
CommitKV proposes a lifecycle-aware KV cache compression method for multi-turn ReAct agents, distinguishing dormant tokens from completed ones via commit transitions to reduce memory use and speed up inference.
The author adds Maple-Preview support to Mference, a tool that streams MoE experts from disk to run large models on low-RAM devices, achieving 40 tps with 500MB RAM on an Air M4.
Highlights a paper introducing ZeRO (Zero Redundancy Optimizer), a memory optimization system for efficiently training extremely large models on limited GPU memory.
Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.
AnchorKV is a new KV cache compression scheme that shrinks the cache by 20x without discarding any tokens, using anchor-residual representations to preserve 99% of full-cache accuracy at the 70B scale. The paper is a preprint under review.
Discusses running Gemma 4 on a device with only 500MB of memory, likely through quantization or other optimization techniques.
Technical write-up on running DeepSeek-V4-Flash-0731 with full 1M context on a single RTX 5090 + 256GB DDR5 desktop using vLLM with CPU/RAM offloading, achieving ~800 tps prefill and ~15 tps decode.
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.
PyTorch 2.13 release brings FlexAttention to Apple Silicon with up to 12x speedups and 4x peak memory reduction for large-vocabulary models via fused LinearCrossEntropyLoss, alongside updates to distributed training, compilation, and on-device inference.
Testing a 1.56TB Mixture-of-Experts model on a 6GB RTX 4050 laptop, requiring patched memory streaming with NVMe to achieve 0.106 tokens/s decode speed.
Valkey uses different encodings like listpack and hashtable for its data types to balance memory and performance; understanding these internal structures can lead to significant memory savings.