offloading

Microsoft Research findings demonstrate that offloading AI inference from robots to edge or cloud systems enhances task success rates, efficiency, and battery life for physical AI applications.

FlexEE introduces a self-speculative and KV-cache-compatible early exiting framework for efficient LLM inference in offloading deployments, achieving significant speedups on Llama models with minimal accuracy degradation.

This paper introduces techniques to manage memory peaks in training large Mixture-of-Experts models with long context lengths, including Pipelined LLEP, Ring-DTP, SCO, and OffloadStreamAdamW, which enable fixed GPU working sets and improve throughput up to 10.4x.

The article explains how to run the DeepSeek-V4.1-Flash 763B model locally with only 64GB RAM by offloading a 200GB Engram to NVMe and using DSpark, achieving 200 TPS on 4 Max-Q cards.

LazyTrain is a research system that optimizes LLM training on limited hardware by formulating checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, achieving ~1.24x TFLOPS improvement over matched baselines.

SpecPrefetch proposes a parameter-efficient expert prefetching framework for sparse MoE models, using a lightweight adapter to predict next-layer experts for asynchronous transfer while preserving native routing semantics. It achieves up to 20% decoding throughput improvement on a Snapdragon 8 Elite device, demonstrating practical benefits for memory-constrained deployment.

Nader Dabit visualizes the speed difference between 1,000 tok/s subagents and 85 tok/s, highlighting that lightning skill offload enables ~5x faster execution by using subagents for implementation while keeping frontier models as planners and reviewers.

The article explores the growing trend of relying on AI for thinking and decision-making, using anecdotes and a reference to a Ken Liu short story to question the loss of human autonomy.

A user shares their experience offloading the KV cache to RAM in llama.cpp, achieving comparable speeds while freeing VRAM for larger models and context windows, suggesting this trade-off is often worthwhile.

Detailed breakdown of running Qwen 3.5 122B MoE on a single RTX 3090 at 35 t/s using a custom llama.cpp fork (ik_llama.cpp) with fused MoE operations and expert offloading to CPU RAM, significantly outperforming stock llama.cpp MTP.

Discusses performance trade-offs of offloading large AI model weights from GPU VRAM to system RAM, comparing different GPU configurations like RTX 5090 vs RTX6000 for models like DeepSeek V4 Pro.

A user shares their experience with llama.cpp server's model offloading, noting performance trade-offs and quiet operation, and asks for resources to understand how the tool manages memory across VRAM and system RAM.

This paper presents an empirical study on scheduling multiple LLMs on shared heterogeneous hardware, focusing on performance implications of CPU-GPU offloading and preemption. It finds that offloading causes non-linear decode degradation, especially for smaller models, and preemption overhead is dominated by model state reload, providing design guidance for future multi-model schedulers.