Reducing LLM Latency
Summary
Techniques and methods for reducing latency in large language models, improving inference speed.
Similar Articles
Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors
This paper introduces HYMELL, a hybrid analytical-machine-learning framework for estimating LLM inference latency and energy across prefill and decode phases, validated on NVIDIA H100 with under 5% error for LLaMA 3 8B.
LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models
LaCache introduces a training-free acceleration framework for diffusion-based LLMs that uses lossless caching and precision-adaptive inference to eliminate redundant computation during denoising steps, achieving up to 40.2× end-to-end speedup while maintaining task accuracy.
Local LLM CPU users... How long is it taking you to do anything?
A discussion about the performance of running large language models locally on CPU, especially with large context sizes, and the challenges of VRAM constraints.
MemDefrag: Latent Memory Defragmentation for Large Language Models
Proposes MemDefrag, a training-free framework that uses a middle-layer tracing signal to defragment latent memory in LLMs, achieving significantly better knowledge retention than existing methods like MemoryLLM and M+.
Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices
This paper proposes an efficient GPU inference method for LLMs with moderate unstructured sparsity, introducing a three-layer matrix storage format and a SpMM kernel that jointly utilizes sparse tensor cores and CUDA cores, achieving up to 1.64× kernel-level speedup over SpInfer and up to 1.41× end-to-end speedup over FlashLLM.