Literature Review: LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load | Bnechmarking LLMs on Phones [R]
Summary
This literature review analyzes performance efficiency trade-offs for LLM inference on mobile devices, NPUs, and GPUs under sustained load, focusing on benchmarking LLMs on phones.
Similar Articles
Efficient On-Device Diffusion LLM Inference with Mobile NPU
This paper presents llada.cpp, an NPU-aware inference framework for accelerating diffusion large language models (dLLMs) on smartphones. It introduces three techniques—Multi-Block Speculative Decoding, Dual-Path Progressive Revision, and Swap-Optimized Memory Runtime—to align dLLM inference with mobile NPU characteristics, achieving 17-42x latency reduction over CPU baseline.
Local LLM Inference Optimization: The Complete Guide
A comprehensive guide to optimizing local LLM inference on consumer hardware, covering tools like llama.cpp, vLLM, and LM Studio, with practical advice on memory hierarchy, layer placement, and common failure modes.
The efficient frontier of LLM inference
The article explains the efficient frontier concept in LLM inference, covering tradeoffs between latency, throughput, and cost, and outlines techniques to manage or enhance efficiency in deployment.
Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
Quant.npu introduces a fully static quantization framework for mobile NPUs, using learnable parameters and rotation matrices to enable efficient low-bit LLM inference without runtime re-computation, achieving up to 15.1% latency reduction.
Literature Review: MELTing point: Mobile Evaluation of Language Transformers | Bnechmarking LLMs on Phones
This literature review introduces MELT, a benchmark for evaluating large language models (LLMs) on mobile devices, highlighting the challenges and opportunities of running transformers on phones.