LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

# LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Source: [https://arxiv.org/html/2609.03079](https://arxiv.org/html/2609.03079)
obeypunctuation=true\]1George Mason University2Global Technology Applied Research, JPMorganChase3University of Kansas,,

Conference:The 32nd Annual International Conference on Mobile Computing and Networking; October 26–30, 2026; Austin, TX, USAThe 32nd Annual International Conference on Mobile Computing and Networking \(MobiCom ’26\), October 26–30, 2026, Austin, TX, USADOI:[10\.1145/3795866\.3844470](https://doi.org/10.1145/3795866.3844470)ISBN:979\-8\-4007\-2505\-0/26/10CCS:Computing methodologies Machine learningCCS:Software and its engineeringRenyuan Liu1, Yuyang Leng1, Kaiyan Liu1, Yuzhou Zhong1, Shaohan Hu2, Chun\-Fu \(Richard\) Chen2, Peijun Zhao2, Heechul Yun3, Shuochao Yao1Affiliation:\[email:[rliu23,yleng2,kliu23,yzhong9@gmu\.edu, shaohan\.hu,richard\.cf\.chen,peijun\.zhao@jpmchase\.com](mailto:rliu23,yleng2,kliu23,[email protected],%0Ashaohan.hu,richard.cf.chen,[email protected]%0A)email:[heechul\.yun@ku\.edu, shuochao@gmu\.edu](mailto:[email protected],%[email protected]%0A)

© cc

###### Abstract\.

On\-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM\. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade\-off: accurate sparse execution decisions require the latest context, whereas efficient computation–I/O overlap requires early prediction\. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads\. We present LeanStream, a streaming speculate\-and\-refine framework for efficient on\-device LLM inference\. LeanStream progressively refines computation, loading, and cache\-retention priorities using partial GPU results, enabling fine\-grained overlap between GPU execution and storage I/O\. We implement LeanStream on both mobile and embedded platforms\. Compared with prior on\-device LLM inference systems, LeanStream reduces memory usage by 4\.8×\\times–7\.5×\\timesat the best throughput achieved by prior work, while further improving token generation throughput by 1\.6×\\times–2\.1×\\times\.

###### Keywords:

Mobile Computing, On\-device Inference

††cc\-license:by## 1\.Introduction

The recent rise of Large Language Models \(LLMs\) has drawn significant attention\. Increasing privacy and security requirements, together with the increasing availability of everyday personal computing devices, have created strong demand for on\-device LLM inference\. Yet this remains challenging on mobile platforms due to their limited memory and compute capacity\. To overcome this challenge, recent work has proposed storing model weights on SSDs or flash memory and dynamically activating and executing only the relevant sub\-models\([Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6);[Song et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib7);[Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8)\)\. By exploiting the widespread activation sparsity observed across a wide range of, if not all, LLMs\([Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1);[Liu et al\., 2025a](https://arxiv.org/html/2609.03079#bib.bib11);[Federici et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib12)\), these approaches can dynamically identify the active weight sub\-matrices, load them from storage into device memory, and compute only the necessary portions on demand\.

Compared with loading and executing the full weights, exploiting dynamic activation sparsity can, in principle, reduce memory and computation costs by up to 80%\([Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1);[Liu et al\., 2025a](https://arxiv.org/html/2609.03079#bib.bib11);[Federici et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib12)\)\. However, realizing these theoretical gains in practice creates a fundamental tension with system\-level optimizations\. To make accurate decisions about which weight blocks or sub\-models should be loaded and executed, the predictor ideally relies on the most recent context, namely the output of the preceding layer\. Yet this dependence on fully updated context limits opportunities for optimizations such as I/O prefetching and pipelining\([Guo et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib13);[Wang et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib14);[Chen et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib23);[Chen et al\., 2026](https://arxiv.org/html/2609.03079#bib.bib18)\), leading to substantial I/O stalls\.

To mitigate this bottleneck, many systems adopt layer\-wise speculative I/O fetching\([Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1);[Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6);[Song et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib7);[Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8)\), where activation patterns are predicted using inputs available before the current layer finishes\. This enables weight transfers to overlap with the computation of the preceding layer, thereby hiding part of the I/O latency\. However, because such speculation does not use the latest context, it often produces less accurate activation predictions, causing the system to load unnecessary weight sub\-matrices and execute more sub\-models than required\. Some systems further mitigate I/O latency by introducing in\-memory weight caches\([Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8);[Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6)\)\. Yet the challenge remains similar: the most informative features for deciding which weight sub\-matrices to retain and execute are often still being produced within the ongoing computation kernel\. Without accurate predictive guidance for cache retention and execution prioritization, these designs can lead to substantial memory redundancy \(e\.g\., around 3 GB for cached weights and predictive models for a 7B LLM\) and significant computational overhead \(e\.g\., more than 3×\\timesthe computation required under the ideal activation pattern\)\([Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8);[Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6)\)\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/intro_beta.png)Figure 1\.Challenges in on\-device LLM inference with activation sparsity\. \(a\) Full\-context prediction achieves high accuracy but forces serialized compute and I/O\. \(b\) Layer\-wise speculative fetching overlaps I/O with compute, but incurs prediction errors and redundant I/O\. \(c\) Speculation with caching reduces I/O stalls, but still introduces memory redundancy and extra computation\. \(d\) Our streaming speculate\-and\-refine pipeline progressively updates context for finer\-grained compute–I/O coordination\.Therefore, instead of forcing a choice between "waiting for the latest context to generate the most accurate and efficient computation and I/O decisions" and "performing long\-horizon prediction to prioritize future computation and data movement", our key insight is to unify the two\. We always begin with an initial prediction based on the most up\-to\-date information available, and then continuously refine computation prioritization and data loading/retention decisions in a fine\-grained streaming manner\. Concretely, while the GPU is still processing the preceding layer, the system leverages partial intermediate results to progressively refine speculative data\-loading and execution priorities, continuously updating I/O requests to reduce redundancy\. At the same time, the I/O subsystem streams the required weights in fine\-grained blocks, enabling the GPU to start executing the highest\-priority available sub\-models as soon as compute resources are free, instead of waiting for the entire sparse weight set of a layer to arrive\.

However, implementing the proposed streaming framework raises several non\-trivial challenges\. First, the speculate\-and\-refine design requires frequent coordination between heterogeneous processors on mobile and embedded SoCs, with the CPU handling activation\-sparsity prediction and I/O while the GPU performs computation\. Existing synchronization mechanisms, such as global barriers \(e\.g\.,cudaDeviceSynchronizein CUDA orclFinishin OpenCL\), incur prohibitive overhead when used at high frequency\. Event\-based primitives \(e\.g\.,cudaEvent\_torcl\_event\) reduce this overhead, but they provide only unidirectional notification from the co\-processor to the CPU host\. This limitation prevents the host from exerting real\-time, bidirectional control over prioritized GPU execution based on dynamic I/O progress\. In addition, because these primitives operate at kernel granularity, they often require manual kernel partitioning, which further increases kernel launch overhead and overall system cost\.

More importantly, although finer\-grained inter\-processor communication can improve activation\-pattern prediction accuracy and reduce idle time, overly frequent coordination sacrifices hardware parallelism and introduces additional overhead\. The system therefore must determine an appropriate coordination frequency\. Yet this choice cannot be fixed offline, because I/O latency is non\-deterministic and varies with runtime factors such as cache miss behavior\. As a result, execution time across the streaming pipeline becomes unpredictable, making adaptive online control essential\.

The second challenge is to efficiently leverage GPU intermediate results to predict dynamic activation patterns and their relative importance\. In contrast to prior methods that make predictions at kernel granularity, our approach operates at a much higher coordination frequency, significantly tightening the latency budget for prediction\. If this process is not sufficiently efficient, prediction latency can itself become a bottleneck, undermining the gains from frequent inter\-device coordination\. This makes a lightweight, high\-performance predictive model critical for system control, so that decision\-making can keep pace with the high\-rate speculate\-and\-refine stream\.

To address these fundamental bottlenecks, we proposeLeanStream, a streaming speculate\-and\-refine framework for efficient on\-device LLM inference\. LeanStream bridges the gap between context\-aware prediction and system\-level efficiency by enabling computation and data\-movement decisions to be made progressively rather than monolithically\. LeanStream starts from an initial prediction using the freshest available context, then continuously updates computation priorities and data loading/retention decisions as intermediate results are produced\. This design enables fine\-grained overlap among I/O and GPU computation, thereby reducing redundant data movement, minimizing processor idle time, and improving end\-to\-end inference efficiency on resource\-constrained mobile and embedded platforms\.

Fine\-Grained Streaming Control\. To support high\-frequency information exchange, we design a lightweight communication and data\-management framework that minimizes coordination overhead between GPU execution and CPU\-side control and I/O\. The framework reduces both synchronization overhead and metadata traffic, enabling high\-grained streaming without incurring additional stalls\. We also rigorously analyze the trade\-off introduced by frequent coordination and design an adaptive online controller to manage it\. By dynamically adjusting the synchronization frequency at runtime, the controller maintains an effective balance between prediction accuracy and hardware parallelism, thereby maximizing end\-to\-end system throughput\.

Lightweight System Control with Stacked Learnable Hashing\. We design a lightweight control mechanism based on stacked learnable hashing for low\-latency, memory\-efficient prediction under fine\-grained streaming execution\. Compared with conventional shallow\-MLP controllers, stacked learnable hashing offers high expressive capacity with substantially lower runtime and memory overhead\. By relying on efficient bitwise operations, in\-register table lookups, and compact output representations, it minimizes prediction latency while preserving strong modeling power\. At the same time, it remains fully compatible with standard supervised learning and can be trained end\-to\-end with backpropagation\. This makes it an effective control primitive for high\-rate speculate\-and\-refine execution\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/dependency_V3.png)Figure 2\.Dependency chain of one\-shot prediction and LeanStream’s partial\-feature refinement\. \(a\) Existing layer\-wise predictors use the residual state enteringMLPn\\mathrm\{MLP\}\_\{n\}to make a fixed single prediction for the neuron priorities ofMLPn\+1\\mathrm\{MLP\}\_\{n\+1\}\. \(b\) LeanStream executesMLPn\\mathrm\{MLP\}\_\{n\}in priority\-ordered stages\. As each stage completes, its output is accumulated with the residual state to form an increasingly complete partial residual feature, which is used to refine the neuron priorities ofMLPn\+1\\mathrm\{MLP\}\_\{n\+1\}\.We evaluate LeanStream on three LLMs: Mistral\-7B\([Jiang et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib47)\), Llama2\-7B\([Touvron et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib48)\), and Qwen2\.5\-7B\([Yang et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib49)\), across two embedded platforms, NVIDIA Jetson AGX Orin and Jetson AGX Xavier, and one mobile platform, the OnePlus 13\. Compared with state\-of\-the\-art LLM inference systems\([Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8);[Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6);[Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1)\), LeanStream reduces memory usage by 4\.8×\\timesto 7\.5×\\timesat the best throughput achieved by prior work, and further improves token generation throughput by 1\.6×\\timesto 2\.1×\\times\.

In summary, this paper makes the following contributions:

- •We present*LeanStream*, a streaming speculate\-and\-refine framework for efficient on\-device LLM inference\.
- •We introduce a*fine\-grained streaming control*design with adaptive online coordination across CPU, GPU, and I/O to reduce stalls while preserving hardware parallelism\.
- •We propose*stacked learnable hashing*, a lightweight predictive mechanism for fast system control under tight latency and memory budgets\.
- •We implement and evaluate LeanStream on mobile and embedded platforms, showing substantial gains in resource utilization and inference efficiency over prior approaches\.

## 2\.Motivation & Related Work

### 2\.1\.Challenges of on\-Device LLM Inference

Deploying LLMs on edge and mobile devices is fundamentally limited by the gap between model size and available DRAM\. In practice, this constraint is even more severe because the operating system and active applications already occupy a substantial fraction of device memory\. Prior work has explored various techniques to improve the efficiency and deployability of learning systems on mobile devices\([Yao et al\., 2017](https://arxiv.org/html/2609.03079#bib.bib21);[Yao et al\., 2018](https://arxiv.org/html/2609.03079#bib.bib22);[Liu et al\., 2024b](https://arxiv.org/html/2609.03079#bib.bib15);[Liu et al\., 2025b](https://arxiv.org/html/2609.03079#bib.bib17);[Liu et al\., 2025c](https://arxiv.org/html/2609.03079#bib.bib19);[Leng et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib16);[Leng et al\., 2026](https://arxiv.org/html/2609.03079#bib.bib20)\)\. To address the more direct challenge of model weights exceeding available DRAM, a common approach is therefore to place model weights on SSDs and fetch them on demand\([Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6);[Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1);[Song et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib7);[Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8);[Federici et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib12)\)\. Most modern LLMs use decoder\-only Transformers, where feed\-forward networks \(FFNs\) dominate model size\. In recent Group Query Attention models\([Roumeliotis et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib24)\), FFNs account for roughly 80% of parameters in Llama3\-8B, Qwen2\-7B, and Mistral\-7B\. Dynamic sparsity exploits matrix\-vector\-dominated token generation and the many*zero\-valued or near\-zero elements*produced by ReLU\-family\([Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1);[Song et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib7);[Song et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib26)\)and SwiGLU\([Zhang et al\., 2022](https://arxiv.org/html/2609.03079#bib.bib25);[Federici et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib12);[Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8)\)activations\. Exploiting these*sparsity patterns*at different levels\([Federici et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib12)\)can skip around 80% of unnecessary computation and reduce data movement with negligible accuracy loss\([Federici et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib12);[Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8)\)\. Moreover,*activation magnitudes*indicate the*relative priority*of loading and computing corresponding weight sub\-matrices\. Yet translating this opportunity into end\-to\-end system gains remains challenging\.

Figure[1](https://arxiv.org/html/2609.03079#S1.F1)illustrates the core systems tension in sparse on\-device LLM inference\. In Figure[1](https://arxiv.org/html/2609.03079#S1.F1)\(a\), full\-context prediction provides the most accurate activation decisions but serializes GPU computation and I/O, leaving hardware underutilized\. Figure[1](https://arxiv.org/html/2609.03079#S1.F1)\(b\) instead predicts the next layer before the current layer finishes, overlapping I/O with computation at the cost of lower prediction accuracy, redundant weight fetching, and extra computation\. Adding an in\-memory cache in Figure[1](https://arxiv.org/html/2609.03079#S1.F1)\(c\) mitigates I/O stalls but still incurs memory and computation redundancy because decisions rely on incomplete context\.

LeanStream, shown in Figure[1](https://arxiv.org/html/2609.03079#S1.F1)\(d\), avoids this trade\-off by progressively refining computation, loading, and cache\-retention priorities as intermediate results become available\. This enables fine\-grained compute–I/O coordination while reducing redundant data movement and computation\.

Figure[2](https://arxiv.org/html/2609.03079#S1.F2)illustrates partial\-feature refinement\. Let𝐱n\\mathbf\{x\}\_\{n\}denote the residual state beforeMLPn\\mathrm\{MLP\}\_\{n\}\. Prior approaches use𝐱n\\mathbf\{x\}\_\{n\}for a one\-shot prediction of the neuron and weight priorities ofMLPn\+1\\mathrm\{MLP\}\_\{n\+1\}\. LeanStream instead partitionsMLPn\\mathrm\{MLP\}\_\{n\}intoKKstages and loads and computes them in descending predicted\-priority order\. Let𝐡n\\mathbf\{h\}\_\{n\}be the input toMLPn\\mathrm\{MLP\}\_\{n\}, and let𝐖n,gate\(k\)\\mathbf\{W\}\_\{n,\\mathrm\{gate\}\}^\{\(k\)\},𝐖n,up\(k\)\\mathbf\{W\}\_\{n,\\mathrm\{up\}\}^\{\(k\)\}, and𝐖n,down\(k\)\\mathbf\{W\}\_\{n,\\mathrm\{down\}\}^\{\(k\)\}denote the gate\-, up\-, and down\-projection weight slices of stagekk\. Its partial output is

\(1\)Δ​𝐦n\(k\)=\[SiLU⁡\(𝐡n​𝐖n,gate\(k\)\)⊙\(𝐡n​𝐖n,up\(k\)\)\]​𝐖n,down\(k\)\.\\Delta\\mathbf\{m\}\_\{n\}^\{\(k\)\}=\\left\[\\operatorname\{SiLU\}\\left\(\\mathbf\{h\}\_\{n\}\\mathbf\{W\}\_\{n,\\mathrm\{gate\}\}^\{\(k\)\}\\right\)\\odot\\left\(\\mathbf\{h\}\_\{n\}\\mathbf\{W\}\_\{n,\\mathrm\{up\}\}^\{\(k\)\}\\right\)\\right\]\\mathbf\{W\}\_\{n,\\mathrm\{down\}\}^\{\(k\)\}\.After the firstjjstages, LeanStream forms the updated prediction feature as

\(2\)𝐱~n\(j\)=𝐱n\+∑k=1jΔ​𝐦n\(k\)\.\\widetilde\{\\mathbf\{x\}\}\_\{n\}^\{\(j\)\}=\\mathbf\{x\}\_\{n\}\+\\sum\_\{k=1\}^\{j\}\\Delta\\mathbf\{m\}\_\{n\}^\{\(k\)\}\.Each𝐱~n\(j\)\\widetilde\{\\mathbf\{x\}\}\_\{n\}^\{\(j\)\}refines the weight priorities ofMLPn\+1\\mathrm\{MLP\}\_\{n\+1\}beforeMLPn\\mathrm\{MLP\}\_\{n\}completes\. Priority\-ordered execution makes informative partial outputs available earlier, improving subsequent prediction refinement\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/motivation_sync_overhead_V8.png)Figure 3\.Synchronization overhead analysis\. \(a\) Existing synchronization primitives introduce execution stalls under frequent coordination\. \(b\) Increasing synchronization frequency significantly raises the GPU execution time of a Mistral\-7B MLP layer with 80% sparsity on Jetson AGX Orin\.Realizing this design requires high\-frequency coordination between GPU computation and I/O, but existing synchronization mechanisms \(cudaDeviceSynchronizein CUDA orclFinishin OpenCL\) are too expensive at that granularity\. As shown in Figure[3](https://arxiv.org/html/2609.03079#S2.F3)\(b\), increasing synchronization frequency sharply increases the execution time of a Mistral\-7B MLP layer with 80% sparsity on Jetson AGX Orin\. This overhead comes from both the synchronization primitive itself and the kernel\-fragmentation cost of splitting monolithic kernels into smaller schedulable units\. These results motivate a custom low\-overhead communication runtime for fine\-grained speculate\-and\-refine execution\.

### 2\.2\.The Inefficiency of Static Coordination

To further optimize hardware efficiency, a common design objective is to fine\-tune the execution schedule to maximize the overlap between weight loading and GPU computation\. This involves identifying the optimal coordination points where the system can interleave I/O requests with compute kernels without introducing significant stalls\([Narayanan et al\., 2019](https://arxiv.org/html/2609.03079#bib.bib27);[Wang et al\., 2022](https://arxiv.org/html/2609.03079#bib.bib28);[Chen et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib29);[Bae et al\., 2021](https://arxiv.org/html/2609.03079#bib.bib30);[Rajbhandari et al\., 2021](https://arxiv.org/html/2609.03079#bib.bib31)\)\.

However, the effectiveness of offline scheduling fundamentally depends on workload determinism, an assumption that does not hold for sparse LLM inference\. As shown in Figure[4](https://arxiv.org/html/2609.03079#S2.F4), our measurements reveal substantial runtime variability that makes static profiling ineffective\. Figure[4](https://arxiv.org/html/2609.03079#S2.F4)\(a\) shows strong spatial heterogeneity in Mistral\-7B: even when 50% of the model weights are pinned in DRAM, cache miss rates still vary significantly across layers\. Figure[4](https://arxiv.org/html/2609.03079#S2.F4)\(b\) further shows pronounced temporal variation within a single layer across different prompts, driven by the input\-dependent nature of activation patterns\. This variability causes the timing relationship between I/O fetching and GPU computation to shift continuously at runtime\. As a result, an offline schedule can quickly become suboptimal when actual cache miss behavior deviates from the profiled average, leading to either hardware underutilization or excessive I/O stalls\. These observations show that effective coordination cannot rely solely on precomputed schedules\. Instead, LeanStream adopts an adaptive online strategy that dynamically adjusts coordination frequency based on real\-time execution feedback\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/motivation_miss_ratio_v3.png)Figure 4\.Cache miss dynamics in Mistral\-7B under a 50% weight\-cache budget\. \(a\) Cache miss rates vary significantly across layers\. \(b\) Cache miss rates for Layer 0 also fluctuate across prompts\. This variability across layers and inputs makes offline profiling unreliable and motivates adaptive online coordination\.
### 2\.3\.Prediction Overhead

LeanStream converts intermediate GPU results into weight\-priority and cache\-eviction decisions to guide loading, computation, and retention\. On mobile heterogeneous platforms, this control is typically performed on the CPU to keep the GPU dedicated to execution\. However, Table[1](https://arxiv.org/html/2609.03079#S3.T1)shows that a state\-of\-the\-art two\-layer MLP predictor on the Jetson AGX Orin CPU can take longer than the sparse GPU MLP computation\. Moreover, its cost grows linearly with finer\-grained coordination\. Thus, conventional neural predictors cannot keep pace with high\-frequency streaming, motivating a substantially lower\-latency learnable control mechanism\.

## 3\.LeanStream Design

### 3\.1\.Overview

Efficient on\-device LLM execution requires fully utilizing GPU computation, DRAM, and SSD bandwidth under tight resource constraints\. LeanStream achieves this through a fine\-grained communication substrate that supports high\-frequency coordination across heterogeneous system components\. As illustrated in Figure[5](https://arxiv.org/html/2609.03079#S3.F5), LeanStream decouples computation from the rigid layer\-by\-layer I/O schedule used in prior sparse execution pipelines\. Partial GPU results are continuously fed into the predictor to identify and reprioritize the most important weights for the next layer, allowing the storage subsystem to refine its fetch decisions on the fly\. In the opposite direction, once any subset of required weights is loaded, the GPU immediately begins executing the corresponding partial computation instead of waiting for the entire layer’s weights to arrive\. This bidirectional coordination overlaps computation and I/O more effectively, reducing stalls and improving end\-to\-end throughput\.

We introduce the thread\-block level fine\-grained synchronization method and the streaming control strategy in Section[3\.2](https://arxiv.org/html/2609.03079#S3.SS2)\. Next, we propose lightweight system control by proposing a stacked learnable hashing method in Section[3\.3](https://arxiv.org/html/2609.03079#S3.SS3)\.

Table 1\.GPU Computation vs\. Prediction Overhead\. CPU predictor follows the two\-layer MLP design in state\-of\-the\-art works\([Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1);[Song et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib7)\)\. Latency is measured on Jetson AGX Orin for Mistral\-7B with 80% sparsity\.
### 3\.2\.Fine\-Grained Streaming Control

To reduce redundancy and idle time, LLM inference should decouple GPU computation from I/O dependencies through fine\-grained, context\-aware coordination\. Conventional approaches rely on kernel partitioning and global device synchronization, which incur substantial overhead\.

We propose a thread\-block level communication mechanism that is both non\-blocking and asymmetric\. This method leverages the unified DRAM memory architecture of SoCs to enable direct inter\-device coordination without kernel splitting\. In Section[3\.2\.1](https://arxiv.org/html/2609.03079#S3.SS2.SSS1), we define the primitives for this thread\-block level communication\. Section[3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2)analyzes the overhead associated with LLM execution under this communication model\. Finally, Section[3\.2\.3](https://arxiv.org/html/2609.03079#S3.SS2.SSS3)presents a strategy for dynamically adjusting the communication frequency to optimize the overall LLM inference stream\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/overview_V10.png)Figure 5\.The overview of LeanStream Framework\.#### 3\.2\.1\.Lightweight Coordination Primitives\.

To achieve coordination that is context\-aware rather than strictly bound to the execution of a specific GPU kernel, we must intervene within the kernel itself, enabling the internal scheduling of the kernel to interact with external information\. Furthermore, we seek a flexible communication mechanism that avoids the mutual waiting inherent in global device\-level synchronization\. This approach does not require strictly peer\-to\-peer or symmetric information exchange\. For example, the I/O unit is not mandated to respond to every individual GPU instruction and is permitted to react to multiple GPU signals simultaneously\. By allowing such decoupled interactions, the LLM inference stream becomes more flexible and the mutual interference between devices is minimized\.

To address this issue, we propose a thread\-block level communication mechanism that is non\-blocking, asymmetric, and fine\-grained\. As illustrated in Figure[6](https://arxiv.org/html/2609.03079#S3.F6), we leverage the unified memory architecture of the SoC, which allows the GPU and CPU to access the same data simultaneously\. We insert flag checks before the execution of a GPU block to verify whether the corresponding I/O block has been loaded by the CPU\. Furthermore, once a block completes its computation, it updates a flag while writing its partial results back to DRAM\. This design allows the GPU to utilize already loaded weights for computation while the I/O unit concurrently loads the remaining data for the current layer, ensuring that the transfer between the GPU and I/O blocks remains asynchronous as shown in Figure[6](https://arxiv.org/html/2609.03079#S3.F6)\(a\)\. Similarly, during I/O idle cycles, the CPU can verify completed computation results\. If multiple GPU thread blocks have finished their tasks, the CPU can aggregate these results to predict and update the I/O selection\. As shown in Figure[6](https://arxiv.org/html/2609.03079#S3.F6)\(b\), the CPU operates asynchronously with GPU computation and can process multiple GPU results within a single update cycle\.

As illustrated in Algorithm[1](https://arxiv.org/html/2609.03079#algorithm1), our mechanism enables a fine\-grained, asynchronous flow by embedding synchronization logic directly within GPU thread\-blocks\. In this model, each block’s "leader thread" performs a non\-blocking check ongpu\_rd\_flg\. If the required weights are pre\-loaded in DRAM, the block immediately initiates computation, bypassing device synchronization stalls\. Upon completion, the block writes back partial results and signals its status viagpu\_wr\_flg, maintaining kernel persistence while providing the CPU with real\-time state visibility\. The I/O worker operates in a complementary fashion by monitoringio\_rd\_flgfor pending prediction tasks\. To maintain internal consistency, apthread\_barriersynchronizes worker lanes before and after parallel I/O submissions\. Within each worker,lane 0manages global atomic flags and triggers sibling threads via a localgroup\_govariable for efficient submission\. Once asynchronous I/O events complete, the worker updatesio\_wr\_flgto release waiting GPU blocks\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/sync5.png)Figure 6\.Thread\-block–level, non\-blocking, asymmetric coordination mechanism for LLM streaming\. \(a\) GPU thread\-blocks perform non\-blocking checks on fine\-grained flags to consume weights that may have been pre\-loaded into DRAM, decoupling weight arrival from compute execution\. \(b\) The CPU asynchronously monitors completed thread\-blocks and aggregates multiple partial results to update I/O selection, supporting an asymmetric mapping where one CPU update responds to an arbitrary number of GPU signals\.Algorithm 1Lightweight Coordination MechanismInput:Atomic flagsgpu\_rd\_flg,gpu\_wr\_flg,io\_rd\_flg,io\_wr\_flg, groupG, layerL, iterationN

Output:Synchronized execution across GPU compute and libaio workers

1FunctionGPU\_Kernel\(layer\_id, group\_id\)

2if*get\_local\_id\(0\) == 0*then

3while*atomic\_load\(gpu\_rd\_flg\[layer\_id\]\[group\_id\]\)≠\\neqready*do

4spin\_wait\(\);

5barrier\(CLK\_GLOBAL\_MEM\_FENCE\);

6/\* Execute sparse computation kernels \*/;

7barrier\(CLK\_GLOBAL\_MEM\_FENCE\);

8if*get\_local\_id\(0\) == 0*then

9atomic\_store\(gpu\_wr\_flg\[layer\_id\]\[group\_id\], 1\);

10FunctionI/O\_Worker\(layer\_id, group\_id, lane\_id\)

11for*it←0\\leftarrow 0toN*do

12pthread\_barrier\_wait\(&iter\_start\_barrier\);if*lane\_id == 0*then

13while*atomic\_load\(io\_rd\_flg\[layer\_id\]\[group\_id\]\)≠\\neqit*do

14spin\_wait\(\);

15group\_go\[group\_id\]←1\\leftarrow 1;

16else

17while*group\_go\[group\_id\] == 0*do

18spin\_wait\(\);

19io\_submit\(ctx, BLK\_PER\_THR, cbs\);

20if*lane\_id == 0*then

21atomic\_store\(io\_wr\_flg\[layer\_id\]\[group\_id\], 1\);

22group\_go\[group\_id\]←0\\leftarrow 0;

23pthread\_barrier\_wait\(&iter\_end\_barrier\);

Crucially, the relationship betweengpu\_wr\_flgandio\_rd\_flgis not necessarily a one\-to\-one mapping, reflecting the non\-symmetric nature of our coordination substrate\. To maximize throughput, the system allows for a many\-to\-one correspondence where multiple completedgpu\_wr\_flgsignals can be aggregated into a singleio\_rd\_flgupdate for a collective I/O prediction\. Conversely, a single I/O completion signal may resolve the dependencies for multiple GPU thread\-blocks simultaneously\. This flexibility allows the communication frequencies of computation and I/O to differ, further decoupling the execution progress of individual hardware units\.

#### 3\.2\.2\.Analysis of Coordination Overheads\.

While our fine\-grained coordination primitive makes speculate\-and\-refine execution possible, prioritized loading and computation introduce additional overhead\. In this section, we analyze how these mechanisms affect both I/O and computation, and use the resulting insights to motivate the coordination\-scheduling design in the next section\.

Prioritizing weight fetching based on real\-time importance scores inherently disrupts the sequential access patterns of both storage and computation\. This is because the scheduler issues I/O requests according to predicted activation magnitudes rather than the physical layout of weights on disk\. As a result, out\-of\-order data movement can increase the number of I/O operations, reduce effective I/O block size, and degrade storage\-bandwidth utilization\([Jeong et al\., 2013](https://arxiv.org/html/2609.03079#bib.bib62);[Agrawal et al\., 2008](https://arxiv.org/html/2609.03079#bib.bib64);[Ji et al\., 2016](https://arxiv.org/html/2609.03079#bib.bib63)\)\. It also complicates the downstream computation pipeline\. To mitigate these overheads, we develop two specialized strategies\.

1\. I/O Placement with Co\-Activated Neurons\.To mitigate the bandwidth loss caused by more frequent I/O requests and smaller transfer sizes, we optimize the physical placement of weight matrices according to neuron co\-activation patterns\. While prior work has noted similar effects\([Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8);[Alizadeh et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib6)\), existing methods largely rely on pairwise co\-activation statistics and thus miss the higher\-order structure required for segment\-level I/O placement\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/control9.png)Figure 7\.The Streaming Control Strategy\.We address this by formulating I/O grouping as a distributional representation learning problem over neuron activation contexts, inspired by word embedding and contrastive representation learning\([Mikolov et al\., 2013](https://arxiv.org/html/2609.03079#bib.bib65);[Pennington et al\., 2014](https://arxiv.org/html/2609.03079#bib.bib66);[Salakhutdinov and Hinton, 2007](https://arxiv.org/html/2609.03079#bib.bib67)\)\. Instead of using heuristic grouping rules, we learn a neuron embedding space for I/O\-aware organization in sparse inference\. The offline embedding procedure is simple: neurons that co\-activate are treated as positive pairs, while neurons that do not co\-activate are treated as negative pairs\. For a neuronnnwith learnable embeddingZnZ\_\{n\}, we sample a positive neuronn\+n^\{\+\}and a negative neuronn−n^\{\-\}with learnable embeddingsZn\+Z\_\{n^\{\+\}\}andZn−Z\_\{n^\{\-\}\}, respectively\. The objective minimizes the distance betweenZnZ\_\{n\}andZn\+Z\_\{n^\{\+\}\}while maximizing the distance toZn−Z\_\{n^\{\-\}\}, as follows:

ℒ=min\{Zn,Zn\+,Zn−\}log\(σ\(−ZnT⋅Zn\+\)\)\+log\(σ\(ZnT⋅Zn−\)\)\\mathcal\{L\}=\\min\_\{\\\{Z\_\{n\},Z\_\{n^\{\+\}\},Z\_\{n^\{\-\}\}\\\}\}\\log\(\\sigma\(\-Z\_\{n\}^\{T\}\\cdot Z\_\{n^\{\+\}\}\)\)\+\\log\(\\sigma\(Z\_\{n\}^\{T\}\\cdot Z\_\{n^\{\-\}\}\)\)whereσ⁡\(⋅\)\\sigma\(\\cdot\)is a sigmoid function\. The learned embeddings capture higher\-order co\-activation structure automatically, after which standard clustering can be used to organize neurons for I/O\-aware placement\. In practice, we group three weights into a block for Qwen2\.5\-7B and two weights into a block for Mistral\-7B and Llama2\-7B\. These settings are used as the default configuration in our experiments\.

2\. Permutation\-Invariant Execution\.On the computation side, out\-of\-order weight arrival would normally require complex indexing logic or dynamic kernel reconstruction\([Kjolstad et al\., 2017](https://arxiv.org/html/2609.03079#bib.bib9);[Liu et al\., 2024a](https://arxiv.org/html/2609.03079#bib.bib10)\)\. LeanStream avoids this overhead by exploiting the permutation invariance of the MLP hidden dimension\. For a standard SwiGLU layer, lethσ⁡\(j\)h\_\{\\sigma\(j\)\}be thejj\-th hidden neuron computed under permutationσ\\sigma:

h\[σ⁡\(j\)\]=SiLU\(X\(W1\)\[:,σ\(j\)\]\)⋅\(X\(Wg​a​t​e\)\[:,σ\(j\)\]\)h\_\{\[\\sigma\(j\)\]\}=\\mathrm\{SiLU\}\(X\(W\_\{1\}\)\_\{\[:,\\sigma\(j\)\]\}\)\\cdot\(X\(W\_\{gate\}\)\_\{\[:,\\sigma\(j\)\]\}\)The output at coordinatemmis the sum over these neurons:

Y^\[:,m\]=∑j=1kh\[σ⁡\(j\)\]\(W2\)\[σ⁡\(j\),m\]=∑t=1kh\[t\]\(W2\)\[t,m\]=Y\[:,m\]\\hat\{Y\}\_\{\[:,m\]\}=\\sum\_\{j=1\}^\{k\}h\_\{\[\\sigma\(j\)\]\}\(W\_\{2\}\)\_\{\[\\sigma\(j\),m\]\}=\\sum\_\{t=1\}^\{k\}h\_\{\[t\]\}\(W\_\{2\}\)\_\{\[t,m\]\}=Y\_\{\[:,m\]\}
This property allows LeanStream to treat out\-of\-order weights as a logically contiguous dense matrix\. Because the SwiGLU output is computed as a sum over hidden neurons, it is invariant to the internal ordering of those neurons\. As a result, the order in which weights arrive from storage can be used directly as the GPU execution order\. The system simply appends arriving weight blocks into a contiguous memory buffer in arrival order, without any re\-indexing or data reshuffling\. This enables the GPU to execute high\-performance dense kernels on the subset of neurons currently available\.

#### 3\.2\.3\.Fine\-Grained Streaming Control Strategy

Runtime variations in sparse prediction and cache misses make a fixed streaming schedule ineffective, while the coordination decision for the current layer affects subsequent\-layer execution\. LeanStream therefore formulates streaming coordination as a finite\-horizon stochastic predictive\-control problem\. Following the receding\-horizon principle of MPC\([Mayne et al\., 2000](https://arxiv.org/html/2609.03079#bib.bib3)\), LeanStream optimizes from the currently observed state and applies only the current\-layer coordination plan\. Future cache and sparse\-prediction behavior is represented by profiled probability distributions and incorporated through stochastic MPC\([Mesbah, 2016](https://arxiv.org/html/2609.03079#bib.bib4)\)\. Solving this optimization online at every layer boundary would be expensive\. Inspired by explicit MPC\([Bemporad et al\., 2002](https://arxiv.org/html/2609.03079#bib.bib5)\), LeanStream synthesizes the state\-feedback policy offline and materializes it as a lookup table\.

1\. State and coordination plan\.AtTn−1G​P​UT^\{GPU\}\_\{n\-1\}, letRnR\_\{n\}denote the selected weight volume of layernnthat remains to be loaded\. The controller state isqn=Q⁡\(Rn\)q\_\{n\}=Q\(R\_\{n\}\), whereQ⁡\(⋅\)Q\(\\cdot\)quantizes the remaining\-load volume to a finite set of controller states\.

LeanStream divides GPU computation and I/O into logical stages and assigns each stage a communication frequency,

\(3\)ϕn=\(𝝎nG,𝝎nIO\)∈Φn​\(qn\)\.\\boldsymbol\{\\phi\}\_\{n\}=\\left\(\\boldsymbol\{\\omega\}\_\{n\}^\{G\},\\boldsymbol\{\\omega\}\_\{n\}^\{\\mathrm\{IO\}\}\\right\)\\in\\Phi\_\{n\}\(q\_\{n\}\)\.These frequencies determine the communication points that partition GPU and I/O into runtime segments\. Lower frequencies create larger segments and reduce coordination overhead\. However, coarse GPU segments can force the GPU to wait until a larger weight chunk is loaded before proceeding, while coarse I/O segments delay the incorporation of refined GPU predictions\. Higher frequencies allow the GPU to start computation earlier and the I/O to incorporate refined GPU predictions sooner, but incur greater coordination overhead\. LeanStream selectsϕn\\boldsymbol\{\\phi\}\_\{n\}to balance these trade\-offs\.

2\. Execution model and stochastic state transition\.Given the controller stateqnq\_\{n\}atTn−1G​P​UT^\{GPU\}\_\{n\-1\}and a candidate coordination planϕ\\boldsymbol\{\\phi\}, we derive the next stateqn\+1q\_\{n\+1\}atTnG​P​UT^\{GPU\}\_\{n\}from the execution between the two time points and the resulting remaining I/O workload\. We divide this execution into two intervals\. Interval I ends when the remaining layer\-nnI/O completes\. Interval II ends when the layer\-nnGPU computation completes\.

During Interval I, layer\-nnI/O does not depend on GPU progress because its final prediction has already been determined atTn−1G​P​UT^\{GPU\}\_\{n\-1\}\. Thus, its duration is modeled asT¯nI​\(qn,𝝎nIO\)\\bar\{T\}\_\{n\}^\{\\mathrm\{I\}\}\(q\_\{n\},\\boldsymbol\{\\omega\}\_\{n\}^\{\\mathrm\{IO\}\}\)\. Each GPU segment can execute only after its required weight volume has been loaded\. These dependencies determine the GPU work completed by the end of Interval I, denoted asC¯nG,I​\(T¯nI,qn,ϕn\)\\bar\{C\}\_\{n\}^\{G,\\mathrm\{I\}\}\(\\bar\{T\}\_\{n\}^\{\\mathrm\{I\}\},q\_\{n\},\\boldsymbol\{\\phi\}\_\{n\}\)\. LetCnGC\_\{n\}^\{G\}denote the fixed total GPU workload of layernn\. The residual GPU workload for Interval II is thereforeC¯nG,II=CnG−C¯nG,I\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\}=C\_\{n\}^\{G\}\-\\bar\{C\}\_\{n\}^\{G,\\mathrm\{I\}\}\.

At the beginning of Interval II, all weights required by layer\-nncomputation have been loaded\. The GPU therefore executes the residual work without current\-layer I/O dependencies, with durationT¯nII​\(C¯nG,II,𝝎nG\)\\bar\{T\}\_\{n\}^\{\\mathrm\{II\}\}\(\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\},\\boldsymbol\{\\omega\}\_\{n\}^\{G\}\), including the segmentation and communication overhead induced by the GPU coordination frequency\.

Therefore, the predicted execution time fromTn−1G​P​UT^\{GPU\}\_\{n\-1\}toTnG​P​UT^\{GPU\}\_\{n\}is

\(4\)T¯n​\(q,ϕ\)=T¯nI\+T¯nII\.\\bar\{T\}\_\{n\}\(q,\\boldsymbol\{\\phi\}\)=\\bar\{T\}\_\{n\}^\{\\mathrm\{I\}\}\+\\bar\{T\}\_\{n\}^\{\\mathrm\{II\}\}\.Given\(qn,ϕn\)\(q\_\{n\},\\boldsymbol\{\\phi\}\_\{n\}\), the current\-layer execution time is deterministic\. LeanStream profiles all feasible\(q,ϕ\)\(q,\\boldsymbol\{\\phi\}\)configurations offline and stores the correspondingT¯I\\bar\{T\}^\{\\mathrm\{I\}\},T¯II\\bar\{T\}^\{\\mathrm\{II\}\},C¯G,I\\bar\{C\}^\{G,\\mathrm\{I\}\}, andC¯G,II\\bar\{C\}^\{G,\\mathrm\{II\}\}\.

During Interval II, LeanStream prefetches layer\-\(n\+1\)\(n\+1\)weights untilTnG​P​UT^\{GPU\}\_\{n\}\. We are interested in the remaining weight volume that still needs to be loaded at this point\. However, even for a fixed layernn, residual GPU workloadC¯nG,II\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\}, and coordination planϕn\\boldsymbol\{\\phi\}\_\{n\}, different prompts can produce different predictions and cache\-miss patterns\. We therefore model the remaining I/O volume as a conditional distribution:

\(5\)Rn\+1∼𝒟n,C¯nG,II,ϕn,qn\+1=Q⁡\(Rn\+1\),R\_\{n\+1\}\\sim\\mathcal\{D\}\_\{n,\\,\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\},\\,\\boldsymbol\{\\phi\}\_\{n\}\},\\qquad q\_\{n\+1\}=Q\(R\_\{n\+1\}\),whereRn\+1R\_\{n\+1\}denotes the remaining layer\-\(n\+1\)\(n\+1\)weight volume atTnG​P​UT^\{GPU\}\_\{n\}with respect to the final prediction\.

In implementation, we quantizeC¯nG,II\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\}into 16 bins and collect 20 samples for each\(n,C¯nG,II,ϕn\)\(n,\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\},\\boldsymbol\{\\phi\}\_\{n\}\)configuration\. Profiling all layers on Jetson AGX Orin takes approximately 90 hours, but this process is performed entirely offline and introduces no runtime decision overhead\. With the deterministic mapping from\(qn,ϕn\)\(q\_\{n\},\\boldsymbol\{\\phi\}\_\{n\}\)toC¯nG,II\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\}, this conditional distribution defines the stochastic state transition fromqnq\_\{n\}toqn\+1q\_\{n\+1\}\.

3\. Finite\-horizon stochastic optimization and explicit policy\.The conditional distribution above induces a state\-action\-dependent stochastic transition\. For compactness, we denote

\(6\)𝒟n​\(q,ϕ\)≜𝒟n,C¯nG,II,ϕ𝒏,\\mathcal\{D\}\_\{n\}\(q,\\boldsymbol\{\\phi\}\)\\triangleq\\mathcal\{D\}\_\{n,\\,\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\},\\,\\boldsymbol\{\\phi\_\{n\}\}\},whereC¯nG,II\\bar\{C\}\_\{n\}^\{G,\\mathrm\{II\}\}is deterministically determined by\(q,ϕ\)\(q,\\boldsymbol\{\\phi\}\)\.

LeanStream minimizes the expected latency over horizonHHusing stochastic dynamic programming with Bellman backward recursion\([Marescot et al\., 2013](https://arxiv.org/html/2609.03079#bib.bib2)\):

\(7\)VnH\(q\)=minϕ∈Φn​\(q\)\[\\displaystyle V\_\{n\}^\{H\}\(q\)=\\min\_\{\\boldsymbol\{\\phi\}\\in\\Phi\_\{n\}\(q\)\}\\Big\[T¯n​\(q,ϕ\)\\displaystyle\\bar\{T\}\_\{n\}\(q,\\boldsymbol\{\\phi\}\)\+𝔼Rn\+1∼𝒟n​\(q,ϕ\)\[Vn\+1H−1\(Q\(Rn\+1\)\)\]\],\\displaystyle\+\\mathbb\{E\}\_\{R\_\{n\+1\}\\sim\\mathcal\{D\}\_\{n\}\(q,\\boldsymbol\{\\phi\}\)\}\\left\[V\_\{n\+1\}^\{H\-1\}\(Q\(R\_\{n\+1\}\)\)\\right\]\\Big\],withVn0​\(q\)=0V\_\{n\}^\{0\}\(q\)=0\. The transition distributions are empirically estimated from 20 profiled samples per configuration\. The minimizing action is computed offline and stored as

\(8\)LUT⁡\[n,q\]=ϕn∗,H​\(q\)\.\\mathrm\{LUT\}\[n,q\]=\\boldsymbol\{\\phi\}\_\{n\}^\{\*,H\}\(q\)\.
4\. Online lookup and application\.AtTn−1G​P​UT^\{GPU\}\_\{n\-1\}, LeanStream measures the realized remaining\-load volume and applies

\(9\)ϕn=LUT⁡\[n,Q⁡\(Rn\)\]\.\\boldsymbol\{\\phi\}\_\{n\}=\\mathrm\{LUT\}\\left\[n,Q\(R\_\{n\}\)\\right\]\.Only the current\-layer coordination plan is executed\. AtTnG​P​UT^\{GPU\}\_\{n\}, LeanStream observes the new realized remaining\-load state and queries the corresponding policy\. The prediction horizon then advances by one layer\. The stochastic optimization is performed offline, while runtime control requires only state measurement and table lookup\.

### 3\.3\.Lightweight System Control with Stacked Learnable Hashing

Our speculate\-and\-refine framework requires a lightweight control mechanism that can frequently use partial GPU computation results to generate control signals for data and computation prioritization\. Specifically, it must estimate the relative ordering of activation magnitudes across the relevant weight sub\-matrices and sub\-computations in activation\-sparse LLM inference, so as to guide I/O prefetching, GPU execution, and in\-memory cache eviction\. As discussed in Section[2\.3](https://arxiv.org/html/2609.03079#S2.SS3), conventional neural predictors are poorly suited for this role: they are too slow for high\-frequency coordination and too memory\-intensive for resource\-constrained devices\. Prior systems report neural predictors consuming more than 1 GB of memory\([Song et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib7);[Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8);[Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1)\)\. Locality\-Sensitive Hashing \(LSH\) offers an attractive alternative because of its low computational and memory overhead\. It has been successfully applied to approximate nearest\-neighbor search\([Indyk and Motwani, 1998](https://arxiv.org/html/2609.03079#bib.bib32);[Gionis et al\., 1999](https://arxiv.org/html/2609.03079#bib.bib33);[Yagnik et al\., 2011](https://arxiv.org/html/2609.03079#bib.bib34)\), large matrix\-multiplication approximation\([Zeng et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib35);[Blalock and Guttag, 2021](https://arxiv.org/html/2609.03079#bib.bib36);[Wei et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib37);[Tang et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib44)\), and even large\-scale neural network training\([Spring and Shrivastava, 2017](https://arxiv.org/html/2609.03079#bib.bib38);[Chen et al\., 2020b](https://arxiv.org/html/2609.03079#bib.bib39);[Chen et al\., 2020a](https://arxiv.org/html/2609.03079#bib.bib40)\)\. Moreover, adaptive hash functions can further reduce LSH query cost in data\-dependent or learned hashing settings\([Andoni and Razenshteyn, 2015](https://arxiv.org/html/2609.03079#bib.bib41);[Dong et al\., 2019](https://arxiv.org/html/2609.03079#bib.bib42);[Andoni et al\., 2015](https://arxiv.org/html/2609.03079#bib.bib43)\)\.

Our key insight is to view learnable LSH as a differentiable indexing\-and\-lookup\-table primitive, where the adaptive hash function provides a learnable index and the table stores learnable binary features\. This allows learnable LSH to replace standard neural primitives such as a single\-layer MLP, but with much lower latency and memory cost\. Stacking such LSH layers increases model capacity through successive nonlinear lookups while retaining efficient CPU inference, since the computation is dominated by bitwise operations and in\-register table accesses\([Wei et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib37);[Blalock and Guttag, 2017](https://arxiv.org/html/2609.03079#bib.bib45);[Wang et al\., 2017](https://arxiv.org/html/2609.03079#bib.bib46)\)\. In addition, when cast as a classification problem, the predictor yields compact outputs, requiring onlylog⁡\(n\)\\log\(n\)bits for annn\-way decision space\. In contrast to conventional neural\-network quantization or binarization\([Hubara et al\., 2016](https://arxiv.org/html/2609.03079#bib.bib52);[Xiao et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib57)\), our approach goes beyond reducing arithmetic precision by replacing much of the neural computation itself with learned indexing and lookup\.

#### 3\.3\.1\.Stacked Learnable Hashing

For thekk\-th learnable hashing layer, we denote the input by𝐱k\\mathbf\{x\}\_\{k\}and the output by𝐲k\\mathbf\{y\}\_\{k\}\. For all layers except the first, the input𝐱k\\mathbf\{x\}\_\{k\}is a binary representation\. Letfkf\_\{k\}be the hash function associated with layerkk, and let𝐓k∈\{±1\}2τ×d\\mathbf\{T\}\_\{k\}\\in\\\{\\pm 1\\\}^\{2^\{\\tau\}\\times d\}be a hash table consisting of2τ2^\{\\tau\}buckets, each represented by a*learnable*dd\-dimensional binary vector\. The layer output is then defined as

\(10\)𝐲k=𝐓k​\[ϕτ​\(fk​\(𝐱𝐤\)\)\]\\mathbf\{y\}\_\{k\}=\\mathbf\{T\}\_\{k\}\\big\[\\phi\_\{\\tau\}\\big\(f\_\{k\}\(\\mathbf\{x\_\{k\}\}\)\\big\)\\big\]whereϕτ​\(⋅\)\\phi\_\{\\tau\}\(\\cdot\)converts a binary code in\{±1\}τ\\\{\\pm 1\\\}^\{\\tau\}into the corresponding integer index in\{0,1,⋯,2τ−1\}\\\{0,1,\\cdots,2^\{\\tau\}\-1\\\}\.

In LeanStream, we adopt hyperplane hashing\([Charikar, 2002](https://arxiv.org/html/2609.03079#bib.bib68)\)to compute the hash code\. Specifically,

\(11\)fk​\(𝐱𝐤\)=sign​\(𝐱k⊛𝐖k\)f\_\{k\}\(\\mathbf\{x\_\{k\}\}\)=\\text\{sign\}\(\\mathbf\{x\}\_\{k\}\\circledast\\mathbf\{W\}\_\{k\}\)where𝐖k\\mathbf\{W\}\_\{k\}is a*learnable*matrix of shaped×τd\\times\\tau\. For all layers except the first,𝐖k\\mathbf\{W\}\_\{k\}is also binary\. Moreover,⊛\\circledastdenotes matrix multiplication implemented with XNOR and bitcount operations for all layers except the first\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/end2end_all_single_v3.png)Figure 8\.Throughput comparison under varying memory budgets on the Scrolls\-Qasper dataset across different devices and models\.Stacked learnable hashing may use either a single layer or multiple layers\. Unless otherwise noted, our default configuration consists of five hashing layers withτ=8\\tau=8andd=256d=256\. The final output layer is task\-dependent\. For classification, the design is simple: the last learnable hashing layer outputs a hash index directly, without an additional feature table, and this index is mapped to a class label\. For regression, by contrast, the output space is quantized rather than directly cast to integers\. We use equal\-probability quantization\([Gray and Neuhoff, 2002](https://arxiv.org/html/2609.03079#bib.bib50)\): we first profile the training labels, remove outliers via percentile\-based clipping, determine the effectiveminandmax, and estimate the corresponding cumulative distribution function \(CDF\)\. We then derive the quantization thresholds from the inverse CDF so that each quantization interval contains approximately the same probability mass\.

#### 3\.3\.2\.End\-to\-End Differentiable Training

To enable standard end\-to\-end supervised training with backpropagation, all components of stacked learnable hashing must be made differentiable\. For the sign function, we adopt the standard straight\-through estimator\([Bengio et al\., 2013](https://arxiv.org/html/2609.03079#bib.bib51);[Hubara et al\., 2016](https://arxiv.org/html/2609.03079#bib.bib52)\)during the backward pass\. Another key component is the lookup operation, which is not naturally differentiable\. To address this, we introduce an alternative formulation that exposes a differentiable view of the lookup and enables gradient propagation:

\(12\)ϕτ​\(fk​\(𝐱𝐤\)\)=arg​maxi⁡\(𝐙\[i\]⋅fk​\(𝐱k\)\)\\phi\_\{\\tau\}\\big\(f\_\{k\}\(\\mathbf\{x\_\{k\}\}\)\\big\)=\\argmax\_\{i\}\\big\(\\mathbf\{Z\}\_\{\[i\]\}\\cdot f\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\\big\)where𝐙∈\{±1\}2τ×τ\\mathbf\{Z\}\\in\\\{\\pm 1\\\}^\{2^\{\\tau\}\\times\\tau\}is a structured matrix whoseii\-th row satisfies𝐙\[i\]=ϕ−1​\(i\)\\mathbf\{Z\}\_\{\[i\]\}=\\phi^\{\-1\}\(i\)\. For example, whenτ=3\\tau=3,𝐙\\mathbf\{Z\}is an8×38\\times 3matrix whose rows enumerate all possible sign patterns, i\.e\.,\[\(−1,−1,−1\);\(−1,−1,\+1\);\(−1,\+1,−1\);⋯;\(\+1,\+1,\+1\)\]\[\(\-1,\-1,\-1\);\(\-1,\-1,\+1\);\(\-1,\+1,\-1\);\\cdots;\(\+1,\+1,\+1\)\]\. We then apply gumbel\-softmax\([Jang et al\., 2016](https://arxiv.org/html/2609.03079#bib.bib53)\), a standard differentiable relaxation ofarg​max\\argmax, to enable end\-to\-end training\. Importantly, the alternative formulation in Eq\. \([12](https://arxiv.org/html/2609.03079#S3.E12)\) is used only during the backward pass for gradient estimation\. At inference time, the compact binary representation is used directly, without expanding it into a2τ2^\{\\tau\}\-dim vector\.

In addition, we treat the final layer differently for classification and regression tasks\. For classification, we use the same alternative formulation as in Eq\. \([12](https://arxiv.org/html/2609.03079#S3.E12)\) and optimize it with the softmax cross\-entropy loss\. For regression, we find that applying binary cross\-entropy loss to the quantized binary representation yields better performance, because the binary code itself preserves hierarchical similarity among quantized values\.

#### 3\.3\.3\.Predictive System Control Tasks in LeanStream

LeanStream supports two predictive tasks for system control in each LLM block\. First, given a partial input feature, it predicts which neurons in the MLP output will be activated\. The resulting probabilistic predictions provide a relative priority for loading the corresponding weight sub\-matrices and for scheduling the computation of MLP sub\-slices\. Second, it predicts eviction decisions for elements in the in\-memory cache\. In the following, we describe the input features and output formats for these two tasks\.

For neuron\-activation prediction, instead of directly feeding the partial or fully updated input feature into the predictor, we first apply a PCA\-based linear transformation to perform rotational dimensionality reduction while preserving the most representative directions\([Abdi and Williams, 2010](https://arxiv.org/html/2609.03079#bib.bib58)\)\. The projection matrix is easily obtained from SVD, and the reduced feature representation can be computed alongside the original MLP execution by fusing this linear projection into the existing GPU kernel, resulting in negligible overhead\. Similar ideas have been adopted in KV\-compression systems\([Lee et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib59)\)\. In practice, we reduce the input\-feature dimension by 70%\.

For the cache\-eviction task, we formulate prediction as a regression problem that estimates the reuse distance of each cache element\. Combined with the element’s insertion time, the predicted reuse distance enables the system to determine its eviction priority\. As input features, we use the inter\-arrival times between consecutive requests to each cache object, a representation that has also been adopted in prior learning\-based cache designs\([Song et al\., 2020](https://arxiv.org/html/2609.03079#bib.bib60);[Yang et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib61)\)\.

## 4\.Evaluation

### 4\.1\.Experimental Setup

Models and Devices\.We evaluate LeanStream on three LLMs: Mistral\-7B\([Jiang et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib47)\), Llama2\-7B\([Touvron et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib48)\), and Qwen2\.5\-7B\([Yang et al\., 2025](https://arxiv.org/html/2609.03079#bib.bib49)\)\. Our testbeds include two embedded platforms, NVIDIA Jetson AGX Orin and Jetson AGX Xavier, both paired with Samsung 980 Pro SSDs, as well as a mobile platform, the OnePlus 13, featuring the Snapdragon 8 Elite chipset and UFS 4\.0 flash storage\. This setup allows us to assess LeanStream across diverse model architectures and hardware environments\.

Baselines\.We compare LeanStream against three primary baselines: DejaVu\([Liu et al\., 2023](https://arxiv.org/html/2609.03079#bib.bib1)\), which predicts and loads the current layer’s weights using information from the previous layer; PowerInfer\-2\([Xue et al\., 2024](https://arxiv.org/html/2609.03079#bib.bib8)\), which combines weight prediction with in\-memory weight caching to mitigate I/O latency; and DejaVu\+, our enhanced DejaVu variant with an added LRU cache module to support varying memory budgets\.

Datasets\.We conduct experiments on three representative generation tasks: long\-document question answering with Scrolls\-Qasper\([Shaham et al\., 2022](https://arxiv.org/html/2609.03079#bib.bib54)\), open\-ended factual generation with TruthfulQA\([Lin et al\., 2022](https://arxiv.org/html/2609.03079#bib.bib55)\), and conversational question answering with CoQA\([Reddy et al\., 2019](https://arxiv.org/html/2609.03079#bib.bib56)\)\. These tasks allow us to evaluate LeanStream across a range of context lengths and output characteristics\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/end2end_3_different_dataset_v5.png)Figure 9\.Throughput comparison of our LeanStream under varying memory budgets across different datasets when streaming is memory\-bound on Jetson AGX Orin\.
### 4\.2\.End\-to\-end Results

Overall Performance\.As shown in Figure[8](https://arxiv.org/html/2609.03079#S3.F8), we compare the performance of LeanStream and the baselines on the Scrolls\-Qasper dataset across different memory budgets and hardware platforms\. The results indicate that LeanStream consistently outperforms all baselines in every tested scenario\. Even when compared to the optimal configuration of PowerInfer\-2, where 50% of the weights are cached in memory, LeanStream achieves up to a 2\.2×\\timesimprovement in throughput\. This performance gap becomes even more significant as the memory budget decreases\. Furthermore, LeanStream can reduce the required memory budget by as much as 5\.9×\\timeswhile still matching the peak throughput achieved by PowerInfer\-2 at its optimal 50% cache setting\.

The performance characteristics vary according to the hardware capabilities of each platform\. The Jetson AGX Orin exhibits a relatively balanced ratio between computation and I/O performance\. The Jetson AGX Xavier features an identical SSD to the Orin, which results in excellent I/O capabilities\. However, its lower computation power compared to the Orin leads to a lower peak throughput\. In contrast, the OnePlus 13 utilizes UFS flash storage, which provides lower I/O bandwidth than the Jetson devices\. Consequently, the OnePlus 13 enters an I/O\-bound state much earlier as the memory budget decreases\.

![Refer to caption](https://arxiv.org/html/2609.03079v1/end2end_2_different_sparsity_v5.png)Figure 10\.End\-to\-end throughput under different sparsity levels on Qasper using Mistral\-7B\.Table 2\.Performance comparison under different sparsity levels\.Cross\-Dataset Analysis\.Since different datasets exhibit identical performance during compute\-bound periods, Figure[8](https://arxiv.org/html/2609.03079#S3.F8)presents the overall results using Scrolls\-Qasper as a representative case\. Figure[9](https://arxiv.org/html/2609.03079#S4.F9)further compares the throughput of the three datasets on Jetson AGX Orin under memory\-bound regime\. We observe that performance varies significantly depending on the combination of model and dataset even under the same memory budget\. This variation occurs because the effectiveness of the prioritization mechanism in LeanStream depends on the specific activation patterns of different dataset and model combinations, which leads to different cache miss ratios\. Consequently, these differences in cache efficiency result in distinct performance levels during the I/O\-bound phase\. In this memory\-bound regime, throughput is primarily dominated by cache misses, and the impact of device I/O bandwidth scales proportionally\. Since all devices exhibit similar performance trends, these results on the Orin platform are representative of the behavior observed on other devices\.

Table 3\.Synchronization overhead when partitioning one MLP layer of Mistral\-7B on Jetson AGX Orin and OnePlus 13\.Sparsity and Accuracy Analysis\.Table[2](https://arxiv.org/html/2609.03079#S4.T2)presents the accuracy achieved by LeanStream and the baselines across various target weight sparsity levels for each dataset\. The results demonstrate that our approach maintains accuracy levels that are nearly identical to both the original dense models and the various baselines\. Figure[10](https://arxiv.org/html/2609.03079#S4.F10)further illustrates the performance of various baselines under different memory budgets and sparsity levels for Mistral\-7B on the Scrolls\-Qasper dataset\. As sparsity decreases, the overall throughput of all models declines because lower sparsity necessitates more computation and increases the volume of weights that must be loaded\. Despite this, LeanStream maintains a performance lead across all tested sparsity levels\. Notably, when the memory budget becomes extremely small, LeanStream remains functional while the other baselines fail due to out\-of\-memory errors\.

### 4\.3\.Component Impact Study

#### 4\.3\.1\.Analysis of Synchronization Overhead

Table[3](https://arxiv.org/html/2609.03079#S4.T3)compares the synchronization overhead of our proposed method against traditional device synchronization mechanisms\. We evaluate these overheads when partitioning a single MLP layer of Mistral\-7B on both the Jetson AGX Orin and the OnePlus 13\. The results indicate that the overhead of our method is consistently lower than that of standard device synchronization\. As the number of splits increases, our approach does experience an increase in overhead due to reduced parallelism\. However, even when the number of splits reaches 16, our overhead remains less than half of the original computation time\. In contrast, the overhead associated with traditional synchronization methods exceeds the total computation time at that same split level\.

#### 4\.3\.2\.Ablation Study of Prioritization Components

![Refer to caption](https://arxiv.org/html/2609.03079v1/component_Add_prioritize.png)Figure 11\.Ablation study of prioritization components on Qasper dataset with 20% weight caching budget on Jetson AGX Orin and OnePlus 13\.Figure[11](https://arxiv.org/html/2609.03079#S4.F11)presents an ablation study of the prioritization components on the Qasper dataset, using a cache budget equal to 20% of the model weight size, on Jetson AGX Orin and OnePlus 13\. Starting from the standard DejaVu implementation, we progressively add prioritized computation, prioritized loading, and prioritized caching\. This step\-by\-step integration allows us to isolate how each component reduces redundant computation and I/O\. The results show that prioritized caching provides the largest performance gain, with an especially pronounced effect on the mobile platform where I/O bandwidth is more limited\. This improvement is consistent with the low compute\-to\-memory\-access ratio of the LLM decoding phase, in which performance is bottlenecked more by weight fetching than by arithmetic throughput\. As a result, intelligently prioritizing which weights remain in cache substantially reduces high\-latency I/O requests and yields the largest throughput gains\.

Table 4\.Cache miss ratio of different memory management schemes for Mistral\-7B on Qasper\. Memory budget denotes the fraction of model weights that can be held in memory relative to the total model weights\.Table 5\.Throughput comparison on the Qasper workload with 20% weight caching on Jetson AGX Orin\.Fixuses a default fixed streaming configuration,Best Staticuses an offline\-searched globally optimal static configuration of compute and I/O blocks, andOursapplies dynamic streaming control\.
#### 4\.3\.3\.Effectiveness of Cache Policies\.

Table[4](https://arxiv.org/html/2609.03079#S4.T4)presents the cache miss ratios for various memory management schemes using Mistral\-7B on the Qasper dataset across two memory budget configurations\. We compare LeanStream against PowerInfer\-2 and standard replacement policies including Least Frequently Used \(LFU\) and Least Recently Used \(LRU\)\. The results show that LeanStream achieves a significantly lower cache miss ratio than all other methods\. At a 25% memory budget, LeanStream maintains a miss ratio of only 0\.11, while the baseline PowerInfer\-2 and standard policies such as LRU exhibit much higher miss ratios of 0\.56 and 0\.81, respectively\. As the memory budget increases to 50%, LeanStream further reduces the cache miss ratio to 0\.05, which represents a nearly fourfold improvement over the PowerInfer\-2 baseline\. These results indicate that our prioritization mechanism is highly effective at identifying and retaining the most critical weights for inference\. By minimizing cache misses, LeanStream significantly reduces the volume that must be loaded, which directly translates to the higher throughput observed in our end\-to\-end evaluations\.

#### 4\.3\.4\.Effectiveness of Dynamic Streaming Control

Table 6\.Comparison of model size, latency, and relative loading redundancy among DNN\-based, BNN\-based, and our Stacked Learnable Hashing predictors on Jetson AGX Orin\. DNNFuses the full current context, while DNNSuses stale context\.Table[5](https://arxiv.org/html/2609.03079#S4.T5)compares the throughput on the Qasper workload with 20% weight caching on the Jetson AGX Orin to evaluate our streaming control mechanism\. The "One Shot" configuration represents a baseline approach that predicts the weight only once at the input of layer\. The "Best Static" configuration uses an offline profile to select a static generally optimized granularity for I/O and computation blocks within the stream\. The results demonstrate that our dynamic approach consistently outperforms the offline optimal setting\.

#### 4\.3\.5\.Efficiency of Stacked Learnable Hashing

Table[6](https://arxiv.org/html/2609.03079#S4.T6)evaluates our stacked learnable hashing mechanism against traditional DNN\-based and BNN\-based prediction methods across three key dimensions on the Jetson AGX Orin\. The results demonstrate that our hashing approach provides a superior balance of efficiency and performance\. In terms of memory footprint, our method achieves a remarkable reduction, requiring only approximately 23 MB to 24 MB across all tested models\. This represents a significant improvement over the 1\.1 GB to 1\.8 GB required by DNN predictors and even the 106 MB to 130 MB required by BNN implementations\.

Furthermore, our approach exhibits the lowest inference latency, processing predictions in under 100μ\\mus\. While BNNs show slightly lower relative loading redundancy in some cases, LeanStream maintains a highly competitive redundancy level between 12% and 14%\. By drastically lowering both size and latency while maintaining effective weight selection, our stacked learnable hashing ensures a highly efficient prediction pipeline that minimizes resource contention on constrained platforms\.

#### 4\.3\.6\.Validation across Refinement Steps

To evaluate progressive refinement, we partition each sparse MLP into eight priority\-ordered stages and update the next\-layer prediction after each stage\. We report Importance Ratio, the normalized importance captured by the predicted sparse set, and Top\-10% Recall\. Figure[12](https://arxiv.org/html/2609.03079#S4.F12)\(a\) shows that both metrics improve with progressive refinement\. Compared with an otherwise identical non\-priority execution order, Priority ordering provides the largest early\-stage gain because high\-importance neurons are computed first, while the gap narrows as both orders approach the complete sparse MLP output\. Figure[12](https://arxiv.org/html/2609.03079#S4.F12)\(b\) shows the same trend across five representative layers\.

#### 4\.3\.7\.Analysis of Impact of Prefill Phase

![Refer to caption](https://arxiv.org/html/2609.03079v1/coqa_refinement_gamma_v4.png)Figure 12\.Progressive prediction refinement for Mistral\-7B on CoQA under 80% activation sparsity\. LeanStream divides the MLP computation into eight stages and executes them in descending priority order\. \(a\) Top\-10% recall and importance ratio averaged across all transformer layers\. \(b\) Per\-layer importance ratio for different layers\. 0/8 represents the initial one\-shot prediction, while 8/8 uses the complete MLP output\.![Refer to caption](https://arxiv.org/html/2609.03079v1/prefill_decode.png)Figure 13\.Impact of various input prefill lengths on total inference latency on the Qasper dataset using Mistral\-7B on Jetson AGX Orin and OnePlus 13 with a fixed output of 256 tokens\.Figure[13](https://arxiv.org/html/2609.03079#S4.F13)illustrates the impact of input prefill length on the total inference latency of Mistral\-7B on Jetson AGX Orin and OnePlus 13\. The results show that prefill overhead is mainly determined by the initial dense weight loading and the corresponding computation\. While prefill may involve a large number of input tokens, it has a substantially higher compute\-to\-I/O ratio than decoding\. Even for a 256\-token prompt followed by 256\-token generation, decoding remains the dominant contributor to total inference latency\. This suggests that, on edge devices, the sequential and memory\-intensive decoding phase is a more critical bottleneck than prompt prefill\.

#### 4\.3\.8\.Energy Consumption and Thermal Behavior

Table[7](https://arxiv.org/html/2609.03079#S5.T7)reports the energy consumption and thermal behavior during sustained inference\. To eliminate variations caused by DVFS and ensure a fair comparison, we fix the CPU and GPU frequencies at 2201 MHz and 713 MHz, respectively, for all systems\. LeanStream consumes slightly more energy per token than PowerInfer\-2 and DejaVu because it maintains higher CPU and GPU utilization to overlap computation with weight loading\. This also leads to higher peak CPU and GPU temperatures\. LeanStream reaches a peak GPU temperature of76∘76^\{\\circ\}C, which remains below the99∘99^\{\\circ\}C thermal\-throttling threshold\. None of the evaluated systems experiences thermal throttling during the two\-hour execution\.

## 5\.Conclusion

We presented*LeanStream*, a streaming speculate\-and\-refine framework for on\-device LLM inference on resource\-constrained mobile and embedded platforms\. By progressively refining execution, loading, and cache decisions from partial GPU results, LeanStream resolves the tension between accurate context\-aware prediction and efficient computation\.

Table 7\.Energy consumption and thermal behavior on Jetson AGX Orin\. Mistral\-7B on Scrolls\-Qasper with 3 GB memory budget\. Each system runs continuously for two hours at an ambient temperature of21∘21^\{\\circ\}C\.
## 6\.Disclaimer

This paper was prepared for informational purposes with contributions from the Global Technology Applied Research center of JPMorgan Chase & Co\. \(JPMC\) and is not a product of its, or its affiliates’, Research Departments\. JPMC and its affiliates make no representations or warranties, express or implied, regarding the completeness, accuracy, or reliability of the information herein, and accept no liability for its use or any related outcomes\. This document does not constitute investment advice, financial research, or a recommendation or offer to buy or sell any security, financial instrument, product, or service\.

## 7\.Acknowledgements

This work is in part supported by the National Science Foundation grants III\-2107200, CNS\-2038658 and CNS\-2038923\.

## References

- H\. Abdi and L\. J\. WilliamsPrincipal component analysis\.Wiley interdisciplinary reviews: computational statistics2\(4\),pp\. 433–459\.Cited by:[§3\.3\.3](https://arxiv.org/html/2609.03079#S3.SS3.SSS3.p2.1)\.
- Agrawalet al\.\(2008\)N\. Agrawal, V\. Prabhakaran, T\. Wobber, J\. D\. Davis, M\. Manasse, and R\. PanigrahyDesign tradeoffs for\{\\\{ssd\}\\\}performance\.In2008 USENIX Annual Technical Conference \(USENIX ATC 08\),Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p2.1)\.
- Alizadehet al\.\(2024\)K\. Alizadeh, S\. I\. Mirzadeh, D\. Belenko, S\. Khatamifard, M\. Cho, C\. C\. Del Mundo, M\. Rastegari, and M\. FarajtabarLlm in a flash: efficient large language model inference with limited memory\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12562–12584\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p1.1),[§1](https://arxiv.org/html/2609.03079#S1.p11.1),[§1](https://arxiv.org/html/2609.03079#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1),[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p3.1)\.
- Andoniet al\.\(2015\)A\. Andoni, P\. Indyk, T\. Laarhoven, I\. Razenshteyn, and L\. SchmidtPractical and optimal lsh for angular distance\.Advances in neural information processing systems28\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Andoni and Razenshteyn \(2015\)A\. Andoni and I\. RazenshteynOptimal data\-dependent hashing for approximate near neighbors\.InProceedings of the forty\-seventh annual ACM symposium on Theory of computing,pp\. 793–801\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Baeet al\.\(2021\)J\. Bae, J\. Lee, Y\. Jin, S\. Son, S\. Kim, H\. Jang, T\. J\. Ham, and J\. W\. Lee\{\\\{flashneuron\}\\\}:\{\\\{ssd\-Enabled\}\\\}\{\\\{large\-batch\}\\\}training of very deep neural networks\.In19th USENIX conference on file and storage technologies \(FAST 21\),pp\. 387–401\.Cited by:[§2\.2](https://arxiv.org/html/2609.03079#S2.SS2.p1.1)\.
- Bemporadet al\.\(2002\)A\. Bemporad, M\. Morari, V\. Dua, and E\. N\. PistikopoulosThe explicit linear quadratic regulator for constrained systems\.Automatica38\(1\),pp\. 3–20\.Cited by:[§3\.2\.3](https://arxiv.org/html/2609.03079#S3.SS2.SSS3.p1.1)\.
- Bengioet al\.\(2013\)Y\. Bengio, N\. Léonard, and A\. CourvilleEstimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§3\.3\.2](https://arxiv.org/html/2609.03079#S3.SS3.SSS2.p1.1)\.
- Blalock and Guttag \(2021\)D\. Blalock and J\. GuttagMultiplying matrices without multiplying\.InInternational Conference on Machine Learning,pp\. 992–1004\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Blalock and Guttag \(2017\)D\. W\. Blalock and J\. V\. GuttagBolt: accelerated data mining with fast vector compression\.InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 727–735\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p2.1)\.
- Charikar \(2002\)M\. S\. CharikarSimilarity estimation techniques from rounding algorithms\.InProceedings of the thiry\-fourth annual ACM symposium on Theory of computing,pp\. 380–388\.Cited by:[§3\.3\.1](https://arxiv.org/html/2609.03079#S3.SS3.SSS1.p2.1)\.
- Chenet al\.\(2020a\)B\. Chen, Z\. Liu, B\. Peng, Z\. Xu, J\. L\. Li, T\. Dao, Z\. Song, A\. Shrivastava, and C\. ReMongoose: a learnable lsh framework for efficient neural network training\.InInternational Conference on Learning Representations,Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Chenet al\.\(2020b\)B\. Chen, T\. Medini, J\. Farwell, C\. Tai, A\. Shrivastava,et al\.Slide: in defense of smart algorithms over hardware acceleration for large\-scale deep learning systems\.Proceedings of Machine Learning and Systems2,pp\. 291–306\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Chenet al\.\(2024\)C\. Chen, X\. Li, Q\. Zhu, J\. Duan, P\. Sun, X\. Zhang, and C\. YangCentauri: enabling efficient scheduling for communication\-computation overlap in large model training via communication partitioning\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 178–191\.Cited by:[§2\.2](https://arxiv.org/html/2609.03079#S2.SS2.p1.1)\.
- Chenet al\.\(2025\)H\. Chen, W\. Xie, B\. Zhang, J\. Tang, J\. Wang, J\. Dong, S\. Chen, Z\. Yuan, C\. Lin, C\. Qiu,et al\.Ktransformers: unleashing the full potential of cpu/gpu hybrid inference for moe models\.InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles,pp\. 1014–1029\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p2.1)\.
- Chenet al\.\(2026\)J\. Chen, C\. Du, R\. Liu, S\. Yao, D\. Yan, J\. Liao, S\. Liu, F\. Wu, and G\. ChenTokenFlow: responsive llm text streaming serving under request burst via preemptive scheduling\.InProceedings of the 21st European Conference on Computer Systems,pp\. 497–513\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p2.1)\.
- Donget al\.\(2019\)Y\. Dong, P\. Indyk, I\. Razenshteyn, and T\. WagnerLearning space partitions for nearest neighbor search\.arXiv preprint arXiv:1901\.08544\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Federiciet al\.\(2025\)M\. Federici, D\. Belli, M\. Van Baalen, A\. Jalalirad, A\. Skliar, B\. Major, M\. Nagel, and P\. WhatmoughEfficient llm inference using dynamic input pruning and cache\-aware masking\.Proceedings of Machine Learning and Systems7\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p1.1),[§1](https://arxiv.org/html/2609.03079#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Gioniset al\.\(1999\)A\. Gionis, P\. Indyk, R\. Motwani,et al\.Similarity search in high dimensions via hashing\.InVldb,Vol\.99,pp\. 518–529\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Gray and Neuhoff \(2002\)R\. M\. Gray and D\. L\. NeuhoffQuantization\.IEEE transactions on information theory44\(6\),pp\. 2325–2383\.Cited by:[§3\.3\.1](https://arxiv.org/html/2609.03079#S3.SS3.SSS1.p3.1)\.
- Guoet al\.\(2023\)L\. Guo, W\. Choe, and F\. X\. LinSti: turbocharge nlp inference at the edge via elastic pipelining\.InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,pp\. 791–803\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p2.1)\.
- Hubaraet al\.\(2016\)I\. Hubara, M\. Courbariaux, D\. Soudry, R\. El\-Yaniv, and Y\. BengioBinarized neural networks\.Advances in neural information processing systems29\.Cited by:[§3\.3\.2](https://arxiv.org/html/2609.03079#S3.SS3.SSS2.p1.1),[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p2.1)\.
- Indyk and Motwani \(1998\)P\. Indyk and R\. MotwaniApproximate nearest neighbors: towards removing the curse of dimensionality\.InProceedings of the thirtieth annual ACM symposium on Theory of computing,pp\. 604–613\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Janget al\.\(2016\)E\. Jang, S\. Gu, and B\. PooleCategorical reparameterization with gumbel\-softmax\.arXiv preprint arXiv:1611\.01144\.Cited by:[§3\.3\.2](https://arxiv.org/html/2609.03079#S3.SS3.SSS2.p1.2)\.
- Jeonget al\.\(2013\)S\. Jeong, K\. Lee, S\. Lee, S\. Son, and Y\. Won\{\\\{i/O\}\\\}stack optimization for smartphones\.In2013 USENIX Annual Technical Conference \(USENIX ATC 13\),pp\. 309–320\.Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p2.1)\.
- Jiet al\.\(2016\)C\. Ji, L\. Chang, L\. Shi, C\. Wu, Q\. Li, and C\. J\. XueAn empirical study of\{\\\{file\-system\}\\\}fragmentation in mobile storage systems\.In8th USENIX Workshop on Hot Topics in Storage and File Systems \(HotStorage 16\),Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p2.1)\.
- Jianget al\.\(2023\)Y\. Jiang, X\. Li, G\. Zhu, H\. Li, J\. Deng, K\. Han, C\. Shen, Q\. Shi, and R\. Zhang6G non\-terrestrial networks enabled low\-altitude economy: opportunities and challenges\.arXiv preprint arXiv:2311\.09047\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p11.1),[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p1.1)\.
- Kjolstadet al\.\(2017\)F\. Kjolstad, S\. Kamil, S\. Chou, D\. Lugato, and S\. AmarasingheThe tensor algebra compiler\.Proceedings of the ACM on Programming Languages1\(OOPSLA\),pp\. 1–29\.Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p5.1)\.
- Leeet al\.\(2024\)W\. Lee, J\. Lee, J\. Seo, and J\. Sim\{\\\{infinigen\}\\\}: Efficient generative inference of large language models with dynamic\{\\\{kv\}\\\}cache management\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 155–172\.Cited by:[§3\.3\.3](https://arxiv.org/html/2609.03079#S3.SS3.SSS3.p2.1)\.
- Lenget al\.\(2023\)Y\. Leng, R\. Liu, H\. Guo, S\. Chen, and S\. YaoScaleflow: efficient deep vision pipeline with closed\-loop scale\-adaptive inference\.InProceedings of the 31st ACM International Conference on Multimedia,pp\. 1698–1706\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Lenget al\.\(2026\)Y\. Leng, R\. Liu, S\. Hu, P\. Zhao, C\. Chen, S\. Chen, and S\. YaoPhysical self\-supervised learning: imu sensing without manual labels\.InProceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services,pp\. 1011–1025\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulqa: measuring how models mimic human falsehoods\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 3214–3252\.Cited by:[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p3.1)\.
- Liuet al\.\(2025a\)J\. Liu, P\. Ponnusamy, T\. Cai, H\. Guo, Y\. Kim, and B\. AthiwaratkunTraining\-free activation sparsity in large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dGVZwyq5tV)Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p1.1),[§1](https://arxiv.org/html/2609.03079#S1.p2.1)\.
- Liuet al\.\(2025b\)R\. Liu, Y\. Leng, K\. Liu, S\. Hu, C\. Chen, P\. Zhao, H\. Yun, and S\. YaoDAF: an efficient end\-to\-end dynamic activation framework for on\-device dnn training\.InProceedings of the 23rd Annual International Conference on Mobile Systems, Applications and Services,pp\. 196–208\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Liuet al\.\(2024a\)R\. Liu, Y\. Leng, S\. Tian, S\. Hu, C\. Chen, and S\. YaoDynaSpa: exploiting spatial sparsity for efficient dynamic dnn inference on devices\.InProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems,pp\. 422–435\.Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p5.1)\.
- Liuet al\.\(2024b\)R\. Liu, Y\. Leng, S\. Tian, S\. Hu, C\. Chen, and S\. YaoDynaspa: exploiting spatial sparsity for efficient dynamic dnn inference on devices\.InProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems,pp\. 422–435\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Liuet al\.\(2025c\)R\. Liu, Y\. Leng, S\. Tian, S\. Hu, R\. Chen, and S\. YaoOn\-device dynamic dnn inference through spatial sparsity exploitation\.GetMobile: Mobile Computing and Communications29\(3\),pp\. 35–38\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Liuet al\.\(2023\)Z\. Liu, J\. Wang, T\. Dao, T\. Zhou, B\. Yuan, Z\. Song, A\. Shrivastava, C\. Zhang, Y\. Tian, C\. Re,et al\.Deja vu: contextual sparsity for efficient llms at inference time\.InInternational Conference on Machine Learning,pp\. 22137–22176\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p1.1),[§1](https://arxiv.org/html/2609.03079#S1.p11.1),[§1](https://arxiv.org/html/2609.03079#S1.p2.1),[§1](https://arxiv.org/html/2609.03079#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.03079#S3.T1),[Table 1](https://arxiv.org/html/2609.03079#S3.T1.10),[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p2.1)\.
- Marescotet al\.\(2013\)L\. Marescot, G\. Chapron, I\. Chadès, P\. L\. Fackler, C\. Duchamp, E\. Marboutin, and O\. GimenezComplex decisions made simple: a primer on stochastic dynamic programming\.Methods in Ecology and Evolution4\(9\),pp\. 872–884\.Cited by:[§3\.2\.3](https://arxiv.org/html/2609.03079#S3.SS2.SSS3.p11.2)\.
- Mayneet al\.\(2000\)D\. Q\. Mayne, J\. B\. Rawlings, C\. V\. Rao, and P\. O\. ScokaertConstrained model predictive control: stability and optimality\.Automatica36\(6\),pp\. 789–814\.Cited by:[§3\.2\.3](https://arxiv.org/html/2609.03079#S3.SS2.SSS3.p1.1)\.
- Mesbah \(2016\)A\. MesbahStochastic model predictive control: an overview and perspectives for future research\.IEEE Control Systems Magazine36\(6\),pp\. 30–44\.Cited by:[§3\.2\.3](https://arxiv.org/html/2609.03079#S3.SS2.SSS3.p1.1)\.
- Mikolovet al\.\(2013\)T\. Mikolov, I\. Sutskever, K\. Chen, G\. S\. Corrado, and J\. DeanDistributed representations of words and phrases and their compositionality\.Advances in neural information processing systems26\.Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p4.1)\.
- Narayananet al\.\(2019\)D\. Narayanan, A\. Harlap, A\. Phanishayee, V\. Seshadri, N\. R\. Devanur, G\. R\. Ganger, P\. B\. Gibbons, and M\. ZahariaPipeDream: generalized pipeline parallelism for dnn training\.InProceedings of the 27th ACM symposium on operating systems principles,pp\. 1–15\.Cited by:[§2\.2](https://arxiv.org/html/2609.03079#S2.SS2.p1.1)\.
- Penningtonet al\.\(2014\)J\. Pennington, R\. Socher, and C\. D\. ManningGlove: global vectors for word representation\.InProceedings of the 2014 conference on empirical methods in natural language processing \(EMNLP\),pp\. 1532–1543\.Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p4.1)\.
- Rajbhandariet al\.\(2021\)S\. Rajbhandari, O\. Ruwase, J\. Rasley, S\. Smith, and Y\. HeZero\-infinity: breaking the gpu memory wall for extreme scale deep learning\.InProceedings of the international conference for high performance computing, networking, storage and analysis,pp\. 1–14\.Cited by:[§2\.2](https://arxiv.org/html/2609.03079#S2.SS2.p1.1)\.
- Reddyet al\.\(2019\)S\. Reddy, D\. Chen, and C\. D\. ManningCoqa: a conversational question answering challenge\.Transactions of the Association for Computational Linguistics7,pp\. 249–266\.Cited by:[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p3.1)\.
- Roumeliotiset al\.\(2023\)K\. I\. Roumeliotis, N\. D\. Tselikas, and D\. K\. NasiopoulosLlama 2: early adopters’ utilization of meta’s new open\-source pretrained model\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Salakhutdinov and Hinton \(2007\)R\. Salakhutdinov and G\. HintonLearning a nonlinear embedding by preserving class neighbourhood structure\.InArtificial intelligence and statistics,pp\. 412–419\.Cited by:[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p4.1)\.
- Shahamet al\.\(2022\)U\. Shaham, E\. Segal, M\. Ivgi, A\. Efrat, O\. Yoran, A\. Haviv, A\. Gupta, W\. Xiong, M\. Geva, J\. Berant,et al\.Scrolls: standardized comparison over long language sequences\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 12007–12021\.Cited by:[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p3.1)\.
- Songet al\.\(2025\)C\. Song, X\. Han, Z\. Zhang, S\. Hu, X\. Shi, K\. Li, C\. Chen, Z\. Liu, G\. Li, T\. Yang,et al\.Prosparse: introducing and enhancing intrinsic activation sparsity within large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 2626–2644\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Songet al\.\(2024\)Y\. Song, Z\. Mi, H\. Xie, and H\. ChenPowerinfer: fast large language model serving with a consumer\-grade gpu\.InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles,pp\. 590–606\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p1.1),[§1](https://arxiv.org/html/2609.03079#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.03079#S3.T1),[Table 1](https://arxiv.org/html/2609.03079#S3.T1.10)\.
- Songet al\.\(2020\)Z\. Song, D\. S\. Berger, K\. Li, and W\. LloydLearning relaxed belady for content distribution network caching\.In17th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 20\),pp\. 529–544\.Cited by:[§3\.3\.3](https://arxiv.org/html/2609.03079#S3.SS3.SSS3.p3.1)\.
- Spring and Shrivastava \(2017\)R\. Spring and A\. ShrivastavaScalable and sustainable deep learning via randomized hashing\.InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 445–454\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Tanget al\.\(2023\)X\. Tang, Y\. Wang, T\. Cao, L\. L\. Zhang, Q\. Chen, D\. Cai, Y\. Liu, and M\. YangLut\-nn: empower efficient neural network inference with centroid learning and table lookup\.InProceedings of the 29th Annual International Conference on Mobile Computing and Networking,pp\. 1–15\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p11.1),[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p1.1)\.
- Wanget al\.\(2025\)H\. Wang, Q\. Zhou, Z\. Hong, and S\. GuoD2MoE: dual routing and dynamic scheduling for efficient on\-device moe\-based llm serving\.InProceedings of the 31st Annual International Conference on Mobile Computing and Networking,pp\. 574–588\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p2.1)\.
- Wanget al\.\(2017\)J\. Wang, T\. Zhang, N\. Sebe, H\. T\. Shen,et al\.A survey on learning to hash\.IEEE transactions on pattern analysis and machine intelligence40\(4\),pp\. 769–790\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p2.1)\.
- Wanget al\.\(2022\)S\. Wang, J\. Wei, A\. Sabne, A\. Davis, B\. Ilbeyi, B\. Hechtman, D\. Chen, K\. S\. Murthy, M\. Maggioni, Q\. Zhang,et al\.Overlap communication with dependent computation via decomposition in large deep learning models\.InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1,pp\. 93–106\.Cited by:[§2\.2](https://arxiv.org/html/2609.03079#S2.SS2.p1.1)\.
- Weiet al\.\(2025\)J\. Wei, S\. Cao, T\. Cao, L\. Ma, L\. Wang, Y\. Zhang, and M\. YangT\-mac: cpu renaissance via table lookup for low\-bit llm deployment on edge\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 278–292\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p2.1)\.
- Xiaoet al\.\(2023\)G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. HanSmoothquant: accurate and efficient post\-training quantization for large language models\.InInternational conference on machine learning,pp\. 38087–38099\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p2.1)\.
- Xueet al\.\(2024\)Z\. Xue, Y\. Song, Z\. Mi, X\. Zheng, Y\. Xia, and H\. ChenPowerinfer\-2: fast large language model inference on a smartphone\.arXiv preprint arXiv:2406\.06282\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p1.1),[§1](https://arxiv.org/html/2609.03079#S1.p11.1),[§1](https://arxiv.org/html/2609.03079#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1),[§3\.2\.2](https://arxiv.org/html/2609.03079#S3.SS2.SSS2.p3.1),[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p2.1)\.
- Yagniket al\.\(2011\)J\. Yagnik, D\. Strelow, D\. A\. Ross, and R\. LinThe power of comparative reasoning\.In2011 International Conference on Computer Vision,pp\. 2431–2438\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2609.03079#S1.p11.1),[§4\.1](https://arxiv.org/html/2609.03079#S4.SS1.p1.1)\.
- Yanget al\.\(2023\)J\. Yang, Z\. Mao, Y\. Yue, and K\. Rashmi\{\\\{gl\-Cache\}\\\}: group\-level learning for efficient and high\-performance caching\.In21st USENIX Conference on File and Storage Technologies \(FAST 23\),pp\. 115–134\.Cited by:[§3\.3\.3](https://arxiv.org/html/2609.03079#S3.SS3.SSS3.p3.1)\.
- Yaoet al\.\(2018\)S\. Yao, Y\. Zhao, H\. Shao, S\. Liu, D\. Liu, L\. Su, and T\. AbdelzaherFastdeepiot: towards understanding and optimizing neural network execution time on mobile and embedded devices\.InProceedings of the 16th ACM Conference on Embedded Networked Sensor Systems,pp\. 278–291\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Yaoet al\.\(2017\)S\. Yao, Y\. Zhao, A\. Zhang, L\. Su, and T\. AbdelzaherDeepiot: compressing deep neural network structures for sensing systems with a compressor\-critic framework\.InProceedings of the 15th ACM conference on embedded network sensor systems,pp\. 1–14\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.
- Zenget al\.\(2023\)Z\. Zeng, M\. Davies, P\. Pulijala, K\. Sankaralingam, and V\. SinghLookupffn: making transformers compute\-lite for cpu inference\.InInternational Conference on Machine Learning,pp\. 40707–40718\.Cited by:[§3\.3](https://arxiv.org/html/2609.03079#S3.SS3.p1.1)\.
- Zhanget al\.\(2022\)Z\. Zhang, Y\. Lin, Z\. Liu, P\. Li, M\. Sun, and J\. ZhouMoefication: transformer feed\-forward layers are mixtures of experts\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 877–890\.Cited by:[§2\.1](https://arxiv.org/html/2609.03079#S2.SS1.p1.1)\.