From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG

# From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG
Source: [https://arxiv.org/html/2608.19535](https://arxiv.org/html/2608.19535)
## From Retrieved Context to Runtime Control: Adaptive Compression for Edge\-based RAGNote:Accepted to appear in the Proceedings of the ACM AI Leadership Summit 2026\.CCS:Computing methodologies Natural language generationCCS:Information systems Retrieval models and rankingCCS:Computer systems organization Embedded systems

Zlatan FericNote:Both authors contributed equally to this research\.Affiliation:Northeastern University,Boston,Massachusetts,USAemail:[feric\.z@northeastern\.edu](mailto:[email protected])Amir TaherinAffiliation:Northeastern University,Boston,Massachusetts,USAemail:[taherin\.a@northeastern\.edu](mailto:[email protected]),Yanzhi WangAffiliation:Northeastern University,Boston,Massachusetts,USAemail:[yanz\.wang@northeastern\.edu](mailto:[email protected])andDavid KaeliAffiliation:Northeastern University,Boston,Massachusetts,USAemail:[d\.kaeli@northeastern\.edu](mailto:[email protected])

© none

###### Abstract\.

Retrieval\-augmented generation \(RAG\) improves language\-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV\-cache footprint, memory traffic, latency, and energy\. Context compression offers a natural remedy by pruning retrieved text before generation\. However, state\-of\-the\-art context\-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time\. This static view ignores both workload variation and the live state of the edge device\. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings\.

This paper proposes a vision for telemetry\-informed adaptive compression in edge RAG, grounded in experimental evidence\. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua\-2 compression\. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per\-query latency and 91% of GPU energy for 7B–8B generators\. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality\. Intermediate compression can reduce GPU energy by up to 53\.2%, and SoC energy by up to 48\.2%, with negligible quality loss\. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry\.

###### Keywords:

Edge RAG, Context Compression, Energy\-Efficient Inference

## 1\.Introduction

Large language models \(LLMs\) have rapidly improved in capability and scale\([18](https://arxiv.org/html/2608.19535#bib.bib34);[2](https://arxiv.org/html/2608.19535#bib.bib35);[33](https://arxiv.org/html/2608.19535#bib.bib36);[48](https://arxiv.org/html/2608.19535#bib.bib37);[41](https://arxiv.org/html/2608.19535#bib.bib52)\), but they remain hampered by hallucinations\([12](https://arxiv.org/html/2608.19535#bib.bib38)\), stale parametric knowledge\([19](https://arxiv.org/html/2608.19535#bib.bib39)\), and poor access to private or domain\-specific information\([7](https://arxiv.org/html/2608.19535#bib.bib16);[22](https://arxiv.org/html/2608.19535#bib.bib14)\)\. Retrieval\-augmented generation \(RAG\) addresses these limitations by conditioning generation on passages retrieved at inference time, allowing the model to produce responses grounded in external, dynamic and domain\-specific knowledge without retraining\([22](https://arxiv.org/html/2608.19535#bib.bib14)\)\. This makes RAG an attractive foundation for practical LLM applications in domains where correctness, freshness, and provenance matter\.

Running RAG at the edge is compelling for privacy\-sensitive data\([23](https://arxiv.org/html/2608.19535#bib.bib25);[39](https://arxiv.org/html/2608.19535#bib.bib26)\), latency\-critical applications such as robotics\([56](https://arxiv.org/html/2608.19535#bib.bib40);[21](https://arxiv.org/html/2608.19535#bib.bib41);[25](https://arxiv.org/html/2608.19535#bib.bib53);[45](https://arxiv.org/html/2608.19535#bib.bib51)\)and augmented reality\([5](https://arxiv.org/html/2608.19535#bib.bib42);[44](https://arxiv.org/html/2608.19535#bib.bib55)\), personal agents\([37](https://arxiv.org/html/2608.19535#bib.bib43);[57](https://arxiv.org/html/2608.19535#bib.bib44)\), and mobile and bandwidth\-limited settings where cloud access may be unreliable\. Recent edge\-RAG systems have improved efficiency on constrained platforms, including mobile devices\([42](https://arxiv.org/html/2608.19535#bib.bib22);[35](https://arxiv.org/html/2608.19535#bib.bib23)\), edge SoCs\([36](https://arxiv.org/html/2608.19535#bib.bib24)\), and wearable systems\([43](https://arxiv.org/html/2608.19535#bib.bib27)\), through optimizations such as faster vector search, compact indexing, and reduced memory footprint\. However, these systems primarily focus on retrieval/index efficiency or fixed content\-reduction strategies\. Instead, we should explore how RAG can run efficiently on edge devices based on the application content\.

This motivates the following questions:*once documents have been retrieved, how much of that context should actually be passed to the generator on an edge device?*RAG improves knowledge access by adding retrieved tokens, but tokens are expensive on edge hardware\. Each retrieved passage increases the prompt length seen by the generator, adding to prefill latency, KV\-cache footprint, memory traffic, and energy per query\([57](https://arxiv.org/html/2608.19535#bib.bib44);[42](https://arxiv.org/html/2608.19535#bib.bib22)\)\. This cost is amplified on constrained edge SoCs, where memory bandwidth, power budget, and thermal headroom are limited\. Moreover, retrieving more context does not guarantee better answers: irrelevant or redundant passages can distract the generator and increase system cost without improving response quality\([15](https://arxiv.org/html/2608.19535#bib.bib45);[26](https://arxiv.org/html/2608.19535#bib.bib46);[55](https://arxiv.org/html/2608.19535#bib.bib47);[46](https://arxiv.org/html/2608.19535#bib.bib54)\)\. Thus, edge RAG comes with a new set of challenging tradeoff:*the system needs enough context to preserve answer quality, but not so much context that generation overuses latency and energy\.*

![Refer to caption](https://arxiv.org/html/2608.19535v1/adaptive_compression_pipeline_v2-cropped.png)

Figure 1\.Telemetry\-informed adaptive compression for edge RAG\. Retrieved passages are compressed before generation, but the compression ratio is selected at runtime by a controller that observes edge\-SoC telemetry and optimizes for latency, energy, thermal, memory, and accuracy constraints\.*Context compression*is a natural knob for this tradeoff: it prunes or rewrites retrieved passages before generation, aiming to preserve query\-relevant evidence while reducing input tokens\. We use this term to distinguish our setting from broader prompt\-compression work\([24](https://arxiv.org/html/2608.19535#bib.bib1)\): only the retrieved context is compressed, while instructions and few\-shot scaffolding remain unchanged\. Unlike coarser controls such as retrieval depth, compression exposes a continuous rate knob: lower rates reduce generation load more aggressively, while higher rates preserve more evidence\.

However, compression on the edge is not free\. The compressor runs on the same SoC as retrieval and generation, adding latency, memory traffic, and energy while competing for shared resources\. Thus, the relevant metric is not gross generation speedup but*net benefit*: the generation savings from a shorter prompt minus the compressor’s own overhead\. A static compression ratio can be a net loss when contexts are short, generators are small, or too few tokens are removed\.

In this paper, we present our vision for*telemetry\-informed adaptive compression*: context compression should be a runtime decision, not a fixed preprocessing step\. As Figure[1](https://arxiv.org/html/2608.19535#S1.F1)illustrates, an edge RAG system could use workload features and live SoC telemetry \(e\.g\., latency, energy, and memory bandwidth\) to decide whether to compress and what rate to use\. Our goal is to demonstrate the potential for such a controller by measuring when compression is beneficial, when it is harmful, and which system signals expose that boundary\.

To evaluate this potential, we measure RAG on Jetson AGX Thor using Llama and Qwen generators from 1B to 8B parameters, Natural Questions and HotpotQA datasets, and LLMLingua\-2 context compression\. Our results show three findings\. First, for 7B–8B models, generation dominates the edge\-RAG budget, accounting for roughly 90% of latency, 91% of GPU energy, and 92% of total SoC energy per query\. Second, compression can provide substantial net savings after accounting for compressor overhead, but its benefit depends on model size, retrieval depth, and dataset\. Third, sweeping the compression rate reveals an adaptive operating region: mild compression can lose energy, overly aggressive compression can hurt answer quality, and intermediate rates can substantially reduce energy \(i\.e\., GPU energy by up to 53\.2% and SoC energy by up to 48\.2%\) with little quality impact\.

This paper makes three contributions: \(1\) a stage\-level characterization of latency and energy in edge RAG on Jetson AGX Thor, identifying when generation becomes the dominant cost; \(2\) a net\-benefit analysis of context compression that includes compressor overhead rather than reporting generation\-only speedups; and \(3\) a telemetry\-informed adaptive\-compression vision, grounded in measured workload\-dependent tradeoffs, for future edge\-RAG systems\.

## 2\.Background and Motivation

### 2\.1\.RAG Pipeline and Context Cost

A RAG pipeline has two phases: an offline indexing phase and an online query phase\. Offline, a document collection is split into passages and stored in a searchable datastore, typically with lexical or vector indexes for efficient retrieval\([40](https://arxiv.org/html/2608.19535#bib.bib13);[17](https://arxiv.org/html/2608.19535#bib.bib12)\)\. Online, a user query is encoded and used to retrieve the top\-ranked passages; a generator LLM then conditions on both the query and the retrieved context to produce the final response\([22](https://arxiv.org/html/2608.19535#bib.bib14);[11](https://arxiv.org/html/2608.19535#bib.bib15)\)\. Modern RAG systems extend this basic retrieve\-then\-generate flow with additional stages such as query rewriting\([28](https://arxiv.org/html/2608.19535#bib.bib18);[49](https://arxiv.org/html/2608.19535#bib.bib19)\), reranking\([30](https://arxiv.org/html/2608.19535#bib.bib20);[31](https://arxiv.org/html/2608.19535#bib.bib21)\), and context compression\([13](https://arxiv.org/html/2608.19535#bib.bib2);[24](https://arxiv.org/html/2608.19535#bib.bib1)\), inserted between retrieval and generation\([7](https://arxiv.org/html/2608.19535#bib.bib16);[10](https://arxiv.org/html/2608.19535#bib.bib17)\)\.

The same retrieved context that improves grounding also creates the main systems cost: every additional passage increases the prompt length seen by the generator\. Longer prompts increase prefill work, KV\-cache footprint and memory traffic, as well as the energy per query, which is especially costly on edge SoCs with limited memory bandwidth, power, and thermal headroom\([57](https://arxiv.org/html/2608.19535#bib.bib44);[42](https://arxiv.org/html/2608.19535#bib.bib22)\)\.*Context compression*is inserted between retrieval and generation to reduce this cost before the generator runs\. Throughout this paper, we use the term*rate*for the fraction of retrieved\-context tokens*kept*, matching the parameter exposed by LLMLingua\-2\([34](https://arxiv.org/html/2608.19535#bib.bib4)\)\. Thus,rate=0\.5\\text\{rate\}=0\.5corresponds to keeping roughly half of the retrieved context, or about2×2\\timescompression in factor notation\. Compression is applied only to the retrieved context; the system prompt, instructions, and few\-shot scaffolding \(system message, few\-shot examples\) are left intact\.

### 2\.2\.Context Compression Methods

Prompt compression methods are commonly divided into*hard*and*soft*compression\([24](https://arxiv.org/html/2608.19535#bib.bib1)\)\. Hard compression directly edits the textual input before inference, either by extracting important tokens or sentences or by summarizing the input\([13](https://arxiv.org/html/2608.19535#bib.bib2);[34](https://arxiv.org/html/2608.19535#bib.bib4)\)\. Soft compression instead maps text into latent tokens or embeddings consumed by the decoder\([29](https://arxiv.org/html/2608.19535#bib.bib6);[8](https://arxiv.org/html/2608.19535#bib.bib7);[3](https://arxiv.org/html/2608.19535#bib.bib8)\)\. We focus on hard compression because it can be inserted into existing RAG pipelines without changing the generator interface and because compressed text length directly affects generation cost\.

Hard compression can be*extractive*or*abstractive*\. Extractive methods remove low\-importance tokens or sentences while preserving the original text, as in the LLMLingua family\([13](https://arxiv.org/html/2608.19535#bib.bib2);[34](https://arxiv.org/html/2608.19535#bib.bib4);[14](https://arxiv.org/html/2608.19535#bib.bib5)\)\. Abstractive methods such as RECOMP generate a new summary of the retrieved passages\([51](https://arxiv.org/html/2608.19535#bib.bib9)\), but this usually adds another autoregressive stage and therefore extra latency and energy\. Prior device\-oriented studies show that compressor cost varies substantially across methods\([52](https://arxiv.org/html/2608.19535#bib.bib10);[27](https://arxiv.org/html/2608.19535#bib.bib11)\), making overhead a first\-order concern on edge devices\. Provence similarly shows the value of filtering context before generation, but does not expose the continuous rate knob needed for our adaptive\-control study\([4](https://arxiv.org/html/2608.19535#bib.bib3)\)\.

We useLLMLingua\-2\([34](https://arxiv.org/html/2608.19535#bib.bib4)\)as a representative compressor because it is extractive, lightweight relative to the generator, and exposes a continuous*rate*parameter\. This lets us sweep rate to measure the tradeoff among generation savings, compressor overhead, and answer quality\. Our goal is not to claim LLMLingua\-2 is the final edge\-RAG compressor, but to use it as a controllable representative for identifying when compression helps, when it is a net loss, and where adaptive policies have room to operate\.

### 2\.3\.Edge RAG and the Missing Runtime View

Efficient edge inference has largely focused on reducing generator cost through quantization, memory reduction, KV\-cache optimization, and runtime scheduling\. RAG broadens this problem: retrieval, optional reranking or compression, and generation all share the same latency, memory, power, and thermal budget\. Thus, a stage such as context compression must be evaluated by its*net*effect, not by generation speedup alone\.

Recent edge\-RAG systems address important parts of this pipeline\. EdgeRAG optimizes retrieval memory and latency through index and embedding techniques\([42](https://arxiv.org/html/2608.19535#bib.bib22)\); MobileRAG uses selective content reduction on mobile devices\([35](https://arxiv.org/html/2608.19535#bib.bib23)\); RoCR accelerates retrieval with edge computing\-in\-memory architectures\([36](https://arxiv.org/html/2608.19535#bib.bib24)\); and wearable RAG systems optimize retrieval energy and memory movement\([43](https://arxiv.org/html/2608.19535#bib.bib27)\)\. This prior work shows that RAG can run on constrained platforms, but they primarily target retrieval/index efficiency, platform\-specific acceleration, or fixed content reduction rather than runtime control of compression under live device constraints\.

Adaptive RAG serving systems such as METIS show that RAG configurations should be selected per query rather than fixed offline\([38](https://arxiv.org/html/2608.19535#bib.bib49)\)\. However, METIS targets server\-side quality\-delay tradeoffs, not edge SoC telemetry, energy, thermal behavior, or the cost of running a compressor on the same device as the generator\. The missing runtime view is therefore deciding*when compression is worth its own cost*: an edge controller must choose whether to compress, and at what rate, based on workload features, retrieved\-context length, model size, quality risk, and hardware state\.

## 3\.Methodology and Evaluation

### 3\.1\.Experimental Setup and Metrics

We evaluate aRAG pipelineon theNVIDIA Jetson AGX Thor\([32](https://arxiv.org/html/2608.19535#bib.bib50)\)using a standard retrieve\-then\-generate flow\. Table[1](https://arxiv.org/html/2608.19535#S3.T1)summarizes the shared configuration\. All stages run on the same SoC, and the compressor, when enabled, is co\-resident with the retriever and generator\. We run two experiment groups: \(1\) an uncompressed stage\-attribution sweep to identify where latency and energy are spent across model sizes, datasets, and retrieval depths, and \(2\) a compression\-rate sweep to determine when LLMLingua\-2 becomes net\-positive and where energy savings begin to impact answer quality\. The experimental setup builds on RAGMark, our stage\-level RAG benchmarking framework\([6](https://arxiv.org/html/2608.19535#bib.bib57)\), and incorporates the GPU/SoC power, thermal, and memory telemetry collection from our edge\-LLM characterization framework, Hydra\([47](https://arxiv.org/html/2608.19535#bib.bib56)\)\.

Table 1\.Experimental SetupWe report compression as a*net*effect\. Compressed RAG is beneficial only when the generation savings from a shorter prompt exceeds the compressor’s own latency and energy on the same SoC\. Therefore, our reported savings, include the compression stage itself, rather than treating generation\-only speedup as an end\-to\-end gain\.

### 3\.2\.Results

We answer three main questions: \(i\) where does the per\-query budget go in an uncompressed pipeline, \(ii\) how do quality and net energy vary as we sweep the compression rate, and \(iii\) how does the value of compression vary with workload characteristics?

Figure 2\.Per\-query share of latency \(left\) and GPU energy \(right\) by stage, on AGX Thor, fp16, no compression\.Generation dominates the budget above 3 B parameters \(Figure[2](https://arxiv.org/html/2608.19535#S3.F2)\)\.At Llama\-3\.1\-8B and Qwen\-2\.5\-7B, the generator alone accounts for 90 % of per\-query latency and*91 %*of per\-query GPU energy\. Embed and retrieve together never exceed 10 % at this scale\. At the 1 B end the picture inverts: Llama\-3\.2\-1B spends 33 % of wall time and 39 % of GPU energy in embed\+retrieve, leaving compression of the generator’s prompt with much less headroom to recover\. The Llama\-vs\-Qwen split is statistically a no\-op at equal parameter counts \(Llama\-3B and Qwen\-3B agree to within 0\.3 pp on the generation share; Llama\-8B and Qwen\-7B to within 0\.3 pp\), so we run the compression sweeps on the Llama family only\.

Figure 3\.Answer F1 \(left axis\) and per\-query GPU energy \(right axis\) vs\. LLMLingua\-2rateon HotpotQA\. Shaded band: adaptive operating room between the two dotted\-line knees\.Two knees frame a wide adaptive operating range \(Figure[3](https://arxiv.org/html/2608.19535#S3.F3)\)\.On every \(model, top\-kk\) panel, F1 is quite similar fromrate=1\.0\\text\{rate\}=1\.0down torate=0\.3\\text\{rate\}=0\.3\(the F1 deltas across this range are inside the±0\.05\\pm 0\.05noise floor\), then collapses by 4–10 absolute points atrate=0\.15\\text\{rate\}=0\.15when the generator can no longer reconstruct meaning from the heavily\-pruned context\. GPU energy changes monotonically: it gets*worse*atrate=0\.9\\text\{rate\}=0\.9because LLMLingua\-2’s roughly fixed130130–310310ms per\-query overhead is not amortized by dropping only 10 % of tokens; it gets better fromrate=0\.7\\text\{rate\}=0\.7onward as the shorter prompt cuts prefill cost\. The two inflection points define an*adaptive operating region*of width≈0\.6\\approx 0\.6on the rate axis \(shaded\)\. The safest aggressive setpoint we observe israte=0\.3\\text\{rate\}=0\.3, which preserves quality and lies right at the quality knee\.

Figure 4\.Net GPU energy delta \(top\) andΔ\\DeltaF1 \(bottom\) atrate=0\.9\\text\{rate\}=0\.9\(mild\) andrate=0\.3\\text\{rate\}=0\.3\(safe\-aggressive\), vs\. the in\-session uncompressed baseline\.The*value*of compressing varies with workload, while the best observed safe\-aggressive rate is stable \(Figure[4](https://arxiv.org/html/2608.19535#S3.F4), Table[2](https://arxiv.org/html/2608.19535#S3.T2)\)\.Atrate=0\.3\\text\{rate\}=0\.3, GPU energy savings grow monotonically with model size and retrieved context length, from\+30%\+30\\,\\%on Llama\-3B atk=5k\{=\}5to\+53%\+53\\,\\%on Llama\-8B atk=10k\{=\}10\. Atrate=0\.9\\text\{rate\}=0\.9the relationship inverts: every configuration increases net energy usage \(from−1%\-1\\,\\%on the heaviest workload to−13%\-13\\,\\%on the lightest\), because the compressor’s computational overhead outweighs the small generation savings from dropping 10 % of tokens\. The samerate=0\.3\\text\{rate\}=0\.3setpoint is the safest aggressive choice on every panel, while the quantitative win it delivers shifts by a factor of∼\\sim1\.8×\\timesacross our model\+context combinations\.

Table 2\.Net savings atrate=0\.3\\text\{rate\}=0\.3vs\. the in\-session uncompressed baseline \(positiveΔ\\Delta= saving on the cost columns, improvement on the quality columns\)\.

## 4\.Discussion and Research Agenda

### 4\.1\.From Adaptive Room to an Online Controller

The two\-knee structure in Figure[3](https://arxiv.org/html/2608.19535#S3.F3)reframes adaptive compression on edge devices\. In our sweep, the best observed safe\-aggressive rate israte=0\.3\\text\{rate\}=0\.3, but the*value*of enabling compression varies substantially with workload\. Heavy workloads, such as 8 B generators withk=10k\{=\}10, recover up to 53% of GPU energy at this setpoint, while lighter workloads recover only 30%\. Mild compression atrate=0\.9\\text\{rate\}=0\.9is a net loss because the compressor’s fixed per\-query cost is not amortized by dropping only a small fraction of tokens\.

This suggests that the first practical controller does not need to search a continuous rate space for every query\. Instead, it can make a small number of workload\-conditioned decisions: skip compression when projected generation savings fall below compressor overhead, userate=0\.3\\text\{rate\}=0\.3for long\-context or high\-cost generations, and reserve more aggressive compression for settings with explicit quality slack\. Such a controller can use cheap signals already available at runtime, including model identity, retrieved\-context length, top\-kk, recent latency, energy, and thermal headroom\. The solution is therefore not only choosing a compression method, but deciding when compression is worth its own cost on the same SoC\.

### 4\.2\.Research Agenda

Three questions naturally arise\. First,how does the energy knee move with lighter compressors?LLMLingua\-2 uses a fixed model pass, making mild compression expensive\. Streaming, token\-level, or hardware\-aware compressors could make compression beneficial at milder rates and shrink the negative\-savings region\.

Second,do the observed knees hold under quantization and larger models?Our measurements use fp16 generators up to 8 B parameters\. Production edge deployments often use 4\-bit quantization, and larger models may shift the balance between generation savings, compressor overhead, and quality loss\.

Third,how should compression interact with the rest of the RAG pipeline?Compression is only one knob\. Retrieval depth, reranker cutoff, source selection, query rewriting, and conditional or iterative RAG policies\([1](https://arxiv.org/html/2608.19535#bib.bib48)\)all consume the same latency, memory, energy, and thermal budget\. An telemetry\-informed controller, that considers multiple RAG knobs, should lead to significant savings\.

Treating compression as an adaptive on\-device resource\-management knob, rather than a fixed preprocessing step, makes efficient grounded generation on the edge an interesting optimization problem\. Our two\-knee structure, evaluated on AGX Thor, offers empirical evidence for building an adaptive compression controller\.

## 5\.Conclusion

This paper presents a vision, grounded in empirical evidence, that context compression should be treated as a runtime systems knob for edge RAG rather than a fixed preprocessing step\. Our measurements on Jetson AGX Thor show why: generation dominates the cost of edge\-RAG pipelines, compression can deliver substantial net energy savings, and the benefit depends on workload and compression rate\. The observed gap between the energy and quality knees demonstrates the potential for telemetry\-informed adaptive compression\. Future edge\-RAG systems should use workload features and live SoC state to decide when compression is worth its own cost and how aggressively it should be applied\.

## References

- Asaiet al\.\(2024\)A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. HajishirziSelf\-rag: learning to retrieve, generate, and critique through self\-reflection\.InInternational conference on learning representations,Vol\.2024,pp\. 9112–9141\.Cited by:[§4\.2](https://arxiv.org/html/2608.19535#S4.SS2.p3.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Chevalieret al\.\(2023\)A\. Chevalier, A\. Wettig, A\. Ajith, and D\. ChenAdapting language models to compress contexts\.arXiv preprint arXiv:2305\.14788\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p1.1)\.
- Chirkovaet al\.\(2025\)N\. Chirkova, T\. Formal, V\. Nikoulina, and S\. ClinchantProvence: efficient and robust context pruning for retrieval\-augmented generation\.arXiv preprint arXiv:2501\.16214\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1)\.
- Daiet al\.\(2025\)Y\. Dai, B\. Qian, Y\. Liu, Y\. Yan, and Y\. ShuEros: real\-time dense mapping made easy on mobile devices\.InProceedings of the 26th International Workshop on Mobile Computing Systems and Applications,pp\. 19–24\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Fericet al\.\(2026\)Z\. Feric, A\. Taherin, B\. Ren, Y\. Wang, J\. Dy, and D\. KaeliRAGMark: a comprehensive framework for benchmarking retrieval\-augmented generation systems\.InIEEE International Symposium on Workload Characterization \(IISWC\),Note:To appearCited by:[§3\.1](https://arxiv.org/html/2608.19535#S3.SS1.p1.1)\.
- Gaoet al\.\(2023\)Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, H\. Wang, and H\. WangRetrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.109972\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Geet al\.\(2023\)T\. Ge, J\. Hu, L\. Wang, X\. Wang, S\. Chen, and F\. WeiIn\-context autoencoder for context compression in a large language model\.arXiv preprint arXiv:2307\.06945\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.5.2.1.1)\.
- Guptaet al\.\(2024\)S\. Gupta, R\. Ranjan, and S\. N\. SinghA comprehensive survey of retrieval\-augmented generation \(rag\): evolution, current landscape and future directions\.arXiv preprintarXiv:2410\.12837\.Note:Preprint\. Available at[https://arxiv\.org/abs/2410\.12837](https://arxiv.org/abs/2410.12837)Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Izacard and Grave \(2020\)G\. Izacard and E\. GraveLeveraging passage retrieval with generative models for open domain question answering\.arXiv preprint arXiv:2007\.01282\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Jiet al\.\(2023\)Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. FungSurvey of hallucination in natural language generation\.ACM computing surveys55\(12\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Jianget al\.\(2023\)H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. QiuLlmlingua: compressing prompts for accelerated inference of large language models\.arXiv preprint arXiv:2310\.05736\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1)\.
- Jianget al\.\(2024a\)H\. Jiang, Q\. Wu, X\. Luo, D\. Li, C\. Lin, Y\. Yang, and L\. QiuLongllmlingua: accelerating and enhancing llms in long context scenarios via prompt compression\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1658–1677\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1)\.
- Jianget al\.\(2024b\)Z\. Jiang, X\. Ma, and W\. ChenLongrag: enhancing retrieval\-augmented generation with long\-context llms\.arXiv preprint arXiv:2406\.15319\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p3.1)\.
- Jinet al\.\(2024\)J\. Jin, Y\. Zhu, X\. Yang, C\. Zhang, and Z\. DouFlashRAG: a modular toolkit for efficient retrieval\-augmented generation research\.External Links:2405\.13576,[Link](https://arxiv.org/abs/2405.13576)Cited by:[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.3.2.1.1)\.
- Johnsonet al\.\(2019\)J\. Johnson, M\. Douze, and H\. JégouBillion\-scale similarity search with gpus\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Kasaiet al\.\(2023\)J\. Kasai, K\. Sakaguchi, R\. Le Bras, A\. Asai, X\. Yu, D\. Radev, N\. A\. Smith, Y\. Choi, K\. Inui,et al\.Realtime qa: what’s the answer right now?\.Advances in neural information processing systems36,pp\. 49025–49043\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. PetrovNatural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.4.2.1.1)\.
- Lanet al\.\(2025\)G\. Lan, K\. Qu, R\. Zurbrügg, C\. Chen, C\. E\. Mower, H\. Bou\-Ammar, and M\. HutterExperience is the best teacher: grounding vlms for robotics through self\-generated memory\.arXiv preprint arXiv:2507\.16713\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Liet al\.\(2024\)J\. Li, C\. Xu, L\. Jia, F\. Wang, C\. Zhang, and J\. LiuEACO\-RAG: towards distributed tiered LLM deployment using edge\-assisted and collaborative RAG with adaptive knowledge update\.CoRRabs/2410\.20299\.External Links:[Link](https://arxiv.org/abs/2410.20299),[Document](https://dx.doi.org/10.48550/arXiv.2410.20299),2410\.20299Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Liet al\.\(2025\)Z\. Li, Y\. Liu, Y\. Su, and N\. CollierPrompt compression for large language models: a survey\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7182–7195\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p1.1)\.
- Linet al\.\(2026\)J\. Lin, A\. Taherin, A\. Akbari, A\. Akbari, L\. Lu, G\. Chen, T\. Padir, X\. Yang, W\. Chen, Y\. Li, X\. Lin, D\. Kaeli, P\. Zhao, and Y\. WangVOTE: vision\-language\-action optimization with trajectory ensemble voting\.External Links:2507\.05116,[Link](https://arxiv.org/abs/2507.05116)Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Liuet al\.\(2024\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the association for computational linguistics12,pp\. 157–173\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p3.1)\.
- Luet al\.\(2024\)Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, X\. Zhang, N\. D\. Lane, and M\. XuSmall language models: survey, measurements, and insights\.arXiv preprint arXiv:2409\.15790\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1)\.
- Maet al\.\(2023\)X\. Ma, Y\. Gong, P\. He, H\. Zhao, and N\. DuanQuery rewriting in retrieval\-augmented large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5303–5315\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Muet al\.\(2023\)J\. Mu, X\. Li, and N\. GoodmanLearning to compress prompts with gist tokens\.Advances in Neural Information Processing Systems36,pp\. 19327–19352\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p1.1)\.
- Nogueira and Cho \(2019\)R\. Nogueira and K\. ChoPassage re\-ranking with bert\.arXiv preprint arXiv:1901\.04085\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Nogueiraet al\.\(2020\)R\. Nogueira, Z\. Jiang, R\. Pradeep, and J\. LinDocument ranking with a pretrained sequence\-to\-sequence model\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 708–718\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- NVIDIA \(2025\)NVIDIAJetson agx thor — technical reference manual\.Note:[https://developer\.nvidia\.com/embedded/jetson\-thor](https://developer.nvidia.com/embedded/jetson-thor)Cited by:[§3\.1](https://arxiv.org/html/2608.19535#S3.SS1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Panet al\.\(2024\)Z\. Pan, Q\. Wu, H\. Jiang, M\. Xia, X\. Luo, J\. Zhang, Q\. Lin, V\. Rühle, Y\. Yang, C\. Lin,et al\.Llmlingua\-2: data distillation for efficient and faithful task\-agnostic prompt compression\.arXiv preprint arXiv:2403\.12968\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1),[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p3.1),[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.6.2.1.1)\.
- Parket al\.\(2025\)T\. Park, G\. Lee, and M\. KimMobileRAG: a fast, memory\-efficient, and energy\-efficient method for on\-device RAG\.CoRRabs/2507\.01079\.External Links:[Link](https://arxiv.org/abs/2507.01079),[Document](https://dx.doi.org/10.48550/arXiv.2507.01079),2507\.01079Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.19535#S2.SS3.p2.1)\.
- Qinet al\.\(2024\)R\. Qin, Z\. Yan, D\. Zeng, Z\. Jia, D\. Liu, J\. Liu, Z\. Zheng, N\. Cao, K\. Ni, J\. Xiong, and Y\. ShiRobust implementation of retrieval\-augmented generation on edge\-based computing\-in\-memory architectures\.CoRRabs/2405\.04700\.External Links:[Link](https://arxiv.org/abs/2405.04700),[Document](https://dx.doi.org/10.48550/arXiv.2405.04700),2405\.04700Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.19535#S2.SS3.p2.1)\.
- Rawassizadeh and Rong \(2023\)R\. Rawassizadeh and Y\. RongODSearch: fast and resource efficient on\-device natural language search for fitness trackers’ data\.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies6\(4\),pp\. 1–25\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Rayet al\.\(2025\)S\. Ray, R\. Pan, Z\. Gu, K\. Du, S\. Feng, G\. Ananthanarayanan, R\. Netravali, and J\. JiangMETIS: fast quality\-aware rag systems with configuration adaptation\.External Links:2412\.10543,[Link](https://arxiv.org/abs/2412.10543)Cited by:[§2\.3](https://arxiv.org/html/2608.19535#S2.SS3.p3.1)\.
- Renet al\.\(2024\)R\. Ren, Y\. Wu, X\. Zhang, J\. Ren, Y\. Shen, S\. Wang, and K\. TsangRetrieval\-augmented generation for mobile edge computing via large language model\.CoRRabs/2412\.20820\.External Links:[Link](https://arxiv.org/abs/2412.20820),[Document](https://dx.doi.org/10.48550/arXiv.2412.20820),2412\.20820Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Robertsonet al\.\(2009\)S\. Robertson H\. Zaragozaet al\.The probabilistic relevance framework: bm25 and beyond\.Foundations and Trends® in Information Retrieval3\(4\),pp\. 333–389\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Rupprechtet al\.\(2026\)T\. Rupprecht, P\. Zhao, A\. Taherin, A\. Akbari, A\. Akbari, Y\. He, T\. Imtiaz, S\. Duffy, J\. Lin, Y\. Chen, R\. Chowdhury, E\. Nan, Y\. Shen, Y\. Cao, H\. Zeng, W\. Chen, G\. Yuan, J\. Dy, S\. Ostadabbas, X\. Zhang, D\. Kaeli, E\. Yeh, and Y\. WangHuman cognition in machines: a unified perspective of world models\.External Links:2604\.16592,[Link](https://arxiv.org/abs/2604.16592)Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Seemakhuptet al\.\(2024\)K\. Seemakhupt, S\. Liu, and S\. M\. KhanEdgeRAG: online\-indexed RAG for edge devices\.CoRRabs/2412\.21023\.External Links:[Link](https://arxiv.org/abs/2412.21023),[Document](https://dx.doi.org/10.48550/arXiv.2412.21023),2412\.21023Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1),[§1](https://arxiv.org/html/2608.19535#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p2.1),[§2\.3](https://arxiv.org/html/2608.19535#S2.SS3.p2.1)\.
- Shaoet al\.\(2025\)K\. Shaoet al\.A memory\-efficient retrieval architecture for RAG\-enabled wearable medical LLMs\-agents\.CoRRabs/2510\.27107\.External Links:[Link](https://arxiv.org/abs/2510.27107),[Document](https://dx.doi.org/10.48550/arXiv.2510.27107),2510\.27107Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.19535#S2.SS3.p2.1)\.
- Sunet al\.\(2020\)Q\. Sun, A\. Taherin, Y\. Siatitse, and Y\. ZhuEnergy\-efficient 360\-degree video rendering on fpga via algorithm\-architecture co\-design\.InProceedings of the 2020 ACM/SIGDA International Symposium on Field\-Programmable Gate Arrays,FPGA ’20,New York, NY, USA,pp\. 97–103\.External Links:ISBN 9781450370998,[Link](https://doi.org/10.1145/3373087.3375317),[Document](https://dx.doi.org/10.1145/3373087.3375317)Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Taherinet al\.\(2026a\)A\. Taherin, J\. Lin, A\. Akbari, A\. Akbari, P\. Zhao, W\. Chen, D\. Kaeli, and Y\. WangCross\-platform scaling of vision\-language\-action models from edge to cloud gpus\.InProceedings of the Great Lakes Symposium on VLSI 2026,pp\. 234–239\.External Links:ISBN 9798400724312,[Link](https://doi.org/10.1145/3787109.3816400)Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Taherinet al\.\(2018\)A\. Taherin, M\. Salehi, and A\. EjlaliReliability\-aware energy management in mixed\-criticality systems\.IEEE Transactions on Sustainable Computing3\(3\),pp\. 195–208\.External Links:[Document](https://dx.doi.org/10.1109/TSUSC.2018.2801123)Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p3.1)\.
- Taherinet al\.\(2026b\)A\. Taherin, S\. Taghipour Anvari, C\. Amante, Y\. Chen, R\. Noroian, Z\. Feric, N\. Bohm Agostini, P\. Zhao, J\. Cano, B\. Ren, Y\. Wang, and D\. KaeliHydra: phase\-aware workload characterization of LLM inference across edge SoC generations, backends, and quantization levels\.InIEEE International Symposium on Workload Characterization \(IISWC\),Note:To appearCited by:[§3\.1](https://arxiv.org/html/2608.19535#S3.SS1.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p1.1)\.
- Wanget al\.\(2023\)L\. Wang, N\. Yang, and F\. WeiQuery2doc: query expansion with large language models\.arXiv preprint arXiv:2303\.07678\.Cited by:[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p1.1)\.
- Wang \(2023\)R\. WangIntfloat/e5\-base\-v2\.Note:[https://huggingface\.co/intfloat/e5\-base\-v2](https://huggingface.co/intfloat/e5-base-v2)Cited by:[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.3.2.1.1)\.
- Xuet al\.\(2023\)F\. Xu, W\. Shi, and E\. ChoiRecomp: improving retrieval\-augmented lms with compression and selective augmentation\.arXiv preprint arXiv:2310\.04408\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1)\.
- Xuet al\.\(2024\)J\. Xu, Z\. Li, W\. Chen, Q\. Wang, X\. Gao, Q\. Cai, and Z\. LingOn\-device language models: a comprehensive review\.arXiv preprint arXiv:2409\.00088\.Cited by:[§2\.2](https://arxiv.org/html/2608.19535#S2.SS2.p2.1)\.
- Yanget al\.\(2024\)A\. Yanget al\.Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.5.2.1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Table 1](https://arxiv.org/html/2608.19535#S3.T1.2.4.2.1.1)\.
- Yoranet al\.\(2024\)O\. Yoran, T\. Wolfson, O\. Ram, and J\. BerantMaking retrieval\-augmented language models robust to irrelevant context\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 29862–29883\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p3.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, R\. Wang, J\. Lin, Z\. Wang, and X\. QiRetrieval\-vla: training\-free in\-context adaptation for vision\-language\-action models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 1358–1367\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1)\.
- Zhenget al\.\(2025\)Y\. Zheng, Y\. Chen, B\. Qian, X\. Shi, Y\. Shu, and J\. ChenA review on edge large language models: design, execution, and applications\.ACM Computing Surveys57\(8\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2608.19535#S1.p2.1),[§1](https://arxiv.org/html/2608.19535#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.19535#S2.SS1.p2.1)\.