// The Efficiency Frontier //

Cool paper on context management.

As agents reuse the same documents and histories across many turns, the cheapest context strategy is not fixed. This work describes a principled rule for picking one per deployment instead of defaulting to whatever topped a benchmark in isolation.

Retrieval and compression methods are almost always benchmarked on accuracy and cost separately, so you never learn when one actually beats another under real load.

The Efficiency Frontier models context strategy selection as a single cost-performance problem, with a log-utility term for diminishing returns from extra context and a reuse parameter N that amortizes preprocessing across repeated queries.

Sweep N and the optimal strategy changes, exposing crossover regions where retrieval, compression, or full context each wins. On 5,000 HotpotQA instances, deployment-aware selection cuts effective token usage about 25 percent at the same performance, and amortized memory compression runs over 50 percent cheaper than full-context prompting in higher-performance settings.

Paper: https://arxiv.org/abs/2605.23071

Learn to build effective AI agents in our academy: https://academy.dair.ai

The Efficiency Frontier: A Unified Framework for Cost–Performance Optimization in LLM Context Management

Source: https://arxiv.org/html/2605.23071 BINQI SHEN1†∗, LIER JIN2†, HANYU CAI1, LAN HU3, YUTING XIN4 1Northwestern University 2Duke University 3Carnegie Mellon University 4University of Minnesota E-mails: [email protected] [email protected] [email protected] [email protected] [email protected] †Equal contribution∗Corresponding author

Abstract

Large language models (LLMs) increasingly rely on long-context processing, but expanding context windows introduces substantial computational and financial costs. Existing context reduction approaches, including retrieval and memory compression methods, are typically evaluated using performance and efficiency metrics independently, limiting systematic comparison and deployment-aware decision-making.

This paper introducesThe Efficiency Frontier, a unified framework for cost–performance optimization in LLM context management. The framework models context strategy selection as a deployment-aware optimization problem that jointly accounts for task performance, token cost, and preprocessing reuse through amortized cost modeling. Unlike existing evaluations that compare methods in isolation, the proposed framework enables decision-oriented analysis of when different context management strategies become preferable under varying operational conditions. Evaluated on 5,000 HotpotQA instances, the framework reveals distinct operational regimes and transition boundaries between retrieval-based and preprocessing-based strategies. Results show that deployment-aware optimization reduces effective token usage by approximately 25% at comparable performance (F​1≈0.78F1\approx 0.78), while amortized memory compression achieves over 50% lower token cost relative to full-context prompting in higher-performance settings. Overall, the proposed framework provides a principled and practical foundation for evaluating and deploying scalable, efficient, and sustainable LLM systems.

IIntroduction

Large language models (LLMs) have achieved rapid progress in recent years, demonstrating strong capabilities across a wide range of natural language processing tasks, such as search, customer support, and knowledge work[1]. However, these advances come with increasing computational and financial costs, driven by both model scale and the growing length of input context[2]. As context windows continue to expand, the computational cost of processing additional tokens often increases faster than the corresponding gains in downstream task performance, making efficient context utilization an increasingly important challenge[3]. At the same time, the environmental impact of large-scale AI systems, including energy and water consumption, has raised growing concerns about their long-term sustainability[4,5]. These challenges highlight the need for more efficient use of context in LLM systems.

Recent work has explored a variety of techniques for reducing context length while preserving task performance, including retrieval-based filtering, summarization, and context compression methods[6,7]. These approaches aim to improve efficiency by selectively retaining the most relevant information while discarding redundant or less informative content[8]. Although these methods have shown promising results, their evaluation remains fragmented. Existing studies typically report performance metrics such as exact match (EM) or F1 score, alongside cost indicators such as token usage or latency[9]. However, these metrics are often considered independently and rarely provide a unified assessment of the trade-off between cost reduction and performance degradation[10]. Moreover, retrieval, compression, and long-context approaches are frequently evaluated under different experimental settings, making direct comparison difficult. As a result, it remains difficult to systematically compare different context reduction strategies or assess when one strategy should be preferred over another under practical deployment constraints[11].

To address this limitation, we propose a unified evaluation framework for methodically assessing the efficiency of context reduction techniques in large language models. We introduce the concept of theEfficiency Frontier, a three-stage evaluation framework that characterizes the trade-off between task performance and computational cost across different context management strategies. Unlike existing approaches that evaluate performance and cost in isolation, our framework provides explicit decision logic for selecting context management strategies, bridging the gap between retrieval-based methods and long-context processing. The framework incorporates a parameterized log-utility metric to model diminishing returns from additional context while accounting for amortized preprocessing cost. By varying a reuse parameter (NN), the framework supports systematic comparison under realistic deployment constraints by identifying crossover regions where different strategies become preferable.

Beyond evaluation, the framework provides practical guidance for both research and practice during context management strategy selection under different cost and reuse conditions, shifting the focus from maximizing context capacity to optimizing context utilization in real-world LLM systems. To illustrate the proposed framework, we conducted experiments on the HotpotQA dataset[12], which features multi-hop reasoning and includes both relevant and distractor context, enabling unified evaluation of context reduction and its impact on model accuracy.

IIRelated Work

II-AEvaluation of Large Language Models

Recent evaluation frameworks for large language models have expanded beyond task accuracy to incorporate additional dimensions such as robustness, fairness, generalization, computational efficiency, and sensitivity to prompting conditions and interaction style. Beyond task performance, frameworks like HELM and recent specialized benchmarks increasingly emphasize multi-dimensional evaluation of model behavior, particularly regarding the trade-offs between accuracy and execution efficiency[9,13]. At the same time, work on efficient and sustainable AI has highlighted the importance of resource-aware evaluation criteria, including computational cost, energy consumption, and latency[14]. For example, Green AI advocates for incorporating efficiency and resource usage into model evaluation as model scale and deployment costs continue to increase[15]. Beyond general behavior and resource usage, recent work has identified the need for evaluating the suitability of alignment systems, defined as their reliability under real-world perturbations[16]. This shift underscores the necessity of moving beyond static benchmarks toward evaluation frameworks that are verifiably robust under deployment conditions.

However, existing methodologies typically treat task effectiveness, computational cost, and deployment efficiency as independent variables. This fragmentation obscures the practical trade-offs involved in real-world deployment, where practitioners must balance task performance and computational cost without standardized evaluation criteria[17,18]. Many studies report performance metrics such as F1 or compression ratio alongside basic cost indicators, but rarely provide deployment-aware, end-to-end comparisons of per-query token or monetary cost against task performance across different context management strategies. This limitation becomes particularly important in long-context settings, where increases in context length can substantially increase computational cost without consistent gains in downstream performance. Recent work on long-context evaluation shows that increasing context or model complexity does not necessarily yield proportional performance gains[19].

II-BContext Length Scaling and Diminishing Returns

As long-context capabilities continue to expand, recent advances in large language models (LLMs) have significantly increased maximum context length, enabling models to process longer sequences and incorporate more information into their reasoning process. While longer context windows can improve performance in tasks requiring multi-hop reasoning or long-range dependencies, empirical evidence suggests that these gains are often subject to diminishing returns[20].

Research has shown that LLMs do not always effectively utilize long input sequences. The “lost in the middle” phenomenon demonstrates that models tend to underutilize information located in the middle of long sequences[19], while more recent studies report degraded performance due to attention dilution and distractor interference as context length increases[21,22]. Large-scale evaluations further show that models often fail to fully utilize the additional context available to them[23].

At the same time, the computational cost of long-context processing grows disproportionately with sequence length due to the quadratic complexity of attention mechanisms[24], whereas performance improvements are often sublinear or inconsistent[19,25,26]. These limitations have motivated growing interest in methods that reduce or selectively process context in order to improve efficiency while preserving task performance. Existing work, however, primarily focuses on improving long-context capabilities or benchmarking performance rather than systematically modeling the trade-offs between context length, computational cost, and downstream performance.

II-CContext Reduction Techniques

To mitigate the high computational cost associated with long-context processing, a growing body of work has explored techniques for reducing context length while preserving task performance. Recent work has proposed various context compression techniques, including token-level compression strategies and instruction-driven routing mechanisms that selectively sparsify input tokens to reduce inference latency[27,28]. Other studies explore reasoning-enhanced adaptation, instruction tuning, and multimodal fusion strategies for improving context understanding and efficient context utilization in complex LLM settings[29,30,31]. Such strategies are increasingly employed to enable real-time deployment for time-constrained applications[32]. In addition, context reduction methods such as semantic sparsification and filtering techniques aim to remove redundant context prior to generation, improving efficiency, robustness, and risk-aware resilience[33,34,35]. Building on these ideas, hybrid retrieval and routing approaches have also been proposed to further improve robustness and context selection[36].

Existing work primarily evaluates retrieval, compression, and long-context processing in isolation, with comparisons often performed under different datasets, prompting settings, or cost assumptions. As a result, it remains difficult to determine when one strategy is more efficient or effective than another under comparable conditions. This lack of standardized evaluation makes it challenging to reason systematically about efficiency-performance trade-offs across context management strategies, motivating the need for a unified evaluation framework.

IIIMethodology

We propose a structured, three-stage framework for systematically evaluating the trade-off between performance and computational cost in context management strategies for large language models. Unlike prior approaches that optimize accuracy or efficiency in isolation, our framework explicitly modelsdecision-making under deployment constraints, enabling strategy selection conditioned on both performance requirements and system usage patterns.

A central contribution is the distinction betweenintrinsic cost(per-query inference cost) andamortized cost(including reusable preprocessing), captured through a reuse parameterNN. This formulation reflects realistic deployment settings, such as shared memory systems, cached summaries, and multi-query workloads, where upfront computation can be reused across queries. As a result, the framework supports evaluation across heterogeneous operational regimes within a unified objective.

III-AEfficiency Frontier Framework

We formulate context management as adecision problem: given a deployment preference over performance and cost, select the strategy and configuration that maximizes utility.

III-A1Cost Model

We model computation as a two-stage process. LetTstage1T_{\text{stage1}}denote context preprocessing cost (e.g., memory compression), andTstage2T_{\text{stage2}}denote per-query inference cost. When context preprocessing is reused acrossNNqueries, the effective cost is:

EffectiveTokens=Tstage2+Tstage1N\text{EffectiveTokens}=T_{\text{stage2}}+\frac{T_{\text{stage1}}}{N}(1) This distinguishesintrinsic cost(per-query inference) fromamortized costunder context reuse.

III-A2Efficiency Score

We define a parameterized utility function that captures the trade-off between performance and cost:

EfficiencyScore​(w)=w⋅F​1−(1−w)⋅log⁡(EffectiveTokens)\text{EfficiencyScore}(w)=w\cdot F1-(1-w)\cdot\log(\text{EffectiveTokens})(2)wherew∈[0,1]w\in[0,1]controls the preference between performance and efficiency. Largerwwemphasizes accuracy, while smallerwwprioritizes lower cost.

This formulation captures two key properties: (i) amortization of preprocessing cost under reuse viaNN, and (ii) diminishing sensitivity to token cost through the logarithmic penalty, reflecting practical tolerance to cost increases at scale.

III-A3Optimization Procedure

The Efficiency Frontier is constructed in three stages:

• •Stage 1: Intra-Strategy Optimization. For each strategy, we evaluate a range of configurations (e.g., compression ratios, retrieval depth) and retain only those that are optimal under someww: arg⁡maxconfig⁡EfficiencyScore​(w)\arg\max_{\text{config}}\;\text{EfficiencyScore}(w)(3)This step removes dominated configurations while preserving all potentially optimal operating points.
• •Stage 2: Candidate Scoring and Evaluation. All retained configurations are evaluated under the amortized cost formulation above, ensuring consistent comparison across strategies with heterogeneous cost structures.
• •Stage 3: Global Decision Optimization. We aggregate candidates across all strategies and compute the globally optimal choice: arg⁡maxstrategy, config⁡EfficiencyScore​(w)\arg\max_{\text{strategy, config}}\;\text{EfficiencyScore}(w)(4) Sweeping overwwyields a sequence of decision transition points, forming the global Efficiency Frontier. This frontier induces (i) a strategy transition map across preference regimes, and (ii) a lookup table mapping target performance levels to minimum achievable cost.

III-BRole of Amortization (NN)

The parameterNNexplicitly models reuse of preprocessing outputs. WhenN=1N=1, all costs are incurred per query, favoring lightweight or zero-cost methods during context retrieval. AsNNincreases, preprocessing costs are amortized, shifting the optimal frontier toward strategies with higher upfront cost but stronger performance, such as memory compression.

This parameter is critical for modeling real-world systems, where reuse patterns vary significantly across applications (e.g., single-turn queries vs. persistent agent memory), and directly influences optimal strategy selection.

III-CContext Management Strategies

We evaluate representative strategies that span distinct mechanisms of context utilization and cost structures:

• •Full-Context Prompting: concatenation of all available context. This serves as a high-cost baseline that maximizes information availability and approximates upper-bound performance under unconstrained context.
• •Oracle Retrieval: construction of context using only ground-truth supporting documents. This provides a non-deployable upper bound that isolates limitations due to context selection from those due to model reasoning.
• •Memory Compression: LLM-based preprocessing that transforms raw context into a condensed representation prior to inference. This introduces explicit Stage 1 cost while reducing Stage 2 cost, enabling analysis of trade-offs between compression fidelity and computational overhead.
• •Zero-Cost Retrieval Methods: TF-IDF (vanilla and query-aware) and semantic embedding retrieval. Vanilla TF-IDF ranks sentences based solely on corpus statistics, independent of the input question, while query-aware TF-IDF incorporates the input question to prioritize context relevant to the specific task. Semantic embedding retrieval encodes both the Full-Context and question into dense vector representations and retrieves the top-kkdocuments based on semantic similarity. These methods perform context selection without LLM-based preprocessing, incurring no Stage 1 token cost and establishing a lower bound on computational overhead.

Together, these strategies cover a broad design space, including full-context baselines, upper-bound references, model-based preprocessing, and lightweight retrieval approaches. This diversity enables thorough analysis of how different mechanisms interact with cost, performance, and reuse.

III-DEvaluation Setup

We instantiate the framework on HotpotQA, a benchmark designed for multi-hop question answering. HotpotQA requires reasoning across multiple documents and includes both relevant and distractor context, making it well-suited for evaluating context selection and reduction strategies.

Specifically, the dataset captures two key challenges: (i)multi-hop reasoning, which requires preserving relational structure across documents, and (ii)irrelevant context filtering, which stresses the ability of strategies to discard noise. These properties make it a representative testbed for context management under realistic conditions.

We sample 5,000 instances using a fixed random seed (42) to balance computational tractability with coverage. All strategies are evaluated using GPT-5.4 mini[37]under a standardized prompt and deterministic inference configuration to isolate the effect of context management. GPT-5.4 mini was selected to balance reasoning capability and computational scalability: the model is sufficiently strong to support reliable multi-hop reasoning while remaining efficient enough to enable large-scale evaluation across thousands of strategy and configuration combinations.

IVExperiments

We apply the proposed framework to analyze how context management strategies behave under varying performance requirements and deployment constraints. Rather than comparing strategies in isolation, the framework reveals how optimal choices depend jointly on target performance, computational budget, and reuse patterns.

Results are presented through three complementary analyses: (i) Efficiency Frontier Analysis, which characterizes intra- and inter-strategy trade-offs, (ii) Decision Patterns Across Regimes, which maps performance targets to deployment-dependent optimal strategies, and (iii) Learnings and Practical Implications, which distills broader system-level insights for LLM deployment. All reported token costs correspond to effective per-question token usage after amortizing preprocessing costs according to Eq. (1) and averaged over the evaluated HotpotQA instances.

IV-AEfficiency Frontier Analysis

Figure 1:Strategy-level Efficiency Frontiers and decision paths. Each panel plots token cost versus task performance (F1). Faint points denote all evaluated configurations, dashed lines indicate the Pareto frontier, and solid lines trace optimal configurations under varying preference weightww. Across strategies, the optimal operating point changes continuously with preference weightww, indicating that no single configuration is universally optimal across deployment settings.Figure 2:Global Efficiency Frontier under different reuse regimes (NN). Each curve traces the sequence of optimal (strategy, configuration) choices as preference weightwwvaries. As reuse increases, preprocessing-based methods such as memory compression become optimal across a broader range of preference weights, replacing lightweight retrieval methods in parts of the frontier.Fig.1provides a unified view of strategy-level Efficiency Frontiers and decision paths. Each panel captures three layers: (i) the full configuration space (all evaluated parameter settings), (ii) the intrinsic Pareto frontier (non-dominated trade-offs), and (iii) the optimal strategy operating path induced by the efficiency score.

This reveals a key structural property:each strategy admits a single intrinsic frontier, but multiple deployment-dependent operating points. As preference weightwwshifts, the optimal configuration moves along the frontier rather than remaining fixed.

• •Query-aware retrieval improves the intrinsic frontier. Compared to vanilla TF-IDF, query-aware retrieval consistently achieves higher performance at comparable cost, shifting the Pareto frontier upward without increasing preprocessing overhead.
• •Memory compression changes position under amortization. Unlike retrieval methods, memory compression introduces substantial preprocessing cost. However, as reuse increases, amortization reduces its effective cost, causing memory compression to occupy a larger portion of the global frontier.

Fig.2aggregates these frontiers into a global view. Each curve traces the sequence of optimal (strategy, configuration) choices as preference weightwwshifts from cost-sensitive to performance-sensitive regimes, with transition points indicating where one strategy becomes more efficient than another.

A consistent pattern emerges: higher reuse (NN) favors strategies with higher upfront cost. AsNNincreases, amortization reduces effective cost, enabling memory compression to dominate across broader balanced regimes. This demonstrates that optimal strategy selection is inherently deployment-dependent rather than fixed: isolated single-query workloads tend to favor lightweight retrieval methods, whereas persistent assistants and enterprise knowledge systems increasingly benefit from preprocessing-heavy approaches such as memory compression.

IV-BDecision Patterns Across Regimes

While the Efficiency Frontier provides a continuous view of optimal decisions across preference weights, practitioners often require discrete guidance: given a target performance level, which strategy should be selected? To address this, we translate the frontier into a decision-oriented view, mapping performance targets to optimal strategies under different deployment conditions.

To better understand how the framework informs decision-making, we group operating points into three practical regimes based on performance requirements: (i) efficiency-oriented regimes, (ii) balanced regimes, and (iii) high-performance regimes. TableIsummarizes representative operating points from the global frontier, translating the continuous frontier into a discrete, deployment-oriented decision guide.

• •Efficiency-oriented regime (F​1<0.78F1<0.78).Lightweight retrieval methods dominate due to minimal cost. In low-reuse settings, TF-IDF QA achieves competitive performance at the lowest token usage.
• •Balanced regime (0.78≤F​1<0.820.78\leq F1<0.82).This regime exhibits the most variation across deployment settings. Under higher reuse, memory compression becomes increasingly favorable, often becoming competitive or dominant in balanced regimes.
• •High-performance regime (F​1≥0.82F1\geq 0.82).Full-Context prompting remains necessary to achieve peak performance. However, this comes at a significant cost increase, often exceeding 2×\timesthat of balanced configurations, indicating diminishing returns.

TABLE I:Dominant strategy across performance regimes and reuse levels.The framework enables direct quantification of efficiency gains across regimes:

• •In the Balanced regime, increasing reuse fromN=1N=1toN=100N=100shifts the optimal operating point from TF-IDF QA (566 EffectiveTokens) to memory compression (424 EffectiveTokens), reducing effective cost by approximately 25% at comparable performance (F​1=0.78F1=0.78).
• •In the High-performance regime (F​1≥0.82F1\geq 0.82), full-context prompting remains necessary, but incurs substantial cost: achieving the highest evaluated performance levels with full-context prompting requires more than 2×\timesthe EffectiveTokens cost of balanced-regime operating points. This highlights a consistent pattern of diminishing returns at higher performance levels.

While the specific transition points and thresholds reported here are derived from HotpotQA, the observed structure, namely the existence of distinct efficiency regimes and strategy transition boundaries, arises from the underlying cost–performance trade-off and is expected to generalize across tasks with similar context characteristics.

IV-CLearnings and Practical Implications

IV-C1A unified framework is necessary

Analysis across all configurations reveals a strongly non-linear relationship between performance and token cost. Achieving higher performance requires disproportionately larger increases in computation: for example, improving from mid-range performance (F​1≈0.78F1\approx 0.78) to high performance (F​1≈0.84F1\approx 0.84) more than doubles token usage.

Evaluating strategies using performance or cost in isolation therefore obscures deployment-dependent trade-offs and can lead to misleading conclusions. The proposed efficiency metric provides a unified view, enabling principled comparison and decision-making across competing strategies.

IV-C2System-level efficiency gains

Aligning strategy selection with deployment conditions yields substantial efficiency improvements. In our setting, this translates to approximately 25% reduction in token usage at comparable performance in the balanced regime (F​1≈0.78F1\approx 0.78) when moving from a low-reuse setting (N=1N=1) to a high-reuse setting (N=100N=100), shifting the optimal operating point from TF-IDF QA (566 EffectiveTokens) to Memory Compression (424 EffectiveTokens). Similarly, near the upper end of the balanced regime (F​1≈0.80F1\approx 0.80), increasing reuse fromN=1N=1toN=100N=100shifts the optimal strategy from Full-Context (1308 EffectiveTokens) to Memory Compression (584 EffectiveTokens), yielding over 50% effective cost reduction. These gains arise not from improving individual strategies, but from selecting the right strategy under the right conditions.

IV-C3Strategy selection is deployment-dependent

No single strategy dominates across all scenarios. Lightweight retrieval methods are optimal in low-cost regimes, while preprocessing-based approaches such as memory compression become increasingly favorable when reuse is present. At the highest performance levels, Full-Context remains necessary despite its cost. This highlights the importance of incorporating deployment factors, such as reuse and performance targets, into evaluation.

IV-C4Operational guidance

The framework supports two complementary modes of use. First, sweeping the preference parameterwwprovides a continuous view of trade-offs between performance and cost. Second, the decision table offers a discrete mapping from target performance to optimal strategy, enabling straightforward integration into system design and deployment pipelines.

IV-C5Implications for sustainable AI systems

By enabling systematic reductions in unnecessary computation, the framework provides a practical pathway toward more efficient and sustainable deployment of large-scale language models. Rather than scaling context indiscriminately, it supports targeted context utilization, reducing computational overhead while preserving task performance.

VConclusion

This work proposes theEfficiency Frontier, a unified framework for evaluating context management strategies in large language models under explicit performance–cost trade-offs. Rather than treating accuracy and computational efficiency as separate objectives, the framework models strategy selection as a deployment-dependent decision problem, jointly accounting for task performance, inference cost, and amortized preprocessing reuse.

Across experiments on HotpotQA, several consistent patterns emerge. First, the relationship between performance and computational cost is strongly non-linear: achieving incremental gains in performance often requires disproportionately larger increases in token usage. Second, no single context management strategy is universally optimal. Lightweight retrieval methods dominate efficiency-oriented regimes, while preprocessing-based methods such as memory compression become increasingly favorable under high reuse settings due to amortization effects. Finally, full-context prompting remains necessary for peak performance, but exhibits clear diminishing returns relative to its computational cost.

These findings address several limitations in existing evaluations of context reduction methods. The proposed framework provides: (i) a unified objective that jointly models performance and cost, (ii) explicit operational transition points between competing strategies, and (iii) deployment-aware recommendations conditioned on reuse patterns and performance targets. By incorporating amortized preprocessing cost through the reuse parameterNN, the framework also captures practical deployment scenarios that are often overlooked in existing benchmarking settings, including persistent memory systems, shared retrieval pipelines, and multi-query inference workloads.

Beyond evaluation, the framework provides practical value for both academic research and real-world deployment. In research settings, the Efficiency Frontier offers a standardized methodology for comparing heterogeneous context management approaches under a common decision framework, enabling more reproducible and deployment-relevant evaluation of emerging LLM optimization methods. In industry settings, the framework can support system-level optimization by guiding strategy selection according to operational constraints such as latency budgets, token cost limits, and query reuse patterns. Experimental results demonstrate that deployment-aware strategy selection can reduce EffectiveTokens usage by approximately 25% at comparable performance in mid-range operating regimes, and by more than 50% relative to full-context baselines under amortized reuse conditions. These improvements are achieved not through larger models or additional training, but through more efficient utilization of context.

More broadly, the proposed framework contributes toward the development of more sustainable large-scale AI systems. As LLM deployment continues to scale across enterprise, scientific, and public-sector applications, computational efficiency is becoming increasingly important not only for economic reasons, but also for infrastructure scalability and environmental sustainability. By formalizing the trade-off between performance and context cost, the Efficiency Frontier shifts the focus from maximizing context length toward optimizing context utilization.

Several directions remain for future work. First, the framework can be extended beyond question answering to other long-context tasks, including agent memory systems, code generation, document reasoning, and conversational assistants. Second, future work may incorporate additional system-level objectives such as latency, energy consumption, hardware utilization, or monetary cost into the optimization framework. Third, the current formulation assumes a fixed utility function; adaptive or learned preference models may better capture application-specific deployment priorities. Finally, while this study evaluates representative retrieval and compression methods, future work may further improve context management by incorporating structured domain knowledge and more specialized optimization objectives into representation learning and compression pipelines. Prior work in domain-aware representation learning has shown that tailored latent modeling frameworks and optimization objectives can improve representation fidelity and capture complex underlying relationships in high-dimensional systems[38,39].

Overall, the proposed Efficiency Frontier framework establishes a principled and practical foundation for deployment-aware optimization of context utilization in large language model systems.

References

• [1]M. Raza, Z. Jahangir, M. B. Riaz, M. J. Saeed, and M. A. Sattar, “Industrial applications of large language models,”Scientific Reports, vol. 15, no. 1, p. 13755, Apr. 2025.
• [2]L. Zhang, X. Liu, Z. Li, X. Pan, P. Dong, R. Fan, R. Guo, X. Wang, Q. Luo, S. Shi, and X. Chu, “Dissecting the runtime performance of the training, fine-tuning, and inference of large language models,” 2023. [Online]. Available:https://arxiv.org/abs/2311.03687
• [3]C. Wu, H. Huang, and Y.-Q. Ni, “Evaluation of tunnel rock mass integrity using multi-modal data and generative large model: Tunnel rip-gpt,”SSRN Electronic Journal, 2025. [Online]. Available:https://ssrn.com/abstract=5348429
• [4]C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Baiet al., “Sustainable ai: Environmental implications, challenges and opportunities,”Proceedings of machine learning and systems, vol. 4, pp. 795–813, 2022.
• [5]P. López-Úbeda, T. Martín-Noguerol, and A. Luna, “Environmental and economic costs behind llms,”Nature Reviews Electrical Engineering, vol. 21, no. 3, pp. 661–663, Mar. 2026.
• [6]H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu, “Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 1658–1677.
• [7]P. Xu, W. Ping, X. Wu, L. McAfee, C. Zhu, Z. Liu, S. Subramanian, E. Bakhturina, M. Shoeybi, and B. Catanzaro, “Retrieval meets long context large language models,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 49 569–49 584.
• [8]D. Jiang, Y. Li, G. Li, and B. Li, “Magma: A multi-graph based agentic memory architecture for ai agents,”arXiv preprint arXiv:2601.03236, 2026.
• [9]“Holistic evaluation of language models,” 2023. [Online]. Available:https://arxiv.org/abs/2211.09110
• [10]N. Pollertlam and W. Kornsuwannawit, “Beyond the context window: A cost-performance analysis of fact-based memory vs. long-context llms for persistent agents,” 2026. [Online]. Available:https://arxiv.org/abs/2603.04814
• [11]D. Jiang, Y. Li, S. Wei, J. Yang, A. Kishore, A. Zhao, D. Kang, X. Hu, F. Chen, Q. Liet al., “Anatomy of agentic memory: Taxonomy and empirical analysis of evaluation and system limitations,”arXiv preprint arXiv:2602.19320, 2026.
• [12]Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” 2018. [Online]. Available:https://arxiv.org/abs/1809.09600
• [13]Z. Zhao and B. M. Chen, “Benchmark for evaluating initialization of visual-inertial odometry,” in2023 42nd Chinese Control Conference (CCC). IEEE, 2023, pp. 3935–3940.
• [14]J. Rao, X. Liu, H. Yan, J. Shen, H. Mo, Y. Dong, Z. Yan, Z. Wang, Z. Lin, X. Meng, Z. Yu, L. Deng, J. Wei, Y. Wang, and M. Zhang, “A data-centric perspective on the lifecycle of large language models,”TechRxiv, vol. 2025, no. 1220, 2025. [Online]. Available:https://www.techrxiv.org/doi/abs/10.36227/techrxiv.176620610.03288677/v1
• [15]R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,”Communications of the ACM, vol. 63, no. 12, pp. 54–63, 2020.
• [16]J. Zang, Y. Wei, R. Bai, S. Jiang, N. Mo, B. Li, Q. Sun, and H. Liu, “Reward auditor: Inference on reward modeling suitability in real-world perturbed scenarios,”arXiv preprint arXiv:2512.00920, 2025.
• [17]W. Sun, Z. Qi, and Q. Shen, “High-recall deep learning: A gated recurrent unit approach to bank account fraud detection on imbalanced data,” in2025 5th International Conference on Digital Society and Intelligent Systems (DSInS), 2025, pp. 207–212.
• [18]J. Cao, Y. Ma, X. Li, Q. Ren, and X. Chen, “Task-specific efficiency analysis: When small language models outperform large language models,” 2026. [Online]. Available:https://arxiv.org/abs/2603.21389
• [19]N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,”Transactions of the association for computational linguistics, vol. 12, pp. 157–173, 2024.
• [20]Y. Du, M. Tian, S. Ronanki, S. Rongali, S. Bodapati, A. Galstyan, A. Wells, R. Schwartz, E. A. Huerta, and H. Peng, “Context length alone hurts llm performance despite perfect retrieval,” 2025. [Online]. Available:https://arxiv.org/abs/2510.05381
• [21]R. Bansal, A. Zhang, R. Tiwari, L. Madaan, S. S. Duvvuri, D. Khatri, D. Brandfonbrener, D. Alvarez-Melis, P. Bhargava, M. S. Kaleet al., “Let’s (not) just put things in context: Test-time training for long-context llms,”arXiv preprint arXiv:2512.13898, 2025.
• [22]S. Gu, “Long context, less focus: A scaling gap in llms revealed through privacy and personalization,”arXiv preprint arXiv:2602.15028, 2026.
• [23]Z. Chen, X. Wu, J. Jia, C. Gao, Q. Fu, D. Zhang, and S. Hu, “Longbench pro: A more realistic and comprehensive bilingual long-context evaluation benchmark,”arXiv preprint arXiv:2601.02872, 2026.
• [24]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017.
• [25]Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Houet al., “Longbench: A bilingual, multitask benchmark for long context understanding,” inProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 3119–3137.
• [26]L. Lai, Z. Cheng, K. Cheng, and X. Qi, “Do transformers always win? an empirical study of semantic embeddings for short-text e-commerce reviews,” in2026 9th International Symposium on Big Data and Applied Statistics (ISBDAS), 2026, pp. 525–529.
• [27]T. Ge, J. Hu, L. Wang, X. Wang, S.-Q. Chen, and F. Wei, “In-context autoencoder for context compression in a large language model,”arXiv preprint arXiv:2307.06945, 2023.
• [28]W. Li, R. Zhang, R. Shao, J. He, and L. Nie, “Cogvla: Cognition-aligned vision-language-action model via instruction-driven routing & sparsification,” inAdvances in Neural Information Processing Systems, 2025.
• [29]Z. Wang, Y. Sun, H. Wang, B. Jing, X. Shen, X. Dong, Z. Hao, H. Xiong, and Y. Song, “Reasoning-enhanced domain-adaptive pretraining of multimodal large language models for short video content governance,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, S. Potdar, L. Rojas-Barahona, and S. Montella, Eds. Suzhou (China): Association for Computational Linguistics, Nov. 2025, pp. 1104–1112. [Online]. Available:https://aclanthology.org/2025.emnlp-industry.77/
• [30]Y. Sun, Y. Li, R. Sun, C. Liu, F. Zhou, Z. Jin, L. Wang, X. Shen, Z. Hao, and H. Xiong, “Audio-enhanced vision-language modeling with latent space broadening for high quality data expansion,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, ser. KDD ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 4872–4881. [Online]. Available:https://doi.org/10.1145/3711896.3737195
• [31]L. Li, S. Jia, J. Wang, Z. Jiang, F. Zhou, J. Dai, T. Zhang, Z. Wu, and J.-N. Hwang, “Human Motion Instruction Tuning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
• [32]Z. Zhao, “Balf: Simple and efficient blur aware local feature detector,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 3362–3372.
• [33]W. Li, R. Zhang, R. Shao, Z. Fang, K. Zhou, Z. Tian, and L. Nie, “Semanticvla: Semantic-aligned sparsification and enhancement for efficient robotic manipulation,” inProceedings of the AAAI Conference on Artificial Intelligence, 2026.
• [34]J. Rao, X. Liu, H. Deng, Z. Lin, Z. Yu, J. Wei, X. Meng, and M. Zhang, “Dynamic sampling that adapts: Iterative dpo for self-aware mathematical reasoning,” 2025. [Online]. Available:https://arxiv.org/abs/2505.16176
• [35]Z. Xue, S. Zhao, Y. Qi, X. Zeng, and Z. Yu, “Resilient routing: Risk-aware dynamic routing in smart logistics via spatiotemporal graph learning,” 2026. [Online]. Available:https://arxiv.org/abs/2601.13632
• [36]Z. Cheng, L. Lai, and Y. Liu, “Resolving the robustness-precision trade-off in financial rag through hybrid document-routed retrieval,” 2026. [Online]. Available:https://arxiv.org/abs/2603.26815
• [37]OpenAI, “GPT-5.4 mini,” OpenAI, Technical Report, 2026. [Online]. Available:https://platform.openai.com/docs/models
• [38]W. Yan, E. Wu, A. G. Schwing, and E. Rosenbaum, “Semantic autoencoder for modeling beol and mol dielectric lifetime distributions,” in2023 IEEE International Reliability Physics Symposium (IRPS). IEEE, 2023, pp. 1–9.
• [39]W. Yan, E. Wu, and E. Rosenbaum, “New loss function for learning dielectric thickness distributions and generative modeling of breakdown lifetime,” in2025 IEEE International Reliability Physics Symposium (IRPS). IEEE, 2025, pp. 1–9.