Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction

# Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction
Source: [https://arxiv.org/html/2608.11242](https://arxiv.org/html/2608.11242)
Zhiqi Wang, Yichi Zhang, Dongwon Lee, Yuchen Yang The Pennsylvania State University \{zhiqi\.wang,yichi\.zhang,dongwon,yuchen\.yang\}@psu\.edu

###### Abstract

When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks\. We identify a class of user\-issued instructions, Session Constraints \(SCs\), such as "do not delete any emails until I confirm," that are meant to constrain LLM’s behavior for the remainder of a session but are silently dropped during compaction\. To quantify this loss, we introduceCompInt, an evaluation suite that evaluates compactors across three long\-context scenarios: multi\-turn chat, agentic trajectory, and long\-horizon research\.

Current compactors retain only 17% of injected SCs on average, and most perform worse than running the same task without compaction\. Retention varies sharply with compactor, prompt, context length, SC phrasing, and injection location, showing that the loss is systematic rather than tied to any single setting\. We propose an SC\-aware extractor that runs alongside the compactor as a plug\-and\-play module, achieving over 90% retention across all three scenarios without modifying the compactor or LLM\. TheCompIntevaluation suite and accompanying implementation are available at[https://github\.com/ZhiqiEliWang/compaction\-integrity](https://github.com/ZhiqiEliWang/compaction-integrity)\.

Lost in Compaction: Evaluating Side\-Constraint Loss under Context Compaction

Zhiqi Wang, Yichi Zhang, Dongwon Lee, Yuchen YangThe Pennsylvania State University\{zhiqi\.wang,yichi\.zhang,dongwon,yuchen\.yang\}@psu\.edu

## 1Introduction

Long context enables LLM\-based AI systems to handle more complex tasksJimenezet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib21)\); Liet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib7)\)\. But open\-weight models and API services specify a maximum context window, and when a session approaches that limit, the canonical solution is to compact the prior contextSteinberger and OpenClaw Contributors \([2025](https://arxiv.org/html/2608.11242#bib.bib15)\); OpenAI \([2026a](https://arxiv.org/html/2608.11242#bib.bib24)\)\. A compactor takes the conversation history as input and produces a summary that replaces it, reducing context pressure while supplying continuous context to subsequent turnsWuet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib27)\); Shinnet al\.\([2023](https://arxiv.org/html/2608.11242#bib.bib17)\); Steinberger and OpenClaw Contributors \([2025](https://arxiv.org/html/2608.11242#bib.bib15)\); Anthropic \([2026](https://arxiv.org/html/2608.11242#bib.bib29)\)\. Because the compacted state is far shorter than the original, what the compactor chooses to retain matters\.

![Refer to caption](https://arxiv.org/html/2608.11242v1/x1.png)Figure 1:Constraint violation induced by context compaction\.\(A\): The user issues aside constraint \(SC\),“confirm with me before any action,”which the agent initially honors by proposing an edit and awaiting approval\. \(B\): When the context window fills, the compactor produces a summary that preserves the main task progress \(calendar reorganized, email sent\) but drops the SC\. \(C\): Operating on the compacted context alone, the agent later executes a calendar change directly, violating the user’s original constraint\.However, compactors are designed around*task continuity*: they preserve the objective, working state, and next steps so subsequent turns can resume the dominant taskAnthropic \([2026](https://arxiv.org/html/2608.11242#bib.bib29)\)\(e\.g\., "please organize my email"\)\. This works for content the compactor recognizes as task\-relevant, but it underserves a different class of context\. Throughout an interaction, users also issue instructions that constrain how the task is carried out rather than what the task is, such as "confirm with me before taking any action\." We call theseSessionConstraints\(SCs\)\. Unlike system prompts, SCs are temporary constraints scoped to the current interaction\. They are not durable memory, so after context compression they remain effective only if the compactor preserves them\. This makes SCs fragile: they are often stated once, phrased as soft preferences, and embedded in task\-focused turns\. A task\-centric compactor may therefore preserve the task but drop the constraint, causing a silent integrity failure: the agent continues the task while violating how the user asked it to proceed\. For example, the user asks not to send an email without approval; after compaction, this constraint is dropped and the agent sends it\. Figure[1](https://arxiv.org/html/2608.11242#S1.F1)illustrates this failure mode\.

To quantify this loss, we introduceCompInt, an evaluation suite organized along four orthogonal axes: \(i\)SC content, drawn from a five\-category taxonomy; \(ii\)framing, varying the strength and explicitness of each constraint; \(iii\)injection condition, varying SC position and multiplicity; and \(iv\)long\-context environment, covering multi\-turn chat, multi\-turn agent workflow, and single\-turn long\-horizon research\. Crossing these axes yields 750 evaluation instances per compaction condition\.

WithCompInt, we investigate four research questions:

- ∙\\bulletRQ1:To what extent do current compaction methods preserve SCs?
- ∙\\bulletRQ2:How do compaction\-system factors \(compactor choice, context length, compression rate\) affect SC retention?
- ∙\\bulletRQ3:How do SC\-side factors \(declaration location, repetition, surface framing, SC type\) affect SC retention?
- ∙\\bulletRQ4:How can we design an SC\-aware compaction system that mitigates this loss?

We show that this loss is systematic across compactors, datasets, and SC framings, and that closing it requires architectural separation rather than better compactor prompts\. Concretely, we contribute:\(i\)a formalization of Session Constraints, a class of user\-issued constraint whose lifetime is one session and whose preservation depends entirely on the compactor, together with a five\-category taxonomy;\(ii\)CompInt, a benchmark for SC integrity under compaction across three long\-context scenarios \(multi\-turn chat, agentic trajectory, and long\-horizon research\), with controlled axes for SC content, framing, and injection condition;\(iii\)a systematic evaluation showing that compactors we tested only retains 17% of injected SCs on average, with retention rate varying by compactor choice, context length, SC framing, and injection location; and\(iv\)an SC\-aware extractor that operates alongside the compactor as a strategy\-layer intervention, achieving over 90% retention across all three datasets\. TheCompIntevaluation suite and accompanying implementation—including dataset construction, experiment runners, analysis code, and the SC\-aware extractor—are available on GitHub\.111[https://github\.com/ZhiqiEliWang/compaction\-integrity](https://github.com/ZhiqiEliWang/compaction-integrity)

## 2Related Work

#### Context Compaction\.

Prior work on context compaction falls into three categories\.\(1\) Context truncation:Shinnet al\.\([2023](https://arxiv.org/html/2608.11242#bib.bib17)\)designed a short\-term memory keeping only the 3 most recent self\-reflections, a method also adopted byYanget al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib16)\); Wuet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib27)\)\.\(2\) Non\-prompt\-based language model compression:Panet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib39)\)train a BERT\-like binary classifier to decide which tokens to preserve, whileLiet al\.\([2023](https://arxiv.org/html/2608.11242#bib.bib44)\)remove tokens with lower perplexity\.\(3\) Prompt\-based LLM summarization:the canonical method in most agentsOpenAI \([2026a](https://arxiv.org/html/2608.11242#bib.bib24)\); Anthropic \([2026](https://arxiv.org/html/2608.11242#bib.bib29)\); Xuet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib37)\), which prompts an LLM with a compaction instruction \(e\.g\., identify main tasks, user instructions and requirements\) to produce a summary of the full context\.

#### Long Context Constraint Following\.

Zhaoet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib8)\)show that LLMs can proactively adhere to user preferences, but this capability degrades when only a few turns separate the preference statement from the query\. Their focus is preference\-following under extended contexts, whereas ours is constraint\-following under compaction, and our taxonomy extends beyond preferences to four additional categories\.Robinetteet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib6)\)propose a verifiable instruction\-following dataset for long\-context adherence,*but evaluate only on static contexts, not the dynamic information loss from compaction that our work targets\.*

## 3Side Constraints

We define a side constraint \(SC\)ssas a clause in a user’s prompt such that\(1\)ssis not part of the user’s task,\(2\)ssis meant to constrain the LLM’s decoding within the same session, and\(3\)sshas no intended use outside that session\. Linguistically,ssis a*generic*directive \(one that targets a kind of action or output\) rather than an*episodic*one \(one that targets a specific action in the current turn\)\. For example, in “Email Sarah and let her know I’ll be late, but show me the draft before sending anything from now on”, “Email Sarah…” is episodic: the email is the task, predicated over one identifiable email\. The clause “show me the draft before sending anything from now on” is generic: “sending anything” denotes a kind of action, so the directive is meant to apply to every send\-like action for the rest of the session, and is not needed to draft the email itself\. This is the SC\.

These three qualifiers make SCs fragile under compaction\. An SC is not the main task, so a task\-centric compactor prompt has no explicit intent to retain it\. It might be phrased as a soft preference by average users who are not familiar with prompt engineering, which a compactor can read as non\-essential\. A compactor that preserves the user’s goal and working state can drop the SC while retaining task\-relevant signals\.

We organize SCs by what the constraint binds\.ActionandInformationbind what the agent does and emits;Processbinds how it reaches an answer or action;Preferencebinds which of several task\-equivalent answers it picks; andOutputbinds textual properties of the response\. The definition of each SC is in Appendix Table[13](https://arxiv.org/html/2608.11242#A6.T13), and the 15 SC examples used in our benchmark are in Appendix Table[12](https://arxiv.org/html/2608.11242#A6.T12)\.

## 4CompInt

OurCompIntis designed to comprehensively evaluate the integrity of SCs after context compaction\. It consists 3 components:Long Context Environment,SC Samples and Probes, andSC Injection\.

![Refer to caption](https://arxiv.org/html/2608.11242v1/x2.png)Figure 2:Overview ofCompInt\.This evaluation suite has three components\.SC Taxonomy\(left\): five categories of side constraints\.Injection Condition\(middle\): each SC is rendered under four framings crossing*Constraint Strength*\(Preferential vs\. Strict\) with*Explicitness*\(Contextualized vs\. Direct\), then injected into a long context at one of four locations \(Top, Middle, Bottom, Multi\)\.Evaluation\(right\): the injected context is compacted, and we measure*Retention*, whether the SC is present in the compacted summary via LLM\-as\-a\-judge, and*Compliance*, whether the model behaves according to the SC on a probing multiple\-choice query\.### 4\.1Preliminaries

Given an LLM\-based agent𝒜\\mathcal\{A\}and a user𝒰\\mathcal\{U\}, at a given timestampttin the conversationHH, the conversation history is

Ht=\[x𝒰0,x𝒜0,…,x𝒰t,x𝒜t\],H^\{t\}=\\left\[x^\{0\}\_\{\\mathcal\{U\}\},x^\{0\}\_\{\\mathcal\{A\}\},\\dots,x^\{t\}\_\{\\mathcal\{U\}\},x^\{t\}\_\{\\mathcal\{A\}\}\\right\],\(1\)wherex𝒰x\_\{\\mathcal\{U\}\}is the user’s input \(i\.e\., instruction\),x𝒜x\_\{\\mathcal\{A\}\}is the agent’s response\.

At any turnt\+1t\+1, the output from the agentx𝒜t\+1x\_\{\\mathcal\{A\}\}^\{t\+1\}is conditioned on the user inputx𝒰t\+1x^\{t\+1\}\_\{\\mathcal\{U\}\}and the history contextHtH^\{t\}:

x𝒜t\+1=L​L​Mθ​\(\[Ht,x𝒰t\+1\]\),x\_\{\\mathcal\{A\}\}^\{t\+1\}=LLM\_\{\\theta\}\(\[H^\{t\},x\_\{\\mathcal\{U\}\}^\{t\+1\}\]\),\(2\)withL​L​MθLLM\_\{\\theta\}being the LLM of the agent parametrized byθ\\theta\. Let the max context length ofL​L​MθLLM\_\{\\theta\}belmaxl\_\{\\text\{max\}\}, and let the system serving it apply a coefficientαl\\alpha\_\{l\}on top oflmaxl\_\{\\text\{max\}\}\. When\|Ht\|≥αl⋅lmax\|H^\{t\}\|\\geq\\alpha\_\{l\}\\cdot l\_\{\\text\{max\}\}, where\|⋅\|\|\\cdot\|denotes token length, a compactorCCproduces a compacted history

H~t=C​\(Ht\),\\tilde\{H\}^\{t\}=C\(H^\{t\}\),\(3\)which replacesHtH^\{t\}in Equation[2](https://arxiv.org/html/2608.11242#S4.E2)for all subsequent turns\.

### 4\.2Long Context Environment

Table 1:Statistics of synthetic long\-context conversations at 100k token length\.All columns report mean values over the subset used in evaluation\. \# Stitched: number of datapoints from the original dataset used for stitching a context\.To evaluate the compaction mechanism under real\-world scenarios, we adopt 3 different datasets as the filler context, each representing a different environment for LLM systems:

- ∙\\bulletWildChatZhaoet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib4)\): anonymous conversations between online users and ChatGPT \(GPT\-4 and GPT\-3\.5 variants, without tool use or reasoning\), serving as an environment for multi\-turn long conversationZhaoet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib8)\)\.
- ∙\\bulletHermes AgentMueller \([2026](https://arxiv.org/html/2608.11242#bib.bib9)\): trajectories of the Hermes agent harnessNous Research \([2026](https://arxiv.org/html/2608.11242#bib.bib28)\)on tool\-calling, multi\-step reasoning, and coding tasks, serving as long\-trajectory collaborative work between a user and an agent\.
- ∙\\bulletOpenResearcherLiet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib7)\): long\-horizon deep research trajectories with a minimal browser tool set \(search, open, find\), serving as a single\-user\-turn, long\-horizon agentic task in which the agent autonomously conducts iterative cycles of reasoning and tool invocation\.

Compaction is often triggered when an LLM system’s context approaches its predefined maximum lengthWuet al\.\([2026](https://arxiv.org/html/2608.11242#bib.bib27)\); Steinberger and OpenClaw Contributors \([2025](https://arxiv.org/html/2608.11242#bib.bib15)\); Anthropic \([2026](https://arxiv.org/html/2608.11242#bib.bib29)\)\. We construct our datasets by cropping and concatenating trajectories to 100K tokens, approximately 80% of a 128K\-token context window, to simulate a near\-limit compaction trigger\. For WildChat and Hermes Agent, we concatenate dataset entries; Appendix Section[A\.1](https://arxiv.org/html/2608.11242#A1.SS1)describes our stitching algorithm and discusses the complementary roles and external validity of the three datasets\. OpenResearcher contains native long\-horizon trajectories, so we select data points longer than 100K tokens and truncate each trajectory at a turn boundary while retaining at least 100K tokens\. These complementary settings span different levels of session coherence, and retention degradation persists across all three\. For each dataset, we construct 50 long\-context instances\.

### 4\.3SC Injection and Probe

SC and Probe ExampleSC:Before you run a command, send a message, or make a change, show me what you’re about to do and wait for my go\-ahead\.
Probe:“Email Sarah and let her know I’ll be late\.”A:Show the drafted email and wait for the user to approve\.B:Send the email\.Which behavior is correct given your instructions?Figure 3:An example of an action\-boundary SC and its probing setup\.AnswerAis the complied option, and answerBis the uncomplied option\. The complete list of SCs and probes is in Appendix Table[12](https://arxiv.org/html/2608.11242#A6.T12)\.We evaluate SC integrity under long context compaction by injecting an SC into the conversation history\. Given an SCss, context to be injectedHt=\[x𝒰0,x𝒜0,…,x𝒰t,x𝒜t\]H^\{t\}=\\left\[x^\{0\}\_\{\\mathcal\{U\}\},x^\{0\}\_\{\\mathcal\{A\}\},\\dots,x^\{t\}\_\{\\mathcal\{U\}\},x^\{t\}\_\{\\mathcal\{A\}\}\\right\], and a location setI⊆\{0,1,…,t\}I\\subseteq\\\{0,1,\\dots,t\\\}, we define the injection operatorInjthat augments each selected user turn:

x~𝒰i=x𝒰i⊕s,∀i∈I,\\tilde\{x\}^\{i\}\_\{\\mathcal\{U\}\}=x^\{i\}\_\{\\mathcal\{U\}\}\\oplus s,\\quad\\forall i\\in I,\(4\)where⊕\\oplusdenotes textual concatenation\. The resulting injected history is

Hs,It=Inj​\(Ht,s,I\),H^\{t\}\_\{s,I\}=\\texttt\{Inj\}\(H^\{t\},s,I\),\(5\)which is identical toHtH^\{t\}except thatx𝒰ix^\{i\}\_\{\\mathcal\{U\}\}is replaced byx~𝒰i\\tilde\{x\}^\{i\}\_\{\\mathcal\{U\}\}for everyi∈Ii\\in I\. LetUt=\{0,1,…,t\}U^\{t\}=\\\{0,1,\\dots,t\\\}denote the set of user turn indices\. We instantiate four canonical injection conditions that vary the location and multiplicity ofssin the context:

- ∙\\bulletTop\(Itop=\{0\}I\_\{\\text\{top\}\}=\\\{0\\\}\):ssis placed in the first user turn\.
- ∙\\bulletMiddle\(Imid=\{⌊t/2⌋\}I\_\{\\text\{mid\}\}=\\\{\\lfloor t/2\\rfloor\\\}\):ssis placed at the median user turn\.
- ∙\\bulletBottom\(Ibot=\{t\}I\_\{\\text\{bot\}\}=\\\{t\\\}\):ssis placed in the most recent user turn prior to compaction\.
- ∙\\bulletMulti\(Imulti∼Uniform​\(Utk\)I\_\{\\text\{multi\}\}\\sim\\text\{Uniform\}\\binom\{U^\{t\}\}\{k\}\):ssis restated atk=min⁡\(r,\|Ut\|\)k=\\min\(r,\|U^\{t\}\|\)user turns drawn uniformly at random without replacement fromUtU^\{t\}, wherer≥2r\\geq 2is the target repetition count\. This condition probes whether redundancy across arbitrary locations improves retention\.

Based on the SC taxonomy \(in Section[3](https://arxiv.org/html/2608.11242#S3)\), we manually crafted 15 SC examples \(3 examples per SC category\)\. Some examples are inspired by existing literatureZhouet al\.\([2023](https://arxiv.org/html/2608.11242#bib.bib14)\); Zhaoet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib8)\)\. The complete examples are listed in Appendix Table[12](https://arxiv.org/html/2608.11242#A6.T12)\.

Besides the SC content itself, how these constraints are delivered also matters\. We extend each example with two binary attributes that control its surface form:

- ∙\\bulletConstraint Strength:Strictprepends “This is an important constraint:” to mark the instruction as a strict requirement;Preferentialomits this prefix\.
- ∙\\bulletExplicitness:Directprepends “For the rest of this session\.” so the scope is named explicitly;Contextualizedleaves the scope implicit\.

### 4\.4Evaluation Metrics

#### Retention\.

Given a contextHs,PtH^\{t\}\_\{s,P\}injected with SCss\(as denoted in Equation[5](https://arxiv.org/html/2608.11242#S4.E5)\), and a compactorCC, we prompt GPT\-5\.4OpenAI \([2026c](https://arxiv.org/html/2608.11242#bib.bib33)\)as llm\-as\-a\-judge to compute

Retain​\(s,Hs,Pt\)∈\{0,1\}\.\\text\{Retain\}\(s,H^\{t\}\_\{s,P\}\)\\in\\\{0,1\\\}\.\(6\)Where 1 indicatesssis semantically presented inHs,PtH^\{t\}\_\{s,P\}\. The judge prompt is in Appendix Section[C\.1](https://arxiv.org/html/2608.11242#A3.SS1)\. We also show that GPT\-5\.4’s judgment of retention is consistent with another LLM and human annotator \(in Appendix[C\.2](https://arxiv.org/html/2608.11242#A3.SS2)\)\.

#### Compliance\.

Beyond evaluating the semantic presence of SC in the output context, we further assess its behavioral compliance\. For each SC, we manually construct a probing question\-answer pair\(xprob,yprob\)\(x\_\{\\text\{prob\}\},y\_\{\\text\{prob\}\}\)formatted as a multiple\-choice question \(MCQ\), following established protocolsBaiet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib35)\); Zhanget al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib34)\); Zhaoet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib8)\)\. Given a context historyHtH^\{t\}, we append the probing queryxprobx\_\{\\text\{prob\}\}as the next user turn and elicit a response fromL​L​MprobLLM\_\{\\text\{prob\}\}:

y′=L​L​Mprob​\(xprob,Ht\),y′∈\{A,B\}y^\{\\prime\}=LLM\_\{\\text\{prob\}\}\(x\_\{\\text\{prob\}\},H^\{t\}\),\\quad y^\{\\prime\}\\in\\\{\\text\{A\},\\text\{B\}\\\}\(7\)A response is considered compliant ify′=yproby^\{\\prime\}=y\_\{\\text\{prob\}\}\. We instantiate four experimental conditions that vary only in the contextKKsupplied toLLMprob\\text\{LLM\}\_\{\\text\{prob\}\}:

- ∙\\bulletLong ctx w/ SC\(Klctx sc=Hs,PtK\_\{\\text\{lctx sc\}\}=H^\{t\}\_\{s,P\}\):LLMprob\\text\{LLM\}\_\{\\text\{prob\}\}receives the full, uncompacted history with the SC injected at locationsPP\.
- ∙\\bulletLong ctx w/o SC\(Klctx=HtK\_\{\\text\{lctx\}\}=H^\{t\}\):LLMprob\\text\{LLM\}\_\{\\text\{prob\}\}receives the full history without any SC, capturing its prior tendency toward the compliant option in the absence of the constraint\.
- ∙\\bulletCompaction\(Kcomp=C​\(Hs,Pt\)K\_\{\\text\{comp\}\}=C\(H^\{t\}\_\{s,P\}\)\): the SC is injected before compaction, reflecting the real\-world setting in which a user\-issued constraint must survive a compaction event\.
- ∙\\bulletUpper\-bound\(Kub=C​\(Ht\)⊕sK\_\{\\text\{ub\}\}=C\(H^\{t\}\)\\oplus s\): the SC is appended to the compacted context, and located immediately before the probing query, establishing the maximum compliance attainable under the compacted\-context setup\.

For each conditiong∈\{lctx sc,lctx,comp,ub\}g\\in\\\{\\text\{lctx sc\},\\text\{lctx\},\\text\{comp\},\\text\{ub\}\\\}, we report thecompliance rateover the evaluation set𝒟\\mathcal\{D\}of sizeNN:

c¯g=1N​∑i=1N𝟙​\[L​L​Mprob​\(xprobi,Kgi\)=yprobi\]\.\\bar\{c\}\_\{g\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\!\\left\[LLM\_\{\\text\{prob\}\}\(x^\{i\}\_\{\\text\{prob\}\},K^\{i\}\_\{g\}\)=y^\{i\}\_\{\\text\{prob\}\}\\right\]\.\(8\)

#### Effect Retention\.

To quantify the effective margin in compliance for a compactor, we baseline\-correct the compliance of compaction groupc¯comp\\bar\{c\}\_\{\\text\{comp\}\}by the no\-SC compliancec¯lctx\\bar\{c\}\_\{\\text\{lctx\}\}and normalize against the upper\-boundc¯ub\\bar\{c\}\_\{\\text\{ub\}\}

Effect Retention=c¯comp−c¯lctxc¯ub−c¯lctx\.\\text\{Effect Retention\}=\\frac\{\\bar\{c\}\_\{\\text\{comp\}\}\-\\bar\{c\}\_\{\\text\{lctx\}\}\}\{\\bar\{c\}\_\{\\text\{ub\}\}\-\\bar\{c\}\_\{\\text\{lctx\}\}\}\.\(9\)By construction,ER=1\\text\{ER\}=1corresponds to lossless behavioral preservation,ER=0\\text\{ER\}=0to complete loss\.

## 5Evaluation

Table 2:SC retention and compliance across datasets and compactors \(N=750\)\.Per\-dataset baseline compliance rates appear in theshaded section headers\.Retention Rate: fraction of injected SCs preserved through compaction\.Compaction Compliance: compliance when probing on the compacted context \(Eq\.[8](https://arxiv.org/html/2608.11242#S4.E8),KcompK\_\{\\text\{comp\}\}\)\.Upper\-bound Compliance: compliance when the SC is appended verbatim post\-compaction \(KubK\_\{\\text\{ub\}\}\)\.Effect Retention: compaction effect normalized by the upper\-bound gain over the SC\-free baseline \(Eq\.[9](https://arxiv.org/html/2608.11242#S4.E9)\)\. GPT\-5\.4\-mini is evaluated on a longer context to match its larger context window, and on fewer trials due to experiment cost; details in Appendix[G](https://arxiv.org/html/2608.11242#A7)\.#### Default evaluation setting\.

We fix the SC framing to thedirectandpreferentialformulation across all main experiments\. This isolates the effect of compactor choice from framing effects, which we study separately as an independent factor in Section[5\.3](https://arxiv.org/html/2608.11242#S5.SS3)\(constraint strength×\\timesexplicitness\)\. We select 6 representative compactors that collectively span a range of methodological paradigms: \(i\) Recent\-5 truncation, which retains only the five most recent turns; \(ii\) LLMLingua\-2Panet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib39)\)configured with 500 tokens as budget \(more discussion in Appendix[B](https://arxiv.org/html/2608.11242#A2)\); together with three LLM\-based compaction configurations, namely \(iii\) gpt\-oss\-120bOpenAI \([2025](https://arxiv.org/html/2608.11242#bib.bib41)\)paired with the Anthropic compaction promptAnthropic \([2026](https://arxiv.org/html/2608.11242#bib.bib29)\), \(iv\) gpt\-oss\-120b paired with the pi\-mono compaction prompt222[Pi\-mono prompt\.](https://github.com/earendil-works/pi/blob/f129ac93c508c2cbe45e8342bbf59ce4ba04acdc/packages/coding-agent/src/core/compaction/compaction.ts#L444), which is employed by the widely adopted LLM agent system OpenClawSteinberger and OpenClaw Contributors \([2025](https://arxiv.org/html/2608.11242#bib.bib15)\), \(v\) Qwen3\-30B\-A3BYang and others \([2025](https://arxiv.org/html/2608.11242#bib.bib42)\)paired with the Anthropic compaction prompt, and \(vi\) Gemma 4\-E4BGemma Team, Google DeepMind \([2026](https://arxiv.org/html/2608.11242#bib.bib45)\)paired with the Anthropic compaction prompt\. These LLMs are deliberately selected to represent models of varying parameter scales, with each supporting a context length of approximately 140K tokens\. By default, the SC is injected exactly once, at the top of the filler context\. For each dataset, we construct 50 filler contexts and evaluate them against 15 manually crafted SC examples, yielding a total of N=750 evaluation instances \(50 contexts×\\times15 SC examples\)\. The complete experimental setup is described in Appendix Section[B](https://arxiv.org/html/2608.11242#A2)\.

### 5\.1RQ1: SCs Have Low Retention Rates under Compaction

To address the question of whether compactors preserve SC’s integrity, we evaluate with all the metrics introduced in Section[4\.4](https://arxiv.org/html/2608.11242#S4.SS4)\. Table[5](https://arxiv.org/html/2608.11242#S5)shows the complete evaluation\.

#### Retention Rate\.

The retention rates reported throughout this table are uniformly low\. \(1\) The non\-LLM\-based compactors \(Recent 5 and LLMLingua2\) exhibit a 0% retention rate across all three datasets\. \(2\) The performance of open\-source LLM\-based compactors varies according to both the prompt and the dataset\. On datasets with fewer turns and tool calls \(Hermes Agents and WildChat\), compactors with the pi\-mono prompt retain SCs more effectively than those with the Anthropic prompt, yielding an average margin of 14\.5%\. Conversely, on WildChat the Anthropic prompt retains SCs better than pi\-mono, though the margin is smaller \(1\.3%\), reflecting the fact that WildChat’s many\-turn nature renders the task substantially harder overall\. \(3\) Commercial LLM\-based compactors attain the highest retention rates but nevertheless fail in many cases\. GPT\-5\.4\-mini reaches a retention rate as high as 98%, yet drops to as low as 6\.7%\. Patterns persist across closed and open source: WildChat remains the most challenging case, and prompt advantages are dataset\-dependent\. Low retention also persists when the OpenResearcher trajectory generator and compactor are both gpt\-oss\-120b \(8\.4%; Appendix Section[B\.4](https://arxiv.org/html/2608.11242#A2.SS4)\), showing that source–compactor mismatch alone cannot explain the observed loss\. The GPT\-5\.4\-mini caveat is discussed in Appendix Section[G](https://arxiv.org/html/2608.11242#A7)\. Moreover, an SC\-targeted prompt improves retention and compliance, but retention remains below 40% on WildChat, compared with 90\.3% for our extractor introduced in later sections \(Appendix Section[B\.3](https://arxiv.org/html/2608.11242#A2.SS3)\)\.

#### Compliance\.

SC retention is a prerequisite for SC\-induced compliance, so we also evaluate compliance across the different groups\. Table[5](https://arxiv.org/html/2608.11242#S5)reports the compliance rates measured on the four groups introduced in Section[4\.4](https://arxiv.org/html/2608.11242#S4.SS4)\. Weaker compactors are often less effective than their long\-context baseline \(the long ctx w/ SC group\), with only GPT\-5\.4\-mini occasionally surpassing it, suggesting that most compactors degrade SC compliance relative to no compaction\. Effective retention further indicates that all evaluated compactors, with the exception of GPT\-5\.4\-mini, fall well short of recovering the level of SC following that the upper bound is able to achieve\. Robustness checks using a tool\-using free\-generation harness and three downstream models show the same pattern \(Appendix[D](https://arxiv.org/html/2608.11242#A4)\), indicating that the result generalizes beyond the MCQ format and the specific downstream model\.

### 5\.2RQ2: Compaction System Side Factors

#### Compaction Rate\.

We see that compaction rate for 100K long context is on average182×182\\timesshorter\. We also observe a notable pattern in the growth of output length\. Specifically, the compactor output grows by only0\.84×0\.84\\timesto1\.28×1\.28\\timeswhile the input expands by10×10\\times\(from 10K to 100K tokens\), rendering the output length nearly invariant to the input length\. Complete compaction rates are in Appendix Table[11](https://arxiv.org/html/2608.11242#A6.T11)\.

#### Context Length\.

![Refer to caption](https://arxiv.org/html/2608.11242v1/x3.png)Figure 4:SC retention rate by input context length\.The hue denotes the compactor, and line style denotes the dataset as the filler\.Building on our earlier observation regarding the largely independent relationship between the inputs and outputs of LLM\-based compactors, we further examine how retention rate varies with input context length\. As shown in Figure[4](https://arxiv.org/html/2608.11242#S5.F4), retention rates reach approximately 90% at a context length of 10K on the Hermes Agent dataset, and decline as the context length increases\. Within each dataset, the relative ordering of compactors remains stable across input lengths: better\-performing compactors retain their advantage regardless of input context length, and the ranking with respect to downstream task performance is likewise preserved as context grows\. This finding suggests a straightforward strategy for improving SC retention, namely performing compaction at shorter contexts rather than deferring until the context approaches its limit\. However, such an approach trades off against the fundamental purpose of compaction, which is to extend the effective context budget\.

### 5\.3RQ3: SC\-Side Factors

#### Injection Location\.

![Refer to caption](https://arxiv.org/html/2608.11242v1/x4.png)Figure 5:SC retention rate by injection location\.Each row corresponds to a dataset–compactor combination\. WC: WildChat; HA: Hermes Agents; R5: Recent\-5; Lingua: LLMLingua\-2; GPT\-A: gpt\-oss\-120b with Anthropic prompt; GPT\-P: gpt\-oss\-120b with pi\-mono prompt; Qwen\-A: Qwen30B\-A3B with Anthropic prompt\.The location at which the SC is injected also influences retention\. As shown in Figure[5](https://arxiv.org/html/2608.11242#S5.F5), most compactors retain the SC better when it is injected toward the bottom of the filler context, with Recent 5 on WildChat performing best because the injection falls within its final\-five\-turn window\. This pattern does not hold for Hermes\-Agent, whose substantially lower retention we attribute to the numerous tool\-calling and interleaved\-thinking turns typical of agentic trajectories\. LLMLingua2, by contrast, achieves near\-zero retention regardless of injection location, as its BERT\-style sliding\-window processing is largely location\-invariant\.

For LLM\-based compactors, we observe that later injection locations yield higher retention rates\. We attribute this effect to the placement of the compaction instruction, which immediately follows the context to be compacted,HtH^\{t\}; consequently, later injection locations put the SC in closer proximity to the compaction prompt\. We further evaluate this behavior at a 50K context length, which yields improved retention rates at the top and middle injection positions but no corresponding improvement at the bottom position relative to the 100K setting\. This finding supports our hypothesis that the proximity between the compaction instruction and the SC statement is responsible for the elevated retention rate\. The corresponding table is provided in Appendix Section[H\.1](https://arxiv.org/html/2608.11242#A8.SS1)\. Repeating the SC across multiple turns also improves retention, but it has limited impact as retention rate and effect retention converges after 30 turns \(Appendix Section[E](https://arxiv.org/html/2608.11242#A5)\)\.

This monotonic retention pattern diverges from prior retrieval\-focused findingHsiehet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib46)\); Liuet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib36)\), which report a U\-shaped pattern\. Our pattern is more consistent withZhaoet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib8)\), which find that LLMs follow instructions more reliably when the query is closer to relevant statements\. The detailed discussion is in Appendix Section[E](https://arxiv.org/html/2608.11242#A5)\.

#### SC Framing\.

Table 3:SC retention rate on Hermes Agent across compactors and framing combinations\.The highest cell per row is inbold\.Another axis ofCompIntinvestigate the framing of the SC on retention\. In practice, SCs are delivered by a user prompt with accompanying framing, and from the user’s perspective, they can be expressed through a variety of prompt engineering techniques\. Table[3](https://arxiv.org/html/2608.11242#S5.T3)reports how different framings affect the retention rate\. For non\-LLM compactors, framing exhibits no obvious effect\. For LLM\-based compactors, by contrast, framing the SC as eitherexplicitorstrictyields measurable improvements in retention, withstrictalone producing a larger gain thanexplicitalone\. Combining both framings provides only a marginal additional benefit, raising the retention rate by an average of 1\.3% relative to using a single framing\. These results suggest that while users should phrase SC explicitly and emphasize the constraint, such framing strategies have limited capacity to enforce retention after context compaction\.

#### SC Type\.

![Refer to caption](https://arxiv.org/html/2608.11242v1/x5.png)Figure 6:SC retention rate by type, averaged over 3 datasets\.Process SCs are retained least across all compactors, while Preference SCs tend to be best\-retained for LLM\-based compactors\. No compactor exceeds 36% average retention on any SC type\.Our SC taxonomy defined 5 types of SCs\. Figure[6](https://arxiv.org/html/2608.11242#S5.F6)shows that retention varies across types and compactors\. Preference SCs tend to be among the best\-retained categories for LLM\-based compactors, reaching 36% for gpt\-oss \(pi\-mono\), though they are not uniformly the highest; for Gemma\-4, Output SCs are retained slightly more often\. Process SCs are consistently retained the least across all compactors\. Crucially, no SC type approaches reliable retention under any compactor we tested: even the best result \(Preference under gpt\-oss \(pi\-mono\), at 36%\) loses nearly two\-thirds of constraints\. This indicates that the retention gap is not isolated to one type of instruction but is a general property of current compaction methods\. Results in Figure[6](https://arxiv.org/html/2608.11242#S5.F6)are averaged over 3 datasets; per\-dataset heatmaps in Appendix[I](https://arxiv.org/html/2608.11242#A9)show the pattern is largely consistent across datasets\.

### 5\.4RQ4: An SC\-Aware Extractor

Table 4:SC extractor performance with Qwen3\.5\-9B across datasets\. Retention is the fraction of injected SCs recovered by the extractor; Latency is the average wall\-clock time per context\.To mitigate the loss of SCs during context compaction, we propose an online constraint extractor that can be readily integrated into LLM systems\. We decompose a context into individual turns and only process user turns, employing a Small Language Model \(SLM\), Qwen3\.5\-9BQwen Team \([2026](https://arxiv.org/html/2608.11242#bib.bib43)\), as the extractorEEthat maintains a growing list of SCsStS^\{t\}\. For each user prompt,EEis prompted to determine whether any SC is specified, and if so, the identified SC is appended toStS^\{t\}\. This decomposes the long context into fine\-grained subtasks well suited to an SLMBelcaket al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib47)\)\. At deployment,EEoperates alongside a general LLM\-based compactorCC: when replacing the original contextHtH^\{t\}in Equation[3](https://arxiv.org/html/2608.11242#S4.E3), we substitute

H~t=C​\(Ht\)⊕St,\\tilde\{H\}^\{t\}=C\(H^\{t\}\)\\oplus S^\{t\},\(10\)whereStS^\{t\}is the running list of SCs extracted fromxU0,…,xUtx^\{0\}\_\{U\},\\ldots,x^\{t\}\_\{U\}\. Retained SCs are thus preserved in a form analogous to the upper\-bound group in Section[4\.4](https://arxiv.org/html/2608.11242#S4.SS4)\. We evaluate retention onStS^\{t\}using the same LLM\-as\-a\-judge setup as in our main evaluation\. As reported in Table[4](https://arxiv.org/html/2608.11242#S5.T4), the extractor achieves high retention across all three datasets, surpassing all open\-source compactors and outperforming GPT\-5\.4\-mini by 62% on WildChat\. Because it processes only user prompts, it is highly efficient on think\-heavy and tool\-call\-heavy contexts, requiring only 0\.03 seconds per 100K context on OpenResearcher \(single user query\) and a maximum of 12\.93 seconds on WildChat \(average 257 user turns per 100K context\)\. The extractor is training\-free; its complete prompt, per\-type results, and deployment workflow are provided in Appendix Section[J](https://arxiv.org/html/2608.11242#A10)\.

## 6Conclusion

In this work, we identify Side Constraints \(SCs\), a class of user\-posed constraints that are frequently lost during the context compaction triggered by long contexts\. We introduce a taxonomy of SCs along with an evaluation suite,CompInt\. From the system perspective, although LLM\-based compactors outperform their non\-LLM\-based counterparts, their ability to retain SCs remains limited, with retention higher on shorter contexts and contexts exhibiting less topic drift\. From the user perspective, repeating an SC, positioning it toward the end of the context, and phrasing it in an explicit and strict manner each increase the likelihood of its retention\. Building on these insights, we propose an SLM\-based SC extractor that reads user prompts and maintains a running list of SCs, substantially improving SC retention and providing a strategy\-layer complement to the context compactor\.

## Limitations

#### Proprietary Model Context Pressure\.

In our experiments, we evaluated GPT\-5\.4\-mini as a representative example of a proprietary model\. We were unable to test at the long\-context setting of 320K tokens, which corresponds to approximately 80% of its disclosed maximum context window \(400K\), as OpenAI’s API consistently returned errors at this length; further discussion is provided in Appendix Section[G](https://arxiv.org/html/2608.11242#A7)\. Extending the long\-context experiment to models supporting even larger context windows is cost\-prohibitive, given that newer flagship models \(e\.g\., GPT\-5\.5 and Claude Opus 4\.6\) offer context windows of up to 1,000K tokens\. As a reference point, our current 220K\-context experiment with GPT\-5\.4\-mini incurred a cost of approximately 800 USD\.

#### Compliance Rate\.

Throughout our experiments, we employed only a single model for probing compliance \(theL​L​MprobLLM\_\{\\text\{prob\}\}in Equation[8](https://arxiv.org/html/2608.11242#S4.E8)\)\. Evaluating compliance across additional probing models in future work could help mitigate potential model\-specific bias in our Effect Retention estimates\.

## Ethical Considerations

The failure mode we study has direct safety implications: when compaction drops session constraints, agents may take actions the user explicitly forbidden, including unauthorized tool calls, disclosure of withheld information, or bypassing user\-required verification steps\. Our motivating example illustrates exactly this risk\.CompIntis designed to surface these failures so that compaction design can be evaluated and improved, and our SC\-aware extractor offers a mitigation\. The benchmark uses synthetic constraints over public datasets and does not introduce attack surfaces beyond those already present in deployed systems\. We releaseCompIntto support measurement and mitigation of constraint loss, not to facilitate adversarial exploitation\.

## References

- Compaction: server\-side context management for long conversations\.Note:[https://platform\.claude\.com/docs/en/build\-with\-claude/compaction](https://platform.claude.com/docs/en/build-with-claude/compaction)Claude API Documentation\. Beta header:compact\-2026\-01\-12\. Accessed: 2026\-05\-11Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1),[§1](https://arxiv.org/html/2608.11242#S1.p2.1),[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.11242#S4.SS2.p2.1),[§5](https://arxiv.org/html/2608.11242#S5.SS0.SSS0.Px1.p1.2)\.
- Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2025\)LongBench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3639–3664\.External Links:[Link](https://aclanthology.org/2025.acl-long.183/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.183),ISBN 979\-8\-89176\-251\-0Cited by:[§4\.4](https://arxiv.org/html/2608.11242#S4.SS4.SSS0.Px2.p1.4)\.
- P\. Belcak, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. C\. Lin, and P\. Molchanov \(2025\)Small language models are the future of agentic ai\.arXiv preprint arXiv:2506\.02153\.Cited by:[§5\.4](https://arxiv.org/html/2608.11242#S5.SS4.p1.7)\.
- Gemma Team, Google DeepMind \(2026\)Gemma 4\.Note:[https://huggingface\.co/google/gemma\-4\-31B](https://huggingface.co/google/gemma-4-31B)Open\-weights release, April 2, 2026\. Apache 2\.0 license\.Cited by:[§5](https://arxiv.org/html/2608.11242#S5.SS0.SSS0.Px1.p1.2)\.
- Google DeepMind \(2026\)Gemini 3\.5 Flash: model card\.Note:Accessed: 2026\-07\-28External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-5-flash/)Cited by:[§C\.2](https://arxiv.org/html/2608.11242#A3.SS2.p1.1)\.
- N\. Hemken, S\. Koneru, F\. Jacob, H\. Hartenstein, and J\. Niehues \(2025\)Can a large language model keep my secrets? a study on LLM\-controlled agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),J\. Zhao, M\. Wang, and Z\. Liu \(Eds\.\),Vienna, Austria,pp\. 746–759\.External Links:[Link](https://aclanthology.org/2025.acl-srw.49/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-srw.49)Cited by:[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.8.6.5.1.1)\.
- C\. Hsieh, Y\. Chuang, C\. Li, Z\. Wang, L\. Le, A\. Kumar, J\. Glass, A\. Ratner, C\. Lee, R\. Krishna,et al\.\(2024\)Found in the middle: calibrating positional attention bias improves long context utilization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 14982–14995\.Cited by:[§H\.2](https://arxiv.org/html/2608.11242#A8.SS2.p1.1),[§5\.3](https://arxiv.org/html/2608.11242#S5.SS3.SSS0.Px1.p3.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)Swe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[§B\.2](https://arxiv.org/html/2608.11242#A2.SS2.SSS0.Px1.p1.1)\.
- Y\. Li, B\. Dong, F\. Guerin, and C\. Lin \(2023\)Compressing context to enhance inference efficiency of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 6342–6353\.Cited by:[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Li, D\. Jiang, X\. Ma, H\. Zhang, P\. Nie, Y\. Zhang, K\. Zou, J\. Xie, Y\. Zhang, and W\. Chen \(2026\)Openresearcher: a fully open pipeline for long\-horizon deep research trajectory synthesis\.arXiv preprint arXiv:2603\.20278\.Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1),[3rd item](https://arxiv.org/html/2608.11242#S4.I1.i3.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the association for computational linguistics12,pp\. 157–173\.Cited by:[§H\.2](https://arxiv.org/html/2608.11242#A8.SS2.p1.1),[§5\.3](https://arxiv.org/html/2608.11242#S5.SS3.SSS0.Px1.p3.1)\.
- Z\. Mueller \(2026\)Creating highly efficient agents: 450M tool\-calling tokens distilled for post\-training from top open\-source models\.Note:[Lambda AI Blog](https://lambda.ai/blog/creating-highly-efficient-agents-450m-tool-calling-tokens-distilled-for-post-training-from-top-open-source-models)Accessed: 2026\-05\-04Cited by:[2nd item](https://arxiv.org/html/2608.11242#S4.I1.i2.p1.1)\.
- Nous Research \(2026\)Hermes agent: the agent that grows with you\.Note:[https://github\.com/NousResearch/hermes\-agent](https://github.com/NousResearch/hermes-agent)Version 0\.13\.0 \(2026\.5\.7\)\. MIT License\. Accessed: 2026\-05\-11Cited by:[2nd item](https://arxiv.org/html/2608.11242#S4.I1.i2.p1.1)\.
- OpenAI \(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§5](https://arxiv.org/html/2608.11242#S5.SS0.SSS0.Px1.p1.2)\.
- OpenAI \(2026a\)GPT\-5\.3\-Codex system card\.Technical reportOpenAI\.Note:OpenAI Deployment Safety Hub\. Accessed: 2026\-05\-11External Links:[Link](https://deploymentsafety.openai.com/gpt-5-3-codex/gpt-5-3-codex.pdf)Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1),[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2026b\)Introducing GPT\-5\.4 mini and nano\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4\-mini\-and\-nano/](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)Accessed: 2026\-05\-11Cited by:[Appendix G](https://arxiv.org/html/2608.11242#A7.p1.1)\.
- OpenAI \(2026c\)Introducing GPT\-5\.4\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4/](https://openai.com/index/introducing-gpt-5-4/)Accessed: 2026\-05\-11Cited by:[§C\.1](https://arxiv.org/html/2608.11242#A3.SS1.p1.1),[§4\.4](https://arxiv.org/html/2608.11242#S4.SS4.SSS0.Px1.p1.3)\.
- Z\. Pan, Q\. Wu, H\. Jiang, M\. Xia, X\. Luo, J\. Zhang, Q\. Lin, V\. Rühle, Y\. Yang, C\. Lin, H\. V\. Zhao, L\. Qiu, and D\. Zhang \(2024\)LLMLingua\-2: data distillation for efficient and faithful task\-agnostic prompt compression\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 963–981\.External Links:[Link](https://aclanthology.org/2024.findings-acl.57/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.57)Cited by:[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.11242#S5.SS0.SSS0.Px1.p1.2)\.
- Y\. Qin, K\. Song, Y\. Hu, W\. Yao, S\. Cho, X\. Wang, X\. Wu, F\. Liu, P\. Liu, and D\. Yu \(2024\)Infobench: evaluating instruction following ability in large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 13025–13048\.Cited by:[§C\.1](https://arxiv.org/html/2608.11242#A3.SS1.p1.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)Alibaba Cloud\. Accessed: 2026\-05\-23Cited by:[§5\.4](https://arxiv.org/html/2608.11242#S5.SS4.p1.7)\.
- P\. K\. Robinette, A\. Hard, S\. Ramaswamy, E\. Amid, R\. Mathews, and T\. T\. Johnson \(2026\)We are what we repeatedly do: improving long context instruction following\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 4855–4884\.Cited by:[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Shao, T\. Li, W\. Shi, Y\. Liu, and D\. Yang \(2024\)Privacylens: evaluating privacy norm awareness of language models in action\.Advances in Neural Information Processing Systems37,pp\. 89373–89407\.Cited by:[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.6.4.5.1.1),[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.7.5.5.1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1),[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Steinberger and OpenClaw Contributors \(2025\)OpenClaw: your own personal ai assistant\.Note:[https://github\.com/openclaw/openclaw](https://github.com/openclaw/openclaw)Open\-source personal AI assistant framework supporting multiple messaging platforms \(WhatsApp, Telegram, Slack, Discord, Signal, iMessage, etc\.\)\. 196k stars, MIT license\. Built for Molty, a space lobster AI assistant\.Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1),[§4\.2](https://arxiv.org/html/2608.11242#S4.SS2.p2.1),[§5](https://arxiv.org/html/2608.11242#S5.SS0.SSS0.Px1.p1.2)\.
- O\. Styles, S\. Miller, P\. Cerda\-Mardini, T\. Guha, V\. Sanchez, and B\. Vidgen \(2024\)WorkBench: a benchmark dataset for agents in a realistic workplace setting\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=4HNAwZFDcH)Cited by:[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.3.1.5.1.1)\.
- X\. Wu, K\. Li, Y\. Zhao, L\. Zhang, L\. Ou, H\. Yin, Z\. Zhang, X\. Yu, D\. Zhang, Y\. Jiang, P\. Xie, F\. Huang, M\. Cheng, S\. Wang, H\. Cheng, and J\. Zhou \(2026\)ReSum: unlocking long\-horizon search intelligence via context summarization\.External Links:2509\.13313,[Link](https://arxiv.org/abs/2509.13313)Cited by:[§1](https://arxiv.org/html/2608.11242#S1.p1.1),[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.11242#S4.SS2.p2.1)\.
- W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang \(2026\)A\-mem: agentic memory for LLM agents\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=FiM0M8gcct)Cited by:[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Yanget al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5](https://arxiv.org/html/2608.11242#S5.SS0.SSS0.Px1.p1.2)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)Swe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhang, L\. Lei, L\. Wu, R\. Sun, Y\. Huang, C\. Long, X\. Liu, X\. Lei, J\. Tang, and M\. Huang \(2024\)SafetyBench: evaluating the safety of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15537–15553\.External Links:[Link](https://aclanthology.org/2024.acl-long.830/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.830)Cited by:[§4\.4](https://arxiv.org/html/2608.11242#S4.SS4.SSS0.Px2.p1.4)\.
- S\. Zhao, M\. Hong, Y\. Liu, D\. Hazarika, and K\. Lin \(2025\)Do LLMs recognize your preferences? evaluating personalized preference following in LLMs\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QWunLKbBGF)Cited by:[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.12.10.5.1.1),[§H\.2](https://arxiv.org/html/2608.11242#A8.SS2.p1.1),[§2](https://arxiv.org/html/2608.11242#S2.SS0.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2608.11242#S4.I1.i1.p1.1),[§4\.3](https://arxiv.org/html/2608.11242#S4.SS3.p2.1),[§4\.4](https://arxiv.org/html/2608.11242#S4.SS4.SSS0.Px2.p1.4),[§5\.3](https://arxiv.org/html/2608.11242#S5.SS3.SSS0.Px1.p3.1)\.
- W\. Zhao, X\. Ren, J\. Hessel, C\. Cardie, Y\. Choi, and Y\. Deng \(2024\)Wildchat: 1m chatgpt interaction logs in the wild\.arXiv preprint arXiv:2405\.01470\.Cited by:[1st item](https://arxiv.org/html/2608.11242#S4.I1.i1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§C\.1](https://arxiv.org/html/2608.11242#A3.SS1.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.10.8.5.1.1),[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.11.9.5.1.1),[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.15.13.5.1.1),[Table 12](https://arxiv.org/html/2608.11242#A6.T12.1.16.14.5.1.1),[§4\.3](https://arxiv.org/html/2608.11242#S4.SS3.p2.1)\.

## Appendix ADataset Details

### A\.1Topic\-cohesive Stitching

The WildChat dataset collects anonymous user conversations spanning a wide range of topics, so naively concatenating adjacent rows yields synthetic long\-context inputs in which successive segments are topically unrelated\. To produce more coherent synthetic conversations, we apply the topic\-cohesive stitching procedure described in Algorithm[1](https://arxiv.org/html/2608.11242#alg1)\. The procedure first computes an L2\-normalized embeddingzi=E​\(xi\)/‖E​\(xi\)‖z\_\{i\}=E\(x\_\{i\}\)/\\\|E\(x\_\{i\}\)\\\|for every conversationxi∈𝒟x\_\{i\}\\in\\mathcal\{D\}using an embedding modelEE, then builds a global k\-NN indexℐ\\mathcal\{I\}over\{zi\}i=1n\\\{z\_\{i\}\\\}\_\{i=1\}^\{n\}that is queried during stitching to retrieve topically similar candidates\. Each synthetic conversation is grown from a seed sample by iteratively appending the nearest unused neighbor of the running centroid, where the centroid is the L2\-normalized arithmetic mean of the embeddings already in the stitched conversation\. We usek=32k=32andE=Qwen/Qwen3\-Embedding\-0\.6BE=\\texttt\{Qwen/Qwen3\-Embedding\-0\.6B\}333[https://huggingface\.co/Qwen/Qwen3\-Embedding\-0\.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)for both the WildChat and Hermes Agent datasets\.

#### Neighbor lookup\.

The neighbor query in line 12 of Algorithm[1](https://arxiv.org/html/2608.11242#alg1)returns the top\-ranked sample inℛ∖𝒰\\mathcal\{R\}\\setminus\\mathcal\{U\}under cosine similarity to the centroidc​\(H~\)c\(\\tilde\{H\}\)\. Because our k\-NN backend returns a fixed\-size top\-wwlist rather than a streaming iterator, we issue an initial query withw=k\+\|H~\|w=k\+\|\\tilde\{H\}\|neighbors, scan the returned list in ranked order for the first sample that lies inℛ∖𝒰\\mathcal\{R\}\\setminus\\mathcal\{U\}, and doublewwand re\-query if every returned neighbor has already been used or removed from the pool\. The window is capped at\|ℐ\|\|\\mathcal\{I\}\|, and a query atw=\|ℐ\|w=\|\\mathcal\{I\}\|that finds no valid candidate raises a failure \(which our experiments never triggered\)\.

#### Length control\.

Algorithm[1](https://arxiv.org/html/2608.11242#alg1)terminates the inner loop once the stitched conversation reaches the target token lengthltl\_\{t\}, but the final appended samplexxcan push the total pastltl\_\{t\}by an arbitrary amount\. This could lead the final concatenated sample having a length greater than the compactor LLM’s max context window\. To bound this overshoot, our implementation enforces a soft upper bound of1\.25​lt1\.25\\,l\_\{t\}on the stitched length\. If the inner loop terminates withℓ​\(H~\)\>1\.25​lt\\ell\(\\tilde\{H\}\)\>1\.25\\,l\_\{t\}, we crop the last sample such that it falls under the1\.25​lt1\.25\\,l\_\{t\}upperbound\.

#### Seed selection\.

For reproducibility, the seed at each outer iteration is chosen deterministically as the lowest\-indexed remaining sample inℛ\\mathcal\{R\}under the original dataset ordering, rather than at random\.

#### Hermes Agent system prompts\.

The Hermes Agent dataset prepends a tool\-defining system message to every conversation\. When stitching Hermes agent rows, we keep the system prompt of the seed conversation and strip the leading system prompt from every subsequently appended conversation, so the stitched conversation contains exactly one system message at position zero\. This step doesn’t apply to the WildChat dataset, as its conversations carry no such prefix\.

Algorithm 1Topic\-cohesive synthetic long\-context construction0:Dataset

𝒟=\{x1,…,xn\}\\mathcal\{D\}=\\\{x\_\{1\},\\dots,x\_\{n\}\\\}, embedding model

EE, target token length

ltl\_\{t\}, number of synthetic conversations

NN, k\-NN parameter

kk
0:Synthetic long\-context dataset

𝒟~\\tilde\{\\mathcal\{D\}\}
1:Compute normalized embeddings

𝒵=\{\(xi,zi\)∣zi=E​\(xi\)/‖E​\(xi\)‖,xi∈𝒟\}\\mathcal\{Z\}=\\\{\(x\_\{i\},z\_\{i\}\)\\mid z\_\{i\}=E\(x\_\{i\}\)/\\\|E\(x\_\{i\}\)\\\|,\\ x\_\{i\}\\in\\mathcal\{D\}\\\}
2:Build a global k\-NN index

ℐ\\mathcal\{I\}over

\{zi\}i=1n\\\{z\_\{i\}\\\}\_\{i=1\}^\{n\}
3:Initialize remaining pool

ℛ←𝒟\\mathcal\{R\}\\leftarrow\\mathcal\{D\}
4:Initialize output dataset

𝒟~←∅\\tilde\{\\mathcal\{D\}\}\\leftarrow\\emptyset
5:while

\|𝒟~\|<N\|\\tilde\{\\mathcal\{D\}\}\|<Nand

ℛ≠∅\\mathcal\{R\}\\neq\\emptysetdo

6:Select a seed sample

xs∈ℛx\_\{s\}\\in\\mathcal\{R\}
7:Initialize stitched conversation

H~←\[xs\]\\tilde\{H\}\\leftarrow\[x\_\{s\}\]
8:Initialize used set

𝒰←\{xs\}\\mathcal\{U\}\\leftarrow\\\{x\_\{s\}\\\}
9:Initialize centroid accumulator

s←zss\\leftarrow z\_\{s\}
10:while

ℓ​\(H~\)<lt\\ell\(\\tilde\{H\}\)<l\_\{t\}and

ℛ∖𝒰≠∅\\mathcal\{R\}\\setminus\\mathcal\{U\}\\neq\\emptysetdo

11:Compute centroid embedding

c​\(H~\)←s/\|H~\|‖s/\|H~\|‖c\(\\tilde\{H\}\)\\leftarrow\\frac\{s/\|\\tilde\{H\}\|\}\{\\\|s/\|\\tilde\{H\}\|\\\|\}
12:Let

x⋆x^\{\\star\}be the highest\-ranked neighbor of

c​\(H~\)c\(\\tilde\{H\}\)under

ℐ\\mathcal\{I\}that lies in

ℛ∖𝒰\\mathcal\{R\}\\setminus\\mathcal\{U\}
13:Append

x⋆x^\{\\star\}to

H~\\tilde\{H\}
14:Update

𝒰←𝒰∪\{x⋆\}\\mathcal\{U\}\\leftarrow\\mathcal\{U\}\\cup\\\{x^\{\\star\}\\\}
15:Update centroid accumulator

s←s\+z⋆s\\leftarrow s\+z^\{\\star\}
16:endwhile

17:Add

H~\\tilde\{H\}to output dataset:

𝒟~←𝒟~∪\{H~\}\\tilde\{\\mathcal\{D\}\}\\leftarrow\\tilde\{\\mathcal\{D\}\}\\cup\\\{\\tilde\{H\}\\\}
18:Remove used samples from pool:

ℛ←ℛ∖𝒰\\mathcal\{R\}\\leftarrow\\mathcal\{R\}\\setminus\\mathcal\{U\}
19:endwhile

20:return

𝒟~\\tilde\{\\mathcal\{D\}\}

### A\.2Dataset Coverage and External Validity

Our datasets are curated from open\-source sources representative of long\-context settings that require context compaction\. WildChat is a real\-world human–AI conversation dataset representing many\-turn conversations; Hermes Agent is a multi\-turn tool\-calling trajectory dataset generated with the Hermes Agent harness, representing agentic tool\-use trajectories; and OpenResearcher is a single\-query, long\-horizon deep\-research trajectory dataset\. These three sources cover qualitatively distinct interaction settings that require compaction\.

WildChat is only one of the three complementary settings\. We include it as a challenging many\-turn conversation setting in which a user may continue to submit different requests within the same chat window over time\. We also evaluate two settings with stronger session coherence: Hermes Agent, which is constructed from substantially fewer stitched agent trajectories, and OpenResearcher, which contains native long\-horizon research trajectories without stitching\. Retention degradation persists across all three datasets, showing that the main finding is not driven solely by discontinuities in WildChat\.

Our comparisons are also controlled within each dataset: the conversation history remains fixed while only compaction and SC availability change\. WildChat’s lower retention may reflect both longer interaction distances and lower session coherence\.

## Appendix BCompactor Setting

### B\.1LLMLingua\-2\-output\-budget

LLMLingua\-2 is a token\-level extractive compressor whose default configuration thresholds a retention probability on each token; consequently, its output length scales with the input length rather than being bounded by an absolute target\. Under the long\-context regime considered byCompInt, this default would produce outputs exceeding 10K tokens, which is incommensurable with the outputs of the LLM\-based compactors evaluated in this work, each of which is prompted to emit a bounded summary irrespective of input length\. Beyond considerations of comparability, an input\-proportional output also conflicts with the practical motivation for compaction in agentic systems, where the explicit objective is to reduce the context occupied by prior turns so that subsequent turns retain sufficient budget to proceed\. We therefore fix LLMLingua\-2’s output budget at 500 tokens, a value consistent with the order of magnitude produced by the LLM\-based compactors under different prompt and different base LLMs in Table[11](https://arxiv.org/html/2608.11242#A6.T11)\.

### B\.2LLM\-based Compactor

#### Serving\.

We serve all the open\-weight model on a server equipped with an AMD EPYC 9334 32\-Core Processor running Ubuntu 22\.04\.5 LTS, with four NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs\. We only use single GPU to perform inference as it is sufficient memory for all local inferences\. Models are served with vLLMKwonet al\.\([2023](https://arxiv.org/html/2608.11242#bib.bib48)\), a fast LLM serving library\. Each compaction experiment run \(750 queries\) takes on average 5 hours with 100K context, gpt\-oss\-120b\. All models are running on their default hyperparameter\.

#### Prompts\.

We adapted two compaction prompts for our experiments: Anthropic’s default compaction prompt, as specified in the official documentation,444[Anthropic compaction prompt](https://platform.claude.com/docs/en/build-with-claude/compaction#custom-summarization-instructions), last accessed May 20, 2026\.and the pi\-mono compaction prompt,555[Pi\-mono compaction prompt](https://github.com/earendil-works/pi/blob/f129ac93c508c2cbe45e8342bbf59ce4ba04acdc/packages/coding-agent/src/core/compaction/compaction.ts#L444)\.which is employed by OpenClaw,666[OpenClaw’s compaction session](https://github.com/openclaw/openclaw/blob/b79effefee925d26ed7aea44afd6dfcdc20a2992/src/agents/pi-embedded-runner/compact.ts#L1103-L1116)\.a publicly available LLM agent framework\.

### B\.3Compactor Prompt Details and SC\-Targeted Ablation

The sources of the compaction prompts used throughout the paper are provided in Appendix Section[B\.2](https://arxiv.org/html/2608.11242#A2.SS2)\. Because prompt details are important for understanding failures to preserve SCs, Table[5](https://arxiv.org/html/2608.11242#A2.T5)compares the structure and constraint\-related content of the Anthropic and pi\-mono prompts with our SC\-targeted Anthropic prompt\.

Table 5:Overview of the compaction prompts evaluated in this work\. The SC\-targeted prompt adds one explicit SC\-preservation instruction to the original Anthropic prompt\.The Anthropic prompt is a general context\-summarization prompt that asks the compactor to preserve information such as the current state, next steps, and learnings\. The pi\-mono prompt requires the summary to follow seven fixed sections, including goals, constraints and preferences, and progress\. It also explicitly instructs the compactor to include “any constraints, preferences, or requirements mentioned by user,” partly targeting SCs\.

To directly test whether an explicit SC\-preservation instruction improves retention, we construct an SC\-targeted prompt by appending the following instruction to the original Anthropic prompt:*“Additionally, preserve every user\-provided session\-level constraint or requirement that should continue to govern future responses\. State each constraint explicitly in the summary, retaining its operative conditions and details\.”*

We conduct a controlled prompt ablation on WildChat with gpt\-oss\-120b and Qwen3\-30B\-A3B\. For each compactor, we hold the contexts, SCs, injection conditions, and all other evaluation settings fixed while varying only whether the Anthropic prompt includes the additional SC\-targeting instruction\.

Table 6:Effect of adding an explicit SC\-preservation instruction to the Anthropic compaction prompt on WildChat\. Parentheses report absolute improvements in percentage points over the corresponding baseline\.The targeted prompt improves both retention and downstream compliance for gpt\-oss\-120b and Qwen3\-30B\-A3B, showing that explicitly targeting SCs is a helpful step toward addressing the retention problem\. Nevertheless, retention remains below 40% for both targeted configurations\. In comparison, our proposed SLM\-based extractor achieves 90\.3% retention on WildChat, 52\.7 percentage points higher than the best targeted\-prompt result\. Thus, SC\-targeted prompting helps but does not close the gap with explicit constraint extraction\.

### B\.4Source–Compactor Compatibility

To assess whether source–compactor mismatch explains the observed retention loss, we examine OpenResearcher trajectories generated by gpt\-oss\-120b\. All compactors use the same Anthropic compaction prompt\. The matched configuration uses gpt\-oss\-120b as both the trajectory generator and the compactor, whereas the mismatched configurations use compactors from other model families\.

Table 7:SC retention under matched and mismatched source–compactor configurations on OpenResearcher\. All compactors use the Anthropic compaction prompt\.†GPT\-5\.4\-mini was evaluated on longer contexts and fewer trials and is therefore not directly comparable to the 100K conditions\.

Retention remains low at 8\.4% even when the trajectory generator and compactor are both gpt\-oss\-120b, showing that source–compactor mismatch alone cannot explain the observed loss\. Under the same 100K\-context conditions, the mismatched Gemma\-4\-E4B and Qwen3\-30B\-A3B compactors achieve 3\.6% and 1\.1%, respectively\. GPT\-5\.4\-mini achieves 26\.0% in the separate 220K\-context evaluation, but this result is based on longer contexts and fewer trials and is therefore not included in a matched\-versus\-mismatched average\. Because matching status changes together with compactor identity, these comparisons do not isolate compatibility from model capability\.

### B\.5Licensing and Fair\-use

The LLM serving platform: vLLM \(Apache 2\.0\) Datasets: WildChat \(AI2 ImpACT License\), Hermes Agent \(MIT\), and OpenResearcher Dataset \(MIT Licensecl\)\. Models: gpt\-oss\-120b \(Apache 2\.0\), Qwen3\-30B and Qwen3\.5\-9B \(Apache 2\.0\), Gemma\-4\-E4B \(Apache 2\.0\), and Qwen3\-Embedding\-0\.6B \(Apache 2\.0\)\. Other compactor: LLMLingua\-2 \(MIT\) Our use of these artifacts is limited to research evaluation and follows the applicable license terms and usage conditions\.CompIntwill be released under MIT License\.

## Appendix CRetention Judge

### C\.1Retention Judge Details

To judge the retention of an SC, we use llm\-as\-a\-judge for retention judge which is also adopted as a standard practice in prior workQinet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib30)\); Zhenget al\.\([2023](https://arxiv.org/html/2608.11242#bib.bib31)\)\. We designed this prompt to query GPT\-5\.4OpenAI \([2026c](https://arxiv.org/html/2608.11242#bib.bib33)\):

SC Retention Judge PromptSystem:You are evaluating whether a single injected Session\-Scoped Side\-Constraint \(SC\) is preserved after context compaction\.Definition:A Session\-Scoped Side\-Constraint \(SC\) is a user\-provided instruction that is intended to constrain the assistant’s behavior throughout the ongoing session, but is not itself the user’s main task objective\.Task:Decide whether the injected SC is still PRESENT in the compacted context as a stated instruction or requirement\.Scope of judgment:∙\\bulletJudge only the presence of the SC in the compacted text\.∙\\bulletDo NOT judge whether the assistant followed or acknowledged the SC\. Assistant non\-compliance inside the compacted context is irrelevant here\.∙\\bulletTreat the compacted context as a static document and ask: can a reader recover the SC as a session\-level instruction from this text?Decision rule:∙\\bulletOutput YES if the compacted context contains the injected SC as a stated instruction, even if paraphrased or shortened\.∙\\bulletOutput NO if the SC is omitted or weakened so much that a downstream assistant could not recover it as a session\-level constraint\.∙\\bulletBe strict on presence: vague thematic similarity is not enough\.∙\\bulletOutput only YES or NO\.User:\[Injected SC\]\{injected\_sc\}\[Compacted Context\]\{compacted\_context\}

### C\.2Retention Judge Correctness

We further validate the correctness of using GPT\-5\.4 as llm\-as\-a\-judge for labeling SC retention\. We compare GPT\-5\.4’s judgment with another independent LLM judge, Gemini\-3\.5\-FlashGoogle DeepMind \([2026](https://arxiv.org/html/2608.11242#bib.bib49)\), and a human annotator to classify retention of summaries samples from Table[5](https://arxiv.org/html/2608.11242#S5)\.

For human annotation, we sampled a balanced set of 50 examples: 25 judged by GPT\-5\.4 as retained and 25 as not retained\. A human annotator then labeled each compacted summary blindly without seeing the judge’s verdict\. For the additional LLM\-as\-ajudge evaluation, we use Gemini\-3\.5\-Flash to judge the same 50 samples used for human annotation, with 1,950 additional randomly sampled examples using the same judge prompt \(as used for GPT\-5\.4\)\. After excluding 15 samples77715 samples were marked asPROHIBITED\_CONTENTand were blocked by Gemini\-3\.5\-Flash’s API platform\. These are samples from WildChat, which includes users’ unsafe queries\., the evaluation contained 1,985 samples\.

Table 8:Agreement and Cohen’sκ\\kappaacross human and llm\-as\-a\-judge comparisons\. GPT refers to GPT\-5\.4, and Gemini refers to Gemini\-3\.5\-Flash\.Table[8](https://arxiv.org/html/2608.11242#A3.T8)shows the agreement among human annotator and 2 judge models\. GPT\-5\.4 judge used throughout the paper achieves 100% agreement with human annotator on 50 samples\. Gemini\-3\.5\-Flash agreed with the human annotator on 48 of the 50 examples, corresponding to 96\.0% agreement and a Cohen’sκ\\kappaof 0\.920\. Both judge’s result shows that proprietary LLMs has high agreement with human’s judgment on SC retention\. We also compares the 2 llm\-as\-a\-judge on a larger sample size, and it yields a Cohen’sκ\\kappaof 0\.926\. These results show that GPT\-5\.4’s judgments are highly consistent with both human annotation and an independent LLM judge\.

## Appendix DRobustness of the Compliance Evaluation

To complement our controlled compliance evaluation, which uses multiple\-choice questions \(MCQs\) with gpt\-oss\-120b as the downstream modelL​L​MprobLLM\_\{\\mathrm\{prob\}\}, we conduct two additional evaluations to validate the generalization beyond MCQs and downstream model effect\. First, a tool\-using agent\-harness evaluation lets the model freely generate a response or tool call from the supplied context, directly testing post\-compaction behavior without fixed answer choices\. Second, a multi\-model evaluation tests whether the finding is specific to gpt\-oss\-120b\. Both evaluations support the same conclusion: the compacted context often does not support compliant downstream behavior, even though the downstream model can follow the SC when it is explicitly provided after compaction\.

### D\.1Tool\-Using Agent\-Harness Evaluation

For this evaluation, we use the subset of SCs whose compliance can be directly determined from generated behavior: IDs 1, 2, 3, 6, 7, and 10 in Table[13](https://arxiv.org/html/2608.11242#A6.T13)\. Under Hermes Agent contexts, we provide the corresponding tool definitions, such asdraft\_emailandget\_user\_profile, through thedeveloperchannel following gpt\-oss’s Harmony format\. Rather than selecting an MCQ answer, the model freely generates a textual response or tool call in thecommentarychannel\.

GPT\-5\.4 evaluates the complete generated output, including both the textual response and any tool call, using three labels:*compliant*,*non\-compliant*, and*not enough information*\(NEI\)\. The reported agent\-harness score excludes NEI samples\. The lower bound treats all NEI samples as non\-compliant, while the upper bound treats them as compliant\. For a controlled comparison, we recompute the MCQ results on the same six\-SC subset\.

Table 9:Compliance under the controlled MCQ evaluation and the tool\-using free\-generation evaluation on the same six\-SC subset\. For free generation, the main score excludes NEI samples\. The reported NEI range treats all NEI samples as non\-compliant for the lower bound and compliant for the upper bound\.The agent\-harness evaluation reproduces the main result without fixed answer choices\. Compliance is31\.8%31\.8\\%when the model receives the compacted context alone, but reaches99\.7%99\.7\\%when the SC is appended after compaction\. This large gap shows that the model can follow the SC when it is available after compaction, whereas the compacted context often does not support compliant behavior\.

Moreover, compliance with the full context and SC is higher in the agent harness than in the MCQ setting \(73\.4%73\.4\\%versus66\.0%66\.0\\%\)\. This result suggests that the MCQ evaluation does not overstate the model’s ability to follow long\-context SCs\.

### D\.2Robustness Across Downstream Models

We repeat the controlled compliance evaluation with gpt\-oss\-120b, Qwen3\-30B\-A3B, and Gemma\-4\-E4B as the downstream modelL​L​MprobLLM\_\{\\mathrm\{prob\}\}\. We use the same Hermes Agent contexts and hold the compaction configuration fixed to gpt\-oss\-120b with the Anthropic compaction prompt, varying only the downstream model\. This controlled setup tests whether the observed low effective retention is specific to how gpt\-oss\-120b interprets the compacted summary\.

Table 10:Controlled compliance evaluation across downstream models on Hermes Agent contexts\. The compactor and compaction prompt are held fixed, and onlyL​L​MprobLLM\_\{\\mathrm\{prob\}\}varies\.Although the absolute compliance rates vary across models, effective retention remains low for all three, ranging from12\.8%12\.8\\%to27\.6%27\.6\\%\. In contrast,KubK\_\{\\mathrm\{ub\}\}is above98%98\\%for every model, indicating that each downstream model can use the SC when it is explicitly provided after compaction\. Therefore, the low effective retention is not specific to how gpt\-oss\-120b interprets the compacted summary\.

## Appendix ERepetition’s effect on SC Retention

![Refer to caption](https://arxiv.org/html/2608.11242v1/x6.png)Figure 7:SC retention rate and effect retention by SC repetition\.The x\-axis shows the number of different locations a SC was injected in the context\. The compactor is gpt\-oss\-120b with Anthropic prompt, and the dataset is WildChat with 100K context\.Repeating a SC notably improves the chance of it being retained in the long context, but the gains are front\-loaded: retention rises rapidly from 1 to 10 repetitions, and both metrics converge by around 30\. This suggests that users can reinforce their constraints by stating them multiple times to increase the chance of retention\.

## Appendix FCompaction Rate

The compaction rate of Hermes Agent dataset and WildChat dataset at various context length is in Table[11](https://arxiv.org/html/2608.11242#A6.T11)\.

Table 11:Mean post\-compaction context length \(in tokens\) under varying pre\-compaction context lengths\.Output length is nearly invariant to input size\.Table 12:Session Constraints \(SCs\) and behavioral probes used to evaluate post\-compaction instruction retention\. Each probe pairs a user query with the compliant \(✓\) and violating \(✗\) agent responses\.Table 13:Proposed taxonomy of Side Constraints \(SCs\)\.
## Appendix GGPT\-5\.4 mini Evaluation and 220K Dataset

We additionally evaluate GPT\-5\.4\-miniOpenAI \([2026b](https://arxiv.org/html/2608.11242#bib.bib32)\), a closed\-source model released by OpenAI\. We selected the second\-largest model in this family in order to keep the experimental cost at a tractable level\. Since ourCompIntevaluates LLM\-based compactors near their context limits, we curate datasets at approximately 80% of the model’s nominal context window\. Given the substantial cost of OpenAI’s API calls under long\-context inputs, we use 40% of the original dataset, namely 20 filler contexts×\\times15 manually crafted examples, yielding 300 evaluation samples\.

According to its documentation, GPT\-5\.4\-mini supports a context limit of 400K tokens\. However, our queries to OpenAI’s official API with samples of approximately 300K tokens consistently failed, returning an error indicating that the input exceeded the model’s context window\. This behavior may be attributable to an internal system prompt, the contents of which are not publicly disclosed in the documentation\. Accordingly, we re\-tested with contexts of approximately 250K tokens, though a subset of queries continued to fail \(since the realized context length fluctuates across samples\)\. Ultimately, we settled on a target of 220K tokens to simulate a near\-limit compaction trigger \(as discussed in Section[4\.2](https://arxiv.org/html/2608.11242#S4.SS2)\) with the assumption that 220K is very close to the limit \(set by OpenAI API\) of the context window\. And the discrepancy of the advertised 400K context and us hitting context window limit at around 250K tokens\.

For the WildChat and Hermes Agent datasets, we apply the same stitching methodology proposed in Section[4\.2](https://arxiv.org/html/2608.11242#S4.SS2)and detailed in Appendix Section[A\.1](https://arxiv.org/html/2608.11242#A1.SS1)\. However, our single user\-turn dataset, OpenResearcher, does not contain rows of 220K\-token context\. Consequently, we construct each entry by concatenating two 110K\-token entries to reach the 220K\-token target\.

Our evaluation with GPT\-5\.4\-mini does not follow the four\-group setup introduced in Section[4\.4](https://arxiv.org/html/2608.11242#S4.SS4)\. This is because the full\-context groups \(Long ctx w/ SC and Long ctx w/o SC\) evaluate compliance without the compactor, which requires an LLM with a context window longer thanHtH^\{t\}andHstH^\{t\}\_\{s\}\. Although we employ gpt\-oss\-120b as the compliance model throughout the paper, it cannot accommodate the context length of 220K tokens required in these settings\.

## Appendix HInjection Location Evaluation

![Refer to caption](https://arxiv.org/html/2608.11242v1/x7.png)Figure 8:SC retention rate by injection location\. Each row corresponds to a dataset–compactor combination\. WC: WildChat; HA: Hermes Agents; R5: Recent\-5; Lingua: LLMLingua\-2; GPT\-A: gpt\-oss\-120b with Anthropic prompt; GPT\-P: gpt\-oss\-120b with pi\-mono prompt; Qwen\-A: Qwen30B\-A3B with Anthropic prompt\.### H\.150K Context

We additionally evaluate the injection location at a 50K context length\. For LLM\-based compactors, retention at the top and middle positions improves substantially, while bottom retention is essentially unchanged\. The improvement decays with position: in HA/GPT\-P, top retention rises from 36% to 59% \(\+\+23\), middle from 57% to 70% \(\+\+13\), and bottom from 80% to 81% \(\+\+1\)\. This gradient is the prediction of a probe\-proximity account: halving the context most reduces the SC\-to\-probe distance for top\-injected SCs, less so for middle, and leaves bottom\-injected SCs essentially as close to the probe as before\. An intrinsic\-property account, in which retention is governed by SC phrasing or type independently of position, predicts no such position\-dependent improvement and is not consistent with the observed pattern\.

### H\.2Attention\-mechanism Discussion

Several of the observations in Section[5\.2](https://arxiv.org/html/2608.11242#S5.SS2)about SC repetition are consistent with prior work on the attention mechanism in long\-context language models\.Hsiehet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib46)\); Liuet al\.\([2024](https://arxiv.org/html/2608.11242#bib.bib36)\)suggest that language models attend more readily to later context than to middle context\. However, our finding of a monotonic increase in retention rate as the SC is injected later does not fully accord with their results, which report that performance is typically highest when the relevant information is located at the top or the bottom of the context\. In our setting, retention is higher when the SC is injected in the middle than at the top\. A key distinction between our setup and theirs is that prior work evaluates information retrieval, whereas our setup evaluates compaction, namely generation conditioned on the full context\. Our finding and set\-up is more consistent with that ofZhaoet al\.\([2025](https://arxiv.org/html/2608.11242#bib.bib8)\), who report that LLMs follow instructions more effectively when the query is positioned closer to the relevant statement\.

## Appendix ISC Type Per Dataset Analysis

![Refer to caption](https://arxiv.org/html/2608.11242v1/x8.png)\(a\)Hermes Agents
![Refer to caption](https://arxiv.org/html/2608.11242v1/x9.png)\(b\)OpenResearcher
![Refer to caption](https://arxiv.org/html/2608.11242v1/x10.png)\(c\)WildChat

Figure 9:SC retention rate by type\.Supplementing Figure[6](https://arxiv.org/html/2608.11242#S5.F6)\.Figure[9](https://arxiv.org/html/2608.11242#A9.F9)disaggregates the averaged retention rates of Figure[6](https://arxiv.org/html/2608.11242#S5.F6)into the three datasets\. The high\-level conclusions from the main text remain visible: the non\-LLM compactors retain essentially no SCs of any type on any dataset, and no compactor approaches reliable retention even in the best per\-dataset cell \(Preference under gpt\-oss \(pi\-mono\) on OpenResearcher, at 55%\)\. The per\-dataset view also surfaces nuances that the average obscures\. The Preference advantage for LLM\-based compactors is largely driven by OpenResearcher; on Hermes Agents, Information \(53%\) slightly exceeds Preference \(51%\) under gpt\-oss \(pi\-mono\), so the averaged ordering should be read as a tendency rather than a strict ranking\. Process SCs remain near the bottom in most rows but are not always strictly the lowest, with Action tying or undercutting Process in several cells\. WildChat is substantially harder than the other two datasets: gpt\-oss \(pi\-mono\), the strongest compactor elsewhere, drops to 0% across all five SC types, and no compactor exceeds 10% on any type, suggesting that retention is shaped not only by SC type and compactor choice but also by characteristics of the underlying conversations\.

## Appendix JSC Extractor Details

### J\.1Training\-Free Serving Setup

Our SLM\-based extractor is training\-free: we use the off\-the\-shelf Qwen3\.5\-9B model with thinking disabled and rely solely on prompting, without fine\-tuning or additional training\. In our experiments, Qwen3\.5\-9B runs on the same serving platform as the LLM compactors described in Appendix Section[B\.2](https://arxiv.org/html/2608.11242#A2.SS2)\.

### J\.2Online Extraction and Registry Update

At every user turn, the extractor receives three inputs: \(1\) the current user message, \(2\) the immediately preceding assistant message for resolving references such as “do that going forward,” and \(3\) the existing session\-level constraint registry for deduplication\. The previous assistant message is used only to resolve references in the current user turn, and the existing registry is used only to suppress duplicate or paraphrased constraints rather than to infer new ones\.

The extractor identifies user instructions or preferences intended to persist across future turns of the same session\. It returns structured JSON containing a canonical one\-sentence formulation and a short supporting evidence span for each extracted SC\. If no SC is detected, it returns an empty list\. Newly identified, non\-duplicate constraints are appended to the session registry\. At compaction time, the registry is appended to the compaction summary, preserving the extracted constraints in a form analogous to the upper\-bound condition\.

### J\.3Prompt

For space, we summarize the main components of the extraction prompt below; the full prompt, including few\-shot examples, is included in our released code\.

SC Extraction Prompt SummarySystem:The extraction prompt instructs the model to detect Session\-Scoped Side\-Constraints \(SCs\) from the current user turn only\. It emphasizes that most turns contain no SC and that the default output should be an empty list\.SC definition:An SC is defined as a user instruction or preference about how the assistant should behave that is intended to persist across future turns of the same session, rather than applying only to the current task\.Persistence criterion:For each candidate instruction, the model is asked to determine whether the instruction would still apply if the user asked an unrelated question several turns later\. The instruction is extracted only when this persistence is clear; uncertain cases are discarded\.Exclusion rules:The prompt tells the model not to extract current\-task instructions, one\-off corrections, local formatting requests, politeness or filler, or background facts that do not imply a behavioral constraint\.Registry handling:The prompt provides the current registry of previously extracted SCs\. The model uses this registry only to suppress duplicate or paraphrased SCs, not to infer new ones\.Disambiguation context:When available, the previous assistant turn is provided only to resolve references in the current user turn, such as “do that going forward\.” The model is instructed not to extract SCs from the assistant turn\.Output format:The model must return JSON with a list of extracted SCs\. Each extracted SC contains a canonical one\-sentence phrasing and a short evidence span from the current user turn\. If no SC is detected, the model returns an empty list\.

### J\.4Retention Across Interaction Environments

Table[4](https://arxiv.org/html/2608.11242#S5.T4)reports retention rates of 90\.3%, 95\.6%, and 95\.1% on WildChat, Hermes Agent, and OpenResearcher, respectively, corresponding to an average of 93\.7%\. The extractor therefore maintains consistently high retention across interaction environments, including WildChat, where the SC\-targeted compaction prompt remains substantially less effective\. On WildChat, the extractor’s 90\.3% retention is 52\.7 percentage points higher than the best targeted\-prompt result of 37\.6% reported in Appendix Section[B\.3](https://arxiv.org/html/2608.11242#A2.SS3)\.

### J\.5Per\-Constraint\-Type Analysis

Table 14:SC extractor retention by constraint type and dataset\. Averages are calculated using the unrounded dataset\-level rates\. Hermes: Hermes Agent, OpenResearch: OpenResearcher\.The extractor achieves consistently high retention across all five SC types and all three interaction environments\. Average retention ranges from 88\.2% for Action constraints to 97\.1% for Preference constraints\. Although Action constraints are the most challenging category, their retention remains above 80% in every dataset\. These results show that the extractor’s aggregate performance is not driven by a single easily detected SC type and generalizes across the full SC taxonomy\.

### J\.6Deployment in an Agent Harness

At deployment, the SLM\-based extractor can be integrated into an agent harness and invoked after each user query\. The extractor updates a session\-scoped constraint registry using the current user message, the immediately preceding assistant message, and the existing registry\. The agent harness retains this registry separately from the main conversation history and appends it to the compaction summary whenever compaction occurs\. This design requires neither modification nor retraining of the primary agent model or compactor and prevents the constraint registry from being overwritten by compaction\.