## TPBench：面向对话压缩的转折点基准（A Turning-Point Benchmark for Dialogue Compression）

# TPBench: A Turning-Point Benchmark for Dialogue Compression
Source: [https://arxiv.org/html/2610.02736](https://arxiv.org/html/2610.02736)
Seunghyun YoonHyuk LimAffiliation:Korea Institute of Energy Technology \(KENTECH\)Affiliation:\{minjipark, syoon, hlim\}@kentech\.ac\.kr

###### Abstract

A compressor can keep the facts of a dialogue and still drop the turn that changed them\. A user corrects a price, reverses a choice, or adds a constraint\. We call this failure turning\-point eviction\. One overall retention score hides it, because that score mixes what the user first wanted with what the user wants now\.

We introduce TPBench, which evaluates three complementary information targets at shared nominal retention budgets\. P1 asks for the user’s initial goal\. P2 asks for the current value of a slot the user revised\. P3 asks for both, in dialogues with a late annotated slot update\. The current\-value answers come from the human dialogue\-state annotations of MultiWOZ and SGD\. The initial\-goal answer is the first sentence of the first user turn\. Neither requires new crowdsourcing\.

The probe\-specific evaluations rank compression methods differently\. On the joint probe at a retained fraction of0\.300\.30, every tested compressed method remains below full context with the main Llama reader\. Deleting the turn that carries the update sharply lowers current\-value accuracy, while deleting one matched irrelevant turn leaves it unchanged\. A Mistral reader repeats the P2/P3 rankings and the joint\-probe gap\. Current\-value recovery is tested on an additional corpus, LongMemEval\-KU, and on Chinese RiSAWOZ: full context has the highest accuracy, and recency has the highest compressed\-method mean in both evaluations\.

## 1Introduction

A user asks an assistant to find a hotel in the east\. Later, the user changes the price to under $50\. A compressor may preserve the hotel request and drop the price change\. Although the compressed dialogue remains topically relevant, it no longer contains the evidence needed to determine the current price constraint\. We refer to this failure as*turning\-point eviction*\.

The initial request and the updated price occupy different parts of the dialogue\. The initial request describes the task\. The updated price records the latest value of a*slot*, or named field\. Price, departure time, and destination are examples of slots\.

Within each comparison, the same language model reads each compressed dialogue\. We call it the*reader*\. The reader answers a question using only the context that a compression method retained\. Its answer reveals which information remains usable after compression\.

TPBench contains three probes:

- •P1 \(initial goal\)asks what the user originally wanted\.
- •P2 \(current value\)asks for the final value of a slot that changed during the dialogue\.
- •P3 \(initial goal and late update\)asks for both answers in dialogues where the final annotated slot update occurs in the second half\.

Figure[1](https://arxiv.org/html/2610.02736#S1.F1)shows why the three results are reported separately\. A context that keeps only the first turn can answer P1\. A context that keeps only the later price statements can answer P2\. P3 requires both parts of the dialogue\.

T0user: Find a hotel in the east\.
initial goalT3user: A moderate price is fine\.
earlier valueT6user: Actually, make it under $50\.
turning pointT7system: I found one under $50\.
answer repeatedA dialogue with one revisionkeeps T0, drops T6 and T7
✓ P1 initial goal✗P2 current valuedrops T0, keeps T6 and T7
✗P1 initial goal ✓ P2 current valuekeeps T0 and T6
✓ P3 asks for both at onceWhat a compressed context supportsFigure 1:An illustrative dialogue separates the three probes\. P1 needs the initial goal, P2 needs the current value, and P3 needs both\. On the left, the heavy box is the turning point, the user turn that revises the slot\. On the right, each box is one compressed context; a check mark means the probe can be answered from it and a cross means it cannot\. The heavy box on the right is the only context that answers P3\.A*turn*is one user or system message, indexed from zero\. We use*turning point*operationally for a user turn with an annotated slot update\. An annotation change need not be a semantic revision: P2 removes changes identified by its alias heuristic, while P3 retains late annotation changes and reports an alias\-exclusion sensitivity analysis\.

Schema\-Guided Dialogue \(SGD\)\([Rastogi et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib35)\)and MultiWOZ 2\.2\([Budzianowski et al\., 2018](https://arxiv.org/html/2610.02736#bib.bib2);[Zang et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib48)\)make these questions reproducible\. Both corpora contain multi\-turn dialogues from several service domains\. Their annotations record active slot values as the conversation changes\. TPBench derives questions from these records\. It then evaluates every compression method with the same reader and scoring rules\.

The benchmark serves two purposes\. First, it identifies which part of a dialogue a method tends to preserve\. Second, it measures whether both parts remain available when an application needs the original task and the latest constraint together\. TPBench supplies probe\-specific comparisons of released compression pipelines for these complementary requirements\.

##### Contributions

- •We separate dialogue\-compression evaluation into three targets: recovering the user’s initial goal, recovering the current value after a revision, and recovering both after a late update\.
- •We build TPBench from the SGD and MultiWOZ dialogue\-state annotations\. Probe construction and scoring are deterministic, and no new crowdsourced labels are needed\.
- •Across seven turn selectors and four KV\-cache methods, the preferred method differs across probe\-specific evaluations\. In the main Llama evaluation, joint recovery remains substantially below full context for every tested turn selector\.
- •We validate the current\-value probe with matched deletion controls, and we test P2/P3 with a second reader and current\-value recovery with an additional corpus and Chinese dialogue\.

## 2Related work

##### Long\-context compression methods

Prompt\-side compression shortens the text sent to a reader\. LLMLingua\-2 scores token importance\([Pan et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib30)\), and LongLLMLingua extends the idea to long inputs by reordering and budgeting across segments\([Jiang et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib19)\)\. ICAE represents context in compact memory slots\([Ge et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib12)\)\. MMR selects relevant and diverse content\([Carbonell and Goldstein, 1998](https://arxiv.org/html/2610.02736#bib.bib5)\), an objective that predates neural readers and is still used to rank passages alongside lexical scoring functions\([Robertson and Zaragoza, 2009](https://arxiv.org/html/2610.02736#bib.bib37)\)and dense sentence encoders\([Devlin et al\., 2019](https://arxiv.org/html/2610.02736#bib.bib8);[Reimers and Gurevych, 2019](https://arxiv.org/html/2610.02736#bib.bib36);[Chen et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib6)\)\.[Li et al\. \(2024b\)](https://arxiv.org/html/2610.02736#bib.bib22)survey the area, and[Nagle et al\. \(2024\)](https://arxiv.org/html/2610.02736#bib.bib29)give a rate\-distortion account of what any black\-box prompt compressor can achieve\. These methods differ in the signal they score: token probability, embedding similarity, or attention weight\. Each signal answers a different question about a turn\. Under a diversity criterion, the turn that states the user’s opening request can lose weight once later turns repeat the topic\. Under a token\-level importance criterion, a brief revision can receive little weight\. Our measurements make this concrete: the selector that recovers the initial goal best is not the selector that recovers the current value best, on either dataset\.

KV\-cache compression reduces the internal state that a transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2610.02736#bib.bib40)\)retains during decoding\. Attention scores guide H2O\([Zhang et al\., 2023](https://arxiv.org/html/2610.02736#bib.bib49)\)and ScissorHands\([Liu et al\., 2023](https://arxiv.org/html/2610.02736#bib.bib24)\)\. Other retention schemes include SnapKV\([Li et al\., 2024a](https://arxiv.org/html/2610.02736#bib.bib23)\), PyramidKV\([Cai et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib4)\), and StreamingLLM\([Xiao et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib44)\), whose attention\-sink argument rests on the position of a token in the sequence rather than its content\. KIVI quantizes the cache\([Liu et al\., 2024c](https://arxiv.org/html/2610.02736#bib.bib25)\)\. ChunkKV prunes groups of cache entries\([Liu et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib28)\)\. A more recent line allocates the budget adaptively rather than uniformly\. Ada\-KV varies the budget across attention heads\([Feng et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib11)\)and DynamicKV varies it across layers\([Zhou et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib52)\)\. KeyDiff evicts by key similarity\([Park et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib32)\), and FastKV separates context reduction from cache compression\([Jo et al\., 2026](https://arxiv.org/html/2610.02736#bib.bib20)\)\. This family and the prompt\-side family retain different objects, a fraction of turns against a fraction of cache entries, so a shared retained fraction does not make their budgets equivalent\. TPBench evaluates representatives from both families under the same probe questions and reader model, supporting descriptive accuracy comparisons across the released pipelines despite their different budget units\.

##### Where information sits in a long input

Position is known to govern what a reader recovers\.[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.02736#bib.bib27)show that accuracy is highest when the relevant passage lies at either end of the input and lowest in the middle, and[Hsieh et al\. \(2024\)](https://arxiv.org/html/2610.02736#bib.bib15)trace part of that pattern to positional attention bias\. This literature treats position as a nuisance to be corrected\. A dialogue turns it into a substantive problem, because the initial goal is anchored near the beginning while a revised value may be established later\. In P3 the final update is constrained to the second half of the dialogue\. A compressor that keeps one end therefore does not fail at random: it fails on a particular probe, which is what TPBench measures\. Several lines of work treat a point of change as the unit that structures a discourse: event boundaries\([Zacks et al\., 2007](https://arxiv.org/html/2610.02736#bib.bib47)\), dialogue\-act modeling\([Stolcke et al\., 2000](https://arxiv.org/html/2610.02736#bib.bib39)\), and topic segmentation in text\([Hearst, 1997](https://arxiv.org/html/2610.02736#bib.bib13)\)and in dialogue\([Park et al\., 2023](https://arxiv.org/html/2610.02736#bib.bib31)\)\. A turn\-level importance score need not explicitly represent the role of an update in changing the dialogue state\.

##### Long\-context and dialogue evaluation benchmarks

LongBench\([Bai et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib1)\)and∞\\inftyBench\([Zhang et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib50)\)test a broad range of long\-context abilities\. LongGenBench targets long\-form generation\([Liu et al\., 2024b](https://arxiv.org/html/2610.02736#bib.bib26)\)\. CLongEval covers Chinese\([Qiu et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib33)\)\. MT\-Bench evaluates multi\-turn instruction following\([Zheng et al\., 2023](https://arxiv.org/html/2610.02736#bib.bib51)\)\. LongMemEval tests long\-term memory across sessions\([Wu et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib43)\), and long\-term dialogue datasets supply the conversational setting for such memory\([Xu et al\., 2022a](https://arxiv.org/html/2610.02736#bib.bib46);[Smith et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib38);[Xu et al\., 2022b](https://arxiv.org/html/2610.02736#bib.bib45);[Jang et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib17)\)\. These benchmarks provide broad task coverage, but do not explicitly separate initial\-goal recovery from later current\-value recovery at shared nominal retention budgets\. TPBench measures these targets separately and adds a probe that requires both at once\.

##### Dialogue\-state and conversational benchmarks

SGD\([Rastogi et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib35)\)and MultiWOZ 2\.2\([Budzianowski et al\., 2018](https://arxiv.org/html/2610.02736#bib.bib2);[Zang et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib48)\)are widely used for dialogue\-state tracking\. The task was organized for years around the Dialog State Tracking Challenges\([Henderson et al\., 2014](https://arxiv.org/html/2610.02736#bib.bib14);[Williams et al\., 2016](https://arxiv.org/html/2610.02736#bib.bib41)\)\. It has been addressed by generation\-based trackers such as TRADE\([Wu et al\., 2019](https://arxiv.org/html/2610.02736#bib.bib42)\)and, more recently, by prompted language models\([Hudeček and Dusek, 2023](https://arxiv.org/html/2610.02736#bib.bib16)\)\. Standard DST evaluates the full state against a schema, and its scoring conventions have themselves been questioned for hiding turn\-level behavior\([Dey et al\., 2022](https://arxiv.org/html/2610.02736#bib.bib9)\)\. TPBench uses changes in that state to create questions\. The reader sees the dialogue and a natural\-language question\. It does not receive the schema\. This difference matters for what a failure means: a DST model that misses an update has misread the dialogue, whereas a reader in our setting may never have been given the turn that carries the update\. The turn\-selection comparisons therefore hold the reader fixed while varying the retained dialogue text\. Cache\-compression comparisons instead vary the reader’s retained attention state\. Information\-state accounts of dialogue treat the state as revised turn by turn\([Larsson and Traum, 2000](https://arxiv.org/html/2610.02736#bib.bib21)\)\. Grounding accounts describe how participants register such revisions\([Clark and Brennan, 1991](https://arxiv.org/html/2610.02736#bib.bib7)\)\. Both motivate treating a revision as a state\-changing event whose evidential role differs from that of the surrounding turns\. Multi\-Session Chat\([Xu et al\., 2022a](https://arxiv.org/html/2610.02736#bib.bib46)\)and Blended Skill Talk\([Smith et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib38)\)cover less structured conversations\. Their records do not provide the slot\-level updates required by the present construction\.

## 3TPBench

Each TPBench example contains a dialogue, a question, and a reference answer\. A turn selector shortens the dialogue text; a KV\-cache method reduces the reader’s retained attention state\. The reader answers using the resulting context\. Within each comparison, methods share the reference answers and reader model\.

The two targets differ in what a compressor has to keep, not only in how hard they are\. Dialogue\-theoretic accounts treat the information state as revised turn by turn[Larsson and Traum \(2000\)](https://arxiv.org/html/2610.02736#bib.bib21), a view also adopted in dialogue state tracking\. We distinguish information anchored in the initial user turn from information determined by the latest slot update\. P1 evaluates recovery of the former, P2 evaluates the latter, and P3 requires both\.

P1 fixes its reference to the first user turn, whereas P2 follows the final annotated update to the queried slot\. A method that keeps the anchor turn can score well on P1 while discarding the evidence needed by P2\. The paired deletions in Section[6\.3](https://arxiv.org/html/2610.02736#S6.SS3)measure the reader’s dependence on the update turn and on the complete set of answer\-bearing turns\. Table[1](https://arxiv.org/html/2610.02736#S3.T1)states the question and the reference answer for each probe\.

Table 1:The three TPBench probes\. The Question column paraphrases each prompt\. Appendix A gives the prompts verbatim\.##### P1: initial goal

The reader answers, “What was the user’s initial goal at the beginning of this conversation?” The reference is the first sentence of the first user turn\. We lowercase alphanumeric tokens, discard tokens shorter than two characters, and remove a predefined stopword list\. The remaining distinct tokens are the content words\. The strict score requires at least two shared content words\. The loose score requires at least one\. TokenF1F\_\{1\}gives a graded view of the same overlap\.

##### P2: current value

The reader is given a slot name, such astrain\-departure, and asked for its current value\. The reference comes from the last annotated update to that slot\. Normalization handles common equivalents such as12 pmand12:00\. The strict score requires equality after normalization\. The loose score also accepts a longer prediction that contains the complete reference at phrase boundaries\.

##### P3: initial goal and late update

P3 scores recovery of the initial\-goal reference and current\-value reference in one answer\. It uses dialogues with at least ten turns\. The final update must occur at index six or later\. It must also occur in the second half of the dialogue\. Both P3 scores require one shared content word for the initial goal\. P3\-strict requires the complete current value at phrase boundaries\. P3\-loose accepts that match or at leastmax⁡\(1,⌊\|V\|/2⌋\)\\max\(1,\\lfloor\|V\|/2\\rfloor\)shared value content words, whereVVis the set of reference value content tokens\. The saved prompt requests “task; slot: value” inside a JSONvaluefield\. It asks for “the user’s task in this conversation,” whereas the goal reference remains the first sentence of the first user turn\. Unlike P1, this wording does not explicitly request the initial goal; in multi\-task dialogues it can invite a current or overall task description\. P3 therefore scores the goal segment more loosely than P1 does\. The P3 scorer parses that field into a goal segment and aslot: valuesegment\. It applies the goal criterion to the goal segment and the value criterion to the value segment\. An answer the parser cannot read scores zero\. Section[6\.2\.3](https://arxiv.org/html/2610.02736#S6.SS2.SSS3)reports the alternative of scoring the whole field at once, together with the share of answers that parse\.

P1 and P2 use different questions and different scoring rules\. Their absolute accuracy values should therefore be read within each probe\. The ranking of methods within P1 can be compared with the ranking within P2\. P3 directly measures joint recovery on one set of examples\. Strict and loose P1 scores report multiword and single\-word overlap, respectively\. P3\-strict uses its own component thresholds; it is not the conjunction of standalone P1\-strict and P2\-strict\. The probes use separately sampled sets with different eligibility rules\. Methods are paired within each probe; P1/P2 ranking comparisons also reflect differences between their sampled sets\.

## 4Constructing the examples

SGD and MultiWOZ 2\.2 provide dialogue\-state tracking \(DST\) annotations that record active slot values throughout a conversation\. These annotations are produced and corrected by people, and MultiWOZ 2\.2 in particular revises the state labels of earlier releases\. Their multi\-domain dialogues contain many revisions, and the state records make those revisions identifiable without collecting new labels\. The two corpora also use different collection procedures and annotation formats\. Using both datasets tests whether the observed effects persist across different annotation formats and data\-collection procedures\.

##### Initial\-goal examples

P1 takes the first sentence of the first user turn\. This reference is fixed by position rather than by an annotation, and it selects one initial\-goal reference for each sampled dialogue\. We use the same sentence when later slot values change\.

##### Current\-value examples

A source question is a stored slot\-update question derived from a dialogue; several source questions can refer to one source dialogue\. P2 construction follows five steps\.

1. 1\.Compare adjacent dialogue states at each user turn\.
2. 2\.Record every slot whose annotated value changes\.
3. 3\.Normalize the old and new values and discard pairs accepted by the corpus\-specific alias\-equivalence heuristic\.
4. 4\.Require the normalized new value to occur as a substring of the dialogue text\.
5. 5\.Keep only the final update to the selected slot\.

The fourth step guards against values that exist only in the annotation\. The fifth step prevents an earlier value from being labeled current\. Each seed draws200200source questions and then applies these five rules to them, so a cell keeps whatever survives\. Effective sample sizes are\{128,125,132\}\\\{128,125,132\\\}for SGD and\{161,168,168\}\\\{161,168,168\\\}for MultiWOZ at seeds\{42,43,44\}\\\{42,43,44\\\}, respectively\. Over the whole source pool the final\-update rule leaves791791SGD and664664MultiWOZ candidates, and the alias rule reduces these to623623and628628\. The value\-occurrence rule removes no further item\.

##### Late updates for P3

P3 selects final annotated slot updates and applies its length and position rules\. Unlike P2, its builder does not apply the alias\-equivalence or value\-occurrence filters\. Its eligible pools contain452452SGD and277277MultiWOZ examples\. Each seed draws200200examples from these eligible pools, so every primary P3 cell has exactly200200examples\. P3 thus measures joint recovery after late annotation changes, including some changes that the P2 alias heuristic treats as equivalent\. Section[6\.2\.2](https://arxiv.org/html/2610.02736#S6.SS2.SSS2)reports a sensitivity analysis excluding those pairs from the saved outputs\. These distinct sampling and eligibility rules explain the P2/P3 sample\-size difference; P3 is not a late\-update subset of a sampled P2 set\.

##### Update turns and answer\-bearing turns

The*update turn*is the user turn where the annotation changes to the final value\. An*answer\-bearing turn*is any user or system turn that states the complete normalized reference\. These sets can differ\. In Figure[1](https://arxiv.org/html/2610.02736#S1.F1), T6 is the update turn\. Both T6 and T7 carry the answer\. The same phrase\-boundary matcher locates answer\-bearing turns and scores loose P2 predictions\.

##### Audit of the update\-turn mapping

The mapping follows the recorded conversion order of the source dialogues\. Every accepted update maps to a corresponding user turn\. Repeated annotations create more than one matching turn for1616MultiWOZ questions\. The recorded conversion order resolves those ties\.

##### Matched deletion controls

The first P2 deletion removes only the update turn\. Its control removes one irrelevant turn from the same speaker\. Word length and dialogue position choose the closest match\. The second deletion removes the update turn and every answer\-bearing turn\. Its control matches both the number of turns and the speaker composition\. Matching succeeds for all128128SGD examples and for159159of the161161MultiWOZ examples\.

##### Leak checks

We check two potential sources of evaluation leakage\. First, the P1 reference is the first sentence of turn 0, so any selector that keeps the earliest turns retains the P1 reference by position\.*Earliest turns*keeps turn00by rule and scores0\.4800\.480on SGD P1\-strict against0\.4870\.487for full context\. P1 therefore measures whether a selector keeps the opening of the dialogue\. Second, the seed sample could over\-represent dialogues with explicit revision wording, which would favour methods keyed to correction cues\. The audit compares the examples a seed actually evaluates against the rest of the final\-update pool of791791and664664candidates\. On seed4242, a fixed user\-side correction\-cue regular expression fires on28\.1%28\.1\\%of the128128evaluated SGD examples versus21\.3%21\.3\\%of the663663other final\-update candidates \(\+6\.9\+6\.9pp\), and on32\.3%32\.3\\%of the161161evaluated MultiWOZ examples versus28\.8%28\.8\\%of the remaining503503\(\+3\.5\+3\.5pp\)\. These complements include candidates removed by P2 filtering, so the comparison combines sampling and eligibility effects\.

## 5Experimental setup

##### Turn selectors

We evaluate seven methods that retain whole dialogue turns\. Four selectors use position alone\. Recency keeps the latest turns\. The earliest\-turn baseline keeps the first turns\. Random selection samples turns uniformly\. Uniform stride spreads retained turns across the dialogue\.

Three selectors use dialogue content\. The main evaluation supplies fixed contexts labeled H2O\-inspired attention proxy, embedding maximal marginal relevance \(MMR\), and LLMLingua\-2\. The released attention proxy averages attention over layers and heads, sums it over query positions, and averages within each turn\([Zhang et al\., 2023](https://arxiv.org/html/2610.02736#bib.bib49)\)\. The released MMR implementation uses a trade\-off of0\.50\.5and the mean turn embedding as its query\([Carbonell and Goldstein, 1998](https://arxiv.org/html/2610.02736#bib.bib5)\)\. The free\-form experiment of Section[6\.6](https://arxiv.org/html/2610.02736#S6.SS6), which we call FMTS, records GPT\-2 as the feature model for these two selectors\. The feature model and the generation environment for the main H2O and MMR caches were not recorded, so that configuration does not establish what produced them\.

The LLMLingua\-2 turn selector uses binary word\-retention labels fromllmlingua\-2\-xlm\-roberta\-large\-meetingbank, run at its own token rate of0\.50\.5\([Pan et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib30)\)\. That rate governs the label model and is separate from the turn budget of the experiments\. It averages those labels within each turn and keeps the highest\-scoring turns\.

The release includes fixed compressed contexts and saved reader outputs\. These support reconstruction of the reader inputs and deterministic rescoring; they do not by themselves establish identical outputs from a new inference run\.

The saved main full\-context and position\-only inputs retain\[Tn\] USER/SYSTEMturn and speaker markers\. The H2O/MMR/LLMLingua\-2 inputs concatenate selected turn texts without those markers\. Together with the floor/round budget difference below, this means the comparisons describe released pipelines at shared nominal budgets; they do not isolate selection logic from rendering and rounding choices\.

##### KV\-cache methods

KV\-cache compression leaves the displayed dialogue intact\. It changes the reader’s stored attention keys and values\. ChunkKV groups cache entries into chunks before pruning\([Liu et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib28)\)\. It is the one cache method we evaluate on P3, in Table[25](https://arxiv.org/html/2610.02736#S6.T25)\. SnapKV, PyramidKV, and StreamingLLM are evaluated on P2\.

The budgetrrdenotes the target fraction retained by a method\. A main\-pool positional selector keepsk=max⁡\(1,round⁡\(nturns​r\)\)k=\\max\(1,\\operatorname\{round\}\(n\_\{\\mathrm\{turns\}\}r\)\)dialogue turns\. The LLMLingua\-2 instead keepsk=max⁡\(1,⌊nturns​r⌋\)k=\\max\(1,\\lfloor n\_\{\\mathrm\{turns\}\}r\\rfloor\)turns, as do H2O and MMR\. Thus the same nominalrrcan produce a one\-turn budget difference\. A KV method keeps the fractionrrof the KV cache\. The two method families therefore share a retained\-fraction convention\. Their physical units differ: retaining a fraction of turns need not retain the same fraction of tokens\.

##### First\-user\-plus\-recent baseline

This selector retains the first USER turn and the latestk−1k\-1other turns, wherek=max⁡\(1,round⁡\(nturns​r\)\)k=\\max\(1,\\operatorname\{round\}\(n\_\{\\mathrm\{turns\}\}r\)\), in chronological order\. It uses only turn position and speaker identity\. Its matched comparison includes full context, earliest turns, recency, and uniform stride \(Section[6\.2\.1](https://arxiv.org/html/2610.02736#S6.SS2.SSS1)\)\.

##### Reader and samples

The main reader is Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib10)\)\. Decoding is greedy\. Turn\-selection results use seeds\{42,43,44\}\\\{42,43,44\\\}\. P1 is evaluated atr=0\.30r\{=\}0\.30\. P2 usesr∈\{0\.10,0\.30\}r\\in\\\{0\.10,0\.30\\\}in the main tables andr∈\{0\.50,0\.70\}r\\in\\\{0\.50,0\.70\\\}in the wider\-budget comparison of Section[6\.4\.1](https://arxiv.org/html/2610.02736#S6.SS4.SSS1)\. P3 usesr=0\.30r\{=\}0\.30\. Mistral\-7B\-Instruct replicates P2 and P3, and the free\-form experiment of Section[6\.6](https://arxiv.org/html/2610.02736#S6.SS6)uses GPT\-5\.4\-mini\. Each seed begins with200200sampled probe examples per dataset and probe\. P2 filtering produces the smaller effective counts reported above\. Turn selectors within each cell share the examples, questions, reader model, and probe\-level decoding settings\. Their turn rounding and context rendering can differ as described above\. The KV\-cache evaluations use the same reader model and probe questions, with cache\-specific execution settings\. Full context uses the reader without compression\.

##### Scoring

The headline P1–P3 scores use deterministic functions to compare the reader’s answers with the references\. Table[2](https://arxiv.org/html/2610.02736#S5.T2)lists the reader, the sample, and the scoring rule of every comparison\.

Table 2:Experiment scope\. Main seeds sample probe sets\. LongMemEval\-KU and RiSAWOZ instead keep the eligible set fixed and vary random selection\. Turn fractions and cache fractions have different physical units\.

## 6Findings

### 6\.1Rankings across the three probes

The best compression method changes with the question\. This section reports the main scores, then asks whether a method falls behind because it throws the answer away or because the reader misses an answer it kept\.

Table 3:Main Llama results atr=0\.30r\{=\}0\.30, averaged over seeds\{42,43,44\}\\\{42,43,44\\\}\. P1/P3 use200200examples per dataset and seed; P2 uses\{128,125,132\}\\\{128,125,132\\\}SGD and\{161,168,168\}\\\{161,168,168\\\}MultiWOZ examples\. P1 columns show strict / loose scores, P2 is strict, and P3land P3sare the loose and strict joint scores of Section[3](https://arxiv.org/html/2610.02736#S3)\. Italic denotes full context; bold denotes the best compressed method in each column\.Figure[2](https://arxiv.org/html/2610.02736#S6.F2)compares the seven turn selectors atr=0\.30r\{=\}0\.30\. On SGD, the H2O proxy and uniform stride tie for the highest P1\-strict score \(0\.4930\.493\), whereas embedding MMR leads P2\-strict \(0\.3090\.309\) and scores0\.1780\.178on P1\-strict\. On MultiWOZ, earliest turns, the H2O proxy, and uniform stride fall within0\.0040\.004of each other on P1\-strict \(0\.4070\.407–0\.4100\.410\), and random selection leads P2\-strict \(0\.3140\.314\)\. Across the seven selectors the Spearman rank correlationρ\\rhobetween P1\-strict and P2\-strict is−0\.34\-0\.34on SGD and\+0\.11\+0\.11on MultiWOZ\. The association is negative on SGD and weakly positive on MultiWOZ\. On both datasets a different selector has the highest mean in each probe\. These comparisons describe target\-specific performance on the sampled probe sets, rather than isolating the effect of changing only the question\. Section[6\.1\.4](https://arxiv.org/html/2610.02736#S6.SS1.SSS4)separates answer retention from reader success conditional on retention\.

SGD1234567P1\-strictP1\-looseP2\-strictMultiWOZ1234567P1\-strictP1\-looseP2\-strictrecencyrandom selectionearliest turnsuniform strideattention/H2O proxyembedding MMRLLMLingua\-2Figure 2:Rank changes among the seven compressed turn selectors atr=0\.30r\{=\}0\.30\. Rank 1 is best and ties take the average rank\. Each line follows one method from P1\-strict through P1\-loose to P2\-strict\. Solid lines are the four selectors that use position alone and dashed lines are the three that use dialogue content, as defined in Section[5](https://arxiv.org/html/2610.02736#S5)\. A line that falls steeply between P1\-loose and P2\-strict marks a method that keeps the opening but loses the revision\. Table[3](https://arxiv.org/html/2610.02736#S6.T3)reports the corresponding accuracies\.##### P1 scoring sensitivity

Full context attains P1\-strict0\.4870\.487on SGD and0\.4300\.430on MultiWOZ; the loose scores are0\.8500\.850and0\.8670\.867\. These thresholds measure lexical recovery at different levels of detail\.

##### Aggregation can hide different strengths

On SGD, earliest turns scores0\.4800\.480on P1\-strict and0\.1560\.156on P2\-strict, while embedding MMR scores0\.1780\.178and0\.3090\.309\. One selector favors the initial goal, the other the current value\. A weighted average changes their order with the probe mix\.

#### 6\.1\.1AI\-assisted semantic evaluation of P1

The lexical P1 rule rejects valid paraphrases, so this evaluation asks whether it understates methods whose answers restate the goal in other words\.

We evaluate8,4008\{,\}400Llama P1 answers atr=0\.30r\{=\}0\.30: two datasets, three seeds, and seven conditions including full context\. The3,0493\{,\}049distinct reference–answer pairs each have one cached judgment\. The judged set covers full context and six turn selectors\. Inputs contain the initial\-goal reference and reader answer, without method identities or expected rankings\.

The judge isgpt\-5\.4\-mini\-2026\-03\-17, with temperature00, reasoning effortnone, and a512512\-token output limit\. Full equivalence accepts paraphrases and requires the action, target, and explicit task\-relevant constraints to retain their meaning\. For example, “find a restaurant” omits constraints from a request for a moderately priced restaurant in the west\. The judge records a short explanation and ambiguous decisions\.

##### Agreement with external human judgments

We assess the equivalence criterion on Answer Equivalence \(AE\)\([Bulian et al\., 2022](https://arxiv.org/html/2610.02736#bib.bib3)\), which supplies human ratings of answers to factual questions\. Fixed, label\-independent samples contain200200development and600600test pairs, with disjoint question IDs across splits\. Multiple human ratings are combined by majority vote; one development pair and five test pairs have tied votes and remain unresolved\. Human labels are excluded from the judge input\. The equivalence prompt and model are identical for development, test, and TPBench\. AE inputs include the question and source passage; P1 inputs supply the initial\-goal reference without the remaining dialogue\.

Development agreement is0\.8590\.859on199199resolved pairs\. On the595595resolved test pairs, agreement is0\.8370\.837, with a question\-clustered bootstrap95%95\\%interval of\[0\.807,0\.866\]\[0\.807,0\.866\]\(10,00010\{,\}000resamples\)\. Balanced accuracy is0\.8410\.841\. The confusion counts are201201true positives,297297true negatives,6464false positives, and3333false negatives, with no uncertain model decisions\. A fixed token\-F1≥0\.5F\_\{1\}\\geq 0\.5baseline attains accuracy0\.7700\.770and balanced accuracy0\.7460\.746on the same pairs\. The paired accuracy gain is6\.76\.7percentage points, with a95%95\\%interval of\[3\.0,10\.4\]\[3\.0,10\.4\]\. All600600sampled test pairs differ under normalized exact match; that baseline always predicts non\-equivalence and scores0\.6070\.607accuracy\. AE evaluates the full equivalence decision in factual QA\.

##### P1 semantic results

Full context scores0\.4820\.482on SGD and0\.1800\.180on MultiWOZ\. The MultiWOZ score is an empirical uncompressed baseline, not an upper bound: the judge accepts fewer than one fifth of the full\-context answers as fully equivalent to the initial reference\. The reader requests a short noun phrase, whereas the judge requires explicit task\-relevant constraints; this difference can penalize omissions\. The low score limits absolute initial\-goal recovery under this reader/reference/judge setup even before compression\. Source\-dialogue\- clustered95%95\\%bootstrap intervals are\[0\.433,0\.533\]\[0\.433,0\.533\]and\[0\.142,0\.221\]\[0\.142,0\.221\]\(5,0005\{,\}000resamples\)\. Repeated appearances across seeds remain in the same cluster\. Uncertain judgments remain in the denominator and receive no credit\.

Table 4:AI\-assisted full semantic equivalence of initial\-goal answers atr=0\.30r\{=\}0\.30, averaged over three seeds \(200200examples per dataset and seed\)\. This analysis covers six selectors and full context\. LLMLingua\-2 is outside the judged set\. Italic denotes full context; bold denotes the best compressed method in each column\.In Table[4](https://arxiv.org/html/2610.02736#S6.T4)the H2O proxy leads compressed\-method equivalence on SGD, and earliest turns leads on MultiWOZ\. Correlations with P2\-strict among these six selectors are−0\.60\-0\.60and\+0\.37\+0\.37, respectively\. The best selector differs between initial\-goal and current\-value recovery on both datasets\.

#### 6\.1\.2What a single average hides

LetR1​\(C\)R\_\{1\}\(C\)andR2​\(C\)R\_\{2\}\(C\)denote the P1 and P2 accuracies of compressorCC\. A weighted averageα​R1​\(C\)\+\(1−α\)​R2​\(C\)\\alpha R\_\{1\}\(C\)\+\(1\-\\alpha\)R\_\{2\}\(C\)depends on the probe mix throughα\\alpha\. Two methods can have nearly equal averages even when one leads on P1 and the other leads on P2\.

##### Concrete instance

At SGDr=0\.30r\{=\}0\.30\(Table[3](https://arxiv.org/html/2610.02736#S6.T3)\), earliest turns scores0\.4800\.480on P1\-strict and0\.1560\.156on P2\-strict, while embedding MMR scores0\.1780\.178and0\.3090\.309\. Their weighted\-score difference is0\.302​α−0\.153​\(1−α\)0\.302\\alpha\-0\.153\(1\-\\alpha\), which changes sign at approximatelyα=0\.336\\alpha=0\.336\. Thus the aggregate ranking depends on the evaluation mix, while the separate scores identify each selector’s strength\.

The P2 order could come from what each method keeps or from how the reader reads what remains\. The next three sections separate the two\.

#### 6\.1\.3Retention of the answer text

This diagnostic checks whether a normalized current value appears as a complete phrase in the compressed dialogue before the reader answers\. It uses the same phrase matcher as P2\-loose and therefore separates text retention from reader accuracy\.

The probe builder uses normalized substring matching, whereas this diagnostic uses phrase boundaries\. Two MultiWOZ items contain only plural forms \(“colleges” and “parks”\) for singular references \(“college” and “park”\), accounting for the159/161159/161retention rate\.

Table 5:Fraction of MultiWOZ P2 examples whose current value remains in the compressed dialogue \(r=0\.30r\{=\}0\.30,n=161n\{=\}161, seed4242\)\. Italic denotes full context; bold denotes the best compressed method in each column\.
#### 6\.1\.4Retention and reading

A method can lose a current value in two ways\. The compressed dialogue may no longer contain the value\. Or it may contain the value and the reader may still answer with something else\. Table[5](https://arxiv.org/html/2610.02736#S6.T5)measures the first way on one cell\. This section joins the two measurements example by example, on both datasets and all three seeds\.

We reuse the phrase matcher of Section[6\.1\.3](https://arxiv.org/html/2610.02736#S6.SS1.SSS3)\. For each example we record whether the current value survives compression and whether the reader answers correctly\. Table[6](https://arxiv.org/html/2610.02736#S6.T6)reports the retention rate and the accuracy on the retained examples\. That accuracy is not a method ranking, because each method is scored on the subset it keeps\.

Retention and reader success both vary across methods\. Conditional accuracy on examples retaining the value ranges from0\.640\.64to0\.830\.83on SGD and0\.570\.57to0\.700\.70on MultiWOZ\. The P2\-strict spreads across methods are0\.1950\.195and0\.1890\.189, respectively\.

Full context reads the retained value at0\.6910\.691on SGD and0\.7900\.790on MultiWOZ\. On MultiWOZ every compressed method reads it less often than full context\. On SGD four of the seven read it more often\.

On SGD, uniform stride has the highest retention \(0\.4020\.402\), whereas embedding MMR has the highest P2\-strict accuracy \(0\.3090\.309\)\. Their conditional accuracies are0\.6640\.664and0\.8080\.808, respectively\. LLMLingua\-2 retains the value in0\.1820\.182of SGD examples and answers0\.8270\.827of that retained subset correctly\. Retention alone therefore does not determine the exact method order\.

Table 6:P2\-strict, answer retention, and accuracy on the retained examples atr=0\.30r\{=\}0\.30, averaged over seeds\{42,43,44\}\\\{42,43,44\\\}\. An answer is retained when the complete current value occurs at phrase boundaries in the compressed dialogue, and “Read\|\|kept” scores only those examples\. Italic denotes full context; bold denotes the highest compressed value in the first two columns of each corpus\.
#### 6\.1\.5Results by slot type

This section asks whether the strongest P2 selector is the same for each kind of revised slot\.

Table[7](https://arxiv.org/html/2610.02736#S6.T7)groups the SGD examples by what the revised slot holds\. Random selection has the highest mean on time slots \(0\.2690\.269\), uniform stride on date slots \(0\.3870\.387\), and embedding MMR on place/name slots \(0\.4170\.417\) and the remaining slots \(0\.3720\.372\)\. Thus the aggregate MMR lead does not hold for every slot category\.

LLMLingua\-2 retains0\.2080\.208of time values and0\.1490\.149of date values, compared with0\.1800\.180on place/name and0\.1910\.191on other slots\. Its retention is therefore close to even across slot types\.

Table 7:SGD P2\-strict and answer retention by slot type atr=0\.30r\{=\}0\.30, averaged over seeds\{42,43,44\}\\\{42,43,44\\\}\. Each entry reads P2\-strict / retention, and the four groups together cover every example\. Bold denotes the highest P2\-strict in each column\.

### 6\.2The P3 gap to full context

P3 evaluates whether a compressed context preserves both the initial goal and a late slot update\. Every compressed method scores below full context on it, and this section reports by how much and tests three ways the gap could be an artifact of the sample or the scorer\.

Uniform stride is the strongest compressed turn selector on both datasets\. It also leads in each of the three seeds\. Its loose score is29\.729\.7percentage points below full context on SGD and28\.528\.5points below on MultiWOZ\. The strict gaps are36\.336\.3and29\.729\.7points\.

Embedding MMR leads SGD P2\. Random selection leads MultiWOZ P2\. Neither method leads P3\. Current\-value recovery alone does not identify the best method when the reader also needs the initial goal\. A matched first\-user\-plus\-recent baseline also leaves a substantial P3 gap to full context \(Section[6\.2\.1](https://arxiv.org/html/2610.02736#S6.SS2.SSS1)\)\.

#### 6\.2\.1A matched first\-user\-plus\-recent baseline

This baseline tests whether anchoring the first user turn and reserving the remaining budget for recent dialogue is sufficient for joint recovery\. A late P3 update is constrained to the second half, but need not be near the end\.

The matched comparison fixes the first USER turn and allocates the remaining turn budget to the most recent turns\. All methods use the same probe examples and Llama reader, with greedy decoding,9696output tokens and eager attention for P1/P2, and128128output tokens with SDPA for P3\. The model revision is0e9e39f249a16976918f6564b8830bc894c89659; inference uses bfloat16, PyTorch 2\.5\.1, Transformers 5\.13\.1, and batches of up to eight requests\.

Table 8:Matched first\-user\-plus\-recent comparison atr=0\.30r\{=\}0\.30, three seeds, scored with the segment\-local P3\-strict rule\. This experiment uses the same200200examples per seed as Table[3](https://arxiv.org/html/2610.02736#S6.T3)but a separate generation run\. Its full\-context values differ from that table by at most0\.0070\.007\. Italic denotes full context; bold denotes the best compressed method in each column\.In Table[8](https://arxiv.org/html/2610.02736#S6.T8)the first\-user\-plus\-recent P3\-strict differences from full context are−39\.8\-39\.8points on SGD \(95%95\\%CI\[−46\.4,−33\.1\]\[\-46\.4,\-33\.1\]\) and−34\.2\-34\.2points on MultiWOZ \(\[−41\.4,−27\.2\]\[\-41\.4,\-27\.2\]\), using5,0005\{,\}000source\-dialogue\-clustered resamples\. It improves on earliest turns and recency, but retaining both ends of the dialogue leaves a large joint\-recovery gap\. Its differences from uniform stride have intervals containing zero\.

The same selector attains P1\-strict0\.4880\.488and P2\-strict0\.1630\.163on SGD, and P1\-strict0\.4430\.443and P2\-strict0\.0880\.088on MultiWOZ\. These scores describe its within\-probe behavior; the two metrics have different scales\.

#### 6\.2\.2Sensitivity to excluding alias\-equivalent updates

The primary P3 pool retains late annotation changes, including pairs that P2’s alias heuristic would exclude\. Applying that same heuristic identifies4848,5353, and5353of the200200SGD examples at seeds4242,4343, and4444, and1919,1717, and2020of the MultiWOZ examples\. These are heuristic matches, not human judgments that every pair is semantically identical\. Examples include “afternoon 12:45” versus “12:45 PM” and “Vancouver, BC” versus “Vancouver\.”

Table[9](https://arxiv.org/html/2610.02736#S6.T9)excludes these pairs from the saved segment\-local predictions\. We do not generate new answers or replenish the samples to200200\. Each number is the arithmetic mean of three seed accuracies with the reduced denominators\. The full\-context advantage and the identities of the highest\-mean selectors remain in these reduced sets\. Two selectors fall much further than the rest on SGD P3\-strict: earliest turns drops from0\.1600\.160to0\.0360\.036and the H2O proxy from0\.1680\.168to0\.0380\.038\. Their scores on the primary pool come mostly from the excluded pairs, so their position in the middle of the SGD ranking depends on which pairs the pool keeps\. The leading selector and the full\-context advantage hold in both pools\.

Table 9:Saved\-output P3 sensitivity after excluding pairs accepted by the P2 alias heuristic\. SGD denominators are152/147/147152/147/147and MultiWOZ181/183/180181/183/180for seeds42/43/4442/43/44\. U denotes uniform stride and L the LLMLingua\-2\. “Best” is the highest mean among seven turn selectors, selected separately for each threshold\.For the saved ChunkKV seed\-42 predictions, the same exclusion leaves152152SGD and181181MultiWOZ examples\. Loose/strict scores are0\.171/0\.1510\.171/0\.151and0\.105/0\.0880\.105/0\.088, respectively\. These are single\-seed diagnostics\.

#### 6\.2\.3Sensitivity to scoring the whole answer field

This check asks how much the P3 results depend on scoring the goal and the value in separate segments\.

The reported P3 scorer parses the answer into a goal segment and aslot: valuesegment\. A deterministic parser accepts one semicolon separating a nonempty goal from a nonemptyslot: valueexpression\. The first colon in the second part separates the label from the value, and subsequent colons, including those in times, are preserved\. An answer the parser cannot read scores zero\. The alternative is to apply both criteria to the complete answer field\. That rule credits an answer whose value words appear in the goal part, or the reverse, so it can pass an answer that places the right content in the wrong place\.

This comparison covers all19,60019\{,\}600saved P3 answers:9,6009\{,\}600each for Llama and Mistral and400400for ChunkKV\. Whole\-field scoring passes1,8961\{,\}896Llama answers, of which203203fail under segment\-local scoring:4141because the parser cannot read them and162162because a match lands outside its segment\. Mistral passes1,2691\{,\}269and loses424424, split273273and151151\. ChunkKV passes5252and loses77, split11and66\.

Table 10:P3\-strict on Llama under the two scorers, averaged over seeds\{42,43,44\}\\\{42,43,44\\\}\(200200examples per dataset and seed\)\. “Segment” is the scorer used in the main tables and “Whole” applies both criteria to the complete answer\. “Parsed” is the share of answers the segment parser reads, and “Seg\.\|\|parsed” scores only those\. Italic denotes full context; bold denotes the best compressed method in each accuracy column\.Whole\-field scoring never lowers a Llama cell, and it raises all but one: MultiWOZ embedding MMR scores0\.1100\.110under both\. The SGD order is unchanged\. On MultiWOZ, some relative orders and ties change\. Uniform stride leads the compressed methods on both datasets under both scorers\.

The full\-context advantage over the strongest compressed method is28\.528\.5to36\.336\.3points across the two datasets and thresholds\. We bootstrap that gap over source dialogues \(5,0005\{,\}000paired resamples\)\. On SGD the interval is\[23\.5,35\.9\]\[23\.5,35\.9\]points under the loose threshold and\[30\.1,42\.5\]\[30\.1,42\.5\]under the strict one\. On MultiWOZ the two intervals are\[21\.3,35\.7\]\[21\.3,35\.7\]and\[22\.4,36\.9\]\[22\.4,36\.9\]\. None of the four contains zero\.

Segment\-local scoring charges for response format, because it gives zero to an answer the parser cannot read\. The share of answers that parse ranges from0\.7250\.725to0\.9770\.977across methods on Llama\. On Mistral it ranges from0\.5830\.583to0\.8520\.852, and full context has the lowest share of any condition\. The last column of Table[10](https://arxiv.org/html/2610.02736#S6.T10)scores only the answers that parse\. The full\-context advantage holds in that column on both datasets, and so does the lead of uniform stride among compressed methods\. The whole\-field rule can credit an answer that puts the right content in the wrong segment\. All three scoring views preserve the Llama full\-context advantage and the uniform\-stride lead shown here\.

On Mistral the two scorers disagree about the leading selector only for MultiWOZ\-loose: whole\-field scoring favors uniform stride, whereas segment\-local scoring favors LLMLingua\-2\. Uniform stride leads both SGD thresholds under both scorers, and LLMLingua\-2 leads MultiWOZ\-strict under both\. Full context remains above the strongest compressed mean by6\.36\.3to13\.013\.0points under segment\-local scoring\.

### 6\.3Deleting the turn that carries the update

A low score shows that a method lost something\. This section identifies which turns the reader depends on, by deleting one turn at a time from the uncompressed dialogue and watching the answer change\.

The evidence analysis first locates the final value throughout each dialogue\. The update turn itself does not contain the complete normalized reference in68/12868/128SGD examples and34/16134/161MultiWOZ examples\. The union of the update turn and answer\-bearing turns contains more than one turn in106/128106/128SGD examples and120/161120/161MultiWOZ examples\. A system turn contains the answer in102/128102/128and108/161108/161examples\. These system turns often confirm or restate the value after the user update\. Evidence for the final value is therefore distributed across update and answer\-bearing turns\.

Two paired comparisons measure the contribution of these turns\. The first removes the update turn and compares it with one matched irrelevant turn\. The second removes the update turn and every answer\-bearing turn\. Its control removes the same number of irrelevant turns with the same speaker composition\. These controls measure the effect of deleting evidence separately from the effect of shortening context\.

Table 11:Strict P2 accuracy in the paired deletion experiment, on the seed\-4242P2 sets\. The single\-turn comparison uses128128SGD and161161MultiWOZ examples\. The comparison that removes the update turn together with all answer\-bearing turns uses128128and159159, respectively\. The displayed full\-context row uses the single\-turn sample; all\-evidence comparisons pair their last two rows\.Table[11](https://arxiv.org/html/2610.02736#S6.T11)shows that removing only the update turn lowers strict accuracy on both datasets\. The single\-turn controls leave accuracy at the full\-context level\. The control prompt differs from full context in every example, because one turn is removed from each\. On SGD the reader nevertheless gives the same answer on all128128examples\. On MultiWOZ four answers change and the two totals coincide\. Removing the update turn together with all answer\-bearing turns produces a much larger decrease\. The matched controls do not show that decrease\. Exact McNemar tests compare each evidence deletion with its matched control among items answered correctly in full context\. All four tests givep≤×10−7p\\leq 2\.4\\\!\\times\\\!10^\{\-7\}\.

The single\-turn intervention reduces accuracy by41\.041\.0points on MultiWOZ and16\.416\.4on SGD, demonstrating the reader’s dependence on the annotated update even when other turns retain answer evidence\.

#### 6\.3\.1Locating update turns and testing their removal

The update\-turn mapping replays the SGD and MultiWOZ conversion with its recorded seed and iteration order\. A question maps to a user turn when its slot, old value, new value, question, and answer match the stored row and that turn contains the corresponding state change\. All5,2275\{,\}227SGD and3,8853\{,\}885MultiWOZ source questions match\. Each of the949949SGD candidates has one possible update turn\. In MultiWOZ,722722have one,1515have two, and one has three because an identical state change is repeated\. The recorded conversion order selects one turn in these1616cases\. Missing update turns, updates with no corresponding dialogue turn, and selected system turns all have count zero\. Requiring the selected turn to be the final update to its slot leaves791791SGD and664664MultiWOZ candidates\.

The fields\_originstores the update turn\. The fields\_evidencestores every user or system turn containing the complete normalized reference at phrase boundaries\. Their union is stored ass\_transition\_support\.

The update\-turn control removes one non\-answer turn with the same speaker and the closest word length and position\. The all\-evidence control reuses that turn and adds distinct non\-answer turns to match the number and speaker composition of the remaining answer\-bearing turns; word length and position again break ties\. This construction succeeds for all128128SGD examples and159/161159/161MultiWOZ examples\. The two unmatched MultiWOZ examples remain in the update\-turn comparison\.

Figure[3](https://arxiv.org/html/2610.02736#S6.F3)summarizes the three conditions and what the reader answers in each\. Every condition uses the same full\-context P2 prompt and whole\-turn deletion\. Llama\-3\.1\-8B\-Instruct runs with greedy decoding,max\_new\_tokens=128=128,max\_input\_tokens=7,168=7\{,\}168, and scaled dot\-product attention \(SDPA\)\. Reader errors and malformed outputs count as wrong\.

full contextremove the update turnremove one matchedirrelevant turninitial goal turnunrelated turnupdate turnanswer\-repeating turninitial goal turnunrelated turnupdate turn removedanswer\-repeating turninitial goal turnunrelated turn removedupdate turnanswer\-repeating turncurrent value0\.801superseded value0\.062no answer0\.000current value0\.391superseded value0\.286no answer0\.000current value0\.801superseded value0\.062no answer0\.000Figure 3:Deleting the update turn moves the reader from the current value to the value it replaced, and the matched control does not\. Each column is one deletion condition, red marks the turn removed, and the heavy boxes mark the condition whose answers change\. Rates are MultiWOZ P2 at seed4242\(n=161n\{=\}161\) from Tables[12](https://arxiv.org/html/2610.02736#S6.T12)and[13](https://arxiv.org/html/2610.02736#S6.T13)\.Table 12:P2 deletion experiment on seed4242, reported as strict / loose accuracy\. SGD uses128128examples throughout\. MultiWOZ uses161161for the first three rows and159159for the last two, because two items have no valid matched control for the all\-evidence comparison\. The displayed full\-context row uses the single\-turn sample\.In Table[12](https://arxiv.org/html/2610.02736#S6.T12), among items answered correctly in full context, update\-turn deletion changes23/8623/86SGD and66/12966/129MultiWOZ answers from correct to wrong, compared with0/860/86and2/1292/129for the controls\. The exact paired McNemarpp\-values are2\.38×10−72\.38\\times 10^\{\-7\}and1\.08×10−191\.08\\times 10^\{\-19\}\. Removing the update turn together with all answer\-bearing turns changes78/8678/86and125/128125/128answers, compared with3/863/86and2/1282/128for the controls; the correspondingpp\-values are1\.03×10−211\.03\\times 10^\{\-21\}and1\.88×10−371\.88\\times 10^\{\-37\}\. These tests compare paired outcomes within the full\-context\-correct subsets\.

##### Comparison with the synthetic set

The synthetic comparison evaluates full context, support deletion, and matched control deletion on100100examples from the free\-form set \(Section[6\.6](https://arxiv.org/html/2610.02736#S6.SS6)\) whose answers are confined to three authored turns\. Strict uses normalized exact match; loose accepts phrase\-bounded containment in either direction for these free\-form answers\. Strict accuracy is0\.800\.80in full context,0\.000\.00after removing those three turns, and0\.800\.80after the matched control deletion; loose accuracy is0\.970\.97,0\.000\.00, and0\.960\.96\. The synthetic set confines answer evidence to the three support turns, whereas SGD and MultiWOZ distribute it across update turns and repeated mentions\.

##### One phrase matcher for evidence and scoring

The loose P2 scorer ands\_evidencecall the same function: the complete normalized reference must occur at phrase boundaries in the candidate answer or turn\. Validation recomputes the scores of40,12440\{,\}124saved slot\-value predictions and agrees with every stored score\. The P3 scorer also applies phrase\-boundary checks when matching the complete reference value\.

#### 6\.3\.2What the reader answers when the evidence is gone

Table[13](https://arxiv.org/html/2610.02736#S6.T13)distinguishes explicit abstention from replies that match the superseded value\. It does not classify the mechanism behind every other incorrect answer\. Explicit no\-answer replies stay at or below0\.0130\.013in every condition, including the condition in which strict accuracy falls to0\.0190\.019\. Replies matching the value the user replaced become more frequent\. That rate rises from0\.0940\.094to0\.2340\.234on SGD and from0\.0620\.062to0\.2860\.286on MultiWOZ when the update turn is removed, and to0\.4450\.445and0\.4840\.484when the answer\-bearing turns are removed as well\. The matched controls produce much smaller changes\. Evidence deletion therefore shifts responses toward superseded values while explicit abstention remains rare\.

Table 13:What the reader answers when evidence is deleted, on the seed\-4242P2 sets\. Abstention is the rate of explicit no\-answer replies and old value uses normalized exact matching against the replaced value\. SGD usesn=128n=128; MultiWOZ usesn=161n=161, andn=159n=159for the last two rows\.

### 6\.4Wider budgets and cache\-level compression

The main tables keep30%30\\%of the turns and compress the prompt\. This section loosens the budget and then switches to methods that compress the reader’s attention cache instead of the text\.

Full context improves P2\-strict over the best compressed turn selector by38\.238\.2points on SGD and46\.746\.7points on MultiWOZ\. For prompt\-side selection, retaining30%30\\%of input tokens would reduce the dense prefill attention term by about11×11\\timesand input KV\-cache memory by about3\.3×3\.3\\times\. These scaling factors apply to token retention, and the experimental budgetrrcounts turns\. They are theoretical scaling factors rather than measured end\-to\-end speedups\.

SnapKV, PyramidKV, and StreamingLLM were tested atr=0\.30r\{=\}0\.30on P2\. On MultiWOZ P2, SnapKV scores0\.6890\.689\. PyramidKV scores0\.6830\.683\. The best turn selector on the same seed scores0\.3290\.329\.

#### 6\.4\.1Wider turn budgets

These budgets ask whether the P2 gap to full context persists when a compressed context may keep more turns\.

The wider\-budget evaluation covers full context and four turn selectors \(recency, random selection, earliest turns, and uniform stride\) atr∈\{0\.50,0\.70\}r\\in\\\{0\.50,0\.70\\\}on both datasets for P2\. Results use seeds\{42,43,44\}\\\{42,43,44\\\},200200examples per seed before P2 filtering, and the Llama reader with greedy decoding and128128output tokens\. Full context uses the same samples within each comparison\.

Table 14:P2 strict / loose accuracy atr∈\{0\.50,0\.70\}r\\in\\\{0\.50,0\.70\\\}, mean±\\pmstandard deviation over three seeds\. The three samples contain\{128,125,132\}\\\{128,125,132\\\}SGD and\{161,168,168\}\\\{161,168,168\\\}MultiWOZ examples\. Full context does not use the budget, so its row repeats across the two columns\. Italic denotes full context; bold denotes the best compressed method in each column within each corpus\.In Table[14](https://arxiv.org/html/2610.02736#S6.T14), raisingrrfrom0\.500\.50to0\.700\.70lifts P2\-strict in all eight dataset–method comparisons, by0\.0650\.065–0\.2350\.235\. On MultiWOZ atr=0\.70r\{=\}0\.70, earliest turns scores0\.741±0\.0350\.741\\pm 0\.035against0\.779±0\.0410\.779\\pm 0\.041for full context, a3\.83\.8\-point gap\. The joint P3 probe is evaluated only atr=0\.30r\{=\}0\.30\.

#### 6\.4\.2KV\-cache compression results

These methods compress the attention cache instead of selecting turns, so this analysis asks whether the probe\-dependent pattern also appears in a different method family\.

We test ChunkKV\([Liu et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib28)\), SnapKV\([Li et al\., 2024a](https://arxiv.org/html/2610.02736#bib.bib23)\), PyramidKV\([Cai et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib4)\), and StreamingLLM\([Xiao et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib44)\)through the kvpress implementation library111[https://github\.com/NVIDIA/kvpress](https://github.com/NVIDIA/kvpress)at seed4242\. All four have P2 results atr=0\.30r\{=\}0\.30\. P3 is reported for ChunkKV\. ChunkKV useschunk\_length=20=20, drop ratio0\.70\.7, window6464, kernel55, and context\-only compression\. Its P3 reader uses eager attention,128128output tokens, and a7,1687\{,\}168\-token input limit\.

Table 15:Strict accuracy for KV\-cache methods atr=0\.30r\{=\}0\.30, seed4242\. The P3 rows use the segment\-local scorer\. Higher is better\. Bold denotes the best KV\-cache method in each row with more than one entry\. The last column repeats the best turn selector on the same seed; MMR/random denotes a tie between embedding MMR and random selection\. A dash marks a condition we did not evaluate\.Atr=0\.30r\{=\}0\.30, SnapKV, PyramidKV, and StreamingLLM produce a reader error on00to55of the reported P2 examples per dataset, and those rows are scored as wrong\. On MultiWOZ P2 they score0\.6890\.689,0\.6830\.683, and0\.5840\.584, compared with0\.3290\.329for the best turn selector on the same seed\. SnapKV and PyramidKV differ by at most0\.0240\.024across the two P2 settings\.

The MultiWOZ P2 sensitivity sweep atr=0\.30r\{=\}0\.30compareschunk\_length∈\{10,20,40\}\\in\\\{10,20,40\\\}and yields strict scores of0\.0870\.087,0\.0870\.087, and0\.0500\.050\. A chunk length of4040therefore costs about four points\. Table[15](https://arxiv.org/html/2610.02736#S6.T15)reports the sweep run of thechunk\_length=20=20setting\.

#### 6\.4\.3ChunkKV chunk length

ChunkKV prunes the cache in contiguous chunks, so the chunk length sets how coarse the pruning is\. Table[16](https://arxiv.org/html/2610.02736#S6.T16)varies it on the MultiWOZ P2 set atr=0\.30r\{=\}0\.30\(n=161n\{=\}161\)\. Chunk lengths of1010and2020give the same strict score, and4040lowers it to0\.0500\.050\. The drop coincides with a higher malformed\-output rate: it rises from0\.1240\.124at chunk length1010to0\.1990\.199at2020and0\.4780\.478at4040, while the abstention rate stays near0\.320\.32\. The main comparison uses chunk length2020\.

Table 16:ChunkKV chunk length on the MultiWOZ P2 set atr=0\.30r\{=\}0\.30\(n=161n\{=\}161, seed4242\)\.

### 6\.5Another reader, corpus, and language

The main results use one reader on two English corpora\. This section repeats the current\-value and joint probes with a second reader, an English long\-term memory corpus, and a Chinese corpus, to see whether the same methods lead\. The reader comparison uses three sampled probe sets\. For LongMemEval\-KU and RiSAWOZ, we evaluate the complete eligible sets and vary the random turn selector over seeds\{42,43,44,45,46\}\\\{42,43,44,45,46\\\}\. Deterministic selectors are evaluated once\. Confidence intervals use10,00010\{,\}000dialogue\-level bootstrap samples\.

##### Cross\-reader \(Mistral\-7B\-Instruct\([Jiang et al\., 2023](https://arxiv.org/html/2610.02736#bib.bib18)\)\)

With Mistral atr=0\.30r\{=\}0\.30, embedding MMR leads SGD P2 and random selection leads MultiWOZ P2\. On P3, uniform stride leads both SGD thresholds and the LLMLingua\-2 leads both MultiWOZ thresholds\. The strongest compressed P3 mean remains6\.36\.3to13\.013\.0points below full context \(Table[18](https://arxiv.org/html/2610.02736#S6.T18)\)\.

##### Cross\-corpus \(LongMemEval knowledge updates, or LongMemEval\-KU\([Wu et al\., 2025](https://arxiv.org/html/2610.02736#bib.bib43)\)\)

All7272eligible questions are evaluated atr=0\.30r\{=\}0\.30\. Full context scores0\.4170\.417on P2\-strict, with a95%95\\%bootstrap interval of\[0\.306,0\.528\]\[0\.306,0\.528\]\. Random selection averages0\.2280\.228over the five selector seeds; earliest turns scores0\.0970\.097\.

##### Cross\-language \(RiSAWOZ Chinese\([Quan et al\., 2020](https://arxiv.org/html/2610.02736#bib.bib34)\)\)

Across all207207eligible dialogues, full context scores0\.2320\.232on P2\-strict\. Recency scores0\.1550\.155, random selection averages0\.1500\.150over five selector seeds, and earliest turns scores0\.0290\.029\.

#### 6\.5\.1Replication with Mistral\-7B\-Instruct

This replication asks whether the probe\-dependent rankings and the P3 gap to full context depend on the particular reader model\.

The Mistral\-7B\-Instruct replication uses the same questions, scorers, prompts, and turn\-selection methods as the Llama P2/P3 evaluation\. Tables[17](https://arxiv.org/html/2610.02736#S6.T17)–[18](https://arxiv.org/html/2610.02736#S6.T18)report means and standard deviations over seeds\{42,43,44\}\\\{42,43,44\\\}\. P3 uses200200examples per seed\. After the P2 filters, the three samples contain\{128,125,132\}\\\{128,125,132\\\}SGD examples and\{161,168,168\}\\\{161,168,168\\\}MultiWOZ examples\. P2 uses greedy decoding with scaled dot\-product attention \(SDPA\); P3 uses greedy decoding with eager attention\. Both allow128128output tokens and7,1687\{,\}168input tokens\. Not every Mistral condition records a model revision, so these comparisons describe the recorded pipelines rather than an isolated selector\-only intervention\.

Table 17:Mistral P2\-strict accuracy, mean±\\pmstandard deviation over three seeds\. Italic denotes full context; bold denotes the best compressed method in each column\.Table 18:Mistral P3 accuracy atr=0\.30r\{=\}0\.30, mean±\\pmstandard deviation over three seeds \(n=200n\{=\}200per seed\)\. Italic denotes full context; bold denotes the best compressed method in each column\.Embedding MMR leads SGD P2 with both readers, and random selection leads MultiWOZ P2 at both budgets\. The deletion experiment uses the Llama reader\.

#### 6\.5\.2Replication on LongMemEval\-KU

This replication asks whether current\-value recovery behaves the same way outside task\-oriented dialogue, where the answer is free\-form text rather than a slot value\.

LongMemEval\-KU provides7272eligible free\-form memory questions\. We evaluate the complete set atr=0\.30r\{=\}0\.30with the Llama reader\. Its answers can contain explicit alternatives such as “X \(or Y\),” so this experiment uses a corpus\-specific scorer\. Strict accepts an explicit answer variant after normalization, a complete variant at word boundaries in a longer prediction, or a numeric\-only prediction with the same complete numeric signature\. Loose also accepts at least half of the reference content tokens\. Numeric JSON scalars are accepted\. The table reports strict accuracy\. Random selection is averaged over selector seeds\{42,43,44,45,46\}\\\{42,43,44,45,46\\\}; deterministic selectors are evaluated once\. Intervals use10,00010\{,\}000dialogue\-level bootstrap samples\.

Table 19:LongMemEval\-KU P2\-strict accuracy atr=0\.30r\{=\}0\.30\(n=72n\{=\}72\)\. Random selection averages five selector seeds\. The reader uses a 32K input window and removes tokens from the beginning if a prompt exceeds it\. Italic denotes full context; bold denotes the best compressed method in each column\.In Table[19](https://arxiv.org/html/2610.02736#S6.T19)the uncompressed, window\-limited baseline has the highest mean, followed by recency\. The random\-selector means range from0\.2080\.208to0\.2500\.250\(standard deviation0\.0160\.016\), placing its five\-seed average between uniform stride and earliest turns\.

#### 6\.5\.3Replication on Chinese RiSAWOZ

This replication asks whether current\-value recovery behaves the same way in another language\.

We evaluate current\-value recovery on all207207eligible RiSAWOZ dialogues atr=0\.30r\{=\}0\.30\. P2 uses normalized exact match\. Random selection is averaged over selector seeds\{42,43,44,45,46\}\\\{42,43,44,45,46\\\}; deterministic selectors are evaluated once\. Intervals use10,00010\{,\}000dialogue\-level bootstrap samples\.

Table 20:RiSAWOZ P2\-strict atr=0\.30r\{=\}0\.30\(n=207n\{=\}207\)\. Random selection averages five selector seeds\. Italic denotes full context; bold denotes the best compressed method in each column\.Table[20](https://arxiv.org/html/2610.02736#S6.T20)gives recency the highest compressed\-method mean; random selection leads on two of the five selector seeds\.

### 6\.6Free\-form questions

This experiment asks whether the method rankings hold when the question is free\-form instead of templated\.

FMTS \(free\-form multi\-turn synthesis\) evaluates free\-form questions alongside the templated P1–P3 probes\. Its questions ask, for example, which budget value the user finally chose and when it changed\. GPT\-5\.4\-mini reads each compressed context and returns a one\- or two\-sentence answer\. A deterministic case\-insensitive matcher scores the answer\. It accepts substring containment in either direction between the prediction and the reference\. The scorer carries a second rule for references that contain the replaced value, and no item in this release meets that condition, so every reported score comes from containment alone\. Containment credits partial answers, so this score measures reference matching rather than answer completeness\. We call it*free\-form answer accuracy*\.

We evaluate100100synthetic questions that require resolving an indirect reference, plus200200free\-form questions each derived from MultiWOZ and SGD\. All use seed4242andr∈\{0\.10,0\.30\}r\\in\\\{0\.10,0\.30\\\}\. The reader comparison covers full context and six turn selectors from the main setup\. LLMLingua\-2 appears only in the diagnostics below, which score the compressed text itself and need no reader\. The reader tables therefore report six selectors and the diagnostics report seven\. FMTS usesk=max⁡\(1,⌊nturns​r⌋\)k=\\max\(1,\\lfloor n\_\{\\mathrm\{turns\}\}r\\rfloor\)retained turns\.

Each FMTS dialogue carries a labelled transition type\. The100100synthetic questions are all value replacements\. The MultiWOZ\-derived set is81\.0%81\.0\\%constraint addition and19\.0%19\.0\\%value replacement\. The SGD\-derived set is52\.5%52\.5\\%constraint addition and47\.5%47\.5\\%value replacement\. No dialogue in the released set removes a constraint or switches to a new goal\.

Table 21:Free\-form answer accuracy with the GPT\-5\.4\-mini reader \(seed4242\)\. Italic denotes full context; bold denotes the best compressed method among the six evaluated selectors in each column\. LLMLingua\-2 appears only in Table[22](https://arxiv.org/html/2610.02736#S6.T22)\. Full\-context answers are generated and scored separately for each budget, producing differences of at most0\.020\.02\.In Table[21](https://arxiv.org/html/2610.02736#S6.T21), among the six evaluated selectors, random selection leads syntheticr=0\.10r\{=\}0\.10and the H2O proxy leads syntheticr=0\.30r\{=\}0\.30\. Embedding MMR leads MultiWOZ\-derivedr=0\.10r\{=\}0\.10and both SGD\-derived budgets; uniform stride leads MultiWOZ\-derivedr=0\.30r\{=\}0\.30\. These comparisons describe only the retained reader conditions\. The experiment also changes the reader, so it does not isolate question format from reader effects\.

Table[22](https://arxiv.org/html/2610.02736#S6.T22)reports two diagnostics computed directly from the compressed contexts, independently of reader\-answer generation\.*Lexical answer retention*accepts case\-insensitive reference containment or overlap with more than half of the reference token set\. It is a heuristic for retained answer content, not an exact\-span test\.*Boundary\-turn recall*is the fraction of annotated change\-boundary turns retained\. Each entry reports these two values in that order\. LLMLingua\-2 has the highest lexical retention on the synthetic set; embedding MMR has the highest on both corpus\-derived sets\. On the synthetic set, embedding MMR has higher boundary\-turn recall but lower lexical retention than LLMLingua\-2\. The two diagnostics therefore capture different properties of the retained context\.

Table 22:FMTS diagnostics on the compressed contexts atr=0\.30r\{=\}0\.30: lexical answer\-retention heuristic / boundary\-turn recall\. Italic denotes full context; bold denotes the highest compressed lexical\-retention value\. These diagnostics do not use generated reader answers\.
### 6\.7Cross\-seed results

This section asks whether the probe disagreement of Section[6\.1](https://arxiv.org/html/2610.02736#S6.SS1)survives a change of scoring rule, a change of budget, or a change of seed\. If it does not, the disagreement would be a property of how we score rather than of what each probe asks for\.

Table[23](https://arxiv.org/html/2610.02736#S6.T23)scores the same P1 answers three ways\. The same two groups of methods separate under all three rules, so the P1 ranking does not depend on how an answer is matched\. Table[24](https://arxiv.org/html/2610.02736#S6.T24)asks the other question at two budgets\. The P1 leaders do not lead here, and the tighter budget lowers every compressed score without changing the leader\. The probe, not the scoring rule and not the budget, decides the ranking\. Table[25](https://arxiv.org/html/2610.02736#S6.T25)gives the same variation for the joint P3 probe\. Uniform stride leads all four columns, and in each one its margin over the next method is larger than either method’s seed standard deviation\. ChunkKV is reported for its seed\-4242run\.

Seed variation is small next to these differences\. The largest between\-method gaps exceed the seed standard deviations, although smaller pairwise differences do not: the three leading MultiWOZ P1 means fall within0\.0040\.004of each other, inside one seed standard deviation\.

Table 23:P1 atr=0\.30r\{=\}0\.30under three scoring rules, mean±\\pmsample standard deviation over seeds\{42,43,44\}\\\{42,43,44\\\}\(200200examples per dataset and seed\)\. Italic denotes full context; bold denotes the best compressed method in each column\.Table 24:P2\-strict accuracy, mean±\\pmstandard deviation over seeds\{42,43,44\}\\\{42,43,44\\\}\. The three samples contain\{128,125,132\}\\\{128,125,132\\\}SGD and\{161,168,168\}\\\{161,168,168\\\}MultiWOZ examples\. Italic denotes full context; bold denotes the best compressed method in each column\.Table 25:P3 accuracy atr=0\.30r\{=\}0\.30under the segment\-local scorer, mean±\\pmstandard deviation over seeds\{42,43,44\}\\\{42,43,44\\\}\(n=200n\{=\}200per seed\)\. Higher is better\. ChunkKV uses seed4242\. Italic denotes full context; bold denotes the best turn selector\.

## 7Conclusion

TPBench separates dialogue compression into three complementary targets: recovering the user’s initial goal, recovering the current value of a revised slot, and recovering both after a late update\. The experiments show that these targets should not be collapsed into a single score\. Compression methods rank differently on initial\-goal and current\-value recovery, and averaging the probes makes the preferred method depend on their relative weights\. On the main Llama P3 evaluation atr=0\.30r\{=\}0\.30, the strongest compressed turn selector remains28\.528\.5–36\.336\.3percentage points below full context across the two datasets and thresholds\. At wider turn budgets the current\-value gap narrows to3\.83\.8points on MultiWOZ, and we did not run the joint probe at those budgets\.

The paired deletion experiments clarify what the current\-value probe measures\. Removing the annotated update turn lowers P2 accuracy, while removing a matched irrelevant turn does not\. The final value is often repeated elsewhere in the dialogue, so removing the update turn alone leaves some usable evidence\. Removing the update turn together with all answer\-bearing turns causes a much larger drop\. These results support the role of annotated update turns and repeated answer\-bearing turns in current\-value recovery\.

Cache\-level compression behaves differently from turn selection on the current\-value probe\. On MultiWOZ, SnapKV and PyramidKV score0\.6890\.689and0\.6830\.683against0\.3290\.329for the best turn selector on the same seed\. On SGD, PyramidKV scores0\.4920\.492, compared with0\.2970\.297for the tied best turn selectors, embedding MMR and random selection\. These comparisons use the same nominal retained fraction, with different physical budget units\. SnapKV, PyramidKV, and StreamingLLM are evaluated on P2 only, so their joint\-probe behavior is unknown\. ChunkKV is also evaluated on P3\.

Compressor evaluation should report initial\-goal, current\-value, and joint recovery separately\. Different selectors favor different targets and datasets\. Uniform stride has the highest mean on the main Llama P3 evaluation among the seven turn selectors\. With Mistral, uniform stride leads P3 on SGD and the LLMLingua\-2 leads on MultiWOZ at both thresholds; the best compressed means remain6\.36\.3–13\.013\.0points below full context\. TPBench provides a reproducible way to expose these trade\-offs and a target for compression methods that explicitly preserve updates and coverage across dialogue regions\.

The deletion results show that update turns contribute beyond matched irrelevant context, while repeated answer\-bearing turns also support recovery\. Removing this evidence increases superseded\-value replies and rarely produces explicit abstention\. The experiments do not independently isolate lexical value retention from the context that establishes which value is current\.

## 8Limitations

TPBench is built from English task\-oriented dialogue with slot\-level state annotations\. P2 and P3 inherit those annotations, and the deletion experiment identifies answer\-bearing turns by normalized phrase matching\. Annotation errors and semantic equivalents outside this matcher affect the evidence sets\. Lexical scoring rejects valid paraphrases, and the AI\-assisted semantic evaluation that complements it is assessed against external human ratings on factual QA\([Bulian et al\., 2022](https://arxiv.org/html/2610.02736#bib.bib3)\)\. Direct human agreement on TPBench is unmeasured, and the bootstrap intervals condition on the saved judgments\.

The main evaluation reads with Llama\-3\.1\-8B\-Instruct for P1–P3 and Mistral\-7B\-Instruct for P2/P3\. LongMemEval\-KU and Chinese RiSAWOZ provide smaller current\-value replications\. The wider\-budget experiments cover four turn selectors, and the KV\-cache experiments use seed4242\. Every transition in the released material is a value replacement or a constraint addition, so constraint removal and goal switching are uncovered\. The results do not establish generality to open\-ended dialogue, multi\-party conversation, or goals that carry no slot annotation\.

Derived dialogue content inherits the source datasets’ privacy and licensing conditions\. LongMemEval\-KU includes personal context in user statements\.

## References

- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172),[Link](https://aclanthology.org/2024.acl-long.172/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Budzianowskiet al\.\(2018\)P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. GašićMultiWOZ \- a large\-scale multi\-domain Wizard\-of\-Oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 5016–5026\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1547),[Link](https://aclanthology.org/D18-1547/)Cited by:[§1](https://arxiv.org/html/2610.02736#S1.p8.1),[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Bulianet al\.\(2022\)J\. Bulian, C\. Buck, W\. Gajewski, B\. Börschinger, and T\. SchusterTomayto, tomahto\. beyond token\-level answer equivalence for question answering evaluation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 291–305\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.20/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.20)Cited by:[§6\.1\.1](https://arxiv.org/html/2610.02736#S6.SS1.SSS1.Px1.p1.1),[§8](https://arxiv.org/html/2610.02736#S8.p1.1)\.
- Caiet al\.\(2025\)Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, B\. Chang, J\. Hu, and W\. XiaoPyramidKV: dynamic KV cache compression based on pyramidal information funneling\.InConference on Language Modeling \(COLM\),Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1),[§6\.4\.2](https://arxiv.org/html/2610.02736#S6.SS4.SSS2.p2.1)\.
- Carbonell and Goldstein \(1998\)J\. Carbonell and J\. GoldsteinThe use of MMR, diversity\-based reranking for reordering documents and producing summaries\.InProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’98,New York, NY, USA,pp\. 335–336\.External Links:[Document](https://dx.doi.org/10.1145/290941.291025),ISBN 1581130155,[Link](https://doi.org/10.1145/290941.291025)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.02736#S5.SS0.SSS0.Px1.p2.1)\.
- Chenet al\.\(2024\)J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. LiuM3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2318–2335\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.137),[Link](https://aclanthology.org/2024.findings-acl.137/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Clark and Brennan \(1991\)H\. H\. Clark and S\. E\. BrennanGrounding in communication\.InPerspectives on Socially Shared Cognition,L\. B\. Resnick, J\. M\. Levine, and S\. D\. Teasley \(Eds\.\),pp\. 127–149\.Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of NAACL\-HLT,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Deyet al\.\(2022\)S\. Dey, R\. Kummara, and M\. DesarkarTowards fair evaluation of dialogue state tracking by flexible incorporation of turn\-level performances\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 318–324\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-short.35),[Link](https://aclanthology.org/2022.acl-short.35/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Fenget al\.\(2025\)Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. ZhouAda\-KV: optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tcisuhGsQZ)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.
- Geet al\.\(2024\)T\. Ge, H\. Jing, L\. Wang, X\. Wang, S\. Chen, and F\. WeiIn\-context autoencoder for context compression in a large language model\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=uREj4ZuGJE)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori A\. Dubeyet al\.The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Appendix A](https://arxiv.org/html/2610.02736#A1.p2.1),[§5](https://arxiv.org/html/2610.02736#S5.SS0.SSS0.Px4.p1.1)\.
- Hearst \(1997\)M\. A\. HearstTextTiling: segmenting text into multi\-paragraph subtopic passages\.Comput\. Linguist\.23\(1\),pp\. 33–64\.External Links:ISSN 0891\-2017Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px2.p1.1)\.
- Hendersonet al\.\(2014\)M\. Henderson, B\. Thomson, and J\. D\. WilliamsThe second dialog state tracking challenge\.InProceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue \(SIGDIAL\),K\. Georgila, M\. Stone, H\. Hastie, and A\. Nenkova \(Eds\.\),Philadelphia, PA, U\.S\.A\.,pp\. 263–272\.External Links:[Document](https://dx.doi.org/10.3115/v1/W14-4337),[Link](https://aclanthology.org/W14-4337/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, Y\. Chuang, C\. Li, Z\. Wang, L\. Le, A\. Kumar, J\. Glass, A\. Ratner, C\. Lee, R\. Krishna, and T\. PfisterFound in the middle: calibrating positional attention bias improves long context utilization\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14982–14995\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.890),[Link](https://aclanthology.org/2024.findings-acl.890/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px2.p1.1)\.
- Hudeček and Dusek \(2023\)V\. Hudeček and O\. DusekAre large language models all you need for task\-oriented dialogue?\.InProceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue,S\. Stoyanchev, S\. Joty, D\. Schlangen, O\. Dusek, C\. Kennington, and M\. Alikhani \(Eds\.\),Prague, Czechia,pp\. 216–228\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.sigdial-1.21),[Link](https://aclanthology.org/2023.sigdial-1.21/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Janget al\.\(2024\)J\. Jang, T\. Kim, and H\. KimMixed\-session conversation with egocentric memory\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 11786–11815\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.689),[Link](https://aclanthology.org/2024.findings-emnlp.689/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§6\.5](https://arxiv.org/html/2610.02736#S6.SS5.SSS0.Px1)\.
- Jianget al\.\(2024\)H\. Jiang, Q\. Wu, X\. Luo, D\. Li, C\. Lin, Y\. Yang, and L\. QiuLongLLMLingua: accelerating and enhancing LLMs in long context scenarios via prompt compression\.InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models,External Links:[Link](https://openreview.net/forum?id=9YvfRrpmyw)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Joet al\.\(2026\)D\. Jo, J\. Song, Y\. Kim, and J\. KimFastKV: decoupling of context reduction and KV cache compression for prefill\-decoding acceleration\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 32167–32186\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1610/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1610),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.
- Larsson and Traum \(2000\)S\. Larsson and D\. R\. TraumInformation state and dialogue management in the TRINDI dialogue move engine toolkit\.Nat\. Lang\. Eng\.6\(3–4\),pp\. 323–340\.External Links:[Document](https://dx.doi.org/10.1017/S1351324900002539),ISSN 1351\-3249,[Link](https://doi.org/10.1017/S1351324900002539)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2610.02736#S3.p2.1)\.
- Liet al\.\(2024a\)Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. ChenSnapKV: LLM knows what you are looking for before generation\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=poE54GOq2l)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1),[§6\.4\.2](https://arxiv.org/html/2610.02736#S6.SS4.SSS2.p2.1)\.
- Liet al\.\(2024b\)Z\. Li, Y\. Liu, Y\. Su, and N\. CollierPrompt compression for large language models: a survey\.External Links:2410\.12388,[Link](https://arxiv.org/abs/2410.12388)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024a\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638),[Link](https://aclanthology.org/2024.tacl-1.9/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024b\)X\. Liu, P\. Dong, X\. Hu, and X\. ChuLongGenBench: long\-context generation benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 865–883\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.48),[Link](https://aclanthology.org/2024.findings-emnlp.48/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)X\. Liu, Z\. Tang, P\. Dong, Z\. Li, Liuyue, B\. Li, X\. Hu, and X\. ChuChunkKV: semantic\-preserving KV cache compression for efficient long\-context LLM inference\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=20JDhbJqn3)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2610.02736#S5.SS0.SSS0.Px2.p1.1),[§6\.4\.2](https://arxiv.org/html/2610.02736#S6.SS4.SSS2.p2.1)\.
- Liuet al\.\(2023\)Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. ShrivastavaScissorhands: exploiting the persistence of importance hypothesis for LLM KV cache compression at test time\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=JZfg6wGi6g)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.
- Liuet al\.\(2024c\)Z\. Liu, J\. Yuan, H\. Jin, S\. \(\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. HuKIVI: a tuning\-free asymmetric 2bit quantization for KV cache\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.
- Nagleet al\.\(2024\)A\. Nagle, A\. Girish, M\. Bondaschi, M\. Gastpar, A\. V\. Makkuva, and H\. KimFundamental limits of prompt compression: a rate\-distortion framework for black\-box language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=TeBKVfhP2M)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Panet al\.\(2024\)Z\. Pan, Q\. Wu, H\. Jiang, M\. Xia, X\. Luo, J\. Zhang, Q\. Lin, V\. Rühle, Y\. Yang, C\. Lin, H\. V\. Zhao, L\. Qiu, and D\. ZhangLLMLingua\-2: data distillation for efficient and faithful task\-agnostic prompt compression\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 963–981\.External Links:[Link](https://aclanthology.org/2024.findings-acl.57/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.57)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.02736#S5.SS0.SSS0.Px1.p3.1)\.
- Parket al\.\(2025\)J\. Park, D\. Jones, M\. Morse, R\. Goel, M\. Lee, and C\. LottKeyDiff: key similarity\-based KV cache eviction for long\-context LLM inference in resource\-constrained environments\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Document](https://dx.doi.org/10.52202/085713-0210)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.
- Parket al\.\(2023\)S\. Park, J\. Seo, and J\. LeeUnsupervised dialogue topic segmentation in hyperdimensional space\.InINTERSPEECH 2023,interspeech\_2023,pp\. 730–734\.External Links:[Document](https://dx.doi.org/10.21437/interspeech.2023-1859),[Link](http://dx.doi.org/10.21437/Interspeech.2023-1859)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px2.p1.1)\.
- Qiuet al\.\(2024\)Z\. Qiu, J\. Li, S\. Huang, X\. Jiao, W\. Zhong, and I\. KingCLongEval: a Chinese benchmark for evaluating long\-context large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 3985–4004\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.230),[Link](https://aclanthology.org/2024.findings-emnlp.230/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Quanet al\.\(2020\)J\. Quan, S\. Zhang, Q\. Cao, Z\. Li, and D\. XiongRiSAWOZ: a large\-scale multi\-domain Wizard\-of\-Oz dataset with rich semantic annotations for task\-oriented dialogue modeling\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 930–940\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.67),[Link](https://aclanthology.org/2020.emnlp-main.67/)Cited by:[§6\.5](https://arxiv.org/html/2610.02736#S6.SS5.SSS0.Px3)\.
- Rastogiet al\.\(2020\)A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. KhaitanTowards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.Proceedings of the AAAI Conference on Artificial Intelligence34,pp\. 8689–8696\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6394)Cited by:[§1](https://arxiv.org/html/2610.02736#S1.p8.1),[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 3982–3992\.External Links:[Link](https://aclanthology.org/D19-1410/),[Document](https://dx.doi.org/10.18653/v1/D19-1410)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Robertson and Zaragoza \(2009\)S\. E\. Robertson and H\. ZaragozaThe probabilistic relevance framework: BM25 and beyond\.Found\. Trends Inf\. Retr\.3,pp\. 333–389\.External Links:[Link](https://api.semanticscholar.org/CorpusID:207178704)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p1.1)\.
- Smithet al\.\(2020\)E\. M\. Smith, M\. Williamson, K\. Shuster, J\. Weston, and Y\. BoureauCan you put it all together: evaluating conversational agents’ ability to blend skills\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 2021–2030\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.183),[Link](https://aclanthology.org/2020.acl-main.183/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Stolckeet al\.\(2000\)A\. Stolcke, K\. Ries, N\. Coccaro, E\. Shriberg, R\. Bates, D\. Jurafsky, P\. Taylor, R\. Martin, C\. Van Ess\-Dykema, and M\. MeteerDialogue act modeling for automatic tagging and recognition of conversational speech\.Computational Linguistics26\(3\),pp\. 339–374\.External Links:[Link](https://aclanthology.org/J00-3003/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px2.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 6000–6010\.External Links:ISBN 9781510860964Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.
- Williamset al\.\(2016\)J\. D\. Williams, A\. Raux, and M\. HendersonThe dialog state tracking challenge series: a review\.Dialogue & Discourse7\(3\),pp\. 4–33\.Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Wuet al\.\(2019\)C\. Wu, A\. Madotto, E\. Hosseini\-Asl, C\. Xiong, R\. Socher, and P\. FungTransferable multi\-domain state generator for task\-oriented dialogue systems\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 808–819\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1078),[Link](https://aclanthology.org/P19-1078/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Wuet al\.\(2025\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongMemEval: benchmarking chat assistants on long\-term interactive memory\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=pZiyCaVuti)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1),[§6\.5](https://arxiv.org/html/2610.02736#S6.SS5.SSS0.Px2)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1),[§6\.4\.2](https://arxiv.org/html/2610.02736#S6.SS4.SSS2.p2.1)\.
- Xuet al\.\(2022a\)J\. Xu, A\. Szlam, and J\. WestonBeyond goldfish memory: long\-term open\-domain conversation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 5180–5197\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.356),[Link](https://aclanthology.org/2022.acl-long.356/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Xuet al\.\(2022b\)X\. Xu, Z\. Gou, W\. Wu, Z\. Niu, H\. Wu, H\. Wang, and S\. WangLong time no see\! open\-domain conversation with long\-term persona memory\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 2639–2650\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.207),[Link](https://aclanthology.org/2022.findings-acl.207/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Zackset al\.\(2007\)J\. M\. Zacks, N\. K\. Speer, K\. M\. Swallow, T\. S\. Braver, and J\. R\. ReynoldsEvent perception: a mind/brain perspective\.Psychological Bulletin133\(2\),pp\. 273–293\.External Links:[Document](https://dx.doi.org/10.1037/0033-2909.133.2.273)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px2.p1.1)\.
- Zanget al\.\(2020\)X\. Zang, A\. Rastogi, S\. Sunkara, R\. Gupta, J\. Zhang, and J\. ChenMultiWOZ 2\.2: a dialogue dataset with additional annotation corrections and state tracking baselines\.InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI,pp\. 109–117\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.nlp4convai-1.13),[Link](https://aclanthology.org/2020.nlp4convai-1.13)Cited by:[§1](https://arxiv.org/html/2610.02736#S1.p8.1),[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2024\)X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. Hao, X\. Han, Z\. Thai, S\. Wang, Z\. Liu, and M\. SunInftyBench: extending long context evaluation beyond 100K tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15262–15277\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.814),[Link](https://aclanthology.org/2024.acl-long.814/)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Re, C\. Barrett, Z\. Wang, and B\. ChenH2O: heavy\-hitter oracle for efficient generative inference of large language models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RkRrPp7GKO)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2610.02736#S5.SS0.SSS0.Px1.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=uccHPGDlao)Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2025\)X\. Zhou, W\. Wang, M\. Zeng, J\. Guo, X\. Liu, L\. Shen, M\. Zhang, and L\. DingDynamicKV: task\-aware adaptive KV cache compression for long context LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8042–8057\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.426/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.426),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2610.02736#S2.SS0.SSS0.Px1.p2.1)\.

## Appendix AReader prompts

The artifact usesstory\_labelandintentfor the P1 initial goal,p3for paper P2, andP1\_lateorlatefor paper P3\. Method identifiersrandom\_seed42,first\_n, anduniform\_stridecorrespond to random selection, earliest turns, and uniform stride\. The seed in a main\-experiment file name selects the probe sample; its random selector is fixed and is namedrandom\_seed42\. In the LongMemEval\-KU and RiSAWOZ files the eligible set is fixed and the seed selects turns, which is why those files carryrandom\_seed42throughrandom\_seed46\.

Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.02736#bib.bib10)\)uses deterministic decoding \(temperature00, top\-pp1\.01\.0\)\. The main turn\-selection P1/P2 runs allow9696output tokens with eager attention\. Turn\-selection P3 and the deletion experiment allow128128output tokens with SDPA\. ChunkKV P3 uses eager attention and128128output tokens; the cache\-level settings are described in Section[6\.4\.2](https://arxiv.org/html/2610.02736#S6.SS4.SSS2)\. The system prompt is identical across probes and methods:

> You are reading a conversation that may be partial or compressed\. Answer ONLY using the provided context\. Reply with a single JSON object and no other text\. Do not add markdown fences\.

The user\-turn template differs per probe:

- •P1:What was the user’s initial goal at the beginning of this conversation? Reply with one short noun phrase\.
- •P2:After the most recent change visible in the context, what is the user’s CURRENT value for⟨\\langleslot natural\-language form⟩\\rangle? Reply with one short value\.
- •P3:In one short reply, state BOTH \(a\) the user’s task in this conversation AND \(b\) the final value the user decided on for⟨\\langleslot natural\-language form⟩\\rangle\. Format your answer as: "⟨\\langletask⟩\\rangle;⟨\\langleslot⟩\\rangle:⟨\\langlefinal value⟩\\rangle"\.

Each user prompt then requests the same JSON fields: a stringvalue, a verbatimsupportspan of at most3030words, and a Booleanabstain\. For P3,valuecontains the requested “task; slot: final value” string\.

## Appendix BToken retention and reading\-cost scaling

Letq=Lcompressed/Lfullq=L\_\{\\mathrm\{compressed\}\}/L\_\{\\mathrm\{full\}\}be the fraction of input tokens retained by a prompt\-side selector\. For a fixed reader architecture and precision, the input KV\-cache size scales asqq, and the dense prefill self\-attention term scales asq2q^\{2\}\. Atq=0\.30q=0\.30, the corresponding full\-to\-compressed ratios are approximately3\.33\.3and11\.111\.1\.

These are theoretical scaling factors for input KV memory and the prefill attention term, not measured end\-to\-end speedups\. The experimental budgetrrspecifies a turn fraction for selectors and a cache\-entry fraction for KV methods\. Turn lengths, prompt overhead, compression work, and decoding also affect serving cost; cache pruning can leave full\-context prefill work intact\.

## Appendix CP1 lexical\-scoring sensitivity and examples

P1\-strict requires at least two shared content tokens, whereas P1\-loose requires one\. TokenF1F\_\{1\}measures graded overlap\. Full\-context strict/loose scores are0\.487/0\.8500\.487/0\.850on SGD and0\.430/0\.8670\.430/0\.867on MultiWOZ \(Table[23](https://arxiv.org/html/2610.02736#S6.T23)\)\.

The scores capture wording rather than full semantic equivalence\. For example, the reference “Could you help me find a doctor” and answer “find a doctor” fail the strict lexical rule but pass the semantic criterion\. Conversely, “Can you find me some free attractions to visit in New York City” and “find free attractions” pass the lexical rule but fail full equivalence because the city constraint is omitted\. Both are actual seed\-4242full\-context answers under the initial\-goal question\. The two scoring views therefore distinguish lexical recovery from semantic completeness\.

## Appendix DReproducibility

##### ChunkKV runs

Two runs of the ChunkKVchunk\_length=20=20setting on MultiWOZ P2 atr=0\.30r\{=\}0\.30score0\.0870\.087and0\.0930\.093, differing on one of the161161examples\. The recorded run state does not identify the source of the difference\. Both runs are in the release; the main tables use the sweep run\.

##### Artifact

Probes, fixed compressed contexts, reader outputs, scorers and code are available at[https://github\.com/kentech\-sail/TPBench](https://github.com/kentech-sail/TPBench)\. The release records hashes for probes, scorers and compressor wrappers, reader revisions, selector seeds and bootstrap settings\. Appendix A maps paper and artifact names\. The Croissant manifest describes records and distributions and passes validation\.

##### Reader execution

Llama retains revision0e9e39f249a16976918f6564b8830bc894c89659\. P1 uses batch size88and a batch token budget of8,0008\{,\}000; P2/P3 are serial\. The release records two software environments: Python 3\.10\.12 with PyTorch 2\.5\.1\+cu121 and Transformers 5\.13\.1, and Python 3\.14\.4 with PyTorch 2\.13\.0\+cu130 and Transformers 5\.14\.1\. Each condition records which one produced it\. Mistral pins revisionc170c708c41dac9275d15a8fff4eca08d52bab71\. It retains bfloat16, greedy serial generation, P2 SDPA/P3 eager attention, and7,1687\{,\}168/128128input/output limits\. Batched diagnostics are archived and are not primary results\. Model and execution provenance are recorded per condition\.

##### Compute resources

The deterministic package checks run on CPU and load no reader model\. The reader experiments used 24 GB\-class GPUs and trained no model, with recorded estimates of1515–2323GPU\-hours for the Mistral turn\-selection runs and99–1212hours for the seed\-4242ChunkKV runs\. LongMemEval\-KU uses the largest input window \(32,76832\{,\}768tokens\) and truncates longer prompts from the beginning\.