基于状态空间编码的线性时间上下文压缩

# Linear-Time Context Compression with State-Space Encoding
Source: [https://arxiv.org/html/2610.33006](https://arxiv.org/html/2610.33006)
Andrey BarskyCarlos Boned RieraAffiliation:Tomás OckierDimosthenis KaratzasAffiliation:Computer Vision Center, UAB, Spain

###### Abstract

Transformer\-based large language models \(LLMs\) impose substantial computation and memory costs as context sizes increase, motivating methods that compress a context into a shorter representation before inference\. Existing soft\-prompt compression methods typically use the same Transformer backbone as used in the target model, where producing the compressed representation leads to significant latency overheads\. This restricts them to the offline setting where context is compressed once and then reused across many queries\. We propose theBoa Compressor, a plug\-and\-playonlinecontext compressor built on a Mamba state\-space model \(SSM\) that encodes arbitrary\-length context in a single linear\-time pass, producing a representation on which we condition a frozen target LLM\. The SSM is trained by distillation, matching the next\-token logit distribution from the LLM ingesting the full context \(as teacher\) to the same LLM ingesting the compressed representation \(as the student\)\. At equal compression ratios, Boa matches or surpasses the performance of existing state\-of\-the\-art offline compression baselines, while being an order of magnitude faster than Transformer\-based implementations, therebyreducingend\-to\-end latency rather than adding compression time overhead\. We demonstrate that Boa also scales well to longer contexts and to compression ratios as aggressive as 16×\\times\. By enabling effective compression that is cheap enough to run on every query at inference time, Boa becomes a practical and general tool for efficient long\-context inference\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.33006v1/Figures/Method.png)Figure 1:Overview of the Boa compressor\. A trainable Boa module encodes the input context using a Mamba 2 model, a temporal stride operation, and key\-value \(KV\) projections\. This compressed KV cache is injected into a frozen student Transformer*decoder*\. The Boa*encoder*is trained by minimising the KL divergence between the student’s output and that of a frozen teacher model\.Large language models are increasingly deployed on long inputs, including multi\-document question answering, retrieval\-augmented generation, agent trajectories, and long\-form reasoning\. However, the Transformer architecture that enables these models has scaling issues at long contexts: self\-attention costs grow quadratically with sequence length during pre\-fill, and the key\-value \(KV\) cache grows linearly during decoding\. With longer inputs, inference cost is increasingly dominated by these time and memory constraints\.

Context compression is an answer to this scaling problem: replacing a long context with a compact representation that preserves the information the model needs to answer\. Effective compression simultaneously reduces pre\-fill cost and cache size, filters noise from long inputs\([Jiang et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib15)\), and extends the effective context size that a model can handle\. The highest information per token can be retained using*soft\-prompt*compression methods, which learn to encode the context into a short sequence of continuous vectors that condition a frozen decoder\. Recent works\([Zhang et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib28);[Ge et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib8);[Li et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib18);[Liu et al\., 2026](https://arxiv.org/html/2610.33006#bib.bib20)\)have demonstrated that much of the performance of the model can be retained even at high compression ratios\.

However, existing methods share a limitation in their choice of compressor architecture\. In most cases, the compressor uses the same Transformer architecture as the target model: that is, the encoder is the LLM itself, or an adaptation of it, or otherwise another Transformer of comparable size\. In practice this restricts soft\-prompt compression to the offline regime, where a context is compressed once and reused across many queries\. This is useful in a range of circumstances, but for a context seen only once \(a common case at inference time\), the cost of compression is comparable to processing the original context itself\.

This limitation is not inherent to soft\-prompt compression methods, but only to the choice of compressor architecture\. State\-space models \(SSMs\) such as Mamba\([Gu and Dao, 2024](https://arxiv.org/html/2610.33006#bib.bib9)\)process a sequence in linear time with a fixed\-size recurrent state, while remaining competitive with Transformers on long\-sequence modelling\. This class of models makes for natural context compressors: they can read an arbitrarily long input cheaply and summarise it into a compact representation, offloading the heavy reasoning to a frozen Transformer decoder\. This makes compression cheap enough to run*online*, as a single\-pass step in the inference path itself, rather than only as*offline*preprocessing\.

We instantiate this idea in the Boa Compressor \(Figure[1](https://arxiv.org/html/2610.33006#S1.F1)\)\. A Mamba encoder reads the context and produces a short sequence of representations that are injected directly into a frozen LLM; the LLM is never modified or fine\-tuned\. Boa is trained by knowledge distillation, matching the frozen model’s own output distribution on the full context, so the compressed representation is optimised to directly reproduce the behaviour of the decoder\. On the MRQA benchmark suite\([Fisch et al\., 2019](https://arxiv.org/html/2610.33006#bib.bib7)\), Boa matches or surpasses state\-of\-the\-art Transformer\-based baselines at equal compression ratios, and degrades far more gracefully as the ratio increases, while cutting total long\-context latency by more than an order of magnitude\.

The remainder of the paper reviews related work in context compression and state\-space models \(Section[2](https://arxiv.org/html/2610.33006#S2)\), details the Boa architecture and training \(Section[3](https://arxiv.org/html/2610.33006#S3)\), reports quality and efficiency results \(Section[4](https://arxiv.org/html/2610.33006#S4)\), and discusses broader implications on compressor design \(Section[5](https://arxiv.org/html/2610.33006#S5)\)\.

## 2Related Work

### 2\.1Context Compression

The objective of compressing an LLM’s input context is to reduce the number of tokens it must process while generating output sufficiently similar to that attained without compression\. There are broadly two families of approach to this:*hard\-prompt*methods that delete or rewrite tokens in natural language, and*soft\-prompt*methods that replace tokens with continuous representations\.

Hard\-prompt compressionmethods keep the input in natural language and remove what is redundant\. Selective\-Context\([Li et al\., 2023](https://arxiv.org/html/2610.33006#bib.bib17)\)prunes low\-information spans using token\-level entropy or perplexity, and the LLMLingua family\([Jiang et al\., 2023](https://arxiv.org/html/2610.33006#bib.bib14);[Jiang et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib15)\)extends this to aggressive token deletion, showing that models can act on inputs even when fluency is broken\. These methods are model\-agnostic and require no training, but they are bounded by what can be expressed in surviving tokens and cannot compress below the granularity of the text itself\.

Soft\-prompt compressionmethods compress natural\-language tokens into a small number of continuous vectors, achieving higher density than hard prompts\. Gisting\([Mu et al\., 2023](https://arxiv.org/html/2610.33006#bib.bib21)\)trains reusable “gist” tokens, though it requires fine\-tuning the target model and handles only short prompts\. AutoCompressor\([Chevalier et al\., 2023](https://arxiv.org/html/2610.33006#bib.bib4)\)scales input length by recurrently propagating summary vectors across segments; however, it too requires fine\-tuning the target model and incurs heavy encoding costs due to performing multiple forward passes over the backbone for a single context\. ICAE\([Ge et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib8)\)removes the need to change the decoder: an LLM adapted with LoRA encodes a context of up to roughly 500 tokens into a handful of memory slots that condition the frozen model, reaching up to 8×\\timescompression with a two\-step training procedure: first pre\-trained with a combination of autoencoder and language modelling losses, then fine\-tuned on instruction data\. 500xCompressor\([Li et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib18)\)shows that placing the compressed representation in the KV space rather than the embedding space is able to achieve far higher effective compression ratios of up to 480×\\timesunder similar conditions\.

Soft prompts have also been specialised for retrieval\-augmented generation \(RAG\), where a passage is compressed once and reused, as in xRAG and COCOM\([Cheng et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib3);[Rau et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib23)\)\.

We compare against two state\-of\-the\-art methods that serve as the closest comparison baselines\.Activation Beacon\([Zhang et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib28)\)compresses the KV activations produced throughout the decoder by splitting the context into chunks, summarising each chunk with special beacon tokens whose activations are retained, and training via compression\-based autoregression at randomly sampled ratios\. While it preserves near\-uncompressed quality up to128​K128\\text\{K\}tokens and reduces cache memory, encoding is computationally expensive, requiring multiple forward passes over the decoder, and compression is structurally bounded by its sliding\-window size\.Semantic\-Anchor Compression \(SAC\)\([Liu et al\., 2026](https://arxiv.org/html/2610.33006#bib.bib20)\)removes the reconstruction\-based pretraining used by ICAE\-style methods, which it shows interferes with the language\-modelling features needed downstream; instead it selects existing context tokens as anchors and aggregates information into them through bidirectional attention\. SAC improves alignment between compressed and original representations and confirms that pretraining on a language\-modelling objective, rather than autoencoding, yields stronger downstream compression\.

Across this family of methods, there is an architectural assumption that the compressor is a Transformer \(often adapted from the target model itself\) and therefore that encoding any context carries Transformer\-order computational costs\. Boa keeps the soft\-prompt idea and the frozen decoder but replaces the encoder with a linear\-time SSM, so the compression step itself is computationally cheap\.

Context distillation\.Knowledge distillation \(KD;[Hinton et al\. 2015](https://arxiv.org/html/2610.33006#bib.bib10)\) has been widely applied for model compression, but has also been applied to conditioning LLMs without full context: by directly optimising per\-prompt soft tokens\([Wingate et al\., 2022](https://arxiv.org/html/2610.33006#bib.bib26)\)or internalising the context into the LLM’s own weights\([Snell et al\., 2022](https://arxiv.org/html/2610.33006#bib.bib24)\)\. Closer to our work is xRAG\([Cheng et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib3)\), which trains a lightweight projection over frozen retrieval embeddings by distilling from the full\-context model\. For Boa we adopt the distillation framework to encode arbitrary contexts with a trainable sequence encoder, as a natural objective for preserving fidelity between compressed and uncompressed contexts, finding it to work better than direct language modelling\.

Online vs\. offline compression\.When a context is to be reused across several queries, it is efficient to compress it once in advance, and the cost of compression is repaid over repeated access\. We call this theofflinesetting\. By contrast, in theonlinesetting, the context is seen for the first and last time at inference, so compression must pay for itself within a single query\. Transformer\-based methods \(most of those described above\) are typically constrained to offline use, as the overhead cost of single\-use compression adds latency rather than removing it\.

Existing work on online compression has been largely confined to the RAG setting: RECOMP\([Xu et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib27)\)summarises passages between retrieval and encoding, and REFRAG\([Lin et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib19)\)selectively expands query\-relevant retrievals based on pre\-indexed encodings\. LongLLMLingua\([Jiang et al\., 2023](https://arxiv.org/html/2610.33006#bib.bib14)\)is a hard\-prompt method that prunes tokens conditioned on the query, but is limited in performance to the granularity of the text being compressed\. These methods are effective specifically when context is highly redundant and independently chunked, with sparse relevance to the query\.

The online setting also offers an advantage that is unavailable offline: since the query is known at compression time, the compressor benefits from conditioning on it and retaining only the information in the context required to answer\. In practice, this advantage has not been fully exploited: existing online compression methods achieve high performance only under narrow assumptions, and offline compression methods are usually too slow to see single\-query benefits \(though we observe in Figure[2](https://arxiv.org/html/2610.33006#S4.F2)that SAC achieves moderate latency gains at longer context lengths\)\. Boa is designed for precisely this setting, achieving the high performance of Transformer\-based offline compressors, with a latency that makes it effective in the online regime\.

### 2\.2State\-Space Models for Compression

State\-space models \(SSMs\) process sequences through a latent recurrent state, enabling linear\-time inference and constant\-size memory\. Selective variants such as Mamba\([Gu and Dao, 2024](https://arxiv.org/html/2610.33006#bib.bib9)\)and Mamba\-2\([Dao and Gu, 2024](https://arxiv.org/html/2610.33006#bib.bib5)\)improve earlier structured SSMs by making state transitions input\-dependent\.

A known limitation of SSMs relative to attention\-based models is weaker associative recall and verbatim copying, due to the fixed size of the recurrent state\([Jelassi et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib13);[Arora et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib1);[Waleffe et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib25)\)\. This motivates our use of the SSM in a hybrid architecture: Boa does not require retrieval from the compressed state itself, but allows a Transformer full attention over the set of compressed state representations, which scales as a fraction of input length\.

RCC\([Huang et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib11)\)applies SSMs to context compression, but processes the input in independent chunks, and fine\-tunes both the encoder and decoder\. In contrast, Boa compresses in a single pass and remains plug\-and\-play by retaining a frozen LLM as the target decoder\.

SQuADNewsQATriviaQASearchQAHotpotQANaturalQAverageNormΔ\\DeltaMethodEMF1EMF1EMF1EMF1EMF1EMF1EMF1EMF1Full context \(ceiling\)87\.2993\.7057\.1772\.0676\.1282\.9281\.5387\.5068\.9583\.7670\.6082\.5273\.6183\.74100100No context \(floor\)5\.7012\.192\.044\.6321\.7024\.9115\.1817\.914\.079\.285\.9711\.849\.1113\.46004×\\timescompressionSAC64\.1979\.6344\.9262\.3964\.9272\.6980\.5785\.8861\.5177\.961\.5275\.2562\.9475\.6283\.4688\.45Beacon67\.0782\.5748\.0567\.2067\.5374\.8263\.8172\.6063\.4179\.4960\.2875\.0261\.6975\.2881\.5287\.96Boa \(ours\)79\.8189\.4352\.7568\.8376\.8782\.6681\.3387\.1763\.8580\.3066\.2079\.3270\.1481\.2994\.6296\.518×\\timescompressionSAC53\.1770\.2837\.7355\.7361\.9870\.0979\.4385\.0557\.6275\.1554\.2169\.9057\.3671\.0374\.8181\.92Beacon64\.6680\.6644\.2364\.2265\.3973\.0161\.1570\.0360\.4077\.5757\.0872\.5958\.8273\.0177\.0784\.73Boa \(ours\)78\.0188\.4052\.3068\.3876\.4882\.3481\.0586\.8462\.5379\.2063\.7477\.5869\.0280\.4692\.8895\.3316×\\timescompressionSAC34\.0151\.3326\.4543\.656\.0864\.8974\.2880\.944\.4862\.9640\.657\.1845\.9860\.1446\.2766\.42Beacon60\.5977\.0541\.5761\.2762\.6370\.6956\.8365\.5856\.8774\.0354\.3770\.1855\.4869\.8071\.8980\.17Boa \(ours\)78\.1688\.1651\.6467\.6476\.3982\.2181\.1886\.9561\.8478\.5663\.5577\.4568\.7980\.1692\.5394\.91Table 1:MRQA*in\-domain*results with Qwen3\-1\.7B, per dataset\. EM / token\-level F1; higher is better\.NormΔ\\Deltanormalises the average between the no\-context floor \(0\) and full\-context ceiling \(100\)\. Boa uses the Mamba\-2 \(1\.3B\) KV variant\.

## 3Method

### 3\.1Overview

Boa compresses a long context into a short sequence of continuous representations that condition the target frozen LLM\. The method consists of a trainable Mamba\-based encoder and a lightweight projection module, while the target Transformer decoder remains frozen\. Given an input sequence of lengthLL, the encoder processes it in a single linear\-time pass\. A strided subset of the resulting hidden states is then projected into the attention space of the decoder to produce the layer\-wise key \(KK\) and value \(VV\) representations\. The decoder attends to this compressed KV cache as contextual information and generates the output\. Figure[1](https://arxiv.org/html/2610.33006#S1.F1)illustrates the overall architecture\. For a compression ratior∈ℕr\\in\\mathbb\{N\}, we use a striderrand producem=⌈L/r⌉m=\\lceil L/r\\rceilcompressed representations\.

### 3\.2Mamba Compressor

The encoder is a pretrained Mamba2\-based model that processes the input in linear time\. Let the context bec=\(x1,…,xL\)c=\(x\_\{1\},\\dots,x\_\{L\}\)with token embeddings\(u1,…,uL\)\(u\_\{1\},\\dots,u\_\{L\}\)\. A selective state\-space layer maintains a hidden statehth\_\{t\}that evolves recurrently with input\-dependent parameters:

ht=A¯t​ht−1\+B¯t​ut,et=Ct​ht,h\_\{t\}=\\bar\{A\}\_\{t\}\\,h\_\{t\-1\}\+\\bar\{B\}\_\{t\}\\,u\_\{t\},\\qquad e\_\{t\}=C\_\{t\}\\,h\_\{t\},\(1\)where\(A¯t,B¯t,Ct\)\(\\bar\{A\}\_\{t\},\\bar\{B\}\_\{t\},C\_\{t\}\)are functions of the input, via Mamba’s selection mechanism, andete\_\{t\}is the encoder hidden state at positiontt\. Since the state is carried forward recurrently, eachete\_\{t\}summarises all tokens up to positionttin a single pass\.

Rather than emit one representation per token, Boa gathers hidden states on a fixed stride\. Anchored at the final token and stepping backward byrr, it selects the index setℐ=\{L,L−r,L−2​r,…\}\\mathcal\{I\}=\\\{\\,L,\\,L\-r,\\,L\-2r,\\,\\dots\\,\\\}with\|ℐ\|=m=⌈L/r⌉\|\\mathcal\{I\}\|=m=\\lceil L/r\\rceil, yielding the compressed sequenceE=\(ei\)i∈ℐE=\(e\_\{i\}\)\_\{i\\in\\mathcal\{I\}\}\. The last position is always the final token of the context, which is typically the most information\-dense position in a causal model\. Earlier positions are periodic checkpoints back through the context\.

For our experiments, we use Mamba\-2 pretrained on roughly300300B tokens\. This gives us a suitable pretrained base encoder to evaluate against prior methods\.

### 3\.3Conditioning the Frozen Decoder

The compressed representations can condition the Transformer decoder in two ways\. The most straightforward is to feed them as input embeddings, i\.e\. as soft prompts that enter at the first layer of the decoder\. The stronger method, which we adopt, is to inject them directly as per\-layer key and value vectors in the attention cache of the decoder\. For each selected stateeie\_\{i\}and decoder layerℓ∈\{1,…,N\}\\ell\\in\\\{1,\\dots,N\\\}, a learned projectionPℓP^\{\\ell\}produces a key/value pair:

\(kiℓ,viℓ\)=Pℓ​\(ei\),\(k\_\{i\}^\{\\ell\},\\,v\_\{i\}^\{\\ell\}\)=P^\{\\ell\}\(e\_\{i\}\),\(2\)and the collectionZ=\{\(kiℓ,viℓ\)\}i∈ℐℓZ=\\\{\(k\_\{i\}^\{\\ell\},v\_\{i\}^\{\\ell\}\)\\\}\_\{i\\in\\mathcal\{I\}\}^\{\\ell\}is assembled into a KV cache packed exactly as one the decoder would have produced for a real prefix of lengthmm, including the rotary position encoding of the model, so that from the perspective of the decoder the compressed representations occupy the firstmmpositions of its context\.

KV injection conditions the model at every layer rather than only in the input embedding, giving each compressed representation substantially more capacity to influence the network than a single soft token\. This mirrors the finding in 500xCompressor\([Li et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib18)\)that KV\-space compression outperforms embedding\-space compression at high ratios\. The key difference in our setting is that the encoder of 500xCompressor, and posterior methods, is the LLM itself, so its keys and values already live in the attention space of the decoder and can be extracted by slicing the model’s own cache\. Our encoder is a separate Mamba, with its own distinct representation space\. Therefore, the decoder keys and values cannot be obtained by directly slicing encoder activations and must instead be learned through a projection module, which serves as the interface between the two architectures\. This design is non\-trivial because injecting learned key\-value representations into every decoder layer modifies the attention computation throughout the network depth and may destabilise generation\. Motivated by prefix tuning and 500xCompressor, we nevertheless condition the frozen decoder directly in KV space\. Our results show that the decoder can effectively exploit keys and values produced by a heterogeneous encoder, achieving performance comparable to compression methods that reuse the target model’s architecture\.

BioASQDROPDuoRCRACERETextbookQAAverageNormΔ\\DeltaMethodEMF1EMF1EMF1EMF1EMF1EMF1EMF1EMF1Full context61\.3077\.3255\.4964\.0956\.6366\.5245\.5559\.3477\.8288\.1364\.8074\.1660\.2771\.59100100No context17\.3524\.0623\.0926\.381\.674\.582\.676\.039\.7415\.2624\.2831\.4813\.1317\.97004×\\timescompressionSAC55\.1268\.9545\.4454\.3948\.5758\.937\.9853\.3168\.4281\.1462\.2870\.6752\.9764\.5684\.5186\.89Beacon51\.9967\.9941\.1851\.1449\.7061\.3136\.0551\.4175\.2786\.7055\.4866\.1051\.6164\.1181\.6386\.05Boa \(ours\)54\.8569\.0542\.6551\.4253\.9064\.2036\.6550\.8361\.7078\.3259\.2168\.3151\.4963\.6981\.3785\.278×\\timescompressionSAC49\.6062\.1239\.8549\.3742\.7053\.0332\.0547\.3546\.5461\.4760\.6169\.2245\.2357\.0968\.1072\.96Beacon53\.6667\.1237\.1347\.9548\.2359\.7233\.6850\.0875\.2786\.2951\.2962\.2549\.8862\.2477\.9682\.56Boa \(ours\)57\.1871\.1840\.4550\.2252\.5062\.5234\.1248\.7755\.7074\.1558\.5568\.5749\.7562\.5777\.6883\.1816×\\timescompressionSAC45\.4756\.5531\.0040\.334\.2444\.1725\.3740\.2938\.2653\.4652\.9561\.9637\.8849\.4652\.5058\.73Beacon53\.7365\.2231\.1341\.4745\.5656\.4431\.1347\.0374\.3585\.6948\.5658\.9147\.4159\.1372\.7276\.76Boa \(ours\)54\.3968\.9143\.3152\.0252\.1062\.5634\.5748\.4853\.6071\.6258\.5567\.8449\.4261\.9176\.9881\.95Table 2:MRQA*out\-of\-domain*results with Qwen3\-1\.7B, per dataset\. EM / F1; higher is better\.NormΔ\\Deltanormalises the average between the no\-context floor \(0\) and full\-context ceiling \(100\)\. Boa uses the Mamba\-2 \(1\.3B\) KV variant\.
### 3\.4Training Objective and Procedure

To align the representations between the Mamba encoder and the frozen Transformer decoder, Boa employs a two\-stage training strategy that updates only the encoder and projection layer\.

Stage 1: Pretraining\.We pretrain the compressor on 10B tokens sampled from FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib22)\)using a standard next\-token cross\-entropy objective, with a max sequence length of 1k tokens, conditioning the frozen decoder on the compressed sequenceEE\. Causal language modelling provides a computationally lightweight base that transfers more effectively to compression than autoencoding\([Liu et al\., 2026](https://arxiv.org/html/2610.33006#bib.bib20)\)\. For our experiments in long\-context encoding, we similarly pretrain on 10B tokens but selectively sampling the longest sequences of this dataset, targeting up to 10k sequence length\.

Stage 2: Distillation Fine\-Tuning\.We fine\-tune the compressor using knowledge distillation from the uncompressed context teacher\([Hinton et al\., 2015](https://arxiv.org/html/2610.33006#bib.bib10)\)across 500k QA pairs from the MRQA training dataset\([Fisch et al\., 2019](https://arxiv.org/html/2610.33006#bib.bib7)\)for standard context compression, as well as dedicated long\-context datasets \(BookSum, GovReport, QASPER and LongAlpaca\)\([Kryściński et al\., 2022](https://arxiv.org/html/2610.33006#bib.bib16);[Huang et al\., 2021](https://arxiv.org/html/2610.33006#bib.bib12);[Dasigi et al\., 2021](https://arxiv.org/html/2610.33006#bib.bib6);[Chen et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib2)\)to evaluate long\-context capabilities, training over samples of 10k sequence length\. For each context, the compressor minimises the KL divergence over the answer token positionst∈\{1,…,\|y\|\}t\\in\\\{1,\\dots,\|y\|\\\}:

ℒ=τ2\|y\|∑t=1\|y\|KL\(pT\(⋅∣c,q,y<t\)∥pS\(⋅∣Z,q,y<t\)\),\\mathcal\{L\}=\\frac\{\\tau^\{2\}\}\{\|y\|\}\\sum\_\{t=1\}^\{\|y\|\}\\ KL\\\!\\left\(p\_\{\\mathrm\{T\}\}\(\\cdot\\mid c,q,y\_\{<t\}\)\\,\\\|\\,p\_\{\\mathrm\{S\}\}\(\\cdot\\mid Z,q,y\_\{<t\}\)\\right\),\(3\)
whereccis the uncompressed context,qqis the query,y<ty\_\{<t\}are the previously generated answer tokens,ZZis the KV projection of the compressed state,τ\\tauis the distillation temperature, and\|y\|\|y\|is the total number of tokens in the generated answer sequence\. Rather than forcing full token reconstruction, this objective directly aligns the compressed state with downstream generation by training the compressor to replicate full\-context teacher behaviour\([Cheng et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib3)\)\. All baselines share the same pretraining budget and data, with early stopping applied during fine\-tuning\. Further implementation and training details are provided in the Appendix\.

## 4Experiments

### 4\.1Setup

Benchmark\.We evaluate on MRQA, a suite of extractive question\-answering datasets \(divided into in\-domain and out\-of\-domain tasks, where out\-of\-domain tasks are seen only at validation time\)\. We report Exact Match \(EM\) and token\-level F1 as fidelity metrics, averaged across tasks, and additionally a normalised score NormΔ\\Deltathat places each configuration on a 0\-100 scale between a no\-context floor and a full\-context ceiling, in order to see the true delta of the compressed context relative to the uncompressed baseline\.

Models\.We use two frozen decoders, Qwen3\-1\.7B and Qwen3\-4B\. For the compressor, we utilise pretrained Mamba\-2 models at two scales: 1\.3B and 2\.7B\. We evaluate compression ratios of 4×\\times, 8×\\times, and 16×\\timeswith the KV\-injection conditioning of Section[3\.3](https://arxiv.org/html/2610.33006#S3.SS3)\. We selected these specific power\-of\-two ratios because they are structurally required by the Activation Beacon baseline; the other evaluated methods are flexible enough to accommodate them, allowing for a fair and direct comparison across all approaches\.

Reference points\.We report compression performance relative to an upper and lower bound with respect to our target LLM model: Qwen3 fine\-tuned using the complete context, which serves as the upper bound and is normalised to 100, and the same model evaluated without context, which serves as the lower bound and is normalised to 0\. Based on our benchmark results across the evaluated state\-of\-the\-art baselines, we selectActivation Beaconas our primary compression reference as it achieves the strongest overall performance \(different to the results reported in[Liu et al\. 2026](https://arxiv.org/html/2610.33006#bib.bib20)due to the use of compression ratios not supported by the Activation Beacon architecture\)\. We additionally compare againstSACto provide a comprehensive comparison with relevant methods in the context\-compression setting\. For methods not originally designed for this architecture, we adapted their implementations to Qwen3 to ensure a fair comparison\.

We do not compare against RCC despite its SSM\-based compressor architecture, as it fine\-tunes the decoder and encoder jointly instead of conditioning a frozen LLM, and so does not provide a matched comparison\. In our benchmark, we trained each method to align with the data distribution of the target tasks\. For Activation Beacon, we adapted the beacon window to accommodate the varying context lengths in MRQA\. Although this configuration differs from that used in the original work, we preserved the same average number of beacon tokens per context\. Unless otherwise stated, all methods are evaluated in the online setting\. Additional details regarding the experimental setup and hyperparameter configurations are presented in the Appendix\.

### 4\.2Compression Fidelity

The spread between the full\-context ceiling and the no\-context floor in Table[1](https://arxiv.org/html/2610.33006#S2.T1)confirms that the task chiefly depends on the context rather than on knowledge stored in model parameters\. In\-domain, Boa scores above both baselines on both metrics at every ratio tested, recovering 95\-96% of the floor\-to\-ceiling F1 range against 72\-88% for Activation Beacon and 66\-88% for SAC\. Out\-of\-domain \(Table[2](https://arxiv.org/html/2610.33006#S3.T2)\) the three methods are much closer: SAC is strongest at 4×\\times, Boa and Beacon are tied at 8×\\times, and Boa dominates at 16×\\times, showing less degradation as compression ratio increases\. The exception is Relation Extraction \(RE\), a task that requires verbatim extraction of text spans\. This is the setting in which attention\-based Transformers exceed SSMs, consistent with prior work\([Jelassi et al\., 2024](https://arxiv.org/html/2610.33006#bib.bib13)\), and here Beacon leads by 13\.6–20\.8 EM at every ratio\.

### 4\.3Compression Ratio Scaling

While peak quality is similar across all methods at low compression ratios \(4×\\times\), Boa degrades much more gracefully as compression ratio increases, losing less than 2% of its in\-domain EM, against roughly 10% for Activation Beacon and 27% for SAC, with the same ordering out\-of\-domain \(Tables[1](https://arxiv.org/html/2610.33006#S2.T1)and[2](https://arxiv.org/html/2610.33006#S3.T2)\)\. Boa at 16×\\timesscores above Beacon at 4×\\timesin\-domain, i\.e\. a four\-fold shorter context at no cost in fidelity\. We refer to the Appendix for detailed visualisations of score degradation across compression ratios\.

Table 3:Scalability to a larger decoder\.Boa vs\. Activation Beacon with a frozenQwen3\-4Bdecoder\.NormΔ\\Deltapools the in\- and out\-of\-domain averages and normalises between the pooled no\-context floor \(0\) and full\-context ceiling \(100\)\.
### 4\.4Inference Latency

Figure[2](https://arxiv.org/html/2610.33006#S4.F2)reports end\-to\-end latency \(encoding, prefill, and generation\) against input length using a frozen Qwen3 backbone \(comparing the 1\.7B and 4B Qwen checkpoints\)\. The baseline curve is full\-context processing by the unmodified LLM\. The other curves report inference latency for each comparison method in the online compression regime \(with single\-use context\)\. Boa operates consistently at or below this baseline, achieving a net speedup without requiring context re\-use as in the offline compression case\. We note that SAC achieves moderate speedups at longer contexts, while Activation Beacon carries a significant inference cost that scales quadratically with prefill length, never reaching viability for online use\. Boa maintains lower compute latency across all evaluated sequence lengths and decoder sizes, while matching or outperforming comparisons in performance at almost every compression ratio\. A more detailed analysis of computational cost, memory overhead and latency is presented in the Appendix\.

![Refer to caption](https://arxiv.org/html/2610.33006v1/Figures/end_to_end_pair_vertical.png)Figure 2:Total processing time versus input length at 8×\\timescompression: Boa against other methods and full\-context processing, up to 240K tokens for Qwen3\-1\.7B and 4B\. We included both windows sizes*w*used for short context and long context for Beacon
### 4\.5Long\-Context Evaluation

Table[4](https://arxiv.org/html/2610.33006#S4.T4)reports results on long context benchmarks \(up to 32K tokens\)\. On summarisation tasks BookSum and GovReport, Boa achieves performance comparable to Activation Beacon\. However, Activation Beacon retains an advantage on the question\-answering task QASPER\. We attribute this gap to a weaker long\-context prior, as the baseline Mamba checkpoint was pretrained solely on sequence lengths up to 2K tokens, whereas Activation Beacon operates directly on full\-attention windows across extended contexts\.

### 4\.6Encoder\-Decoder Scale Ratios

Table[3](https://arxiv.org/html/2610.33006#S4.T3)evaluates Boa across two compressor scales \(1\.3B and 2\.7B parameters\) using a frozen Qwen3\-4B decoder\. Note that while Activation Beacon inherently uses the full 4B Qwen backbone for both encoding and decoding, Boa achieves comparable performance with significantly smaller, lightweight encoders, even when the LLM decoder is scaled up to compensate\. Doubling our compressor to 2\.7B leaves in\-domain accuracy unchanged \(≤\\leq0\.7 EM\) and yields only small out\-of\-domain gains \(1\.2–1\.9 EM\)\. Extended results and full tables are provided in the Appendix\.

### 4\.7Ablations

Table[5](https://arxiv.org/html/2610.33006#S4.T5)separates the architectural choices from the training objective\. The choice of conditioning space is an order of magnitude more impactful than the choice of encoder family, which is consistent with 500xCompressor\([Li et al\., 2025](https://arxiv.org/html/2610.33006#bib.bib18)\)\. As a training objective, distillation is substantially more effective in the fine\-tuning stage than as a off\-policy pretraining signal: moving it to pretraining recovers less than half of the in\-domain difference between the full recipe and language modelling alone\.

Table[6](https://arxiv.org/html/2610.33006#S5.T6)shows that the three components act on different axes and are not interchangeable: distillation fine\-tuning contributes strongly to in\-domain performance, KV injection makes the largest difference for out\-of\-domain performance, and language\-modelling pretraining contributes moderately to both\. Each removal falls below the Activation Beacon baseline on at least one split\.

Table 4:Long\-contextresults at 32K sequence length under 8×\\timesand 16×\\timescompression ratios against Activation Beacon\.Table 5:Ablationsat8×8\\times\(Qwen3\-1\.7B\)\. Top: encoder×\\timesconditioning space\. Bottom: training objective\.

## 5Discussion

Our experiments in Section[4](https://arxiv.org/html/2610.33006#S4)show that Boa matches state\-of\-the\-art compression performance across a range of benchmarks while substantially reducing inference time\. These results, along with our ablations, demonstrate that KV cache injection works well for compression in spite of an architectural mismatch between the frozen LLM decoder and a trained, separate encoder\. This suggests that small, efficient encoders can be effectively paired with larger reasoning decoders, avoiding the LLM prefill bottleneck and enabling online compression even for very large Transformer models\.

Our approach presents clear scaling potential for future research\. Boa achieves competitive fidelity despite a significant pretraining deficit relative to existing methods\. While techniques that reuse the target LLM benefit from representations honed over trillions of tokens and extended context windows, our Mamba encoders were trained on approximately300​B300\\text\{B\}tokens and exposed only to2​K2\\text\{K\}\-token sequences\. The observation that our approach matches or surpasses state\-of\-the\-art compressors under these constraints suggests that the current results serve as a baseline for future optimisation\.

Several directions for future work naturally follow from this foundation\. First, initialising the compressor with state\-space models trained on larger token budgets and natively longer sequences may address current capacity limitations and further improve compression performance\. Additionally, the Boa architecture could be extended to support adaptive compression of varying token slices\. Rather than relying on a fixed compression ratio with a static strided slice parameter, future iterations could dynamically adjust the compression rate based on the semantic complexity of the context\.

Table 6:Component ablationof Boa at8×8\\times\. Each row removes one component from the full model\.
## 6Conclusion

We presented Boa, a context compressor that uses a linear\-time state\-space model to encode arbitrary input context into a representation that conditions a frozen Transformer decoder, trained through knowledge distillation, with no modification or fine\-tuning of the decoder\. Boa matches or surpasses a strong compression baseline at matched compression ratios, degrades far more gracefully as the ratio increases, and reduces inference latency and memory by large margins\. This makes compression practically viable online at inference time rather than only as offline preprocessing\.

## References

- Arora et al\. \(2024\)Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré\.Zoology: Measuring and improving recall in efficient language models\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*, 2024\.URL[https://openreview\.net/forum?id=LY3ukUANko](https://openreview.net/forum?id=LY3ukUANko)\.
- Chen et al\. \(2024\)Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia\.LongLoRA: Efficient fine\-tuning of long\-context large language models\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*, 2024\.URL[https://openreview\.net/forum?id=6PmJoRfdaK](https://openreview.net/forum?id=6PmJoRfdaK)\.
- Cheng et al\. \(2024\)Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si\-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao\.xRAG: Extreme context compression for retrieval\-augmented generation with one token\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 37, pages 109487–109516\. Curran Associates, Inc\., 2024\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/file/c5cf13bfd3762821ef7607e63ee90075\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/c5cf13bfd3762821ef7607e63ee90075-Paper-Conference.pdf)\.
- Chevalier et al\. \(2023\)Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen\.Adapting language models to compress contexts\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 3829–3846, Singapore, 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.emnlp\-main\.232\.
- Dao and Gu \(2024\)Tri Dao and Albert Gu\.Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, volume 235 of*Proceedings of Machine Learning Research*, pages 10041–10071\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/dao24a\.html](https://proceedings.mlr.press/v235/dao24a.html)\.
- Dasigi et al\. \(2021\)Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A\. Smith, and Matt Gardner\.A dataset of information\-seeking questions and answers anchored in research papers\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*, pages 4599–4610\. Association for Computational Linguistics, 2021\.doi:10\.18653/v1/2021\.naacl\-main\.365\.
- Fisch et al\. \(2019\)Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen\.MRQA 2019 shared task: Evaluating generalization in reading comprehension\.In*Proceedings of the 2nd Workshop on Machine Reading for Question Answering*, pages 1–13, Hong Kong, China, 2019\. Association for Computational Linguistics\.doi:10\.18653/v1/D19\-5801\.
- Ge et al\. \(2024\)Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si\-Qing Chen, and Furu Wei\.In\-context autoencoder for context compression in a large language model\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*, 2024\.URL[https://openreview\.net/forum?id=uREj4ZuGJE](https://openreview.net/forum?id=uREj4ZuGJE)\.
- Gu and Dao \(2024\)Albert Gu and Tri Dao\.Mamba: Linear\-time sequence modeling with selective state spaces\.In*First Conference on Language Modeling \(COLM\)*, 2024\.URL[https://openreview\.net/forum?id=tEYskw1VY2](https://openreview.net/forum?id=tEYskw1VY2)\.
- Hinton et al\. \(2015\)Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.doi:10\.48550/arXiv\.1503\.02531\.
- Huang et al\. \(2024\)Chensen Huang, Guibo Zhu, Xuepeng Wang, Yifei Luo, Guojing Ge, Haoran Chen, Dong Yi, and Jinqiao Wang\.Recurrent context compression: Efficiently expanding the context window of LLM\.*arXiv preprint arXiv:2406\.06110*, 2024\.doi:10\.48550/arXiv\.2406\.06110\.
- Huang et al\. \(2021\)Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang\.Efficient attentions for long document summarization\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*, pages 1419–1436\. Association for Computational Linguistics, 2021\.doi:10\.18653/v1/2021\.naacl\-main\.112\.
- Jelassi et al\. \(2024\)Samy Jelassi, David Brandfonbrener, Sham M\. Kakade, and Eran Malach\.Repeat after me: Transformers are better than state space models at copying\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, volume 235 of*Proceedings of Machine Learning Research*, pages 21502–21521\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/jelassi24a\.html](https://proceedings.mlr.press/v235/jelassi24a.html)\.
- Jiang et al\. \(2023\)Huiqiang Jiang, Qianhui Wu, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\.LLMLingua: Compressing prompts for accelerated inference of large language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 13358–13376, Singapore, 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.emnlp\-main\.825\.
- Jiang et al\. \(2024\)Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\.LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\), Volume 1: Long Papers*, pages 1658–1677, Bangkok, Thailand, 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.91\.
- Kryściński et al\. \(2022\)Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev\.BookSum: A collection of datasets for long\-form narrative summarization\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, pages 6536–6558, Abu Dhabi, United Arab Emirates, 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.findings\-emnlp\.488\.
- Li et al\. \(2023\)Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin\.Compressing context to enhance inference efficiency of large language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 6342–6353, Singapore, 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.emnlp\-main\.391\.
- Li et al\. \(2025\)Zongqian Li, Yixuan Su, and Nigel Collier\.500xCompressor: Generalized prompt compression for large language models\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\), Volume 1: Long Papers*, pages 25081–25091, Vienna, Austria, 2025\. Association for Computational Linguistics\.doi:10\.18653/v1/2025\.acl\-long\.1219\.
- Lin et al\. \(2025\)Xiaoqiang Lin, Aritra Ghosh, Bryan Kian Hsiang Low, Anshumali Shrivastava, and Vijai Mohan\.REFRAG: Rethinking RAG based decoding\.*arXiv preprint arXiv:2509\.01092*, 2025\.doi:10\.48550/arXiv\.2509\.01092\.
- Liu et al\. \(2026\)Xin Liu, Runsong Zhao, Pengcheng Huang, Xinyu Liu, Junyi Xiao, Chunyang Xiao, Tong Xiao, Shengxiang Gao, Zhengtao Yu, and Jingbo Zhu\.Autoencoding\-free context compression for LLMs via contextual semantic anchors\.In*The Fourteenth International Conference on Learning Representations \(ICLR\)*, 2026\.URL[https://openreview\.net/forum?id=8Pi6Du0n7F](https://openreview.net/forum?id=8Pi6Du0n7F)\.
- Mu et al\. \(2023\)Jesse Mu, Xiang Lisa Li, and Noah Goodman\.Learning to compress prompts with gist tokens\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 36, pages 19327–19352\. Curran Associates, Inc\., 2023\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/file/3d77c6dcc7f143aa2154e7f4d5e22d68\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/3d77c6dcc7f143aa2154e7f4d5e22d68-Paper-Conference.pdf)\.
- Penedo et al\. \(2024\)Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf\.The FineWeb datasets: Decanting the web for the finest text data at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*, 2024\.URL[https://openreview\.net/forum?id=n6SCkn2QaG](https://openreview.net/forum?id=n6SCkn2QaG)\.
- Rau et al\. \(2025\)David Rau, Shuai Wang, Hervé Déjean, Stéphane Clinchant, and Jaap Kamps\.Context embeddings for efficient answer generation in retrieval\-augmented generation\.In*Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining \(WSDM\)*, pages 493–502, Hannover, Germany, 2025\. Association for Computing Machinery\.doi:10\.1145/3701551\.3703527\.
- Snell et al\. \(2022\)Charlie Snell, Dan Klein, and Ruiqi Zhong\.Learning by distilling context\.*arXiv preprint arXiv:2209\.15189*, 2022\.doi:10\.48550/arXiv\.2209\.15189\.
- Waleffe et al\. \(2024\)Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro\.An empirical study of Mamba\-based language models\.*arXiv preprint arXiv:2406\.07887*, 2024\.doi:10\.48550/arXiv\.2406\.07887\.
- Wingate et al\. \(2022\)David Wingate, Mohammad Shoeybi, and Taylor Sorensen\.Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, pages 5621–5634, Abu Dhabi, United Arab Emirates, 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.findings\-emnlp\.412\.
- Xu et al\. \(2024\)Fangyuan Xu, Weijia Shi, and Eunsol Choi\.RECOMP: Improving retrieval\-augmented LMs with context compression and selective augmentation\.In*The Twelfth International Conference on Learning Representations \(ICLR\)*, 2024\.URL[https://openreview\.net/forum?id=mlJLVigNHp](https://openreview.net/forum?id=mlJLVigNHp)\.
- Zhang et al\. \(2025\)Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou\.Long context compression with Activation Beacon\.In*The Thirteenth International Conference on Learning Representations \(ICLR\)*, 2025\.URL[https://openreview\.net/forum?id=1eQT9OzfNQ](https://openreview.net/forum?id=1eQT9OzfNQ)\.

## Appendix AImplementation and Training Details

### A\.1MRQA Training

Table[8](https://arxiv.org/html/2610.33006#A1.T8)reports the full configuration of the two training stages of the MRQA Boa pipeline: \(i\)*pretraining*of the Mamba2 compressor with a next\-token language\-modeling objective, and \(ii\)*knowledge\-distillation \(KD\) fine\-tuning*on MRQA against the frozen teacher\. In both stages the Qwen3\-1\.7B decoder is kept frozen and the compressed context is injected through the decoder’s KV cache\. All hyperparameters below are shared across*every*compression ratio \(4×4\\times,8×8\\times,16×16\\times\); the ratio is controlled by a singlestrideargument and is the only value that changes between runs\. The fine\-tuning stage is initialized from the pretrained compressor checkpoint of the matching ratio\.

#### Frozen teacher: Qwen3\-1\.7B full fine\-tuning on MRQA\.

The frozen decoder used throughout is Qwen3\-1\.7B, obtained by fully fine\-tuning \(FullFT\) the base Qwen3\-1\.7B checkpoint on MRQA\. We train for a single epoch, which we find sufficient for convergence on this dataset\. This FullFT checkpoint is the teacher/decoder used by every Boa pretraining and fine\-tuning run reported here\. Table[7](https://arxiv.org/html/2610.33006#A1.T7)summarizes its configuration\.

Table 7:Full fine\-tuning configuration of the Qwen3\-1\.7B teacher on MRQA\.

### A\.2Long\-Context Training

Table 8:Training configuration of the two Boa stages on MRQA: Language Modeling \(LM\) pretraining of the Mamba2 compressor \(left\) and KL knowledge\-distillation fine\-tuning against the frozen Qwen3\-1\.7B teacher \(right\)\. All values are shared across every compression ratio \(4×/8×/16×4\\times/8\\times/16\\times\); onlystride\(and, for fine\-tuning, the matching pretrained init checkpoint\) changes between runs\.The initial setup used for the long\-context training is the same as specified above, with the same model configurations\. The training setup is changed in order to increase the sequence length employed, mainly adding full activation checkpointing, a float8 variant of AdamW, and making use of common PyTorch optimisation techniques; Table[9](https://arxiv.org/html/2610.33006#A1.T9)reports the resulting configuration\.

#### Long\-context pretraining data\.

We sample the longest sequences from the FineWeb\-Edu 100B pool of data\. Since we truncate the context during training if it surpasses our chosen sequence length, we sampled more than 10B tokens from the 100B pool in order to reach our 10B\-token training budget target\.

#### Long\-context fine\-tuning data\.

The following datasets are used for the fine\-tuning phase using KD in the long\-context setups: LongAlpaca\-12k, BookSum, GovReport, and Qasper\. BookSum and GovReport correspond to summarisation tasks, while Qasper and LongAlpaca correspond to question answering\. Figure[3](https://arxiv.org/html/2610.33006#A1.F3)provides the sequence\-length distribution of the data used for evaluation and training during this phase\. During training, at10,24010\{,\}240tokens we start truncating the context of the given sample\. Token counts are calculated using the tokeniser employed by Boa\.

Table 9:Training configuration of the two Boa stages on long\-context data: Language Modeling \(LM\) pretraining of the Mamba2 compressor \(left\) and KL knowledge\-distillation fine\-tuning against the frozen Qwen3\-1\.7B teacher \(right\)\. All values are shared for both8×8\\timesand16×16\\timescompression ratios; onlystride\(and, for fine\-tuning, the matching pretrained init checkpoint\) changes between runs\.![Refer to caption](https://arxiv.org/html/2610.33006v1/Figures/kd_context_data.png)Figure 3:Sequence\-length distributions of the long\-context KD fine\-tuning data, shown as empirical cumulative distributions \(each point is one sample; theyy\-axis gives the proportion of samples at or below its length, thexx\-axis the context length in Boa tokens, log\-scaled\)\.Top:training corpora \(LongAlpaca\-12k, BookSum, GovReport, Qasper\); the dashed line marks the10,24010\{,\}240\-token training sequence length, and samples beyond it \(drawn desaturated\) are truncated during training\.Bottom:evaluation splits\. GovReport is the longest on average and has the most samples above the training sequence length\. We evaluate on the full sequence length of each sample without truncation\.

### A\.3Hardware

All runs were trained on the accelerated \(ACC\) partition of the MareNostrum 5 supercomputer using32 nodes\. Each node is equipped with4×\\timesNVIDIA Hopper H100 with 64 GB HBM2, two Intel Xeon Platinum 8460Y\+ \(40C, 2\.3 GHz\) CPUs, and 512 GB of DDR5 memory, interconnected with NDR200 InfiniBand\. This yields a total of128 H100 GPUs\. Training uses distributed data parallelism \(DDP\) across all 128 GPUs, with the frozen decoder run in bfloat16\.

## Appendix BScalability to a Larger Decoder: Full Per\-Dataset Results

The scalability table in the main paper reports in\-domain and out\-of\-domain averages for Boa and Activation Beacon with a frozenQwen3\-4Bdecoder, at compressor scales 1\.3B and 2\.7B\. Tables[10](https://arxiv.org/html/2610.33006#A2.T10)and[11](https://arxiv.org/html/2610.33006#A2.T11)below give the full per\-dataset breakdown behind those averages\.

Table 10:MRQA*in\-domain*results with a frozenQwen3\-4Bdecoder, per dataset\. EM / token\-level F1; higher is better\.NormΔ\\Deltanormalises the average between the no\-context floor \(0\) and full\-context ceiling \(100\)\.Table 11:MRQA*out\-of\-domain*results with a frozenQwen3\-4Bdecoder, per dataset\. EM / F1; higher is better\.NormΔ\\Deltanormalises the average between the no\-context floor \(0\) and full\-context ceiling \(100\)\.
## Appendix CLatency and Memory Efficiency

Figure[4](https://arxiv.org/html/2610.33006#A3.F4)profiles wall\-clock latency and peak GPU memory for Boa, SAC, Activation Beacon \(window sizes 128 and 2048\), and uncompressed full\-context inference, all at8×8\\timescompression with a frozen Qwen3\-1\.7B decoder in bfloat16 on an NVIDIA RTX PRO 6000 \(Blackwell\)\. Each data point is the median of three timed runs \(one discarded warmup\), measured betweentorch\.cuda\.synchronize\(\)barriers\.

#### Context processing time \(panel a\)\.

Different methods split the work between an explicitcompress\(\)call and the decoder’s prefill in different ways: SAC runs a separable encoder whose cost lands entirely in compression time, whereas Boa and Activation Beacon time their compression inside the decoder forward pass, reporting zero standalone compression time\. To ensure a fair comparison, panel \(a\) plots the sum of compression time and time\-to\-first\-token \(tcompress\+tttftt\_\{\\text\{compress\}\}\+t\_\{\\text\{ttft\}\}\), which captures the full elapsed time from raw context tokens to the first generated token regardless of where each method draws its internal boundary\. Boa remains the fastest method\.

#### End\-to\-end latency \(panel b\)\.

Panel \(b\) adds the decode phase \(64 generated tokens\) to the context processing cost\. Because all methods pay a similar decode overhead, the relative ordering is preserved\.

#### Peak GPU memory \(panel c\)\.

Panel \(c\) reportstorch\.cuda\.max\_memory\_allocatedper run\. All compression methods remain closely clustered well below the full\-context baseline\. Activation Beacon with a window of 128 tokens achieves the lowest memory footprint, stemming from its sliding\-window design; however, when the window is widened to 2048 for faster inference, its memory consumption rises to the level of Boa\. Overall, Boa trades moderately higher memory for substantially lower latency\.

![Refer to caption](https://arxiv.org/html/2610.33006v1/Figures/latency_memory_strip.png)Figure 4:Latency and memory comparison at8×8\\timescompression with Qwen3\-1\.7B \(bfloat16, RTX PRO 6000 Blackwell\)\.\(a\)Context processing time \(tcompress\+tttftt\_\{\\text\{compress\}\}\+t\_\{\\text\{ttft\}\}\): elapsed time from raw context to first generated token, log scale\.\(b\)End\-to\-end latency including 64 decoded tokens, log scale\.\(c\)Peak GPU memory, linear scale\. Full context \(grey dashed\) is the uncompressed reference, not a competing method\.

## Appendix DCompression Ratio Scaling: Graceful Degradation

Boa degrades far more gracefully than the baselines as the compression ratio increases from 4×\\timesto 16×\\times\. Figure[5](https://arxiv.org/html/2610.33006#A4.F5)plots in\-domain normalised F1 \(NormΔ\\Delta\) against compression ratio for Boa and Activation Beacon, making the effect visible directly: Boa’s curve is nearly flat, while Activation Beacon’s drops sharply, and the gap widens as the ratio grows\.

![Refer to caption](https://arxiv.org/html/2610.33006v1/Figures/gracefull_degradation.png)Figure 5:In\-domain normalised F1 \(NormΔ\\Delta\) as a function of compression ratio \(4×\\times, 8×\\times, 16×\\times\) for Boa and Activation Beacon\. Boa loses less than 2% of its normalised F1 across the full range, while Activation Beacon loses roughly 10%\.