Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

# Good Memory has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy
Source: [https://arxiv.org/html/2609.00103](https://arxiv.org/html/2609.00103)
Shmuel BermanAffiliation:Department of Computer ScienceAffiliation:Princeton UniversityEmail:[sb6870@princeton\.edu](mailto:)Jia DengAffiliation:Department of Computer ScienceAffiliation:Princeton UniversityEmail:[jiadeng@princeton\.edu](mailto:)

###### Abstract

Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video\. However, accuracy alone misses properties that matter for real long\-horizon tasks\. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system’s*capacity*—its raw accuracy at a specific budget—via three axes we call ECC:*efficiency*—the computation, in FLOPs, needed to answer from memory;*compression*—whether compressible inputs are remembered more accurately or efficiently; and*calibration*—whether the system abstains in response to its own uncertainty and the cost of an error\. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both\. Among a broader set of memory backbones, several non\-Transformer architectures achieve better compression\-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons\.

*Keywords*memory⋅\\cdotvision\-language models⋅\\cdotlong\-context⋅\\cdotbenchmark⋅\\cdotcalibration⋅\\cdotcompression

## 1Introduction

Memory is widely viewed as an important missing capability for LLMs and VLMs, and is a subject of active research\([Du et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib8);[Long et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib21);[Hong et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib15);[Sinha et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib24)\)\. But what does it mean for an LLM or VLM to have a good memory and how can we evaluate it? This question may sound trivial and appears to have a settled solution already widely practiced by existing benchmarks\([Packer et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib1);[Hu et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib2);[Wang et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib3)\)\. The solution is to measure accuracy of answering questions about a long text or video—if a model achieves high accuracy, it has a good memory\.

But is it so? In this paper, we argue that it is not—measuring accuracy alone is insufficient and does not constitute proper evaluation of memory\. We argue that additional aspects must be measured, which include Efficiency, Compression, and Calibration \(ECC\)\.

Figure 1:Accuracy alone cannot tell two memory systems apart; ECC can\.Two black\-box memory systems, A and B, answer the same recall task and score nearly the same on standard accuracy, yet differ on the axes beyond accuracy:*Efficiency*\(compute spent per query\),*Compression*\(accuracy gained when the input has structure rather than being random\), and*Calibration*\(abstaining when the answer cannot be recalled and an error is costly\)\. System B is the stronger memory on all three\. Numbers are illustrative\.Efficiency—the computational resources needed to access memorized information—is important to measure in addition to accuracy, because without it, strong memory can be trivial\. If given unlimited time, a capable but memoryless model can achieve high accuracy by simply brute\-force checking every chunk of the original input, like how a person can re\-read every page of a book to find the answer\. Intuitively, good memory implies fast and effortless recall of information\. A proper evaluation of memory must consider compute cost in addition to accuracy\.

Now, if a system can accurately and efficiently answer questions about long inputs, does it necessarily have a good memory? Surprisingly not, unless the questions are chosen appropriately, because a system with poor memory can still do well by exploiting biases in the data it sees and the questions it is asked\. For example, if faces occur only occasionally in a long video but every question is about them, a system that memorizes faces and nothing else beats one that remembers everything else\. What if we make the questions completely random such that every piece of information will be queried with equal likelihood? In this case, given long enough input and finite storage, choosing what to remember will not help, and the only way to outperform others is*compression*—exploiting the redundancy and regularity in the input \(e\.g\., recurring green\-yellow\-red patterns of a traffic light\)\.

This suggests that proper evaluation of memory should separate compression from*saliency*—priors on what information is more important to retain\. Such priors are highly task\- and domain\-dependent, and can in fact be counterproductive if questions are adversarially chosen\. However, existing evaluations have not been designed to separate saliency as a confounder, and it is often unclear whether a system performs well due to saliency or compression\.

Memory in real systems is also finite and cannot possibly remember everything even with compression\. Thus a useful memory system needs to have*calibration*: it abstains in response to its own uncertainty and the cost of an error\. A memory module should be able to abstain from responding when it detects the requested information was not preserved or is uncertain and there is a cost to incorrect retrieval\. However, existing benchmarks have not been designed to evaluate calibration because in most cases the models are not provided an abstain option or a cost for an incorrect answer\.

We present ECCBench, a*general*evaluation of memory that takes all of these aspects of memory into account\. We measure*efficiency*through the computation required to produce an answer\. Our dataset lets us measure*compression*by varying the information content of the input streams and comparing relative accuracy\. Finally, we measure*calibration*by giving the system an explicit abstention option with an associated cost of error\. Together, these dimensions provide a more complete picture of how effectively a system can store, retrieve, and use information\. Figure[1](https://arxiv.org/html/2609.00103#S1.F1)illustrates the three axes\. In the task, a system watches a stream and is later asked whether a short candidate appeared in it\. Our code is available[here](https://github.com/princeton-vl/ECCBench)and the dataset[here](https://huggingface.co/datasets/anonstreammem/substream-recollection/tree/main)\.

Our evaluation reveals significant weaknesses in VLMs\. While some VLMs compress their memory on text, the ones we tested generally cannot on video\. They are also poorly calibrated across both modalities; increasing the cost of an incorrect answer has little effect on their coverage\.

Motivated by our VLM results, we evaluate a diverse set of memory backbones beyond the sequence lengths seen during training\. We find that several architectures achieve substantially better compression\-calibration tradeoffs than RoPE Transformers\. Our experiments show that the underlying architecture itself can substantially change the accuracy, compression, and calibration properties of memory\.

## 2Related Work

##### Long\-context and long\-video benchmarks\.

There is a rich corpus of work that evaluates memory by asking questions about a long document or video\. Natural benchmarks use question answering, summarization, and long\-video understanding, while synthetic benchmarks insert small targets, or “needles,” and ask models to retrieve, order, or count them\([Bai et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib5);[An et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib4);[Zhang et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib35);[Fu et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib9);[Wu et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib32);[Lin et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib20);[Kamradt, 2023](https://arxiv.org/html/2609.00103#bib.bib39);[Hsieh et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib16);[Zhao et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib40);[Wang et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib30)\)\. These test memory through accuracy but confound memory with the ability to predict what will matter\.[Mangalam et al\. \(2023\)](https://arxiv.org/html/2609.00103#bib.bib22)formalize a related idea with the*temporal certificate*, the smallest portion of a video needed to verify an answer\. While this is related, the quantity we define in Section[3](https://arxiv.org/html/2609.00103#S3)captures the features of a stream that must be kept over the entire question distribution\.

##### Definitions and evaluations of memory\.

Prior work uses*memory*in several senses\. Early neural systems describe a state that can be read and written, while agent frameworks distinguish episodic memory for past events from semantic and procedural memory\([Weston et al\., 2015](https://arxiv.org/html/2609.00103#bib.bib31);[Graves et al\., 2014](https://arxiv.org/html/2609.00103#bib.bib11);[Sumers et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib50)\), and recent work treats model components themselves as associative memories\([Behrouz et al\., 2025a](https://arxiv.org/html/2609.00103#bib.bib51)\)\. Our definition abstracts over these mechanisms and defines a memory system as one that observes a stream, retains some state, and later answers from it\. Benchmarks of episodic memory instead test useful abilities over conversations and interactions, including recall, temporal reasoning, updating, and conflict resolution\([Maharana et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib57);[Wu et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib58);[Hu et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib2);[Tan et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib59);[Long et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib21)\)\. Some already measure abstention or efficiency, but usually as properties of the full agent, where retrieval, reasoning, and storage policy are intertwined\. HELM provides an important broader precedent for evaluating models along several dimensions rather than accuracy alone\([Liang et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib52)\)\. Its dimensions are general properties of language\-model behavior and do not identify what an implementation\-agnostic memory should provide\. We specify these memory\-specific properties through capacity, efficiency, compression, and cost\-sensitive abstention, and apply them to both text and video\.

##### Associative recall and compression\.

Associative recall is a long\-standing memory test in which a cue is used to recover stored information, and remains common for comparing sequence architectures\([Hopfield, 1982](https://arxiv.org/html/2609.00103#bib.bib56);[Graves et al\., 2014](https://arxiv.org/html/2609.00103#bib.bib11);[Arora et al\., 2024a](https://arxiv.org/html/2609.00103#bib.bib53);[Arora et al\., 2024b](https://arxiv.org/html/2609.00103#bib.bib54)\)\. ECCBench uses this as the base unit of our task in different modalities because it requires little semantic knowledge or multi\-step reasoning, allowing other properties of memory to be assessed independently\. Compression has also been used differently in prior work, including lossless coding and methods that reduce stored activations, cache entries, or tokens, and is mainly used in relation to storage space\([Delétang et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib7);[Rae et al\., 2020](https://arxiv.org/html/2609.00103#bib.bib23);[Xiao et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib33);[Tang et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib27)\)\. Prior work observed that repetition improves recall in recurrent models\([Arora et al\., 2024c](https://arxiv.org/html/2609.00103#bib.bib55)\), but does not generalize this to predictable structure\. Our notion of compression is behavioral and can be determined empirically\.

##### Memory architectures and agents\.

Many architectures aim to retain long histories using recurrent state, learned updates, or a reduced attention cache, while memory\-augmented models and agents keep internal or external stores that can be searched, summarized, or edited\([Katharopoulos et al\., 2020](https://arxiv.org/html/2609.00103#bib.bib17);[Sun et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib25);[Gu and Dao, 2024](https://arxiv.org/html/2609.00103#bib.bib13);[Dao and Gu, 2024](https://arxiv.org/html/2609.00103#bib.bib42);[Yang et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib44);[Peng et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib45);[Sun et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib26);[Behrouz et al\., 2025b](https://arxiv.org/html/2609.00103#bib.bib41);[Zhang et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib47);[Lewis et al\., 2020](https://arxiv.org/html/2609.00103#bib.bib37);[Packer et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib1);[Long et al\., 2026](https://arxiv.org/html/2609.00103#bib.bib21)\)\. ECCBench treats memory as a black box and evaluates what can be recalled, at what total computation, whether predictable experience helps, and whether the system recognizes uncertain recalls\. The same framework can therefore evaluate a Transformer context window, a recurrent model, a compressed cache, or a retrieval system such as RAG; our controlled backbone experiments further hold the task and training distribution fixed when comparing architectures\.

## 3Background

We first formalize memory as an online problem and define the properties we use to evaluate it\. A memory system observes an input, keeps some record of it, and later answers questions using its state\. The input may arrive piecemeal and a question may be asked at any point about anything already seen\. We write the stream asx1:Lx\_\{1:L\}, with eachxix\_\{i\}a single observation, such as a frame or a piece of text\.LLis not revealed to the system in advance\.

A memory system must hold an internal state, although it is not necessarily of bounded size\. It provides two operations, a write and a read, where a write folds the next observation into the state and a read takes a questionqqand answers from that state alone:

St\+1=𝖶𝗋𝗂𝗍𝖾⁡\(St,xt\+1\),\(y^,St\+\)=𝖱𝖾𝖺𝖽⁡\(St,q\),y^∈𝒴∪\{⊥\}\.S\_\{t\+1\}=\\mathsf\{Write\}\(S\_\{t\},x\_\{t\+1\}\),\\qquad\(\\hat\{y\},S\_\{t\}^\{\+\}\)=\\mathsf\{Read\}\(S\_\{t\},q\),\\qquad\\hat\{y\}\\in\\mathcal\{Y\}\\cup\\\{\\bot\\\}\.\(1\)Here𝒴\\mathcal\{Y\}is the set of allowed answers and⊥\\botis abstention\. A write is a general operation that may append, evict, or reorganize the memory state, and a read may occur at anyttand return an updated state\.

The only assumption we make about a memory system is that we can count the arithmetic it uses; otherwise we treat it like a black box\. That arithmetic is counted in floating\-point operations,FwF^\{\\mathrm\{w\}\}for reading the stream andFrF^\{\\mathrm\{r\}\}for answering\. For instance, this may correspond in the pretrained VLMs to prefill and decode\. We report the totalF=Fw\+FrF=F^\{\\mathrm\{w\}\}\+F^\{\\mathrm\{r\}\}, since a system may spend early so that answering is cheap or defer most processing until a question arrives\. The optimal allocation between the two depends on how many queries each stream has to serve, which is task specific\. We assume, in counting FLOPs, there is one query per stream\. However, our framework extends to separatingFwF^\{\\mathrm\{w\}\}andFrF^\{\\mathrm\{r\}\}and measuring the properties of memory against them\. We do not make any other assumptions about the memory system, so it can apply to a context window, a recurrent state, a retrieval index, or an explicitly edited store\([Lewis et al\., 2020](https://arxiv.org/html/2609.00103#bib.bib37)\)\.

Let𝒟\\mathcal\{D\}be a distribution over input streams and𝒬\\mathcal\{Q\}a distribution over queries about them\. No independence assumption is made, so𝒟\\mathcal\{D\}may contain arbitrary relationships among tokens or frames\. For streams drawn from𝒟\\mathcal\{D\}and queries drawn from𝒬\\mathcal\{Q\}, letCCbe the probability of a correct answer andRRthe probability that the system responds\. The probability of an incorrect answer isR−CR\-C, and the probability of abstention is1−R1\-R\.

Under this setting, a good memory has four desirable attributes\. The first, capacity, is measured by accuracy; the three we add are the E, C, and C of the title\.

##### Capacity\.

A system has high capacity if it can accurately recall a large amount of incompressible input\. For a stream lengthLLand FLOPs budgetFF, we measure capacity as the correct answer rateCCon a matched maximum\-entropy distribution𝒟max\\mathcal\{D\}\_\{\\max\}\. ComparingCCacross stream lengths at fixedFFshows how much incompressible input the system can remember\. The distribution𝒟max\\mathcal\{D\}\_\{\\max\}provides no useful repetition or other shortcut; in our synthetic examples, we define it as a uniform distribution over the vocabulary\. Capacity under an unlimited FLOPs budget is analogous to raw accuracy in other benchmarks\.

##### Efficiency\.

A system is efficient if it reaches a given correct answer rate at low cost, where we define cost as computation\. We use total FLOPsFFas the cost metric and measure efficiency as the smallestFFneeded to reach a targetCC\. We use FLOPs rather than state size or latency, since storage \(off of specialized hardware such as GPUs\) is cheap, and latency depends heavily on hardware and implementation\([Gu and Dao, 2024](https://arxiv.org/html/2609.00103#bib.bib13);[Dao, 2024](https://arxiv.org/html/2609.00103#bib.bib6)\)\. All of our numbers assume one query per stream\. In other settings, it may be more useful to use other metrics such as latency or the depth of computation, but using FLOPs is general and lets us consistently compare different architectures\.

##### Compression\.

A system can compress if it uses regularity in the observed stream to improve recall or reduce how much computation it uses\.

We measure compression by comparing equal\-length streams drawn from distributions with different amounts of regularity\. At fixedLL,FF, and𝒬\\mathcal\{Q\}, the accuracy gained over incompressible input is

Comp⁡\(L,F,𝒟\)=C𝒟​\(L,F\)−C𝒟max​\(L,F\),\\operatorname\{Comp\}\(L,F;\\mathcal\{D\}\)=C\_\{\\mathcal\{D\}\}\(L,F\)\-C\_\{\\mathcal\{D\}\_\{\\max\}\}\(L,F\),\(2\)whereC𝒟​\(L,F\)C\_\{\\mathcal\{D\}\}\(L,F\)is the correct answer rate on streams from𝒟\\mathcal\{D\}\. Alternatively, it can be measured as FLOPs saved when reaching the sameCC, although this is usually more difficult to measure\.

For the discrete streams in the synthetic subset of our benchmark, we measure regularity with the empirical Lempel–Ziv rateh^LZ\\hat\{h\}\_\{\\mathrm\{LZ\}\}, in bits per symbol\([Ziv and Lempel, 1978](https://arxiv.org/html/2609.00103#bib.bib36);[Kontoyiannis, 1998](https://arxiv.org/html/2609.00103#bib.bib19)\): about0\.70\.7on the low\-entropy streams we generate, and44bits on𝒟max\\mathcal\{D\}\_\{\\max\}, which is generated from a uniform distribution\.

We distinguish compression from*saliency*, which is task specific\. Saliency exploits knowledge of what questions are likely to be asked to decide what information is worth retaining\. Compression instead exploits regularity in the stream to retain that information more efficiently\.

To make this distinction precise, consider the stream distribution𝒟\\mathcal\{D\}and question distribution𝒬\\mathcal\{Q\}\. For a streamxx, let𝒦𝒟,𝒬​\(x\)\\mathcal\{K\}\_\{\\mathcal\{D\},\\mathcal\{Q\}\}\(x\)denote the information that must be retained to answer any question that𝒬\\mathcal\{Q\}may ask\. Two streamsxxandx′x^\{\\prime\}in the support of𝒟\\mathcal\{D\}have the same𝒦𝒟,𝒬​\(x\)\\mathcal\{K\}\_\{\\mathcal\{D\},\\mathcal\{Q\}\}\(x\)exactly when

y⁡\(x,q\)=y⁡\(x′,q\)for every question​q​that​𝒬​may ask,y\(x,q\)=y\(x^\{\\prime\},q\)\\qquad\\text\{for every question \}q\\text\{ that \}\\mathcal\{Q\}\\text\{ may ask\},wherey⁡\(x,q\)y\(x,q\)is the correct answer toqqonxx\. Thus,𝒦𝒟,𝒬​\(x\)\\mathcal\{K\}\_\{\\mathcal\{D\},\\mathcal\{Q\}\}\(x\)preserves exactly the differences between streams that can affect a future answer\. These need not correspond to particular tokens, frames, or regions\. We write𝒦\\mathcal\{K\}for short\.

This definition lets us disentangle saliency from compression capability\. If𝒬\\mathcal\{Q\}is narrow, many differences between streams never matter, and a system can discard them using a saliency prior\. On the other hand, if some questions are more likely it can further favor the information those questions require\. Compression instead exploits regularity in𝒟\\mathcal\{D\}to reduce how much must be stored to preserve the distinctions that do matter\. On natural tasks,𝒟\\mathcal\{D\},𝒬\\mathcal\{Q\}, and therefore𝒦\\mathcal\{K\}are generally unknown\. Our synthetic task controls them explicitly, together with the entropy of𝒟\\mathcal\{D\}, allowing us to isolate compression from saliency\.

##### Calibration\.

A system is calibrated if it answers when its memory is reliable and abstains when the risk of error is high\. We define calibration here in a selective sense, and do not attempt to elicit a system’s stated confidence\. Because the value of abstention depends on the cost of an error, we measure calibration through coverage,RR, and accuracy\-on\-answered,C/RC/R, at different reward and cost profiles\([El\-Yaniv and Wiener, 2010](https://arxiv.org/html/2609.00103#bib.bib38)\)\.

Calibration is measured through responsiveness and selectivity\. WriteRλR\_\{\\lambda\}for coverage when a wrong answer costsλ\\lambda\. Answering is worth it exactly when the system’s confidence exceedsτ=λ/\(1\+λ\)\\tau=\\lambda/\(1\+\\lambda\), so raisingλ\\lambdashould lower coverage\.*Responsiveness*is the drop in coverage,Rλlo−RλhiR\_\{\\lambda\_\{\\mathrm\{lo\}\}\}\-R\_\{\\lambda\_\{\\mathrm\{hi\}\}\}; a system that ignores the cost of an error scores zero on it\.*Selectivity*asks whether the system knows which questions to abstain from by comparing its accuracy on the dropped questions against its accuracy on the kept ones\.

## 4Experiments

ECCBench has two evaluation sets: a synthetic set with controlled entropy, and a set of natural videos\. The first tests both textual and video memory in distinct subsets\. We then evaluate nine pretrained VLM families on the text and video versions of ECCBench under the compute\-aware protocol of Section[3](https://arxiv.org/html/2609.00103#S3), using token counts to compute the FLOPs each model spent on the answer\. We additionally score a RAG\-like pipeline as one more memory system; full details are in Appendix[B\.7](https://arxiv.org/html/2609.00103#A2.SS7)\.

### 4\.1Data

\(a\) Synthetic textstream8, 1,4, 6, 2, 1, 1, 2, 13, 1, 2, 9, 3, 1, 2, 7, 0, 5, 2, …candidate4, 6, 2\(b\) Synthetic videostream![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_1.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_2.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_3.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_4.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_5.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_6.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_7.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_stream_8.png)⋯\\cdotscandidate![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_cand_1.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_cand_2.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_cand_3.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_cand_4.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/sv_cand_5.png)\(c\) Natural videostream![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/nat_1.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/nat_2.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/nat_3.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/nat_4.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/nat_5.png)![Refer to caption](https://arxiv.org/html/2609.00103v1/images/frames/nat_6.png)⋯\\cdotscandidate“Did the person open the fridge in this video?”Figure 2:One example from each evaluation set\.Each set pairs a stream with a candidate, and the model judges whether the candidate occurred in the stream\. On both synthetic sets the candidate is in the same modality as the stream—text against text in \(a\), rendered frames against rendered frames in \(b\)—so no cross\-modal transfer is required; highlighting marks the positions matched\. The natural set \(c\) cannot be built that way, since its queries come from activity annotations: the stream is video and the candidate a text question\. Frames in \(c\) are EPIC\-Kitchens\-100 \(CC BY\-NC 4\.0\), from our natural\-video split\.##### Synthetic subset\.

Our primary evaluation uses discrete streamsx1:Lx\_\{1:L\}over a vocabulary𝒱\\mathcal\{V\}, with\|𝒱\|=16\|\\mathcal\{V\}\|=16\. For the max\-entropy distribution𝒟max\\mathcal\{D\}\_\{\\max\}, symbols are sampled i\.i\.d\. and uniformly from𝒱\\mathcal\{V\}\. The low\-entropy distributions𝒟\\mathcal\{D\}are variable\-ordernn\-gram processes whose generated streams haveh^LZ≈0\.7\\hat\{h\}\_\{\\mathrm\{LZ\}\}\\approx 0\.7bits per symbol\([Ziv and Lempel, 1978](https://arxiv.org/html/2609.00103#bib.bib36)\)\. The query distribution𝒬\\mathcal\{Q\}samples balanced positive and negative candidates; positives are copied from random stream positions and negatives are checked to be absent \(Appendix[A\.1](https://arxiv.org/html/2609.00103#A1.SS1)\)\. Each stream has 16 queries asking whether a candidate occurred as an exact contiguous subsequence\. Because of our construction,𝒦𝒟,𝒬​\(x\)\\mathcal\{K\}\_\{\\mathcal\{D\},\\mathcal\{Q\}\}\(x\)is known precisely: the questions range over all candidates of lengths three to six, and the answers depend only on which of them occur in the stream, so that is all a system needs to retain, and how often one occurs does not matter\. Generator details are in Appendix[A\.1](https://arxiv.org/html/2609.00103#A1.SS1)\.

Text streams are comma\-separated integers and video streams render one letter per frame on one of three labeled lanes\. The candidates are the same modality as the overall stream\. Rendering and query details are in Appendix[A\.3](https://arxiv.org/html/2609.00103#A1.SS3)\. Figure[2](https://arxiv.org/html/2609.00103#S4.F2)shows one example from each set\.

##### Natural video subset\.

We build𝒟nat\\mathcal\{D\}\_\{\\mathrm\{nat\}\}from EPIC\-Kitchens\-100 and SoccerNet clips and take𝒬nat\\mathcal\{Q\}\_\{\\mathrm\{nat\}\}from their activity annotations\. Each question asks whether a particular activity occurred in the clip\. It is important to note that we can only make qualitative claims about compression on natural video\. For natural video we do not know𝒟\\mathcal\{D\},𝒬\\mathcal\{Q\}, and therefore what must be retained, precisely, so an apparent advantage on low\-density video can come from compression, from a useful saliency prior, or from both\. We also do not know its true underlying entropy\. We measure its information content by encoding the videos and reconstructing them and split the videos at the median into a low\-density and a high\-density band; the full details of generation and the information measure are in Appendix[B\.6](https://arxiv.org/html/2609.00103#A2.SS6)and Appendix[B\.6\.2](https://arxiv.org/html/2609.00103#A2.SS6.SSS2)\. Though qualitative, this tests whether our synthetic results extend to real\-world data\.

### 4\.2Can VLMs compress?

We first ask whether VLMs can compress their memory, and if their ability to do so changes over text and video\. On each of our subsets, we measurecapacityon the densest, or highest entropy, streams andcompressionrelative to the lower entropy ones\. Both are read against efficiency, or FLOPs\. Abstentions count as incorrect unless stated otherwise\. Full details of the evaluations are in Appendix[B](https://arxiv.org/html/2609.00103#A2)\.

We do not evaluate any closed\-source models on these splits because we view efficiency as a key part of memory and we cannot calculate how much computation they use\. However, we do evaluate them on an even more compressible subset and replicate our main findings in Appendix[A\.2](https://arxiv.org/html/2609.00103#A1.SS2)\.

#### 4\.2\.1Text

##### VLMs have low capacity on text, and compress their memories\.

The results of our experiments on text are in Figure[3](https://arxiv.org/html/2609.00103#S4.F3)\. On low\-entropy text, most VLMs are highly accurate\. However, their performance drops precipitously on the high\-entropy sequences, with only three VLMs scoring above chance, which we define throughout as at least60%60\\%accuracy against a50%50\\%chance rate \(Appendix[B\.4](https://arxiv.org/html/2609.00103#A2.SS4)\)\.

##### VLMs compress by increasing accuracy, not efficiency\.

There are two ways to take advantage of regularity: higher accuracy for the same amount of computation, or the same accuracy for less compute\. Of the three models above chance on both entropy sets, only InternVL3\.5 \(38B\) reaches similar accuracy at fewer FLOPs on the low entropy set\. Other models fail to match their 60%\+ accuracy on regular data when evaluated on irregular data, regardless of the amount of compute tested\.

##### High capacity implies compression in our tests, but not the reverse\.

Nearly all models with high capacity, or accuracy on max\-entropy streams, are also strong compressors\. However, the reverse is not true; for instance, Qwen3\-VL \(8B\) is barely above chance, at60%60\\,\\%, on max\-entropy data atL=1024L\{=\}1024\.

![Refer to caption](https://arxiv.org/html/2609.00103v1/images/flops_acc_text_L512_L1024.png)Figure 3:On synthetic text, several models compress their memories and a few have high capacity\.Accuracy vs\. FLOPs atL=512L\{=\}512\(left\) andL=1024L\{=\}1024\(right\) symbols\. Blue solid: low\-entropy; orange dashed: max\-entropy\. The compression values can be read as the gap above the orange line and below the blue line at a given FLOPs value; when the orange line is above, this represents negative compression\. Right\-edge integers: final accuracy per condition\.

#### 4\.2\.2Video

##### VLMs do not compress their memory on synthetic video\.

Our results for synthetic video are in Figure[4](https://arxiv.org/html/2609.00103#S4.F4)\. Most models perform only as accurately as chance at both entropy levels, and the models that are performant gain little from structure\. Only one VLM, InternVL3\.5 Thinking \(30B, A3B\), has a gap larger than ten points, although the retrieval\-based system we tested \(Appendix[B\.7](https://arxiv.org/html/2609.00103#A2.SS7)\) compresses atL=64L=64\. Capacity is also much lower on average as compared to text\.

![Refer to caption](https://arxiv.org/html/2609.00103v1/images/flops_acc_video_L16_L64.png)Figure 4:On synthetic video, accuracy falls from 16 to 64 frames; at 64 frames only GLM\-4\.5V stays above 70%, while most VLMs fall near chance and gain nothing from predictable structure\.Accuracy vs\. FLOPs atL=16L\{=\}16\(left\) andL=64L\{=\}64\(right\) frames\. Blue solid: low\-entropy; orange dashed: max\-entropy\. Right\-edge integers: final accuracy per condition\. Models with low accuracy on a short number of frames are not plotted to ensure we only test models with stronger perception \(Appendix[B\.5](https://arxiv.org/html/2609.00103#A2.SS5)\); complete numbers are in Appendix[B\.4](https://arxiv.org/html/2609.00103#A2.SS4)\.
##### Models do not compress natural video\.

Our results for natural video are in Figure[5](https://arxiv.org/html/2609.00103#S4.F5)\. Similar to synthetic video, only a single VLM, InternVL3\.5 \(8B\), demonstrates a strong ability to compress, although it is a different model than InternVL3\.5 \(38B\) that compresses text\. The complete per\-model contrasts are in Appendix Table[8](https://arxiv.org/html/2609.00103#A2.T8)\. Many VLMs, in fact, show anti\-compression, or a higher accuracy on dense videos than on sparser ones\. Part of the reason for this is the bands’ label priors, since the low band has fewer yes answers and some models have a bias towards no, but balanced accuracy, which weights yes and no questions equally, only moves the mean contrast, low\-density accuracy minus high\-density accuracy averaged over models, from\+0\.45\+0\.45to\+1\.34\+1\.34points, which is very weak compression\.

![Refer to caption](https://arxiv.org/html/2609.00103v1/images/natvid_density_by_flops.png)Figure 5:On natural video, accuracy is higher than synthetic video but memory compression is nearly absent\.Accuracy against FLOPs per response, one row per model and one column per clip length\. Blue solid: low density; orange dashed: high density\. Each curve is the running fraction of the cell answered correctly as its responses are spent cheapest first, so a wrong answer costs compute without changing the curve\. The right\-hand value is the cell’s accuracy at maximum compute\. FLOPs follow Appendix[B\.3](https://arxiv.org/html/2609.00103#A2.SS3)\.

### 4\.3Are VLMs calibrated?

Next, we ask if VLMs are calibrated and test this by repeating the prior experiments with an additional prompt that adds an explicit error cost\. Arm A gives a payoff of\+3\+3for a correct answer and−1\-1for an incorrect answer, so break\-even confidence should be at25%25\\,\\%\. Arm B gives \+1/−\-4, which raises it to80%80\\,\\%\. The arms differ only by one appended sentence and run on identical streams\. The payoffs and the exact prompts are in Appendix[B\.8](https://arxiv.org/html/2609.00103#A2.SS8)\.

Figure 6:VLMs are poorly calibrated on all modalities\.CoverageRRunder each arm, per model, for all three subsets\. Nearly all models are unaffected by the increased cost of an expensive answer, and no model abstains meaningfully more often\.Figure 7:VLMs only slightly optimize their payoffs\.Each model shown here scores at least60%60\\,\\%on its answered questions, and is asked to answer given a payoff scheme of\+3\+3/−1\-1\(arm A\) or\+1\+1/−4\-4\(arm B\)\.*Left*: does the model abstain more when a wrong answer costs four times as much?*Middle*: were the questions it abstained from on arm B the ones it got wrong on arm A?*Right*: both arms scored under arm B’s payoff of\+1\+1correct,−4\-4wrong,00for declining, with the higher of the two in bold\. The arm prompts are in Appendix[B\.8](https://arxiv.org/html/2609.00103#A2.SS8)\.##### Quadrupling the penalty for a wrong answer barely moves coverage\.

Despite being explicitly provided a large difference in cost \(the reward\-to\-penalty ratio has a twelvefold difference between the arms\), no model meaningfully changes its overall behavior \(Figure[6](https://arxiv.org/html/2609.00103#S4.F6)\)\. Given the number of models we evaluate, the few models that do abstain slightly more under higher cost may partly reflect natural variation in the data distribution\. In fact, many models actually abstain more when the expected reward of guessing is greater; thus they are not, in general,responsive\. Figure[7](https://arxiv.org/html/2609.00103#S4.F7)in its rightmost graph shows that many models received a lower expected payoff than if they had abstained on every question\.

##### Most models increase their payoff only slightly when told the payoff scheme\.

Although VLMs areoverallmiscalibrated, the little movement there is goes in the right direction\. In the rightmost column of Figure[7](https://arxiv.org/html/2609.00103#S4.F7), we evaluate every model’s responses using the payoff scheme for arm B\. For most models, the responses produced when the model is told that the payoff scheme is arm B achieve a higher arm\-B payoff than the responses produced when the model is instead told that the payoff scheme is arm A\. Models generally adjust their behavior in the direction encouraged by the payoff scheme, but the resulting improvement in payoff is small\. The middle column shows that the questions a model drops under arm B are mostly ones it got wrong under arm A, so most models, but not all, areselective\.

### 4\.4Do other backbones have better memory than RoPE Transformers?

Our results in the prior section indicate that VLMs are missing several aspects of strong memory, especially on video\. We thus test whether architecture is one reason VLMs lack so much of what strong memory needs\. We use a token\-level version of ECCBench with the same task structure and vocabulary as the rest of the benchmark\. This is not a direct comparison with the much larger pretrained VLMs, because we train only on this task and use small models\. Nevertheless, it gives insight into the properties of each backbone’s memory and how they extend beyond the lengths they were trained on\.

Our goal is to identify which architectures can continue to compress efficiently on novel streams from a distribution they did not observe in their training data\. To do so, we train on max\-entropy streams ofL≤1024L\\leq 1024, and evaluate only on longer ones \(L=2048L\{=\}2048and40964096\) of both max\- and low\-entropy\. We evaluate architectural backbones spanning attention\-based, linear\-recurrent, state\-space, memory\-augmented, and recurrent families \(Appendix Table[11](https://arxiv.org/html/2609.00103#A3.T11); full setup in Appendix[C](https://arxiv.org/html/2609.00103#A3)\)\. Our metric of confidence is the output softmax score following prior work\([Guo et al\., 2017](https://arxiv.org/html/2609.00103#bib.bib14);[Wang, 2023](https://arxiv.org/html/2609.00103#bib.bib29)\), which allows us to trace its coverage versus accuracy\. The results forL=2048L\{=\}2048appear in Figure[8](https://arxiv.org/html/2609.00103#S4.F8), and forL=4096L\{=\}4096in Appendix Figure[10](https://arxiv.org/html/2609.00103#A3.F10)\.

##### Several models compress despite training only on incompressible noise\.

Most surprising is that many models dynamically develop the ability to compress despite being trained only on incompressible noise\. We can see this for individual models with high compressibility, such as RetNet and TTT\-MLP, both with a gap of around 30% between peak low\-entropy accuracy and peak max\-entropy accuracy \(Figure[8](https://arxiv.org/html/2609.00103#S4.F8)\)\.

##### Compression cannot be measured for models near ceiling\.

Note that the definition of compression capability isC𝒟​\(L,F\)−C𝒟max​\(L,F\)C\_\{\\mathcal\{D\}\}\(L,F\)\-C\_\{\\mathcal\{D\}\_\{\\max\}\}\(L,F\)\. However, if both are nearly 100% accurate, it is difficult to measure compression because the difference is much smaller\. Memory Mosaic is so accurate that the difference on different splits is negligible \(Figure[8](https://arxiv.org/html/2609.00103#S4.F8)\), though there is a small difference onL=4096L\{=\}4096\(Appendix Figure[10](https://arxiv.org/html/2609.00103#A3.F10)\)\.

##### Non\-Transformer backbones compress better than the RoPE Transformer\.

We can read the two parts of Figure[8](https://arxiv.org/html/2609.00103#S4.F8)together to determine which models can compress by increasing the frontier of their coverage curve when processing low\-entropy streams\. Notably, RoPE Transformers have extremely low capacity, in line with the poor length extrapolation of rotary encoding\([Kazemnejad et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib61)\)\. Although replacing this positional encoding with a local convolution vastly increases capacity, their compression is outmatched by many models; DeltaNet, Test\-Time Training Layers, and Mamba, among others, have pronouncedly higher accuracy on the left graph\.

##### Non\-Transformer models are better calibrated than the Transformer\.

On both graphs in Figure[8](https://arxiv.org/html/2609.00103#S4.F8), RoPE Transformers lie near the bottom, and their coverage must be increased drastically before accuracy increases above fifty percent\. Although the Transformer with the convolution has a higher accuracy\-coverage curve, it spans a narrow coverage range, so lowering coverage increases accuracy very little\. In scenarios where incorrect answers are extremely expensive, it is sometimes desirable to abstain on nearly all queries except ones in which the memory contains a confident answer\. Lowering coverage buys very little for the Transformer as compared to models like the GRU, TTT\-MLP, or GLA, for instance\.

![Refer to caption](https://arxiv.org/html/2609.00103v1/images/backbone_calib_L2048_L4096_elow.png)Figure 8:Non\-Transformers generally show better properties than RoPE Transformers\.AtL=2048L\{=\}2048the non\-Transformer models stay accurate while still declining to answer when they are unsure—something the RoPE Transformer does not manage\. Memory Mosaic, TTT\-MLP, and Mamba\-2 define the low\-entropy Pareto frontier, but on the max\-entropy split a convolutional Transformer joins them\. The legend abbreviates:*Transf\. Conv4*and*Transf\. RoPE*are the Transformer with a small convolution and the Transformer with rotary encoding, and*Comp\. Transf\.*,*G\. DeltaNet*,*Mem\. Mosaic*,*Mamba*,*RWKV*and*TTT*are Compressive Transformer, Gated DeltaNet, Memory Mosaic, Mamba\-2, RWKV\-7 and TTT\-MLP\. Filled circles:D=64D\{=\}64; filled triangles:D=128D\{=\}128whereDDis the model width\. Left panel: low\-entropy split; right panel: max\-entropy split\.

## 5Conclusion

We defined three properties of strong memory beyond accuracy and developed a general\-purpose evaluation of memory that is applicable to VLMs, retrieval\-augmented models, and other intelligent systems\. Our evaluation results show that VLMs compress text but are weak compressors on video, with limited capacity, and they also lack the ability to adjust their abstention rates in response to explicit costs\. We also trained a range of memory backbones from scratch on the same task, and found that several non\-Transformer architectures handle sequences beyond their training length better than the RoPE Transformer does\. More broadly, this work provides a way to look at memory beyond aggregate accuracy, and helps understand which aspects of memory matter for downstream tasks\.

## References

- C\. An, S\. Gong, M\. Zhong, X\. Zhao, M\. Li, J\. Zhang, L\. Kong, and X\. QiuL\-Eval: instituting standardized evaluation for long context language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 14388–14411\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.776),[Link](https://aclanthology.org/2024.acl-long.776/)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Aroraet al\.\(2024a\)S\. Arora, S\. Eyuboglu, A\. Timalsina, I\. Johnson, M\. Poli, J\. Zou, A\. Rudra, and C\. RéZoology: measuring and improving recall in efficient language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2312\.04927Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Aroraet al\.\(2024b\)S\. Arora, S\. Eyuboglu, M\. Zhang, A\. Timalsina, S\. Alberti, D\. Zinsley, J\. Zou, A\. Rudra, and C\. RéSimple linear attention language models balance the recall\-throughput tradeoff\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235\.External Links:2402\.18668Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Aroraet al\.\(2024c\)S\. Arora, A\. Timalsina, A\. Singhal, B\. Spector, S\. Eyuboglu, X\. Zhao, A\. Rao, A\. Rudra, and C\. RéJust read twice: closing the recall gap for recurrent language models\.arXiv preprint arXiv:2407\.05483\.External Links:2407\.05483Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172),[Link](https://aclanthology.org/2024.acl-long.172/)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Behrouzet al\.\(2025a\)A\. Behrouz, M\. Razaviyayn, P\. Zhong, and V\. MirrokniNested learning: the illusion of deep learning architectures\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38\.Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Behrouzet al\.\(2025b\)A\. Behrouz, P\. Zhong, and V\. MirrokniTitans: learning to memorize at test time\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38,pp\. 113506–113543\.External Links:[Document](https://dx.doi.org/10.52202/085713-3786)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Choet al\.\(2014\)K\. Cho, B\. van Merriënboer, C\. Gulcehre, D\. Bahdanau, F\. Bougares, H\. Schwenk, and Y\. BengioLearning phrase representations using RNN encoder–decoder for statistical machine translation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1724–1734\.External Links:[Document](https://dx.doi.org/10.3115/v1/D14-1179)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px2)\.
- Dao and Gu \(2024\)T\. Dao and A\. GuTransformers are SSMs: generalized models and efficient algorithms through structured state space duality\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235,pp\. 10041–10071\.External Links:2405\.21060Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px3),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Dao \(2024\)T\. DaoFlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations \(ICLR\),External Links:2307\.08691,[Link](https://arxiv.org/abs/2307.08691)Cited by:[§3](https://arxiv.org/html/2609.00103#S3.SS0.SSS0.Px2.p1.1)\.
- Delétanget al\.\(2024\)G\. Delétang, A\. Ruoss, P\. Duquenne, E\. Catt, T\. Genewein, C\. Mattern, J\. Grau\-Moya, L\. K\. Wenliang, M\. Aitchison, L\. Orseau, M\. Hutter, and J\. VenessLanguage modeling is compression\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=jznbgiynus)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Duet al\.\(2025\)J\. Du, W\. Sun, D\. Lan, J\. Hu, and Y\. ChengMoM: linear sequence modeling with mixture\-of\-memories\.External Links:2502\.13685,[Link](https://arxiv.org/abs/2502.13685)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px11),[§1](https://arxiv.org/html/2609.00103#S1.p1.1)\.
- El\-Yaniv and Wiener \(2010\)R\. El\-Yaniv and Y\. WienerOn the foundations of noise\-free selective classification\.Journal of Machine Learning Research11,pp\. 1605–1641\.External Links:[Link](https://www.jmlr.org/papers/v11/el-yaniv10a.html)Cited by:[§3](https://arxiv.org/html/2609.00103#S3.SS0.SSS0.Px4.p1.1)\.
- Fuet al\.\(2025\)C\. Fu, Y\. Dai, Y\. Luo, L\. Li, S\. Ren, R\. Zhang, Z\. Wang, C\. Zhou, Y\. Shen, M\. Zhang, P\. Chen, Y\. Li, S\. Lin, S\. Zhao, K\. Li, T\. Xu, X\. Zheng, E\. Chen, C\. Shan, R\. He, and X\. SunVideo\-MME: the first\-ever comprehensive evaluation benchmark of multi\-modal LLMs in video analysis\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 24108–24118\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2025/html/Fu_Video-MME_The_First-Ever_Comprehensive_Evaluation_Benchmark_of_Multi-modal_LLMs_in_CVPR_2025_paper.html)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Gebruet al\.\(2021\)T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. Daumé III, and K\. CrawfordDatasheets for datasets\.Communications of the ACM64\(12\),pp\. 86–92\.External Links:[Document](https://dx.doi.org/10.1145/3458723),[Link](https://doi.org/10.1145/3458723)Cited by:[§A\.4](https://arxiv.org/html/2609.00103#A1.SS4.p1.1)\.
- Graveset al\.\(2014\)A\. Graves, G\. Wayne, and I\. DanihelkaNeural turing machines\.External Links:1410\.5401,[Link](https://arxiv.org/abs/1410.5401)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Greveet al\.\(2017\)A\. Greve, E\. Cooper, A\. Kaula, M\. C\. Anderson, and R\. HensonDoes prediction error drive one\-shot declarative learning?\.Journal of Memory and Language94,pp\. 149–165\.External Links:[Document](https://dx.doi.org/10.1016/j.jml.2016.11.001)Cited by:[Figure 9](https://arxiv.org/html/2609.00103#A1.F9.2),[Figure 9](https://arxiv.org/html/2609.00103#A1.F9.3),[§A\.2](https://arxiv.org/html/2609.00103#A1.SS2.p1.1)\.
- Gu and Dao \(2024\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.InFirst Conference on Language Modeling \(COLM\),External Links:2312\.00752,[Link](https://arxiv.org/abs/2312.00752)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2609.00103#S3.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§4\.4](https://arxiv.org/html/2609.00103#S4.SS4.p2.1)\.
- Honget al\.\(2025\)K\. Hong, A\. Troynikov, and J\. HuberContext rot: how increasing input tokens impacts LLM performance\.Technical reportChroma\.External Links:[Link](https://research.trychroma.com/context-rot)Cited by:[§A\.2](https://arxiv.org/html/2609.00103#A1.SS2.p2.1),[§1](https://arxiv.org/html/2609.00103#S1.p1.1)\.
- Hopfield \(1982\)J\. J\. HopfieldNeural networks and physical systems with emergent collective computational abilities\.Proceedings of the National Academy of Sciences79\(8\),pp\. 2554–2558\.External Links:[Document](https://dx.doi.org/10.1073/pnas.79.8.2554)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)Y\. Hu, Y\. Wang, and J\. McAuleyEvaluating memory in LLM agents via incremental multi\-turn interactions\.InInternational Conference on Learning Representations \(ICLR\),External Links:2507\.05257,[Link](https://openreview.net/forum?id=DT7JyQC3MR)Cited by:[§1](https://arxiv.org/html/2609.00103#S1.p1.1),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Kamradt \(2023\)G\. KamradtNeedle in a haystack – pressure testing LLMs\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)GitHub repositoryCited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Katharopouloset al\.\(2020\)A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. FleuretTransformers are RNNs: fast autoregressive transformers with linear attention\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.119,pp\. 5156–5165\.External Links:[Link](https://proceedings.mlr.press/v119/katharopoulos20a.html)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Kazemnejadet al\.\(2023\)A\. Kazemnejad, I\. Padhi, K\. Natesan Ramamurthy, P\. Das, and S\. ReddyThe impact of positional encoding on length generalization in transformers\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 24892–24928\.External Links:[Link](https://openreview.net/forum?id=Drrl2gcjzl)Cited by:[§C\.1](https://arxiv.org/html/2609.00103#A3.SS1.p2.1),[§4\.4](https://arxiv.org/html/2609.00103#S4.SS4.SSS0.Px3.p1.1)\.
- Köcheret al\.\(2025\)C\. Köcher, A\. Kozachinskiy, A\. W\. Lin, M\. Sälzer, and G\. ZetzscheNoPE: the counting power of transformers with no positional encodings\.arXiv preprint arXiv:2505\.11199\.Note:Cited from arXiv v1 \(the later v2 is retitled “The Counting Power of Transformers”\)\.External Links:2505\.11199,[Link](https://arxiv.org/abs/2505.11199v1)Cited by:[§C\.1](https://arxiv.org/html/2609.00103#A3.SS1.p2.1)\.
- Kontoyiannis \(1998\)I\. KontoyiannisAsymptotic recurrence and waiting times for stationary processes\.Journal of Theoretical Probability11\(3\),pp\. 795–811\.External Links:[Document](https://dx.doi.org/10.1023/A%3A1022610816550),[Link](https://doi.org/10.1023/A:1022610816550)Cited by:[§3](https://arxiv.org/html/2609.00103#S3.SS0.SSS0.Px3.p3.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2609.00103#S3.p3.1)\.
- Lianget al\.\(2023\)P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Ré, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. KoreedaHolistic evaluation of language models\.Transactions on Machine Learning Research \(TMLR\)\.External Links:2211\.09110Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Linet al\.\(2026\)J\. Lin, Z\. Fang, C\. Chen, Z\. Wan, F\. Luo, P\. Li, Y\. Liu, and M\. SunStreamingBench: assessing the gap for MLLMs to achieve streaming video understanding\.InIEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),External Links:[Link](https://arxiv.org/abs/2411.03628)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Longet al\.\(2026\)L\. Long, Y\. He, W\. Ye, Y\. Pan, Y\. Lin, H\. Li, J\. Zhao, and W\. LiSeeing, listening, remembering, and reasoning: a multimodal agent with long\-term memory\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=PMz29A7Muq)Cited by:[§1](https://arxiv.org/html/2609.00103#S1.p1.1),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating very long\-term conversational memory of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13851–13870\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.747)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Mangalamet al\.\(2023\)K\. Mangalam, R\. Akshulakov, and J\. MalikEgoSchema: a diagnostic benchmark for very long\-form video language understanding\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=JVlWseddak)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Oquabet al\.\(2024\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jegou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. BojanowskiDINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=a68SUt6zFt)Cited by:[§B\.6\.2](https://arxiv.org/html/2609.00103#A2.SS6.SSS2.Px2.p1.1)\.
- Packeret al\.\(2023\)C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. GonzalezMemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.External Links:[Link](https://arxiv.org/abs/2310.08560)Cited by:[§1](https://arxiv.org/html/2609.00103#S1.p1.1),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Penget al\.\(2025\)B\. Peng, R\. Zhang, D\. Goldstein, E\. Alcaide, X\. Du, H\. Hou, J\. Lin, J\. Liu, J\. Lu, W\. Merrill, G\. Song, K\. Tan, S\. Utpala, N\. Wilce, J\. S\. Wind, T\. Wu, D\. Wuttke, and C\. Zhou\-ZhengRWKV\-7 “goose” with expressive dynamic state evolution\.InSecond Conference on Language Modeling \(COLM\),External Links:2503\.14456Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px8),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Raeet al\.\(2020\)J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, C\. Hillier, and T\. P\. LillicrapCompressive transformers for long\-range sequence modelling\.InInternational Conference on Learning Representations \(ICLR\),External Links:1911\.05507,[Link](https://arxiv.org/abs/1911.05507)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px13),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Schlaget al\.\(2021\)I\. Schlag, K\. Irie, and J\. SchmidhuberLinear transformers are secretly fast weight programmers\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.139,pp\. 9355–9366\.External Links:2102\.11174Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px6)\.
- Sinhaet al\.\(2026\)A\. Sinha, A\. Arun, S\. Goel, S\. Staab, and J\. GeipingThe illusion of diminishing returns: measuring long horizon execution in LLMs\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=3lm8lWYxiq)Cited by:[§1](https://arxiv.org/html/2609.00103#S1.p1.1)\.
- Sumerset al\.\(2024\)T\. R\. Sumers, S\. Yao, K\. Narasimhan, and T\. L\. GriffithsCognitive architectures for language agents\.Transactions on Machine Learning Research \(TMLR\)\.External Links:2309\.02427Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Sunet al\.\(2025\)Y\. Sun, X\. Li, K\. Dalal, J\. Xu, A\. Vikram, G\. Zhang, Y\. Dubois, X\. Chen, X\. Wang, S\. Koyejo, T\. Hashimoto, and C\. GuestrinLearning to \(Learn at test time\): RNNs with expressive hidden states\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267,pp\. 57503–57522\.External Links:2407\.04620,[Link](https://arxiv.org/abs/2407.04620)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px10),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Sunet al\.\(2023\)Y\. Sun, L\. Dong, S\. Huang, S\. Ma, Y\. Xia, J\. Xue, J\. Wang, and F\. WeiRetentive network: a successor to transformer for large language models\.External Links:2307\.08621,[Link](https://arxiv.org/abs/2307.08621)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px5),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Tanet al\.\(2025\)H\. Tan, Z\. Zhang, C\. Ma, X\. Chen, Q\. Dai, and Z\. DongMemBench: towards more comprehensive evaluation on the memory of LLM\-based agents\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 19336–19352\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.989)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Tanget al\.\(2024\)J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. HanQuest: query\-aware sparsity for efficient long\-context LLM inference\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235\.External Links:[Link](https://arxiv.org/abs/2406.10774)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px1.p2.1)\.
- Wang \(2023\)C\. WangCalibration in deep learning: a survey of the state\-of\-the\-art\.External Links:2308\.01222,[Link](https://arxiv.org/abs/2308.01222)Cited by:[§4\.4](https://arxiv.org/html/2609.00103#S4.SS4.p2.1)\.
- Wanget al\.\(2025\)W\. Wang, Z\. He, W\. Hong, Y\. Cheng, X\. Zhang, J\. Qi, M\. Ding, X\. Gu, S\. Huang, B\. Xu, Y\. Dong, and J\. TangLVBench: an extreme long video understanding benchmark\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 22958–22967\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2025/html/Wang_LVBench_An_Extreme_Long_Video_Understanding_Benchmark_ICCV_2025_paper.html)Cited by:[§1](https://arxiv.org/html/2609.00103#S1.p1.1)\.
- Wanget al\.\(2024\)W\. Wang, S\. Zhang, Y\. Ren, Y\. Duan, T\. Li, S\. Liu, M\. Hu, Z\. Chen, K\. Zhang, L\. Lu, X\. Zhu, P\. Luo, Y\. Qiao, J\. Dai, W\. Shao, and W\. WangNeedle in a multimodal haystack\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Vol\.37\.External Links:2406\.07230,[Link](https://arxiv.org/abs/2406.07230)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Westonet al\.\(2015\)J\. Weston, S\. Chopra, and A\. BordesMemory networks\.InInternational Conference on Learning Representations \(ICLR\),External Links:1410\.3916,[Link](https://arxiv.org/abs/1410.3916)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongMemEval: benchmarking chat assistants on long\-term interactive memory\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=pZiyCaVuti)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024\)H\. Wu, D\. Li, B\. Chen, and J\. LiLongVideoBench: a benchmark for long\-context interleaved video\-language understanding\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Vol\.37,pp\. 28828–28857\.External Links:[Document](https://dx.doi.org/10.52202/079017-0907),[Link](https://openreview.net/forum?id=3G1ZDXOI4f)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2309.17453)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px3.p1.1)\.
- Xuet al\.\(2025\)M\. Xu, T\. Ao, J\. He, J\. Lu, G\. Shi, and M\. ZhengDeltaFormer: unlock the state space of transformer\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38,pp\. 171027–171057\.Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px9)\.
- Yanget al\.\(2025\)S\. Yang, J\. Kautz, and A\. HatamizadehGated delta networks: improving Mamba2 with delta rule\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:2412\.06464Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px7),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Yanget al\.\(2024a\)S\. Yang, B\. Wang, Y\. Shen, R\. Panda, and Y\. KimGated linear attention transformers with hardware\-efficient training\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235,pp\. 56501–56523\.External Links:2312\.06635Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px4)\.
- Yanget al\.\(2024b\)S\. Yang, B\. Wang, Y\. Zhang, Y\. Shen, and Y\. KimParallelizing linear transformers with the delta rule over sequence length\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37\.External Links:2406\.06484,[Link](https://arxiv.org/abs/2406.06484)Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px6)\.
- Zhanget al\.\(2025\)J\. Zhang, N\. Nolte, R\. Sadhukhan, B\. Chen, and L\. BottouMemory mosaics\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:2405\.06394Cited by:[§C\.2](https://arxiv.org/html/2609.00103#A3.SS2.SSS0.Px12),[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2024\)X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. K\. Hao, X\. Han, Z\. L\. Thai, S\. Wang, Z\. Liu, and M\. Sun∞\\inftybench: Extending long context evaluation beyond 100k tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 15262–15277\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.814),[Link](https://aclanthology.org/2024.acl-long.814/)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2025\)Z\. Zhao, H\. Lu, Y\. Huo, Y\. Du, T\. Yue, L\. Guo, B\. Wang, W\. Chen, and J\. LiuNeedle in a video haystack: a scalable synthetic evaluator for video MLLMs\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=ZJo6Radbqq)Cited by:[§2](https://arxiv.org/html/2609.00103#S2.SS0.SSS0.Px1.p1.1)\.
- Ziv and Lempel \(1978\)J\. Ziv and A\. LempelCompression of individual sequences via variable\-rate coding\.IEEE Transactions on Information Theory24\(5\),pp\. 530–536\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1978.1055934)Cited by:[§3](https://arxiv.org/html/2609.00103#S3.SS0.SSS0.Px3.p3.1),[§4\.1](https://arxiv.org/html/2609.00103#S4.SS1.SSS0.Px1.p1.1)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2609.00103#S1)
2. [2Related Work](https://arxiv.org/html/2609.00103#S2)
3. [3Background](https://arxiv.org/html/2609.00103#S3)
4. [4Experiments](https://arxiv.org/html/2609.00103#S4)1. [4\.1Data](https://arxiv.org/html/2609.00103#S4.SS1) 2. [4\.2Can VLMs compress?](https://arxiv.org/html/2609.00103#S4.SS2)1. [4\.2\.1Text](https://arxiv.org/html/2609.00103#S4.SS2.SSS1) 2. [4\.2\.2Video](https://arxiv.org/html/2609.00103#S4.SS2.SSS2) 3. [4\.3Are VLMs calibrated?](https://arxiv.org/html/2609.00103#S4.SS3) 4. [4\.4Do other backbones have better memory than RoPE Transformers?](https://arxiv.org/html/2609.00103#S4.SS4)
5. [5Conclusion](https://arxiv.org/html/2609.00103#S5)
6. [References](https://arxiv.org/html/2609.00103#bib)
7. [ABenchmark Details](https://arxiv.org/html/2609.00103#A1)1. [A\.1Entropy\-controlled generator](https://arxiv.org/html/2609.00103#A1.SS1) 2. [A\.2EasyHuman](https://arxiv.org/html/2609.00103#A1.SS2) 3. [A\.3Video rendering and question variants](https://arxiv.org/html/2609.00103#A1.SS3) 4. [A\.4Dataset documentation](https://arxiv.org/html/2609.00103#A1.SS4)
8. [BPretrained VLMs Experimental Details](https://arxiv.org/html/2609.00103#A2)1. [B\.1Evaluation prompts](https://arxiv.org/html/2609.00103#A2.SS1) 2. [B\.2Compute resources](https://arxiv.org/html/2609.00103#A2.SS2) 3. [B\.3Per\-model FLOPs equations](https://arxiv.org/html/2609.00103#A2.SS3) 4. [B\.4Per\-model synthetic accuracy](https://arxiv.org/html/2609.00103#A2.SS4) 5. [B\.5Short\-LLperception check](https://arxiv.org/html/2609.00103#A2.SS5) 6. [B\.6Natural\-video evaluation](https://arxiv.org/html/2609.00103#A2.SS6)1. [B\.6\.1Construction](https://arxiv.org/html/2609.00103#A2.SS6.SSS1) 2. [B\.6\.2Density bands](https://arxiv.org/html/2609.00103#A2.SS6.SSS2) 7. [B\.7RAG\-Pipeline for Video Retrieval](https://arxiv.org/html/2609.00103#A2.SS7) 8. [B\.8Calibration: the two payoff arms](https://arxiv.org/html/2609.00103#A2.SS8)
9. [CBackbone Architecture Experimental Details](https://arxiv.org/html/2609.00103#A3)1. [C\.1Configurations and training](https://arxiv.org/html/2609.00103#A3.SS1) 2. [C\.2FLOPs formulas and per\-architecture values](https://arxiv.org/html/2609.00103#A3.SS2) 3. [C\.3Calibration: coverage vs\. accuracy at fixed FLOPs](https://arxiv.org/html/2609.00103#A3.SS3)
10. [DLimitations and Broader Impact](https://arxiv.org/html/2609.00103#A4)

## Appendix ABenchmark Details

Our benchmark, ECCBench and its splits with Croissant metadata, is released under CC\-BY\-4\.0 at[https://huggingface\.co/datasets/anonstreammem/substream\-recollection](https://huggingface.co/datasets/anonstreammem/substream-recollection)\. The natural\-video manifests reference upstream EPIC\-Kitchens\-100 and SoccerNet sources under their original research\-use terms\.

### A\.1Entropy\-controlled generator

Streams are drawn over a fixed alphabet𝒱\\mathcal\{V\}with\|𝒱\|=16\|\\mathcal\{V\}\|=16symbols\. Each stream is produced by sampling a variable\-ordernn\-gram rule set \(rule lengths 1–8\) and resampling until the realized stream falls within a target entropy band\. Bands are set by the empirical Lempel–Ziv rateh^LZ\\hat\{h\}\_\{\\mathrm\{LZ\}\}of the realized sequence rather than by the generator’s stationary entropy, because what a model can exploit is the compressibility of the sequence itself\.

We use two entropy bands:

- •Low entropy:narrow rule sets with strong repetition\. Empirical meanh^LZ=0\.67\\hat\{h\}\_\{\\mathrm\{LZ\}\}=0\.67bits/symbol \(std0\.160\.16\), pooled across all evaluated lengthsL∈\{8,16,32,64,128,256,512,1024,2048,4096\}L\\in\\\{8,16,32,64,128,256,512,1024,2048,4096\\\}, observed range\[0\.33,0\.99\]\[0\.33,0\.99\]\.
- •Max\-entropy:symbols sampled i\.i\.d\. from the uniform distribution over𝒱\\mathcal\{V\}; we fix the band at exactlylog2⁡16=4\\log\_\{2\}16=4bits/symbol\.

A rule set is a small dictionary mapping contexts to distributions\. Each context is a string of one to eight symbols from𝒱\\mathcal\{V\}, meaning the generator is based on variable n\-grams\. For each context we specify up to four next\-symbol probabilities, and the remaining probability mass spreads uniformly across the other symbols\. As a concrete example, a two\-symbol context might assign probability 0\.8 to symbol8, probability 0\.18 to symbol13, and the leftover 0\.02 uniform over the remaining fourteen symbols\. As generation proceeds, at each position we find the longest suffix of the current prefix that matches some context in the rule set, then sample the next symbol from that context’s distribution\. When no context matches, we draw uniformly from𝒱\\mathcal\{V\}\. We resample rule sets until the full stream’s empirical LZ entropyh^LZ\\hat\{h\}\_\{\\mathrm\{LZ\}\}falls inside the target band\. We keep the first stream meeting this criterion\.

Each \(length, band\) cell contains 20 sequences, the same at every evaluated length\. Each sequence is paired with 16 membership queries, which are a roughly equal mix of true positives and true negatives\. Candidates are three to six symbols long\. A positive is a contiguous subsequence copied from a random position in the stream\. A negative is sampled uniformly and is kept only if it does not occur anywhere in it\. The video variant uses the same candidates rendered as frames\.

Two examples of length 16, low entropy \(h^LZ≈0\.7\\hat\{h\}\_\{\\mathrm\{LZ\}\}\\approx 0\.7\) first and max entropy \(log2⁡\|𝒱\|=4\.0\\log\_\{2\}\|\\mathcal\{V\}\|=4\.0\) second:

> 8, 1, 4, 6, 2, 1, 1, 2, 1, 1, 2, 1, 1, 2, 1, 1 7, 1, 13, 13, 5, 2, 7, 13, 11, 9, 8, 7, 0, 0, 15, 12

### A\.2EasyHuman

One of the motivations of our task is the desire to have human\-like memory\. We create a split, called EasyHuman, that asks whether models share a strength people are known to have: remembering a rare event that breaks a predictable pattern\[[Greve et al\., 2017](https://arxiv.org/html/2609.00103#bib.bib12)\]\. Each stream repeats a short pattern of length 3–5 with one or two short deviations, and queries target the deviation and corruptions of it \(Figure[9](https://arxiv.org/html/2609.00103#A1.F9); construction below\)\. EasyHuman is named for how obvious its redundancy is to a person, but we did not run a human study\.

![Refer to caption](https://arxiv.org/html/2609.00103v1/images/easyhuman_with_example.png)Figure 9:Prior work suggests people remember a single break in a repeating pattern well\[[Greve et al\., 2017](https://arxiv.org/html/2609.00103#bib.bib12)\]; models struggle as length grows\.The EasyHuman task: a short pattern repeats with one deviation\.Models partly compress text and do not compress video at all\.Results are shown in Table[1](https://arxiv.org/html/2609.00103#A1.T1)for text and video, whereLLdenotes the stream length \(in number of comma\-separated numbers for text streams, in frames for video streams\)\. Despite the extreme redundancy of the inputs, no open\-weights model reliably exceeds 77% accuracy in text, and video accuracy is near chance\. Text accuracy degrades only mildly fromL=256L=256toL=1024L=1024, consistent with mild context degradation\[[Hong et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib15)\]\. On video, models do very poorly and almost never abstain, which points to memory rather than perception for the models that pass the short\-video check \(Appendix[B\.5](https://arxiv.org/html/2609.00103#A2.SS5)\)\.

Frontier models also cannot compress video\.Claude Opus 4\.7, Gemini 3\.1 Pro and GPT\-5\.4 Pro, run at comparable reasoning budgets, all clear90%90\\,\\%on EasyHuman text at both lengths, well above any open\-weight model\. Still, none of them is distinguishable from chance on video\. They are the last three rows of Table[1](https://arxiv.org/html/2609.00103#A1.T1)and because they report no FLOPs they take no part in the compute\-aware analysis\.

Table 1:Models can compress text, but not videos\.EasyHuman accuracy \(%\) for text atL=256L\{=\}256andL=1024L\{=\}1024and video atL=256L\{=\}256\. Abstentions count as incorrect\. No video cell is distinguishable from chance, including the three frontier API models, which lead on text by seventeen points atL=256L\{=\}256and twenty\-one atL=1024L\{=\}1024\.##### Construction\.

The EasyHuman split consists of highly regular streams with a small number of deviations, and queries that focus on the deviation and its corruptions\.

We use the following structures\. In all cases, the deviations are chosen such that they do not already exist in the stream\.

- •Repeating Pattern\. A uniformly drawn motif of33–55symbols is repeated to fill the stream, with one or two deviations of the same length placed anywhere in it\.
- •Counting Disruption\. The base motif is a contiguous count by11between two integers, either ascending \(e\.g\.,2,3,4,52,3,4,5\) or descending \(e\.g\.,13,12,1113,12,11\); motif lengths are33–55symbols, and the motif is repeated to fill the stream\. Deviations are also33–55symbols long and are drawn from the uniform distribution\.

### A\.3Video rendering and question variants

Each symbol becomes one letter on one of three vertical conveyor belts \(Figure[2](https://arxiv.org/html/2609.00103#S4.F2)b\), rendered at448×448448\\times 448pixels and 1 frame per second, so a length\-LLstream is anLL\-frame,LL\-second video\.

##### Construction invariants that force the whole stream to be retained\.

It is important to note that while the entire video in pixel space need not be retained \(e\.g\., the texture of the conveyor belt is not asked about\), we can identify the exact properties of what must be remembered and quantify their information content\. Three properties of the renderer, enforced in the generation code, ensure that recovering𝒦⁡\(x\)\\mathcal\{K\}\(x\)requires the whole symbol stream\.*\(i\) One symbol, one frame,*in the main video, any prefix and every candidate alike, with no padding and no repeated glyphs\.*\(ii\) Candidates do not leak information:*a genuine candidate cannot be told from a fake one by its appearance, only by whether it occurred\.*\(iii\) Negatives are absent*from the rendered stream\.

##### Why letters in video and numbers in text\.

We use numbers \(0–15\) for text streams and letters for the video glyphs\. In pilot trials, VLMs read rendered letters off video frames more reliably than rendered digits at the same font size\. The text track has no such handicap because tokenizers process digits cleanly\. Keeping the underlying alphabet at 16 symbols, we picked the most readable rendering for each modality, and do not believe that changing these decisions would make a significant difference in our results\.

The synthetic video split has two question variants, weighted equally in every reported accuracy:

- •Sequential\.The candidate is a separate short video rendered with the same belt geometry, frame rate, and glyph style as the main stream\. A true positive sequential candidate is a contiguous subsequence of the main video with its symbols, belts,*and*original temporal order all preserved; the candidate is never a true subsequence shown in scrambled order\.
- •Spatial\.A true positive is identical to sequential, but a false answer will have a sequence in which the glyphs are correct but one of them appears on the wrong conveyor belt\.

The model receives the same prompt format in both variants\. The distinction lies only in how candidates are constructed and the types of incorrect answers\. The model must therefore decide whether a candidate appears as a subsequence either with or without order matching, without knowing in advance which is being asked\.

### A\.4Dataset documentation

A Croissant 1\.0metadata\.jsonin the HuggingFace release is the datasheet for ECCBench and covers the standard Datasheets\-for\-Datasets categories\[[Gebru et al\., 2021](https://arxiv.org/html/2609.00103#bib.bib10)\], including the Responsible\-AI fields\. The release also contains the manifest that is needed for our codebase: a top\-levelquestions\_dataset\.jsonindex and a per\-bucketmanifest\.json\. Errata and reproduction issues may be reported through the issue tracker of the code repository at[https://github\.com/princeton\-vl/ECCBench](https://github.com/princeton-vl/ECCBench), linked from the dataset card\. Corrections and expanded splits will appear in the HuggingFace revision history\.

## Appendix BPretrained VLMs Experimental Details

Here we document the per\-model evaluation protocol used in the VLM experiments \(Section[4](https://arxiv.org/html/2609.00103#S4)\)\. We describe the prompts \(Section[B\.1](https://arxiv.org/html/2609.00103#A2.SS1)\), the FLOPs measurement methodology \(Section[B\.3](https://arxiv.org/html/2609.00103#A2.SS3)\), the natural\-video evaluation \(Section[B\.6](https://arxiv.org/html/2609.00103#A2.SS6)\), and the two\-arm calibration protocol \(Section[B\.8](https://arxiv.org/html/2609.00103#A2.SS8)\)\.

### B\.1Evaluation prompts

We use the following three prompts\. The model may answer\{0\}for yes,\{1\}for no, or\{2\}for abstain\. Outputs are parsed by extracting the last bracketed token; if no bracketed answer is emitted within the model’s allowed token budget, the response is treated as an implicit abstention\.111The video prompt below uses the spelling “conveyer” rather than the standard “conveyor”; this misspelling was present in the prompt seen by every model\.

##### Prompt for video questions\.

```
"In the main video and option video, letters moved down three
 conveyer belts."
"A letter that appeared in any of the videos on a specific
 conveyer belt ALWAYS stayed on the same conveyer belt and
 only moved down, not left or right."
"The letters appeared sequentially. Only one was on the screen
 at a time."
"The letters continued moving down the conveyer belt until
 they left the screen."
"Examine the main video and the option video closely."
"Did this option video show events that exactly appeared
 SOMEWHERE in the main video, down to the order they appeared
 and which conveyer belt each letter was on?"
"Write {0} if yes, {1} if no, or {2} if unsure.
 Please put your answer in curly braces, e.g, {2}."
```

##### Motion language in the video prompt\.

The prompt describes letters as “moving down” the conveyor belts and continuing until they “left the screen\.” In the rendered videos used throughout this paper, each letter is visible in only one frame—the single frame in which it occupies its assigned belt—and the lane progression is implied by sequential frames rather than by any within\-frame motion \(Section[A\.3](https://arxiv.org/html/2609.00103#A1.SS3)\)\. Consistent with this, the prompt tells the model that only one letter is on screen at a time\.

##### Prompt for text questions\.

```
"You are given a main sequence of tokens and a candidate
 sequence."
"Your task: Determine if the candidate sequence appears as a
 contiguous subsequence within the main sequence, in the exact
 same order."
"The candidate must appear with all its tokens consecutive and
 in order - no gaps, no extra tokens in between."
"Answer {0} if the candidate appears as a contiguous
 subsequence, {1} if it does not, or {2} if you are unsure.
 Please put your answer in curly braces, e.g, {2}."
```

The main token sequence and the candidate sequence are supplied to the model alongside this prompt \(the candidate as a separateCandidate sequence:line\); the prompt text itself is fixed across all lengths and entropy bands\.

##### Prompt for natural\-video questions\.

The annotation’s question is inserted as written; abstention uses\{uncertain\}\.

```
Watch this video closely, paying attention to the entire video.
{question}

Write {0} if yes, {1} if no or {uncertain} if you are unsure.
Please put your answer in curly braces, e.g, {0}.
```

### B\.2Compute resources

All experiments ran on an internal cluster of A40, H200, L40, and RTX 3090 GPUs\. Pretrained VLM inference jobs allocated 2–7 GPUs per job depending on model size\. Each backbone uses a 36 wall\-clock\-hour training budget across four runs per architecture, on single\-GPU nodes \(Section[C](https://arxiv.org/html/2609.00103#A3)\)\. The full project consumed approximately 60,000 GPU\-hours\.

### B\.3Per\-model FLOPs equations

Every FLOPs axis in this paper is calculated analytically\. For each response we evaluate a formula whose inputs are the shape of the query \(how many frames or stream symbols the model read, and how long its answer was\) and the model’s architectural constants \(Table[4](https://arxiv.org/html/2609.00103#A2.T4)\)\. Nothing on these axes comes from a profiler, and no quantity is fitted to data\. The aim is a like\-for\-like comparison of how much arithmetic each architecture must perform to answer a query, so all models are priced under a single set of rules, given in full below\. A reader with this section alone can reproduce our numbers given our assumptions\.

##### Constants\.

Architectural constants are sourced from each model’s released checkpoint configuration\. Where a checkpoint’s code defaults disagree with the shapes of its trained weights, we use the shape of the actual weights\.

##### What counts as a FLOP\.

A matrix multiplication of shapes\(A,B\)⋅\(B,C\)\(A,B\)\\cdot\(B,C\)costs2​A​B​C2ABCFLOPs, two operations per multiply–add, and that is all we count\. Elementwise operations — normalization, softmax, activation functions, rotary position rotations, residual additions, embedding lookups — count as zero\.

##### The four cost terms\.

Although there are standard, simpler ways of counting per\-model FLOPs in LLMs, the vision encoder complicates these\. Additionally, the small differences in how these models chunk up data can make enormous differences on video data\. We thus decide to use analytical formulas for FLOPs computation which take these into account\. We can split the cost of a model into the vision encoder, the connector that maps vision features into the language model’s input space, the language model’s reading of the prompt \(prefill\), and its generation of the answer \(decode\):

Ftotal​\(L,Tin,Tout\)=Fvis\+Fconn\+Fpre\+Fdec,F\_\{\\mathrm\{total\}\}\(L,T\_\{\\mathrm\{in\}\},T\_\{\\mathrm\{out\}\}\)\\;=\\;F\_\{\\mathrm\{vis\}\}\\;\+\\;F\_\{\\mathrm\{conn\}\}\\;\+\\;F\_\{\\mathrm\{pre\}\}\\;\+\\;F\_\{\\mathrm\{dec\}\},forLLinput frames,TinT\_\{\\mathrm\{in\}\}prompt tokens andToutT\_\{\\mathrm\{out\}\}generated tokens\. Symbols:dv,fv,Lvd\_\{v\},f\_\{v\},L\_\{v\}are the vision transformer’s width, feed\-forward width and number of blocks;VVis the number of tokens entering one vision block;dt,Ltd\_\{t\},L\_\{t\}are the language model’s width and number of layers;nq,nk​vn\_\{q\},n\_\{kv\}its query and key/value head counts;hth\_\{t\}its head dimension, withdq=nq​htd\_\{q\}=n\_\{q\}h\_\{t\}anddk​v=nk​v​htd\_\{kv\}=n\_\{kv\}h\_\{t\}\(ht=128h\_\{t\}=128for every model we evaluate\);ftf\_\{t\}its feed\-forward width \(per expert, for mixture\-of\-experts models\);EEthe number of experts with top\-kkactive per token;VvocV\_\{\\mathrm\{voc\}\}the vocabulary size;Φ\\Phithe per\-token feed\-forward cost; andN=Tin\+VLLMN=T\_\{\\mathrm\{in\}\}\+V\_\{\\mathrm\{LLM\}\}the context the language model reads, whereVLLMV\_\{\\mathrm\{LLM\}\}is the number of vision tokens the connector emits\.

##### Vision encoder\.

Each frame is cut into square patches, and each patch becomes one token\. One vision block processingVVtokens, each attending overSSpositions, costs

Fblk​\(V,S\)=8​V​dv2⏟query, key, value, output projections\+4​V​S​dv⏟attention itself\+FFFN​\(V\),F\_\{\\mathrm\{blk\}\}\(V,S\)\\;=\\;\\underbrace\{8Vd\_\{v\}^\{2\}\}\_\{\\text\{query, key, value, output projections\}\}\\;\+\\;\\underbrace\{4VSd\_\{v\}\}\_\{\\text\{attention itself\}\}\\;\+\\;F\_\{\\mathrm\{FFN\}\}\(V\),FFFN​\(V\)=\{4​V​dv​fvtwo\-matrix feed\-forward,6​V​dv​fvthree\-matrix \(gated\) feed\-forward,F\_\{\\mathrm\{FFN\}\}\(V\)=\\begin\{cases\}4Vd\_\{v\}f\_\{v\}&\\text\{two\-matrix feed\-forward\},\\\\ 6Vd\_\{v\}f\_\{v\}&\\text\{three\-matrix \(gated\) feed\-forward\},\\end\{cases\}and the tower also pays once for turning raw pixels into patch features,2​Vpre​dpat​dv2V\_\{\\mathrm\{pre\}\}d\_\{\\mathrm\{pat\}\}d\_\{v\}\. HereVpreV\_\{\\mathrm\{pre\}\}is the number of patches before any class token is prepended, anddpatd\_\{\\mathrm\{pat\}\}is the number of pixel values per patch:3​pv23p\_\{v\}^\{2\}for patch sizepvp\_\{v\}, doubled where two frames are packed together\.

The spanSSrepresents how vision cost grows with video length\. In every model here it is bounded, whether a frame group, a tile or a crop:

- •Qwen2\.5\-VL, MIMO\-VL\.Two consecutive frames are packed into one temporal group ofV=1024V=1024tokens\. Four blocks attend across the whole group \(S=1024S=1024\); the remaining 28 attend within 64\-token windows \(S=64S=64\)\.
- •Qwen3\-VL, Qwen3\-VL Thinking, Qwen3\-Omni\.All 27 blocks attend within one two\-frame temporal group ofV=S=784V=S=784tokens\.
- •GLM\-4\.5V\.V=S=1024V=S=1024, one two\-frame temporal group\.
- •InternVL3\.5\.V=S=1025V=S=1025per4482448^\{2\}tile: 1024 patches plus one class token\.
- •MiniCPM\-V 2\.6 / 4\.5, LongVILA, Phi\-4\-MM\.V=S=1024V=S=1024per vision\-tower pass, which is one image crop; Phi\-4\-MM runs two crops per frame\.

Because each span is fixed, the number of spans grows in proportion to the number of frames and the vision term is*linear*inLLfor every model\.

##### How many tokens reach the language model\.

VLLMV\_\{\\mathrm\{LLM\}\}follows each model’s own preprocessing:

- •Qwen2\.5\-VL, MIMO\-VL, GLM\-4\.5V\.Two frames per temporal group; each2×22\\times 2block of patches is merged into one token, giving256256tokens per group and256​⌈L/2⌉256\\lceil L/2\\rceilin total\.
- •Qwen3\-VL, Qwen3\-Omni\.Two frames per group,2×22\\times 2merging over a28×2828\\times 28patch grid:196​⌈L/2⌉196\\lceil L/2\\rceil\.
- •InternVL3\.5\.One4482448^\{2\}tile per frame, then pixel\-unshuffle, which folds each2×22\\times 2block of patch features into a single wider feature:256​L256L\.
- •MiniCPM\-V 2\.6\.A resampler — a fixed set of6464learned query vectors that read the frame’s patches by cross\-attention — condenses each frame to6464tokens:64​L64L\.
- •MiniCPM\-V 4\.5\.The same resampler reads several frames at once\. The pack size grows with clip length asp=min⁡\(6,max⁡\(1,⌈L/180⌉\)\)p=\\min\\\!\\left\(6,\\,\\max\(1,\\lceil L/180\\rceil\)\\right\), and each call still emits6464tokens, soVLLM=64​⌈L/p⌉V\_\{\\mathrm\{LLM\}\}=64\\lceil L/p\\rceil\. The last pack is priced at its true size when fewer thanppframes remain\.
- •LongVILA\.Frames are projected to256256tokens each, then averaged in groups along time; each surviving group carries its256256tokens plus one separator token, so257257in all\. The group size is chosen from the clip length,pool⁡\(L\)=4⋅bucket⁡\(L/256\)\\mathrm\{pool\}\(L\)=4\\cdot\\mathrm\{bucket\}\(L/256\)wherebucket⁡\(⋅\)\\mathrm\{bucket\}\(\\cdot\)rounds up to the next power of two in\{1,2,4,8\}\\\{1,2,4,8\\\}, givingVLLM=257​⌊L/pool⁡\(L\)⌋V\_\{\\mathrm\{LLM\}\}=257\\lfloor L/\\mathrm\{pool\}\(L\)\\rfloor\.
- •Phi\-4\-MM\.Each frame yields one whole\-frame view and one cropped view, contributing545545tokens per frame after pooling and the row separators the model inserts\.

##### Connector\.

We account for the connector per token it emits\. Writingm=4​dvm=4d\_\{v\}for the width of a merged2×22\\times 2patch block: the Qwen2\.5\-VL, MIMO\-VL and LongVILA mergers cost2​m2\+2​m​dt2m^\{2\}\+2md\_\{t\}per token; GLM\-4\.5V’s merger follows its projection with a gated feed\-forward of width1094410944,2​m​dt\+2​dt2\+6​dt⋅109442md\_\{t\}\+2d\_\{t\}^\{2\}\+6d\_\{t\}\\cdot 10944; the Qwen3 towers run a main merger plus three DeepStack adapters — extra copies of the same merger that inject features from partway up the tower into early decoder layers — for four passes of2​m2\+2​m​dt2m^\{2\}\+2md\_\{t\}; InternVL3\.5 applies a two\-layer projection after pixel\-unshuffle,2​m​dt\+2​dt22md\_\{t\}\+2d\_\{t\}^\{2\}; Phi\-4\-MM projects and pools,2​dv​dt\+2​dt22d\_\{v\}d\_\{t\}\+2d\_\{t\}^\{2\}\. The MiniCPM resamplers are cross\-attention rather than projections: withKKpatch tokens entering a call that emits6464tokens, the cost is2​K​dv​dt\+4​K​dt2\+4⋅64​dt​K\+6⋅64​dt22Kd\_\{v\}d\_\{t\}\+4Kd\_\{t\}^\{2\}\+4\\cdot 64\\,d\_\{t\}K\+6\\cdot 64\\,d\_\{t\}^\{2\}\.

##### Reading the prompt \(prefill\)\.

Define the per\-token cost of one pass through the language model’s layers, excluding attention itself, and the coefficient multiplying attention:

B=Lt​\[2​dt​\(dq\+2​dk​v\)\+2​dq​dt\+Φ\],A=4​Lt​nq​ht\.B\\;=\\;L\_\{t\}\\Bigl\[\\,2d\_\{t\}\(d\_\{q\}\+2d\_\{kv\}\)\\;\+\\;2d\_\{q\}d\_\{t\}\\;\+\\;\\Phi\\Bigr\],\\qquad A\\;=\\;4L\_\{t\}n\_\{q\}h\_\{t\}\.ReadingNNtokens then costs

Fpre​\(N\)=B​N\+A​P​\(N\),P⁡\(N\)=12​N​\(N\+1\),F\_\{\\mathrm\{pre\}\}\(N\)\\;=\\;B\\,N\\;\+\\;A\\,P\(N\),\\qquad P\(N\)\\;=\\;\\tfrac\{1\}\{2\}N\(N\+1\),whereP⁡\(N\)P\(N\)counts the query–key pairs a causal model can use\.

##### Generating the answer \(decode\)\.

Generation reuses a cache of the keys and values that were already computed, so each new token costs one pass through the layers plus attention over the cache\. Reading the prompt already produces the distribution for the first generated token, so producingToutT\_\{\\mathrm\{out\}\}tokens applies the output vocabulary projectionToutT\_\{\\mathrm\{out\}\}times but runs the layers onlyTout−1T\_\{\\mathrm\{out\}\}\-1times, thett\-th of those attending overN\+tN\+tcached positions:

Fdec​\(N,Tout\)=Tout​Fvoc\+\(Tout−1\)​B\+A⁡\[\(Tout−1\)​N\+12​Tout​\(Tout−1\)\],Fvoc=2​dt​Vvoc,F\_\{\\mathrm\{dec\}\}\(N,T\_\{\\mathrm\{out\}\}\)\\;=\\;T\_\{\\mathrm\{out\}\}F\_\{\\mathrm\{voc\}\}\\;\+\\;\(T\_\{\\mathrm\{out\}\}\-1\)B\\;\+\\;A\\Bigl\[\(T\_\{\\mathrm\{out\}\}\-1\)N\+\\tfrac\{1\}\{2\}T\_\{\\mathrm\{out\}\}\(T\_\{\\mathrm\{out\}\}\-1\)\\Bigr\],\\qquad F\_\{\\mathrm\{voc\}\}=2d\_\{t\}V\_\{\\mathrm\{voc\}\},with the vocabulary projectionFvocF\_\{\\mathrm\{voc\}\}we count as part of decode\.

##### Per\-token feed\-forward cost\.

For models with one gated feed\-forward per layer,Φ=6​dt​ft\\Phi=6d\_\{t\}f\_\{t\}\. For mixture\-of\-experts models, where a small router scores the experts and only the topkkrun,

Φ=2​dt​E⏟router\+k⋅6​dt​ft\+1shared⋅6​dt​fs,\\Phi\\;=\\;\\underbrace\{2d\_\{t\}E\}\_\{\\text\{router\}\}\\;\+\\;k\\cdot 6d\_\{t\}f\_\{t\}\\;\+\\;\\mathbb\{1\}\_\{\\mathrm\{shared\}\}\\cdot 6d\_\{t\}f\_\{s\},the last term covering an always\-active shared expert where the checkpoint has one \(GLM\-4\.5V,fs=1408f\_\{s\}=1408; Qwen3\-Omni and InternVL3\.5 \(30B, A3B\) have none\)\. Two models are not uniform down their depth\. GLM\-4\.5V’s first decoder layer is an ordinary feed\-forward of width1094410944and its remaining4545layers are mixture\-of\-experts, so its body cost isLt​\[2​dt​\(dq\+2​dk​v\)\+2​dq​dt\]\+6​dt⋅10944\+\(Lt−1\)​ΦL\_\{t\}\\bigl\[2d\_\{t\}\(d\_\{q\}\+2d\_\{kv\}\)\+2d\_\{q\}d\_\{t\}\\bigr\]\+6d\_\{t\}\\cdot 10944\+\(L\_\{t\}\-1\)\\Phi\. Phi\-4\-MM’s vision LoRA adapters \(r=256r=256\) — small low\-rank matrices added alongside the main weights whenever an image is present — run at inference and are added toBBon all four fused text projections of every layer\.

##### The cost of a response\.

In our evaluation, we calculate the FLOPs necessary at the point of final commitment\. Because a model commits to an answer at the point its text emits an option marker, whatever it writes afterwards does not affect the decision the accuracy axis scores\. For a response ofToutT\_\{\\mathrm\{out\}\}tokens whose last marker falls at characteriiofℓ\\ellcharacters, we charge

Tans=max⁡\(1,Tout⋅iℓ\),T\_\{\\mathrm\{ans\}\}\\;=\\;\\max\\\!\\left\(1,\\;T\_\{\\mathrm\{out\}\}\\cdot\\frac\{i\}\{\\ell\}\\right\),andTans=Tout/2T\_\{\\mathrm\{ans\}\}=T\_\{\\mathrm\{out\}\}/2when no marker appears, as in a truncated response\.

##### Reading and writing costs\.

Recall in Section[3](https://arxiv.org/html/2609.00103#S3)we definedF=Fw\+FrF=F^\{\\mathrm\{w\}\}\+F^\{\\mathrm\{r\}\}as the total cost of memory\. Our accounting assumes a query occurs once per stream, and is independent of our actual inference setup\. Withqqqueries per stream the write side amortizes asFw/qF^\{\\mathrm\{w\}\}/q, and because the write dominates, this rescales every VLM almost uniformly\. For a VLM, which takes the whole stream into its context when the question arrives, the write cost of Section[3](https://arxiv.org/html/2609.00103#S3)is the prefill of Section[B\.3](https://arxiv.org/html/2609.00103#A2.SS3)and the read cost is the decode\.

Reading itself has two parts\. Most of the work, the projections and the feed\-forward, costs the same for every token no matter how much came before it, so it grows in proportion toNN\. We write that per\-token price asBB\. Attention, though, grows quadratically: the number of pairs the model has to score isP⁡\(N\)=12​N​\(N\+1\)P\(N\)=\\tfrac\{1\}\{2\}N\(N\+1\)\. We write the price of one such pair asAA\. Reading therefore costs

Fpre​\(N\)=B​N⏟grows with​N\+A​P​\(N\)⏟grows with​N2\.F\_\{\\mathrm\{pre\}\}\(N\)\\;=\\;\\underbrace\{B\\,N\}\_\{\\text\{grows with \}N\}\\;\+\\;\\underbrace\{A\\,P\(N\)\}\_\{\\text\{grows with \}N^\{2\}\}\.
In general, reading dominates, especially on video\. While there are some exceptions, notably reasoning models, many more tokens must be processed during the input than during the decode phase\. Table[2](https://arxiv.org/html/2609.00103#A2.T2)gives the three FLOPs price variables for every VLM we evaluate, and Table[3](https://arxiv.org/html/2609.00103#A2.T3)the writing and answering costs they produce; MF, GF and TF are millions, billions and trillions of operations\.

Table 2:The numbers we use to account for FLOPs from each VLM\.BBis what one token costs to push through the model’s layers without attention;AAis what one query–key pair costs, and attention pays it once for every pair;FvocF\_\{\\mathrm\{voc\}\}is the output projection to the vocabulary distribution\.Table 3:What reading and writing cost, for every VLM atL=64L\{=\}64frames with a three\-token answer\.NNis the size of the input the model must take in, in tokens\.
##### Checkpoint context lengths\.

AtL=1024L=1024for video, the contextNNexceeds the largest position the checkpoint was trained on for the three InternVL3\.5 models \(6\.4×6\.4\\times\), Phi\-4\-MM \(4\.3×4\.3\\times\), MiniCPM\-V 2\.6 and GLM\-4\.5V \(2\.0×2\.0\\times\), Qwen3\-Omni \(1\.5×1\.5\\times\), and marginally Qwen2\.5\-VL and MIMO\-VL \(1\.02×1\.02\\times\)\. The models that assign positions along separate time and space axes \(the Qwen family, GLM\-4\.5V\) advance their time index once per frame group, so their largest position index is still below the token count, whereas the InternVL and MiniCPM models index positions along a single axis and must extrapolate heavily outside of their given context length\.

Table 4:Constants entering the per\-response FLOPs formulas, read from each model’s released checkpoint configuration\.Lv,dv,fvL\_\{v\},d\_\{v\},f\_\{v\}: vision transformer depth, width, feed\-forward width\.Lt,dt,nq/nk​v,ftL\_\{t\},d\_\{t\},n\_\{q\}/n\_\{kv\},f\_\{t\}: language model depth, width, query / key\-value head counts, and feed\-forward width \(per expert where the model is mixture\-of\-experts\)\. “MoE” gives the expert countEEand the number active per token, with “\+sh” marking an additional always\-active shared expert\.VvocV\_\{\\mathrm\{voc\}\}is the vocabulary size, which is used only through the output projectionFvoc=2​dt​VvocF\_\{\\mathrm\{voc\}\}=2d\_\{t\}V\_\{\\mathrm\{voc\}\}\. “FFN style” gives the language model’s / the vision tower’s feed\-forward form: G = two matrices, SG = three matrices \(gated\)\.

### B\.4Per\-model synthetic accuracy

The figures in Section[4\.2](https://arxiv.org/html/2609.00103#S4.SS2)plot accuracy against compute, and Table[5](https://arxiv.org/html/2609.00103#A2.T5)gives the same accuracies as numbers, including models omitted from the figures for space or for scoring below51%51\\,\\%in every cell\. Abstentions count as wrong\.

Table 5:Per\-model synthetic accuracy \(%\) at the lengths plotted in Figures[3](https://arxiv.org/html/2609.00103#S4.F3)and[4](https://arxiv.org/html/2609.00103#S4.F4), both entropy levels, computed from exactly the rows those figures plot\. Abstentions count as wrong\. InternVL3\.5 \(38B\) was evaluated on synthetic video only atL=8L\{=\}8\(Table[6](https://arxiv.org/html/2609.00103#A2.T6)\)\.
### B\.5Short\-LLperception check

Most models can read short streams, so their collapse at longerLLis a memory failure\.In order to make sure that the models have the basic skills required to perform our task, we conduct a short perception check\. AtL=8L\{=\}8almost no memory is needed, and the overall cohort averages86\.8%86\.8\\,\\%on text against71\.071\.0and65\.1%65\.1\\,\\%on the two video variants \(Table[6](https://arxiv.org/html/2609.00103#A2.T6)\)\. We take each model’sL=8L\{=\}8score as its baseline and interpret the drop at longer lengths as the memory loss we study\.

Table 6:Raw accuracy \(%\) atL=8L\{=\}8on the low\-entropy band, for text and for both video question variants; abstentions count as wrong\. Qwen2\.5\-VL’s text score atL=8L\{=\}8comes from writing the yes and no markers swapped\.
### B\.6Natural\-video evaluation

#### B\.6\.1Construction

The natural\-video benchmark consists of binary membership questions over short clips drawn primarily from EPIC\-Kitchens\-100, with a smaller share from SoccerNet in the same yes/no/abstain answer format as the rest of ECCBench\.

##### Frame extraction at 1 fps\.

Every video is rendered as a sequence of frames extracted at 1 frame per second\. The frame count therefore equals the duration in seconds for videos at native speed\. Each model receives the full extracted frame sequence as input; we do not sub\-sample down to a fixed small number of frames before feeding the model—a common practice in long\-video benchmarks where a “long” video is presented as 8, 16, or 32 uniformly sampled frames regardless of source length\. Under that convention, a model never actually processes a long video; it processes a fixed\-budget summary of one\. However, under our protocol, a model evaluated atL=1024L\{=\}1024processes 1024 frames per video, not a sub\-sampled fraction of them\.

For most videos the source clip is at least as long as the target frame count in seconds\. For a small minority of videos in the longer buckets, the longest available source clip of the right event class is shorter than the target length\. In those cases we apply a uniform slowdown to the source clip—extracting the same content at sub\-real\-time speed so that 1 frame per second still yields the target frame count\. The slowdown factor is recorded per video\.

##### Length buckets\.

We create six buckets,L∈\{8,16,64,128,512,1024\}L\\in\\\{8,16,64,128,512,1024\\\}frames, holding 252 videos each atL∈\{8,16,64\}L\\in\\\{8,16,64\\\}, 110 each atL∈\{128,512\}L\\in\\\{128,512\\\}and 52 atL=1024L\{=\}1024; at 1 fps the native duration in seconds equals the frame count\. Bucket sizes shrink at long lengths because long EPIC\-Kitchens\-100 segments containing the chosen events are scarcer\. Per\-source composition: atL∈\{8,16,64\}L\\in\\\{8,16,64\\\}, 236 of 252 videos per bucket are from EPIC\-Kitchens\-100 and 16 from SoccerNet; atL∈\{128,512\}L\\in\\\{128,512\\\}, 100 of 110 are EPIC, 10 are SoccerNet; atL=1024L=1024, 42 of 52 are EPIC, 10 are SoccerNet\.

##### Slowdown distribution\.

L=8L=8andL=16L=16contain no slowed videos\.L=64L=64contains 2 slowed videos out of 252\.L=128L=128contains 3 of 110\.L=512L=512contains 15 of 110\.L=1024L=1024contains 42 of 52, with slowdown factors ranging from1\.2×1\.2\\timesto2\.9×2\.9\\times\.

##### Event categories\.

Each video is associated with one of eleven event categories:open\_fridge,open\_cupboard,open\_drawer,pour\_water,cut\_nontomato,rinse\_X,turn\_on\_tap,wipe\_surface,clean\_counter,wash\_plate, andred\_card\. The first ten are kitchen\-action categories sampled from EPIC\-Kitchens\-100 action annotations;red\_cardis sampled from SoccerNet event annotations\. Positive and negative video\-question pairs are balanced per category within each bucket\.

##### Question format\.

Each question takes the form*“Did the person \[action\] in this video?”*for kitchen events, or*“Did a red card appear in this video?”*for the SoccerNet category; the prompt is given in full in Section[B\.1](https://arxiv.org/html/2609.00103#A2.SS1)and the arm sentences in Section[B\.8](https://arxiv.org/html/2609.00103#A2.SS8)\. Ground\-truth labels come from the source annotations\.

#### B\.6\.2Density bands

Natural footage has no known ground\-truth entropy, so information density is measured by fixing a coding*rate*and reading off the resulting*distortion*\(Section[4\.1](https://arxiv.org/html/2609.00103#S4.SS1)\)\. We use a cutoff to assign each video to the low or the high band; every question asked about a clip inherits this designation, similarly to our synthetic set\.

##### Encoding\.

Each clip is re\-encoded with x264 at3232kbit per frame, at the corpus’s native640×360640\\times 360, forced to a constant11fps\. Rate control is two\-pass with no cap on the instantaneous rate, since single\-pass average\-bitrate control does not converge on clips this short and a cap forces undershoot\. The keyframe interval exceeds every clip length, so no periodic intra frame is inserted\. We kept scene\-cut detection enabled\. The preset isveryslowwith up to88B\-frames and1616reference frames, because temporal reach is the property under test\.

##### Distortion\.

We compute and compare distortion in the embedding space of DINOv2\[[Oquab et al\., 2024](https://arxiv.org/html/2609.00103#bib.bib60)\]\. Withztz\_\{t\}andz^t\\hat\{z\}\_\{t\}theℓ2\\ell\_\{2\}\-normalized patch\-mean DINOv2 embeddings of the original and the decoded framett, the per\-frame distortion is

δt=1−⟨zt,z^t⟩,\\delta\_\{t\}\\;=\\;1\-\\langle z\_\{t\},\\,\\hat\{z\}\_\{t\}\\rangle,and the clip score is the mean ofδt\\delta\_\{t\}over frames, dropping frame00, which is coded from scratch\.

##### Banding\.

We define a clip as low\-density if its score falls below a cutoff of0\.04010\.0401and high\-density otherwise\. Table[7](https://arxiv.org/html/2609.00103#A2.T7)gives the composition of each length and its distortion statistics\.

Table 7:Composition of the low\- and high\-density bands using the cutoff of0\.04010\.0401\. The yes\-rate is the percentage of examples with positive ground\-truth answers\.The rationale behind our metric for compressibility\.The metric should agree with intuitive notions of compressibility while extending to images that differ in pixel space but carry the same information for natural language questions\. For instance, duplicating frames in a video should score as more compressible on average \(relative to a video of the same length without duplicated frames\)\. The comparison in an embedding space fits these desiderata\.

VLMs do not show the consistent ability to compress natural video\.We show the contrast, defined as the accuracy on the low\-density split minus the high\-density split, in Table[8](https://arxiv.org/html/2609.00103#A2.T8)\. InternVL3\.5 \(8B\) does show a large contrast, but other models do not\. InternVL3\.5 \(8B\) was evaluated atL=64L\{=\}64andL=128L\{=\}128only\. The two bands also differ in label prior \(Table[7](https://arxiv.org/html/2609.00103#A2.T7)\), so we also score every cell with both answer classes weighted equally: the mean contrast moves from\+0\.45\+0\.45to\+1\.34\+1\.34points, positive in 21 of 41 cells\. InternVL3\.5 \(8B\)’s gap between low and high still survives at\+19\.0\+19\.0atL=128L\{=\}128\.

Table 8:Each cell is accuracy on the low\-density band minus accuracy on the high\-density band in percentage points\. Positive means the model did better on the compressible clips\. The signs are close to evenly split and the mean over all 41 cells is\+0\.45\+0\.45\.†\\daggermarks a cell where the model scores under5%5\\,\\%in both bands\. – marks combinations that were not evaluated\.

### B\.7RAG\-Pipeline for Video Retrieval

Our goal is to provide a general framework that is applicable to any system that reads a stream and later answers from it\. We use a RAG\-like model to show this\. To use it, each video is cut into 16\-frame windows at a stride of 4 and every window is embedded once with Qwen3\-VL\-Embedding\-8B\. At query time, the candidate is embedded the same way and then the top\-ranked window \(top three atL≥128L\{\\geq\}128\) is retrieved, overlapping windows are merged, and the composed short clip goes to Qwen3\-Omni \(30B, A3B\) with the same prompts as every baseline, plus one sentence disclosing that the clip was retrieved to provide some context for calibration\.

The embedder shares the Qwen3\-VL \(8B\) architecture, so it is priced with the same formulas as the other models\. Embedding one 16\-frame window costs32\.932\.9trillion operations\. Embedding a query costs0\.560\.56trillion on natural video, where the query is a sentence, and1111trillion on synthetic video, where the query is the candidate clip itself\. Qwen3\-Omni then answers as if on a clip of the retrieved length exactly as in Section[B\.3](https://arxiv.org/html/2609.00103#A2.SS3)\.

Table 9:The retrieval pipeline against full\-video Qwen3\-Omni\. We define the per\-answer cost as the query embed plus the inference call, with the index amortized away\. One query charges the whole index to a single question, which is our convention\. Break\-even is the number of questions a stream must serve before the pipeline’s total cost drops below re\-reading the stream per question\.Table[9](https://arxiv.org/html/2609.00103#A2.T9)shows three main results\. On natural video, the retrieval pipeline matches or exceeds the full\-video baseline in all three settings while reducing cost per answer by4\.7×4\.7\\times–42\.5×42\.5\\timesonce the index is amortized over more questions than the break\-even column, with larger savings for longer videos because the model reads only retrieved windows rather than the full stream\. On synthetic video, accuracy is3\.43\.4points below the baseline, showing that retrieval becomes the main bottleneck when many windows look similar\. For yes\-questions, if the retriever finds the relevant event, the model usually answers correctly, so yes accuracy \(36\.8%36\.8\\,\\%\) closely tracks the retriever hit rate \(33%33\\,\\%\)\. No\-questions are harder because the event is absent: the retriever must still return its closest match, and the model rarely rejects that window as irrelevant, causing no\-question accuracy to fall from64\.9%64\.9\\,\\%atL=64L\{=\}64to46\.8%46\.8\\,\\%atL=512L\{=\}512\. Intuitively, retrieval also makes calibration harder because even weak matches are presented as evidence of something matching and the embedder never abstains\. We theorize that providing retrieval similarity or confidence scores, as well as other metadata about the sequence, might help the model distinguish strong evidence from a merely best\-available match\.

### B\.8Calibration: the two payoff arms

##### Exact prompt modification\.

We append a new sentence to the original prompt in Section[B\.1](https://arxiv.org/html/2609.00103#A2.SS1)and nothing else changes\.\{X\}is whichever abstention token the base prompt already used—\{uncertain\}on natural video and\{2\}on both synthetic templates\. We take a model whose answering does not change between the arms to have ignored the cost of an error\.

```
arm A:
Scoring: a correct answer earns +3 points, a wrong answer
costs -1 point, and {X} earns 0 points. Answer only if you
are more than 25% confident; otherwise answer {X}.

arm B:
Scoring: a correct answer earns +1 point, a wrong answer
costs -4 points, and {X} earns 0 points. Answer only if you
are more than 80% confident; otherwise answer {X}.
```

##### All arm pairs\.

In Table[10](https://arxiv.org/html/2609.00103#A2.T10), we report the runs behind Section[4\.3](https://arxiv.org/html/2609.00103#S4.SS3)\. The main text reports a subset of these cells; all cells are listed here\. The*gap*column of Table[10](https://arxiv.org/html/2609.00103#A2.T10)asks whether the questions arm B dropped were the harder ones\. We sort arm A’s answers into kept and declined, count right and wrong in each group, and compute how likely a split at least this lopsided would be if arm B had declined without regard to difficulty \(a one\-sided Fisher test\); a smallppmeans the declined questions really were harder\. The sixteen questions about one stream tend to succeed or fail together, so treating them as independent overstates our certainty\. As a check, we redraw each cell4,0004\{,\}000times, picking as many streams as the cell has at random with replacement, keeping every question of each picked stream, and recomputing the gap; synthetic cells pool their two lengths and natural\-video cells are per length\.

Table 10:All calibration cells\.nnis the number of items; coverage is the fraction answered under each arm; accuracy is over answered items only\.*gap*is arm A’s accuracy on the questions it kept minus its accuracy on those arm B declined, shown only where arm B declined at least five questions arm A had answered;ppis a one\-sided Fisher test on that split\. The test and the stream\-level check are described above\.
##### Three ways a model can abstain\.

We are evaluating model behavior, and thus count a lack of an answer as an abstention, but three distinct behaviors are counted as abstention\.

*\(i\) Context overflow\.*We allow the models to respond for up to 8,192 tokens; if a model overflows this limit without answering, it counts as an abstention\.

*\(ii\) Format non\-compliance or no answer\.*If a model does not give an answer in the proper bracketed format, or gives no answer at all, we count it as an abstention\.

*\(iii\) Explicit abstention\.*The model may abstain explicitly by outputting the format corresponding to that option\.

## Appendix CBackbone Architecture Experimental Details

### C\.1Configurations and training

Table[11](https://arxiv.org/html/2609.00103#A3.T11)lists the architectural settings for the two embedding widths used in the backbone evaluation\.

Note that we also trained a Transformer with no positional encoding at all\[[Kazemnejad et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib61),[Köcher et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib18)\]\. It never learned the task at either width, staying at chance on the training distribution, so it is not reported\. We instead use a Transformer with a small convolution to evaluate the effects of RoPE\.

shared parametersModelheadsFFNconvArchitecture\-specificD=128D\{=\}128Compressive Transf\.4——block and memory length 128; compression\[8,4\]\[8,4\]at lengths\[32,32\]\[32,32\]—DeltaFormer41—chunked attention; qk\-norm;ropeθ=104\\text\{rope\}\_\{\\theta\}\{=\}10^\{4\}—DeltaNet414expandk=expandv=1\\text\{expand\}\_\{k\}\{=\}\\text\{expand\}\_\{v\}\{=\}1; L2 qk\-norm—Gated DeltaNet444expandv=2\\text\{expand\}\_\{v\}\{=\}2; gating; short\-conv preprocessing—GLA41—expandk=0\.5\\text\{expand\}\_\{k\}\{=\}0\.5\(dk=8d\_\{k\}\{=\}8\);expandv=1\\text\{expand\}\_\{v\}\{=\}1—Mamba\-2——4dstate=128d\_\{\\text\{state\}\}\{=\}128; expand 2dstate=256d\_\{\\text\{state\}\}\{=\}256Memory Mosaic4——block size 256; 6 persistent memories of size 128size 256MoM4——16 memories; top\-kk3; capacity 1\.5; Gated DeltaNet experts—RetNet414expandk=1\\text\{expand\}\_\{k\}\{=\}1;expandv=2\\text\{expand\}\_\{v\}\{=\}2; output gate—RWKV\-744———GRU———hidden width equals embedding width—Transformer \(RoPE\)41—causal; rotary encoding—Transformer \(NoPE \+ conv\)414no positional encoding; depthwise conv mixer—TTT\-MLP41—window 16; mini\-batch 256; gatemini\-batch 128Table 11:Backbone hyperparameters, evaluated on sequence lengths 2048 and 4096, beyond the 32–1024 training window\. Each architecture is three layers deep, has dropout 0, and uses a head dimension of 16\. Going fromD=64D\{=\}64toD=128D\{=\}128therefore doubles the width and the head count; the last column lists the settings that change beyond that\. A dash means the parameter does not apply to that architecture or is not counted\.##### Training\.

We train three\-layer versions of all architectures for up to thirty\-six wall\-clock hours\. For each model, we take the best of four training\-configuration runs and use that one for evaluation\. To simulate a streaming memory backbone, where the model is prepared to answer at any time step, we train on sequence lengths 32–1024\.

All jobs reuse the same synthetic membership dataset: 32 queries per sequence, 500,000 randomly generated sequences, and a vocabulary of 16 symbols\. We reserve 5 % of examples for validation, train with cosine LR schedules \(1 % warmup\), bf16\-mixed precision, and deterministic dataloading\. Batch sizes range from 12 to 128 and learning rates from 2e\-4 to 7e\-4, chosen per architecture and width \(theD=128D\{=\}128runs generally use 2\.5e\-4 and theD=64D\{=\}64runs 6e\-4–7e\-4\); batch size for the largest configurations is set by GPU memory\. We train for up to 125 epochs and stop once validation accuracy reaches 0\.97\. Per\-run values for every evaluated checkpoint are released with the configurations\.

### C\.2FLOPs formulas and per\-architecture values

The backbone accounting is kept separate from the VLM accounting, since these models have no vision tower and are compared only among themselves\. Elementwise operations, including normalization, RoPE, activation, and softmax, are omitted under our matmul\-FLOP convention\.

##### Transformer\.

This derivation covers both transformer rows\. Rotary positional encoding is elementwise and therefore free under this convention, so the RoPE Transformer and a no\-encoding Transformer of the same shape cost the same; the width\-4 causal depthwise convolution in the convolutional one adds8​D8DFLOPs per token per layer, or24​D​T24DTover three layers, about0\.16%0\.16\\,\\%of the total\.

Each layer contains causal multi\-head attention\[[Vaswani et al\., 2017](https://arxiv.org/html/2609.00103#bib.bib28)\]followed by anr=1r\{=\}1MLP\. Both use pre\-normalization and residual connections\. Each attention head has widthdh=D/Hd\_\{h\}=D/H, and causal attention containsP⁡\(T\)=T⁡\(T\+1\)/2P\(T\)=T\(T\+1\)/2valid query–key pairs\.

ForN=3N=3layers,

F=3⋅\[12​T​D2\+2​D​T​\(T\+1\)\]/106F=3\\cdot\[12TD^\{2\}\+2DT\(T\+1\)\]/10^\{6\}MFLOPs\.

##### GRU\[[Cho et al\., 2014](https://arxiv.org/html/2609.00103#bib.bib48)\]\.

Each layer is a GRU with hidden widthDD\. At every token, it computes input and recurrent projections for the reset gate, update gate, and candidate state\.

Therefore,

F=12​N​T​D2/106F=12NTD^\{2\}/10^\{6\}MFLOPs,

withN=3N=3\.

##### Mamba\-2\[[Dao and Gu, 2024](https://arxiv.org/html/2609.00103#bib.bib42)\]\.

Each layer is one Mamba\-2 block with:

di​n​n​e​r=2​Dd\_\{inner\}=2Dds​t​a​t​e=2​Dd\_\{state\}=2Dp=64p=64H=di​n​n​e​r/pH=d\_\{inner\}/pdi​n=8​D\+Hd\_\{in\}=8D\+H𝑐𝑜𝑛𝑣​\_​𝑑𝑖𝑚=6​D\\mathit\{conv\\\_dim\}=6D
Here,p=64p=64is the Mamba\-2 SSM head width\.

For recurrent inference:

For three layers,

F=3​T​\[2​D⋅di​n\+8⋅𝑐𝑜𝑛𝑣​\_​𝑑𝑖𝑚\+20​D2\]/106F=3T\[2D\\cdot d\_\{in\}\+8\\cdot\\mathit\{conv\\\_dim\}\+20D^\{2\}\]/10^\{6\}MFLOPs\.

##### GLA\[[Yang et al\., 2024a](https://arxiv.org/html/2609.00103#bib.bib43)\]\.

GLA hasH=D/16H=D/16heads,dk=8d\_\{k\}=8,dv=16d\_\{v\}=16, and a rank\-16 decay projection\.

F=3​c​T/106F=3cT/10^\{6\}MFLOPs\.

##### RetNet\[[Sun et al\., 2023](https://arxiv.org/html/2609.00103#bib.bib25)\]\.

RetNet hasH=D/16H=D/16,dk=16d\_\{k\}=16,dv=32d\_\{v\}=32, and a 16×\\times32 state per head\.

F=3​c​T/106F=3cT/10^\{6\}MFLOPs\.

##### DeltaNet\[[Schlag et al\., 2021](https://arxiv.org/html/2609.00103#bib.bib49),[Yang et al\., 2024b](https://arxiv.org/html/2609.00103#bib.bib34)\]\.

DeltaNet applies

vo​l​d=St−1𝖳​ktv\_\{old\}=S\_\{t\-1\}^\{\\mathsf\{T\}\}k\_\{t\}

and then writes the correction before reading the output\. Each head hasdk=dv=16d\_\{k\}=d\_\{v\}=16\.

q, k, v projections6​D26D^\{2\}output projection2​D22D^\{2\}beta projection D→\\toHD2/8D^\{2\}/8three width\-4 convolutions24​D24Drecurrent delta coreold\-value read32​D32Dcorrection write32​D32Doutput read32​D32Dtotal =6​H⋅dk⋅dv6H\\cdot d\_\{k\}\\cdot d\_\{v\}96​D96DSwiGLU MLP, I=2561536​D1536DPer layer per tokenc=8\.125​D2\+1656​Dc=8\.125D^\{2\}\+1656D
F=3​c​T/106F=3cT/10^\{6\}MFLOPs\.

##### Gated DeltaNet\[[Yang et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib44)\]\.

Each head hasdk=16d\_\{k\}=16,dv=32d\_\{v\}=32, and a 16×\\times32 state\. The learned global decay is elementwise and therefore not counted\.

F=3​c​T/106F=3cT/10^\{6\}MFLOPs\.

##### RWKV\-7\[[Peng et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib45)\]\.

RWKV\-7 maintains a 16×\\times16 state per head\. Its recurrent core requires four state\-sized matmul passes:

The complete count is:

F=T​ctot/106F=Tc\_\{\\mathrm\{tot\}\}/10^\{6\}MFLOPs\.

##### DeltaFormer\[[Xu et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib46)\]\.

Stage 1 performs strictly causal attention over previously corrected values; Stage 2 performs ordinary causal attention\.

For three layers,

F=3⋅Fl​a​y​e​r/106F=3\\cdot F\_\{layer\}/10^\{6\}MFLOPs\.

##### TTT\-MLP\[[Sun et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib26)\]\.

Each layer contains cached sliding\-window attention, a TTT\-MLP fast\-weight memory, and a ratio\-1 GELU MLP\. Each head stores a16→64→1616\\to 64\\to 16fast\-weight MLP\. Mini\-batch sizebbdetermines the point at which each token’s gradient is evaluated, but the fast weights conceptually update after every token\.

Let

PW​\(T\)=∑t=1Tmin⁡\(t,W\+1\)P\_\{W\}\(T\)=\\sum\_\{t=1\}^\{T\}\\min\(t,W\+1\)\.

ForT\>WT\>W,

PW​\(T\)=\(W\+1\)​T−W⁡\(W\+1\)/2P\_\{W\}\(T\)=\(W\+1\)T\-W\(W\+1\)/2\.

WithW=f=16W=f=16:

For three layers,

F=3⋅Fl​a​y​e​r/106F=3\\cdot F\_\{layer\}/10^\{6\}MFLOPs\.

##### MoM\[[Du et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib8)\]\.

Each token updates three routed memories and one shared memory\. Every memory uses a Gated DeltaNet state withdk=dv=16d\_\{k\}=d\_\{v\}=16\. The q/k/v, decay, and beta projections share weights and produce identical pre\-convolution features, so they are computed only once\. The four memories retain separate convolution and recurrent states\.

F=3​c​T/106F=3cT/10^\{6\}MFLOPs\.

##### Memory Mosaic\[[Zhang et al\., 2025](https://arxiv.org/html/2609.00103#bib.bib47)\]\.

Under cached inference, the two LeakyAvg operations reduce to elementwise EMA state updates and do not cost FLOPs\. We still count context attention as quadratic\.

context key, value, output projections6​T​D26TD^\{2\}persistent key and output projections4​T​D24TD^\{2\}context attentionT\(T−\-1\)/2 strictly causal pairs2​D​T​\(T−1\)2DT\(T\-1\)six persistent\-memory reads, M=2Dscore and value matmuls48​T​D248TD^\{2\}Per layerFl​a​y​e​r=58​T​D2\+2​D​T​\(T−1\)F\_\{layer\}=58TD^\{2\}\+2DT\(T\-1\)
For three layers,

F=3⋅Fl​a​y​e​r/106F=3\\cdot F\_\{layer\}/10^\{6\}MFLOPs\.

##### Compressive Transformer\[[Rae et al\., 2020](https://arxiv.org/html/2609.00103#bib.bib23)\]\.

Each layer processes the sequence in 128\-token blocks\. New tokens attend causally within the block and attend to thembm\_\{b\}past memory slots \(short\-term, compressed, and long\-term memories\); after each full block, maintenance compresses evicted slots and updates the long\-term memory\.

For a full block, letmbm\_\{b\}be the number of past memory slots available before blockbb\. The valid attention\-pair count is

Pb=128⋅mb\+128⋅129/2=128⋅mb\+8256P\_\{b\}=128\\cdot m\_\{b\}\+128\\cdot 129/2=128\\cdot m\_\{b\}\+8256\.

The warm\-up schedule is:

mb=0,128,144,160,164,…,192,193,193,…m\_\{b\}=0,128,144,160,164,\\dots,192,193,193,\\dots

Memory maintenance costs:

from block 28×\\timescompression plus cached K/V320​D2320D^\{2\}from block 44×\\timescompression plus cached K/V48​D248D^\{2\}from block 12long\-term update plus cached K/V32​D2\+16​D32D^\{2\}\+16D
The final one\-token block attends to 193 past slots and costs

48​D2\+1164​D48D^\{2\}\+1164D\.

Forn≥12n\\geq 12full blocks followed by one token,

F=3⋅\[∑b\(6144​D2\+6​D⋅Pb\)\+\(n−1\)⋅320​D2\+\(n−3\)⋅48​D2\+\(n−11\)⋅\(32​D2\+16​D\)\+48​D2\+1164​D\]/106F=3\\cdot\[\\sum\_\{b\}\(6144D^\{2\}\+6D\\cdot P\_\{b\}\)\+\(n\-1\)\\cdot 320D^\{2\}\+\(n\-3\)\\cdot 48D^\{2\}\+\(n\-11\)\\cdot\(32D^\{2\}\+16D\)\+48D^\{2\}\+1164D\]/10^\{6\}MFLOPs\.

Plugging the hyperparameters above, the 64/128\-width configs described in the backbone section, into these expressions reproduces the per\-query values in Table[12](https://arxiv.org/html/2609.00103#A3.T12)\.

Table 12:Backbone model computational complexity across embedding dimensions\. Implementation\-independent multiply\-add FLOPs for one causal forward pass of the backbone atT=L\+1T=L\+1; elementwise operations, normalization and softmax are free under this convention, and machinery that exists only to parallelize an exactly equivalent computation \(dense chunk matrices, parallel scans, triangular solves, dual\-form updates, padding\) is excluded\. These are the values used for the Pareto frontiers in Figures[8](https://arxiv.org/html/2609.00103#S4.F8)and[10](https://arxiv.org/html/2609.00103#A3.F10)\.

### C\.3Calibration: coverage vs\. accuracy at fixed FLOPs

Reading the plots: for a fixed MFLOPs budget, \(i\) capacity shifts the whole curve up and to the right, \(ii\) compression appears as a low\- vs\. max\-entropy gap at the same budget, and \(iii\) calibration appears when coverage drops on the max\-entropy stream before accuracy collapses—the model abstains instead of confidently guessing\.

![Refer to caption](https://arxiv.org/html/2609.00103v1/images/coverage_accuracy_L4096_low_high.png)Figure 10:Same coverage\-vs\-accuracy convention and legend abbreviations as Figure[8](https://arxiv.org/html/2609.00103#S4.F8), now atL=4096L=4096\. Both panels use FLOPs budgets≤\\leq30,000,≤\\leq10,000,≤\\leq1,750 MFLOPs \(left: low\-entropy split; right: max\-entropy split\)\. The≤\\leq30,000 tier admits all configurations\.

## Appendix DLimitations and Broader Impact

##### Limitations\.

Our study codifies a general test of memory, but we do not test all types of questions that cover memorization\. We omit natural text data, and while we do use natural video data, we draw the bulk of our conclusions from synthetic data\. ECCBench isolates associative recognition and not the full range of behaviors commonly called memory\. We do not test free\-form recall, temporal reasoning, interference between experiences, or whether remembered information can subsequently support complex reasoning\. ECC is intended to apply to these settings, but our experiments do not establish that the empirical findings transfer to them\.

We measure efficiency using arithmetic FLOPs and assume one query per stream\. FLOPs provide an implementation\-independent approximation to computation, but there are many other qualities that matter like memory bandwidth and communication costs\. Moreover, systems that amortize an expensive write over many subsequent queries may rank differently under a multi\-query workload\.

Our synthetic experiments are designed to remove other skills and priors because what must be retained is known exactly, but they still rely on the ability of the model to extract and retain that information effectively\. Thus, it is possible some models are able to store certain glyphs more compactly due to their priors, which would inflate their capacity score\.

Our compressibility measure for natural video is a heuristic, validated only on sanity checks such as slowed\-down clips scoring as more compressible\. We chose it because no established measure of compressibility in feature space, rather than pixel space, exists\.

We do not report complete results for closed\-source models because we cannot access how many FLOPs they use\. However, our definitions and axes of memory still apply to them\.

Our backbone study uses small models, which limits the ability to generalize to LLM\-scale, where models have billions of parameters\. At that scale, it is possible that other capabilities emerge, and that the differences we observe are more or less pronounced\. Some backbones may benefit from a wider hyperparameter search than the four configurations we tried\. Nonetheless, as many studies of small models do generalize to larger ones, we believe it will be useful in motivating more efficient memory\.

##### Broader impact\.

There have been few formal attempts to classify memory in a way that is comparable across contexts with diverse computational, certainty, and accuracy requirements\. We hope that measuring these properties explicitly will encourage the development of systems that use computational resources more effectively and are more willing to withhold an answer under uncertainty\. Better memory can make AI systems more capable and persistent, which may amplify both beneficial and harmful applications, and more study is needed into inducing aligned long\-horizon behavior\.

## Acknowledgments

This project was partially supported by the National Science Foundation\.