Uncheatable Eval：基于动态压缩的语言模型评估方法

# Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Source: [https://arxiv.org/html/2609.27510](https://arxiv.org/html/2609.27510)
Yudong LiAffiliation:Tsinghua UniversityLinlin ShenAffiliation:Shenzhen University

###### Abstract

Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results\. Reliable evaluation is particularly challenging for base models, whose limited instruction\-following ability complicates task\-based assessment\. We introduce*Uncheatable Eval*, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination\. Drawing on the relationship between a model’s predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text\. We evaluate 80 models across 14 text categories, study how compression changes with context length, and examine the correlation between compression rate and zero\-shot MMLU accuracy\. Our results yield three main findings: \(1\) compression performance follows a consistent scaling trend with model size; \(2\) attention\-based, hybrid, and recurrent models differ in how their compression performance changes as more context becomes available; and \(3\) lower compression rates are strongly associated with higher zero\-shot MMLU accuracy\. Code is available at[https://github\.com/Jellyfish042/uncheatable\_eval](https://github.com/Jellyfish042/uncheatable_eval)\.

*K*eywordslanguage model evaluation⋅\\cdotdata contamination⋅\\cdotdynamic benchmarks⋅\\cdotdata compression

![Refer to caption](https://arxiv.org/html/2609.27510v1/figures/main_fig.png)Figure 1:Overview of Uncheatable Eval\. Left: Data construction and compression\-rate evaluation\. Right: Compression rates of six representative models across five text types\.## 1Introduction

Effective evaluation is essential to the development of large language models\. However, the vast internet corpora used to train these models may contain benchmark questions, answers, and solutions\. Such contamination can inflate scores through memorization, making it difficult to distinguish genuine generalization from recall\. Longitudinal studies using model training cutoffs find evidence of this effect in code and mathematics data\[[1](https://arxiv.org/html/2609.27510#bib.bib30)\]; surveys and controlled studies show that contamination occurs at several semantic levels and is difficult to detect reliably\[[2](https://arxiv.org/html/2609.27510#bib.bib2),[3](https://arxiv.org/html/2609.27510#bib.bib7),[4](https://arxiv.org/html/2609.27510#bib.bib14)\]\.

Dynamic benchmarks reduce this risk by collecting newly published data or generating new test examples\. LiveBench and LiveCodeBench use recently released questions and code problems\[[5](https://arxiv.org/html/2609.27510#bib.bib3),[6](https://arxiv.org/html/2609.27510#bib.bib4)\]\. Other recent benchmarks continuously update their evaluation data to reflect changes in facts and knowledge\[[7](https://arxiv.org/html/2609.27510#bib.bib18),[8](https://arxiv.org/html/2609.27510#bib.bib24),[9](https://arxiv.org/html/2609.27510#bib.bib25)\]\. Newer test data reduce contamination risk, while task\-based evaluation can obscure base\-model capabilities because base models have limited instruction\-following ability\.

We introduce Uncheatable Eval, a dynamic benchmark that measures how well pretrained language models predict newly published text \(Figure[1](https://arxiv.org/html/2609.27510#S0.F1)\)\. Drawing on the relationship between prediction and lossless compression, we view each language model as a lossless compressor: its next\-token probabilities can guide arithmetic coding, assigning shorter codes to more probable continuations\[[10](https://arxiv.org/html/2609.27510#bib.bib29)\]\. A model that predicts the observed text more accurately therefore requires fewer bits to encode it\. We evaluate this ability through compression rate, calculated from the ideal code length relative to the original text size\. This directly evaluates base models on new text without task instructions or reference answers\.

To apply this approach across diverse forms of text, we collect newly published fiction, source code, scientific papers, news, and encyclopedia articles within specified time windows\. We then process these texts through a data curation pipeline that combines quality filtering, deduplication, and text normalization, with balanced sampling across languages for the multilingual subset\. Results for 80 models on the resulting datasets reveal a systematic relationship between model size and compression rate, well described by a power law with an additive constant\. A 54\-model analysis on longer scientific documents shows different changes in CR across attention\-based, hybrid, and recurrent models as more preceding text becomes available\. A comparison with zero\-shot MMLU further reveals a strong association between better compression of technical text and higher MMLU accuracy\.

Our main contributions can be summarized as follows:

1. 1\.We introduce Uncheatable Eval, a dynamic benchmark that evaluates base language models through compression of newly published text collected from diverse sources and curated using a data processing pipeline developed for the benchmark\.
2. 2\.We report 80 models spanning attention\-based, hybrid, and recurrent families across 14 text categories, providing a broad comparison of compression performance across model sizes and text domains\.
3. 3\.We analyze how CR varies with model size and document position and how it correlates with zero\-shot MMLU accuracy\.

## 2Related Work

#### Data contamination\.

Benchmark questions and answers can enter model training data and inflate test scores\[[3](https://arxiv.org/html/2609.27510#bib.bib7),[11](https://arxiv.org/html/2609.27510#bib.bib9),[12](https://arxiv.org/html/2609.27510#bib.bib16)\]\. Contamination ranges from exact copies to semantic overlap, so lexical matching alone is insufficient\[[2](https://arxiv.org/html/2609.27510#bib.bib2),[13](https://arxiv.org/html/2609.27510#bib.bib1)\]\. Comparisons across training cutoffs and controlled leakage experiments reveal score effects and detection limits\[[1](https://arxiv.org/html/2609.27510#bib.bib30),[14](https://arxiv.org/html/2609.27510#bib.bib21)\]\. Yet overlap does not have the same performance effect for every model, and inflated scores need not reorder leaderboards\[[15](https://arxiv.org/html/2609.27510#bib.bib17),[16](https://arxiv.org/html/2609.27510#bib.bib23)\]\. Other approaches detect contamination from model behavior\[[17](https://arxiv.org/html/2609.27510#bib.bib8),[4](https://arxiv.org/html/2609.27510#bib.bib14)\], score multiple contamination levels\[[18](https://arxiv.org/html/2609.27510#bib.bib13)\], or control screening errors\[[19](https://arxiv.org/html/2609.27510#bib.bib15)\]\. Proposed defenses vary test instances or interactions\[[20](https://arxiv.org/html/2609.27510#bib.bib20),[21](https://arxiv.org/html/2609.27510#bib.bib19),[22](https://arxiv.org/html/2609.27510#bib.bib11),[23](https://arxiv.org/html/2609.27510#bib.bib22)\], limit benchmark exposure\[[24](https://arxiv.org/html/2609.27510#bib.bib10)\], or design contamination\-resistant datasets\[[25](https://arxiv.org/html/2609.27510#bib.bib26)\]\. These studies motivate collecting new evaluation data and show that publication recency alone does not establish a clean test set\.

#### Dynamic benchmarks\.

Dynamic benchmarks regularly add new test data to reduce the risk of contamination\. LiveBench and LiveCodeBench collect recently released questions and programming problems\[[5](https://arxiv.org/html/2609.27510#bib.bib3),[6](https://arxiv.org/html/2609.27510#bib.bib4)\], while AntiLeakBench constructs questions using new real\-world knowledge\[[26](https://arxiv.org/html/2609.27510#bib.bib12)\]\. Related benchmarks test whether models can use updated evidence, track changing facts, and adapt to new knowledge\[[7](https://arxiv.org/html/2609.27510#bib.bib18),[8](https://arxiv.org/html/2609.27510#bib.bib24),[9](https://arxiv.org/html/2609.27510#bib.bib25)\]\.

FreshBench is the closest prior work to ours\. It measures the probabilities models assign to text published at different times and their accuracy in predicting future events\[[27](https://arxiv.org/html/2609.27510#bib.bib5)\]\. Its main question is how performance varies with the age of the data relative to a model’s release, including whether models perform worse on more recent data\. Uncheatable Eval also evaluates newly published text, but focuses on comparing base models through compression rate\. We also examine compression by model size and document position\.

#### Compression\-based evaluation\.

Language models can be evaluated directly by measuring how well they predict text\. Paloma measures perplexity across diverse domains under consistent evaluation conditions\[[28](https://arxiv.org/html/2609.27510#bib.bib6)\]\. Delétang et al\. explain the connection between prediction and lossless compression: a model that assigns higher probabilities to the observed text can encode it using fewer bits\[[10](https://arxiv.org/html/2609.27510#bib.bib29)\]\. Li et al\. use compression to evaluate models on text published after their training cutoffs and examine how performance changes across time periods\[[29](https://arxiv.org/html/2609.27510#bib.bib27)\]\. Huang et al\. find that models that compress text better also tend to score higher on knowledge, code, and mathematics benchmarks\[[30](https://arxiv.org/html/2609.27510#bib.bib28)\]\. Uncheatable Eval extends this approach to a regularly updated benchmark covering diverse text types and model families\.

## 3Uncheatable Eval

### 3\.1Data Collection and Curation

Uncheatable Eval collects newly published text for each evaluation to reduce the risk that models have already encountered the test data during training\. The collection covers fiction, news, encyclopedia articles, scientific papers, source code, and documentation from publicly available sources\. Table[1](https://arxiv.org/html/2609.27510#S3.T1)lists the evaluation categories;*Other*groups source\-code files outside the named language categories\.

The collected text is processed through a curation pipeline to improve quality and limit repetition\. Quality filters remove samples that are too short or contain anomalous character sequences, while near\-duplicate removal uses MinHash locality\-sensitive hashing to limit repeated content\. Unicode text is standardized using NFC normalization\. For the non\-English encyclopedia subset, sampling is balanced across languages to prevent the results from being dominated by a single language\.

Because tokenizers can assign different numbers of tokens to the same text, a length limit based on one tokenizer may leave samples too long for another\. We therefore check each sample with a diverse set of tokenizers and truncate it to meet a common token budget under all of them\. This produces a shared text sample for evaluation while accounting for differences in tokenization\.

Table 1:Evaluation categories and labels used in the result tables\.
### 3\.2Compression\-Based Evaluation

Uncheatable Eval measures how well a model predicts text by treating the model as a lossless compressor\. In arithmetic coding, the model’s next\-token probabilities determine how the text is encoded: more probable sequences require fewer bits\[[10](https://arxiv.org/html/2609.27510#bib.bib29)\]\. A model that predicts the observed text more accurately can therefore compress it more efficiently\. We measure this efficiency using the ideal code length implied by the model’s probabilities\.

Specifically, letssbe a document andx1:Tx\_\{1:T\}its token sequence under the evaluated model’s tokenizer\. The model predicts each token from the preceding tokens, assigning the document the probability

pθ\(x1:T\)=∏t=1Tpθ\(xt∣x<t\)\.p\_\{\\theta\}\(x\_\{1:T\}\)=\\prod\_\{t=1\}^\{T\}p\_\{\\theta\}\(x\_\{t\}\\mid x\_\{<t\}\)\.\(1\)The corresponding ideal code length, in bits, is

Bθ\(s\)=−log2pθ\(x1:T\)=−∑t=1Tlog2pθ\(xt∣x<t\)\.B\_\{\\theta\}\(s\)=\-\\log\_\{2\}p\_\{\\theta\}\(x\_\{1:T\}\)=\-\\sum\_\{t=1\}^\{T\}\\log\_\{2\}p\_\{\\theta\}\(x\_\{t\}\\mid x\_\{<t\}\)\.\(2\)Thus, code length is the next\-token negative log\-likelihood expressed in base 2, linking the evaluation directly to the pretraining objective\. It can be computed from the probabilities of the observed tokens without asking the model to follow instructions or generate an answer\. We use this theoretical code length throughout; it excludes finite\-precision coding overhead and the storage cost of model parameters\.

To compare documents of different lengths and models with different tokenizers, we normalize the code length by the original document’s UTF\-8 size\. Ifnbyte​\(s\)n\_\{\\mathrm\{byte\}\}\(s\)is the number of UTF\-8 bytes inss, its uncompressed size is8​nbyte​\(s\)8n\_\{\\mathrm\{byte\}\}\(s\)bits\. We define the compression rate \(CR\) as the percentage of this size needed for the ideal encoding:

CRθ​\(s\)=100⋅Bθ​\(s\)8​nbyte​\(s\),\\mathrm\{CR\}\_\{\\theta\}\(s\)=100\\cdot\\frac\{B\_\{\\theta\}\(s\)\}\{8n\_\{\\mathrm\{byte\}\}\(s\)\},\(3\)For example, a CR of 10% means that the ideal encoding occupies one tenth of the original UTF\-8 size\. Lower CR indicates better compression and, on the same text, better prediction\. The UTF\-8 denominator puts models on a common scale; each score reflects the model–tokenizer pair\.

For a collection of documents𝒮\\mathcal\{S\}, compression rate is the ratio of their total ideal code length to their total original size:

CRθ​\(𝒮\)=100⋅∑s∈𝒮Bθ​\(s\)8​∑s∈𝒮nbyte​\(s\)\.\\mathrm\{CR\}\_\{\\theta\}\(\\mathcal\{S\}\)=100\\cdot\\frac\{\\sum\_\{s\\in\\mathcal\{S\}\}B\_\{\\theta\}\(s\)\}\{8\\sum\_\{s\\in\\mathcal\{S\}\}n\_\{\\mathrm\{byte\}\}\(s\)\}\.\(4\)We use this byte\-weighted rate to summarize performance within each category\. To compare models across categories, we report an Overall score that takes the unweighted arithmetic mean of the category rates\. This gives each category equal weight, so categories containing more bytes do not dominate the overall comparison\.

### 3\.3Long\-Context Evaluation

Long\-context evaluation examines how compression changes as a model processes more of a document\. We retain the loss assigned to each token and align it to the token’s UTF\-8 bytes\. Letℓs,t=−ln⁡pθ​\(xs,t∣xs,<t\)\\ell\_\{s,t\}=\-\\ln p\_\{\\theta\}\(x\_\{s,t\}\\mid x\_\{s,<t\}\)be the token loss for documentss, and letms,tm\_\{s,t\}be the number of UTF\-8 bytes produced by that token\. We assignℓs,t/ms,t\\ell\_\{s,t\}/m\_\{s,t\}to each of its bytes\. The byte\-wise compression rate at positioniiis then

CRθ,i=1008​ln⁡2⋅1Ni∑s:i≤\|s\|ℓs,ts​\(i\)ms,ts​\(i\),\\mathrm\{CR\}\_\{\\theta,i\}=\\frac\{100\}\{8\\ln 2\}\\cdot\\frac\{1\}\{N\_\{i\}\}\\sum\_\{s:\\,i\\leq\|s\|\}\\frac\{\\ell\_\{s,t\_\{s\}\(i\)\}\}\{m\_\{s,t\_\{s\}\(i\)\}\},\(5\)wherets​\(i\)t\_\{s\}\(i\)is the token containing byteiiandNiN\_\{i\}is the number of documents that reach that position\. This construction compares the same byte positions across models with different tokenizers\.

## 4Experiments and Results

### 4\.1Experimental setup

#### Data and models\.

We evaluate 80 models on a July 2026 dataset with 14 text categories collected from publicly available sources\. Each category contains 500 samples, for a total of 7,000 samples\. During data preparation, each sample is truncated to at most 3,584 tokens under each of eight tokenizers, producing a shared text sample for all evaluated models\. Table[2](https://arxiv.org/html/2609.27510#S4.T2)presents 25 popular models, and Appendix[A](https://arxiv.org/html/2609.27510#A1)reports all 80\.

#### Evaluation metrics\.

We report compression rate \(CR\) as a percentage, with lower values indicating better compression\. Within each category, CR is computed from the total ideal code length and total UTF\-8 size using Eq\.[4](https://arxiv.org/html/2609.27510#S3.E4)\. For the scaling analyses, we instead combine code lengths and byte counts across categories before computing CR\. We also analyze five text types separately: Scientific Paper, Code, Encyclopedia, News, and English Fiction\. These contain five, five, two, one, and one categories, respectively\.

#### Long\-context evaluation\.

The main evaluation truncates each July 2026 document to at most 3,584 tokens under each tokenizer\. For long\-context evaluation, we retain up to 32,768 UTF\-8 bytes of the same documents\. We evaluate 54 models on 500 documents from each of four scientific\-paper categories: computer science, mathematics, physics, and other scientific papers\. All four categories contain documents reaching 32 KiB\. We compute byte\-wise CR within each category using Eq\.[5](https://arxiv.org/html/2609.27510#S3.E5); pooled curves weight each category by its number of contributing documents at each byte position\.

### 4\.2Main results

Table 2:Main results of Uncheatable Eval\. All score cells are compression rate \(CR, %\); lower is better\.ModelParams\.General textScientific PaperCodeOverallEFNewsEENEEBioCSMathSci\-OPhysC\+\+JSMDOtherPyOverall\>20BGemma\-4\-31B31\.39\.177\.466\.986\.096\.436\.595\.986\.166\.313\.163\.707\.863\.984\.156\.00Qwen3\.5\-35B\-A3B\-Base34\.79\.738\.257\.617\.496\.436\.525\.786\.096\.243\.333\.757\.924\.044\.246\.24Gemma\-4\-26B\-A4B25\.89\.527\.687\.226\.416\.626\.926\.226\.446\.563\.293\.878\.344\.174\.386\.26Mistral\-Small\-24B24\.09\.367\.857\.346\.826\.446\.796\.086\.306\.333\.444\.048\.394\.304\.496\.28Seed\-OSS\-36B\-Base36\.29\.618\.027\.488\.516\.526\.835\.976\.366\.573\.263\.898\.124\.074\.286\.39Nemotron\-3\-Nano\-30B\-A3B31\.69\.707\.967\.337\.406\.606\.846\.196\.436\.493\.444\.158\.494\.444\.506\.43OLMo\-3\-32B32\.29\.608\.057\.719\.246\.586\.876\.186\.426\.604\.094\.858\.885\.074\.916\.79∼\\sim14BRWKV7\-G1J\-13\.3B13\.39\.308\.017\.456\.876\.626\.686\.136\.326\.483\.594\.078\.144\.304\.426\.31Ministral\-3\-14B13\.99\.868\.227\.646\.996\.446\.806\.036\.336\.363\.554\.218\.604\.464\.656\.44∼\\sim7BQwen3\.5\-9B\-Base9\.010\.158\.608\.028\.246\.676\.766\.066\.346\.503\.654\.088\.284\.334\.516\.59Ministral\-3\-8B8\.910\.208\.427\.907\.346\.647\.026\.256\.546\.583\.714\.428\.934\.664\.846\.68RWKV7\-G1J\-7\.2B7\.29\.698\.347\.847\.486\.947\.056\.546\.676\.843\.914\.448\.754\.674\.816\.71Meta\-Llama\-3\.1\-8B8\.09\.898\.017\.667\.456\.837\.356\.366\.736\.773\.944\.739\.394\.885\.086\.79Qwen3\-8B\-Base8\.210\.218\.618\.198\.126\.927\.226\.156\.686\.803\.634\.338\.994\.574\.746\.80Falcon\-H1\-7B\-Base7\.610\.168\.338\.009\.096\.887\.236\.176\.656\.713\.964\.699\.414\.905\.126\.95marin\-8b\-base8\.09\.918\.027\.8610\.456\.917\.256\.446\.756\.754\.335\.179\.635\.255\.467\.16∼\\sim3BQwen3\.5\-4B\-Base4\.210\.659\.058\.498\.956\.997\.076\.416\.666\.833\.984\.438\.794\.664\.836\.98RWKV7\-G1J\-2\.9B2\.910\.028\.678\.228\.057\.287\.446\.967\.057\.234\.254\.869\.365\.055\.217\.12Ministral\-3\-3B3\.810\.898\.908\.418\.106\.987\.446\.696\.947\.004\.084\.879\.625\.085\.257\.16Falcon\-H1\-3B\-Base3\.110\.848\.878\.6510\.257\.447\.726\.647\.127\.224\.525\.2710\.275\.535\.687\.57<2<2BRWKV7\-G1J\-1\.5B1\.510\.569\.168\.768\.847\.717\.917\.467\.507\.694\.695\.3510\.105\.535\.677\.64Qwen3\.5\-2B\-Base1\.911\.519\.849\.2810\.097\.627\.757\.087\.307\.514\.675\.149\.825\.355\.527\.75Falcon\-H1\-1\.5B\-Deep\-Base1\.611\.279\.369\.1811\.337\.908\.247\.107\.587\.835\.075\.8811\.066\.166\.208\.16Qwen3\.5\-0\.8B\-Base0\.812\.5210\.8110\.2311\.478\.388\.507\.818\.018\.235\.425\.9410\.956\.126\.278\.62Falcon\-H1\-0\.5B\-Base0\.512\.9610\.3510\.5318\.959\.009\.298\.188\.588\.926\.207\.2713\.107\.547\.479\.88Table 2:Main results of Uncheatable Eval\. All score cells are compression rate \(CR, %\); lower is better\. \(continued\)Abbreviations: EF, English fiction; EE/NEE, English/non\-English encyclopedia; Bio, biology preprints; CS, computer science papers; Sci\-O, other scientific papers; JS, JavaScript; MD, Markdown; Py, Python\. Other denotes source code outside C\+\+, JavaScript, and Python\. Params\. gives total parameters in billions\. Overall is the unweighted mean of the 14 category CRs\. Within each parameter\-size group, bold marks the lowest displayed CR in each column, including ties\.

Table[2](https://arxiv.org/html/2609.27510#S4.T2)presents compression rates for 25 representative models across the 14 text categories, together with their Overall scores; Appendix[A](https://arxiv.org/html/2609.27510#A1)reports the complete results for all 80 models\. Qwen models\[[31](https://arxiv.org/html/2609.27510#bib.bib33),[32](https://arxiv.org/html/2609.27510#bib.bib34)\]are consistently strong on scientific papers and code: Qwen3\.5\-35B\-A3B\-Base leads several scientific categories, while Qwen3\.5 and Qwen3 remain among the strongest code compressors at their respective scales\. RWKV7\-G1J\[[33](https://arxiv.org/html/2609.27510#bib.bib31)\]shows the complementary pattern\. Its models perform particularly well on general text, including the non\-English encyclopedia category, where the family often leads similarly sized alternatives\.

Mixture\-of\-experts models also perform competitively with far fewer active parameters than total parameters\. Qwen3\.5\-35B\-A3B\-Base, Gemma\-4\-26B\-A4B\[[34](https://arxiv.org/html/2609.27510#bib.bib35)\], and Nemotron\-3\-Nano\-30B\-A3B\[[35](https://arxiv.org/html/2609.27510#bib.bib36)\]reach compression rates comparable to dense models with similar total sizes, and Qwen3\.5\-35B\-A3B\-Base approaches the best Overall score in the table\.

### 4\.3Long\-context evaluation

We evaluate 54 models on documents from four scientific\-paper categories and report the complete long\-context results in Appendix[B](https://arxiv.org/html/2609.27510#A2)\. We focus here on Qwen3\-8B\-Base, Falcon\-H1\-7B, and RWKV7\-G1J\-7\.2B as representatives of attention\-based, hybrid, and recurrent architectures, respectively\[[31](https://arxiv.org/html/2609.27510#bib.bib33),[36](https://arxiv.org/html/2609.27510#bib.bib32),[33](https://arxiv.org/html/2609.27510#bib.bib31)\]\. Figure[2](https://arxiv.org/html/2609.27510#S4.F2)compares the three models’ byte\-wise CR as document position increases from 1 to 32 KiB, both in absolute terms and relative to Qwen3\-8B\-Base\.

Figure 2:Long\-context compression for Qwen3\-8B\-Base, Falcon\-H1\-7B, and RWKV7\-G1J\-7\.2B on four scientific\-paper categories\. \(a\) Byte\-wise CR pooled over documents that reach each position, shown in logarithmically spaced bins from 1 to 32 KiB\. \(b\) CR differences from Qwen3\-8B\-Base\.Table 3:Long\-context slopes for similar\-sized models near 7–8B on four scientific\-document categories\.Figure[2](https://arxiv.org/html/2609.27510#S4.F2)shows that all three models achieve lower CR at later positions, but at different rates\. To quantify this trend, we summarize the byte\-wise rates in five non\-overlapping intervals: 1–2, 2–4, 4–8, 8–16, and 16–32 KiB\. Within each interval, we again weight every byte position by its document count\. We then fit

CR¯j=α\+β​log2​xj,\\overline\{\\mathrm\{CR\}\}\_\{j\}=\\alpha\+\\beta\\log\_\{2\}x\_\{j\},\(6\)whereCR¯j\\overline\{\\mathrm\{CR\}\}\_\{j\}is the weighted mean for intervaljjandxjx\_\{j\}is its geometric midpoint\. Ordinary least squares assigns equal weight to the five interval means\. The slopeβ\\betameasures the change in CR, in percentage points, for each doubling of byte position\. We exclude positions below 1 KiB, where little preceding context is available\. Table[3](https://arxiv.org/html/2609.27510#S4.T3)reports these slopes\.

Among the representative models at 7–8B, Falcon\-H1 improves most rapidly, followed by Qwen3 and RWKV7\-G1J, with slopes of−0\.454\-0\.454,−0\.426\-0\.426, and−0\.314\-0\.314, respectively\. Their CR decreases by 1\.807, 1\.700, and 1\.266 percentage points between the first and last fitted intervals\. This ordering holds in all four categories; each representative model improves most rapidly on mathematics and most slowly on computer science\.

Relative to Qwen3\-8B\-Base, Falcon\-H1\-7B shifts from 0\.025 CR points higher in the 1–2 KiB interval to 0\.082 points lower at 16–32 KiB\. RWKV7\-G1J\-7\.2B shifts from 0\.067 points lower to 0\.367 points higher\. RWKV’s short\-context advantage therefore decays as document position grows, even though its absolute CR continues to improve\. Our observation is consistent with von Oswald et al\.\[[37](https://arxiv.org/html/2609.27510#bib.bib38)\], who find that all evaluated linear language models outperform attention\-based models at early sequence positions but gradually lose this advantage as the sequence grows\.

To examine the role of training context, we additionally evaluate older RWKV7\-G1A\-1\.5B/2\.9B and RWKV7\-G0A2\-7\.2B/G0A3\-13\.3B checkpoints trained with 4,096\-token contexts, alongside matched\-size RWKV7\-G1J checkpoints trained with 16,384 tokens\. The short\-to\-long increase in RWKV’s CR gap relative to Qwen3 is smaller for G1J \(0\.33–0\.55 percentage points\) than for the older checkpoints \(0\.41–0\.65\), indicating that longer training contexts reduce relative decay in these comparisons\.

### 4\.4Scaling Laws for Compression Rate

We fit CR as a power law in total parameter countPP\(in billions\) with an additive constant, following a common scaling\-law form\[[38](https://arxiv.org/html/2609.27510#bib.bib37)\]:

CR⁡\(P\)=a​Pb\+c\.\\mathrm\{CR\}\(P\)=aP^\{b\}\+c\.\(7\)We estimateaa,bb, andccby minimizing squared errors\. The exponentbbcontrols how quickly CR declines with model size, whileccis the asymptotic CR asPPincreases\. For mixture\-of\-experts models,PPcounts all experts\. We fit the 80 models’ byte\-weighted CR across all 14 categories and separately for each text type\.

![Refer to caption](https://arxiv.org/html/2609.27510v1/group_scaling_curves.png)Figure 3:CR versus total parameters for 80 models\. Each panel shows a power\-law\-plus\-constant fit\. For each text type, CR is computed from the total code length and UTF\-8 bytes across its categories\.For the aggregate score, we obtain

CR^​\(P\)=3\.200​P−0\.290\+5\.328,R2=0\.915\.\\widehat\{\\mathrm\{CR\}\}\(P\)=3\.200P^\{\-0\.290\}\+5\.328,\\qquad R^\{2\}=0\.915\.\(8\)The fitted curve accounts for 91\.5% of the cross\-model CR variation and has a root mean squared error of 0\.293 percentage points\. The fitted CR curve flattens as model size grows \(Figure[3](https://arxiv.org/html/2609.27510#S4.F3)\)\.

Table 4:Power\-law\-plus\-constant fits to all 80 models by data type\. All types use byte\-weighted CR within the included categories\.Parameters correspond toCR⁡\(P\)=a​Pb\+c\\mathrm\{CR\}\(P\)=aP^\{b\}\+c, withPPmeasured in billions\. RMSE is measured across all 80 models in CR percentage points\.

Scaling strength varies by text type: code has the steepest exponent \(−0\.317\-0\.317\), while English fiction has the shallowest \(−0\.146\-0\.146\)\. The fits account for 72\.1%–89\.7% of the cross\-model variation \(Table[4](https://arxiv.org/html/2609.27510#S4.T4)\), showing that model size explains substantially more variation for some types than for others\.

At the category level, fit quality varies widely:R2R^\{2\}ranges from 0\.594 for non\-English encyclopedia articles to 0\.935 for biology preprints \(Appendix[C](https://arxiv.org/html/2609.27510#A3)\)\.

We also fit Eq\.[7](https://arxiv.org/html/2609.27510#S4.E7)to the parameter–CR Pareto frontier, retaining models for which no evaluated model has at most as many parameters and lower CR\. The all\-category frontier contains 20 models and followsCR^​\(P\)=3\.244​P−0\.304\+5\.016\\widehat\{\\mathrm\{CR\}\}\(P\)=3\.244P^\{\-0\.304\}\+5\.016\(R2=0\.992R^\{2\}=0\.992\)\. Across text types, the frontier exponent ranges from−0\.575\-0\.575for encyclopedia articles to−0\.244\-0\.244for English fiction\. These fits describe the best observed CR at each scale within our model pool; the frontier and its fitted asymptote may change as new models are evaluated \(Appendix[D](https://arxiv.org/html/2609.27510#A4)\)\.

### 4\.5Correlation with MMLU

We evaluate all 80 models on zero\-shot MMLU\. For each question, we choose the answer with the highest next\-token logit among A–D and compute accuracy over 14,042 questions\. We compare this accuracy with each model’s CR pooled across five Scientific Paper and four Code categories\.

We measure the association with Spearman correlation\. We obtain 95% confidence intervals by bootstrapping models 10,000 times for the pooled score and 5,000 times for each category\.

![Refer to caption](https://arxiv.org/html/2609.27510v1/mmlu_compression_correlation.png)Figure 4:Cross\-model correlations\. \(a\) Technical\-text CR versus zero\-shot MMLU accuracy for 80 base models\. \(b\) Pairwise Spearman correlations of residual CR across five text types and the same 80 models, after separate fits against log parameter count\.Across 80 models, lower technical\-text CR is associated with higher zero\-shot MMLU accuracy \(Spearmanρ=−0\.884\\rho=\-0\.884, 95% confidence interval\[−0\.942,−0\.783\]\[\-0\.942,\-0\.783\]; Figure[4](https://arxiv.org/html/2609.27510#S4.F4)\(a\)\)\. The correlation is similar when CR is pooled across all 14 categories \(ρ=−0\.871\\rho=\-0\.871\), and each of the nine technical categories shows the same pattern \(ρ=−0\.883\\rho=\-0\.883to−0\.853\-0\.853\)\.

### 4\.6Correlation analyses across text types

We compute pairwise Pearson correlations among the five text types across 80 models\. All ten correlations are positive, ranging fromr=0\.803r=0\.803to0\.9690\.969\. Scientific Paper and Code correlate most strongly \(r=0\.969r=0\.969\), followed by News and English fiction \(r=0\.963r=0\.963\)\.

We then fit each type’s CR against Overall CR and examine model residuals\. A negative residual means better compression on that type than the model’s Overall score predicts\.The cross\-type residual correlations reveal two strongly related pairs: Scientific Paper and Code \(ρ=0\.838\\rho=0\.838\), and News and English fiction \(ρ=0\.812\\rho=0\.812; Figure[4](https://arxiv.org/html/2609.27510#S4.F4)\(b\)\)\. Encyclopedia lies between them, correlating with both pairs but more strongly with News and English fiction \(ρ=0\.484\\rho=0\.484–0\.5500\.550\) than with Scientific Paper and Code \(ρ=0\.315\\rho=0\.315–0\.4120\.412\)\. Code and News show almost no residual correlation \(ρ=−0\.015\\rho=\-0\.015\)\.

## 5Conclusion

We introduce Uncheatable Eval, a dynamic benchmark that evaluates base language models through lossless compression of newly published text\. By converting next\-token probabilities into code lengths, it measures predictive ability without task instructions or reference answers\. Across 80 models and 14 text categories, CR declines with model size and varies across text types\. Our long\-context analysis shows that attention\-based, hybrid, and recurrent models differ in how their CR changes as more context becomes available\. Lower CR on technical text is also strongly associated with higher zero\-shot MMLU accuracy\. These results support compression of newly published text as a direct way to compare base models across domains\.

## 6Limitations

Although newly published text reduces contamination risk, we cannot guarantee its absence from training: publication dates and model training cutoffs may be incomplete, and models may have seen copies of the same content\. Our raw\-text prediction protocol also limits evaluation to base models, because compression rate does not capture the instruction\-following abilities acquired through post\-training\. Moreover, this contamination risk grows as the dataset ages: new models may train on its public texts, so we must refresh the evaluation data regularly\.

## References

- \[1\]M\. Roberts, H\. Thakur, C\. Herlihy, C\. White, and S\. Dooley\(2023\)Data contamination through the lens of time\.External Links:2310\.10628,[Link](https://arxiv.org/abs/2310.10628)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p1.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]C\. Xu, S\. Guan, D\. Greene, and M\. Kechadi\(2024\)Benchmark data contamination of large language models: a survey\.External Links:2406\.04244,[Link](https://arxiv.org/abs/2406.04244)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p1.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]C\. Deng, Y\. Zhao, X\. Tang, M\. Gerstein, and A\. Cohan\(2024\)Investigating data contamination in modern benchmarks for large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 8706–8719\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.482),[Link](https://aclanthology.org/2024.naacl-long.482/)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p1.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]Y\. V\. Fu, O\. Uzuner, M\. Yetisgen, and F\. Xia\(2025\)Does data contamination detection work \(well\) for llms? a survey and evaluation on detection assumptions\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 5250–5271\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.291),[Link](https://aclanthology.org/2025.findings-naacl.291/)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p1.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey, Shubh\-Agrawal, S\. Singh Sandha, S\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. Goldblum\(2025\)LiveBench: a challenging, contamination\-limited llm benchmark\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p2.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. I\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.External Links:2403\.07974,[Link](https://arxiv.org/abs/2403.07974)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p2.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]C\. Xu, C\. Jin, Y\. Niu, N\. Yan, Y\. Mei, S\. Guan, L\. Chen, and M\. Kechadi\(2026\)LiveFact: a dynamic, time\-aware benchmark for LLM\-driven fake news detection\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11881–11910\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.546),[Link](https://aclanthology.org/2026.acl-long.546/)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p2.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p1.1)\.
- \[8\]J\. Kim, H\. Lee, D\. Zhou, S\. H\. Park, S\. Yoon, T\. Bui, F\. Dernoncourt, S\. Cha, and M\. Seo\(2026\)Can large language models keep up? benchmarking online adaptation to continual knowledge streams\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 42240–42272\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1956),[Link](https://aclanthology.org/2026.acl-long.1956/)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p2.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]H\. Liu, L\. Cao, and Y\. Li\(2026\)RAG or learning? understanding the limits of LLM adaptation under continuous knowledge drift in the real world\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 11234–11252\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.546),[Link](https://aclanthology.org/2026.findings-acl.546/)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p2.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]G\. Delétang, A\. Ruoss, P\. Duquenne, E\. Catt, T\. Genewein, C\. Mattern, J\. Grau\-Moya, L\. K\. Wenliang, M\. Aitchison, L\. Orseau, M\. Hutter, and J\. Veness\(2023\)Language modeling is compression\.External Links:2309\.10668,[Link](https://arxiv.org/abs/2309.10668)Cited by:[§1](https://arxiv.org/html/2609.27510#S1.p3.1),[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2609.27510#S3.SS2.p1.1)\.
- \[11\]Y\. Li, Y\. Guo, F\. Guerin, and C\. Lin\(2024\)An open\-source data contamination report for large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 528–541\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.30),[Link](https://aclanthology.org/2024.findings-emnlp.30/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]O\. Sainz, J\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. Agirre\(2023\)NLP evaluation in trouble: on the need to measure llm data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10776–10787\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.722),[Link](https://aclanthology.org/2023.findings-emnlp.722/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]S\. Chen, Y\. Chen, Z\. Li, Y\. Jiang, Z\. Wan, Y\. He, D\. Ran, T\. Gu, H\. Li, T\. Xie, and B\. Ray\(2025\)Benchmarking large language models under data contamination: a survey from static to dynamic evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10080–10098\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.511),[Link](https://aclanthology.org/2025.emnlp-main.511/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]N\. S\. Hidayat, M\. D\. Al Kautsar, A\. F\. Wicaksono, and F\. Koto\(2025\)Simulating training data leakage in multiple\-choice benchmarks for LLM evaluation\.InProceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems,pp\. 21–39\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.eval4nlp-1.3),[Link](https://aclanthology.org/2025.eval4nlp-1.3/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]A\. K\. Singh, M\. Y\. Kocyigit, A\. Poulton, D\. Esiobu, M\. Lomeli, G\. Szilvasy, and D\. Hupkes\(2024\)Evaluation data contamination in llms: how do we measure it and \(when\) does it matter?\.External Links:2411\.03923,[Link](https://arxiv.org/abs/2411.03923)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]X\. Xiao and Y\. Cheng\(2026\)Contamination inflates scores but rarely reorders large language model leaderboards\.External Links:2609\.02899,[Link](https://arxiv.org/abs/2609.02899)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]Y\. Dong, X\. Jiang, H\. Liu, Z\. Jin, B\. Gu, M\. Yang, and G\. Li\(2024\)Generalization or memorization: data contamination and trustworthy evaluation for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12039–12050\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.716),[Link](https://aclanthology.org/2024.findings-acl.716/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]C\. Xu, N\. Yan, S\. Guan, C\. Jin, Y\. Mei, Y\. Guo, and T\. Kechadi\(2025\)DCR: quantifying data contamination in llms evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 23002–23020\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1173),[Link](https://aclanthology.org/2025.emnlp-main.1173/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[19\]Z\. Zhang, Q\. Liu, S\. Liang, N\. Li, Z\. Hu, W\. Gao, R\. Li, Z\. Huang, L\. Rutkowski, B\. Yu, and D\. Tao\(2026\)Controllable contamination detection for reliable llm evaluation with statistical guarantees\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,pp\. 30122–30143\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1390),[Link](https://aclanthology.org/2026.acl-long.1390/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]K\. Qian, S\. Wan, C\. Tang, Y\. Wang, X\. Zhang, M\. Chen, and Z\. Yu\(2024\)VarBench: robust language model benchmarking through dynamic variable perturbation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 16131–16161\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.946),[Link](https://aclanthology.org/2024.findings-emnlp.946/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]Z\. Yu, C\. Gao, W\. Yao, Y\. Wang, W\. Ye, J\. Wang, X\. Xie, Y\. Zhang, and S\. Zhang\(2024\)KIEval: a knowledge\-grounded interactive evaluation framework for large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5967–5985\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.325),[Link](https://aclanthology.org/2024.acl-long.325/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]Q\. Zhu, Q\. Cheng, R\. Peng, X\. Li, R\. Peng, T\. Liu, X\. Qiu, and X\. Huang\(2024\)Inference\-time decontamination: reusing leaked benchmarks for large language model evaluation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 9113–9129\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.532),[Link](https://aclanthology.org/2024.findings-emnlp.532/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]J\. Chai, Z\. Yu, and J\. Sakuma\(2026\)When benchmarks leak: inference\-time decontamination for LLMs\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 44743–44760\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.2071),[Link](https://aclanthology.org/2026.acl-long.2071/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]A\. Jacovi, A\. Caciularu, O\. Goldman, and Y\. Goldberg\(2023\)Stop uploading test data in plain text: practical strategies for mitigating data contamination by evaluation benchmarks\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5075–5084\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.308),[Link](https://aclanthology.org/2023.emnlp-main.308/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[25\]A\. Al\-Lawati, J\. Lucas, D\. Lee, and S\. Wang\(2026\)LLM benchmark datasets should be contamination\-resistant\.Note:Accepted to ICML 2026 Position Paper TrackExternal Links:2605\.19999,[Link](https://arxiv.org/abs/2605.19999)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px1.p1.1)\.
- \[26\]X\. Wu, L\. Pan, Y\. Xie, R\. Zhou, S\. Zhao, Y\. Ma, M\. Du, R\. Mao, A\. T\. Luu, and W\. Y\. Wang\(2025\)AntiLeakBench: preventing data contamination by automatically constructing benchmarks with updated real\-world knowledge\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,pp\. 18403–18419\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.901),[Link](https://aclanthology.org/2025.acl-long.901/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]C\. Zhu, N\. Chen, Y\. Gao, Y\. Zhang, P\. Tiwari, and B\. Wang\(2025\)Is your llm outdated? a deep look at temporal generalization\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7433–7457\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.381),[Link](https://aclanthology.org/2025.naacl-long.381/)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px2.p2.1)\.
- \[28\]I\. Magnusson, A\. Bhagia, V\. Hofmann, L\. Soldaini, A\. H\. Jha, O\. Tafjord, D\. Schwenk, E\. P\. Walsh, Y\. Elazar, K\. Lo, D\. Groeneveld, I\. Beltagy, H\. Hajishirzi, N\. A\. Smith, K\. Richardson, and J\. Dodge\(2024\)Paloma: a benchmark for evaluating language model fit\.InAdvances in Neural Information Processing Systems 37,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/760b2d94398aa61468aa3bc11506d9ea-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px3.p1.1)\.
- \[29\]Y\. Li, Y\. Guo, F\. Guerin, and C\. Lin\(2024\)Evaluating large language models for generalization and robustness via data compression\.External Links:2402\.00861,[Link](https://arxiv.org/abs/2402.00861)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px3.p1.1)\.
- \[30\]Y\. Huang, J\. Zhang, Z\. Shan, and J\. He\(2024\)Compression represents intelligence linearly\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=SHMj84U5SH)Cited by:[§2](https://arxiv.org/html/2609.27510#S2.SS0.SSS0.Px3.p1.1)\.
- \[31\]A\. Yang, A\. Li, B\. Yang,et al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.2](https://arxiv.org/html/2609.27510#S4.SS2.p2.1),[§4\.3](https://arxiv.org/html/2609.27510#S4.SS3.p1.1)\.
- \[32\]Qwen Team\(2026\)Qwen3\.5: towards native multimodal agents\.Note:Official model release pageExternal Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.2](https://arxiv.org/html/2609.27510#S4.SS2.p2.1)\.
- \[33\]B\. Peng, R\. Zhang, D\. Goldstein,et al\.\(2025\)RWKV\-7 “goose” with expressive dynamic state evolution\.External Links:2503\.14456,[Link](https://arxiv.org/abs/2503.14456)Cited by:[§4\.2](https://arxiv.org/html/2609.27510#S4.SS2.p2.1),[§4\.3](https://arxiv.org/html/2609.27510#S4.SS3.p1.1)\.
- \[34\]Gemma Team, S\. El Abd, V\. Aggarwal,et al\.\(2026\)Gemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§4\.2](https://arxiv.org/html/2609.27510#S4.SS2.p3.1)\.
- \[35\]NVIDIA, A\. Blakeman, A\. Grattafiori,et al\.\(2025\)Nemotron 3 nano: open, efficient mixture\-of\-experts hybrid mamba\-transformer model for agentic reasoning\.External Links:2512\.20848,[Link](https://arxiv.org/abs/2512.20848)Cited by:[§4\.2](https://arxiv.org/html/2609.27510#S4.SS2.p3.1)\.
- \[36\]J\. Zuo, M\. Velikanov, I\. Chahed,et al\.\(2025\)Falcon\-H1: a family of hybrid\-head language models redefining efficiency and performance\.External Links:2507\.22448,[Link](https://arxiv.org/abs/2507.22448)Cited by:[§4\.3](https://arxiv.org/html/2609.27510#S4.SS3.p1.1)\.
- \[37\]J\. von Oswald, N\. Scherrer, S\. Kobayashi, L\. Versari, S\. Yang, S\. Mittal, M\. Schlegel, K\. Maile, Y\. Schimpf, O\. Sieberling, A\. Meulemans, R\. A\. Saurous, G\. Lajoie, C\. Frenkel, R\. Pascanu, B\. Agüera y Arcas, and J\. Sacramento\(2026\)MesaNet: sequence modeling by locally optimal test\-time training\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/2c3d5a4eff20d24390a445762d61bfbb-Abstract-Conference.html)Cited by:[§4\.3](https://arxiv.org/html/2609.27510#S4.SS3.p4.1)\.
- \[38\]T\. Henighan, J\. Kaplan, M\. Katz, M\. Chen, C\. Hesse, J\. Jackson,et al\.\(2020\)Scaling laws for autoregressive generative modeling\.External Links:2010\.14701,[Link](https://arxiv.org/abs/2010.14701)Cited by:[§4\.4](https://arxiv.org/html/2609.27510#S4.SS4.p1.1)\.

## Appendix AComplete Model\-by\-Category Results

Table[5](https://arxiv.org/html/2609.27510#A1.T5)reports category\-level results for the 80\-model cohort used in the main cross\-model analyses\.

Table 5:July 2026 category\-level results for the 80\-model analysis cohort\. All score cells are compression rate \(CR, %\); lower is better\.ModelParams\.General textScientific PaperCodeOverallEFNewsEENEEBioCSMathSci\-OPhysC\+\+JSMDOtherPyOverall\>20BGemma\-4\-31B31\.39\.177\.466\.986\.096\.436\.595\.986\.166\.313\.163\.707\.863\.984\.156\.00Qwen3\.5\-35B\-A3B\-Base34\.79\.738\.257\.617\.496\.436\.525\.786\.096\.243\.333\.757\.924\.044\.246\.24Gemma\-4\-26B\-A4B25\.89\.527\.687\.226\.416\.626\.926\.226\.446\.563\.293\.878\.344\.174\.386\.26Mistral\-Small\-24B24\.09\.367\.857\.346\.826\.446\.796\.086\.306\.333\.444\.048\.394\.304\.496\.28Gemma\-3\-27B27\.49\.537\.647\.156\.356\.626\.906\.076\.396\.443\.464\.088\.584\.414\.616\.30Seed\-OSS\-36B\-Base36\.29\.618\.027\.488\.516\.526\.835\.976\.366\.573\.263\.898\.124\.074\.286\.39Nemotron\-3\-Nano\-30B\-A3B31\.69\.707\.967\.337\.406\.606\.846\.196\.436\.493\.444\.158\.494\.444\.506\.43Qwen2\.5\-32B32\.89\.698\.107\.747\.646\.626\.965\.846\.406\.523\.414\.118\.554\.294\.476\.45Qwen3\-30B\-A3B\-Base30\.59\.828\.337\.867\.576\.697\.045\.906\.476\.543\.414\.078\.764\.354\.536\.52OLMo\-3\-32B32\.29\.608\.057\.719\.246\.586\.876\.186\.426\.604\.094\.858\.885\.074\.916\.79marin\-32b\-base32\.59\.477\.727\.409\.636\.787\.236\.336\.636\.744\.134\.909\.185\.025\.116\.88∼\\sim14BRWKV7\-G1J\-13\.3B13\.39\.308\.017\.456\.876\.626\.686\.136\.326\.483\.594\.078\.144\.304\.426\.31Ministral\-3\-14B13\.99\.868\.227\.646\.996\.446\.806\.036\.336\.363\.554\.218\.604\.464\.656\.44Qwen3\-14B\-Base14\.89\.898\.357\.907\.666\.686\.985\.936\.456\.553\.444\.108\.624\.354\.526\.53Mistral\-Nemo\-Base\-240712\.29\.367\.937\.517\.236\.597\.126\.336\.576\.653\.754\.418\.984\.664\.866\.57gemma\-3\-12b\-pt12\.29\.817\.817\.456\.836\.857\.136\.356\.636\.673\.684\.328\.944\.644\.826\.57Qwen2\.5\-14B14\.89\.828\.157\.847\.836\.777\.146\.036\.576\.693\.594\.318\.854\.484\.666\.62Gravity\-16B\-A3B\-Base16\.210\.228\.588\.389\.886\.827\.256\.626\.787\.014\.024\.889\.255\.055\.177\.14Llama\-2\-13b\-hf13\.09\.857\.997\.718\.387\.267\.797\.077\.227\.314\.375\.2410\.025\.375\.627\.23∼\\sim7BQwen3\.5\-9B\-Base9\.010\.158\.608\.028\.246\.676\.766\.066\.346\.503\.654\.088\.284\.334\.516\.59Ministral\-3\-8B8\.910\.208\.427\.907\.346\.647\.026\.256\.546\.583\.714\.428\.934\.664\.846\.68RWKV7\-G1J\-7\.2B7\.29\.698\.347\.847\.486\.947\.056\.546\.676\.843\.914\.448\.754\.674\.816\.71Meta\-Llama\-3\.1\-8B8\.09\.898\.017\.667\.456\.837\.356\.366\.736\.773\.944\.739\.394\.885\.086\.79Qwen3\-8B\-Base8\.210\.218\.618\.198\.126\.927\.226\.156\.686\.803\.634\.338\.994\.574\.746\.80gemma\-4\-E4B7\.910\.138\.117\.907\.247\.247\.436\.907\.007\.203\.824\.449\.104\.714\.926\.87Falcon\-H1\-7B\-Base7\.610\.168\.338\.009\.096\.887\.236\.176\.656\.713\.964\.699\.414\.905\.126\.95Mistral\-7B\-v0\.17\.29\.878\.037\.789\.146\.947\.476\.596\.896\.974\.104\.859\.474\.995\.247\.02Qwen2\.5\-7B7\.610\.348\.568\.338\.707\.177\.536\.376\.947\.103\.864\.659\.424\.814\.967\.05marin\-8b\-base8\.09\.918\.027\.8610\.456\.917\.256\.446\.756\.754\.335\.179\.635\.255\.467\.16Hunyuan\-7B\-Pretrain7\.510\.708\.808\.358\.407\.357\.606\.817\.137\.344\.044\.729\.364\.895\.047\.18Apertus\-8B\-25098\.110\.488\.467\.957\.057\.447\.897\.557\.437\.754\.175\.009\.705\.155\.427\.25Minitron\-8B8\.310\.118\.398\.118\.557\.227\.757\.387\.287\.494\.175\.1710\.035\.355\.477\.32Olmo\-Hybrid\-7B7\.410\.278\.558\.2710\.607\.037\.346\.646\.877\.094\.615\.469\.745\.655\.407\.39ZAYA1\-base8\.810\.658\.768\.449\.647\.427\.787\.127\.327\.454\.325\.1910\.015\.355\.547\.50OLMo\-3\-7B7\.310\.418\.678\.3711\.087\.137\.456\.776\.997\.224\.715\.589\.935\.775\.507\.54Llama\-2\-7b\-hf6\.710\.188\.238\.038\.987\.598\.147\.477\.577\.674\.665\.5910\.555\.705\.967\.59Falcon3\-7B\-Base7\.510\.648\.988\.6712\.747\.077\.486\.526\.896\.984\.245\.1310\.295\.245\.537\.60∼\\sim3BQwen3\.5\-4B\-Base4\.210\.659\.058\.498\.956\.997\.076\.416\.666\.833\.984\.438\.794\.664\.836\.98Spark\-X2\.5\-4B\-Base4\.110\.949\.038\.498\.547\.247\.316\.626\.947\.163\.964\.398\.534\.564\.617\.02RWKV7\-G1J\-2\.9B2\.910\.028\.678\.228\.057\.287\.446\.967\.057\.234\.254\.869\.365\.055\.217\.12Nanbeige4\.2\-3B\-Base4\.211\.139\.018\.2910\.417\.017\.136\.476\.716\.864\.004\.648\.794\.794\.817\.15Qwen3\-4B\-Base4\.010\.669\.038\.638\.797\.247\.526\.406\.967\.113\.874\.609\.454\.844\.987\.15Ministral\-3\-3B3\.810\.898\.908\.418\.106\.987\.446\.696\.947\.004\.084\.879\.625\.085\.257\.16gemma\-3\-4b\-pt4\.310\.548\.288\.147\.797\.457\.726\.987\.207\.284\.224\.949\.885\.245\.407\.22Llama\-3\.2\-3B3\.210\.458\.468\.258\.377\.387\.896\.977\.287\.344\.435\.3110\.235\.435\.597\.38gemma\-4\-E2B5\.110\.778\.518\.387\.847\.728\.037\.557\.557\.794\.445\.1710\.095\.415\.607\.49Nanbeige4\-3B\-Base3\.911\.439\.188\.7810\.787\.277\.506\.777\.047\.184\.214\.909\.465\.215\.237\.50Qwen2\.5\-3B3\.110\.859\.008\.809\.507\.617\.996\.827\.387\.574\.255\.1110\.145\.255\.377\.55Falcon\-H1\-3B\-Base3\.110\.848\.878\.6510\.257\.447\.726\.647\.127\.224\.525\.2710\.275\.535\.687\.57MiniCPM5\-2B\-Base2\.511\.559\.518\.929\.627\.677\.757\.197\.357\.664\.204\.999\.665\.255\.317\.62SmolLM3\-3B\-Base3\.110\.488\.788\.519\.397\.718\.177\.577\.657\.954\.395\.4010\.415\.665\.647\.69Llama\-3\.1\-Minitron\-4B\-Width4\.510\.478\.678\.3910\.457\.498\.037\.627\.557\.764\.385\.4310\.455\.615\.697\.71gemma\-2\-2b2\.610\.748\.538\.598\.827\.628\.287\.547\.677\.774\.605\.4510\.845\.795\.987\.73Minitron\-4B\-Base4\.210\.668\.778\.599\.457\.628\.207\.977\.748\.014\.535\.6510\.775\.845\.917\.84stablelm\-3b\-4e1t2\.810\.468\.568\.3810\.037\.688\.357\.717\.767\.894\.605\.6711\.076\.026\.017\.87Index\-1\.9B2\.211\.049\.198\.968\.797\.878\.327\.117\.687\.714\.996\.3311\.426\.366\.458\.02Llama\-3\.1\-Minitron\-4B\-Depth4\.510\.829\.028\.7711\.417\.878\.428\.057\.938\.184\.655\.7710\.985\.966\.038\.13<2<2BRWKV7\-G1J\-1\.5B1\.510\.569\.168\.768\.847\.717\.917\.467\.507\.694\.695\.3510\.105\.535\.677\.64Spark\-X2\.5\-1\.7B\-Base1\.711\.699\.689\.199\.567\.827\.957\.217\.547\.764\.485\.039\.515\.185\.227\.70Qwen3\.5\-2B\-Base1\.911\.519\.849\.2810\.097\.627\.757\.087\.307\.514\.675\.149\.825\.355\.527\.75Qwen3\-1\.7B\-Base1\.711\.339\.589\.249\.807\.788\.086\.957\.497\.674\.325\.1510\.295\.375\.497\.75Qwen2\.5\-1\.5B1\.511\.369\.399\.2710\.468\.038\.417\.227\.777\.994\.565\.4910\.745\.615\.728\.00Youtu\-LLM\-2B\-Base2\.012\.129\.819\.1212\.777\.857\.937\.417\.547\.774\.435\.219\.855\.325\.318\.03Falcon\-H1\-1\.5B\-Deep\-Base1\.611\.279\.369\.1811\.337\.908\.247\.107\.587\.835\.075\.8811\.066\.166\.208\.16Llama\-3\.2\-1B1\.211\.269\.088\.989\.508\.088\.657\.738\.008\.095\.116\.0811\.406\.196\.328\.18Falcon\-H1\-1\.5B\-Base1\.611\.449\.489\.3311\.568\.058\.397\.287\.737\.995\.246\.0711\.316\.356\.408\.33SmolLM2\-1\.7B1\.710\.829\.159\.1513\.408\.128\.657\.868\.048\.364\.855\.8211\.366\.086\.138\.41stablelm\-2\-1\_6b1\.611\.069\.018\.8910\.778\.248\.898\.118\.278\.425\.366\.2811\.636\.356\.558\.42Qwen3\.5\-0\.8B\-Base0\.812\.5210\.8110\.2311\.478\.388\.507\.818\.018\.235\.425\.9410\.956\.126\.278\.62MiniCPM5\-1B\-Base1\.112\.2310\.309\.8212\.168\.418\.647\.938\.158\.405\.116\.1411\.386\.386\.768\.70Qwen3\-0\.6B\-Base0\.612\.4110\.6510\.3011\.668\.658\.917\.748\.298\.535\.015\.9611\.536\.156\.288\.72gemma\-3\-1b\-pt1\.011\.809\.159\.309\.178\.819\.299\.078\.768\.996\.467\.2312\.397\.587\.308\.95RWKV7\-G1D\-0\.4B0\.511\.6310\.159\.9010\.498\.989\.268\.738\.799\.005\.846\.8112\.066\.966\.978\.97Qwen2\.5\-0\.5B0\.512\.6410\.5510\.5312\.729\.249\.548\.308\.879\.165\.426\.5012\.276\.576\.649\.21ERNIE\-4\.5\-0\.3B\-Base\-PT0\.413\.2211\.1510\.9513\.639\.299\.748\.929\.179\.356\.187\.1612\.727\.267\.479\.73SmolLM2\-360M0\.412\.3010\.3610\.4016\.509\.239\.768\.979\.119\.475\.987\.0113\.137\.377\.269\.77Falcon\-H1\-0\.5B\-Base0\.512\.9610\.3510\.5318\.959\.009\.298\.188\.588\.926\.207\.2713\.107\.547\.479\.88RWKV7\-G1D\-0\.1B0\.212\.8211\.3511\.1912\.5010\.1610\.409\.939\.9010\.196\.988\.0513\.658\.238\.1410\.25gemma\-3\-270m0\.313\.4510\.5410\.9611\.7010\.3410\.7810\.2110\.1510\.338\.379\.4514\.7910\.008\.8310\.71SmolLM2\-135M0\.113\.3811\.4411\.4418\.7210\.3710\.9710\.5310\.3610\.726\.938\.3914\.589\.448\.7711\.15Table 5:July 2026 category\-level results for the 80\-model analysis cohort\. All score cells are compression rate \(CR, %\); lower is better\. \(continued\)Abbreviations follow Table[2](https://arxiv.org/html/2609.27510#S4.T2)\. Overall is the macro\-average of the 14 category CRs\. Rows are grouped by total parameter count and sorted by unrounded Overall within each group\. Bold marks the best displayed value in each group, including ties at the shown precision\. Params\. is total parameters in billions\.

## Appendix BLong\-Context Results across Model Sizes

Figure 5:Long\-context CR for 33 models in the first three size groups\. Each row shows absolute CR \(left\) and the difference from the Qwen3 model in that size group \(right\), pooled across four scientific\-paper categories\.Figure 6:Long\-context CR for 21 models in the last two size groups\. Each row shows absolute CR \(left\) and the difference from the Qwen3 model in that size group \(right\), pooled across four scientific\-paper categories\.
## Appendix CComplete Category\-Level Scaling Results

![Refer to caption](https://arxiv.org/html/2609.27510v1/category_scaling_curves.png)Figure 7:Category\-level scaling curves\. Each panel shows CR for all 80 models and the fit defined in Eq\.[7](https://arxiv.org/html/2609.27510#S4.E7)\.Table 6:Category\-level power\-law\-plus\-constant fits to all 80 models in the July 2026 cohort\.Parameters correspond toCR⁡\(P\)=a​Pb\+c\\mathrm\{CR\}\(P\)=aP^\{b\}\+c, withPPmeasured in billions\. RMSE is measured across all 80 models in CR percentage points\.

## Appendix DPareto\-Frontier Scaling Results

For the all\-category score and each text type, we fit Eq\.[7](https://arxiv.org/html/2609.27510#S4.E7)only to models on the parameter–CR Pareto frontier\. A model lies on the frontier unless another evaluated model has at most as many parameters and lower CR\.

![Refer to caption](https://arxiv.org/html/2609.27510v1/pareto_scaling_curves.png)Figure 8:Pareto\-frontier scaling by text type\. Diamonds mark frontier models, circles mark other models, and dashed lines fit the frontier models\.Table 7:Pareto\-frontier scaling fits by data type on the July 2026 80\-model cohort\.The all\-category frontier contains 20 models from 0\.135B to 31\.3B parameters\. Fitting these 20 models givesCR^​\(P\)=3\.244​P−0\.304\+5\.016\\widehat\{\\mathrm\{CR\}\}\(P\)=3\.244P^\{\-0\.304\}\+5\.016\(R2=0\.992R^\{2\}=0\.992; RMSE: 0\.125 CR percentage points\)\. Type\-level frontiers contain 12–23 models, with exponents from−0\.575\-0\.575for encyclopedia articles to−0\.244\-0\.244for English fiction\.

## Appendix ETokenization Statistics

Table 8:Native\-token segmentation diagnostics for the 80\-model analysis cohort\. Residualρ\\rhois the Spearman correlation between the residuals of CR and bytes per token after each is regressed onlog10\\log\_\{10\}parameter count with an intercept\.