Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

# Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?
Source: [https://arxiv.org/html/2608.10690](https://arxiv.org/html/2608.10690)
Qingjie Zhang1,2, Xingzhang Ren2, Zixuan Chen1, Jinfeng Li3, Yuefeng Chen3,

Yitong Yang3, Hui Xue3, Dayiheng Liu2\*, and Han Qiu1\*

1Tsinghua University2Qwen Team, Alibaba Group3Alibaba Group

Emails: \{qj\-zhang24@mails\., qiuhan@\}tsinghua\.edu\.cn\*Corresponding authors

###### Abstract

Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released\. Prior work has inferred corpus mixtures or traced specific token groups from released tokenizer vocabularies; in contrast, we estimate corpus ratios for arbitrary target tokens\. We first show that BPE tokenizers trained on different corpora share stable token ID–ratio distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora\. We then propose Quantile\-Guided Density Estimation \(QGDE\), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token\-level estimates\. In controlled settings and a realistic setting using the released SmolLM tokenizer, QGDE achieves mean relative errors as low as 3\.00% for token\-level estimation and 3\.08% after aggregation into category\-level mixtures\. These results suggest that released tokenizer vocabularies provide a useful signal for fine\-grained corpus estimation beyond coarse composition inference\. The code of QGDE is available at[https://github\.com/qingjiesjtu/QGDE](https://github.com/qingjiesjtu/QGDE)\.

Can Released LLM Vocabularies Support Token\-Level Estimation of Hidden Corpora?

Qingjie Zhang1,2, Xingzhang Ren2, Zixuan Chen1, Jinfeng Li3, Yuefeng Chen3, Yitong Yang3, Hui Xue3, Dayiheng Liu2\*, and Han Qiu1\* 1Tsinghua University2Qwen Team, Alibaba Group3Alibaba Group Emails: \{qj\-zhang24@mails\., qiuhan@\}tsinghua\.edu\.cn\*Corresponding authors

## 1Introduction

The composition of pretraining corpora is essential for interpreting LLM performance, since it shapes basic capabilities such as multilingual and domain\-specific performance\(Xieet al\.,[2023](https://arxiv.org/html/2608.10690#bib.bib24); Hoffmannet al\.,[2022](https://arxiv.org/html/2608.10690#bib.bib44); Liet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib45); Pettyet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib25)\)\. However, even models with released weights often leave the corpus opaque\(Touvronet al\.,[2023](https://arxiv.org/html/2608.10690#bib.bib46); Biet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib28); Liuet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib2)\)\. A more accessible signal is the tokenizer vocabulary, which is often released and commonly trained with byte\-pair encoding \(BPE\)\(Sennrichet al\.,[2016](https://arxiv.org/html/2608.10690#bib.bib10)\)on corpus statistics, as in the ChatGPT\(Singhet al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib3)\), Qwen\(Baiet al\.,[2023](https://arxiv.org/html/2608.10690#bib.bib27); Yanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib1)\), DeepSeek\(Biet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib28); Liuet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib2)\)families\.

Prior work has shown that released BPE vocabularies can reveal corpus information111Tokenizer\-training corpora may differ from pretraining corpora, but still reflect composition choices of developers\.\.Hayaseet al\.\([2024](https://arxiv.org/html/2608.10690#bib.bib17)\)recover corpus mixture from BPE merge rules, but it remains coarse\-grained rather than estimating the ratio of each token\.Zhanget al\.\([2025](https://arxiv.org/html/2608.10690#bib.bib21)\)use token IDs to speculate about the prevalence of polluted Chinese tokens, but their setting targets a specific token group rather than general tokens\. This motivates our central question:

Can released tokenizer vocabularies support general token\-level estimation of hidden corpora?

![Refer to caption](https://arxiv.org/html/2608.10690v1/x1.png)Figure 1:Overview of our work\. QGDE transfers ID\-ratio structure from known corpora to estimate token and category ratios of hidden corpora\.[Figure 1](https://arxiv.org/html/2608.10690#S1.F1)illustrates our work\. To estimate token ratios, we first compare BPE vocabularies trained on different corpora and find that their token ID–ratio distributions share a stable global shape\. This motivates transferring this distribution from known corpora to a target tokenizer trained on hidden corpora\. We then proposeQuantile\-Guided Density Estimation\(QGDE\), which fits multiple quantile trends to approximate the known ID–ratio distribution and enables token\-level estimates for the target tokenizer\. Our main contributions are:

- ∙\\bulletWe show that token ID–ratio distributions are transferable across BPE vocabularies trained on different languages and domains, providing a usable signal for estimating hidden corpus ratios from released vocabularies\.
- ∙\\bulletWe propose QGDE, a general token\-level estimator that fits multiple quantile trends to approximate the transferable distribution and uses local density weighting to predict corpus ratios for arbitrary target tokens\.
- ∙\\bulletIn both controlled settings and a realistic setting, QGDE achieves relative errors as low as 3\.00% for token\-level estimation and 3\.08% for category\-level estimation after aggregation, surpassing three compared methods\.

## 2Background

### 2\.1Related Work

#### Training data transparency and tokenizer signals\.

LLMs are trained on corpora whose exact composition is often undisclosed, making it difficult to interpret the data sources, language coverage, and domain coverage behind released models\(Bommasaniet al\.,[2023](https://arxiv.org/html/2608.10690#bib.bib4); Longpreet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib5); Dodgeet al\.,[2021](https://arxiv.org/html/2608.10690#bib.bib47); Xuet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib6)\)\. Existing transparency methods often rely on model outputs or query access to reveal memorized examples, extract training data, or detect benchmark overlap\(Carliniet al\.,[2021](https://arxiv.org/html/2608.10690#bib.bib7),[2022](https://arxiv.org/html/2608.10690#bib.bib8); Yanget al\.,[2023](https://arxiv.org/html/2608.10690#bib.bib9)\)\. In contrast, released tokenizers provide a corpus signal because they are learned from corpus statistics: frequent adjacent token pairs are merged earlier, and the resulting vocabulary and token IDs can encode distributional traces of the tokenizer\-training corpus\(Sennrichet al\.,[2016](https://arxiv.org/html/2608.10690#bib.bib10); Hayaseet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib17); Zhanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib21)\)\.

#### From coarse mixture to token\-level estimation\.

Data Mixture Inference \(DMI\)\(Hayaseet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib17)\)treats released BPE merge rules as a composition signal: it compares tokenizer merge statistics with candidate corpora and estimates a small vector of category proportions\. This output is inherently coarse\-grained, over languages, domains, or data sources, rather than a ratio estimate for each token\. PoCTrace\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib21)\)exploits the relation between token IDs and frequency: it fits a single ID–ratio median trend to estimate the prevalence of polluted Chinese tokens\. However, its use case remains tied to a specific token group and a coarse trend estimate\. We instead study general token\-level corpus ratio estimation for arbitrary tokens\.

### 2\.2Problem Formulation

Let𝒱∗=\{\(vi,ti\)\}i=1N\\mathcal\{V\}^\{\*\}=\\\{\(v\_\{i\},t\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote the released vocabulary of a target tokenizer, whereviv\_\{i\}is a token string andtit\_\{i\}is its token ID\. The tokenizer\-training corpus𝒞∗\\mathcal\{C\}^\{\*\}is hidden\. Letri∗r\_\{i\}^\{\*\}denote the corpus ratio ofviv\_\{i\}in𝒞∗\\mathcal\{C\}^\{\*\}\. The hidden corpus therefore induces an unknown ID\-to\-ratio mappingf∗​\(ti\)=ri∗f^\{\*\}\(t\_\{i\}\)=r\_\{i\}^\{\*\}\.

We assume access to one or moreknown corpora𝒟\\mathcal\{D\}\. For each known corpus or mixture of known corpora, token ratios are observable after training and applying a BPE tokenizer\(Sennrichet al\.,[2016](https://arxiv.org/html/2608.10690#bib.bib10); Hayaseet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib17)\)\. These known corpora provide empirical ID–ratio pairs that can be used to approximate the unknown mappingf∗f^\{\*\}\.

Our goal is to learn an estimatorf^\\widehat\{f\}from the known ID–ratio pairs, such that for arbitrary tokens in the released target vocabulary,

f^​\(ti\)=r^i≈f∗​\(ti\)=ri∗\.\\widehat\{f\}\(t\_\{i\}\)=\\widehat\{r\}\_\{i\}\\approx f^\{\*\}\(t\_\{i\}\)=r\_\{i\}^\{\*\}\.\(1\)
After estimating token\-level ratios, we aggregate them over any token setS⊆𝒱∗S\\subseteq\\mathcal\{V\}^\{\*\}, e\.g\.,R^​\(S\)=∑i∈Sr^i\\widehat\{R\}\(S\)=\\sum\_\{i\\in S\}\\widehat\{r\}\_\{i\}\. When token sets are defined by languages or domains, this yields corpus mixture estimates\.

## 3Is ID\-Ratio Relationship Transferable?

The formulation above relies on a basic assumption: ID–ratio pairs observed in known corpora should provide useful evidence for the hidden target corpus\. Since BPE token IDs reflect merge order and thus corpus statistics\(Sennrichet al\.,[2016](https://arxiv.org/html/2608.10690#bib.bib10); Hayaseet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib17); Zhanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib21)\), the key question iswhether the ID–ratio relationship is transferable across tokenizers trained on different corpora\.

![Refer to caption](https://arxiv.org/html/2608.10690v1/x2.png)Figure 2:Token ID–ratio distributions are transferable across different corpora\.Left: ID–ratio scatter plots in log–log space\.Middle: Similarity across different languages\.Right: Similarity across different domain\.We compare ID–ratio distributions under two controlled settings:same domain but different languages, and same language but different domains\. Specifically, we train BPE tokenizers on English, French, Japanese, and Chinese slices from mC4\(Xueet al\.,[2021](https://arxiv.org/html/2608.10690#bib.bib23)\)for the language setting, and on FineWeb \(Web\)\(Penedoet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib32)\), Wikipedia \(Wiki\)\(Wikimedia Foundation,[2023](https://arxiv.org/html/2608.10690#bib.bib35)\), CodeParrot \(Code\)\(Hugging Face,[2021](https://arxiv.org/html/2608.10690#bib.bib34)\), and OpenWebMath \(Math\)\(Pasteret al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib33)\)for the domain setting\. For each tokenizer, we remove the initial vocabulary tokens, since special tokens and alphabet symbols do not reflect BPE merge order\. We then compute each token’s ratio as its count divided by the total token count\.

We plot the token ID–ratio distributions in log–log space in[Figure 2](https://arxiv.org/html/2608.10690#S3.F2)\. The scatter plots show thatthe distributions are visually similar, even for English and Japanese, and for Code and Wiki, which differ substantially in language or domain structure \(see[Figure 7](https://arxiv.org/html/2608.10690#A3.F7)for all eight corpora\)\. This suggests a transferable ID–ratio distribution\.

To quantify this similarity, we convert each scatter plot into a two\-dimensional probability density over\(log⁡ID,log⁡ratio\)\\left\(\\log\\mathrm\{ID\},\\log\\mathrm\{ratio\}\\right\)\(see[Appendix B](https://arxiv.org/html/2608.10690#A2)\) and define a directional transfer similarity score\. LetPSP\_\{S\}andPTP\_\{T\}denote the ID–ratio densities of a source tokenizerSSand a target tokenizerTT\. We compute

Sim​\(S→T\)=exp⁡\(−DKL​\(PT∥PS\)H​\(PT\)\),\\mathrm\{Sim\}\(S\\to T\)=\\exp\\left\(\-\\frac\{D\_\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|P\_\{S\}\)\}\{H\(P\_\{T\}\)\}\\right\),\(2\)whereH​\(PT\)H\(P\_\{T\}\)is the entropy of the target density andDKLD\_\{\\mathrm\{KL\}\}denotes Kullback–Leibler divergence\(Murphy,[2022](https://arxiv.org/html/2608.10690#bib.bib42)\)\. This score normalizes the transfer divergence by the target entropy and maps it to\(0,1\]\(0,1\], where larger values indicate that the source distribution better explains the target distribution\.

The heatmap in[Figure 2](https://arxiv.org/html/2608.10690#S3.F2)shows thattoken ID–ratio distributions are broadly transferable\. In the language setting, transfer is strongest between languages with more similar linguistic structures: English and French are nearly interchangeable, and Japanese and Chinese are also highly similar\. Transfer across these groups is weaker, but still substantial, with similarities around 0\.8 in the harder English/French\-to\-Japanese/Chinese directions\. In the domain setting, all pairs remain highly similar, with the largest gap appearing between Code and Wiki, mirroring the larger structural gap between programming language and encyclopedic text\. Additional single\- and mixed\-source transfer profiles are reported in[Appendix C](https://arxiv.org/html/2608.10690#A3)\. Overall, the ID–ratio relationship has a stable global shape, motivating an estimator that exploits the shared global ID–ratio trend across corpora\.

## 4Quantile\-Guided Density Estimation

Since token ID–ratio distributions share a global shape, we estimate token ratios by transferring this distributional structure from known corpora with observed token ratios\. Such transfer is inevitably imperfect:the key is to preserve the shared ID–ratio distribution\.

Inspired by prior work\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib21)\)that uses quantile regression to obtain ratio ranges from a single median or boundary curve, we fit multiple quantile trends to better approximate the shared ID–ratio distribution\. We then use local density weighting to convert these quantile trends into a token point estimate, as illustrated in[Figure 3](https://arxiv.org/html/2608.10690#S4.F3)\.

![Refer to caption](https://arxiv.org/html/2608.10690v1/Figs/quantile_guided_density_overview.png)Figure 3:Overview of quantile\-guided density estimation\.Left: the ID–ratio distribution in log–log space\.Middle: multiple quantile trends approximate the global ID–ratio distribution and produce candidate estimates at a target token ID\.Right: Gaussian density kernels around each candidate assign local weights to the estimates\.### 4\.1Fitting ID–Ratio Trends with Quantiles

Following Zipf’s law\(Piantadosi,[2014](https://arxiv.org/html/2608.10690#bib.bib22); Saichevet al\.,[2009](https://arxiv.org/html/2608.10690#bib.bib30)\), which states that word frequency is approximately inversely proportional to frequency rank, a log–log transformation turns this inverse relationship into an approximately linear trend\. We therefore model token ID and corpus ratio in log–log space, where quantile regression can fit linear trends at different quantile levels\.

Given a tokenizer trained on known corpora with observed token counts, we represent each tokenjjas\(xj,yj\)=\(log⁡tj,log⁡rj\)\(x\_\{j\},y\_\{j\}\)=\(\\log t\_\{j\},\\log r\_\{j\}\), wheretjt\_\{j\}is the token ID andrjr\_\{j\}is the token’s corpus ratio\. A single median curve, corresponding to the 0\.5 quantile, can capture the central tendency of this relation\. However,it reduces tokens with similar IDs to a single typical ratio, discarding their local ratio variation\. This local variation is exactly the distributional signal needed for token\-level estimation\.

We instead model the shared ID–ratio relationshipwith a family of quantile trends\. Let𝒯\\mathcal\{T\}denote a set of candidate quantile levels\. For eachτ∈𝒯\\tau\\in\\mathcal\{T\}, we fit a log\-linear quantile curve over the known ID–ratio points using standard quantile regression\(Regression,[2017](https://arxiv.org/html/2608.10690#bib.bib41)\),qτ​\(x\)=aτ\+bτ​xq\_\{\\tau\}\(x\)=a\_\{\\tau\}\+b\_\{\\tau\}x\(see details in[Appendix A](https://arxiv.org/html/2608.10690#A1)\)\. For a target token with IDtit\_\{i\}, theτ\\tau\-th trend gives a candidate log\-ratio estimate:

zi,τ=qτ​\(log⁡ti\)=aτ\+bτ​log⁡ti\.z\_\{i,\\tau\}=q\_\{\\tau\}\(\\log t\_\{i\}\)=a\_\{\\tau\}\+b\_\{\\tau\}\\log t\_\{i\}\.\(3\)The quantile family generalizes single\-curve token\-ID estimators: lower quantiles describe conservative low\-ratio hypotheses, upper quantiles describe high\-ratio hypotheses, and intermediate quantiles describe the dense central region\. Thus, at each token ID, a dense quantile family defines multiple plausible candidate estimates\. The remaining question is which quantile levels best approximate the shared ID–ratio distribution\.

### 4\.2Selecting Quantile Anchors

To avoid redundant or poorly supported trends, we select a small set of representative quantile anchors𝒯K⋆=\{τ1,…,τK\}\\mathcal\{T\}\_\{K\}^\{\\star\}=\\\{\\tau\_\{1\},\\ldots,\\tau\_\{K\}\\\}\.These anchors should cover the global ID–ratio distribution without collapsing onto redundant regions or passing through consistently sparse regions\.We therefore select anchors by directly maximizing their coverage over the known ID–ratio points𝒫=\{\(xj,yj\)\}j=1n\\mathcal\{P\}=\\\{\(x\_\{j\},y\_\{j\}\)\\\}\_\{j=1\}^\{n\}\.

For a candidate anchor set𝒯K\\mathcal\{T\}\_\{K\}, a point is covered if it falls within a vertical band of widthhyh\_\{y\}around at least one selected quantile trend\. We define Quantile Anchor CoverageCCas

C​\(𝒯K\)=∑\(xj,yj\)∈𝒫𝟏​\[minτ∈𝒯K⁡\|yj−qτ​\(xj\)\|<hy\]\.C\(\\mathcal\{T\}\_\{K\}\)=\\sum\_\{\(x\_\{j\},y\_\{j\}\)\\in\\mathcal\{P\}\}\\mathbf\{1\}\\\!\\left\[\\min\_\{\\tau\\in\\mathcal\{T\}\_\{K\}\}\|y\_\{j\}\-q\_\{\\tau\}\(x\_\{j\}\)\|<h\_\{y\}\\right\]\.\(4\)This score counts each known ID–ratio point at most once, even if it is close to multiple selected trends, so redundant anchors receive little extra benefit\. We choose the anchor set by maximizing this Quantile Anchor Coverage:

𝒯K⋆=arg⁡max𝒯K⊂𝒯,\|𝒯K\|=K⁡C​\(𝒯K\)\.\\mathcal\{T\}\_\{K\}^\{\\star\}=\\arg\\max\_\{\\mathcal\{T\}\_\{K\}\\subset\\mathcal\{T\},\\,\|\\mathcal\{T\}\_\{K\}\|=K\}C\(\\mathcal\{T\}\_\{K\}\)\.\(5\)In implementation, we perform a grid search over quantile levels\. Because the maximization depends on both the selected quantile set and the number of anchorsKK,[Section 5](https://arxiv.org/html/2608.10690#S5)analyzes which anchors are selected and how many anchors are needed\.

### 4\.3Local Density Weighting

After selecting quantile anchors, each target token has multiple candidate estimates\{zi,τ\}τ∈𝒯K⋆\\\{z\_\{i,\\tau\}\\\}\_\{\\tau\\in\\mathcal\{T\}\_\{K\}^\{\\star\}\}\. These candidates are plausible under the global ID–ratio trends, but the global trends do not determine how much each candidate should contribute to a particular token\. Therefore,we assign them soft weights by measuring how much local density support each candidate receives from nearby known ID–ratio points\.

For target tokentit\_\{i\}, we collect nearby known ID–ratio points𝒩i=\{\(xj,yj\)∣\|xj−log⁡ti\|<hx\}\\mathcal\{N\}\_\{i\}=\\\{\(x\_\{j\},y\_\{j\}\)\\mid\|x\_\{j\}\-\\log t\_\{i\}\|<h\_\{x\}\\\}\. Within this local ID window, we compute the unnormalized local support using Gaussian kernel\(Silverman,[2018](https://arxiv.org/html/2608.10690#bib.bib29)\):

Wi,τ=∑\(xj,yj\)∈𝒩iexp⁡\(−\(yj−zi,τ\)22​hy2\)\.W\_\{i,\\tau\}=\\sum\_\{\(x\_\{j\},y\_\{j\}\)\\in\\mathcal\{N\}\_\{i\}\}\\exp\\left\(\-\\frac\{\(y\_\{j\}\-z\_\{i,\\tau\}\)^\{2\}\}\{2h\_\{y\}^\{2\}\}\\right\)\.\(6\)This kernel is a soft version of the vertical neighborhood used in anchor coverage, centered at the candidate estimateszi,τz\_\{i,\\tau\}\. This weighting matches the intuition in[Figure 3](https://arxiv.org/html/2608.10690#S4.F3): at a fixed token ID, Gaussian density kernels compare how strongly the surrounding points support each quantile candidate\.

By normalizing the weights across all quantile trends, the estimated ratio of tokentit\_\{i\}is the density\-weighted average of the candidate estimates:

y^i=∑τ∈𝒯K⋆Wi,τ∑τ′∈𝒯K⋆Wi,τ′​zi,τ\.\\widehat\{y\}\_\{i\}=\\sum\_\{\\tau\\in\\mathcal\{T\}\_\{K\}^\{\\star\}\}\\frac\{W\_\{i,\\tau\}\}\{\\sum\_\{\\tau^\{\\prime\}\\in\\mathcal\{T\}\_\{K\}^\{\\star\}\}W\_\{i,\\tau^\{\\prime\}\}\}z\_\{i,\\tau\}\.\(7\)Thus, the global quantile trends provide candidate estimates, and local density weighting combines them using token\-specific neighborhood evidence\. As a result, it turns the range\-style signal used in prior token\-ID frequency estimation\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib21)\)into a fine\-grained token\-level point estimate\.

## 5Quantile Anchor Configuration

In[Equation 5](https://arxiv.org/html/2608.10690#S4.E5), we select quantile anchors to cover the known ID–ratio distribution while avoiding redundant or poorly supported trends\. Before evaluating token\-level estimates, we first examine how this Quantile Anchor Coverage \(QAC\) objective configures the anchors in practice\. The following subsections address two practical questions\.

### 5\.1Which Anchors Cover Better?

We first inspect the fixed\-KKselection problem\. TakingK=3K=3as an example, we grid search over candidate triplets using the QAC objective in[Equation 5](https://arxiv.org/html/2608.10690#S4.E5)\. Since the triplet search space is three\-dimensional,[Figure 4](https://arxiv.org/html/2608.10690#S5.F4)visualizes a pairwise projection: each cell fixes two anchors\(τa,τb\)\(\\tau\_\{a\},\\tau\_\{b\}\)and reports the best coverage obtained by any triplet containing that pair\.

![Refer to caption](https://arxiv.org/html/2608.10690v1/x3.png)Figure 4:Pairwise projection of Quantile Anchor Coverage for selecting three quantile anchors\. The star marks the selected triplet\(0\.50,0\.70,0\.90\)\(0\.50,0\.70,0\.90\)\.[Figure 4](https://arxiv.org/html/2608.10690#S5.F4)shows that coverage is substantially lower when anchors concentrate in sparse or redundant regions\. By contrast, the high\-coverage area lies around middle\-to\-high quantile levels\. In thisK=3K=3configuration, the triplet\(0\.50,0\.70,0\.90\)\(0\.50,0\.70,0\.90\)achieves the highest coverage, covering53\.1%53\.1\\%of the sampled known ID–ratio points under the chosen vertical bandwidth\. This indicates that QAC does not simply spread anchors uniformly across quantile levels; instead, it favors anchors whose trends jointly pass through well\-supported regions of the known ID–ratio distribution\.

### 5\.2How Many Anchors Are Enough?

We then study how the number of quantile anchors affects coverage\. For each anchor numberKK, we maximizeC​\(𝒯K\)C\(\\mathcal\{T\}\_\{K\}\)and report both the best coverage and the marginal gain overK−1K\-1anchors in[Figure 5](https://arxiv.org/html/2608.10690#S5.F5)\. Coverage increases as more anchors are added, but the marginal gain decreases rapidly and becomes nearly zero atK=14K=14\. This saturation suggests that additional anchors eventually pass through regions that are already covered by the selected trends, rather than adding substantial new support from the known ID–ratio distribution\.

![Refer to caption](https://arxiv.org/html/2608.10690v1/x4.png)Figure 5:Effect of the number of quantile anchors on Quantile Anchor Coverage\. Coverage increases withKK, but marginal gains quickly saturate\.Overall, QAC favors complementary anchors rather than sparse or redundant ones\. Its coverage then saturates asKKgrows, indicating diminishing returns from adding more anchors\.

SourceSingle\-source70%\-mixed sourceTarget\-likeS1S\_\{1\}\-onlyS2S\_\{2\}\-onlyS3S\_\{3\}\-onlyS4S\_\{4\}\-onlyAvg\.±\\pmStd\.S1S\_\{1\}\-majorS2S\_\{2\}\-majorS3S\_\{3\}\-majorS4S\_\{4\}\-majorAvg\.±\\pmStd\.Uniform\\rowcolorgray\!12Language Sources\(S1=S\_\{1\}=En,S2=S\_\{2\}=Fr,S3=S\_\{3\}=Zh,S4=S\_\{4\}=Ja\)Transfer21\.4321\.4031\.2537\.7127\.95±\\pm6\.9222\.1919\.5833\.5322\.3324\.41±\\pm5\.3821\.72PoCTrace24\.9425\.0110\.339\.4017\.42±\\pm7\.5618\.8214\.218\.279\.7512\.76±\\pm4\.1210\.45QGDE Avg\.5\.805\.846\.335\.215\.80±\\pm0\.405\.933\.914\.654\.644\.78±\\pm0\.734\.68K=3K=314\.1118\.059\.1710\.95\\cellcolorceladon\!3213\.07±\\pm3\.3817\.5410\.027\.2910\.78\\cellcolorceladon\!3211\.41±\\pm3\.77\\cellcolorceladon\!3211\.42K=4K=414\.1911\.957\.987\.24\\cellcolorceladon\!3210\.34±\\pm2\.859\.655\.796\.576\.76\\cellcolorceladon\!247\.19±\\pm1\.46\\cellcolorceladon\!247\.29K=5K=57\.467\.917\.996\.08\\cellcolorceladon\!247\.36±\\pm0\.767\.553\.475\.905\.68\\cellcolorceladon\!245\.65±\\pm1\.45\\cellcolorceladon\!245\.75K=6K=64\.484\.447\.084\.65\\cellcolorceladon\!165\.16±\\pm1\.118\.364\.614\.554\.18\\cellcolorceladon\!245\.43±\\pm1\.70\\cellcolorceladon\!163\.94K=7K=73\.463\.446\.275\.03\\cellcolorceladon\!164\.55±\\pm1\.194\.243\.264\.904\.47\\cellcolorceladon\!164\.22±\\pm0\.60\\cellcolorceladon\!164\.44K=8K=84\.694\.205\.494\.95\\cellcolorceladon\!164\.83±\\pm0\.463\.832\.924\.033\.40\\cellcolorceladon\!83\.54±\\pm0\.43\\cellcolorceladon\!83\.45K=9K=93\.953\.765\.634\.03\\cellcolorceladon\!164\.34±\\pm0\.753\.032\.793\.893\.50\\cellcolorceladon\!83\.31±\\pm0\.42\\cellcolorceladon\!83\.39K=10K=103\.853\.144\.844\.07\\cellcolorceladon\!83\.98±\\pm0\.613\.532\.793\.393\.52\\cellcolorceladon\!83\.31±\\pm0\.31\\cellcolorceladon\!83\.51K=11K=113\.203\.175\.103\.54\\cellcolorceladon\!83\.75±\\pm0\.793\.572\.823\.513\.09\\cellcolorceladon\!83\.25±\\pm0\.31\\cellcolorceladon\!83\.00K=12K=122\.912\.945\.293\.76\\cellcolorceladon\!83\.72±\\pm0\.973\.092\.773\.713\.22\\cellcolorceladon\!83\.20±\\pm0\.34\\cellcolorceladon\!83\.09K=13K=133\.543\.435\.493\.98\\cellcolorceladon\!84\.11±\\pm0\.823\.262\.763\.903\.44\\cellcolorceladon\!83\.34±\\pm0\.41\\cellcolorceladon\!83\.28K=14K=143\.783\.665\.644\.23\\cellcolorceladon\!164\.33±\\pm0\.793\.502\.934\.093\.67\\cellcolorceladon\!83\.55±\\pm0\.42\\cellcolorceladon\!83\.55\\rowcolorgray\!12Domain Sources\(S1=S\_\{1\}=Web,S2=S\_\{2\}=Wiki,S3=S\_\{3\}=Math,S4=S\_\{4\}=Code\)Transfer13\.7613\.7553\.0212\.7823\.33±\\pm17\.1510\.4217\.5920\.969\.9714\.73±\\pm4\.7014\.19PoCTrace30\.4127\.5037\.7221\.1329\.19±\\pm5\.9627\.5126\.3425\.2222\.2025\.32±\\pm1\.9724\.32QGDE Avg\.11\.458\.0616\.7724\.2815\.14±\\pm6\.127\.346\.318\.7111\.288\.41±\\pm1\.867\.44K=3K=329\.0818\.6628\.4431\.44\\cellcolorceladon\!3226\.91±\\pm4\.8922\.2117\.3824\.2223\.79\\cellcolorceladon\!3221\.90±\\pm2\.71\\cellcolorceladon\!3222\.24K=4K=425\.1217\.2927\.4332\.78\\cellcolorceladon\!3225\.66±\\pm5\.5719\.5910\.8521\.7022\.15\\cellcolorceladon\!3218\.57±\\pm4\.56\\cellcolorceladon\!2411\.89K=5K=517\.8710\.4526\.0631\.42\\cellcolorceladon\!3221\.45±\\pm7\.974\.664\.8914\.5217\.99\\cellcolorceladon\!2410\.52±\\pm5\.87\\cellcolorceladon\!2412\.76K=6K=618\.6011\.2223\.7630\.94\\cellcolorceladon\!3221\.13±\\pm7\.215\.914\.857\.6415\.04\\cellcolorceladon\!248\.36±\\pm3\.98\\cellcolorceladon\!165\.42K=7K=713\.197\.0320\.6431\.67\\cellcolorceladon\!2418\.13±\\pm9\.184\.474\.484\.7310\.96\\cellcolorceladon\!166\.16±\\pm2\.78\\cellcolorceladon\!84\.44K=8K=85\.334\.5220\.4331\.81\\cellcolorceladon\!2415\.52±\\pm11\.344\.524\.694\.6711\.45\\cellcolorceladon\!166\.33±\\pm2\.96\\cellcolorceladon\!84\.44K=9K=95\.154\.5016\.7731\.28\\cellcolorceladon\!2414\.43±\\pm10\.894\.484\.894\.4711\.02\\cellcolorceladon\!166\.21±\\pm2\.78\\cellcolorceladon\!84\.52K=10K=104\.494\.668\.6330\.05\\cellcolorceladon\!2411\.96±\\pm10\.584\.454\.844\.455\.00\\cellcolorceladon\!84\.69±\\pm0\.24\\cellcolorceladon\!84\.76K=11K=114\.704\.598\.8110\.63\\cellcolorceladon\!167\.18±\\pm2\.624\.495\.014\.584\.49\\cellcolorceladon\!84\.64±\\pm0\.22\\cellcolorceladon\!84\.99K=12K=124\.544\.677\.3210\.53\\cellcolorceladon\!166\.76±\\pm2\.444\.454\.714\.464\.45\\cellcolorceladon\!84\.52±\\pm0\.11\\cellcolorceladon\!84\.67K=13K=134\.634\.577\.619\.54\\cellcolorceladon\!166\.59±\\pm2\.104\.454\.604\.514\.51\\cellcolorceladon\!84\.52±\\pm0\.05\\cellcolorceladon\!84\.60K=14K=144\.654\.525\.309\.27\\cellcolorceladon\!165\.94±\\pm1\.954\.464\.564\.534\.51\\cellcolorceladon\!84\.52±\\pm0\.04\\cellcolorceladon\!84\.54

Table 1:Mean relative error \(MRE\) \(%\) of token\-level ratio estimation across language and domain source mixtures\. Single\-source uses one known corpus; 70%\-mixed uses a 70%\-dominant known\-corpus mixture, with the remaining three sources mixed equally; Target\-like uses the same uniform mixture as the target corpora\. Darker green indicates higher error in the QGDEKK\-ablation rows\.## 6Token\-Level Ratio Estimation

We now test whether QGDE turns the quantile trends into accurate token\-level ratio estimates\.

### 6\.1Evaluation Setting

The evaluation is under different source compositions\. Here, source refers to the known corpus mixture used to fit the estimator, while the target corpora are held fixed as uniform mixtures and used only for evaluation\.

We use the same language and domain categories as in[Section 3](https://arxiv.org/html/2608.10690#S3), but evaluate on target corpora drawn from different datasets\. For languages, mC4 serves as the source side and OSCAR\(Suarezet al\.,[2020](https://arxiv.org/html/2608.10690#bib.bib31)\)as the target side over English, French, Japanese, and Chinese\. For domains, Web, Wiki, Code, and Math are paired with target corpora from the same broad domains: RedPajama\-C4\(Weberet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib37)\), BookCorpus\(Zhuet al\.,[2015](https://arxiv.org/html/2608.10690#bib.bib39)\), RedPajama\-GitHub\(Weberet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib37)\), and FineWebMath\(Allalet al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib38)\)\.

We report mean relative error in[Section 5\.2](https://arxiv.org/html/2608.10690#S5.SS2), where lower values indicate better estimates\. We compare QGDE with two baselines: direct ID\-ratio transfer, which copies the source ratio profile by token position, and PoCTrace\(Zhanget al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib21)\), which estimates ratios from a single median ID–ratio trend\.

### 6\.2Token\-Level Results

[Section 5\.2](https://arxiv.org/html/2608.10690#S5.SS2)yields four main observations\.

#### QGDE outperforms baselines\.

QGDE consistently improves token\-level estimation over both baselines\. Direct ID\-ratio transfer copies the source ratio profile and therefore remains high\-error, showing that distributional transferability does not justify token\-wise ratio copying\. PoCTrace avoids direct copying by fitting a median ID–ratio trend, but a single trend cannot represent the vertical spread of plausible ratios at each token ID\. In contrast, QGDE combines multiple quantile trends with local density weighting\. The laterKK\-rows are substantially lower than both baselines in most source\-mixture settings, especially for mixed known\-corpus settings\.

#### More quantile anchors help, then saturate\.

The ablation onKKconfirms that multiple quantile anchors are necessary, but that their benefit saturates\. Moving fromK=3K=3to largerKKsharply reduces error, especially in the domain block: the single\-source average decreases from 26\.91 to 5\.94, and the 70%\-mixed average decreases from 21\.90 to 4\.52\. Later rows fluctuate within a narrower band, so the bestKKis not universal across source compositions; nevertheless,K=14K=14provides a reasonable high\-coverage default once the QAC gain has saturated\. This echoes the QAC analysis in[Section 5](https://arxiv.org/html/2608.10690#S5): once additional anchors provide little new coverage of the known ID–ratio distribution, they also yield diminishing improvements in token\-level estimation error\.

#### Mixed sources help, but exact ratios matter less\.

Mixed sources are generally preferable to single\-source settings because they expose the estimator to a broader ID–ratio distribution\. This is most visible in the domain block, where the QGDE average drops from 15\.14 for single\-source settings to 8\.41 for 70%\-mixed settings\. Within the mixed\-source group, however, the exact dominant component matters much less: atK=14K=14, the 70%\-mixed columns have low standard deviation, with Std\. of 0\.42 for languages and 0\.04 for domains\. Even matching the target mixture exactly \(Target\-like at the last column\) is not always optimal; what matters more is using a mixed source that covers multiple source components\.

#### Language ratios are easier to predict than domain ratios\.

The language block reaches low error with fewer anchors, whereas the domain block has much larger errors at smallKKand only approaches a similar range after more anchors are used\. This difference reflects how closely the source and target ID–ratio relationships match\. In the language setting, mC4 and OSCAR differ as corpora, but the language\-specific signals that shape token IDs are largely stable across them\. In the domain setting, the matched source and target corpora are less aligned\. Source and target corpora may differ in collection pipelines, so the same broad domain label does not guarantee a similar ID–ratio relationship\.

## 7Aggregating Token Ratios into Mixtures

A useful token\-level estimator should also support corpus mixture estimation\. We test this by aggregating QGDE’s token\-level estimates into language or domain proportions\.

### 7\.1Aggregation Procedure

We convert target token ratios into category proportions by distributing each estimated token ratior^i\\widehat\{r\}\_\{i\}according to how the token appears across known source categories\. For example, a token that appears mostly in one category contributes most of its estimated ratio to that category, while a token that appears across all categories is split according to its relative counts in the known corpora\. Formally, we define the category assignment weight as

πc,i=nc,i∑c′∈𝒞nc′,i,c∈𝒞\.\\pi\_\{c,i\}=\\frac\{n\_\{c,i\}\}\{\\sum\_\{c^\{\\prime\}\\in\\mathcal\{C\}\}n\_\{c^\{\\prime\},i\}\},\\qquad c\\in\\mathcal\{C\}\.\(8\)Here,𝒞\\mathcal\{C\}is the set of source categories, andnc,in\_\{c,i\}is the count of target tokenviv\_\{i\}in the known corpus for categorycc\.

We then normalize the estimated token ratios over token setℐ\\mathcal\{I\}and distribute each token ratio to categories usingπc,i\\pi\_\{c,i\}\. The estimated mixture proportion of categoryccis therefore

α^c=∑i∈ℐr^i∑j∈ℐr^j​πc,i,c∈𝒞\.\\widehat\{\\alpha\}\_\{c\}=\\sum\_\{i\\in\\mathcal\{I\}\}\\frac\{\\widehat\{r\}\_\{i\}\}\{\\sum\_\{j\\in\\mathcal\{I\}\}\\widehat\{r\}\_\{j\}\}\\pi\_\{c,i\},\\qquad c\\in\\mathcal\{C\}\.\(9\)
![Refer to caption](https://arxiv.org/html/2608.10690v1/x5.png)Figure 6:Category\-level mixture estimation, where QGDE matches the ground truth more closely in both language and domain settings\. Each setting compares the ground truth, DMI, and QGDE at threeKKvalues; center labels report mean relative error \(MRE\)\.
### 7\.2Category\-Level Results

[Figure 6](https://arxiv.org/html/2608.10690#S7.F6)evaluates category\-level mixture estimation under uniform language and domain targets\. We compare QGDE aggregation with DMI\(Hayaseet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib17)\), a source\-independent baseline that specifically designed for estimating macro mixture proportions from tokenizer statistics\.

#### Token\-level estimates support mixture estimation\.

The figure shows that QGDE token\-level ratios can be aggregated into meaningful category\-level proportions\. Across both language and domain targets, the QGDE pies track the ground\-truth mixtures closely than the baseline estimates\. This indicates that the token\-level signal recovered by QGDE is not only useful for individual token prediction, but also remains informative after aggregation to categories\.

#### QGDE outperforms baseline DMI\.

DMI gives visibly skewed estimates: in the language setting, it overestimates English and French while underestimating Chinese and Japanese; in the domain setting, it strongly overestimates Web and underestimates the remaining categories\. QGDE substantially reduces these errors, lowering language error from 9\.09 to about 3\.0 and domain error from 15\.14 to a much smaller range\.

#### Anchor gains are weaker after aggregation\.

The effect of increasingKKis less pronounced at the category level than at the token level\. In the language setting, error decreases fromK=3K=3toK=8K=8, but changes only slightly atK=14K=14\. In the domain setting, the trend is even less monotonic, because aggregation can shift estimated ratios among overlapping domain vocabularies even when token\-level predictions improve\. Thus, additional anchors still help by improving the underlying token estimates, but their gains are partially smoothed or redistributed by category\-level aggregation\. The full source\-composition andKK\-sweep results are reported in[Appendix E](https://arxiv.org/html/2608.10690#A5)\.

## 8Validation on the SmolLM Tokenizer

Beyond the controlled language and domain settings, we also test QGDE in a realistic setting:a released tokenizer along with training corpora\. Such validation is uncommon because LLM releases rarely include training corpora\. SmolLM\(Allalet al\.,[2025](https://arxiv.org/html/2608.10690#bib.bib38)\)is a useful exception\.

We use the SmolLM tokenizer as the target tokenizer\. Token\-level ground truth is computed from the training corpus, and category\-level ground truth uses the released component proportions: FineWeb\-edu 87\.30%, Cosmopedia\-v2 11\.11%, and Python\-edu 1\.59%\. Since the exact component corpora cannot be used as known corpora, we fit the estimators on corpora matched to the three components: RedPajama\-C4\(Weberet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib37)\), a Wikipedia–arXiv mixture\(Wikimedia Foundation,[2023](https://arxiv.org/html/2608.10690#bib.bib35); Cornell University,[2020](https://arxiv.org/html/2608.10690#bib.bib36)\), and CodeParrot\(Hugging Face,[2021](https://arxiv.org/html/2608.10690#bib.bib34)\), respectively\. We evaluate three known\-corpus mixture ratios over these proxies: a uniform mixture, a pretrain\-like mixture based on\(Soldainiet al\.,[2024](https://arxiv.org/html/2608.10690#bib.bib40)\), and a target\-like mixture that matches the released SmolLM component proportions as a diagnostic setting\.[Section 8](https://arxiv.org/html/2608.10690#S8)shows the results\.

SourceUniformPretrain\-likeTarget\-like\\rowcolorgray\!12 Token\-levelTransfer9\.0610\.537\.71PoCTrace13\.8418\.6820\.33QGDE5\.785\.735\.72\\rowcolorgray\!12 Category\-levelDMI9\.119\.119\.11QGDE6\.085\.905\.93

Table 2:Validation on the released SmolLM tokenizer, where QGDE achieves the lowest error for both token\-level and category\-level estimation\.At the token\-level, QGDE outperforms both direct ID\-ratio transfer and PoCTrace, reaching 5\.72–5\.78 MRE\. The small spread across Uniform, Pretrain\-like, and Target\-like source mixture also echoes the controlled experiments: once the known\-corpus mixture covers the relevant components, the exact mixture ratio is less important\.

At the category level, QGDE also improves over the source\-independent DMI baseline, reducing error from 9\.11 to about 5\.9–6\.1\. These estimates are less exact than in the controlled category\-level experiments because the component proportions are highly imbalanced; in particular, Python\-edu accounts for only 1\.59%\. Nevertheless, QGDE still recovers a better mixture estimate\.

## 9Conclusion

Released LLM vocabularies provide a useful signal for estimating hidden corpus composition beyond coarse category proportions\. We show that BPE tokenizers share stable token ID–ratio distributions across corpora, and introduce QGDE to transfer this structure through quantile trends and local density weighting\. Across controlled settings and the released SmolLM tokenizer, QGDE achieves relative errors as low as 3\.00% for token\-level estimation and 3\.08% after aggregation into category\-level mixtures\.

## Limitations

#### Scarcity of ground truth for released LLM tokenizers\.

ChatGPT, Qwen, and DeepSeek release tokenizer vocabularies, but not their training corpora\. This prevents direct evaluation of token\-level ratio estimates on these models\. We therefore validate QGDE in controlled settings and on SmolLM, a rare released tokenizer with available training data\. These experiments show that the approach is effective when ground truth is available\. Broader validation will become possible if more LLM training corpora are released\.

## Ethics Statement

ACL Ethics Policy is respected in this work\. This work studies corpus ratio estimation from released tokenizer vocabularies and known corpora\. We use publicly available or controlled corpora for research purposes, and we respect the terms, conditions, and copyright requirements of the corresponding data sources\. No human subjects or private personal data are involved\. The proposed methods are intended for research use, transparency analysis, and auditing of corpus composition signals from released tokenizers\.

We adhere to the Association for Computational Linguistics \(ACL\) guidelines on responsible NLP research222[https://aclrollingreview\.org/responsibleNLPresearch/](https://aclrollingreview.org/responsibleNLPresearch/), with particular attention to transparency, research\-use framing, and responsible handling of corpus\-derived evidence\.

## Use of AI Assistants

The authors used AI assistants for language polishing, LaTeX editing, and phrasing suggestions during paper preparation\. All substantive claims, experimental results, analyses, citations, and final text were reviewed and verified by the authors\.

## References

- L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav,et al\.\(2025\)SmolLM2: when smol goes big–data\-centric training of a small language model\.Cited by:[§6\.1](https://arxiv.org/html/2608.10690#S6.SS1.p2.1),[§8](https://arxiv.org/html/2608.10690#S8.p1.1)\.
- Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- X\. Bi, D\. Chen, G\. Chen, S\. Chen, D\. Dai, C\. Deng, H\. Ding, K\. Dong, Q\. Du, Z\. Fu,et al\.\(2024\)Deepseek llm: scaling open\-source language models with longtermism\.arXiv preprint arXiv:2401\.02954\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- R\. Bommasani, K\. Klyman, S\. Longpre, S\. Kapoor, N\. Maslej, B\. Xiong, D\. Zhang, and P\. Liang \(2023\)The foundation model transparency index\.arXiv preprint arXiv:2310\.12941\.Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. Zhang \(2022\)Quantifying memorization across neural language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson,et al\.\(2021\)Extracting training data from large language models\.In30th USENIX security symposium \(USENIX Security 21\),pp\. 2633–2650\.Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- Cornell University \(2020\)arXiv dataset\.Note:[https://www\.kaggle\.com/datasets/Cornell\-University/arxiv](https://www.kaggle.com/datasets/Cornell-University/arxiv)Cited by:[§8](https://arxiv.org/html/2608.10690#S8.p2.1)\.
- J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld, M\. Mitchell, and M\. Gardner \(2021\)Documenting large webtext corpora: a case study on the colossal clean crawled corpus\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 1286–1305\.Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- J\. Hayase, A\. Liu, Y\. Choi, S\. Oh, and N\. A\. Smith \(2024\)Data mixture inference attack: bpe tokenizers reveal training data compositions\.Advances in Neural Information Processing Systems37,pp\. 8956–8983\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px2.p1.1),[§2\.2](https://arxiv.org/html/2608.10690#S2.SS2.p2.2),[§3](https://arxiv.org/html/2608.10690#S3.p1.1),[§7\.2](https://arxiv.org/html/2608.10690#S7.SS2.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- Hugging Face \(2021\)CodeParrot\.Note:[https://huggingface\.co/codeparrot](https://huggingface.co/codeparrot)Cited by:[§3](https://arxiv.org/html/2608.10690#S3.p2.1),[§8](https://arxiv.org/html/2608.10690#S8.p2.1)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora,et al\.\(2024\)Datacomp\-lm: in search of the next generation of training sets for language models\.Advances in Neural Information Processing Systems37,pp\. 14200–14282\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- S\. Longpre, R\. Mahari, A\. Chen, N\. Obeng\-Marnu, D\. Sileo, W\. Brannon, N\. Muennighoff, N\. Khazam, J\. Kabbara, K\. Perisetla,et al\.\(2024\)A large\-scale audit of dataset licensing and attribution in ai\.Nature Machine Intelligence6\(8\),pp\. 975–987\.Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- K\. P\. Murphy \(2022\)Probabilistic machine learning: an introduction\.MIT Press\.Cited by:[Appendix B](https://arxiv.org/html/2608.10690#A2.p3.6),[§3](https://arxiv.org/html/2608.10690#S3.p4.8)\.
- K\. Paster, M\. Dos Santos, Z\. Azerbayev, and J\. Ba \(2024\)Openwebmath: an open dataset of high\-quality mathematical web text\.Vol\.2024\.Cited by:[§3](https://arxiv.org/html/2608.10690#S3.p2.1)\.
- G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, T\. Wolf,et al\.\(2024\)The fineweb datasets: decanting the web for the finest text data at scale\.Vol\.37\.Cited by:[§3](https://arxiv.org/html/2608.10690#S3.p2.1)\.
- J\. Petty, S\. van Steenkiste, and T\. Linzen \(2024\)How does code pretraining affect language model task performance?\.arXiv preprint arXiv:2409\.04556\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- S\. T\. Piantadosi \(2014\)Zipf’s word frequency law in natural language: a critical review and future directions\.Psychonomic bulletin & review21\(5\),pp\. 1112–1130\.Cited by:[§4\.1](https://arxiv.org/html/2608.10690#S4.SS1.p1.1)\.
- Q\. Regression \(2017\)Handbook of quantile regression\.Boca Raton, FL, USA: CRC\.Cited by:[Appendix A](https://arxiv.org/html/2608.10690#A1.p1.1),[§4\.1](https://arxiv.org/html/2608.10690#S4.SS1.p3.5)\.
- A\. I\. Saichev, Y\. Malevergne, and D\. Sornette \(2009\)Theory of zipf’s law and beyond\.Vol\.632,Springer Science & Business Media\.Cited by:[§4\.1](https://arxiv.org/html/2608.10690#S4.SS1.p1.1)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Neural machine translation of rare words with subword units\.InProceedings of the 54th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 1715–1725\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2608.10690#S2.SS2.p2.2),[§3](https://arxiv.org/html/2608.10690#S3.p1.1)\.
- B\. W\. Silverman \(2018\)Density estimation for statistics and data analysis\.Routledge\.Cited by:[§4\.3](https://arxiv.org/html/2608.10690#S4.SS3.p2.2)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- L\. Soldaini, R\. Kinney, A\. Bhagia, D\. Schwenk, D\. Atkinson, R\. Authur, B\. Bogin, K\. Chandu, J\. Dumas, Y\. Elazar,et al\.\(2024\)Dolma: an open corpus of three trillion tokens for language model pretraining research\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15725–15788\.Cited by:[§8](https://arxiv.org/html/2608.10690#S8.p2.1)\.
- P\. O\. Suarez, L\. Romary, and B\. Sagot \(2020\)A monolingual approach to contextualized word embeddings for mid\-resource languages\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 1703–1714\.Cited by:[§6\.1](https://arxiv.org/html/2608.10690#S6.SS1.p2.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- M\. Weber, D\. Y\. Fu, Q\. Anthony, Y\. Oren, S\. Adams, A\. Alexandrov, X\. Lyu, H\. Nguyen, X\. Yao, V\. Adams,et al\.\(2024\)Redpajama: an open dataset for training large language models\.Vol\.37\.Cited by:[§6\.1](https://arxiv.org/html/2608.10690#S6.SS1.p2.1),[§8](https://arxiv.org/html/2608.10690#S8.p2.1)\.
- Wikimedia Foundation \(2023\)Wikipedia dumps\.Note:[https://dumps\.wikimedia\.org/](https://dumps.wikimedia.org/)Cited by:[§3](https://arxiv.org/html/2608.10690#S3.p2.1),[§8](https://arxiv.org/html/2608.10690#S8.p2.1)\.
- S\. M\. Xie, H\. Pham, X\. Dong, N\. Du, H\. Liu, Y\. Lu, P\. S\. Liang, Q\. V\. Le, T\. Ma, and A\. W\. Yu \(2023\)Doremi: optimizing data mixtures speeds up language model pretraining\.Vol\.36,pp\. 69798–69818\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- C\. Xu, S\. Guan, D\. Greene, M\. Kechadi,et al\.\(2024\)Benchmark data contamination of large language models: a survey\.arXiv preprint arXiv:2406\.04244\.Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. Raffel \(2021\)MT5: a massively multilingual pre\-trained text\-to\-text transformer\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies,pp\. 483–498\.Cited by:[§3](https://arxiv.org/html/2608.10690#S3.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p1.1)\.
- S\. Yang, W\. Chiang, L\. Zheng, J\. E\. Gonzalez, and I\. Stoica \(2023\)Rethinking benchmark and contamination for language models with rephrased samples\.arXiv preprint arXiv:2311\.04850\.Cited by:[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1)\.
- Q\. Zhang, D\. Wang, H\. Qian, L\. Yan, T\. Zhang, K\. Xu, Q\. Li, M\. Huang, H\. Li, and H\. Qiu \(2025\)Speculating llms’ chinese training data pollution from their tokens\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 26124–26144\.Cited by:[§1](https://arxiv.org/html/2608.10690#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2608.10690#S2.SS1.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.10690#S3.p1.1),[§4\.3](https://arxiv.org/html/2608.10690#S4.SS3.p3.2),[§4](https://arxiv.org/html/2608.10690#S4.p2.1),[§6\.1](https://arxiv.org/html/2608.10690#S6.SS1.p3.1)\.
- Y\. Zhu, R\. Kiros, R\. Zemel, R\. Salakhutdinov, R\. Urtasun, A\. Torralba, and S\. Fidler \(2015\)Aligning books and movies: towards story\-like visual explanations by watching movies and reading books\.Cited by:[§6\.1](https://arxiv.org/html/2608.10690#S6.SS1.p2.1)\.

## Appendix AQuantile Trend Fitting Details

In[Section 4\.1](https://arxiv.org/html/2608.10690#S4.SS1), QGDE fits one log\-linear trend for each quantile level over the known ID–ratio points using quantile regression\(Regression,[2017](https://arxiv.org/html/2608.10690#bib.bib41)\)\. This section gives the optimization objective used to fit those trends\.

Let𝒫=\{\(xj,yj\)\}j=1n\\mathcal\{P\}=\\\{\(x\_\{j\},y\_\{j\}\)\\\}\_\{j=1\}^\{n\}denote the known ID–ratio points, wherexj=log⁡tjx\_\{j\}=\\log t\_\{j\},yj=log⁡rjy\_\{j\}=\\log r\_\{j\},tjt\_\{j\}is the token ID, andrjr\_\{j\}is the observed ratio in the known corpus\. For each quantile levelτ∈𝒯\\tau\\in\\mathcal\{T\}, we estimate the trend coefficients by

\(aτ,bτ\)=arg⁡mina,b​∑j=1nρτ​\(yj−a−b​xj\),\(a\_\{\\tau\},b\_\{\\tau\}\)=\\arg\\min\_\{a,b\}\\sum\_\{j=1\}^\{n\}\\rho\_\{\\tau\}\\left\(y\_\{j\}\-a\-bx\_\{j\}\\right\),\(10\)whereρτ​\(⋅\)\\rho\_\{\\tau\}\(\\cdot\)is the asymmetric loss:

ρτ​\(u\)=\{τ​u,u≥0,\(τ−1\)​u,u<0\.\\rho\_\{\\tau\}\(u\)=\\begin\{cases\}\\tau u,&u\\geq 0,\\\\ \(\\tau\-1\)u,&u<0\.\\end\{cases\}\(11\)The fitted trend isqτ​\(x\)=aτ\+bτ​xq\_\{\\tau\}\(x\)=a\_\{\\tau\}\+b\_\{\\tau\}x\. The asymmetric loss encourages approximately aτ\\taufraction of known points to lie below the trend, so differentτ\\tauvalues trace different vertical levels of the ID–ratio distribution\. A single median trend captures only the central tendency; QGDE fits a family of trends so that the vertical spread of plausible ratios is preserved before local density weighting\.

## Appendix BTransfer Similarity Computation

In[Section 3](https://arxiv.org/html/2608.10690#S3),[Figure 2](https://arxiv.org/html/2608.10690#S3.F2)quantifies whether ID–ratio distributions transfer across tokenizers\. We compute this score by converting each scatter plot into a probability densityPDP\_\{D\}over ID–ratio space\.

For each tokenizerDD, each tokenviv\_\{i\}is represented as\(xi,yi\)=\(log⁡ti,log⁡ri\)\(x\_\{i\},y\_\{i\}\)=\(\\log t\_\{i\},\\log r\_\{i\}\), wheretit\_\{i\}is its token ID andrir\_\{i\}is its corpus ratio\. We place all tokenizers on a shared gridℬ\\mathcal\{B\}and convert each tokenizer into a smoothed two\-dimensional histogram:

PD​\(b\)=∑vi∈VD𝟏​\[\(xi,yi\)∈b\]\+ϵ\|VD\|\+ϵ​\|ℬ\|,b∈ℬ\.P\_\{D\}\(b\)=\\frac\{\\sum\_\{v\_\{i\}\\in V\_\{D\}\}\\mathbf\{1\}\\\!\\left\[\(x\_\{i\},y\_\{i\}\)\\in b\\right\]\+\\epsilon\}\{\|V\_\{D\}\|\+\\epsilon\|\\mathcal\{B\}\|\},\\qquad b\\in\\mathcal\{B\}\.\(12\)Here,ϵ\\epsilonis a small smoothing constant\. SettingD=SD=SorD=TD=Tgives the source and target densitiesPSP\_\{S\}andPTP\_\{T\}used in the main text\.

The score is directional:Sim​\(S→T\)\\mathrm\{Sim\}\(S\\to T\)asks how well the source densityPSP\_\{S\}explains the target densityPTP\_\{T\}\. We therefore computeDKL​\(PT∥PS\)D\_\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|P\_\{S\}\), normalize it by the target entropyH​\(PT\)H\(P\_\{T\}\), and map the result to\(0,1\]\(0,1\], where higher values indicate stronger transfer similarity\(Murphy,[2022](https://arxiv.org/html/2608.10690#bib.bib42)\)\.

## Appendix CSingle\-source and Mixed\-source Transfer Similarity

In[Section 3](https://arxiv.org/html/2608.10690#S3), the main transfer analysis compares single\-category tokenizers\. Because the token\-level experiments also use mixed known\-corpus sources,[Table 3](https://arxiv.org/html/2608.10690#A3.T3)and[Table 4](https://arxiv.org/html/2608.10690#A3.T4)report transfer similarity from both single\-source and mixed\-source tokenizers to single\-category targets\. Rows specify the source tokenizer’s training mixture, and columns specify the target tokenizer whose ID–ratio density is explained\.

Source RatioTargetWeb:Wiki:Code:MathWebWikiCodeMath100:0:0:01\.000\.930\.870\.950:100:0:00\.941\.000\.850\.910:0:100:00\.860\.781\.000\.930:0:0:1000\.960\.900\.921\.0070:10:10:100\.960\.930\.840\.9110:70:10:100\.930\.970\.820\.8910:10:70:100\.940\.960\.820\.8910:10:10:700\.970\.910\.870\.9425:25:25:250\.950\.950\.830\.91Table 3:Directional transfer similarity of domains\.Source RatioTargetEn:Fr:Ja:ZhEnFrJaZh100:0:0:01\.000\.990\.780\.800:100:0:00\.991\.000\.780\.800:0:100:00\.850\.851\.000\.960:0:0:1000\.850\.850\.961\.0070:10:10:100\.950\.950\.890\.8810:70:10:100\.910\.910\.940\.9310:10:70:100\.840\.840\.970\.9610:10:10:700\.840\.840\.970\.9625:25:25:250\.870\.870\.970\.95Table 4:Directional transfer similarity of languages\.The mixed\-source profiles support the token\-level results in[Section 5\.2](https://arxiv.org/html/2608.10690#S5.SS2)\. First, ID–ratio distributions remain broadly transferable after mixing: most mixed\-source similarities are still above 0\.8, and many domain similarities are close to or above 0\.9\. Second, mixed sources retain useful similarity to multiple targets rather than only to a single self\-matched target\. This helps explain why mixed known\-corpus sources are generally more stable for token\-level estimation\. Third, the same structure as in the main heatmap remains visible: English–French and Japanese–Chinese form stronger language clusters, while Code is the hardest domain target because its ID–ratio distribution differs more from natural language corpora\.

![Refer to caption](https://arxiv.org/html/2608.10690v1/Figs/id_ratio_scatter_grid_all_preview.png)Figure 7:Token ID–ratio scatter plots for all eight controlled tokenizers\. The top row shows language\-specific mC4 tokenizers, and the bottom row shows domain\-specific English tokenizers\.
## Appendix DAdditional ID–Ratio Scatter Plots

In[Section 3](https://arxiv.org/html/2608.10690#S3),[Figure 2](https://arxiv.org/html/2608.10690#S3.F2)shows representative ID–ratio scatter plots to motivate transferability\.[Figure 7](https://arxiv.org/html/2608.10690#A3.F7)provides the full set of eight controlled tokenizers: the top row shows language\-specific mC4 tokenizers, and the bottom row shows domain\-specific English tokenizers\.

Across both rows, the point clouds follow a similar downward log–log shape, while their thickness, location, and tail behavior vary by language and domain\. These patterns support the main\-text conclusion that the ID–ratio relationship is broadly shared but not identical across corpora\.

## Appendix ECategory\-Level Estimation Details

In[Section 7](https://arxiv.org/html/2608.10690#S7),[Figure 6](https://arxiv.org/html/2608.10690#S7.F6)shows selected category\-level estimates for the uniform language and domain targets\. We present the complete source\-composition andKK\-sweep results in this section, showing that the main conclusions do not depend on a single displayedKKor source mixture\.

SourceSingle\-source70%\-mixed sourceTarget\-likeS1S\_\{1\}\-onlyS2S\_\{2\}\-onlyS3S\_\{3\}\-onlyS4S\_\{4\}\-onlyAvg\.±\\pmStd\.S1S\_\{1\}\-majorS2S\_\{2\}\-majorS3S\_\{3\}\-majorS4S\_\{4\}\-majorAvg\.±\\pmStd\.Uniform\\rowcolorgray\!12Language Sources\(S1=S\_\{1\}=En,S2=S\_\{2\}=Fr,S3=S\_\{3\}=Zh,S4=S\_\{4\}=Ja\)DMI9\.09\(source\-independent\)QGDE Avg\.3\.253\.213\.223\.143\.21±\\pm0\.043\.203\.163\.143\.143\.16±\\pm0\.023\.12K=3K=33\.703\.583\.333\.43\\cellcolorceladon\!323\.51±\\pm0\.143\.593\.473\.213\.47\\cellcolorceladon\!323\.44±\\pm0\.14\\cellcolorceladon\!323\.42K=4K=43\.623\.533\.263\.23\\cellcolorceladon\!323\.41±\\pm0\.173\.393\.273\.163\.25\\cellcolorceladon\!243\.27±\\pm0\.08\\cellcolorceladon\!163\.17K=5K=53\.433\.413\.263\.17\\cellcolorceladon\!243\.32±\\pm0\.113\.313\.253\.153\.14\\cellcolorceladon\!163\.22±\\pm0\.07\\cellcolorceladon\!83\.11K=6K=63\.323\.313\.223\.08\\cellcolorceladon\!243\.23±\\pm0\.103\.293\.213\.113\.08\\cellcolorceladon\!163\.17±\\pm0\.08\\cellcolorceladon\!83\.07K=7K=73\.203\.183\.193\.11\\cellcolorceladon\!163\.17±\\pm0\.033\.133\.113\.143\.12\\cellcolorceladon\!163\.12±\\pm0\.01\\cellcolorceladon\!83\.11K=8K=83\.183\.083\.183\.11\\cellcolorceladon\!163\.13±\\pm0\.043\.153\.133\.143\.09\\cellcolorceladon\!163\.13±\\pm0\.02\\cellcolorceladon\!83\.10K=9K=93\.093\.013\.193\.07\\cellcolorceladon\!83\.09±\\pm0\.063\.073\.083\.123\.09\\cellcolorceladon\!83\.09±\\pm0\.02\\cellcolorceladon\!83\.09K=10K=103\.103\.103\.183\.08\\cellcolorceladon\!83\.11±\\pm0\.043\.103\.093\.093\.10\\cellcolorceladon\!83\.09±\\pm0\.00\\cellcolorceladon\!83\.09K=11K=113\.113\.113\.203\.07\\cellcolorceladon\!163\.12±\\pm0\.053\.123\.103\.113\.08\\cellcolorceladon\!83\.10±\\pm0\.02\\cellcolorceladon\!83\.04K=12K=123\.053\.043\.223\.10\\cellcolorceladon\!83\.10±\\pm0\.073\.063\.053\.133\.09\\cellcolorceladon\!83\.08±\\pm0\.03\\cellcolorceladon\!83\.06K=13K=133\.083\.063\.233\.12\\cellcolorceladon\!163\.12±\\pm0\.073\.083\.063\.153\.10\\cellcolorceladon\!83\.10±\\pm0\.03\\cellcolorceladon\!83\.08K=14K=143\.173\.143\.243\.13\\cellcolorceladon\!163\.17±\\pm0\.043\.123\.083\.153\.11\\cellcolorceladon\!83\.12±\\pm0\.03\\cellcolorceladon\!83\.09\\rowcolorgray\!12Domain Sources\(S1=S\_\{1\}=Web,S2=S\_\{2\}=Wiki,S3=S\_\{3\}=Math,S4=S\_\{4\}=Code\)DMI15\.14\(source\-independent\)QGDE Avg\.5\.445\.375\.405\.345\.39±\\pm0\.045\.445\.375\.485\.395\.42±\\pm0\.055\.37K=3K=35\.255\.445\.275\.24\\cellcolorceladon\!85\.30±\\pm0\.085\.335\.405\.405\.26\\cellcolorceladon\!165\.35±\\pm0\.06\\cellcolorceladon\!85\.32K=4K=45\.545\.265\.395\.27\\cellcolorceladon\!165\.37±\\pm0\.115\.235\.405\.425\.33\\cellcolorceladon\!165\.34±\\pm0\.08\\cellcolorceladon\!85\.28K=5K=55\.505\.355\.295\.27\\cellcolorceladon\!165\.35±\\pm0\.095\.605\.325\.515\.32\\cellcolorceladon\!325\.44±\\pm0\.12\\cellcolorceladon\!85\.29K=6K=65\.475\.475\.415\.30\\cellcolorceladon\!325\.42±\\pm0\.075\.505\.365\.465\.27\\cellcolorceladon\!245\.40±\\pm0\.09\\cellcolorceladon\!325\.43K=7K=75\.515\.395\.355\.34\\cellcolorceladon\!245\.40±\\pm0\.075\.445\.395\.515\.42\\cellcolorceladon\!325\.44±\\pm0\.05\\cellcolorceladon\!325\.46K=8K=85\.535\.415\.305\.34\\cellcolorceladon\!245\.40±\\pm0\.095\.465\.395\.525\.43\\cellcolorceladon\!325\.45±\\pm0\.05\\cellcolorceladon\!245\.39K=9K=95\.425\.305\.405\.38\\cellcolorceladon\!245\.37±\\pm0\.055\.435\.395\.525\.43\\cellcolorceladon\!325\.44±\\pm0\.05\\cellcolorceladon\!165\.37K=10K=105\.385\.345\.425\.37\\cellcolorceladon\!245\.38±\\pm0\.035\.475\.385\.545\.49\\cellcolorceladon\!325\.47±\\pm0\.06\\cellcolorceladon\!165\.37K=11K=115\.355\.345\.505\.38\\cellcolorceladon\!245\.39±\\pm0\.065\.475\.365\.535\.43\\cellcolorceladon\!325\.45±\\pm0\.06\\cellcolorceladon\!165\.36K=12K=125\.365\.355\.565\.37\\cellcolorceladon\!325\.41±\\pm0\.095\.455\.355\.485\.43\\cellcolorceladon\!325\.43±\\pm0\.05\\cellcolorceladon\!165\.36K=13K=135\.455\.405\.485\.38\\cellcolorceladon\!325\.43±\\pm0\.045\.475\.325\.455\.40\\cellcolorceladon\!325\.41±\\pm0\.06\\cellcolorceladon\!245\.40K=14K=145\.485\.365\.475\.44\\cellcolorceladon\!325\.44±\\pm0\.055\.495\.365\.465\.43\\cellcolorceladon\!325\.44±\\pm0\.05\\cellcolorceladon\!325\.41

Table 5:Mean relative error \(MRE\) \(%\) of category\-level mixture estimation under uniform targets\. The source\-mixture columns follow the same layout as[Section 5\.2](https://arxiv.org/html/2608.10690#S5.SS2)\. DMI is source\-independent and therefore shown as a single merged value\. Darker green indicates higher error in the QGDEKK\-ablation rows\.[Appendix E](https://arxiv.org/html/2608.10690#A5)follows the same layout as[Section 5\.2](https://arxiv.org/html/2608.10690#S5.SS2): single\-source columns use one known corpus, 70%\-mixed columns use one dominant known corpus with the other three mixed in equally, and the Target\-like column uses the same uniform mixture as the target\. We report category\-level mean relative error, scaled by 100\.

#### QGDE remains below DMI\.

Across both language and domain targets, QGDE gives substantially lower category\-level error than the source\-independent DMI baseline\. In the language setting, DMI has error 9\.09, while the QGDE averages are around 3\.1–3\.3 across source mixtures\. In the domain setting, DMI has error 15\.14, while QGDE stays around 5\.3–5\.5\.

#### Anchor gains are weaker after aggregation\.

TheKK\-sweep is less monotonic than in token\-level estimation\. For languages, increasingKKimproves the early rows but the results quickly cluster near 3\.1\. For domains, most QGDE rows remain in a narrow band around 5\.3–5\.5\. This supports the observation in[Section 7](https://arxiv.org/html/2608.10690#S7)that category\-level aggregation smooths and redistributes token\-level improvements, so additional anchors have a weaker visible effect after aggregation\.

#### Language mixtures remain easier\.

The language block consistently has lower error than the domain block\. This matches the main results: language categories provide more separable token\-level evidence, whereas domain categories share more vocabulary and therefore make aggregate mixture estimation harder\.