Off-Policy Evaluation for Semantic ID Recommenders: Does the Model's Own Code Hierarchy Help?

# Off-Policy Evaluation for Semantic ID Recommenders:Does the Model’s Own Code Hierarchy Help?
Source: [https://arxiv.org/html/2608.28905](https://arxiv.org/html/2608.28905)
###### Abstract\.

Generative recommenders increasingly emit*semantic IDs*\(SIDs\): each item is a short sequence of hierarchical discrete codes from a residual quantizer, decoded autoregressively\. Before spending scarce A/B\-test, a team may decide*offline*which decoder or reranking variants are worth testing \- a job for off\-policy evaluation \(OPE\)\. We ask a simple question: can the model’s*own*SID tree serve as the action abstraction for that OPE? Our answer has three parts\. \(i\) Under the near\-argmax logging real recommenders use, per\-item OPE is hopeless \- as item\-level effective sample size is usually small on production logs \- but marginalizing items to code\-prefix*clusters*restores estimable support and cuts error\. \(ii\) This gain is thanks to*coarsening*, not to the hierarchy specifically; but the SID tree is what makes coarsening*feasible*in a generative system \- each cluster’s mass is exactly and cheaply returned by the decoder, whereas flat clustering requires enumerating item/leaf masses that a code\-only decoder does not directly expose\. \(iii\) Resolution depth is the operative knob \- coarser under scarce support \- and a conditional bias bound links the coarsening bias to the quantizer’s*worst\-case*reconstruction residual and the target \- logging divergence\.

## 1\.Introduction

Generative retrieval \- encoding each item as a short sequence of hierarchical discrete codes called*semantic ID*\(SID\), and decoding next item SID autoregressively \- is moving into production recommenders\([Rajput et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib9);[Wang et al\., 2024](https://arxiv.org/html/2608.28905#bib.bib18)\)\. Every change to such a system \(e\.g\. new decoder, decoding temperature, or reranking rule\) shifts the distribution of items it surfaces\. Deciding which candidate changes deserve scarce, costly A/B\-test is the role of*off\-policy evaluation*\(OPE\): estimate a candidate policy’s online reward from logs collected under the incumbent, so candidates can be ranked and filtered before live test\.

The obstacle is how production systems log\. Exploration is typically*near\-argmax*\- weighted by item score, not uniform \- so a handful of items dominate each impression and the importance weights of per\-item inverse propensity scoring \(IPS\) explode\. On our production logs the item\-level effective sample size \(ESS\) is only≈0\.003\{\\approx\}0\.003of the raw count: per\-item OPE is not a usable decision tool\. Our scope is the*item\-selection*policy on a fixed catalog \(logging and target choose items from the same pool\); we do not address a changed candidate generator, which alters the action set and may surface items with no logged support\([Wang et al\., 2025](https://arxiv.org/html/2608.28905#bib.bib17)\)\.

A SID model hands us a ready\-made action abstraction: its code prefixes form a nested partition of the catalog\. Marginalizing per\-item propensities up to a code\-prefix*cluster*pools support and makes weights estimable again, at the cost of a within\-cluster bias\. Crucially, an autoregressive decoder already produces each prefix’s probability*exactly*, as a product of per\-level conditionals, without summing over the catalog \- so code depth is a native, cheap resolution knob\. Whether a SID tree is*reward\-coherent*enough for this to pay off is not obvious, and depends on the codes\.

We provide following contributions: i\) We frame OPE where the action abstraction*is*the model’s SID tree, and give a production diagnostic showing per\-item OPE is infeasible under realistic near\-argmax logging\. ii\) We propose then to marginalize to code prefixes; a conditional bias bound \(Prop\.[3\.1](https://arxiv.org/html/2608.28905#S3.Thmtheorem1)\) ties the coarsening bias of cluster marginal\-IPS to the quantizer’s worst\-case reconstruction residual and the target \- logging divergence\. iii\) We conduct series of experiments including support recovery, distinct values of SID tree, cross\-domain comparison of the estimators, embedding sources, quantizers and depth selection from logged data\.

#### Related work\.

Large\-action OPE supplies the estimators we reuse at code granularity:*MIPS*reweights over a coarser action embedding, trading variance for marginalization bias\([Saito and Joachims, 2022](https://arxiv.org/html/2608.28905#bib.bib11)\);*OffCEM*adds a reward model plus a residual weight, so the bias is only what the model misses\([Saito et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib12)\);*Policy Convolution*\(PC\) smooths over an embedding kernel whose bandwidth plays the role of resolution\([Sachdeva et al\., 2024](https://arxiv.org/html/2608.28905#bib.bib10)\);*POTEC*decomposes into clusters for policy*learning*\([Saito et al\., 2025](https://arxiv.org/html/2608.28905#bib.bib13)\)\. All take a flat embedding or ad\-hoc clustering as given\. The nearest generative\-retrieval work adapts doubly\-robust estimation*inside*an off\-policy REINFORCE learning objective over dense two\-tower embeddings, not over a code tree\([Matveev et al\., 2026](https://arxiv.org/html/2608.28905#bib.bib8)\); to our knowledge no prior work evaluates policies on the SID tree itself\. Others*learn*action embeddings\([Cief et al\., 2024a](https://arxiv.org/html/2608.28905#bib.bib2)\)or select estimators and hyperparameters from data\([Udagawa et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib16);[Felicioni et al\., 2024](https://arxiv.org/html/2608.28905#bib.bib5)\)\- building blocks for the open depth\-selection step \(RQ5\)\. Closest in spirit, the marginal density\-ratio estimator\([Taufiq et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib15)\)learns its ratio*from*the base per\-action propensities, so it reintroduces the item\-level dependence a code\-only decoder does not expose\. We instead test the retriever’s*native*tree\.

## 2\.OPE on Semantic ID Trees

*Setup and notation\.*A logging policyπ0​\(a∣x\)\\pi\_\{0\}\(a\\mid x\)generates logs\{\(xi,ai,ri\)\}i=1M\\\{\(x\_\{i\},a\_\{i\},r\_\{i\}\)\\\}\_\{i=1\}^\{M\}: contextxix\_\{i\}, chosen itemaia\_\{i\}from catalog𝒜\\mathcal\{A\}, rewardrir\_\{i\}with meanq⁡\(x,a\)q\(x,a\)\. We want the value of a target policy,V⁡\(πe\)=𝔼x​𝔼a∼πe​\[q⁡\(x,a\)\]V\(\\pi\_\{e\}\)=\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{a\\sim\\pi\_\{e\}\}\[q\(x,a\)\]\. Each itemaahas a SIDc⁡\(a\)=\(c1,…,cL\)c\(a\)=\(c\_\{1\},\\dots,c\_\{L\}\)from a residual quantizer of an item embeddingz⁡\(a\)z\(a\); the length\-ℓ\\ellprefixc≤ℓ​\(a\)c\_\{\\leq\\ell\}\(a\)names a*cluster*Ac=\{a′:c≤ℓ​\(a′\)=c\}A\_\{c\}=\\\{a^\{\\prime\}:c\_\{\\leq\\ell\}\(a^\{\\prime\}\)=c\\\}\. A policy’s mass on that cluster isπ⁡\(c∣x\)=∑a∈Acπ⁡\(a∣x\)\\pi\(c\\mid x\)=\\sum\_\{a\\in A\_\{c\}\}\\pi\(a\\mid x\)\. Item\-level IPS reweights byπe​\(a\)/π0​\(a\)\\pi\_\{e\}\(a\)/\\pi\_\{0\}\(a\)and needs per\-item overlap; the self\-normalized variant \(SNIPS\) divides by the summed weights to curb variance\.

*Marginalizing to the tree\.*The core move is to reweight at cluster granularity instead:V^ℓ=1M​∑iπe​\(c≤ℓ​\(ai\)∣xi\)π0​\(c≤ℓ​\(ai\)∣xi\)​ri\\hat\{V\}\_\{\\ell\}=\\frac\{1\}\{M\}\\sum\_\{i\}\\tfrac\{\\pi\_\{e\}\(c\_\{\\leq\\ell\}\(a\_\{i\}\)\\mid x\_\{i\}\)\}\{\\pi\_\{0\}\(c\_\{\\leq\\ell\}\(a\_\{i\}\)\\mid x\_\{i\}\)\}\\,r\_\{i\}, a hierarchical MIPS estimator\([Saito and Joachims, 2022](https://arxiv.org/html/2608.28905#bib.bib11)\)\. It corrects the target’s distribution*over*clusters while leaving the logging distribution*within*each cluster untouched, so its only bias is within\-cluster target \- logging mismatch \(Section[3](https://arxiv.org/html/2608.28905#S3)\)\. It requires only*cluster positivity*\-π0​\(c∣x\)\>0\\pi\_\{0\}\(c\\mid x\)\>0whereverπe​\(c∣x\)\>0\\pi\_\{e\}\(c\\mid x\)\>0\- far weaker than per\-item positivity\. Coarserℓ\\ellincreases support but raises the within\-cluster bias: depth is the bias \- variance knob\.

*Why the tree, not any clustering\.*For an autoregressive SID policy the cluster mass is not a sum over𝒜\\mathcal\{A\}but the exact prefix probabilityπ⁡\(c≤ℓ∣x\)=∏t≤ℓπ⁡\(codet∣x,code<t\)\\pi\(c\_\{\\leq\\ell\}\\mid x\)=\\prod\_\{t\\leq\\ell\}\\pi\(\\mathrm\{code\}\_\{t\}\\mid x,\\mathrm\{code\}\_\{<t\}\), available inO⁡\(ℓ\)O\(\\ell\)directly from the decoder\. Forming the same mass for a*flat*clustering would require the item\-level propensitiesπ0​\(a∣x\)\\pi\_\{0\}\(a\\mid x\)and anO⁡\(\|𝒜\|\)O\(\|\\mathcal\{A\}\|\)sum \- and a generative retriever emits codes, not a normalized item distribution, so those item propensities are typically unavailable\. The SID tree is thus the cluster representation one can actually run online\. \(If several items share a full code, that leaf is simply a small cluster; nothing changes\.\) We verify this equivalence is exact, not approximate: building an autoregressive tree policy and reading masses as prefix products reproduces the item\-summed cluster marginals to machine precision at every level \(App\.[A](https://arxiv.org/html/2608.28905#A1)\) \- so the item\-sum cluster masses used in our controlled experiments are precisely what a decoder returns for free\.

## 3\.Bias and Quantization Quality

*Intuition\.*Because marginalization exactly matches the target’s cluster masses, the estimator can only err*inside*clusters: its bias is how muchπe\\pi\_\{e\}andπ0\\pi\_\{0\}disagree within a cluster, weighted by how much reward varies there \- the informal counterpart of OffCEM’s within\-cluster residual effect\([Saito et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib12)\)\. Reward\-coherent codes make that variation small\. Quantization quality is the natural proxy: items in a cluster share a reconstructionz^ℓ\\hat\{z\}\_\{\\ell\}, so if reward is smooth in the embedding, within\-cluster reward spread is controlled by the reconstruction residual\.

*Bound\.*Letz^ℓ​\(a\)\\hat\{z\}\_\{\\ell\}\(a\)reconstructz⁡\(a\)z\(a\)from its firstℓ\\ellcodes, letδℓ​\(a\)=‖z⁡\(a\)−z^ℓ​\(a\)‖\\delta\_\{\\ell\}\(a\)=\\\|z\(a\)\-\\hat\{z\}\_\{\\ell\}\(a\)\\\|, and letδℓmax=maxa⁡δℓ​\(a\)\\delta^\{\\max\}\_\{\\ell\}=\\max\_\{a\}\\delta\_\{\\ell\}\(a\)be the worst\-case residual;εℓ=𝔼a∼μ​δℓ​\(a\)2\\varepsilon\_\{\\ell\}=\\mathbb\{E\}\_\{a\\sim\\mu\}\\,\\delta\_\{\\ell\}\(a\)^\{2\}is the mean\-squared reconstruction error a quantizer is trained to minimize\. WriteTVℓin\(x\)=∑cπe\(c∣x\)TV\(π~e\(⋅∣c,x\),π~0\(⋅∣c,x\)\)\\mathrm\{TV\}^\{\\mathrm\{in\}\}\_\{\\ell\}\(x\)=\\sum\_\{c\}\\pi\_\{e\}\(c\\mid x\)\\,\\mathrm\{TV\}\\big\(\\tilde\{\\pi\}\_\{e\}\(\\cdot\\mid c,x\),\\tilde\{\\pi\}\_\{0\}\(\\cdot\\mid c,x\)\\big\)for theπe\\pi\_\{e\}\-weighted*within\-cluster*disagreement, whereπ~\(⋅∣c,x\)\\tilde\{\\pi\}\(\\cdot\\mid c,x\)is the conditional a policy induces inside clusterAcA\_\{c\}\. Assume\(A0\)cluster positivity, known propensities and𝔼\[r∣x,a\]=q\(x,a\)\\mathbb\{E\}\[r\\mid x,a\]=q\(x,a\);\(A1\)*no direct effect*,q​\(x,a\)=q~​\(x,z​\(a\)\)q\(x,a\)=\\tilde\{q\}\(x,z\(a\)\)\- MIPS’s assumption\([Saito and Joachims, 2022](https://arxiv.org/html/2608.28905#bib.bib11)\), imposed here at the*full*embedding so coarsening bias is bounded rather than zero;\(A2\)q~​\(x,⋅\)\\tilde\{q\}\(x,\\cdot\)isLqL\_\{q\}\-Lipschitz on a set containing every item embedding and cluster reconstruction\.

###### Proposition 3\.1\.

LetBias⁡\(V^ℓ\)=𝔼⁡\[V^ℓ\]−V⁡\(πe\)\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)=\\mathbb\{E\}\[\\hat\{V\}\_\{\\ell\}\]\-V\(\\pi\_\{e\}\)be the error of the level\-ℓ\\ellcluster marginal\-IPS estimator \(MIPShier\\mathrm\{MIPS\}\_\{\\mathrm\{hier\}\}\) against the true target value\. Under \(A0\) \- \(A2\),

\|Bias⁡\(V^ℓ\)\|\\displaystyle\\big\|\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)\\big\|≤2​Lq​δℓmax​𝔼x​\[TVℓin​\(x\)\]≤4​Lq​δℓmax​𝔼x​\[TV⁡\(πe,π0\)\],\\displaystyle\\;\\leq\\;2L\_\{q\}\\,\\delta^\{\\max\}\_\{\\ell\}\\;\\mathbb\{E\}\_\{x\}\\big\[\\mathrm\{TV\}^\{\\mathrm\{in\}\}\_\{\\ell\}\(x\)\\big\]\\leq\\;4L\_\{q\}\\,\\delta^\{\\max\}\_\{\\ell\}\\;\\mathbb\{E\}\_\{x\}\\big\[\\mathrm\{TV\}\(\\pi\_\{e\},\\pi\_\{0\}\)\\big\],and the constant in the second, item\-level form cannot be improved \(proof in App\.[B](https://arxiv.org/html/2608.28905#A2)\)\. If in additionδℓmax≤κ​εℓ\\delta^\{\\max\}\_\{\\ell\}\\leq\\kappa\\sqrt\{\\varepsilon\_\{\\ell\}\}, both forms hold withδℓmax\\delta^\{\\max\}\_\{\\ell\}replaced byκ​εℓ\\kappa\\sqrt\{\\varepsilon\_\{\\ell\}\}\.

*Reading the bound\.*Two quantities control the bias, and mean reconstruction error is neither\. The*worst\-case*residual comes first: a finer codebook tightens the bound only insofar as it improves the worst\-reconstructed items, so it can lowerεℓ\\varepsilon\_\{\\ell\}while leavingδℓmax\\delta^\{\\max\}\_\{\\ell\}untouched \- and reaching theεℓ\\varepsilon\_\{\\ell\}form costs a codebook\-specificκ\\kappa, soεℓ\\varepsilon\_\{\\ell\}is not comparable across tokenizers or sources\. The second is*within\-cluster*disagreement: coarsening creates no bias where policies agree conditionally, however far apart they are across clusters\. The item\-level form discards that, hence the ordering above\.

Three limits, detailed in App\.[B](https://arxiv.org/html/2608.28905#A2): the bound covers raw cluster marginal\-IPS only; it beats the trivial\|Bias\|≤1\|\\mathrm\{Bias\}\|\\leq 1only while the quantity2​Lq​δℓmax​𝔼x​\[TVℓin\]2L\_\{q\}\\delta^\{\\max\}\_\{\\ell\}\\mathbb\{E\}\_\{x\}\[\\mathrm\{TV\}^\{\\mathrm\{in\}\}\_\{\\ell\}\]stays below11; and \(A1\) holds only approximately in our testbeds, so we use it to reason about the coarsening*mechanism*, not to predict measured bias\.*Variance*moves the other way: cluster marginal\-IPS has the usual marginalized\-weight second moment\([Saito and Joachims, 2022](https://arxiv.org/html/2608.28905#bib.bib11)\), which falls as coarsening pools support\. Closed\-form variance is unwieldy for the estimators we headline, so we track it via ESS \(an inverse\-variance proxy; RQ1’s collapse is a weight\-variance blow\-up\) and bootstrap CIs\.

## 4\.Experiments

We answer six questions\.RQ1: is per\-item OPE feasible on real reco logs?RQ2: does code\-cluster marginalization recover support, and how do known estimators compare at code granularity?RQ3:*when*does the hierarchy help, and how does the best resolution move with support?RQ4: how do code source and tokenizer affect it?RQ5: can adaptive resolution beat a fixed level?RQ6: does it generalize to a second dataset?

### 4\.1\.Datasets and protocol

Production logs\(motivation, RQ1\)\. A large e\-commerce recommender explores by score\-weighted \(Plackett \- Luce\) sampling, so served\-item propensities are recoverable from the logged normalizer\. Lacking a randomized\-exposure oracle, we use these logs for support diagnostics only \(App\.[A](https://arxiv.org/html/2608.28905#A1)\)\.

KuaiRand\(*primary*controlled testbed, RQ2 \- RQ5\)\. Its random\-exposure slice\([Gao et al\., 2022](https://arxiv.org/html/2608.28905#bib.bib6)\)shows items uniformly at random, so per\-item click rates are unbiased and the oracle is identified without exposure confounding\. We split by row: one half sets the click rates building the logging and target policies, the disjoint half is the evaluation truthqevalq\_\{\\mathrm\{eval\}\}\- both the data\-generating process \(DGP\) for rewardsr∼Bernoulli⁡\(qeval​\(a\)\)r\\sim\\mathrm\{Bernoulli\}\(q\_\{\\mathrm\{eval\}\}\(a\)\)and the oracle valueV⁡\(πe\)=∑aπe​\(a\)​qeval​\(a\)=0\.499V\(\\pi\_\{e\}\)=\\sum\_\{a\}\\pi\_\{e\}\(a\)\\,q\_\{\\mathrm\{eval\}\}\(a\)=0\.499\. No estimator sees evaluation outcomes through the policies: the protocol is*leakage\-free*\(App\.[A](https://arxiv.org/html/2608.28905#A1)\)\. It is a catalog bandit over\|𝒜\|=7339\|\\mathcal\{A\}\|\{=\}7339items with content\-metadata, semantic, collaborative and fused code trees \(§[4\.5](https://arxiv.org/html/2608.28905#S4.SS5)\)\.

Amazon Reviews\(cross\-domain replication, RQ6\)\. Musical Instruments from Amazon Reviews 2023\([Hou et al\., 2024](https://arxiv.org/html/2608.28905#bib.bib7)\):\|𝒜\|=21009\|\\mathcal\{A\}\|\{=\}21009items with≥10\{\\geq\}10reviews per held\-out half, same reward template withq⁡\(a\)=P⁡\(5\-star∣a\)q\(a\)=P\(\\text\{5\-star\}\\mid a\)as oracle and DGP\. Ratings are self\-selected rather than randomly exposed, so that oracle is confounded and Amazon*replicates*the pattern rather than identifying it\.

Estimators\.All are existing methods, run at two granularities marked by a subscript \(ESTitem\\mathrm\{EST\}\_\{\\mathrm\{item\}\}on items,ESThier\\mathrm\{EST\}\_\{\\mathrm\{hier\}\}on SID prefix clusters\): item\-level IPS and SNIPS; cluster IPS and SNIPS \(MIPShier\\mathrm\{MIPS\}\_\{\\mathrm\{hier\}\},SNIPShier\\mathrm\{SNIPS\}\_\{\\mathrm\{hier\}\}\), reweighting byπe​\(c\)/π0​\(c\)\\pi\_\{e\}\(c\)/\\pi\_\{0\}\(c\); the cluster direct methodDMhier\\mathrm\{DM\}\_\{\\mathrm\{hier\}\}; OffCEM\([Saito et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib12)\), whose original configuration pairs the cluster\-level residual weight with an*action*\-level reward model \- a standard doubly\-robust form\([Dudík et al\., 2011](https://arxiv.org/html/2608.28905#bib.bib4)\)\- reported asOffCEMhieract\\mathrm\{OffCEM\}^\{\\mathrm\{act\}\}\_\{\\mathrm\{hier\}\}, alongside a coarsened cluster\-level variantOffCEMhier\\mathrm\{OffCEM\}\_\{\\mathrm\{hier\}\}we add for symmetry; and PC\([Sachdeva et al\., 2024](https://arxiv.org/html/2608.28905#bib.bib10)\), an embedding kernel whose bandwidth is a continuous resolution knob \(so nohier\\mathrm\{hier\}variant\), at untunedhh\. App\.[A](https://arxiv.org/html/2608.28905#A1)gives formulas and the two deliberate departures from published OffCEM\.

A*seed*is one simulated log \(MMdraws fromπ0\\pi\_\{0\}plus simulated rewards\); RMSE is taken over seeds,200200for the headline comparison and support sweep,100100per fit across1010fits for the quantizer ablation,8080for Amazon\. We report mean±\\pmthe half\-width of a95%95\\%bootstrap percentile interval and call a difference*significant*when its*paired*interval excludes zero, with no multiplicity correction\.

Default tree\.Codes are residual K\-Means unless stated; RQ4 finds the tokenizer does not move these results\.

Depth selection\.Unless stated otherwisehier\\mathrm\{hier\}estimators pick the level by error against the*oracle*on a disjoint validation split of seeds, so they are*oracle\-selected diagnostics*\- an upper bound on what depth selection can buy\. Logged\-data\-only, deployable alternative \(SLOPE\([Su et al\., 2020](https://arxiv.org/html/2608.28905#bib.bib14)\), estimator variance alone\) is reported \(RQ3\)\.

### 4\.2\.Per\-item OPE is infeasible \(RQ1\)

RQ1 asks whether per\-item OPE is usable on real recommender logs\. It is not, which motivates everything that follows\. On production logs the served item’s propensity is highly concentrated \(median0\.04−0\.150\.04\-0\.15;6−16%6\-16\\%of impressions above0\.90\.9\), so the item\-level effective sample size \(ESS\) is only≈0\.003\{\\approx\}0\.003of the sample \(Table[5](https://arxiv.org/html/2608.28905#A1.T5), App\.[A](https://arxiv.org/html/2608.28905#A1)\)\. So per\-item IPS is not a credible tool, motivating for coarsening\.

### 4\.3\.Coarsening recovers support and improves accuracy \(RQ2\)

RQ2 has two parts: does marginalizing to code clusters restore the support item\-level OPE lacks, and does the estimator choice matter?

Support is restored, and converts into accuracy\.On the candidate\-logged production slice, coarsening lifts the greedy\-target ESS by∼1\.8×\{\\sim\}1\.8\\times\(Table[5](https://arxiv.org/html/2608.28905#A1.T5)\)\. On the KuaiRand oracle, where error is measurable, Table[1\(a\)](https://arxiv.org/html/2608.28905#S4.T1.st1)compares both granularities:SNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}attains0\.1550\.155RMSE against0\.0870\.087forSNIPShier\\mathrm\{SNIPS\}\_\{\\mathrm\{hier\}\}\.

At code granularity, no estimator dominates\.Hierarchical MIPS/SNIPS, cluster DM, and both OffCEM variants perform close to one another\. PC, which uses no code tree, beats the best cluster estimator by a paired−0\.009\-0\.009ath=0\.1h\{=\}0\.1, but that advantage is bandwidth\-selected \- PC degrades ath=0\.2h\{=\}0\.2\- and it needs the item\-level propensities of §[2](https://arxiv.org/html/2608.28905#S2), so a code\-only deployment cannot count on it\. What generalizes across the estimators is change in granularity\.

Table 1\.RQ2, KuaiRand, leakage\-free \(\|𝒜\|=7339\|\\mathcal\{A\}\|\{=\}7339,M=5000M\{=\}5000;hier\\mathrm\{hier\}at the oracle\-selected level\)\. \(a\) Estimator comparison\. \(b\) SID tree vs\. flatkk\-means at matched cluster counts \(metadata,OffCEMhieract\\mathrm\{OffCEM\}^\{\\mathrm\{act\}\}\_\{\\mathrm\{hier\}\}\), each row refitted under1010shared clustering seeds;Δ\\Deltais the paired tree−\{\-\}flat difference\.\(a\)
\(b\)

*Is the gain due to the tree, or to coarsening in general?*To separate the two we replace the SID tree with a flatkk\-means clustering of the*same*embedding at matched cluster counts \(Table[1\(b\)](https://arxiv.org/html/2608.28905#S4.T1.st2)\)\. Flat clustering recovers the item→\\tocluster gain in full, and at no level is either representation significantly more accurate than the other: all four paired differences contain zero\. Support recovery is therefore a property of coarsening, not of the hierarchy \- and the two representations are, on accuracy, interchangeable\.

The tree’s consistent advantage is instead feasibility\. A deployed generative retriever reads cluster masses off its decoder exactly, inO⁡\(ℓ\)O\(\\ell\), whereas a flat clustering of the same catalog needs the item\-level propensities such a system does not expose \(§2\)\. That is a separate axis from the RMSE comparison above\.

*Ranking candidates, not just estimating values\.*Gating is a ranking problem: a team must order candidates, not measure any one precisely\. We rank1616simulated variants against the oracle \(Table[2](https://arxiv.org/html/2608.28905#S4.T2)\)\.

The method applies to changes sharing a single SID tree \- a new decoder, decoding temperature, or reranking rule \- since both policies’ masses come from the same partition; a re\-tokenization would move the tree itself and is out of scope\. Within that scope we test a*cross\-cluster*family, whose variants shift preference along embedding directions and so move mass between prefixes\. HereSNIPShier\\mathrm\{SNIPS\}\_\{\\mathrm\{hier\}\}ranks better thanSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}, with the margin widening as support grows scarcer \(Kendallτ\\tau0\.130\.13vs\.0\.080\.08atM=1000M\{=\}1000\)\.

Two boundaries apply\. When variants differ only*within*clusters, coarsening averages away the very differences that distinguish them, andSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}ranks at least as well\. And depth here is pre\-specified at level 3 rather than validated, while the deployment\-regret gap falls within confidence intervals\. We therefore claim improved rank correlation on cross\-cluster changes, a narrower result than a general selection guarantee\.

Table 2\.RQ2 \(selection\)\. Ranking1616candidate variants off one log against the oracle ranking \(level 3, pre\-specified\)\. For realistic*cross\-cluster*variants,SNIPShier\\mathrm\{SNIPS\}\_\{\\mathrm\{hier\}\}ranks better thanSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}, more so under scarce support\.
### 4\.4\.When does the hierarchy help? \(RQ3\)

RQ2 established that coarsening helps on average; RQ3 asks when, and how far to coarsen\. Available support governs both\.

Fig\.[1](https://arxiv.org/html/2608.28905#S4.F1)varies the code level directly\. Under scarce support, code clustering halvesSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}RMSE and an intermediate level minimizes error \- coarse enough to pool support, fine enough to keep clusters reward\-coherent; optimum moves finer with more data\.

Figure 1\.RQ3\. KuaiRand RMSE by code level \(coarse→\\tofine\), metadata source, two support regimes \(200200seeds,95%95\\%bootstrap bands\)\. Under scarce support \(left\) an intermediate level is best; with more data \(right\) the optimum moves finer\.Table[3](https://arxiv.org/html/2608.28905#S4.T3)varies support directly\. RaisingMMfrom10001000to1000010000narrows the granularity gap: coarsening buys most where support is scarcest, which is what the bias \- variance account predicts\.

Table 3\.RQ3\. KuaiRand support sweep, leakage\-free, content\-metadata codes \(\|𝒜\|=7339\|\\mathcal\{A\}\|\{=\}7339,200200seeds; RMSE\)\.*oracle*isSNIPShier\\mathrm\{SNIPS\}\_\{\\mathrm\{hier\}\}at the oracle\-selected level \(an upper bound\);*SLOPE*selects that level from logged data alone \- applied toMIPShier\\mathrm\{MIPS\}\_\{\\mathrm\{hier\}\}, whose per\-level standard errors the rule needs \- and is therefore deployable\. SLOPE beatsSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}at everyMM, capturing6767\-81%81\\%of the available gain\. TheM=5000M\{=\}5000row is Table[1\(a\)](https://arxiv.org/html/2608.28905#S4.T1.st1)’s configuration;0\.0860\.086vs\.0\.0870\.087is the independent run’s seed draw\.The table also answers the question a deployment actually faces, since the oracle\-selected column is not available in practice\. Choosing the level with SLOPE \- from logged data alone \- still beatsSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}at everyMM, recovering6767\-81%81\\%of the gap between item\-level OPE and the oracle\-selected upper bound\. So depth selection is not a barrier, though the residual gap to the oracle is open \(RQ5\)\.

The pattern also replicates cross\-domain \(RQ6\): on Amazon Reviews it holds at every support level, with narrower margins and a confounded oracle, so we read it as directional rather than a second identification \(Table[6](https://arxiv.org/html/2608.28905#A1.T6), App\.[A](https://arxiv.org/html/2608.28905#A1)\)\. The scarce regime is the production\-relevant one, since the binding quantity is*effective*support and near\-argmax logging drives item\-ESS to a small fraction of the raw count even on1717M impressions \(RQ1\)\. KuaiRand is not calibrated to production \- domains, rewards, and policies differ, and its simulated item\-level ESS is lower \- yet it spans low\-effective\-support regime\.

### 4\.5\.Code construction and tokenizer \(RQ4\)

RQ4 varies the two choices upstream of the estimator: what to embed \(*content\-metadata*,*semantic*,*collaborative*,*fused*\) and how to quantize it \(residualkk\-means or a trained RQ\-VAE\)\. Separating them needs care, because the quantizer’s fit is itself random: varying only that draw moves best\-level RMSE by up to0\.030\.03\-0\.060\.06on the metadata tree \- more than the tokenizer contrasts we set out to measure \- so intervals over simulation seeds alone exclude the dominant variance component\. We refit every cell under1010independent quantizer seeds, paired across tokenizers \(Table[4](https://arxiv.org/html/2608.28905#S4.T4)\)\.

Under that accounting*source*is a large effect and*tokenizer*is not\. Best\-level RMSE ranges from0\.0840\.084\(metadata\) to0\.1310\.131\(collaborative\), a spread of0\.0470\.047against the0\.0080\.008separating cluster estimators in Table[1\(a\)](https://arxiv.org/html/2608.28905#S4.T1.st1), whereas the paired RQ\-VAE\-kk\-means differences are−0\.003\-0\.003,−0\.001\-0\.001and\+0\.000\+0\.000for metadata, semantic and collaborative, all containing zero; only*fused*differs, at−0\.004\-0\.004, one of four nominal tests\.

Table 4\.RQ4\. Quantizer ablation \(KuaiRand, leakage\-free,M=5000M\{=\}5000,100100simulation seeds per fit\), oracle\-selected\-levelOffCEMhieract\\mathrm\{OffCEM\}^\{\\mathrm\{act\}\}\_\{\\mathrm\{hier\}\}RMSE\. Each cell is refitted under1010independent quantizer seeds; the interval is over*those seeds*, so it covers variability in the fitted tree and not only in the evaluation\.Δ\\Deltais the paired RQ\-VAE−k\{\-\}k\-means difference \(both fitted under the same seed\)\. Source separates the sources by up to0\.0470\.047; every tokenizer difference except*fused*contains zero\.
### 4\.6\.Closing the depth\-selection gap is the open step \(RQ5\)

SLOPE already captures most of the available gain \(RQ3\), but a gap to oracle selection remains, and the reason it resists closing is the pathology that motivates the paper\. Choosing a depth trades bias against variance; variance is easy, and all four routes to the bias that we tried are blocked the same way \(App\.[A](https://arxiv.org/html/2608.28905#A1)\): a per\-item plug\-in dies on less than one logged draw per item; a cluster\-level reward\-model residual stays estimable but yields no reliable gain over a variance\-only rule; and cross\-validated OPE\([Cief et al\., 2024b](https://arxiv.org/html/2608.28905#bib.bib3)\)needs an*unbiased*\- hence item\-level \- validation estimator, which lands at0\.1330\.133against a variance\-only rule’s0\.0960\.096\(metadata,M=5000M\{=\}5000\)\.

The pattern is structural:*estimating the coarsening bias requires resolution at a granularity that near\-argmax logging has already destroyed*\- the RQ1 collapse reappearing one level up\. We read the residual gap as a consequence of the logging regime rather than an unfinished search over selectors\. Averaging across levels needs no bias estimate at all and is the mitigation we would default to, though we do not measure it head\-to\-head here\.

*One route does escape it\.*If the bias cannot be inferred from the logs it can be*bought*: a small bucket logged from the target policy supplies the unbiased reference the logs cannot\. Per candidate that is circular \- it is the deployment gating exists to avoid, and with dense rewards the bucket estimatesV⁡\(πe\)V\(\\pi\_\{e\}\)better than the estimator it calibrates \- so it pays only if the chosen depth*transfers*, which is plausible because depth is a property of the tree and the logging regime rather than of the candidate\. It does transfer: calibrating on one candidate and reusing that depth on others reproduces per\-candidate selection in6565of8080candidate×\{\\times\}seed cells,attains the best fixed level, and never loses to SLOPE while beating it significantly where SLOPE misfires\.A team can therefore calibrate the depth once \- from an A/B test it has*already*run, that being on\-policy data for the policy tested \- and reuse it offline for later candidates, amortizing the cost over all of them\. Gains are modest where SLOPE is already near the best fixed level, and a transferred depth is fixed by construction, forgoing per\-log adaptivity\.

## 5\.Discussion and Open Problems

The recipe is narrow but useful: when item support collapses, marginalize to SID prefixes, which a decoder over that tree makes cheap\. Since the RQ5 failures share one cause \- the bias signal is gone at the resolution the logging policy destroyed \- the promising directions supply a trustworthy reference instead of inferring one: pseudo\-task constructions whose ground truth holds by construction\([Udagawa et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib16);[Felicioni et al\., 2024](https://arxiv.org/html/2608.28905#bib.bib5)\), and the amortized on\-policy calibration of §[4\.6](https://arxiv.org/html/2608.28905#S4.SS6)\. The first thing one would instead try does*not*work, and is worth recording: a uniformly randomized bucket is significantly worse than a variance\-only rule at every size up to half the budget, because its label weights\|𝒜\|​πe​\(a\)\|\\mathcal\{A\}\|\\pi\_\{e\}\(a\)grow in variance with how peakedπe\\pi\_\{e\}is \(App\.[A](https://arxiv.org/html/2608.28905#A1)\)\. Exploration must be aimed at the target\. Our rewards are also dense by construction, which flatters on\-policy references; whether an affordable bucket still ranks depths under sparse rewards is the sharpest open question\. Natural extensions are a tree\-structured PC using the SID hierarchy as its kernel, recursive OffCEM, and support\-constrained off\-policy*learning*; the chief external\-validity limit is that the candidate\-logged production slice has\|𝒜\|<73\|\\mathcal\{A\}\|<73, so large\-catalog claims rest on the semi\-synthetic testbeds\.

## References

- Cief et al\.\(2024a\)Matej Cief, Jacek Golebiowski, Philipp Schmidt, Ziawasch Abedjan, and Artur Bekasov\. 2024a\.Learning Action Embeddings for Off\-Policy Evaluation\. In*European Conference on Information Retrieval*\. 108–122\.
- Cief et al\.\(2024b\)Matej Cief, Branislav Kveton, and Michal Kompan\. 2024b\.Cross\-Validated Off\-Policy Evaluation\.*arXiv:2405\.15332*\(2024\)\.
- Dudík et al\.\(2011\)Miroslav Dudík, John Langford, and Lihong Li\. 2011\.Doubly Robust Policy Evaluation and Learning\. In*ICML*\.
- Felicioni et al\.\(2024\)Nicolò Felicioni, Michael Benigni, and Maurizio Ferrari Dacrema\. 2024\.AutoOPE: Automated off\-policy estimator selection\.*arXiv preprint arXiv:2406\.18022*\(2024\)\.
- Gao et al\.\(2022\)Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He\. 2022\.KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos\. In*CIKM*\. 3953–3957\.
- Hou et al\.\(2024\)Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley\. 2024\.Bridging Language and Items for Retrieval and Recommendation\. In*arXiv:2403\.03952*\.
- Matveev et al\.\(2026\)Artem Matveev, Sergei Makeev, Aleksei Krasilnikov, Vladimir Baikalov, Sergei Liamaev, and Kirill Khrylchenko\. 2026\.Long\-Term Optimization for Large\-Scale Generative Retrieval with Off\-Policy REINFORCE\. In*Workshop on End\-End Customer Journey Optimization @ KDD*\.arXiv:2607\.02818\.
- Rajput et al\.\(2023\)Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q Tran, Jonah Samost, Maciej Kula, Ed H Chi, and Maheswaran Sathiamoorthy\. 2023\.Recommender Systems with Generative Retrieval\. In*NeurIPS*\.
- Sachdeva et al\.\(2024\)Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley\. 2024\.Off\-Policy Evaluation for Large Action Spaces via Policy Convolution\. In*The Web Conference \(WWW\)*\.
- Saito and Joachims \(2022\)Yuta Saito and Thorsten Joachims\. 2022\.Off\-Policy Evaluation for Large Action Spaces via Embeddings\. In*ICML*\.
- Saito et al\.\(2023\)Yuta Saito, Qingyang Ren, and Thorsten Joachims\. 2023\.Off\-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling\. In*ICML*\.
- Saito et al\.\(2025\)Yuta Saito, Jihan Yao, and Thorsten Joachims\. 2025\.POTEC: Off\-Policy Learning for Large Action Spaces via Two\-Stage Policy Decomposition\. In*ICLR*\.
- Su et al\.\(2020\)Yi Su, Pavithra Srinath, and Akshay Krishnamurthy\. 2020\.Adaptive Estimator Selection for Off\-Policy Evaluation\. In*International Conference on Machine Learning \(ICML\)*\.
- Taufiq et al\.\(2023\)Muhammad Faaiz Taufiq, Arnaud Doucet, Rob Cornish, and Jean\-François Ton\. 2023\.Marginal Density Ratio for Off\-Policy Evaluation in Contextual Bandits\. In*Advances in Neural Information Processing Systems*, Vol\. 36\.
- Udagawa et al\.\(2023\)Takuma Udagawa, Haruka Kiyohara, Yusuke Narita, Yuta Saito, and Kei Tateno\. 2023\.Policy\-Adaptive Estimator Selection for Off\-Policy Evaluation\. In*Proceedings of the AAAI Conference on Artificial Intelligence*, Vol\. 37\. 10025–10033\.
- Wang et al\.\(2025\)Peiyao Wang, Zhan Shi, Amina Shabbeer, and Ben London\. 2025\.Off\-Policy Evaluation of Candidate Generators in Two\-Stage Recommender Systems\. In*RecSys*\.
- Wang et al\.\(2024\)Wenjie Wang, Honghui Bao, Fuli Feng, Jizhi Zhang, Xun Wang, Xiangnan He, and Tat\-Seng Chua\. 2024\.Learnable Item Tokenization for Generative Recommendation\. In*CIKM*\.

## Appendix AData, Estimators, and Diagnostics

Production logs\.Rows come from an exploration path that draws a slate by Plackett \- Luce sampling with weightswa=scoreaαw\_\{a\}=\\mathrm\{score\}\_\{a\}^\{\\alpha\}over top\-NNcandidates; the request normalizerZZand exponentα∈\{3,4,6\}\\alpha\\in\\\{3,4,6\\\}are logged, so the rank\-1 propensity is exactlyπ0​\(a\)=scoreaα/Z\\pi\_\{0\}\(a\)=\\mathrm\{score\}\_\{a\}^\{\\alpha\}/Zwhen the scored pool is available\. Tier A has∼\{\\sim\}17M rank\-1 rows for the concentration diagnostic \(Table[5](https://arxiv.org/html/2608.28905#A1.T5), left\)\. Tier B is ann=29n\{=\}29k slice with a recorded candidate pool \(\|𝒜\|<73\|\\mathcal\{A\}\|<73\); its cluster masses are renormalized within the recorded pool \(conditional\-pool propensities, not full\-catalog\), and it shows the coarsening ESS relief \(Table[5](https://arxiv.org/html/2608.28905#A1.T5), right\)\. We encode item text withharrier\-oss\-v1\-0\.6b\(10241024\-d\) and Residual K\-Means\.

Table 5\.RQ1/RQ2 production diagnostics\. Left: served\-item propensity concentration by logging exponentα\\alpha\(item ESS/n≈0\.003n\{\\approx\}0\.003atα=4\\alpha\{=\}4\)\. Right: coarsening lifts greedy\-target ESS \(n=29n\{=\}29k\)\.KuaiRand\.KuaiRand\-Pure\([Gao et al\., 2022](https://arxiv.org/html/2608.28905#bib.bib6)\); the random\-exposure log \(≈1\.19\{\\approx\}1\.19M interactions,∼\{\\sim\}7\.6K items\) gives unbiased click rates\.*Leakage control:*split by row \- one half sets the click ratesqpolicyq\_\{\\text\{policy\}\}defining the policies, the disjoint half is the evaluation truthqevalq\_\{\\text\{eval\}\}\(DGP and oracleV=∑aπe​\(a\)​qeval​\(a\)V\{=\}\\sum\_\{a\}\\pi\_\{e\}\(a\)q\_\{\\text\{eval\}\}\(a\)\); both use beta\-binomial shrinkage; items need≥20\{\\geq\}20impressions per half \(\|𝒜\|=7339\|\\mathcal\{A\}\|\{=\}7339\)\. Code sources:*content\-metadata*\(multi\-hot top\-200 tags\+\+type/music/log\-duration; engagement features excluded as they leak reward\),*semantic*\(captions\+\+cover text, same encoder\),*collaborative*\(64\-d truncated\-SVD of the user×\\timesitem matrix from a disjoint pre\-period log\), and*fused*\(ℓ2\\ell\_\{2\}\-normalized text∥\\\|collaborative\)\. Logging/target are context\-free softmaxes overs⁡\(a\)=logit⁡\(qpolicy​\(a\)\)s\(a\)=\\mathrm\{logit\}\(q\_\{\\text\{policy\}\}\(a\)\):π0∝softmax⁡\(β0​\(s\+noise\)\)\\pi\_\{0\}\\propto\\mathrm\{softmax\}\(\\beta\_\{0\}\(s\+\\text\{noise\}\)\)\(β0=6\\beta\_\{0\}\{=\}6, noise standard deviation0\.70\.7\) andπe∝softmax⁡\(βe​s\)\\pi\_\{e\}\\propto\\mathrm\{softmax\}\(\\beta\_\{e\}s\)\(βe=10\\beta\_\{e\}\{=\}10\); sweeps varyβe,M,\|𝒜\|\\beta\_\{e\},M,\|\\mathcal\{A\}\|\. Headline tables use200200paired seeds \(fixed, deterministic, so every number regenerates exactly and paired differences share draws\); sweeps use4040\. Default codes: Residual K\-Means,44levels, codebook size88\(a graded coarse\-to\-fine ladder; aK∈\{8,16,32,64\}K\\in\\\{8,16,32,64\\\}sweep shows largerKKis non\-monotone, as a large first codebook can eliminate useful coarse pooling\)\. The RQ\-VAE tokenizer \(RQ4\) is a small MLP encoder/decoder with anLL\-level straight\-through residual vector\-quantizer, codebookskk\-means\-initialized, trained300300epochs with reconstruction\+\+commitment losses on the same item embeddings; it is fit per source, so its0\.0720\.072flat\-baseline run \(RQ2\) and its0\.0770\.077Table[4](https://arxiv.org/html/2608.28905#S4.T4)run are independent trainings\.

Estimators\.IPSitem/SNIPSitem\\mathrm\{IPS\}\_\{\\mathrm\{item\}\}/\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\};MIPShier/SNIPShier\\mathrm\{MIPS\}\_\{\\mathrm\{hier\}\}/\\mathrm\{SNIPS\}\_\{\\mathrm\{hier\}\}andDMhier\\mathrm\{DM\}\_\{\\mathrm\{hier\}\}per level; the OffCEM estimator with a55\-fold cross\-fitted reward model at cluster level \(g^​\(c\)\\hat\{g\}\(c\),OffCEMhier\\mathrm\{OffCEM\}\_\{\\mathrm\{hier\}\}\) or action level \(f^​\(a\)\\hat\{f\}\(a\)from Ridge on the embedding,OffCEMhieract\\mathrm\{OffCEM\}^\{\\mathrm\{act\}\}\_\{\\mathrm\{hier\}\}\) \- cross\-fitting keeps the out\-of\-fold residual from cancelling against a constant cluster weight; and Policy Convolution with a mass\-preserving \(column\-stochastic\) kernel\. Our reward model is fit*pointwise*\(ridge regression torr\), not with the original two\-step pairwise local\-correctness objective of\([Saito et al\., 2023](https://arxiv.org/html/2608.28905#bib.bib12)\); enforcing local correctness within clusters could tighten OffCEM’s edge at coarse levels and is left to future work\. This keeps OffCEM a fair member of the estimator panel rather than a separately tuned method, consistent with our aim of isolating the granularity effect\. Both OffCEM residuals are*self\-normalized*,∑iwc​\(ri−g^\)/∑iwc\\sum\_\{i\}w\_\{c\}\(r\_\{i\}\-\\hat\{g\}\)/\\sum\_\{i\}w\_\{c\}: with a raw residual weight, an RQ\-VAE that over\-fragments semantic/collaborative codes leaves near\-empty clusters \(π0​\(c\)∼10−21\\pi\_\{0\}\(c\)\{\\sim\}10^\{\-21\}, weight∼3\.6×104\{\\sim\}3\.6\\times 10^\{4\}\) and the action\-level residual explodes; self\-normalization keeps every level bounded\. Self\-normalization trades a finite\-sample bias for this variance control\.

Autoregressive\-policy check\.Building an autoregressive tree policy \(per\-node softmax over child subtrees\) and reading cluster masses as prefix products matches the exact per\-item cluster marginals to machine precision \(<10−15<10^\{\-15\}\) at every level \- so a generative decoder supplies the cluster weights for free\.

Uniformly randomized bucket\.Charged against the same total budget as the logs, a uniform exploration bucket is significantly worse than SLOPE at every size up to half the budget: its label weights are\|𝒜\|​πe​\(a\)\|\\mathcal\{A\}\|\\pi\_\{e\}\(a\), so the label’s standard error stays near0\.120\.12even atε=0\.50\\varepsilon\{=\}0\.50, against between\-level gaps of0\.020\.02\-0\.050\.05\. Uniform exploration is an inefficient way to evaluate a concentrated target\.

Reward\-space diagnostic\.Theπe\\pi\_\{e\}\-weighted within\-cluster standard deviation ofqqtracks the exact coarsening bias about as well as input\-spaceεℓ\\varepsilon\_\{\\ell\}across our1616source×\{\\times\}level cells \(Spearman0\.520\.52vs\.0\.470\.47, a gap not resolvable atn=16n\{=\}16\), but is at least on a comparable scale, whereasεℓ\\varepsilon\_\{\\ell\}lives in each embedding’s own units \(collaborativeεℓ≈17\\varepsilon\_\{\\ell\}\{\\approx\}17vs\. text≈0\.45\{\\approx\}0\.45, yet lower bias\)\. Both are weak, and both are*descriptive*: they use oracle rewards, so neither is a deployable selector\.

Single\-draw quantizer readings \(RQ4\)\.Individual quantizer seeds produce apparent RQ\-VAE gains as large as0\.040\.04on the collaborative tree \- the favourable tail of a distribution centred on zero\. Table[4](https://arxiv.org/html/2608.28905#S4.T4)’s intervals also show the semantic and collaborative fits are considerably more stable than the metadata one, so the0\.030\.03\-0\.060\.06figure quoted in §[4\.5](https://arxiv.org/html/2608.28905#S4.SS5)is metadata\-driven\.

Depth\-selector failures \(RQ5\)\.Four routes to the coarsening bias, all blocked\. \(i\) A per\-item plug\-in of the exact bias identity \(App\.[B](https://arxiv.org/html/2608.28905#A2), Step 1\) needs per\-item reward estimates, but atM=5000M\{=\}5000over73397339items there is less than one draw per item, so the estimate degenerates to a constant and the criterion collapses onto variance alone\. \(ii\) A cluster\-level reward\-model residual reusing OffCEM’s cross\-fitted model is the one bias signal that stays estimable, and once clustering variability is accounted for it gives no reliable improvement over a variance\-only rule\. \(iii\) Cross\-validated OPE\([Cief et al\., 2024b](https://arxiv.org/html/2608.28905#bib.bib3)\)needs an unbiased item\-level validation estimator, and reaches only0\.1330\.133against a variance\-only rule’s0\.0960\.096; substituting a self\-normalized estimator restores usability but forfeits the guarantee, and then merely matches SLOPE\.

Replication on Amazon \(RQ6\)The support\-dependent pattern replicates on Amazon \(Table[6](https://arxiv.org/html/2608.28905#A1.T6)\):OffCEMhieract\\mathrm\{OffCEM\}^\{\\mathrm\{act\}\}\_\{\\mathrm\{hier\}\}beatsSNIPSitem\\mathrm\{SNIPS\}\_\{\\mathrm\{item\}\}when support\-limited and ties with data\. Gains are smaller than on KuaiRand because the Amazon logging/target pair has milder item\-overlap collapse, and Musical Instruments is among the most support\-stressed sizeable categories in our scan \- so this is a conservative, directional replication, not a causal\-identification claim from observational ratings\.

Table 6\.RQ6\. Amazon Reviews \(Musical Instruments, metadata codes,8080seeds;OffCEMhieract\\mathrm\{OffCEM\}^\{\\mathrm\{act\}\}\_\{\\mathrm\{hier\}\}at the oracle\-selected level\)\. The support\-dependent pattern replicates\.
## Appendix BProof of Proposition[3\.1](https://arxiv.org/html/2608.28905#S3.Thmtheorem1)

*Notation\.*V^ℓ\\hat\{V\}\_\{\\ell\}is the cluster marginal\-IPS estimator of §[3](https://arxiv.org/html/2608.28905#S3)at levelℓ\\ell, whose prefixes partition𝒜\\mathcal\{A\}into clustersAcA\_\{c\}\. We fix a contextxxthroughout \(all quantities conditional onxx; take𝔼x\\mathbb\{E\}\_\{x\}at the end\) and suppress it\. For any policy writeπ⁡\(c\)=∑a∈Acπ⁡\(a\)\\pi\(c\)=\\sum\_\{a\\in A\_\{c\}\}\\pi\(a\)andπ~​\(a∣c\)=π⁡\(a\)/π⁡\(c\)\\tilde\{\\pi\}\(a\\mid c\)=\\pi\(a\)/\\pi\(c\)\. The reconstructionz^ℓ\\hat\{z\}\_\{\\ell\}is constant on each cluster with valueζc\\zeta\_\{c\}, soδℓ​\(a\)=‖z⁡\(a\)−ζc⁡\(a\)‖\\delta\_\{\\ell\}\(a\)=\\\|z\(a\)\-\\zeta\_\{c\(a\)\}\\\|\. Five steps: write the bias exactly, center it per cluster, bound the centered reward gap, collect terms into the within\-cluster form, and relax to the item\-level form\.

*Step 1 \(exact bias\)\.*By \(A0\), withrir\_\{i\}conditionally unbiased forq⁡\(ai\)q\(a\_\{i\}\),

𝔼⁡\[V^ℓ\]=∑aπ0​\(a\)​πe​\(c​\(a\)\)π0​\(c​\(a\)\)​q​\(a\)=∑aρ⁡\(a\)​q​\(a\),\\mathbb\{E\}\[\\hat\{V\}\_\{\\ell\}\]=\\sum\_\{a\}\\pi\_\{0\}\(a\)\\tfrac\{\\pi\_\{e\}\(c\(a\)\)\}\{\\pi\_\{0\}\(c\(a\)\)\}\\,q\(a\)=\\sum\_\{a\}\\rho\(a\)\\,q\(a\),whereρ⁡\(a\):=π0​\(a\)​πe​\(c⁡\(a\)\)/π0​\(c⁡\(a\)\)\\rho\(a\):=\\pi\_\{0\}\(a\)\\pi\_\{e\}\(c\(a\)\)/\\pi\_\{0\}\(c\(a\)\)\. Thenρ\\rhois a distribution andρ⁡\(Ac\)=πe​\(c\)\\rho\(A\_\{c\}\)=\\pi\_\{e\}\(c\)for everycc:*ρ\\rhoandπe\\pi\_\{e\}share cluster marginals*, andρ~\(⋅∣c\)=π~0\(⋅∣c\)\\tilde\{\\rho\}\(\\cdot\\mid c\)=\\tilde\{\\pi\}\_\{0\}\(\\cdot\\mid c\)\. SinceV⁡\(πe\)=∑aπe​\(a\)​q​\(a\)V\(\\pi\_\{e\}\)=\\sum\_\{a\}\\pi\_\{e\}\(a\)q\(a\),

\(1\)Bias⁡\(V^ℓ\)=∑a\(ρ⁡\(a\)−πe​\(a\)\)​q​\(a\)\.\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)=\\textstyle\\sum\_\{a\}\\big\(\\rho\(a\)\-\\pi\_\{e\}\(a\)\\big\)q\(a\)\.
*Step 2 \(per\-cluster centering\)\.*As∑a∈Ac\(ρ⁡\(a\)−πe​\(a\)\)=0\\sum\_\{a\\in A\_\{c\}\}\(\\rho\(a\)\-\\pi\_\{e\}\(a\)\)=0, subtract the cluster constantqc⋆=q~​\(x,ζc\)q\_\{c\}^\{\\star\}=\\tilde\{q\}\(x,\\zeta\_\{c\}\)fromqqin \([1](https://arxiv.org/html/2608.28905#A2.E1)\):

\(2\)Bias⁡\(V^ℓ\)=∑c∑a∈Ac\(ρ⁡\(a\)−πe​\(a\)\)​\(q⁡\(a\)−qc⋆\)\.\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)=\\textstyle\\sum\_\{c\}\\sum\_\{a\\in A\_\{c\}\}\\big\(\\rho\(a\)\-\\pi\_\{e\}\(a\)\\big\)\\big\(q\(a\)\-q\_\{c\}^\{\\star\}\\big\)\.
*Step 3 \(Lipschitz residual\)\.*By \(A1\) \- \(A2\),qc⋆q\_\{c\}^\{\\star\}is defined and\|q⁡\(a\)−qc⋆\|≤Lq​‖z⁡\(a\)−ζc‖=Lq​δℓ​\(a\)\|q\(a\)\-q\_\{c\}^\{\\star\}\|\\leq L\_\{q\}\\\|z\(a\)\-\\zeta\_\{c\}\\\|=L\_\{q\}\\delta\_\{\\ell\}\(a\)\.

*Step 4 \(within\-cluster form\)\.*Bounding each reward gap in \([2](https://arxiv.org/html/2608.28905#A2.E2)\) byLq​δℓ​\(a\)≤Lq​δℓmaxL\_\{q\}\\delta\_\{\\ell\}\(a\)\\leq L\_\{q\}\\delta^\{\\max\}\_\{\\ell\}and taking absolute values gives\|Bias⁡\(V^ℓ\)\|≤Lq​δℓmax​∑a\|ρ⁡\(a\)−πe​\(a\)\|\|\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)\|\\leq L\_\{q\}\\delta^\{\\max\}\_\{\\ell\}\\sum\_\{a\}\|\\rho\(a\)\-\\pi\_\{e\}\(a\)\|\. Becauseρ\\rhoandπe\\pi\_\{e\}share cluster marginals andρ~=π~0\\tilde\{\\rho\}=\\tilde\{\\pi\}\_\{0\}within each cluster \(Step 1\),

∑a\|ρ⁡\(a\)−πe​\(a\)\|=∑cπe​\(c\)​∑a∈Ac\|π~0​\(a∣c\)−π~e​\(a∣c\)\|=2​TVℓin,\\textstyle\\sum\_\{a\}\|\\rho\(a\)\-\\pi\_\{e\}\(a\)\|=\\sum\_\{c\}\\pi\_\{e\}\(c\)\\sum\_\{a\\in A\_\{c\}\}\\big\|\\tilde\{\\pi\}\_\{0\}\(a\\mid c\)\-\\tilde\{\\pi\}\_\{e\}\(a\\mid c\)\\big\|=2\\,\\mathrm\{TV\}^\{\\mathrm\{in\}\}\_\{\\ell\},so\|Bias⁡\(V^ℓ\)\|≤2​Lq​δℓmax​TVℓin\|\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)\|\\leq 2L\_\{q\}\\delta^\{\\max\}\_\{\\ell\}\\mathrm\{TV\}^\{\\mathrm\{in\}\}\_\{\\ell\}\. Only within\-cluster disagreement enters, which is why coarse levels stay low\-bias when codes are reward\-aligned\. Taking𝔼x\\mathbb\{E\}\_\{x\}gives the first claim\.□\\square

*Step 5 \(item\-level relaxation, and tightness\)\.*Marginalization contracts total variation:∑a\|ρ⁡\(a\)−π0​\(a\)\|=∑cπ0​\(c\)​\|πe​\(c\)π0​\(c\)−1\|=∑c\|πe​\(c\)−π0​\(c\)\|≤∑a\|πe​\(a\)−π0​\(a\)\|\\sum\_\{a\}\|\\rho\(a\)\-\\pi\_\{0\}\(a\)\|=\\sum\_\{c\}\\pi\_\{0\}\(c\)\\big\|\\tfrac\{\\pi\_\{e\}\(c\)\}\{\\pi\_\{0\}\(c\)\}\-1\\big\|=\\sum\_\{c\}\|\\pi\_\{e\}\(c\)\-\\pi\_\{0\}\(c\)\|\\leq\\sum\_\{a\}\|\\pi\_\{e\}\(a\)\-\\pi\_\{0\}\(a\)\|, i\.e\.TV⁡\(ρ,π0\)≤TV⁡\(πe,π0\)\\mathrm\{TV\}\(\\rho,\\pi\_\{0\}\)\\leq\\mathrm\{TV\}\(\\pi\_\{e\},\\pi\_\{0\}\)\. The triangle inequality then gives

TVℓin=TV⁡\(ρ,πe\)≤TV⁡\(ρ,π0\)\+TV⁡\(π0,πe\)≤2​TV​\(πe,π0\),\\mathrm\{TV\}^\{\\mathrm\{in\}\}\_\{\\ell\}=\\mathrm\{TV\}\(\\rho,\\pi\_\{e\}\)\\leq\\mathrm\{TV\}\(\\rho,\\pi\_\{0\}\)\+\\mathrm\{TV\}\(\\pi\_\{0\},\\pi\_\{e\}\)\\leq 2\\,\\mathrm\{TV\}\(\\pi\_\{e\},\\pi\_\{0\}\),hence\|Bias⁡\(V^ℓ\)\|≤4​Lq​δℓmax​𝔼x​\[TV⁡\(πe,π0\)\]\|\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\ell\}\)\|\\leq 4L\_\{q\}\\delta^\{\\max\}\_\{\\ell\}\\mathbb\{E\}\_\{x\}\[\\mathrm\{TV\}\(\\pi\_\{e\},\\pi\_\{0\}\)\]\. The constant cannot be improved: the ratioTV⁡\(ρ,πe\)/TV⁡\(πe,π0\)\\mathrm\{TV\}\(\\rho,\\pi\_\{e\}\)/\\mathrm\{TV\}\(\\pi\_\{e\},\\pi\_\{0\}\)approaches22when one cluster’s mass ratioπe​\(c\)/π0​\(c\)\\pi\_\{e\}\(c\)/\\pi\_\{0\}\(c\)diverges and the within\-cluster conditionals are mutually singular\.□\\square

*Remark \(εℓ\\varepsilon\_\{\\ell\}\)\.*Aκ\\kappawithδℓmax≤κ​εℓ\\delta^\{\\max\}\_\{\\ell\}\\leq\\kappa\\sqrt\{\\varepsilon\_\{\\ell\}\}exists for any fixed codebook \(takeκ=δℓmax/εℓ\\kappa=\\delta^\{\\max\}\_\{\\ell\}/\\sqrt\{\\varepsilon\_\{\\ell\}\}\), but nothing forces oneκ\\kappato serve a family of codebooks\. Comparingεℓ\\varepsilon\_\{\\ell\}across quantizers or embedding sources therefore does not compare their bias bounds, and a refinement that lowers meanεℓ\\varepsilon\_\{\\ell\}without improving the worst\-reconstructed items does not tighten the bound at all\. Replacing residualkk\-means by an RQ\-VAE changes the partition,εℓ\\varepsilon\_\{\\ell\},δℓmax\\delta^\{\\max\}\_\{\\ell\}, andκ\\kappatogether, and its training loss is not the input\-spaceεℓ\\varepsilon\_\{\\ell\}here, so tokenizer ablations do not by themselves validate the bound\.