Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

# Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Source: [https://arxiv.org/html/2607.22012](https://arxiv.org/html/2607.22012)
Yuta Natsubori Hakuhodo DY Holdings, Inc\. yuta\.natsubori@hakuhodo\.co\.jp &Masataka Ushiku Hakuhodo DY Holdings, Inc\. masataka\.ushiku@hakuhodo\.co\.jp &Yuta Saito Cornell University ys522@cornell\.edu

###### Abstract

Off\-Policy Evaluation and Learning \(OPE/L\) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data\. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such as few\-shot data, deterministic logging policies, and new actions\. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges\. Existing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited/no exploration in the logged data\. To enable OPE/L even under these unsolved challenges, we propose a new problem setup ofCross\-Domain OPE/L, where we have access not only to the logged data from thetarget domainin which the new policy will be implemented but also to logged datasets collected from other domains\. This novel formulation is widely applicable because we can often use historical data not only from the target hospital, country, device, or user segment but also from other hospitals, countries, devices, or segments\. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially enhanced OPE/L in the previously unsolved situations in our empirical evaluations\.

## 1Introduction

Many decision\-making systems \(e\.g\., recommendation, medication, budget allocation\) interact with the environment through a contextual bandit process in which a policy observes the context, takes action, and obtains rewards\. Off\-Policy Evaluation and Learning \(OPE/L\) has gained attention as a technique for estimating and learning new policies without deploying them, using only historical logged data\. We can find many real applications of these techniques since they do not require costly and risky online A/B testing and exploration to enable the data\-driven policy evaluation and learning lifecycle\(Mehrotraet al\.,[2018](https://arxiv.org/html/2607.22012#bib.bib37); Gilotteet al\.,[2018](https://arxiv.org/html/2607.22012#bib.bib35); Saitoet al\.,[2021](https://arxiv.org/html/2607.22012#bib.bib32); Kiyoharaet al\.,[2024a](https://arxiv.org/html/2607.22012#bib.bib34)\)\.

While recent advances have led to the development of a number of estimators and policy gradient methods\(Saito and Joachims,[2021](https://arxiv.org/html/2607.22012#bib.bib31); Ueharaet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib38)\), most of these are based on inverse propensity scoring \(IPS\), reward regression, or their mixture\(Dudíket al\.,[2014](https://arxiv.org/html/2607.22012#bib.bib25); Wanget al\.,[2017](https://arxiv.org/html/2607.22012#bib.bib48); Suet al\.,[2020a](https://arxiv.org/html/2607.22012#bib.bib46)\)\. These estimators heavily rely on a theoretical assumption calledcommon supportto provide a low\-bias estimate\. Due to the common support assumption, we can only evaluate and learn new policies regarding actions that have already been sufficiently explored by the logging policy\(Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29); Felicioniet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib30)\)\. Therefore, in challenging but realistic cases where the logging policy is completely deterministic or there are new actions, existing methods simply cannot evaluate and choose under\-explored and new actions at all due to the lack of their reward information in the historical data\(Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29)\)\. In addition, the use of importance weighting often causes severe variance issues, particularly when the sample size is small, such asfew\-shotdata and the action space is large\(Saito and Joachims,[2022](https://arxiv.org/html/2607.22012#bib.bib28); Ciefet al\.,[2024a](https://arxiv.org/html/2607.22012#bib.bib42); Sachdevaet al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib39)\)\.

There have been several previous efforts to enable OPE\(Felicioniet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib30)\)and OPL\(Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29)\)under the violation of common support \(also known as support deficiency\), but they cannot handle deterministic logging policies and completely new actions\. It might also be useful to use some structure in the action or reward space to relax the requirement of common support, as studied bySaito and Joachims \([2022](https://arxiv.org/html/2607.22012#bib.bib28)\); Saitoet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib20)\); Ciefet al\.\([2024a](https://arxiv.org/html/2607.22012#bib.bib42)\); Taufiqet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib40)\); Sachdevaet al\.\([2024](https://arxiv.org/html/2607.22012#bib.bib39)\); Kiyoharaet al\.\([2024b](https://arxiv.org/html/2607.22012#bib.bib43)\), but this useful structure is not always learnable\.

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/draw.png)Figure 1:Comparison ofConventional OPE/LandCross\-Domain OPE/L \(Ours\)\.To enable effective OPE/L even with deterministic logging policies and new actions, we propose a new problem formulation calledCross\-Domain OPE/Lwhere we aim to evaluate and optimize the value of new policies under the target domain, but have access to historical logged data collected previously in source domains \(as depicted in Figure[1](https://arxiv.org/html/2607.22012#S1.F1)\)\. There are many cases where we have access to such multiple\-logged datasets\. In the medical field, an example scenario is when the impact of a treatment on a patient’s prognosis is recorded in several hospitals of different sizes and with different patient demographics\. Another example is in recommender systems where new content or function may become available in certain countries or for a subset of users \(e\.g\., active members\), and these can be source domains when we aim to perform OPE/L regarding new countries and other users, such as relatively new members\. If we have access to these logged datasets collected in source domains with more data and exploration in the action space, we can leverage these data to perform OPE/L more effectively towards the target domain even when it has less logged data, a deterministic logging policy, and new actions that existing estimators cannot address\.

After formally presenting the new formulation, we propose a new estimator, calledCross\-domainOff\-PolicyEvaluation \(COPE\)and respective policy gradient method to solve OPE/L by effectively leveraging useful information from source domains to estimate and optimize the value of new policies in the target domain\. To achieve effective transfer from source domains, our methods are based on a decomposition of the expected reward function into the domain\-cluster effect and domain\-specific effect\. The domain\-cluster effect is a component in the reward function that domains in the same cluster have in common, while the domain\-specific effect represents the causal effect not modeled merely by the domain\-cluster effect\. The COPE estimator unbiasedly estimates the domain\-cluster effect by applyingmultiple importance weighting\(Owen,[2013](https://arxiv.org/html/2607.22012#bib.bib17); Agarwalet al\.,[2017](https://arxiv.org/html/2607.22012#bib.bib45)\)using data not only from the target domain but also leveraging data from the source domains that are in the same cluster as the target domain\. It also deals with the domain\-specific effect via reward regression using the logged data from the target domain, which reduces the bias of the estimator depending on the regression accuracy\. We also extend the COPE estimator as a policy gradient estimator to perform OPL using both target and source data, enabling us to perform policy learning even under completely deterministic logging and new actions involved\. Theoretical analysis shows that COPE has analyzable bias and, in particular, that COPE can be unbiased even when the target domain has a deterministic logging policy and new actions\. Empirical evaluation using a real\-world recommendation dataset demonstrates that COPE outperforms existing estimators and policy learning methods by properly leveraging data from both the target and source domains, particularly when there are few target data and many actions that are not previously explored in the target domain\.

Key Related Work\.This section differentiates our work from two closely related studies\. A more comprehensive overview of related work is available in Appendix[A](https://arxiv.org/html/2607.22012#A1)\.

First, we discuss the work byUeharaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib33)\), which studied OPE/L under covariate shift, where the context distributions differ between the logged data \(phist​\(x\)p^\{\\text\{hist\}\}\(x\)\) and the evaluation environment \(peval​\(x\)p^\{\\text\{eval\}\}\(x\)\), while the reward distributionp​\(r\|x,a\)p\(r\|x,a\)remains unchanged\. Under this setup,Ueharaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib33)\)proposes methods to estimate the value of new policies when deployed in the evaluation environment, using only the logged data collected under the historical distributionphist​\(x\)p^\{\\text\{hist\}\}\(x\)\. In contrast, we aim to estimate the same estimand, but leveraging data from both target and source domains \(where there could be multiple source domains\), with each domain having its own unique context and reward distributions\. By leveraging this new setup, our primary goal is to address challenging scenarios in the target domain, such as new actions, deterministic logging, and extremely small logged data, which clearly differ from the motivations ofUeharaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib33)\)\.

We also distinguish our contributions from those ofSaitoet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib20)\), which developed the OffCEM estimator to handle OPE in large action spaces within a single domain setup\. Although our main idea of reward function decomposition is inspired by OffCEM, its application to solve the problem of Cross\-Domain OPE/L would not be possible without our unique formulation\. Moreover, we propose an extension of our estimator to an OPL method, whereasSaitoet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib20)\)focused solely on the OPE problem\. Therefore, our work is the first to formulate the problem of cross\-domain OPE/L and leverage the respective version of reward function decomposition to solve non\-trivial challenges, such as new actions and deterministic logging, and thus offers several unique contributions from both methodological and empirical angles\.

Finally, we discuss an important distinction between our work and those that address limited overlap or deficient support issues\(Hansen,[2008](https://arxiv.org/html/2607.22012#bib.bib12); Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29); Wu and Fukumizu,[2021](https://arxiv.org/html/2607.22012#bib.bib10); Felicioniet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib30)\)\. Limited overlap refers to a situation where certain actions that can be taken under a new policy have zero probability of being observed under the logging policy\. Limited overlap is problematic because, without any data about actions in the logged data, importance weighting techniques become biased\. It is important to note that our work aims to address even more challenging scenarios compared to the general limited overlap problem, namely, completely deterministic logging and new actions\. Deterministic logging refers to a situation where the logging policy selects a specific action with probability one, i\.e\., there is no stochasticity\. This is more difficult because the typical limited overlap issue still allows the logging policy to be stochastic\. Completely new actions present an even greater challenge, as they refer to actionsaathat have zero probability of being observed in the logged data for any context\. To the best of our knowledge, no previous work specifically addresses the issues of completely deterministic logging and new actions\. We tackle these extremely challenging scenarios by newly formulating the problem of cross\-domain OPE/L\.

## 2Conventional Formulation

We begin with describing the conventional formulation of OPE in the contextual bandit setting\. Here, a decision maker repeatedly observes a contextx∈𝒳x\\in\\mathcal\{X\}drawn from an unknown distributionp​\(x\)p\(x\)\. Given a contextxx, a fixed and potentially stochastic policyπ​\(a\|x\)\\pi\(a\|x\)selects an actionaafrom a finite action space denoted by𝒜\\mathcal\{A\}\. Rewardrris then observed following an unknown distributionp​\(r\|x,a\)p\(r\|x,a\)and we useq​\(x,a\):=𝔼​\[r\|x,a\]q\(x,a\):=\\mathbb\{E\}\[r\|x,a\]to denote the expected reward given context and action\. We define the performance measure of policyπ\\piby the expected reward under its deployment as

V​\(π\):=𝔼p​\(x\)​π​\(a\|x\)​p​\(r\|x,a\)​\[r\]=𝔼p​\(x\)​π​\(a\|x\)​\[q​\(x,a\)\],V\(\\pi\):=\\mathbb\{E\}\_\{p\(x\)\\pi\(a\|x\)p\(r\|x,a\)\}\[r\]=\\mathbb\{E\}\_\{p\(x\)\\pi\(a\|x\)\}\[q\(x,a\)\],\(1\)which is often called thepolicy value\. The logged bandit data we can use to perform OPE/L can be denoted by𝒟:=\{\(xi,ai,ri\)\}i=1n\\mathcal\{D\}:=\\\{\(x\_\{i\},a\_\{i\},r\_\{i\}\)\\\}^\{n\}\_\{i=1\}, which containsnnindependent observations drawn from the data distribution induced by the logging policyπ0\\pi\_\{0\}, i\.e\.,p​\(𝒟\)=∏i=1np​\(xi\)​π0​\(ai\|xi\)​p​\(ri\|xi,ai\)p\(\\mathcal\{D\}\)=\\prod\_\{i=1\}^\{n\}p\(x\_\{i\}\)\\pi\_\{0\}\(a\_\{i\}\|x\_\{i\}\)p\(r\_\{i\}\|x\_\{i\},a\_\{i\}\)\.

The aim of OPE is to develop an estimatorV^\\hat\{V\}that can accurately estimate the policy value of a new policyπ\\piusing only𝒟\\mathcal\{D\}\. We measure the accuracy ofV^\\hat\{V\}by the mean squared error \(MSE\) defined as

MSE​\(V^​\(π;𝒟\)\)\\displaystyle\\mathrm\{MSE\}\(\\hat\{V\}\(\\pi;\\mathcal\{D\}\)\):=𝔼p​\(𝒟\)​\[\(V​\(π\)−V^​\(π;𝒟\)\)2\]=Bias​\(V^​\(π;𝒟\)\)2\+𝕍𝒟​\[\(V^​\(π;𝒟\)\)\],\\displaystyle:=\\mathbb\{E\}\_\{p\(\\mathcal\{D\}\)\}\[\(V\(\\pi\)\-\\hat\{V\}\(\\pi;\\mathcal\{D\}\)\)^\{2\}\]=\\mathrm\{Bias\}\(\\hat\{V\}\(\\pi;\\mathcal\{D\}\)\)^\{2\}\+\\mathbb\{V\}\_\{\\mathcal\{D\}\}\[\(\\hat\{V\}\(\\pi;\\mathcal\{D\}\)\)\],where𝔼p​\(𝒟\)​\[⋅\]\\mathbb\{E\}\_\{p\(\\mathcal\{D\}\)\}\[\\cdot\]takes the expectation over the distribution of𝒟\\mathcal\{D\}\.

Limitations of Existing Estimators\.As an existing method for OPE, we first describe IPS\(Horvitz and Thompson,[1952](https://arxiv.org/html/2607.22012#bib.bib27)\), which estimates the policy value by re\-weighting the rewards as

V^IPS​\(π;𝒟\):=1n​∑i=1nπ​\(ai\|xi\)π0​\(ai\|xi\)​ri=1n​∑i=1nw​\(xi,ai\)​ri,\\displaystyle\\hat\{V\}\_\{\\rm\{IPS\}\}\(\\pi;\\mathcal\{D\}\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\\pi\(a\_\{i\}\|x\_\{i\}\)\}\{\\pi\_\{0\}\(a\_\{i\}\|x\_\{i\}\)\}r\_\{i\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}w\(x\_\{i\},a\_\{i\}\)r\_\{i\},\(2\)wherew​\(x,a\):=π​\(a\|x\)/π0​\(a\|x\)w\(x,a\):=\\pi\(a\|x\)/\\pi\_\{0\}\(a\|x\)is called theimportance weight\. It is widely known that IPS is unbiased, i\.e\.,𝔼p​\(𝒟\)​\[V^IPS​\(π;𝒟\)\]=V​\(π\)\\mathbb\{E\}\_\{p\(\\mathcal\{D\}\)\}\[\\hat\{V\}\_\{\\rm\{IPS\}\}\(\\pi;\\mathcal\{D\}\)\]=V\(\\pi\), under the common support condition\.

###### Condition 2\.1\(Common Support\)\.

The logging policyπ0\\pi\_\{0\}is said to have common support for policyπ\\piifπ​\(a\|x\)\>0⟹π0​\(a\|x\)\>0\\pi\(a\|x\)\>0\\implies\\pi\_\{0\}\(a\|x\)\>0for alla∈𝒜a\\in\\mathcal\{A\}andx∈𝒳x\\in\\mathcal\{X\}\.

This assumption for unbiasedness of IPS is never satisfied in situations when the logging policyπ0\\pi\_\{0\}is deterministic or there are new actions that were not available when the logged data was collected\. Under the violation of Condition[2\.1](https://arxiv.org/html/2607.22012#S2.Thmassum1), IPS produces the following bias\(Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29)\)

\|Bias​\(V^IPS​\(π;𝒟\)\)\|=𝔼p​\(x\)​\[∑a∈𝒰0​\(x,π0\)π​\(a\|x\)​q​\(x,a\)\],\\displaystyle\|\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\rm\{IPS\}\}\(\\pi;\\mathcal\{D\}\)\)\|=\\mathbb\{E\}\_\{p\(x\)\}\[\\sum\_\{a\\in\\mathcal\{U\}\_\{0\}\(x,\\pi\_\{0\}\)\}\\pi\(a\|x\)q\(x,a\)\],\(3\)where𝒰0​\(x,π0\):=\{a∈𝒜∣π0​\(a\|x\)=0\}\\mathcal\{U\}\_\{0\}\(x,\\pi\_\{0\}\):=\\\{a\\in\\mathcal\{A\}\\mid\\pi\_\{0\}\(a\|x\)=0\\\}is the set ofdeficientactions for contextxxunderπ0\\pi\_\{0\}\. This bias is due to the fact that IPS cannot evaluate the actions that are not explored by the logging policy\. To deal with this bias unavoidable for IPS,Sachdevaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib29)\)suggests using a reward regression modelq^​\(x,a\)≈q​\(x,a\)\\hat\{q\}\(x,a\)\\approx q\(x,a\)in the form of Doubly Robust \(DR\):

V^DR​\(π;𝒟,q^\):=1n​∑i=1n\{w​\(xi,ai\)​\(ri−q^​\(xi,ai\)\)\+𝔼π​\(a\|xi\)​\[q^​\(xi,a\)\]\}\.\\displaystyle\\hat\{V\}\_\{\\rm\{DR\}\}\(\\pi;\\mathcal\{D\},\\hat\{q\}\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\\{w\(x\_\{i\},a\_\{i\}\)\(r\_\{i\}\-\\hat\{q\}\(x\_\{i\},a\_\{i\}\)\)\+\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}\)\}\[\\hat\{q\}\(x\_\{i\},a\)\]\\\}\.\(4\)While the bias caused by support deficiency and the variance can be smaller ifq^​\(x,a\)\\hat\{q\}\(x,a\)is accurate\(Dudíket al\.,[2014](https://arxiv.org/html/2607.22012#bib.bib25); Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29)\), it is almost impossible to achieve such accuracy when only a small \(like few\-shot\) dataset is available and there are new actions in the target domain\.

## 3Cross\-Domain Off\-Policy Evaluation and Learning

To achieve yet effective OPE/L even with no exploration and few\-shot data in the target environment, this section proposes a new OPE problem that aims to estimate the value of a new policy, but we can have access not only to the historical data collected from the domain in which the new policy will be implemented \(targetdomain\), but also the datasets from other domains \(sourcedomains\) that have varying data generation processes \(DGPs\)\. We also develop an estimator to effectively leverage logged data from both the target and source domains and extend it to a policy gradient method\.

Here, we introduceKKdifferent domains\{k\}k=1K\\\{k\\\}\_\{k=1\}^\{K\}where the logged dataset is collected under potentially different logging policies in each domain\. One out ofKKdomains is the target domainTT, where we are interested in deploying a new policy, and the rest are source domainsSS\. The DGP within each domainkkcan be formulated as a respective contextual bandit process, i\.e\., we first observe the contextxk∈𝒳x^\{k\}\\in\\mathcal\{X\}from an unknownpk​\(x\)p^\{k\}\(x\)\. Given contextxkx^\{k\}, logging policyπ0k​\(a\|x\)\\pi\_\{0\}^\{k\}\(a\|x\)chooses actionaka^\{k\}, and then the rewardrkr^\{k\}is observed from an unknownpk​\(r\|x,a\)p^\{k\}\(r\|x,a\)\. Following this process, we observe logged bandit data denoted by𝒟k:=\{\(xik,aik,rik\)\}i=1nk∼p​\(𝒟k\)=∏i=1nkpk​\(xi\)​π0k​\(ai\|xi\)​pk​\(ri\|xi,ai\)\\mathcal\{D\}^\{k\}:=\\\{\(x\_\{i\}^\{k\},a\_\{i\}^\{k\},r\_\{i\}^\{k\}\)\\\}\_\{i=1\}^\{n^\{k\}\}\\sim p\(\\mathcal\{D\}^\{k\}\)=\\prod\_\{i=1\}^\{n\_\{k\}\}p^\{k\}\(x\_\{i\}\)\\pi\_\{0\}^\{k\}\(a\_\{i\}\|x\_\{i\}\)p^\{k\}\(r\_\{i\}\|x\_\{i\},a\_\{i\}\)for each domain, wherenkn^\{k\}is the sample size of domainkk\. Note that we sometimes use𝒟:=⋃k=1K𝒟k\\mathcal\{D\}:=\\bigcup\_\{k=1\}^\{K\}\\mathcal\{D\}^\{k\}to denote available logged data across all domains\.

We are now interested in evaluating a new policyπ​\(a\|x\)\\pi\(a\|x\)in the target domain whose policy value is defined using the target distributions,\(pT​\(x\),pT​\(r\|x,a\)\)\(p^\{T\}\(x\),p^\{T\}\(r\|x,a\)\), as follows\.

VT​\(π\):=𝔼pT​\(x\)​π​\(a\|x\)​pT​\(r\|x,a\)​\[r\]=𝔼pT​\(x\)​π​\(a\|x\)​\[qT​\(x,a\)\],V^\{T\}\(\\pi\):=\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)p^\{T\}\(r\|x,a\)\}\[r\]=\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\[q^\{T\}\(x,a\)\],\(5\)whereqT​\(x,a\):=𝔼pT​\(r\|x,a\)​\[r\|x,a\]q^\{T\}\(x,a\):=\\mathbb\{E\}\_\{p^\{T\}\(r\|x,a\)\}\[r\|x,a\]denotes the expected reward in the target domain given contextxxand actionaa\. Note here that we can regard the conventional formulation described in Section[2](https://arxiv.org/html/2607.22012#S2)as a special case where there exists only the target domain\. Thus, IPS and DR can readily be defined in our formulation by using only the target domain data𝒟T\\mathcal\{D\}^\{T\}asV^IPS​\(π;𝒟T\)\\hat\{V\}\_\{\\rm\{IPS\}\}\(\\pi;\\mathcal\{D\}^\{T\}\)andV^DR​\(π;𝒟T,q^T\)\\hat\{V\}\_\{\\rm\{DR\}\}\(\\pi;\\mathcal\{D\}^\{T\},\\hat\{q\}^\{T\}\)\.

In contrast, a possible approach to perform the existing estimators with source domain data is to naively integrate the datasets from all domains when performing IPS or DR as below\.

V^IPS−ALL​\(π;𝒟\)\\displaystyle\\hat\{V\}\_\{\\rm\{IPS\-ALL\}\}\(\\pi;\\mathcal\{D\}\):=1N​∑k=1K∑i=1nkwk​\(xik,aik\)​rik,\\displaystyle:=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}w^\{k\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)r\_\{i\}^\{k\},\(6\)V^DR−ALL​\(π;𝒟\)\\displaystyle\\displaystyle\\hat\{V\}\_\{\\rm\{DR\-ALL\}\}\(\\pi;\\mathcal\{D\}\):=1N​∑k=1K∑i=1nk\{wk​\(xik,aik\)​\(rik−q^​\(xik,aik\)\)\+𝔼π​\(a\|xik\)​\[q^​\(xik,a\)\]\},\\displaystyle:=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}\\Big\\\{w^\{k\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\(r\_\{i\}^\{k\}\-\\hat\{q\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\)\+\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}^\{k\}\)\}\[\\hat\{q\}\(x\_\{i\}^\{k\},a\)\]\\Big\\\},\(7\)wherewk​\(x,a\):=π​\(a\|x\)/π0k​\(a\|x\)w^\{k\}\(x,a\):=\\pi\(a\|x\)/\\pi\_\{0\}^\{k\}\(a\|x\)andN:=∑k=1KnkN:=\\sum\_\{k=1\}^\{K\}n^\{k\}\. IPS\-ALL and DR\-ALL use information explored in source domains to estimate the policy value in the target domain with smaller variance by using more data\. However, they completely ignore the differences in DGPs across different domains, possibly introducing substantial bias as we will demonstrate in our experiments\.

### 3\.1The COPE Estimator

This section proposes a new estimator called COPE, which effectively integrates logged bandit data from target and source domains to estimate the value of new policies in the target domain even with less data and no exploration within the target domain\. The key to deriving our estimator is how to pool information from source domains to deal with the aforementioned challenging scenarios while not introducing much bias by taking into account the differences in DGPs\. We achieve this by the following decomposition of the expected reward function\.

qk​\(x,a\)=g​\(x,a,ϕ​\(k\)\)⏟domain\-cluster effect\+h​\(x,a,k\)⏟domain\-specific effect,\\displaystyle q^\{k\}\(x,a\)=\\underbrace\{g\(x,a,\\phi\(k\)\)\}\_\{\\textit\{domain\-cluster effect\}\}\+\\underbrace\{h\(x,a,k\)\}\_\{\\textit\{domain\-specific effect\}\},\(8\)whereϕ​\(k\)\\phi\(k\)is a function to cluster similar domains\. We can, for example, obtain a clustering function by heuristically performing an off\-the\-shelf clustering algorithm based on the empirical average of the rewards of each domain \(r¯k:=∑i=1nkri/nk\\bar\{r\}^\{k\}:=\\sum\_\{i=1\}^\{n\_\{k\}\}r\_\{i\}/n\_\{k\}\) as the domain embedding, which generally works well in our experiments\. The domain\-cluster effectg​\(x,a,ϕ​\(k\)\)g\(x,a,\\phi\(k\)\)in Eq\. \([8](https://arxiv.org/html/2607.22012#S3.E8)\) is a factor of the expected reward function that domains in the same cluster have in common, while the domain\-specific effecth​\(x,a,k\)h\(x,a,k\)is the effect not solely modeled by the domain\-cluster effect and thus is dependent on each individual domainkk\. For example, in a medical problem, the cluster effect could capture the shared effect of medical treatment within similar hospitals, while the domain\-specific effect models how effective an action is for a particular hospital compared to the average in the cluster\. Based on this decomposition, we design a new estimator by applying different estimation strategies between the domain\-cluster \(gg\) and specific \(hh\) effects\. Specifically, our estimator unbiasedly estimates the domain\-cluster effect by applying themultiple importance weightingtechnique\(Owen,[2013](https://arxiv.org/html/2607.22012#bib.bib17); Agarwalet al\.,[2017](https://arxiv.org/html/2607.22012#bib.bib45)\)and the domain\-specific effect by reward regression as

V^COPE​\(π;𝒟ϕ​\(T\)\)\\displaystyle\\hat\{V\}\_\{\\rm\{COPE\}\}\(\\pi;\\mathcal\{D\}^\{\\phi\(T\)\}\):=1nϕ​\(T\)​∑k∈ϕ​\(T\)∑i=1nkπ​\(aik\|xik\)pϕ​\(T\)​\(aik\|xik\)​\(rik−q^T​\(xik,aik\)\)\\displaystyle:=\\frac\{1\}\{n^\{\\phi\(T\)\}\}\\sum\_\{k\\in\\phi\(T\)\}\\sum\_\{i=1\}^\{n^\{k\}\}\\frac\{\\pi\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\{\{\\color\[rgb\]\{0,0\.4,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.4,0\}p^\{\\phi\(T\)\}\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\}\(r\_\{i\}^\{k\}\-\\hat\{q\}^\{T\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\)\+1nT​∑i=1nT∑aT∈𝒜π​\(aT\|xiT\)​q^T​\(xiT,aT\),\\displaystyle\\hskip 85\.35826pt\+\\frac\{1\}\{n^\{T\}\}\\sum\_\{i=1\}^\{n^\{T\}\}\\sum\_\{a^\{T\}\\in\\mathcal\{A\}\}\\pi\(a^\{T\}\|x\_\{i\}^\{T\}\)\\hat\{q\}^\{T\}\(x\_\{i\}^\{T\},a^\{T\}\),\(9\)where𝒟ϕ​\(T\)\\mathcal\{D\}^\{\\phi\(T\)\}represents the logged data of domains belonging to the same cluster as the target domain \(i\.e\.,ϕ​\(T\)\\phi\(T\), which we call thetarget cluster\), andnϕ​\(T\):=∑k∈ϕ​\(T\)nkn^\{\\phi\(T\)\}:=\\sum\_\{k\\in\\phi\(T\)\}n^\{k\}is the total sample size of logged data within the target clusterϕ​\(T\)\\phi\(T\)\. It should also be noted that we use the joint distribution of context and action under the logging policy averaged within the target clusterϕ​\(T\)\\phi\(T\)in the denominator of the first term of the COPE estimator, which is formally defined as

pϕ​\(T\)​\(a\|x\):=1nϕ​\(T\)​∑k∈ϕ​\(T\)nk​pk​\(x\)pT​\(x\)​π0k​\(a\|x\),\\displaystyle\{\\color\[rgb\]\{0,0\.4,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.4,0\}p^\{\\phi\(T\)\}\(a\|x\)\}:=\\frac\{1\}\{n^\{\\phi\(T\)\}\}\\sum\_\{k\\in\\phi\(T\)\}n^\{k\}\\frac\{p^\{k\}\(x\)\}\{p^\{T\}\(x\)\}\\pi\_\{0\}^\{k\}\(a\|x\),\(10\)which results in what is called multiple importance weighting\(Owen,[2013](https://arxiv.org/html/2607.22012#bib.bib17); Agarwalet al\.,[2017](https://arxiv.org/html/2607.22012#bib.bib45)\), i\.e\.,π​\(ak\|xk\)/pϕ​\(T\)​\(ak\|xk\)\\pi\(a^\{k\}\|x^\{k\}\)/p^\{\\phi\(T\)\}\(a^\{k\}\|x^\{k\}\)in COPE to deal with varying distributions in the target cluster\. In our experiments, we estimate the weight based only on observable logged data using techniques from density ratio estimation\(Sugiyamaet al\.,[2012](https://arxiv.org/html/2607.22012#bib.bib15); Kanamoriet al\.,[2012](https://arxiv.org/html/2607.22012#bib.bib14)\), as detailed in Appendix[C](https://arxiv.org/html/2607.22012#A3)\.

Our estimator is expected to substantially outperform the naive application of existing estimators when there is only limited/no exploration and less data in the target domain\. First, COPE has a lower bias than IPS and DR by estimating the value of deficient and new actions by using data from source domains that are in the same cluster as the target domain, i\.e\.,ϕ​\(T\)\\phi\(T\)\. COPE also provides substantial variance reduction in the case of few\-shot data in the target domain by transferring information from source domains, while IPS and DR use only the target domain data\. Moreover, COPE has lower bias than IPS\-ALL and DR\-ALL because it does not naively integrate data from all domains as IPS\-ALL and DR\-ALL, but rather applies multiple importance weighting within the data in the target cluster, with adjustments for the varying context and action distributions across different domains\.

In the following, we analyze the statistical properties of COPE and discuss its intriguing interpretation as a strict generalization of existing estimators\.

###### Condition 3\.1\(Common Cluster Support\)\.

The logging policyπ0T\\pi\_\{0\}^\{T\}is said to have common cluster support for policyπ\\piifπ​\(a\|x\)\>0⟹pϕ​\(T\)​\(a\|x\)\>0\\pi\(a\|x\)\>0\\implies p^\{\\phi\(T\)\}\(a\|x\)\>0for alla∈𝒜a\\in\\mathcal\{A\}andx∈𝒳x\\in\\mathcal\{X\}\.

Note that, when Condition[2\.1](https://arxiv.org/html/2607.22012#S2.Thmassum1)is true, Condition[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1)is always true, indicating that Condition[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1)is more relaxed\. In particular, when the logging policy is deterministic or there are new actions in the target domain, Condition[2\.1](https://arxiv.org/html/2607.22012#S2.Thmassum1)is never satisfied, while Condition[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1)can still be satisfied when there is some exploration in the target clusterϕ​\(T\)\\phi\(T\)\(not necessarily in the target domain\)\.

Based on this milder support condition, we provide the bias of COPE as below\.

###### Theorem 3\.1\(Bias of COPE\)\.

Under Condition[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1), COPE has the following bias\.

𝔼pT​\(x\)​π​\(a\|x\)​\[\{\(∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)pT​\(x\)​π0k​\(a\|x\)pϕ​\(T\)​\(a\|x\)⏟:=η​\(k\)​Δq,q^k​\(x,a\)\)−Δq,q^T​\(x,a\)\}\],\\displaystyle\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\\bigg\[\\bigg\\\{\\bigg\(\\sum\_\{k\\in\\phi\(T\)\}\\underbrace\{\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\frac\{p^\{k\}\(x\)\}\{p^\{T\}\(x\)\}\\frac\{\\pi\_\{0\}^\{k\}\(a\|x\)\}\{p^\{\\phi\(T\)\}\(a\|x\)\}\}\_\{:=\\eta\(k\)\}\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\\bigg\)\-\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\bigg\\\}\\bigg\],\(11\)whereΔq,q^k​\(x,a\):=qk​\(x,a\)−q^k​\(x,a\)\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\):=q^\{k\}\(x,a\)\-\\hat\{q\}^\{k\}\(x,a\)is an estimation error of the regression modelq^\\hat\{q\}given contextxxand actionaafor domainkk\. Note that∑k∈ϕ​\(T\)η​\(k\)=1\\sum\_\{k\\in\\phi\(T\)\}\\eta\(k\)=1\. See Appendix[B\.2](https://arxiv.org/html/2607.22012#A2.SS2)for the proof\.

Theorem[3\.1](https://arxiv.org/html/2607.22012#S3.Thmthm1)suggests that the bias is characterized by the difference between the weighted average of the regression error in the target cluster \(∑k∈ϕ​\(T\)η​\(k\)​Δq,q^k​\(x,a\)\\sum\_\{k\\in\\phi\(T\)\}\\eta\(k\)\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\) and the error in the target domain \(Δq,q^T​\(x,a\)\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\)\. Therefore, ifΔq,q^k​\(x,a\)≈Δq,q^T​\(x,a\)⟹qk​\(x,a\)−qT​\(x,a\)≈q^k​\(x,a\)−q^T​\(x,a\)\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\\approx\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\implies q^\{k\}\(x,a\)\-q^\{T\}\(x,a\)\\approx\\hat\{q\}^\{k\}\(x,a\)\-\\hat\{q\}^\{T\}\(x,a\)holds for∀k∈ϕ​\(T\)\\forall k\\in\\phi\(T\)\(i\.e\.,q^k​\(x,a\)\\hat\{q\}^\{k\}\(x,a\)accurately preserves the relative reward differences of actions,qk​\(x,a\)−qT​\(x,a\)q^\{k\}\(x,a\)\-q^\{T\}\(x,a\), withinϕ​\(T\)\\phi\(T\)\), the bias of COPE becomes small\. This is because COPE already unbiasedly estimates the domain\-cluster effect via its multiple importance weighting, and thus it is sufficient for the regression model to estimate only relative reward differences of actions between domains inϕ​\(T\)\\phi\(T\)and the target domainTTto make the global estimate low\-bias\. More formally, Theorem[3\.1](https://arxiv.org/html/2607.22012#S3.Thmthm1)implies the unbiasedness of COPE under theCPCcondition\.

###### Condition 3\.2\(Conditional Pairwise Correctness; CPC\)\.

A regression modelq^k​\(x,a\)\\hat\{q\}^\{k\}\(x,a\)and domain clustering functionϕ​\(k\)\\phi\(k\)satisfy conditional pairwise correctness if the following holds true:

qk​\(x,a\)−qT​\(x,a\)=q^k​\(x,a\)−q^T​\(x,a\),q^\{k\}\(x,a\)\-q^\{T\}\(x,a\)=\\hat\{q\}^\{k\}\(x,a\)\-\\hat\{q\}^\{T\}\(x,a\),for allx∈𝒳x\\in\\mathcal\{X\},a∈𝒜a\\in\\mathcal\{A\}, andkks\.t\.k∈ϕ​\(T\)k\\in\\phi\(T\)\.

###### Corollary 3\.1\.

Under Conditions[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1)and[3\.2](https://arxiv.org/html/2607.22012#S3.Thmassum2), COPE is unbiased, i\.e\.,𝔼𝒟ϕ​\(T\)​\[V^COPE​\(π;𝒟ϕ​\(T\)\)\]=V​\(π\)\.\\mathbb\{E\}\_\{\\mathcal\{D\}^\{\\phi\(T\)\}\}\[\\hat\{V\}\_\{\\rm\{COPE\}\}\(\\pi;\\mathcal\{D\}^\{\\phi\(T\)\}\)\]=V\(\\pi\)\.See Appendix[B\.2](https://arxiv.org/html/2607.22012#A2.SS2)for the proof\.

Condition[3\.2](https://arxiv.org/html/2607.22012#S3.Thmassum2)requires that the regression modelq^\\hat\{q\}should only correctly preserve the relative reward differenceqk​\(x,a\)−qT​\(x,a\)q^\{k\}\(x,a\)\-q^\{T\}\(x,a\)between domains inϕ​\(T\)\\phi\(T\)and the target domainTT\. Therefore, it does not necessitate an accurate estimate of the absolute value of the reward function, which is a general requirement for regression\-based estimators such as the direct method \(DM\)\. Note that CPC needs only an accurate estimation of the relative reward difference within the target cluster, and thus it does not require anything about source domains outside the particular cluster\.

It is important to clarify that we do not expect Condition 3\.2 to hold in practice\. It is simply a condition that helps to understand when COPE can be unbiased\. This is why we first present Theorem 3\.1, which characterizes the bias of COPE without assuming Condition 3\.2\. It is also important to note that in challenging cases, such as deterministic logging and the presence of new actions, existing estimators, including IPS and DR, produce much bias, but our estimator leverages source domain data and is much more robust to the bias arising due to those challenges, as we will demonstrate\.

It is also worth mentioning that the size of the target cluster plays a crucial role in deciding the bias\-variance tradeoff of COPE\. When\|ϕ​\(T\)\|\|\\phi\(T\)\|is small, the bias of the estimator becomes small\. This is because decreasing the cluster size makes CPC milder\. In the extreme case where\|ϕ​\(T\)\|=1\|\\phi\(T\)\|=1, COPE becomes unbiased irrespective of the accuracy of the regression model because CPC requires nothing in that case\. In contrast, the variance of COPE is likely to increase with a small cluster because it leads to a higher variation of its importance weights\.

Here, we also provide an intriguing interpretation of COPE as a spectrum of DR and DR\-ALL, with the number of domains in the cluster\|ϕ​\(T\)\|\|\\phi\(T\)\|being a lever of the tradeoff\. More specifically, in the extreme case with\|ϕ​\(T\)\|=1\|\\phi\(T\)\|=1, meaning that there is only the target domain in the cluster, COPE reduces to DR because the averaged joint distributionpϕ​\(T\)​\(a\|x\)p^\{\\phi\(T\)\}\(a\|x\)reduces to the logging policy of the target domain, i\.e\.,π0T​\(a\|x\)\\pi\_\{0\}^\{T\}\(a\|x\)\. On the other hand, when\|ϕ​\(T\)\|=K\|\\phi\(T\)\|=K, meaning that the clusterϕ​\(T\)\\phi\(T\)contains allKKdomains, COPE reduces to DR\-ALL\. As a strict generalization of different versions of DR, COPE never performs worse than these existing methods with a good \(if not perfect\) tuning of the cluster size\|ϕ​\(T\)\|\|\\phi\(T\)\|\. Note that the cluster size\|ϕ​\(T\)\|\|\\phi\(T\)\|is a hyperparameter of COPE, and we can tune it based only on available logged data by using recent advancements in the relevant literature\(Suet al\.,[2020b](https://arxiv.org/html/2607.22012#bib.bib47); Udagawaet al\.,[2023](https://arxiv.org/html/2607.22012#bib.bib50); Felicioniet al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib51); Ciefet al\.,[2024b](https://arxiv.org/html/2607.22012#bib.bib52)\)\.

### 3\.2Extension to Cross\-Domain Off\-Policy Learning

In addition to the OPE counterpart, we can formulate the problem of learning a new policy to optimize the expected reward in the target domain asmaxθ⁡VT​\(πθ\)\{\\color\[rgb\]\{0,0\.4,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.4,0\}\\max\_\{\\theta\}V^\{T\}\(\\pi\_\{\\theta\}\)\}whereθ∈ℝd\\theta\\in\\mathbb\{R\}^\{d\}is the policy parameter\. A typical approach to solving the policy learning problem is the policy\-based approach, which updates the policy parameter by iterative gradient ascent asθt\+1←θt\+∇θVT​\(πθ\)\\theta\_\{t\+1\}\\leftarrow\\theta\_\{t\}\+\\nabla\_\{\\theta\}V^\{T\}\(\\pi\_\{\\theta\}\)\. Since the true gradient∇θVT​\(πθ\)=𝔼pT​\(x\)​πθ​\(a\|x\)​\[qT​\(x,a\)​∇θlog⁡πθ​\(a\|x\)\]\\nabla\_\{\\theta\}V^\{T\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\_\{\\theta\}\(a\|x\)\}\[q^\{T\}\(x,a\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|x\)\]is unknown, we need to estimate it from observable data, which has been done by applying IPS or DR in the existing literature\(Suet al\.,[2019](https://arxiv.org/html/2607.22012#bib.bib49); Metelliet al\.,[2021](https://arxiv.org/html/2607.22012#bib.bib18)\)\. However, similarly to the case with OPE, under the violation of common support with deterministic logging and new actions, policy gradient estimators based on IPS and DR produce bias\. Moreover, in the presence of new actions that become newly available when learning a new policy for the target domain, these conventional methods cannot evaluate and choose such new actions at all, even though some of them might have high expected rewards\.

To solve these seemingly intractable problems in OPL, we can indeed readily extend COPE as a policy\-gradient estimator to learn a new policy that optimizes the policy value of the target domain\.

∇θV^COPE−PGT​\(πθ;𝒟ϕ​\(T\)\)\\displaystyle\\nabla\_\{\\theta\}\\hat\{V\}^\{T\}\_\{\\rm\{COPE\-PG\}\}\(\\pi\_\{\\theta\};\\mathcal\{D\}^\{\\phi\(T\)\}\):=1nϕ​\(T\)​∑k∈ϕ​\(T\)∑i=1nkπθ​\(aik\|xik\)pϕ​\(T\)​\(aik\|xik\)​\(rik−q^​\(xik,aik\)\)​∇θlog​πθ​\(aik\|xik\)\\displaystyle:=\\frac\{1\}\{n^\{\\phi\(T\)\}\}\\sum\_\{k\\in\\phi\(T\)\}\\sum\_\{i=1\}^\{n^\{k\}\}\\frac\{\\pi\_\{\\theta\}\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\{p^\{\\phi\(T\)\}\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\(r\_\{i\}^\{k\}\-\\hat\{q\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\)\\nabla\_\{\\theta\}\\mathrm\{log\}\\pi\_\{\\theta\}\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\+1nT​∑i=1nT𝔼πθ​\(aT\|xiT\)​\[q^​\(xiT,aT\)​∇θlog​πθ​\(aT\|xiT\)\]\\displaystyle\\qquad\+\\frac\{1\}\{n^\{T\}\}\\sum\_\{i=1\}^\{n^\{T\}\}\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\(a^\{T\}\|x\_\{i\}^\{T\}\)\}\[\\hat\{q\}\(x\_\{i\}^\{T\},a^\{T\}\)\\nabla\_\{\\theta\}\\mathrm\{log\}\\pi\_\{\\theta\}\(a^\{T\}\|x\_\{i\}^\{T\}\)\]\(12\)As already discussed, our policy\-gradient estimator in Eq\. \([12](https://arxiv.org/html/2607.22012#S3.E12)\) particularly uses data from source domains that belong to the target cluster through multiple importance weighting, enabling to even learn the value of actions with little or no previous exploration\.

## 4Empirical Analysis

Dataset\.This section empirically demonstrates the advantages of COPE and COPE\-PG against existing ideas on a real\-world public dataset called KuaiRec\(Gaoet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib21)\)collected from a recommendation system on a video\-sharing app\. The small matrix of the dataset consists of 1,411 users \(denoted asu∈𝒰u\\in\\mathcal\{U\}\), 3,327 items, and 4,676,570 interactions, with a density of 99\.6%, which enables OPE/L experiments without synthetic reward functions\. We randomly select 30 actions that have at least one interaction with all users for our experiments\. We use the user features and watch ratio recorded in the original data as the contextxux\_\{u\}and expected rewardq​\(xu,a\)q\(x\_\{u\},a\), respectively\.

Setup\.To simulate our problem of cross\-domain OPE/L, we sample users from a domain\-specific distributionpk​\(u\)p^\{k\}\(u\)for each domainkk, which is defined aspk​\(u\):=exp​\(αk⋅q¯​\(u\)\)∑u′∈𝒰exp​\(αk⋅q¯​\(u′\)\),p^\{k\}\(u\):=\\frac\{\\mathrm\{exp\}\(\\alpha^\{k\}\\cdot\\bar\{q\}\(u\)\)\}\{\\sum\_\{u^\{\\prime\}\\in\\mathcal\{U\}\}\\mathrm\{exp\}\(\\alpha^\{k\}\\cdot\\bar\{q\}\(u^\{\\prime\}\)\)\},whereq¯​\(u\):=\(1/\|𝒜\|\)​∑a∈𝒜q​\(xu,a\)\\bar\{q\}\(u\):=\(1/\|\\mathcal\{A\}\|\)\\sum\_\{a\\in\\mathcal\{A\}\}q\(x\_\{u\},a\)andαk\\alpha^\{k\}is a parameter that controls the user distribution in each domain\. A domain with a large positiveαk\\alpha^\{k\}has a higher density of users with a largerq​\(u\)q\(u\)\. We use different values ofαk\\alpha^\{k\}for different domains to vary the context distributions across domains\.

We sample an actionaka^\{k\}based on the domain\-specific logging policyπ0k\\pi\_\{0\}^\{k\}, which is defined as below\.

π0k​\(a\|xu\):=exp​\(βk⋅\(q​\(xu,a\)\+ηu,a\)\)∑a′∈𝒜exp​\(βk⋅\(q​\(xu,a′\)\+ηu,a′\)\),\\displaystyle\\pi\_\{0\}^\{k\}\(a\\,\|\\,x\_\{u\}\):=\\frac\{\\mathrm\{exp\}\(\\beta^\{k\}\\cdot\(q\(x\_\{u\},a\)\+\\eta\_\{u,a\}\)\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\mathrm\{exp\}\(\\beta^\{k\}\\cdot\(q\(x\_\{u\},a^\{\\prime\}\)\+\\eta\_\{u,a^\{\\prime\}\}\)\)\},\(13\)whereβk\\beta^\{k\}andηu,a\\eta\_\{u,a\}are sampled from a uniform distribution within range\[−2\.0,2\.0\]\[\-2\.0,2\.0\]and\[−3\.0,3\.0\]\[\-3\.0,3\.0\], respectively\. In contrast, the new policyπ\\piis defined via the epsilon\-greedy rule as

π​\(a\|x\)=\(1−ϵ\)⋅𝕀​\{a=argmaxa′∈A​q​\(xu,a′\)\}\+ϵ/\|𝒜\|,\\displaystyle\\pi\(a\\,\|\\,x\)=\(1\-\\epsilon\)\\cdot\\mathbb\{I\}\\\{a=\\mathrm\{argmax\}\_\{a^\{\\prime\}\\in A\}\\,q\(x\_\{u\},a^\{\\prime\}\)\\\}\+\\epsilon/\|\\mathcal\{A\}\|,\(14\)whereϵ∈\[0,1\]\\epsilon\\in\[0,1\]controls the quality ofπ\\piand we setϵ=0\.2\\epsilon=0\.2as default\. We sample the rewardrkr^\{k\}from a normal distribution with meanq​\(xu,a\)q\(x\_\{u\},a\)and standard deviationσ=1\\sigma=1\. Iterating this procedurenkn^\{k\}times in each domain generate𝒟k\\mathcal\{D\}^\{k\}withnkn^\{k\}independent copies of\(uk,xuk,ak,rk\)\(u^\{k\},x^\{k\}\_\{u\},a^\{k\},r^\{k\}\)\.

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/legend_ope.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_mse_graph_na.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_bias_graph_na.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_variance_graph_na.png)

Figure 2:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with varying ratios of new actions in the target domain\.![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_mse_graph_dp_0920.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_bias_graph_dp_0920.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_variance_graph_dp_0920.png)

Figure 3:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with varying numbers of users whose logging policy is deterministic in the target domain\.![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_mse_graph_ss.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_bias_graph_ss.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_variance_graph_ss.png)

Figure 4:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with varying \(per\-action\) logged data sizes in the target domain\.Baselines\.We compare COPE with IPS\(T\), IPS\-ALL, DR\(T\), and DR\-ALL where IPS\(T\) and DR\(T\) perform off\-policy estimation using only the target domain data𝒟T\\mathcal\{D\}^\{T\}\. We also include the Direct Method \(DM\) \(V^DM​\(π;𝒟T,q^T\)\\hat\{V\}\_\{\\rm\{DM\}\}\(\\pi;\\mathcal\{D\}^\{T\},\\hat\{q\}^\{T\}\)\) and DM\-ALL \(V^DM−ALL​\(π;𝒟,q^\)\\hat\{V\}\_\{\\rm\{DM\-ALL\}\}\(\\pi;\\mathcal\{D\},\\hat\{q\}\)\) as baselines, which are rigorously defined in the appendix\. We use Random Forest\(Breiman,[2001](https://arxiv.org/html/2607.22012#bib.bib22)\)implemented in scikit\-learn\(Pedregosaet al\.,[2011](https://arxiv.org/html/2607.22012#bib.bib23)\)along with 3\-fold cross\-fitting\(Newey and Robins,[2018](https://arxiv.org/html/2607.22012#bib.bib24)\)to obtainq^T​\(x,a\)\\hat\{q\}^\{T\}\(x,a\)for DR and DM, andq^​\(x,a\)\\hat\{q\}\(x,a\)for DR\-ALL, DM\-ALL, and COPE\. In addition, for COPE, we use\|ϕ​\(T\)\|=4\|\\phi\(T\)\|=4, where we define the target clusterϕ​\(T\)\\phi\(T\)by the set of domains for which the difference in the empirical average of the rewards,\|r¯k−r¯T\|\|\\bar\{r\}^\{k\}\-\\bar\{r\}^\{T\}\|, is small\.

Results:Cross\-Domain OPE\.The following reports and discusses the MSE, squared bias, and variance of the OPE estimators computed over 200 sets of logged data, each replicated with different seeds\. The shaded regions in the MSE plots represent the 95% confidence intervals estimated via bootstrap\. Note that we setK=10K=10for the number of domains andnk=100n^\{k\}=100for the logged data size of each domain as the default experimental parameters\.

First, we compare the estimators under varying ratios of new actions in the target domain,\|𝒰0T\|/𝒜\|∈\{0,0\.2,0\.4,0\.6,0\.8\}\|\\mathcal\{U\}\_\{0\}^\{T\}\|/\\mathcal\{A\}\|\\in\\\{0,0\.2,0\.4,0\.6,0\.8\\\}, where𝒰0T:=\{a∈𝒜\|π0T​\(a\|x\)=0,∀x\}\\mathcal\{U\}\_\{0\}^\{T\}:=\\\{a\\in\\mathcal\{A\}\\,\|\\,\\pi\_\{0\}^\{T\}\(a\|x\)=0,\\forall x\\\}in Figure[4](https://arxiv.org/html/2607.22012#S4.F4)\. The results indicate that estimators using only logged data from the target domain, particularly IPS\(T\) and DR\(T\), produce larger bias as the ratio increases, while estimators that naively use logged data from all domains, such as IPS\-ALL, DR\-ALL, and DM\-ALL, exhibit consistently high bias due to their inability to deal with differences in DGPs\. In contrast, COPE consistently performs the best without being affected by the presence of new actions and producing much bias\. Note that the MSEs of IPS and DR do not deteriorate rapidly as the number of new actions increases\. This is because, although the bias of IPS and DR increases with new actions, their variance decreases as the number of supported actions decreases \(a similar phenomenon is observed inSaito and Joachims \([2022](https://arxiv.org/html/2607.22012#bib.bib28)\)\)\.

Next, we compare the estimators under varying numbers of users whose logging policy is deterministic\. We applied a logging policy that deterministically selects a single action to a subset of the users, while we apply a stochastic logging policy in Eq\. \([13](https://arxiv.org/html/2607.22012#S4.E13)\) to the rest\. Figure[4](https://arxiv.org/html/2607.22012#S4.F4)shows that COPE is robust against an increasing number of users having a deterministic logging policy and maintains a small bias even when a deterministic policy is applied to about 70% \(≃1000/\|𝒰\|\\simeq 1000/\|\\mathcal\{U\}\|\) of all users\. On the other hand, IPS\(T\) and DR\(T\) exhibit an increasing bias as the number of users with a deterministic logging policy grows, which is consistent with Eq\. \([3](https://arxiv.org/html/2607.22012#S2.E3)\)\. This is similar to the pattern observed with the increasing ratio of new actions and demonstrates the larger bias reduction by COPE in harder scenarios with more new actions and users with deterministic logging\.

We also compare the estimators under varying per\-action data sizes \(nT/\|𝒜\|n^\{T\}/\|\\mathcal\{A\}\|\) in Figure[4](https://arxiv.org/html/2607.22012#S4.F4)\. We can see that COPE achieves the most accurate estimation, particularly with smaller logged data sizes in the target data\. Specifically, when there is only a single data point per action, i\.e\.,nT/\|𝒜\|=1n^\{T\}/\|\\mathcal\{A\}\|=1, COPE outperforms the best baseline \(DM\) by a substantial margin \(MSE​\(V^DM​\(T\)\)MSE​\(V^COPE\)=6\.79\\frac\{\\mathrm\{MSE\}\(\\hat\{V\}\_\{\\rm\{DM\(T\)\}\}\)\}\{\\mathrm\{MSE\}\(\\hat\{V\}\_\{\\rm\{COPE\}\}\)\}=6\.79\)\. Note also that the best baselines are different for differentnT/\|𝒜\|n^\{T\}/\|\\mathcal\{A\}\|, while COPE generally performs the best in any per\-action data size\. This powerful and stable behavior of COPE is due to its variance reduction by using data from multiple domains while accounting for varying DGPs to avoid producing bias\.

Results:Cross\-Domain OPL\.We now compare the estimators in terms of their effectiveness in OPL when used to estimate the policy gradient∇θVT​\(πθ\)\\nabla\_\{\\theta\}V^\{T\}\(\\pi\_\{\\theta\}\)\. In the following, we report the target policy values relative to those of the logging policy, computed over 350 sets of logged data, each replicated with different seeds\. The default configurations are the same as those in the OPE experiments\.

We first compare OPL methods under varying ratios of new actions in the target domain in Figure[6](https://arxiv.org/html/2607.22012#S5.F6)\(left\)\. The results indicate that COPE\-PG generally outperforms other methods without being affected by the presence of new actions\. Additionally, Figure[6](https://arxiv.org/html/2607.22012#S5.F6)\(center\) illustrates the policy value within new actions,𝔼pT​\(x\)​\[∑a∈𝒰0Tπθ​\(a\|x\)​q​\(x,a\)\]\\mathbb\{E\}\_\{p^\{T\}\(x\)\}\[\\sum\_\{a\\in\\mathcal\{U\}\_\{0\}^\{T\}\}\\pi\_\{\\theta\}\(a\|x\)q\(x,a\)\]\(Figure[6](https://arxiv.org/html/2607.22012#S5.F6)\(right\) reports the normalized version:𝔼pT​\(x\)​\[∑a∈𝒰0Tπθ​\(a\|x\)​q​\(x,a\)/∑a∈𝒰0Tπθ​\(a\|x\)\]\\mathbb\{E\}\_\{p^\{T\}\(x\)\}\[\\sum\_\{a\\in\\mathcal\{U\}\_\{0\}^\{T\}\}\\pi\_\{\\theta\}\(a\|x\)q\(x,a\)/\\sum\_\{a\\in\\mathcal\{U\}\_\{0\}^\{T\}\}\\pi\_\{\\theta\}\(a\|x\)\]\)\. We observe that methods using only logged data from the target domain, such as Reg\-based\(T\), IPS\-PG\(T\), and DR\-PG\(T\), are not able to gain policy values from new actions, which is reasonable\. In contrast, methods that additionally leverage logged data from the source domains, such as IPS\-ALL, DR\-ALL, and COPE\-PG, develop policies that select new actions relatively well, with COPE\-PG achieving particularly efficient selection of new actions, even though there is no exploration of such actions in the target domain\.

We then compare the OPL methods under varying numbers of users with a deterministic logging policy in Figure[6](https://arxiv.org/html/2607.22012#S5.F6)\(left\)\. We observed that the policy value of IPS\-PG\(T\) and DR\-PG\(T\) deteriorates as the degree of determinism increases\. This is due to the fact that under deterministic logging policies and severe common support violations, they become substantially biased\. On the other hand, COPE\-PG consistently maintains the best policy value, unaffected by the increasing number of users with a deterministic logging policy due to its milder support condition \(Condition[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1)\)\.

We also compare the OPL methods under varying per\-action training data sizes \(nT/\|𝒜\|n^\{T\}/\|\\mathcal\{A\}\|\) in the target domain in Figure[6](https://arxiv.org/html/2607.22012#S5.F6)\(center\)\. As we can observe from the figure, most existing methods show that the policy value becomes higher as the training data size increases, but COPE\-PG generally performs the best\. It is particularly impressive to see the stable performance of COPE\-PG, while no baseline methods generally perform well for varying training data sizes\. Specifically, IPS\-PG\(ALL\) and DR\-PG\(ALL\) are the second best, following COPE\-PG whennT/\|𝒜\|=1n^\{T\}/\|\\mathcal\{A\}\|=1, while they perform even worse than IPS\-PG\(T\) and DR\-PG\(T\) whennT/\|𝒜\|=30n^\{T\}/\|\\mathcal\{A\}\|=30\. In contrast, IPS\-PG\(T\) and DR\-PG\(T\) perform similarly to COPE\-PG whennT/\|𝒜\|=30n^\{T\}/\|\\mathcal\{A\}\|=30while they perform substantially worse whennT/\|𝒜\|=1n^\{T\}/\|\\mathcal\{A\}\|=1\. Given the stable performance of COPE\-PG, there is no particular reason to prioritize other methods over ours in anynT/\|𝒜\|n^\{T\}/\|\\mathcal\{A\}\|in terms of the OPL effectiveness\.

We finally evaluate the effectiveness of COPE\-PG with varying sizes of the target cluster\|ϕ​\(T\)\|\|\\phi\(T\)\|\(a key hyperparameter of COPE\-PG\) in Figure[6](https://arxiv.org/html/2607.22012#S5.F6)\(right\)\. Note that the baseline methods are independent of this parameter, so their results remain flat\. The figure shows that the size of the target cluster indeed affects the performance of COPE\-PG, with the best performance observed at moderate cluster sizes, such as 6 and 8\. However, it should also be noted that COPE\-PG still outperforms the baseline methods even with overly small \(\|ϕ​\(T\)\|=2\|\\phi\(T\)\|=2\) or large \(\|ϕ​\(T\)\|=10\|\\phi\(T\)\|=10\) cluster sizes, demonstrating the robustness of our method to potential failures in tuning of the parameter\.

## 5Conclusion and Future Work

This paper studied OPE/L in challenging scenarios such as with deterministic logging policies, new actions, and few\-shot logged data, which existing formulations and methods cannot address\. To solve those challenging issues, we formulated the problem of cross\-domain OPE/L and proposed a novel estimator and policy\-gradient method based on a reward function decomposition to leverage data explored in source domains without introducing much bias\. In particular, our methods are able to evaluate and optimize new policies even with deterministic logging policies or new actions in the target domain with analyzable bias\.

As future work, even though the heuristic domain clustering of using the empirical averaged rewards as domain embeddings worked satisfactorily in our experiments, it would be valuable to develop a more principled method to perform clustering of domains where we can possibly apply techniques from relevant work in OPE\(Penget al\.,[2023](https://arxiv.org/html/2607.22012#bib.bib5); Sachdevaet al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib39); Kiyoharaet al\.,[2024b](https://arxiv.org/html/2607.22012#bib.bib43)\)\.

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/legend_opl.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real_world_opl-policy_value-na_0930.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real_world_opl-new_actions_policy_value-na_cut0_0930.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real_world_opl-normalized_new_actions_policy_value-na_cut0_0930.png)

Figure 5:Comparison of the \(left\) the test policy valuesVT​\(π\)V^\{T\}\(\\pi\)\(normalized byV​\(π0\)V\(\\pi\_\{0\}\)\), \(center\) the test policy values within new actions, \(rigt\) the normalized test policy values within new actions, under varying ratios of new actions in the target domain\.![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real_world_opl-policy_value-dp_0930.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real_world_opl-policy_value-ss_0930.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real_world_opl-policy_value-cl_0930.png)

Figure 6:Comparison of the test policy valuesVT​\(π\)V^\{T\}\(\\pi\)\(normalized byV​\(π0\)V\(\\pi\_\{0\}\)\) of the OPL methods under \(left\) varying numbers of users whose logging is deterministic, \(center\) varying \(per\-action\) training data sizes in the target domain, \(right\) varying sizes of the target cluster\.
## References

- Effective evaluation using logged bandit feedback from multiple loggers\.InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 687–696\.Cited by:[§1](https://arxiv.org/html/2607.22012#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p1.13),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p1.7)\.
- L\. Breiman \(2001\)Random forests\.Machine learning45,pp\. 5–32\.Cited by:[§4](https://arxiv.org/html/2607.22012#S4.p4.8)\.
- M\. Cief, J\. Golebiowski, P\. Schmidt, Z\. Abedjan, and A\. Bekasov \(2024a\)Learning action embeddings for off\-policy evaluation\.InEuropean Conference on Information Retrieval,pp\. 108–122\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1)\.
- M\. Cief, M\. Kompan, and B\. Kveton \(2024b\)Cross\-validated off\-policy evaluation\.arXiv preprint arXiv:2405\.15332\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p3.1),[§B\.3](https://arxiv.org/html/2607.22012#A2.SS3.p6.1),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p10.9)\.
- M\. Dudík, D\. Erhan, J\. Langford, and L\. Li \(2014\)Doubly robust policy evaluation and optimization\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1),[§2](https://arxiv.org/html/2607.22012#S2.p4.6)\.
- N\. Felicioni, M\. Benigni, and M\. F\. Dacrema \(2024\)AutoOPE: automated off\-policy estimator selection\.arXiv preprint arXiv:2406\.18022\.Cited by:[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p10.9)\.
- N\. Felicioni, M\. Ferrari Dacrema, M\. Restelli, and P\. Cremonesi \(2022\)Off\-policy evaluation with deficient support using side information\.Advances in Neural Information Processing Systems35\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1),[§1](https://arxiv.org/html/2607.22012#S1.p9.1)\.
- G\. Gabbianelli, G\. Neu, and M\. Papini \(2024\)Importance\-weighted offline learning done right\.InProceedings of the 35th International Conference on Algorithmic Learning Theory,Vol\.237,pp\. 614–634\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p3.1)\.
- C\. Gao, S\. Li, W\. Lei, J\. Chen, B\. Li, P\. Jiang, X\. He, J\. Mao, and T\. Chua \(2022\)Kuairec: a fully\-observed dataset and insights for evaluating recommender systems\.InProceedings of the 31st ACM International Conference on Information and Knowledge Management,pp\. 540–550\.Cited by:[§4](https://arxiv.org/html/2607.22012#S4.p1.3)\.
- A\. Gilotte, C\. Calauzènes, T\. Nedelec, A\. Abraham, and S\. Dollé \(2018\)Offline a/b testing for recommender systems\.InProceedings of the Eleventh ACM International Conference on Web Search and Data Mining,pp\. 198–206\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p1.1)\.
- B\. B\. Hansen \(2008\)The prognostic analogue of the propensity score\.Biometrika95\(2\),pp\. 481–488\.Cited by:[§1](https://arxiv.org/html/2607.22012#S1.p9.1)\.
- D\. G\. Horvitz and D\. J\. Thompson \(1952\)A generalization of sampling without replacement from a finite universe\.Journal of the American Statistical Association47\(260\),pp\. 663–685\.Cited by:[§2](https://arxiv.org/html/2607.22012#S2.p3.3)\.
- T\. Kanamori, T\. Suzuki, and M\. Sugiyama \(2012\)Statistical analysis of kernel\-based least\-squares density\-ratio estimation\.Machine Learning86,pp\. 335–367\.Cited by:[Appendix C](https://arxiv.org/html/2607.22012#A3.p1.5),[Appendix C](https://arxiv.org/html/2607.22012#A3.p3.1),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p1.13)\.
- H\. Kiyohara, R\. Kishimoto, K\. Kawakami, K\. Kobayashi, K\. Nakata, and Y\. Saito \(2024a\)Towards assessing and benchmarking risk\-return tradeoff of off\-policy evaluation\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p1.1)\.
- H\. Kiyohara, M\. Nomura, and Y\. Saito \(2024b\)Off\-policy evaluation of slate bandit policies via optimizing abstraction\.InProceedings of the ACM on Web Conference 2024,pp\. 3150–3161\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1),[§5](https://arxiv.org/html/2607.22012#S5.p2.1)\.
- H\. Kiyohara, M\. Uehara, Y\. Narita, N\. Shimizu, Y\. Yamamoto, and Y\. Saito \(2023\)Off\-policy evaluation of ranking policies under diverse user behavior\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 1154–1163\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1)\.
- J\. Liu, Z\. Zhang, Z\. Wei, Z\. Zhuang, Y\. Kang, S\. Gai, and D\. Wang \(2024\)Beyond ood state actions: supported cross\-domain offline reinforcement learning\.InProceedings of the 38th AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 13945–13953\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p3.1)\.
- B\. London and T\. Sandler \(2019\)Bayesian counterfactual risk minimization\.InProceedings of the 36th International Conference on Machine Learning,Vol\.97,pp\. 4125–4133\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p3.1)\.
- R\. Mehrotra, J\. McInerney, H\. Bouchard, M\. Lalmas, and F\. Diaz \(2018\)Towards a fair marketplace: counterfactual evaluation of the trade\-off between relevance, fairness & satisfaction in recommendation systems\.InProceedings of the 27th ACM International Conference on Information and Knowledge Management,pp\. 2243–2251\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p1.1)\.
- A\. M\. Metelli, A\. Russo, and M\. Restelli \(2021\)Subgaussian and differentiable importance sampling for off\-policy evaluation and learning\.Advances in Neural Information Processing Systems34\.Cited by:[§3\.2](https://arxiv.org/html/2607.22012#S3.SS2.p1.4)\.
- W\. K\. Newey and J\. R\. Robins \(2018\)Cross\-fitting and fast remainder rates for semiparametric estimation\.arXiv preprint arXiv:1801\.09138\.Cited by:[§4](https://arxiv.org/html/2607.22012#S4.p4.8)\.
- A\. B\. Owen \(2013\)Monte carlo theory, methods and examples\.Stanford\.Cited by:[§1](https://arxiv.org/html/2607.22012#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p1.13),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p1.7)\.
- F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg,et al\.\(2011\)Scikit\-learn: machine learning in python\.The Journal of Machine Learning Research12,pp\. 2825–2830\.Cited by:[§4](https://arxiv.org/html/2607.22012#S4.p4.8)\.
- J\. Peng, H\. Zou, J\. Liu, S\. Li, Y\. Jiang, J\. Pei, and P\. Cui \(2023\)Offline policy evaluation in large action spaces via outcome\-oriented action grouping\.InProceedings of the ACM Web Conference 2023,pp\. 1220–1230\.Cited by:[§5](https://arxiv.org/html/2607.22012#S5.p2.1)\.
- N\. Sachdeva, Y\. Su, and T\. Joachims \(2020\)Off\-policy bandits with deficient support\.InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 965–975\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1),[§1](https://arxiv.org/html/2607.22012#S1.p9.1),[§2](https://arxiv.org/html/2607.22012#S2.p4.1),[§2](https://arxiv.org/html/2607.22012#S2.p4.5),[§2](https://arxiv.org/html/2607.22012#S2.p4.6)\.
- N\. Sachdeva, L\. Wang, D\. Liang, N\. Kallus, and J\. McAuley \(2024\)Off\-policy evaluation for large action spaces via policy convolution\.InProceedings of the ACM on Web Conference 2024,pp\. 3576–3585\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1),[§5](https://arxiv.org/html/2607.22012#S5.p2.1)\.
- Y\. Saito and T\. Joachims \(2021\)Counterfactual learning and evaluation for recommender systems: foundations, implementations, and recent advances\.InProceedings of the 15th ACM Conference on Recommender Systems,pp\. 828–830\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1)\.
- Y\. Saito and T\. Joachims \(2022\)Off\-policy evaluation for large action spaces via embeddings\.InProceedings of the 39th International Conference on Machine Learning,pp\. 19089–19122\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1),[§4](https://arxiv.org/html/2607.22012#S4.p6.2)\.
- Y\. Saito, Q\. Ren, and T\. Joachims \(2023\)Off\-policy evaluation for large action spaces via conjunct effect modeling\.InProceedings of the 40th International Conference on Machine learning,pp\. 29734–29759\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1),[§1](https://arxiv.org/html/2607.22012#S1.p8.1)\.
- Y\. Saito, T\. Udagawa, H\. Kiyohara, K\. Mogi, Y\. Narita, and K\. Tateno \(2021\)Evaluating the robustness of off\-policy evaluation\.InProceedings of the 15th ACM Conference on Recommender Systems,pp\. 114–123\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p1.1)\.
- O\. Sakhi, I\. Aouali, P\. Alquier, and N\. Chopin \(2024\)Logarithmic smoothing for pessimistic off\-policy evaluation, selection and learning\.arXiv preprint arXiv:2405\.14335\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p3.1)\.
- Y\. Su, M\. Dimakopoulou, A\. Krishnamurthy, and M\. Dudík \(2020a\)Doubly robust off\-policy evaluation with shrinkage\.InProceedings of the 37th International Conference on Machine Learning,Vol\.119,pp\. 9167–9176\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1)\.
- Y\. Su, P\. Srinath, and A\. Krishnamurthy \(2020b\)Adaptive estimator selection for off\-policy evaluation\.InProceedings of the 37th International Conference on Machine Learning,Vol\.119,pp\. 9196–9205\.Cited by:[§B\.3](https://arxiv.org/html/2607.22012#A2.SS3.p6.1),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p10.9)\.
- Y\. Su, L\. Wang, M\. Santacatterina, and T\. Joachims \(2019\)Cab: continuous adaptive blending for policy evaluation and learning\.InProceedings of the 36th International Conference on Machine Learning,Vol\.97,pp\. 6005–6014\.Cited by:[§3\.2](https://arxiv.org/html/2607.22012#S3.SS2.p1.4)\.
- M\. Sugiyama, T\. Suzuki, and T\. Kanamori \(2012\)Density ratio estimation in machine learning\.Cambridge University Press\.Cited by:[Appendix C](https://arxiv.org/html/2607.22012#A3.p1.5),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p1.13)\.
- A\. Swaminathan and T\. Joachims \(2015a\)Counterfactual risk minimization: learning from logged bandit feedback\.InProceedings of the 32nd International Conference on Machine Learning,Vol\.37,pp\. 814–823\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.22012#A1.p3.1)\.
- A\. Swaminathan and T\. Joachims \(2015b\)The self\-normalized estimator for counterfactual learning\.Advances in Neural Information Processing Systems28\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1)\.
- M\. F\. Taufiq, A\. Doucet, R\. Cornish, and J\. Ton \(2023\)Marginal density ratio for off\-policy evaluation in contextual bandits\.Advances in Neural Information Processing Systems36\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p3.1)\.
- T\. Udagawa, H\. Kiyohara, Y\. Narita, Y\. Saito, and K\. Tateno \(2023\)Policy\-adaptive estimator selection for off\-policy evaluation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 10025–10033\.Cited by:[§B\.3](https://arxiv.org/html/2607.22012#A2.SS3.p6.1),[§3\.1](https://arxiv.org/html/2607.22012#S3.SS1.p10.9)\.
- M\. Uehara, M\. Kato, and S\. Yasui \(2020\)Off\-policy evaluation and learning for external validity under a covariate shift\.Advances in Neural Information Processing Systems33\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p2.5),[§1](https://arxiv.org/html/2607.22012#S1.p7.4)\.
- M\. Uehara, C\. Shi, and N\. Kallus \(2022\)A review of off\-policy evaluation in reinforcement learning\.arXiv preprint arXiv:2212\.06355\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1)\.
- Y\. Wang, A\. Agarwal, and M\. Dudık \(2017\)Optimal and adaptive off\-policy evaluation in contextual bandits\.InProceedings of the 34th International Conference on Machine Learning,pp\. 3589–3597\.Cited by:[Appendix A](https://arxiv.org/html/2607.22012#A1.p1.1),[§1](https://arxiv.org/html/2607.22012#S1.p2.1)\.
- P\. Wu and K\. Fukumizu \(2021\)β\\beta\-Intact\-vae: identifying and estimating causal effects under limited overlap\.arXiv preprint arXiv:2110\.05225\.Cited by:[§1](https://arxiv.org/html/2607.22012#S1.p9.1)\.
- Z\. Yang, H\. Gao, D\. Gao, L\. Yang, L\. Yang, X\. Cai, W\. Ning, and G\. Zhang \(2024\)MLoRA: multi\-domain low\-rank adaptive network for ctr prediction\.InProceedings of the 18th ACM Conference on Recommender Systems,pp\. 287–297\.Cited by:[§D\.1](https://arxiv.org/html/2607.22012#A4.SS1.p2.10)\.

## Appendix ARelated Work

Off\-policy evaluation \(OPE\) and learning \(OPL\) have gained particular attention in contextual bandit settings as they offer a safe and cost\-efficient alternative to online A/B testing\(Mehrotraet al\.,[2018](https://arxiv.org/html/2607.22012#bib.bib37); Gilotteet al\.,[2018](https://arxiv.org/html/2607.22012#bib.bib35); Saitoet al\.,[2021](https://arxiv.org/html/2607.22012#bib.bib32); Kiyoharaet al\.,[2024a](https://arxiv.org/html/2607.22012#bib.bib34)\)and are of high practical relevance\. While several practical estimators and policy gradient methods \(most of which are based on importance weighting\) already exist\(Saito and Joachims,[2021](https://arxiv.org/html/2607.22012#bib.bib31); Ueharaet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib38); Dudíket al\.,[2014](https://arxiv.org/html/2607.22012#bib.bib25); Wanget al\.,[2017](https://arxiv.org/html/2607.22012#bib.bib48); Suet al\.,[2020a](https://arxiv.org/html/2607.22012#bib.bib46)\)for each of OPE or OPL, the effectiveness of these methods can decline in challenging but realistic scenarios\. First, high bias may occur when the logging policy fails to satisfy the common support condition\(Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29)\), which becomes even more severe when the logging policy is deterministic or when there are new actions\. Several recent works have addressed violations of common support \(also known as support deficiency\) by exploiting additional information about actions, restricting the action space, applying reward extrapolation, or limiting the policy space, but they cannot handle deterministic logging policies or entirely new actions\(Felicioniet al\.,[2022](https://arxiv.org/html/2607.22012#bib.bib30); Sachdevaet al\.,[2020](https://arxiv.org/html/2607.22012#bib.bib29)\)\. It may also be helpful to use some structure in the action or reward space to relax the common support requirement, as explored bySaito and Joachims \([2022](https://arxiv.org/html/2607.22012#bib.bib28)\); Saitoet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib20)\); Ciefet al\.\([2024a](https://arxiv.org/html/2607.22012#bib.bib42)\); Taufiqet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib40)\); Sachdevaet al\.\([2024](https://arxiv.org/html/2607.22012#bib.bib39)\); Kiyoharaet al\.\([2024b](https://arxiv.org/html/2607.22012#bib.bib43)\)\. For example,Saito and Joachims \([2022](https://arxiv.org/html/2607.22012#bib.bib28)\)leverage additional information about the actions in the form of action embeddings, andKiyoharaet al\.\([2023](https://arxiv.org/html/2607.22012#bib.bib41)\)define importance weights in a low\-dimensional slate abstraction space\. However, this useful structure is not always learnable\. In addition, existing estimators and methods based on importance weighting face significant variance issues, particularly when the logged dataset is small and the action space is large\(Saito and Joachims,[2022](https://arxiv.org/html/2607.22012#bib.bib28); Ciefet al\.,[2024a](https://arxiv.org/html/2607.22012#bib.bib42); Sachdevaet al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib39)\)\. Weight clipping\(Swaminathan and Joachims,[2015a](https://arxiv.org/html/2607.22012#bib.bib53); Suet al\.,[2020a](https://arxiv.org/html/2607.22012#bib.bib46)\)or normalization\(Swaminathan and Joachims,[2015b](https://arxiv.org/html/2607.22012#bib.bib54)\)might be applied to mitigate variance, but they often introduce a substantial amount of bias in estimation\.

Another line of work is OPE/L under covariate shift, as researched byUeharaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib33)\)\. This research focuses on cases where the context distributions differ between the logged data \(phist​\(x\)p^\{\\text\{hist\}\}\(x\)\) and the evaluation environment \(peval​\(x\)p^\{\\text\{eval\}\}\(x\)\), while the reward distributionp​\(r\|x,a\)p\(r\|x,a\)remains unchanged\. Under this setup,Ueharaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib33)\)propose doubly robust and efficient estimators by leveraging an estimator of the density ratio between the historical and evaluation data distributions\. In contrast, we aim to estimate the same estimand using data from both target and source domains \(which may include multiple source domains\), with each domain having unique context distributionspk​\(x\)p^\{k\}\(x\)and reward distributionspk​\(r\|x,a\)p^\{k\}\(r\|x,a\)\. By leveraging this new setup, our primary goal is to address challenging scenarios in the target domain, such as new actions, deterministic logging policies, and extremely small logged data, which clearly differ from the motivations ofUeharaet al\.\([2020](https://arxiv.org/html/2607.22012#bib.bib33)\)\.

Finally, we mention the work byCiefet al\.\([2024b](https://arxiv.org/html/2607.22012#bib.bib52)\)andLiuet al\.\([2024](https://arxiv.org/html/2607.22012#bib.bib16)\), which formulate the problem of cross\-domain offline reinforcement learning \(offline RL\)\. The goals of their formulations are somewhat related to ours, as they aim to learn a policy that performs well in the target domain by leveraging data from both target and source domains\. However, they focus on the offline RL setup and consider only shifts in transition dynamics\. In contrast, we focus on the contextual bandit setup and address varying context and reward distributions across domains\. We also analyze and experiment with the problem of OPE, whileCiefet al\.\([2024b](https://arxiv.org/html/2607.22012#bib.bib52)\)andLiuet al\.\([2024](https://arxiv.org/html/2607.22012#bib.bib16)\)focus solely on policy learning\. Furthermore, it is important to note that we explore the problem of cross\-domain OPE/OPL to tackle the unsolved issues of new actions, deterministic logging, and limited logged data, whereas previous work more generally considers the cross\-domain setup\. Note that we did not compare pessimistic OPL methods\(Swaminathan and Joachims,[2015a](https://arxiv.org/html/2607.22012#bib.bib53); London and Sandler,[2019](https://arxiv.org/html/2607.22012#bib.bib8); Gabbianelliet al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib6); Sakhiet al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib7)\)because they are not relevant to our context\. Our main motivation is to solve the prevalent problem of \(completely\) deterministic logging and new actions, issues that pessimistic techniques do not aim to address\. However, our proposed method could easily be combined with a pessimistic approach if one wants to do so\.

## Appendix BOmitted Proofs

### B\.1Proof of Theorem[3\.1](https://arxiv.org/html/2607.22012#S3.Thmthm1)

###### Proof\.

The bias of COPE under Condition[3\.1](https://arxiv.org/html/2607.22012#S3.Thmassum1)is derived as

Bias​\(V^COPE​\(π;Dϕ​\(T\)\)\)\\displaystyle\\mathrm\{Bias\}\(\\hat\{V\}\_\{\\mathrm\{COPE\}\}\(\\pi;D^\{\\phi\(T\)\}\)\)=𝔼​\[V^COPE​\(π;Dϕ​\(T\)\)\]−VT​\(π\)\\displaystyle=\\mathbb\{E\}\[\\hat\{V\}\_\{\\mathrm\{COPE\}\}\(\\pi;D^\{\\phi\(T\)\}\)\]\-V^\{T\}\(\\pi\)=1nϕ​\(T\)​∑k∈ϕ​\(T\)∑i=1nk𝔼pk​\(x\)​π0k​\(a\|x\)​pk​\(r\|x,a\)​\[π​\(aik\|xik\)pϕ​\(T\)​\(aik\|xik\)​\(rik−q^​\(xik,aik\)\)\]\\displaystyle=\\frac\{1\}\{n^\{\\phi\(T\)\}\}\\sum\_\{k\\in\\phi\(T\)\}\\sum\_\{i=1\}^\{n^\{k\}\}\\mathbb\{E\}\_\{p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)p^\{k\}\(r\|x,a\)\}\[\\frac\{\\pi\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\{\{p^\{\\phi\(T\)\}\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\}\(r\_\{i\}^\{k\}\-\\hat\{q\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\)\]\+1nT​∑i=1nT𝔼pT​\(x\)​π0T​\(a\|x\)​pT​\(r\|x,a\)​\[𝔼π​\(a\|xi\)​q^​\(xiT,aT\)\]\\displaystyle\\phantom\{=\}\+\\frac\{1\}\{n^\{T\}\}\\sum\_\{i=1\}^\{n^\{T\}\}\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\_\{0\}^\{T\}\(a\|x\)p^\{T\}\(r\|x,a\)\}\[\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}\)\}\\hat\{q\}\(x\_\{i\}^\{T\},a^\{T\}\)\]−𝔼pT​\(x\)​π​\(a\|x\)​pT​\(r\|x,a\)​\[r\]\\displaystyle\\phantom\{=\}\-\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)p^\{T\}\(r\|x,a\)\}\[r\]=∑k∈ϕ​\(T\)nknϕ​\(T\)​𝔼pk​\(x\)​π0k​\(a\|x\)​pk​\(r\|x,a\)​\[π​\(ak\|xk\)pϕ​\(T\)​\(ak\|xk\)​\(rk−q^​\(xk,ak\)\)\]\\displaystyle=\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\mathbb\{E\}\_\{p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)p^\{k\}\(r\|x,a\)\}\[\\frac\{\\pi\(a^\{k\}\|x^\{k\}\)\}\{\{p^\{\\phi\(T\)\}\(a^\{k\}\|x^\{k\}\)\}\}\(r^\{k\}\-\\hat\{q\}\(x^\{k\},a^\{k\}\)\)\]\+𝔼pT​\(x\)​π0T​\(a\|x\)​pT​\(r\|x,a\)​\[∑a∈𝒜π​\(a\|xT\)​q^​\(xT,a\)\]\\displaystyle\\phantom\{=\}\+\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\_\{0\}^\{T\}\(a\|x\)p^\{T\}\(r\|x,a\)\}\[\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\|x^\{T\}\)\\hat\{q\}\(x^\{T\},a\)\]−𝔼pT​\(x\)​π​\(a\|x\)​\[qT​\(x,a\)\]\\displaystyle\\phantom\{=\}\-\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\[q^\{T\}\(x,a\)\]=∑k∈ϕ​\(T\)nknϕ​\(T\)​𝔼pk​\(x\)​π0k​\(a\|x\)​\[π​\(ak\|xk\)pϕ​\(T\)​\(ak\|xk\)​Δq,q^k​\(x,a\)\]\\displaystyle=\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\mathbb\{E\}\_\{p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\}\[\\frac\{\\pi\(a^\{k\}\|x^\{k\}\)\}\{\{p^\{\\phi\(T\)\}\(a^\{k\}\|x^\{k\}\)\}\}\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\]\+∑x∈𝒳∑a∈𝒜pT​\(x\)​π​\(a\|x\)​q^​\(x,a\)\\displaystyle\\phantom\{=\}\+\\sum\_\{x\\in\\mathcal\{X\}\}\\sum\_\{a\\in\\mathcal\{A\}\}p^\{T\}\(x\)\\pi\(a\|x\)\\hat\{q\}\(x,a\)−∑x∈𝒳∑a∈𝒜pT​\(x\)​π​\(a\|x\)​qT​\(x,a\)\\displaystyle\\phantom\{=\}\-\\sum\_\{x\\in\\mathcal\{X\}\}\\sum\_\{a\\in\\mathcal\{A\}\}p^\{T\}\(x\)\\pi\(a\|x\)q^\{T\}\(x,a\)=∑x∈𝒳∑k∈ϕ​\(T\)∑a∈𝒜pT​\(x\)​π​\(a\|x\)∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)​π0k​\(a\|x\)​nknϕ​\(T\)​pk​\(x\)​π0k​\(a\|x\)​Δq,q^k​\(x,a\)\\displaystyle=\\sum\_\{x\\in\\mathcal\{X\}\}\\sum\_\{k\\in\\phi\(T\)\}\\sum\_\{a\\in\\mathcal\{A\}\}\\frac\{p^\{T\}\(x\)\\pi\(a\|x\)\}\{\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)−∑x∈𝒳∑a∈𝒜pT​\(x\)​π​\(a\|x\)​Δq,q^T​\(x,a\)\\displaystyle\\phantom\{=\}\-\\sum\_\{x\\in\\mathcal\{X\}\}\\sum\_\{a\\in\\mathcal\{A\}\}p^\{T\}\(x\)\\pi\(a\|x\)\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)=∑x∈𝒳pT\(x\)\{\(∑k∈ϕ​\(T\)∑a∈𝒜π​\(a\|x\)∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)​π0k​\(a\|x\)nknϕ​\(T\)pk\(x\)π0k\(a\|x\)Δq,q^k\(x,a\)\)\\displaystyle=\\sum\_\{x\\in\\mathcal\{X\}\}p^\{T\}\(x\)\\\{\(\\sum\_\{k\\in\\phi\(T\)\}\\sum\_\{a\\in\\mathcal\{A\}\}\\frac\{\\pi\(a\|x\)\}\{\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\)−∑a∈𝒜π\(a\|x\)Δq,q^T\(x,a\)\}\\displaystyle\\phantom\{=\}\-\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\|x\)\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\\}=∑x∈𝒳pT\(x\)∑a∈𝒜π\(a\|x\)\{\(∑k∈ϕ​\(T\)1∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)​π0k​\(a\|x\)nknϕ​\(T\)pk\(x\)π0k\(a\|x\)Δq,q^k\(x,a\)\)\\displaystyle=\\sum\_\{x\\in\\mathcal\{X\}\}p^\{T\}\(x\)\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\|x\)\\\{\(\\sum\_\{k\\in\\phi\(T\)\}\\frac\{1\}\{\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\)−Δq,q^T\(x,a\)\}\\displaystyle\\phantom\{=\}\-\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\\}=𝔼pT​\(x\)​π​\(a\|x\)​\[\{\(∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)pT​\(x\)​π0k​\(a\|x\)pϕ​\(T\)​\(a\|x\)​Δq,q^k​\(x,a\)\)−Δq,q^T​\(x,a\)\}\]\.\\displaystyle=\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\\left\[\\left\\\{\\left\(\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\frac\{p^\{k\}\(x\)\}\{p^\{T\}\(x\)\}\\frac\{\\pi\_\{0\}^\{k\}\(a\|x\)\}\{p^\{\\phi\(T\)\}\(a\|x\)\}\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\\right\)\-\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\right\\\}\\right\]\.∎

### B\.2Proof of Corollary[3\.1](https://arxiv.org/html/2607.22012#S3.Thmcorr1)

###### Proof\.

Starting from the final expression of[B\.1](https://arxiv.org/html/2607.22012#A2.SS1), the bias of COPE under the additional Condition[3\.2](https://arxiv.org/html/2607.22012#S3.Thmassum2)is calculated as

𝔼pT​\(x\)​π​\(a\|x\)​\[\{\(∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)pT​\(x\)​π0k​\(a\|x\)pϕ​\(T\)​\(a\|x\)​Δq,q^k​\(x,a\)\)−Δq,q^T​\(x,a\)\}\]\\displaystyle\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\\left\[\\left\\\{\\left\(\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\frac\{p^\{k\}\(x\)\}\{p^\{T\}\(x\)\}\\frac\{\\pi\_\{0\}^\{k\}\(a\|x\)\}\{p^\{\\phi\(T\)\}\(a\|x\)\}\\Delta\_\{q,\\hat\{q\}\}^\{k\}\(x,a\)\\right\)\-\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\right\\\}\\right\]=𝔼pT​\(x\)​π​\(a\|x\)​\[\{\(∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)pT​\(x\)​π0k​\(a\|x\)pϕ​\(T\)​\(a\|x\)​Δq,q^T​\(x,a\)\)−Δq,q^T​\(x,a\)\}\]∵Condition[3\.2](https://arxiv.org/html/2607.22012#S3.Thmassum2)\\displaystyle=\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\\left\[\\left\\\{\\left\(\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\frac\{p^\{k\}\(x\)\}\{p^\{T\}\(x\)\}\\frac\{\\pi\_\{0\}^\{k\}\(a\|x\)\}\{p^\{\\phi\(T\)\}\(a\|x\)\}\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\right\)\-\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\right\\\}\\right\]\\because\\text\{Condition~\\ref\{ass:cpc\}\}=𝔼pT​\(x\)​π​\(a\|x\)​\[Δq,q^T​\(x,a\)​\{\(∑k∈ϕ​\(T\)nknϕ​\(T\)​pk​\(x\)pT​\(x\)​π0k​\(a\|x\)pϕ​\(T\)​\(a\|x\)\)−1\}\]\\displaystyle=\\mathbb\{E\}\_\{p^\{T\}\(x\)\\pi\(a\|x\)\}\\left\[\\Delta\_\{q,\\hat\{q\}\}^\{T\}\(x,a\)\\left\\\{\\left\(\\sum\_\{k\\in\\phi\(T\)\}\\frac\{n^\{k\}\}\{n^\{\\phi\(T\)\}\}\\frac\{p^\{k\}\(x\)\}\{p^\{T\}\(x\)\}\\frac\{\\pi\_\{0\}^\{k\}\(a\|x\)\}\{p^\{\\phi\(T\)\}\(a\|x\)\}\\right\)\-1\\right\\\}\\right\]=0,\\displaystyle=0,which concludes the proof\. ∎

### B\.3Bias of the Baseline Methods

Here, we can derive the bias of IPS\-ALL by calculating its expectation as

𝔼𝒟​\[V^IPS−ALL​\(π;𝒟\)\]\\displaystyle\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\\hat\{V\}\_\{\\mathrm\{IPS\-ALL\}\}\(\\pi;\\mathcal\{D\}\)\]=1N​∑k=1K∑i=1nk𝔼pk​\(x\)​π0k​\(a\|x\)​\[wk​\(xk,ak\)​qk​\(x,a\)\]\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}\\mathbb\{E\}\_\{p^\{k\}\(x\)\\pi\_\{0\}^\{k\}\(a\|x\)\}\[w^\{k\}\(x^\{k\},a^\{k\}\)q^\{k\}\(x,a\)\]=1N​∑k=1K∑i=1nk𝔼pk​\(x\)​πk​\(a\|x\)​\[qk​\(x,a\)\]∵wk​\(xk,ak\)=πk​\(a\|x\)π0k​\(a\|x\)\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}\\mathbb\{E\}\_\{p^\{k\}\(x\)\\pi^\{k\}\(a\|x\)\}\[q^\{k\}\(x,a\)\]\\;\\because w^\{k\}\(x^\{k\},a^\{k\}\)=\\frac\{\\pi^\{k\}\(a\\,\|\\,x\)\}\{\\pi^\{k\}\_\{0\}\(a\\,\|\\,x\)\}=∑k=1KnkN​𝔼pk​\(x\)​π​\(a\|x\)​\[qk​\(x,a\)\]\.\\displaystyle=\\sum\_\{k=1\}^\{K\}\\frac\{n^\{k\}\}\{N\}\\mathbb\{E\}\_\{p\_\{k\}\(x\)\\pi\(a\|x\)\}\[q^\{k\}\(x,a\)\]\.
Therefore, the bias of IPS\-ALL is:

Bias⁡\(V^IPS​\(π;𝒟\)\)=\|𝔼pT​\(x\)​π​\(a\|x\)​\[qT​\(x,a\)\]−\(∑knkN​𝔼pk​\(x\)​π​\(a\|x\)​\[qk​\(x,a\)\]\)\|\.\\operatorname\{Bias\}\(\\hat\{V\}\_\{\\mathrm\{IPS\}\}\(\\pi;\\mathcal\{D\}\)\)=\\left\|\\mathbb\{E\}\_\{p\_\{T\}\(x\)\\pi\(a\|x\)\}\[q^\{T\}\(x,a\)\]\-\\left\(\\sum\_\{k\}\\frac\{n^\{k\}\}\{N\}\\mathbb\{E\}\_\{p\_\{k\}\(x\)\\pi\(a\|x\)\}\[q^\{k\}\(x,a\)\]\\right\)\\right\|\.
We observe that they are no longer guaranteed to remain unbiased in our setup\. This bias becomes significant when the reward distributions across domains differ substantially because the expectation of IPS\-ALL is characterized by the weighted sum of the policy values of all the domains involved\. In contrast, the bias becomes zero when the reward distributions across all domains are identical \(which is no longer a cross\-domain setup\)\. Based on this derivation and Theorem 3\.1, we can compare the bias of COPE and that of IPS/DR\-ALL as follows:

- •COPE performs better when the distributions across domains are substantially different and also when the pairwise estimation byq^​\(x,a\)\\hat\{q\}\(x,a\)is accurate\.
- •IPS/DR\-ALL performs better when the distributions across domains are similar, i\.e\., when the cross\-domain setup is close to a typical single\-domain setup\.

When the setup is close to the typical single\-domain scenario, effective estimators such as DR and its extensions are already well\-known\. Therefore, we focused more on novel and challenging scenarios where the distributions across domains are not necessarily similar, as baseline methods tend to fail in these cases as shown above and in our experiments\.

In practice, we do not precisely know which of COPE and baselines are better\. This is not the case only for our setup, but also for the other OPE problems\. A typical example is that we never know which of IPS and DM is better, because the comparison depends on many \(potentially unknown\) parameters such as the reward noise and logged data size\. This is why there exists an orthogonal line of work around estimator selection in OPE\(Suet al\.,[2020b](https://arxiv.org/html/2607.22012#bib.bib47); Udagawaet al\.,[2023](https://arxiv.org/html/2607.22012#bib.bib50); Ciefet al\.,[2024b](https://arxiv.org/html/2607.22012#bib.bib52)\), and we can use those methods to identify the appropriate estimator tailored to the specific problem when we use OPE in general\.

## Appendix CDensity Ratio Estimation

Our implementation in the experiments relies on one of the most standard methods, unconstrained Least\-Squares Importance Fitting \(uLSIF\), to perform density ratio estimation proposed in\(Kanamoriet al\.,[2012](https://arxiv.org/html/2607.22012#bib.bib14); Sugiyamaet al\.,[2012](https://arxiv.org/html/2607.22012#bib.bib15)\)\. The method considers minimizing the following squared error betweenssandrrwheressis an estimator for the ratio andw​\(x\)=pk​\(x\)/pT​\(x\)w\(x\)=p^\{k\}\(x\)/p^\{T\}\(x\)is the ratio we need to estimate in order to usepϕ​\(T\)​\(x,a\)p^\{\\phi\(T\)\}\(x,a\)in Eq\. \([10](https://arxiv.org/html/2607.22012#S3.E10)\):

𝔼​\[\(s​\(x\)−w​\(x\)\)2\]=𝔼pk​\(x\)​\[\(w​\(x\)\)2\]−2​𝔼pT​\(x\)​\[s​\(x\)\]\+𝔼pk​\(x\)​\[\(s​\(x\)\)2\]\\mathbb\{E\}\[\(s\(x\)\-w\(x\)\)^\{2\}\]=\\mathbb\{E\}\_\{p^\{k\}\(x\)\}\[\(w\(x\)\)^\{2\}\]\-2\\mathbb\{E\}\_\{p^\{T\}\(x\)\}\[s\(x\)\]\+\\mathbb\{E\}\_\{p^\{k\}\(x\)\}\[\(s\(x\)\)^\{2\}\]
The first term of the equation does not affect the result of the optimization and we can ignore it, i\.e\., the density ratio is estimated through the following minimization problem:

s∗=arg​mins∈𝒮​\[12​𝔼pT​\(x\)​\[\(s​\(x\)\)2\]−𝔼pk​\(x\)​\[s​\(x\)\]\]s^\{\*\}=\\underset\{s\\in\\mathcal\{S\}\}\{\\operatorname\*\{arg\\,min\}\}\\,\\left\[\\frac\{1\}\{2\}\\mathbb\{E\}\_\{p^\{T\}\(x\)\}\[\(s\(x\)\)^\{2\}\]\-\\mathbb\{E\}\_\{p^\{k\}\(x\)\}\[s\(x\)\]\\right\]
To solve the empirical version of this optimization,Kanamoriet al\.\([2012](https://arxiv.org/html/2607.22012#bib.bib14)\)use kernel\-based hypotheses to estimate the density ratio nonparametrically with a regularization term as usual\.[https://github\.com/hoxo\-m/densratio\_py](https://github.com/hoxo-m/densratio_py), which we relied on in our experiments, is one of the well\-known public implementations of the method\.

## Appendix DAdditional Experiment Setup and Results

### D\.1Additional Results on Real\-World Data

This section compares the OPE estimators under varying sizes of the target cluster\|ϕ​\(T\)\|∈\{2,4,6,8,10\}\|\\phi\(T\)\|\\in\\\{2,4,6,8,10\\\}\(a key hyperparameter of COPE\) in Figure[7](https://arxiv.org/html/2607.22012#A4.F7)\. Note that the baseline estimators are independent of this parameter, so their results remain constant\. We observe that COPE, when using a smaller target cluster size, has a small bias but a large variance\. This is because the smaller target cluster size reduces the bias caused by differences in the distribution across domains\. In contrast, the smaller size also decreases the amount of available logged data, leading to an increase in variance\. Additionally, the figure suggests that COPE with a larger target cluster size exhibits higher variance\. This occurs because as the number of domains included in the COPE estimation increases, larger variances are more likely to arise from the density ratio terms\. Nevertheless, the results show that COPE outperforms the baseline estimators, even with small \(\|ϕ​\(T\)\|=2\|\\phi\(T\)\|=2\) or large \(\|ϕ​\(T\)\|=10\|\\phi\(T\)\|=10\) cluster sizes, as seen in the OPL experiment\.

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/legend_ope.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_mse_graph_cl.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_bias_graph_cl.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/real-world_variance_graph_cl.png)

Figure 7:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with with varying size of the target cluster\.Moreover, we excludeq¯​\(u\)\\bar\{q\}\(u\)from the definition of the user distributionpk​\(u\)p^\{k\}\(u\), following an observation that this might favor our methods\. Instead, we define a linear function of the user featurefk​\(u\)=βk​xuf^\{k\}\(u\)=\\beta^\{k\}x\_\{u\}, where the coefficient vectorβk\\beta^\{k\}is sampled from a normal distribution separately for each domainkk\. We then define the user distributionpk​\(u\)p^\{k\}\(u\)for each domain aspk​\(u\):=exp⁡\(fk​\(u\)\)∑u′∈𝒰exp⁡\(fk​\(u′\)\)p^\{k\}\(u\):=\\frac\{\\exp\(f^\{k\}\(u\)\)\}\{\\sum\_\{u^\{\\prime\}\\in\\mathcal\{U\}\}\\exp\(f^\{k\}\(u^\{\\prime\}\)\)\}\. This procedure follows the multi\-domain recommendation literature, which uses user features to define the domains in their experiments\(Yanget al\.,[2024](https://arxiv.org/html/2607.22012#bib.bib11)\)\. More importantly, this approach avoids any dependence of the distribution onq¯​\(u\)\\bar\{q\}\(u\)\. We conducted both OPE and OPL experiments based on this new user distribution, and the results are summarized in Tables[5](https://arxiv.org/html/2607.22012#A4.T5)to[5](https://arxiv.org/html/2607.22012#A4.T5)\. The results indicate that our method mostly outperforms the baseline methods that use only the target domain data or all the data across target and source domains in both OPE and OPL experiments \(bold fonts indicate the best method, if the best and second best are not significantly different, we use bold fonts for both\), even withpk​\(u\)p^\{k\}\(u\)independent ofq¯​\(u\)\\bar\{q\}\(u\)\.

Furthermore, we can extend the definition of the user distribution as:

pk​\(u\):=exp⁡\(λ​q¯​\(u\)\+\(1−λ\)​fk​\(u\)\)∑u′∈𝒰exp\(λq¯\(u’\)\+\(1−λ\)fk\(u′\)\)\)p^\{k\}\(u\):=\\frac\{\\exp\(\\lambda\\bar\{q\}\(u\)\+\(1\-\\lambda\)f^\{k\}\(u\)\)\}\{\\sum\_\{u^\{\\prime\}\\in\\mathcal\{U\}\}\\exp\(\\lambda\\bar\{q\}\(u’\)\+\(1\-\\lambda\)f^\{k\}\(u^\{\\prime\}\)\)\)\}whereλ\\lambdacontrols the degree to which the user distribution depends onq¯​\(u\)\\bar\{q\}\(u\)\. Whenλ=1\\lambda=1, the setup reduces to our original experiment\. Table[5](https://arxiv.org/html/2607.22012#A4.T5)shows the result of the OPL experiment for varyingλ\\lambdavalues\. We should mention here that the difference between the proposed method and the baselines decreases whenλ\\lambdais small, which is reasonable\. These additional observations, however, demonstrate the robustness of COPE to varying user distributions, even with a simple clustering procedure\. Nevertheless, we believe a more principled and effective approach to domain clustering for our method likely exists\. As mentioned in Section 5, we consider the development of a clustering algorithm tailored to Cross\-Domain OPE and OPL to be a valuable direction for future research\.

Table 1:MSE in OPE for varying ratios of new actions in the target domain
Table 2:MSE in OPE for varying numbers of users with deterministic logging in the target domain
Table 3:Test \(Relative\) Policy Value in OPL for varying ratios of new actions in the target domain
Table 4:Test \(Relative\) Policy Value in OPL for varying numbers of users with deterministic logging in the target domain
Table 5:Test \(Relative\) Policy Value in OPL for varyingλ\\lambdavalues

### D\.2Synthetic Experiment

This section empirically demonstrates the advantages of COPE over existing approaches using synthetic data\. The benefits of using synthetic data include the ability to accurately compare estimators based on ground\-truth policy values and the flexibility to control experiment configurations in order to test a wide range of scenarios\. To generate the synthetic data, we first randomly cluster a total ofK=30K=30domains, corresponding to the definition ofϕ​\(k\)\\phi\(k\)\. We then sample a 5\-dimensional domain embeddingeke^\{k\}for each domainkkfrom the standard normal distribution\. We also sample 10\-dimensional context vectorsxkx^\{k\}from a normal distribution with meanμk\\mu\_\{k\}and a standard deviation ofσx=1\\sigma\_\{x\}=1\. The mean parameterμk\\mu\_\{k\}of the context distribution is sampled from a uniform distribution within the range\[−1\.0,1\.0\]\[\-1\.0,1\.0\]\.

We synthesize the expected reward function by using domain embeddingeeas an input as below\.

q​\(x,a,e;λ\)=λ​g​\(x,a,c\)\+h​\(x,a,e\),q\(x,a,e;\\lambda\)=\\lambda g\(x,a,c\)\+h\(x,a,e\),\(15\)whereλ\\lambdais an experiment parameter that controls the influence of the domain\-cluster effect \(gg\) compared to the domain\-specific effect \(hh\), where we useλ=0\.5\\lambda=0\.5as the default parameter\. Specifically, we used the following functions asg​\(⋅,⋅,⋅\)g\(\\cdot,\\cdot,\\cdot\)\(domain\-cluster effect\) andh​\(⋅,⋅,⋅\)h\(\\cdot,\\cdot,\\cdot\)\(domain\-specific effect\), respectively\.

g​\(x,a,c\)\\displaystyle g\(x,a,c\):=x⊤​Mx,ac​one​\_​hota\+θx,c⊤​x\+θa,c⊤​one​\_​hota\+θc⊤​one​\_​hotc,\\displaystyle:=x^\{\\top\}M^\{c\}\_\{x,a\}\\mathrm\{one\\\_hot\}\_\{a\}\+\\theta\_\{x,c\}^\{\\top\}x\+\\theta\_\{a,c\}^\{\\top\}\\mathrm\{one\\\_hot\}\_\{a\}\+\\theta\_\{c\}^\{\\top\}\\mathrm\{one\\\_hot\}\_\{c\},h​\(x,a,e\)\\displaystyle h\(x,a,e\):=θe⊤​e\+x⊤​Mx,e​e\+one​\_​hota⊤​Ma,e​e,\\displaystyle:=\\theta\_\{e\}^\{\\top\}e\+x^\{\\top\}M\_\{x,e\}e\+\\mathrm\{one\\\_hot\}\_\{a\}^\{\\top\}M\_\{a,e\}e,whereone​\_​hota\\mathrm\{one\\\_hot\}\_\{a\}is the one\-hot encoding of the action andone​\_​hotc\\mathrm\{one\\\_hot\}\_\{c\}is the one\-hot encoding of the domain\-clustercc\.Mx,ac,θx,c,θa,c,θcM^\{c\}\_\{x,a\},\\theta\_\{x,c\},\\theta\_\{a,c\},\\theta\_\{c\}are parameter matrices or vectors in domain\-cluster effect function sampled from a uniform distribution with range\[−1,1\]\[\-1,1\]separately for each given domain clustercc\. Also,Mx,e,Ma,eM\_\{x,e\},M\_\{a,e\}are parameter matrices in domain\-specific effect function sampled from a uniform distribution with range\[−1,1\]\[\-1,1\], andθe\\theta\_\{e\}is parameter vector sampled from a uniform distribution with range\[−10,10\]\[\-10,10\]separately for each given domain clustercc\.

Based on the expected reward function, we sample the rewardrkr^\{k\}from a normal distribution with meanq​\(x,a,e;λ\)q\(x,a,e;\\lambda\)and standard deviationσr=1\\sigma\_\{r\}=1\.

We define the logging policyπ0k\\pi\_\{0\}^\{k\}for each domainkkby applying softmax to the expected reward functionq​\(x,a,e;λ\)q\(x,a,e;\\lambda\)as

π0k​\(a\|x\):=exp\(βk⋅\(q\(x,a,e;λ\)\+ηx,a\)∑a′∈𝒜exp\(βk⋅\(q\(x,a′,e;λ\)\+ηx,a\)\\pi\_\{0\}^\{k\}\(a\|x\):=\\frac\{\\mathrm\{exp\}\(\\beta^\{k\}\\cdot\(q\(x,a,e;\\lambda\)\+\\eta\_\{x,a\}\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\mathrm\{exp\}\(\\beta^\{k\}\\cdot\(q\(x,a^\{\\prime\},e;\\lambda\)\+\\eta\_\{x,a\}\)\}\(16\)whereβk\\beta^\{k\}andηx,a\\eta\_\{x,a\}are sampled from a uniform distribution within range\[−0\.5,0\.5\]\[\-0\.5,0\.5\]\. In contrast, the evaluation policyπ\\piis defined as

π​\(a\|x\)=\(1−ϵ\)⋅𝕀​\{a=argmaxa′∈Aq​\(x,a′,e;λ\)\}\+ϵ\|𝒜\|,\\displaystyle\\pi\(a\|x\)=\(1\-\\epsilon\)\\cdot\\mathbb\{I\}\\\{a=\\mathrm\{argmax\}\_\{a^\{\\prime\}\\in A\}\\quad q\(x,a^\{\\prime\},e;\\lambda\)\\\}\+\\frac\{\\epsilon\}\{\|\\mathcal\{A\}\|\},\(17\)whereϵ∈\[0,1\]\\epsilon\\in\[0,1\]controls the quality ofπ\\piand we setϵ=0\.2\\epsilon=0\.2as default\.

#### D\.2\.1Baseline Estimators

Below, we define the baseline estimators compared in our \(both synthetic and real\-world\) experiments\.

Direct Method \(DM\)\.The DM estimator, which uses only logged data from the target domain, is defined as follows:

V^DM​\(π;𝒟T,q^T\):=1nT​∑i=1nT𝔼π​\(a\|xi\)​\[q^T​\(xi,a\)\],\\hat\{V\}\_\{\\rm\{DM\}\}\(\\pi;\\mathcal\{D\}^\{T\},\\hat\{q\}^\{T\}\):=\\frac\{1\}\{n^\{T\}\}\\sum\_\{i=1\}^\{n^\{T\}\}\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}\)\}\[\\hat\{q\}^\{T\}\(x\_\{i\},a\)\],whereq^T​\(x,a\)\\hat\{q\}^\{T\}\(x,a\)estimatesqT​\(x,a\)q^\{T\}\(x,a\)based on logged bandit data from the target domain\. Additionally, the DM\-ALL estimator, which uses logged data from both the target and source domains, is defined as follows:

V^DM−ALL​\(π;𝒟,q^\):=1N​∑k=1K∑i=1nk𝔼π​\(a\|xik\)​\[q^​\(xik,a\)\],\\hat\{V\}\_\{\\rm\{DM\-ALL\}\}\(\\pi;\\mathcal\{D\},\\hat\{q\}\):=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}^\{k\}\)\}\[\\hat\{q\}\(x\_\{i\}^\{k\},a\)\],whereq^​\(x,a\)\\hat\{q\}\(x,a\)estimatesq​\(x,a\)q\(x,a\)based on all domains logged bandit data\.

Inverse Propensity Score \(IPS\)\.The IPS estimator, which uses only logged data from the target domain, is defined as follows:

V^IPS​\(π;𝒟T\):=1n​∑i=1nπ​\(ai\|xi\)π0T​\(ai\|xi\)​ri=1n​∑i=1nwT​\(xi,ai\)​ri,\\hat\{V\}\_\{\\rm\{IPS\}\}\(\\pi;\\mathcal\{D\}^\{T\}\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\\pi\(a\_\{i\}\|x\_\{i\}\)\}\{\\pi\_\{0\}^\{T\}\(a\_\{i\}\|x\_\{i\}\)\}r\_\{i\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}w^\{T\}\(x\_\{i\},a\_\{i\}\)r\_\{i\},wherewT​\(x,a\):=π​\(a\|x\)/π0T​\(a\|x\)w^\{T\}\(x,a\):=\\pi\(a\|x\)/\\pi\_\{0\}^\{T\}\(a\|x\)is importance weight of the target domain\. The IPS\-ALL estimator, which uses logged data from both the target and source domains, is defined as follows:

V^IPS−ALL​\(π;𝒟\):=1N​∑k=1K∑i=1nkπ​\(aik\|xik\)π0k​\(aik\|xik\)​rik:=1N​∑k=1K∑i=1nkwk​\(xik,aik\)​rik,\\hat\{V\}\_\{\\rm\{IPS\-ALL\}\}\(\\pi;\\mathcal\{D\}\):=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}\\frac\{\\pi\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}\{\\pi\_\{0\}^\{k\}\(a\_\{i\}^\{k\}\|x\_\{i\}^\{k\}\)\}r\_\{i\}^\{k\}:=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}w^\{k\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)r\_\{i\}^\{k\},wherewk​\(x,a\):=π​\(a\|x\)/π0k​\(a\|x\)w^\{k\}\(x,a\):=\\pi\(a\|x\)/\\pi\_\{0\}^\{k\}\(a\|x\)is importance weight of each domain andN:=∑k=1KnkN:=\\sum\_\{k=1\}^\{K\}n^\{k\}\.

Doubly Robust \(DR\)\.The DR estimator, which uses only logged data from the target domain, is defined as follows:

V^DR​\(π;𝒟T,q^T\):=1n​∑i=1n\{w​\(xi,ai\)​\(ri−q^T​\(xi,ai\)\)\+𝔼π​\(a\|xi\)​\[q^T​\(xi,a\)\]\}\.\\hat\{V\}\_\{\\rm\{DR\}\}\(\\pi;\\mathcal\{D\}^\{T\},\\hat\{q\}^\{T\}\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\\{w\(x\_\{i\},a\_\{i\}\)\(r\_\{i\}\-\\hat\{q\}^\{T\}\(x\_\{i\},a\_\{i\}\)\)\+\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}\)\}\[\\hat\{q\}^\{T\}\(x\_\{i\},a\)\]\\\}\.The DR\-ALL estimator, which uses logged data from both the target and source domains, is defined as follows:

V^DR−ALL​\(π;𝒟,q^\):=1N​∑k=1K∑i=1nk\{wk​\(xik,aik\)​\(rik−q^​\(xik,aik\)\)\+𝔼π​\(a\|xik\)​\[q^​\(xik,a\)\]\}\.\\hat\{V\}\_\{\\rm\{DR\-ALL\}\}\(\\pi;\\mathcal\{D\},\\hat\{q\}\):=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{n^\{k\}\}\\\{w^\{k\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\(r\_\{i\}^\{k\}\-\\hat\{q\}\(x\_\{i\}^\{k\},a\_\{i\}^\{k\}\)\)\+\\mathbb\{E\}\_\{\\pi\(a\|x\_\{i\}^\{k\}\)\}\[\\hat\{q\}\(x\_\{i\}^\{k\},a\)\]\\\}\.

#### D\.2\.2Experiment Results on Synthetic Data

In the following, we report and discuss the MSE, squared bias, and variance of the estimators computed over 150 sets of logged data, each replicated with different seeds\. Note that the default parameters in our synthetic validation are\|𝒜\|=20\|\\mathcal\{A\}\|=20,K=30K=30,nk=100n^\{k\}=100, and\|cT\|=9\|c^\{T\}\|=9\.

First, we compare the estimators under varying ratios of new actions in the target domain,\|𝒰0T\|/\|𝒜\|∈\{0,0\.2,0\.4,0\.6,0\.8\}\|\\mathcal\{U\}\_\{0\}^\{T\}\|/\|\\mathcal\{A\}\|\\in\\\{0,0\.2,0\.4,0\.6,0\.8\\\}, where𝒰0T:=a∈𝒜\|π0T\(a\|x\)=0,∀x\\mathcal\{U\}\_\{0\}^\{T\}:=\{a\\in\\mathcal\{A\}\\,\|\\,\\pi\_\{0\}^\{T\}\(a\\,\|\\,x\)=0,\\forall x\}\(see Figure[10](https://arxiv.org/html/2607.22012#A4.F10)\)\. The results show that COPE consistently outperforms the other estimators without being affected by the presence of new actions\. As in the real\-world data experiment, the estimators using only logged data from the target domain, particularly IPS\(T\) and DR\(T\), produce a large bias as the ratio increases\. Estimators that naively use logged data from all domains, such as IPS\(ALL\), DR\(ALL\), and DM\(ALL\), exhibit consistently high bias due to their inability to handle differences in DGPs\.

Second, we compare the estimators under varying percentages of samples with deterministic logging, as shown in Figure[10](https://arxiv.org/html/2607.22012#A4.F10)\. In the experiment, we calculate the percentages \(20%, 40%, 60%, 80%\) of a particular one\-dimensional context in the target domain\. Then, if the value of the context in the target logged data is smaller than the percentile point, we apply a deterministic logging policy; otherwise, we apply a probabilistic logging policy defined by Eq\. \([16](https://arxiv.org/html/2607.22012#A4.E16)\)\. The results indicate that COPE is robust against an increasing degree of deterministic logging policy and maintains a small bias\. In contrast, IPS\(T\) and DR\(T\) show increasing bias as the percentile of the target domain context grows, similar to the real\-world OPE experiment with varying numbers of users whose logging is deterministic in the target domain\.

Finally, we compare the estimators under varying logged data sizes of the target domain \(nTn^\{T\}\) in Figure[10](https://arxiv.org/html/2607.22012#A4.F10)\. We observe that most existing estimators show a decrease in MSE as the logged data size in the target domain increases, but COPE generally achieves the most accurate estimation\. Notably, whennT=50n^\{T\}=50, COPE outperforms the best baseline DM\(T\) and the second\-best baseline DR\(T\) by a substantial margin\(MSE​\(V^​DM​\(T\)\)MSE​\(V^​COPE\)=2\.46,MSE​\(V^​DR​\(T\)\)MSE​\(V^​COPE\)=3\.29\)\(\\frac\{\\mathrm\{MSE\}\(\\hat\{V\}\{\\rm\{DM\(T\)\}\}\)\}\{\\mathrm\{MSE\}\(\\hat\{V\}\{\\rm\{COPE\}\}\)\}=2\.46,\\frac\{\\mathrm\{MSE\}\(\\hat\{V\}\{\\rm\{DR\(T\)\}\}\)\}\{\\mathrm\{MSE\}\(\\hat\{V\}\{\\rm\{COPE\}\}\)\}=3\.29\)\.

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/legend_ope.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_mse_graph_newaction.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_bias_graph_newaction.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_variance_graph_newaction.png)

Figure 8:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with varying ratios of new actions in the target domain\.![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_mse_graph_deterministic.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_bias_graph_deterministic.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_variance_graph_deterministic.png)

Figure 9:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with varying percentages of samples with a deterministic logging\.![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_mse_graph_samplesize.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_bias_graph_samplesize.png)

![Refer to caption](https://arxiv.org/html/2607.22012v1/figures/syns_variance_graph_samplesize.png)

Figure 10:MSE\(left\), Squared Bias\(center\), and Variance\(right\) with varying logged data sizes in the target domain\.