Heckman-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting

# Selection on Unobservables Defeats Importance Weighting
Source: [https://arxiv.org/html/2607.05806](https://arxiv.org/html/2607.05806)
## Heckman\-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting

\(July 2026\)

###### Abstract

Training data for machine\-learning models is routinely collected by a selection process the model never sees: loans are observed only when granted, outcomes only when a test was ordered, regions of input space sampled only when someone chose to sample them\. The field’s default fixes — importance weighting, covariate\-shift correction, MAR imputation — assume selection is ignorable given observables\. Econometrics solved the harder problem in 1979: Heckman’s two\-equation model jointly fits a probit selection equation and an outcome equation linked through correlated errors, and the inverse\-Mills\-ratio term corrects the outcome model for selection on*unobservables*— the case where importance weighting is structurally helpless\. We instantiate that machinery for deep epistemic uncertainty: a deep outcome network, a linear selection head, and a joint bivariate\-normal likelihood over all units \(observed and not\), ensembled for epistemic variance\. Our implementations first reproduce the seven\-digit Stata reference output for both the classic two\-step and the maximum\-likelihood estimator on the RAND Health Insurance data \(max coefficient deviation2\.8×10−62\.8\\times 10^\{\-6\}two\-step,5\.0×10−75\.0\\times 10^\{\-7\}MLE\)\. In a controlled generator where sampling probability depends on an unobservable correlated \(ρ\\rhoswept0→0\.90\\to 0\.9\) with the outcome noise, deep ensembles, MC dropout, and GP baselines are overconfident exactly where data was avoided for reasons correlated with the outcome \(coverage of nominal\-90% intervals falls to 64\.4% atρ=0\.9\\rho=0\.9\), and importance weighting with*oracle*propensities does not fix it \(43\.1%\) — reweighting corrects the covariate distribution, not the conditional bias𝔼​\[y∣x,s=1\]≠𝔼​\[y∣x\]\\mathbb\{E\}\[y\\mid x,s\{=\}1\]\\neq\\mathbb\{E\}\[y\\mid x\]\. The Heckman\-corrected predictive distribution restores coverage \(88\.9%\) when the selection equation contains an*instrument*— a variable affecting selection but not the outcome — and degrades gracefully but measurably without one \(40\.3%\); we sweep this honesty curve rather than hide it\. The joint MLE needs a warm\-up schedule to match the two\-step’s stability with deep feature maps \(otherwise the flexible outcome net absorbs the correction\), a methods finding we characterize\. On real tabular data with induced, documented MNAR selection, the corrected intervals are the best\-calibrated \(lowest region\-ECE\) of every non\-oracle method in selected\-against regions on both datasets; baselines that match its raw coverage do so only by over\-widening everywhere, which region\-ECE exposes\. A discussion vignette fits the same two\-equation model to public benchmark reporting panels \(Papers\-with\-Code tables\), where the no\-instrument pathology our controlled experiments quantify appears in the wild: most fits hit the\|ρ^\|=1\|\\hat\{\\rho\}\|=1boundary\. We state plainly which identification regime a practitioner is in, and release the faithfulness\-gated implementation\.

## 1Introduction

Every published treatment of distribution shift in uncertainty quantification \(UQ\) that we are aware of corrects for selection on*observables*: importance weighting and covariate\-shift adaptation reweight by a propensity that depends onxx\[[23](https://arxiv.org/html/2607.05806#bib.bib23),[24](https://arxiv.org/html/2607.05806#bib.bib24),[4](https://arxiv.org/html/2607.05806#bib.bib4)\], and evaluations of predictive uncertainty under shift perturb the marginalp​\(x\)p\(x\)\[[18](https://arxiv.org/html/2607.05806#bib.bib18)\]\. The harder and, in deployed systems, common case is selection on*unobservables*: the probability that a unit enters the training set depends on latent determinants of the outcome itself\. A bank observes repayment only for granted loans, and the loan officer’s side\-information correlates with default risk; a hospital records disease severity only for admitted patients; a simulation campaign samples parameter regions an expert already believed well\-behaved\. In all these cases𝔼​\[y∣x,s=1\]≠𝔼​\[y∣x\]\\mathbb\{E\}\[y\\mid x,s\{=\}1\]\\neq\\mathbb\{E\}\[y\\mid x\], so*no reweighting of the selected sample — even with exact propensities — can recover the population regression*\. A model trained on such data is not merely data\-poor in the selected\-against regions; it is systematically wrong there, and an epistemic\-uncertainty method that widens intervals with data sparsity alone will be confidently wrong\.

Econometrics has owned this problem sinceHeckman \[[9](https://arxiv.org/html/2607.05806#bib.bib9)\]\(building on[25](https://arxiv.org/html/2607.05806#bib.bib25),[8](https://arxiv.org/html/2607.05806#bib.bib8)\): model selection and outcome jointly,

si\\displaystyle s\_\{i\}=𝟏​\[wi⊤​γ\+ui\>0\],\\displaystyle=\\mathbf\{1\}\\\!\\left\[w\_\{i\}^\{\\top\}\\gamma\+u\_\{i\}\>0\\right\],yi\\displaystyle y\_\{i\}=f​\(xi\)\+εiobserved iff​si=1,\\displaystyle=f\(x\_\{i\}\)\+\\varepsilon\_\{i\}\\quad\\text\{observed iff \}s\_\{i\}=1,\(1\)with\(ui,εi\)\(u\_\{i\},\\varepsilon\_\{i\}\)bivariate normal,Corr​\(ui,εi\)=ρ\\mathrm\{Corr\}\(u\_\{i\},\\varepsilon\_\{i\}\)=\\rho\. The selected\-sample regression then satisfies𝔼​\[y∣x,w,s=1\]=f​\(x\)\+ρ​σ​λ​\(w⊤​γ\)\\mathbb\{E\}\[y\\mid x,w,s\{=\}1\]=f\(x\)\+\\rho\\sigma\\,\\lambda\(w^\{\\top\}\\gamma\)whereλ=ϕ/Φ\\lambda=\\phi/\\Phiis the inverse Mills ratio: the bias term is an explicit, estimable function, and the populationf​\(x\)f\(x\)is recoverable\. This paper instantiates that machinery with deep feature maps and asks whether it repairs epistemic uncertainty where importance weighting cannot\.

### The identification caveat, up front\.

Heckman identification is robust when the selection equation contains an*exclusion restriction*— an instrumentzzthat affects selection but not the outcome\. Without one, identification rides entirely on the bivariate\-normal functional form and is notoriously fragile\[[20](https://arxiv.org/html/2607.05806#bib.bib20),[27](https://arxiv.org/html/2607.05806#bib.bib27),[28](https://arxiv.org/html/2607.05806#bib.bib28)\]\. Every experimental grid in this paper therefore contains instrument\-present*and*instrument\-absent conditions, and we report the degradation curve rather than the best case\. In ML settings natural instruments exist and should be named: which sensor, site, annotator, or batch collected the datum; acquisition\-policy randomness such as an exploration epsilon; rung\- assignment randomness in successive halving \(exploited in the companion paper\[[11](https://arxiv.org/html/2607.05806#bib.bib11)\]\)\. A practitioner without any such variable should read our no\-instrument curves as their expected reality\.

### Contributions\.

1. 1\.Faithfulness\-gated implementation\(§[4](https://arxiv.org/html/2607.05806#S4)\): probit, two\-step, and joint\-MLE Heckman estimators that reproduce the seven\-digit Stata reference output ofCameron and Trivedi \[[3](https://arxiv.org/html/2607.05806#bib.bib3)\]on the RAND HIE data \(max deviation2\.8×10−62\.8\\times 10^\{\-6\}/5\.0×10−75\.0\\times 10^\{\-7\}\) and statsmodels’ probit to10−810^\{\-8\}; pre\-registered as a gate before any experiment\.
2. 2\.Deep Heckman UQ\(§[3](https://arxiv.org/html/2607.05806#S3)\): a deep outcome net with a linear selection head trained by the joint bivariate\-normal likelihood over all units, plus a two\-step variant \(probit, then an inverse\-Mills feature\); ensembles of either give a selection\-corrected predictive distribution\.
3. 3\.The controlled demonstration\(§[5](https://arxiv.org/html/2607.05806#S5)\): under selection on unobservables, oracle\-propensity importance weighting fails where the Heckman correction succeeds — coverage 43\.1% vs 88\.9% atρ=0\.9\\rho=0\.9\(nominal 90%\) in selected\-against regions — and the pre\-registered kill condition \(IW matching Heckman would falsify the premise\) does not fire\. Without the instrument the correction degrades to 40\.3%; we quantify the honesty curve inρ\\rho\.
4. 4\.A methods finding\(§[5](https://arxiv.org/html/2607.05806#S5)\): with deep feature maps the two\-step estimator is markedly more stable than the joint MLE, which requires a warm\-up schedule to avoid absorbing the correction term into the flexible outcome function\.
5. 5\.Semi\-real validation and a real\-world vignette\(§[6](https://arxiv.org/html/2607.05806#S6)–[7](https://arxiv.org/html/2607.05806#S7)\): induced, fully documented MNAR selection on real tabular tasks where the corrected intervals are the best\-calibrated \(lowest region\-ECE\) non\-oracle method in selected\-against regions, and where we show raw coverage misleads; and a Heckman analysis of public benchmark reporting panels in which the no\-instrument boundary pathology \(\|ρ^\|=1\|\\hat\{\\rho\}\|=1in 8 of 13 fits\) appears exactly as the controlled experiments predict\.

## 2Background and related work

### Heckman selection models\.

The two\-step estimator\[[9](https://arxiv.org/html/2607.05806#bib.bib9)\]fits a probit forP​\(s=1∣w\)=Φ​\(w⊤​γ\)P\(s\{=\}1\\mid w\)=\\Phi\(w^\{\\top\}\\gamma\), then regressesyyon\[x,λ​\(w⊤​γ^\)\]\[x,\\lambda\(w^\{\\top\}\\hat\{\\gamma\}\)\]over the selected sample;σ\\sigmaandρ\\rhoare recovered from the residual variance and the Mills coefficientβλ=ρ​σ\\beta\_\{\\lambda\}=\\rho\\sigma\. The joint MLE maximizes the bivariate\-normal likelihood

ℓ=∑si=0log⁡Φ​\(−wi⊤​γ\)\+∑si=1\[log⁡ϕσ​\(yi−f​\(xi\)\)\+log⁡Φ​\(wi⊤​γ\+ρ​ei1−ρ2\)\],ei=yi−f​\(xi\)σ\.\\ell=\\sum\_\{s\_\{i\}=0\}\\log\\Phi\(\-w\_\{i\}^\{\\top\}\\gamma\)\+\\sum\_\{s\_\{i\}=1\}\\left\[\\log\\phi\_\{\\sigma\}\(y\_\{i\}\-f\(x\_\{i\}\)\)\+\\log\\Phi\\\!\\left\(\\tfrac\{w\_\{i\}^\{\\top\}\\gamma\+\\rho e\_\{i\}\}\{\\sqrt\{1\-\\rho^\{2\}\}\}\\right\)\\right\],\\quad e\_\{i\}=\\tfrac\{y\_\{i\}\-f\(x\_\{i\}\)\}\{\\sigma\}\.\(2\)Surveys and critiques:Vella \[[27](https://arxiv.org/html/2607.05806#bib.bib27)\], Puhani \[[20](https://arxiv.org/html/2607.05806#bib.bib20)\], Winship and Mare \[[28](https://arxiv.org/html/2607.05806#bib.bib28)\]\. The selection\-on\-observables special case \(ρ=0\\rho=0\) reduces to missing at random\[[22](https://arxiv.org/html/2607.05806#bib.bib22),[15](https://arxiv.org/html/2607.05806#bib.bib15)\]\.

### Covariate shift and importance weighting\.

When selection depends only on observables, reweighting by inverse propensities is consistent\[[23](https://arxiv.org/html/2607.05806#bib.bib23),[24](https://arxiv.org/html/2607.05806#bib.bib24)\];Cortes et al\. \[[4](https://arxiv.org/html/2607.05806#bib.bib4)\]give the sample\-selection\-bias learning theory in exactly this importance\-weighting framework\. All of it conditions on ignorability givenxx; none of it addressesρ≠0\\rho\\neq 0\.

### Selective labels and PU learning\.

The selective\-labels literature\[[13](https://arxiv.org/html/2607.05806#bib.bib13)\]shows that decisions hide outcomes and proposes contraction\-based*evaluation*; it diagnoses the problem rather than repairing the predictive distribution\. Positive\-unlabeled learning\[[6](https://arxiv.org/html/2607.05806#bib.bib6),[2](https://arxiv.org/html/2607.05806#bib.bib2)\]concerns missing labels for a binary class under \(usually\) selected\-completely\-at\-random assumptions, not the MNAR sampling of\(x,y\)\(x,y\)pairs\.

### Flexible\-ML selection models\.

Closest in spirit is a recent line that fits Heckman/Tobit selection models with flexible ML rather than linear equations:O’Neill \[[17](https://arxiv.org/html/2607.05806#bib.bib17)\]give a Type\-2 Tobit sample\- selection model with Bayesian additive regression trees, a Dirichlet\- process\-mixture error model that relaxes bivariate normality, and individual credible intervals\. It is the nearest prior work and we distinguish it precisely: it is a tree ensemble \(not a deep net\), targets econometric estimation and treatment effects \(not epistemic uncertainty under distribution shift\), and does not study the selected\-vs\-well\-sampled region split or the importance\-weighting failure that is our headline\. Our contribution is orthogonal — selection\-corrected*epistemic UQ*with deep feature maps — and complementary: their non\-normal error model is a natural way to relax our Gaussian assumption \(§[6](https://arxiv.org/html/2607.05806#S6)\)\.

### Deep uncertainty\.

Deep ensembles\[[14](https://arxiv.org/html/2607.05806#bib.bib14)\], MC dropout\[[7](https://arxiv.org/html/2607.05806#bib.bib7)\], GPs\[[21](https://arxiv.org/html/2607.05806#bib.bib21)\], and quantile\-calibration metrics\[[12](https://arxiv.org/html/2607.05806#bib.bib12)\]are the baselines and measurement tools we use;Ovadia et al\. \[[18](https://arxiv.org/html/2607.05806#bib.bib18)\]evaluate them under covariate shift\. To our knowledge — after a kill\-test against a 3M\-abstract index and an adversarial web sweep \(2026\-07\-04, re\-swept before submission; §[10](https://arxiv.org/html/2607.05806#S10)\) — no deep\-learning instantiation of selection\-corrected epistemic UQ exists; the nearest lines are those above plus the flexible\-ML Tobit work just cited, and we claim exactly the gap between them:*when the sampling of training points is correlated with unobserved determinants of the outcome, importance weighting cannot fix the bias even with oracle propensities; a joint selection/outcome error model can, given an instrument\.*

## 3Method: Heckman\-corrected predictive distributions

### Architecture\.

The outcome functionfθf\_\{\\theta\}is a deep MLP; the selection indexgψ​\(w\)=w⊤​γ\+bg\_\{\\psi\}\(w\)=w^\{\\top\}\\gamma\+bis linear inw=\[x,z\]w=\[x,z\]\(a deliberately shallow selection head; the instrumentzzenters here and only here\)\. Two global parameters\(log⁡σ,atanh​ρ\)\(\\log\\sigma,\\mathrm\{atanh\}\\,\\rho\)complete the model\.

### Joint MLE\.

We minimize−ℓ\-\\ellof Eq\. \([2](https://arxiv.org/html/2607.05806#S2.E2)\) over all units — unselected units contributelog⁡Φ​\(−gψ\)\\log\\Phi\(\-g\_\{\\psi\}\)— with Adam\. With a flexiblefθf\_\{\\theta\}the correction term is at risk of being absorbed into the outcome function early in training \(the likelihood is then locally flat inρ\\rho\); freezingρ=0\\rho=0for a short warm\-up \(100 of 1500 epochs\), so thatfθf\_\{\\theta\}first fits the selected\-sample conditional mean, resolves this: post\-warm\-up, thezz\-variation of the Mills term separates the correction fromfθf\_\{\\theta\}\. This is the pre\-registered methods contingency \(§[5](https://arxiv.org/html/2607.05806#S5)\); without warm\-up tuning,ρ^\\hat\{\\rho\}collapses toward0\.

### Two\-step\.

A classic probit onwwover all units givesγ^\\hat\{\\gamma\}; the network then fitsy≈fθ​\(x\)\+c​λ​\(w⊤​γ^\)y\\approx f\_\{\\theta\}\(x\)\+c\\,\\lambda\(w^\{\\top\}\\hat\{\\gamma\}\)on selected units by Gaussian NLL withcca free scalar, andσ\\sigmais recovered by the classic residual correction\. The population prediction isfθ​\(x\)f\_\{\\theta\}\(x\)alone\.

### Corrected predictive distribution\.

An ensemble ofK=5K=5independently initialized models gives

y^​\(x\)=1K​∑kfθk​\(x\),Var^​\(x\)=Vark​\[fθk​\(x\)\]⏟epistemic\+1K​∑kσ^k2⏟aleatoric,\\hat\{y\}\(x\)=\\tfrac\{1\}\{K\}\\sum\_\{k\}f\_\{\\theta\_\{k\}\}\(x\),\\qquad\\widehat\{\\mathrm\{Var\}\}\(x\)=\\underbrace\{\\mathrm\{Var\}\_\{k\}\[f\_\{\\theta\_\{k\}\}\(x\)\]\}\_\{\\text\{epistemic\}\}\+\\underbrace\{\\tfrac\{1\}\{K\}\\sum\_\{k\}\\hat\{\\sigma\}\_\{k\}^\{2\}\}\_\{\\text\{aleatoric\}\},\(3\)i\.e\., exactly the deep\-ensemble recipe with the population function in place of the selected\-sample one\. Baselines \(§[5](https://arxiv.org/html/2607.05806#S5)\) get the same ensemble treatment\.

## 4A\-E0: the faithfulness gate

House rule: no experiment runs until the estimators reproduce classic econometric results\. On theCameron and Trivedi \[[3](https://arxiv.org/html/2607.05806#bib.bib3)\]RAND HIE specification \(n=5574n=5574, 4281 selected; no exclusion restriction — their example of identification by functional form\), our two\-step matches their published seven\-digit Stata output with maximum absolute deviation2\.8×10−62\.8\\times 10^\{\-6\}across all coefficients,ρ\\rho,σ\\sigma, andβλ\\beta\_\{\\lambda\}; our MLE matches with maximum deviation5\.0×10−75\.0\\times 10^\{\-7\}\(log\-likelihood difference 0\.000;ρ^=0\.7356\\hat\{\\rho\}=0\.7356,σ^=1\.5701\\hat\{\\sigma\}=1\.5701\)\. The probit head agrees with statsmodels to10−810^\{\-8\}; a statsmodels\-composed two\-step on theMroz \[[16](https://arxiv.org/html/2607.05806#bib.bib16)\]data \(Greene’s example 22\.8 spec\) is matched to10−810^\{\-8\}; and on synthetic data with known truth \(n=105n=10^\{5\},ρ=0\.6\\rho=0\.6\) both estimators recover\(β,ρ,σ\)\(\\beta,\\rho,\\sigma\)while naive OLS on the selected sample does not, with the correctρ=0\\rho=0MAR control\. Eleven tests, all green \(tests/\)\.

## 5A\-E1: controlled demonstration

### Generator\.

x∼U​\[−3,3\]x\\sim U\[\-3,3\]; instrumentz∼N​\(0,1\)z\\sim N\(0,1\);\(ε,u\)\(\\varepsilon,u\)bivariate normal withCorr=ρ\\mathrm\{Corr\}=\\rho,σ=0\.5\\sigma=0\.5;y=sin⁡\(1\.5​x\)\+0\.5​x\+εy=\\sin\(1\.5x\)\+0\.5x\+\\varepsilon; selection indexg0​\(x\)\+α​z\+ug\_\{0\}\(x\)\+\\alpha z\+uwithg0g\_\{0\}decreasing inxx, so high\-xxregions are smoothly selected against \(P​\(s=1∣x\)P\(s\{=\}1\\mid x\)spans≈0\.05\\approx 0\.05–0\.950\.95; overall 50% selected\)\. Grid:ρ∈\{0,0\.3,0\.6,0\.9\}×α∈\{0,1\}×8\\rho\\in\\\{0,0\.3,0\.6,0\.9\\\}\\times\\alpha\\in\\\{0,1\\\}\\times 8seeds;n=2000n=2000training pool; metrics on a fresh uniform test set against population draws, split by region \(P​\(s=1∣x\)≤0\.3P\(s\{=\}1\\mid x\)\\leq 0\.3is “selected\-against”\)\. Methods: deep ensembles\[[14](https://arxiv.org/html/2607.05806#bib.bib14)\], MC dropout\[[7](https://arxiv.org/html/2607.05806#bib.bib7)\], an RBF GP\[[21](https://arxiv.org/html/2607.05806#bib.bib21)\], an importance\-weighted ensemble with*oracle*propensitiesP​\(s=1∣x,z\)P\(s\{=\}1\\mid x,z\)from the generator, a selection\-blind two\-head ablation \(the Heckman architecture withρ≡0\\rho\\equiv 0: is it the joint error model or just the extra head?\), and the two Heckman ensembles\. Figure[1](https://arxiv.org/html/2607.05806#S5.F1)shows the setup and the headline curves; Figure[2](https://arxiv.org/html/2607.05806#S5.F2)the instrument\-present/absent contrast\.

![Refer to caption](https://arxiv.org/html/2607.05806v1/x1.png)Figure 1:\(a\) One draw atρ=0\.9\\rho=0\.9: baselines fit the selected sample faithfully and are therefore biased with tight intervals in the selected\-against region \(shaded\); the Heckman ensemble recoversf0f\_\{0\}\. \(b\) Coverage of nominal\-90% intervals in selected\-against regions vsρ\\rho\(instrument present, 8 seeds, mean±\\pmsd\)\.![Refer to caption](https://arxiv.org/html/2607.05806v1/x2.png)Figure 2:The honesty curves\. \(a\) With instrument \(α=1\\alpha=1\) coverage in selected\-against regions is restored by the Heckman ensembles\. \(b\) Without the instrument \(α=0\\alpha=0\) identification rides on functional form; the MLE’s coverage looks deceptively high but only because weak identification inflates its intervals — panel \(c\) shows the mean is still badly biased\. \(c\) Bias of the predictive mean against the truef0f\_\{0\}in selected\-against regions \(solid: with instrument; dashed: without\): with the instrument the Heckman bias is≈0\\approx 0; without it, all methods including Heckman stay biased\.
### \(i\) Baselines are miscalibrated where it matters\.

Atρ=0\.9\\rho=0\.9with instrument present, deep\-ensemble coverage in selected\-against regions is 64\.4% \(nominal 90%;ρ=0\.6\\rho=0\.6: 77\.8%\), with predictive\-mean bias 0\.441 inyy\-units \(σ=0\.5\\sigma=0\.5\) — all members converge to the same biased𝔼​\[y∣x,s=1\]\\mathbb\{E\}\[y\\mid x,s\{=\}1\], so ensemble spread cannot signal the error\. MC dropout and the GP behave alike \(Fig\.[1](https://arxiv.org/html/2607.05806#S5.F1)b\)\. Atρ=0\\rho=0\(MAR control\) everything is fine — the failure is specifically selection on unobservables, not sparsity\.

### \(ii\) Oracle importance weighting does not fix it\.

The IW ensemble receives the exact generator propensitiesP​\(s=1∣x,z\)P\(s\{=\}1\\mid x,z\)and still covers only 43\.1% \(bias 0\.678\) — worse than the unweighted ensemble, since upweighting the sparse selected\-against points concentrates training on units whose noise is most strongly selected\. This is the pre\-registered kill condition and it does not fire: IW with oracle propensities does not match the Heckman correction under selection on unobservables \(paired Wilcoxon on selected\-against coverage,p=0\.008p=0\.008,n=8n=8seeds; the Wilcoxon floor atn=8n=8isp=0\.008p=0\.008\)\.

### \(iii\) The Heckman correction does\.

With the instrument, the two\-step ensemble covers 88\.9% atρ=0\.9\\rho=0\.9\(88\.8% at0\.60\.6\) with bias 0\.002 — an order of magnitude better calibrated than the baselines in region\-ECE \(0\.036 vs the deep ensemble’s 0\.266, approaching the oracle’s 0\.012\)\. The joint\-MLE ensemble recovers the correlation \(ρ^=0\.90±0\.02\\hat\{\\rho\}=0\.90\\pm 0\.02at true0\.90\.9\) and covers 89\.1%, matching the two\-step here\. The blind two\-head ablation stays at 69\.3% — the gain is the joint error model, not the extra head\.

### \(iv\) Without the instrument\.

Atα=0\\alpha=0the exclusion restriction is gone and identification rides entirely on the bivariate\-normal form\. Point estimates stay badly biased for every method, Heckman included: atρ=0\.9\\rho=0\.9the MLE’s selected\-against mean bias is 0\.582 and the two\-step’s 0\.778, versus 0\.002*with*the instrument, andρ^\\hat\{\\rho\}collapses to 0\.14 \(true0\.90\.9\)\. The interval coverage tells a deceptive\-looking story that we flag rather than exploit: the MLE’s selected\-against coverage is*higher*without the instrument \(84\.7%\) than the two\-step’s \(40\.3%\), but only because weak identification scatters its ensemble members and inflates the predictive variance — it covers by being*uncertain*, not by being right, as the large residual bias shows\. The two\-step, which lacks that variance cushion, exposes the damage directly \(40\.3% vs 28\.8% for the ensemble\)\. The lesson is the classic one\[[20](https://arxiv.org/html/2607.05806#bib.bib20)\], sharpened: without an instrument, neither the point estimate nor the coverage should be trusted, and coverage can actively mislead\. Practitioners should locate themselves on Figure[2](https://arxiv.org/html/2607.05806#S5.F2)before trusting any correction\.

### Methods finding: two\-step vs joint MLE\.

The joint MLE with deep feature maps is stable*only with the warm\-up schedule*: frozenρ=0\\rho=0for the first 100 epochs, the outcome net first fits the selected\-sample mean and the Mills term then separates cleanly, givingρ^=0\.90±0\.02\\hat\{\\rho\}=0\.90\\pm 0\.02\(true0\.90\.9\) and coverage 89\.1% that matches the two\-step\. Without warm\-up,fθf\_\{\\theta\}absorbs the correction andρ^\\hat\{\\rho\}collapses toward0\(we observedρ^→0\.16\\hat\{\\rho\}\\to 0\.16at true0\.80\.8in tuning\)\. The two\-step needs no such schedule — its hard separation of the probit fit from the outcome fit is inherently robust — so we recommend it as the default and report both throughout\.

## 6A\-E2/E3: semi\-real tabular MNAR

We induce MNAR selection on two real regression datasets — California housing\[[19](https://arxiv.org/html/2607.05806#bib.bib19)\]and UCI wine quality\[[5](https://arxiv.org/html/2607.05806#bib.bib5)\]— by a fully documented rule \(code inheckesel/synth\.py\): the selection index combines the standardized linear prediction ofyy\(so high\-predicted regions are selected against, loan\-style\), the*residual*of that fit scaled byρ\\rho\(the unobservable\), an instrumentz∼N​\(0,1\)z\\sim N\(0,1\), and a threshold set for 50% selection\. Because we induce the selection, ground truth exists for all rows; held\-out test rows are never selected on\. Grid:ρ∈\{0,0\.5,0\.8\}\\rho\\in\\\{0,0\.5,0\.8\\\}; 8 seeds at the headlineρ=0\.8\\rho=0\.8, 3 elsewhere \(house rule\); the same seven methods plus a*skyline*ensemble trained on the complete pre\-selection pool \(an oracle upper reference\)\.

![Refer to caption](https://arxiv.org/html/2607.05806v1/x3.png)Figure 3:Semi\-real MNAR \(ρ=0\.8\\rho=0\.8, mean±\\pmsd over 8 seeds\)\. \(a\) Region\-ECE in selected\-against regions — the calibration metric — where the two\-step Heckman is the best non\-oracle method on both datasets\. \(b\) Selected\-against coverage is muddier: conservative baselines can match it\. \(c\) But those baselines get there by over\-widening*everywhere*— the GP covers 98\.0% of a nominal\-90% interval in well\-sampled California regions — which region\-ECE penalizes and coverage alone does not\.Real heteroscedastic data makes the coverage metric treacherous, so we lead with region\-ECE \(calibration error by region\), the metric that cannot be gamed by global over\-widening\. Atρ=0\.8\\rho=0\.8the corrected two\-step ensemble has the lowest selected\-against ECE of every non\-oracle method on*both*datasets — 0\.051 \(California\) and 0\.138 \(wine\), versus the deep ensemble’s 0\.201 / 0\.223, oracle\-IW’s 0\.235 / 0\.252, MC dropout’s 0\.175 / 0\.200, and the GP’s 0\.114 / 0\.164 — approaching the skyline’s 0\.026 / 0\.049 \(paired one\-sided Wilcoxon, two\-step below each of deep\-ensemble, IW, and dropout, pooled over datasets:p<0\.001p<0\.001in all three\)\. By raw*coverage*the picture is muddier \(Fig\.[3](https://arxiv.org/html/2607.05806#S6.F3)b\): MC dropout and the GP sometimes match or exceed the two\-step’s selected\-against coverage \(77\.5% / 64\.1%\) — but they do so by widening intervals indiscriminately, over\-covering the*well\-sampled*regions \(98\.0% / 93\.2% on California at nominal 90%; Fig\.[3](https://arxiv.org/html/2607.05806#S6.F3)c\) rather than by locating and correcting the selection bias\. Oracle\-IW remains the worst\-calibrated method in selected\-against regions on both datasets, as the theory predicts\. The correction is not free of cost — even the skyline undercovers wine \(non\-Gaussian noise\), and no method reaches nominal coverage on these hard tasks — but it is the only one that targets the selection bias directly rather than blunting all its intervals\. Atρ=0\\rho=0the selected\-against coverages coincide \(73\.9% vs 71\.8% on California\), so the correction costs nothing under MAR\. As a naturally selected companion, the RAND HIE fit of §[4](https://arxiv.org/html/2607.05806#S4)estimatesρ^=0\.7356\\hat\{\\rho\}=0\.7356on real medical\-expenditure data — with the caveat, stated byCameron and Trivedi \[[3](https://arxiv.org/html/2607.05806#bib.bib3)\]themselves, that without an instrument this rests on functional form; we make no ground\-truth claim there\.

## 7A\-E4: benchmark reporting panels \(a vignette\)

Public model×\\timestask benchmark matrices are missing\-not\-at\-random panels: a paper reports its method on a chosen subset of standard datasets\. On the frozen Papers\-with\-Code evaluation dump\[[1](https://arxiv.org/html/2607.05806#bib.bib1)\]we build, per task, a panel over the eight most\-reported datasets \(modal accuracy\-like metric per dataset,zz\-scored among reported values\) and fit the classic Heckman MLE: outcome==cellzz\-score on a leave\-one\-out ability proxy and dataset dummies; selection==reported indicator on the same covariates\. There is*no*exclusion restriction here — precisely the regime our controlled experiments warn about — and the fits behave accordingly: 8 of 13 panel and pairwise fits hit the\|ρ^\|=1\|\\hat\{\\rho\}\|=1boundary, the textbook no\-instrument pathology\. The interior estimates are suggestive — ImageNet→\\toImageNet\-ReaL reporting selects positively on unobservables \(ρ^=0\.87±0\.09\\hat\{\\rho\}=0\.87\\pm 0\.09\) — but sign\-unstable across panels \(machine translation:−0\.90\-0\.90\)\. We present this as a methodological caution, not an estimate:*benchmark\-panel corrections need a reporting instrument*\(venue page limits and dataset release dates are candidates\), and until then, leaderboard aggregations inherit an unidentified selection bias\.

![Refer to caption](https://arxiv.org/html/2607.05806v1/x4.png)Figure 4:Heckmanρ^\\hat\{\\rho\}on Papers\-with\-Code reporting panels \(anchor→\\totarget pairs\)\. Squares: fits on the\|ρ^\|=1\|\\hat\{\\rho\}\|=1boundary\. Without an instrument, most fits degenerate — the real\-world face of Figure[2](https://arxiv.org/html/2607.05806#S5.F2)b\.
## 8What we do not claim

We make no causal\-discovery claims: the correction assumes the two\-equation structure, it does not find it\. We make no real\-MNAR ground\-truth claims beyond the induced\-selection experiments; on RAND HIE and the reporting panels the truth is unknowable without instruments, and we say so\. We do not claim the bivariate\-normal error model is correct for any real dataset — only that, where its identification conditions hold even approximately, it beats reweighting approaches that are structurally unable to address the problem\. We do not claim superiority under selection on observables \(ρ=0\\rho=0\), where importance weighting is consistent and simpler\.

## 9Limitations

The outcome noise is modeled as homoscedastic Gaussian; real data \(§[6](https://arxiv.org/html/2607.05806#S6)\) already shows the cost, as even the skyline undercovers\. Heteroscedastic and non\-Gaussian extensions \(e\.g\.,tt\-copula errors\) are natural next steps\. The selection head is linear; a nonlinear selection index with a valid instrument is compatible with the likelihood but untested here\. Ensembles of size 5 approximate epistemic variance coarsely\. The vignette panels aggregate heterogeneous metrics and papers; they are illustrative only\.

## 10Reproducibility

All code, tests, committed result JSONs, and this manuscript’s auto\-generated numbers \(numbers\.tex; regenerate\-and\-diff passes at submission\) are in the repository; every number in the paper is machine\-generated fromresults/\*\.json\(discipline inherited from[10](https://arxiv.org/html/2607.05806#bib.bib10)\)\. Experiments run on one RTX 3080 \(10 GB\) in under six GPU\-hours total\. Datasets: RandHIE and Mroz87 from the sampleSelection R package’s public mirrors\[[26](https://arxiv.org/html/2607.05806#bib.bib26)\]; the Cameron–Trivedi Stata reference output is committed alongside\. Archived release:[DOI: 10\.5281/zenodo\.21210466](https://doi.org/10.5281/zenodo.21210466)\. Pre\-submission recency sweep logged inPLAN\.md\.

## References

- pwc \[2021\]Papers with code: Evaluation tables \(frozen public snapshot\)\.[https://github\.com/paperswithcode/paperswithcode\-data](https://github.com/paperswithcode/paperswithcode-data); mirrored at[https://huggingface\.co/datasets/felixleungsc/paperswithcode\-data\-evaluation\-tables](https://huggingface.co/datasets/felixleungsc/paperswithcode-data-evaluation-tables), 2021\.Accessed 2026\-07\-05; Papers with Code was sunset in 2025, the snapshot is the last public dump\.
- Bekker and Davis \[2020\]Jessa Bekker and Jesse Davis\.Learning from positive and unlabeled data: A survey\.*Machine Learning*, 109:719–760, 2020\.
- Cameron and Trivedi \[2005\]A\. Colin Cameron and Pravin K\. Trivedi\.*Microeconometrics: Methods and Applications*\.Cambridge University Press, 2005\.
- Cortes et al\. \[2008\]Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh\.Sample selection bias correction theory\.In*International Conference on Algorithmic Learning Theory \(ALT\)*, pages 38–53, 2008\.
- Cortez et al\. \[2009\]Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis\.Modeling wine preferences by data mining from physicochemical properties\.*Decision Support Systems*, 47\(4\):547–553, 2009\.
- Elkan and Noto \[2008\]Charles Elkan and Keith Noto\.Learning classifiers from only positive and unlabeled data\.In*Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pages 213–220, 2008\.
- Gal and Ghahramani \[2016\]Yarin Gal and Zoubin Ghahramani\.Dropout as a Bayesian approximation: Representing model uncertainty in deep learning\.In*International Conference on Machine Learning*, pages 1050–1059, 2016\.
- Heckman \[1974\]James J\. Heckman\.Shadow prices, market wages, and labor supply\.*Econometrica*, 42\(4\):679–694, 1974\.
- Heckman \[1979\]James J\. Heckman\.Sample selection bias as a specification error\.*Econometrica*, 47\(1\):153–161, 1979\.
- Howe \[2026a\]Gunner Levi Howe\.Rice’s formula for event\-driven networks: Predicting and training temporal sparsity via differentiable level\-crossing rates\.*Zenodo preprint*, 2026a\.doi:10\.5281/zenodo\.21205335\.
- Howe \[2026b\]Gunner Levi Howe\.Survivor bias in learning\-curve surrogates: Successive halving selects on noise, 2026b\.Companion paper \(paper2/ of this program\)\.
- Kuleshov et al\. \[2018\]Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon\.Accurate uncertainties for deep learning using calibrated regression\.In*International Conference on Machine Learning*, pages 2796–2804, 2018\.
- Lakkaraju et al\. \[2017\]Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan\.The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables\.In*Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pages 275–284, 2017\.
- Lakshminarayanan et al\. \[2017\]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell\.Simple and scalable predictive uncertainty estimation using deep ensembles\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Little and Rubin \[2002\]Roderick J\. A\. Little and Donald B\. Rubin\.*Statistical Analysis with Missing Data*\.Wiley, 2nd edition, 2002\.
- Mroz \[1987\]Thomas A\. Mroz\.The sensitivity of an empirical model of married women’s hours of work to economic and statistical assumptions\.*Econometrica*, 55\(4\):765–799, 1987\.
- O’Neill \[2025\]Eoghan O’Neill\.Type 2 Tobit sample selection models with Bayesian additive regression trees\.*arXiv preprint arXiv:2502\.03600*, 2025\.
- Ovadia et al\. \[2019\]Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D\. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek\.Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.
- Pace and Barry \[1997\]R\. Kelley Pace and Ronald Barry\.Sparse spatial autoregressions\.*Statistics & Probability Letters*, 33\(3\):291–297, 1997\.
- Puhani \[2000\]Patrick Puhani\.The Heckman correction for sample selection and its critique\.*Journal of Economic Surveys*, 14\(1\):53–68, 2000\.
- Rasmussen and Williams \[2006\]Carl Edward Rasmussen and Christopher K\. I\. Williams\.*Gaussian Processes for Machine Learning*\.MIT Press, 2006\.
- Rubin \[1976\]Donald B\. Rubin\.Inference and missing data\.*Biometrika*, 63\(3\):581–592, 1976\.
- Shimodaira \[2000\]Hidetoshi Shimodaira\.Improving predictive inference under covariate shift by weighting the log\-likelihood function\.*Journal of Statistical Planning and Inference*, 90\(2\):227–244, 2000\.
- Sugiyama et al\. \[2007\]Masashi Sugiyama, Matthias Krauledat, and Klaus\-Robert Müller\.Covariate shift adaptation by importance weighted cross validation\.*Journal of Machine Learning Research*, 8:985–1005, 2007\.
- Tobin \[1958\]James Tobin\.Estimation of relationships for limited dependent variables\.*Econometrica*, 26\(1\):24–36, 1958\.
- Toomet and Henningsen \[2008\]Ott Toomet and Arne Henningsen\.Sample selection models in R: Package sampleSelection\.*Journal of Statistical Software*, 27\(7\), 2008\.
- Vella \[1998\]Francis Vella\.Estimating models with sample selection bias: A survey\.*Journal of Human Resources*, 33\(1\):127–169, 1998\.
- Winship and Mare \[1992\]Christopher Winship and Robert D\. Mare\.Models for sample selection bias\.*Annual Review of Sociology*, 18:327–350, 1992\.