selection-bias

This paper unifies selection bias and order sensitivity in multiple-choice LLM evaluation by showing the pooled selection-rate standard deviation factorizes into flip rate and positional skew. Across 41 checkpoints and 12 benchmarks, it shows prior correction mainly reduces positional skew while ordering aggregation mainly reduces the flip rate, recommending both be reported.

这篇论文将多选题评测中的选择偏差和顺序敏感性分解为翻转率（flip rate）和位置偏斜（positional skew）两个因素，并发现先验校正主要降低位置偏斜，而顺序聚合主要降低翻转率，建议同时报告这两个因素以区分缓解效果。

This paper shows that chain-of-thought instructions can distort evaluation in vision-language models when answer decoding happens before reasoning generation, leading to significant accuracy drops and selection bias.

This paper analyzes how localizing edits in human corrections can amplify relative selection bias in model training, providing theoretical insights and empirical evidence from machine translation experiments.

This simulation study quantifies selection bias in retail inflation estimation and compares correction methods, finding that stratification generally outperforms inverse probability weighting in long-tail contexts with severe positivity violations.

A detailed discussion on how evaluation sets degrade when used for selection in agentic loops, with three defenses: counting trials with deflated scoring, progressively revealing eval data, and placing the scorer behind an uncallable tool boundary.

This preregistered replication tests whether the monotonicity effect on label agreement in NLI generalizes from selected low-agreement items to unselected populations, finding that the effect reverses and is small, suggesting the earlier finding was conditional on selection.

This paper introduces Heckman-corrected epistemic uncertainty to address selection on unobservables in machine learning, demonstrating that importance weighting fails when selection depends on unobservables correlated with outcomes. The method restores calibration in controlled experiments and real data, outperforming standard UQ baselines.

This paper systematically evaluates reject inference methods in credit scoring and identifies a failure mode where accuracy improves while recall collapses, creating an illusion of improvement while rejection quality deteriorates. It proposes a controlled exploration strategy that breaks the feedback loop and shows that even minimal exploration rates are sufficient to diagnose the problem.