Tag
The article examines a warning in the DeepSeek-V4 paper that retrying interrupted LLM requests introduces length bias, and validates it by generating 100,000 poems, finding that retries can make responses shorter.
The paper analyzes length bias in likelihood-based multiple-choice evaluation, showing that standard and length-normalized accuracy introduce biases toward shorter and longer answers respectively. It proposes Bayesian accuracy, a scoring rule that uses an explicit prior over answer length to remove linear length effects, demonstrating consistently lower empirical bias across benchmarks.