Tag
This paper investigates how query timing affects positional biases (primacy and recency) in LLMs compared to humans, finding that LLMs exhibit opposite biases depending on when they are queried, with newer models showing exacerbated effects.
The paper tests label-free strategies to reduce positional bias in multiple-choice benchmarks but finds they do not reliably improve accuracy or reduce bias. Only showing all options with an LLM matcher preserves baseline performance, while cyclic permutation sometimes helps.
Introduces LPES, a layer-specific positional embedding scaling method that mitigates the 'lost-in-the-middle' problem in LLMs by assigning distinct scaling factors per layer using a genetic algorithm with Bézier curves, achieving up to 11.2% accuracy gain without fine-tuning or latency increase.
This paper introduces a psychometric datasheet protocol for evaluating LLM judges as measurement instruments, measuring dark current, positional false preference, stable cross-sensitivity, and target sensitivity. A case study on three open-weight models reveals significant differences in judge quality and behavior.
Researchers from Jilin University systematically evaluate positional bias in multi-video summarization using MLLMs, constructing a benchmark from ActivityNet and News videos and assessing nine models with metrics including Coverage, Directional Positional Bias, and Middle-Edge Gap. Results show positional effects are domain- and model-dependent, and increasing visual or generation budget does not uniformly resolve the imbalance.
This paper identifies a 'positional copying' shortcut where small language models answer arithmetic questions by copying the last number before the answer delimiter, bypassing actual reasoning. This effect explains why shuffling CoT steps retains performance; it accounts for 89-92% of teacher-forcing accuracy in 1-3B models on GSM8K.
SDSR proposes lightweight self-describing structured data with dual-layer guidance to exploit LLM primacy bias, achieving 100% routing accuracy without vector DBs.