model-bias

#model-bias

When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

arXiv cs.CL ↗ · 2026-08-21 Cached

This paper studies how irrelevant text context biases predictions in multimodal large language models, showing that context-induced decision margins follow an affine transformation of context-free margins, offering insights into model sensitivity.

0 favorites 0 likes

#model-bias

I had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0.

Reddit r/LocalLLaMA ↗ · 2026-06-28

An open evaluation setup with 55 LLMs blind-grading each other reveals statistically significant same-family rating bias across 8 model families, with Mistral penalizing its own models most severely. The study highlights issues with aggregate leaderboards and proposes improvements like within-response mixed-effects models.

0 favorites 0 likes

model-bias

When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

I had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0.

Submit Feedback