Tag
This paper proposes a novel method to mitigate scoring bias in LLM-as-a-Judge by having LLMs randomly generate numbers to measure their latent numerical bias, then rectifying token generation probabilities accordingly. Experiments across four tasks show the method outperforms baselines and reveals that scoring bias varies across models, tasks, and score ranges.
A technical blog post exploring randomness, Linux entropy, and building a tool called morerandom that uses WASM plugins to feed the system entropy pool.
This paper presents an experiment where GPT-4.1 is asked to pick a random number between 1 and 100, 10,000 times, and the resulting distribution is analyzed for bias compared to a uniform baseline.