Inconsistencies in Unsloth's LiveCodeBench Results

I compared two published benchmark charts for DeepSeek-V4.1-Flash-1017 and noticed something rather unusual. All four Unsloth quantizations AND the Q8 baseline simultaneously improved by approximately 2–3 percentage points on LiveCodeBench v6. Meanwhile, the GSQ-RCO results changed much less consistently. 1. The numbers Model First chart Updated chart Change Unsloth IQ1_S 88.95% 91.0% +2.05 Unsloth IQ1_M 88.76% 91.2% +2.44 Unsloth IQ2_XXS 88.19% 91.3% +3.11 Unsloth Q2_K_XL 89.33% 91.7% +2.37 Q8 baseline 88.95% 91.7% +2.75 Every single Unsloth model improved, including the 8-bit baseline. The first chart's percentages correspond exactly to 525 outcomes (175 problems × 3 generations). For example, 467/525 = 88.95%. Unsloth's updated methodology reports 12 generations per problem, corresponding to 2,100 outcomes per model. If the updated results include those original three generations, the remaining nine must have achieved approximately: Model Initial 3 Additional 9 Improvement IQ1_S 88.95% 91.62–91.75% +2.67–2.79 IQ1_M 88.76% 92.00–92.06% +3.24–3.30 IQ2_XXS 88.19% 92.32–92.38% +4.13–4.19 Q2_K_XL 89.33% 92.44–92.51% +3.11–3.17 Q8 baseline 88.95% 92.57–92.63% +3.62–3.68 The ranges account for the updated chart rounding scores to one decimal place. 2. Statistical probability Assume each generation is an independent Bernoulli trial, the true success probability is unchanged, and the initial 525 outcomes are included in the final 2,100. Using a one-sided hypergeometric test, the probability of observing an initial result at least this low for each model is approximately: Model Probability IQ1_S 3.38–4.14% IQ1_M 1.44–1.62% IQ2_XXS 0.262–0.304% Q2_K_XL 1.57–1.77% Q8 baseline 0.621–0.713% If the models' fluctuations are also independent, multiplying these probabilities produces: P ≈ 1.24 × 10⁻¹⁰ to 2.59 × 10⁻¹⁰ That's approximately 1 in 3.9–8.0 billion under the simplified independent-sampling model. Of course, the models are correlated and evaluated on the same tasks, so this should not be interpreted as the actual probability without the raw data. Also, these calculations assume the original outcomes are included in the updated averages. Nevertheless, the synchronized improvements are striking. 3. GSQ-RCO behaved differently Model First chart Updated chart Change GSQ-RCO IQ1_M 88.0% 87.6% −0.4 GSQ-RCO IQ2_XXS 88.2% 88.9% +0.7 GSQ-RCO IQ2_XS 88.6% 90.5% +1.9 Unlike Unsloth, these results do not exhibit a consistent increase. According to this comment from the Unsloth team, the earlier chart represents just one run, followed by additional evaluations and the evaluation settings remained unchanged. However, the percentages in that first chart correspond exactly to 525 outcomes, suggesting three generations per problem. Both original benchmark charts attached for comparison.