We compared different LLMs on IMO 2026 [R]

Reddit r/MachineLearning Papers
llm-benchmark mathematical-ability multi-agent-harness open-source research imo hallucination

Summary

This study evaluates frontier and open-weight LLMs on IMO 2026 problems, demonstrating that specialized harnesses like AutoFyn significantly improve performance of sub-frontier models, though hallucination issues persist on the hardest problem.

There are a few reasons why problems from International Mathematical Olympiad function as a good benchmark for LLMs: - The problems are new, not included in the training data of any model - Hard math problems are quite a good proxy for general intelligence capability - These are complex multi-step tasks that can benefit from orchestration / harness engineering Results: Frontier models (sol and fable) were able to get perfect / nearly perfect score regardless of harness. For both sonnet and opus, the webapp performance was quite poor, improved by provider harness (claude code) and even further improved using AutoFyn, a customizable multi-agent harness we developed. Even with harness, we were not able to match the performance of the frontier models. Open weight model GLM performed roughly at the same level as sonnet without harness, and improved similarly with AutoFyn. Numerical scores are available in the attached paper below. https://preview.redd.it/fy4ayale5nfh1.png?width=2155&format=png&auto=webp&s=040e466171a2f9480d4d6578b34f4d930b41e292 Grading was done by a different frontier model as well as manual verification (we are former IMO medalists, able to sanity check the results). There were cases when the model claimed a false solution (on P3 by sonnet, for example), so hallucination issue still persists in a verifiable domain like math. On the hardest problem: P3's key reduction was missed by every sub-frontier model in every harness, including a 20-hour run that proved everything else and stalled at the identical step. The harness supplied retrieval and verification, not a key idea needed for the solution. Paper: https://github.com/SignalPilot-Labs/AutoFyn/blob/main/results/imo-2026/autofyn-beyond-model-imo26-report.pdf Audit Trails: https://github.com/SignalPilot-Labs/AutoFyn/tree/main/results/imo-2026
Original Article

Similar Articles

Estimating worst case frontier risks of open weight LLMs

OpenAI Blog

OpenAI researchers study worst-case frontier risks of releasing open-weight LLMs through malicious fine-tuning (MFT) in biology and cybersecurity domains, finding that open-weight models underperform frontier closed-weight models and don't substantially advance harmful capabilities.

Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment

arXiv cs.CL

This paper investigates whether standard benchmarks underestimate LLM performance by re-evaluating hallucination detection datasets using an LLM-first, human-adjudicated assessment method. The study finds that incorporating LLM reasoning into the adjudication process improves agreement and suggests that model-assisted re-evaluation yields more reliable benchmarks for ambiguity-prone tasks.