Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3

Hugging Face Daily Papers Papers
inference-optimization reasoning model-scaling majority-voting imo-problems

Summary

This paper analyzes inference-time optimization techniques for AIMO 3, finding that model capability dominates over prompt engineering and diverse sampling strategies. The study reveals that high-temperature sampling already decorrelates errors maximally, leaving no room for prompt-based improvements, and identifies a 6-point selection loss gap between individual model pass@20 and majority voting consensus.

Majority voting over multiple LLM attempts improves mathematical reasoning, but correlated errors limit the effective sample size. A natural fix is to assign different reasoning strategies to different voters. The approach, Diverse Prompt Mixer, is tested on the AIMO 3 competition: 3 models, 23+ experiments, 50 IMO-level problems, one H100 80 GB, 5-hour limit. Every prompt-level intervention fails. High-temperature sampling already decorrelates errors; weaker strategies reduce accuracy more than they reduce correlation. Across an 8-point capability gap at equal N=8 and every optimization tested, model capability dominates. The gap between the best majority-vote score (42/50) and pass@20 (~45.5) is selection loss, not prompt loss. A verifier-based selector could close it. Prompt engineering cannot.
Original Article
View Cached Full Text

Cached at: 04/20/26, 08:28 AM

Paper page - Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3

Source: https://huggingface.co/papers/2603.27844

https://huggingface.co/papers/2603.27844#model-capability-dominates-inference-time-optimization-lessons-from-aimo-3Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3

fig3_p_vs_score (https://cdn-uploads.huggingface.co/production/uploads/64705d3890482b0e0f6591ed/PFAaNjsMA8B86SLl-y925.png)

Diverse Prompt Mixer assigns different reasoning strategies to majority-voting members to decorrelate errors. Tested on 50 IMO-level problems (1×H100, 5-hour limit, 3 models, 23+ experiments). It does not work.

Why it fails: High-temperature sampling already pushes pairwise error correlation to zero or below (mean ρ̂ = −0.348 across 19 computable points). There is no correlation headroom left. Diverse prompts reduce per-attempt accuracy more than they reduce correlation.

What dominates: At equal N=8, the 8-point model capability gap (gpt-oss-120b at 39.3 vs. gpt-oss-20b at 31.0) is 4× larger than any prompt optimization (±2 points). Scaling N past the compute budget backfires.

Where the real gap is: The model’s pass@20 ≈ 45.5, but majority voting peaks at 42. Six points of selection loss. A verifier-based selector could close it. Prompt engineering cannot.

Similar Articles

AIMO Interpretability Challenge

arXiv cs.AI

The AIMO Interpretability Challenge is a competition aimed at distinguishing robust from spurious reasoning in frontier mathematical language models using interpretability methods, providing new problems, model access, and computing infrastructure.

Multi Tier MoE Caching

Reddit r/LocalLLaMA

Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.

The Capability Frontier: Benchmarks Miss 82% of Model Performance

arXiv cs.AI

The paper introduces the Capability Frontier, a Pareto frontier over models that corrects for biases in single-model and single-run evaluations, showing that standard benchmarks miss up to 82% of model performance and that collective LLM capabilities are substantially underestimated.

Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space

arXiv cs.LG

This paper frames model merging as probabilistic inference under a product-of-experts scenario, showing that existing methods are special cases and proposing a heavy-tailed Cauchy expert design that better captures real residual behavior, achieving significant improvements over state-of-the-art baselines.