test-time-compute

This article argues that LLM benchmark performance is increasingly a function of test-time compute, and that current evaluation methods fail to capture capability improvements when controlling for inference budget. It advocates for plotting performance vs. tokens, cost, or time, and discusses implications for safety evaluations.

This paper proposes consequence-aware test-time compute allocation, routing higher-stakes software engineering tasks to larger compute budgets based on predicted failure cost rather than just difficulty. Evaluated on SWE-bench Lite and Multi-SWE-bench mini, the approach reduces cost-weighted loss by 22–33% compared to difficulty-aware routing.

This paper introduces a prefix-level trajectory evaluation protocol to distinguish harmful overthinking from verbose but harmless overthinking in large reasoning models, showing that continued reasoning after reaching the correct answer can destabilize performance. The authors find that early stopping improves accuracy by up to 21% on multimodal benchmarks, and identify logical drift and visual reinterpretation as key causes of correctness deviations.

FineVerify is a self-verification framework for agentic search that decomposes questions into sub-questions, verifies sampled candidates, and selects the best one, achieving substantial accuracy improvements over baselines on multiple benchmarks, including enabling GPT-5-mini to surpass GPT-5 on BrowseComp-Plus.

A critique arguing that training LLMs on human-generated data limits their ability to discover novel solutions via test-time compute, and that true AGI requires models that can explore hypothesis spaces more broadly, similar to AlphaZero.

The paper 'Generative Recursive Reasoning' introduces a method that scales test-time compute by sampling multiple latent reasoning trajectories in parallel, enabling the model to explore diverse hypotheses and avoid deterministic collapse. This approach improves performance on tasks such as Sudoku, ARC AGI, N Queens, and graph coloring, and can also generate valid Sudoku boards and MNIST digits.

Equilibrium Reasoners (EqR) introduce a novel framework for scalable reasoning by learning task-conditioned attractors in latent dynamical systems, achieving over 99% accuracy on Sudoku-Extreme by unrolling up to 40,000 layers.

GridProbe is a training-free inference paradigm for Long-Video VLMs that adaptively selects relevant frames using posterior probing, achieving sub-quadratic attention costs with minimal accuracy loss.

This technical report introduces ZAYA1-8B, a mixture-of-experts reasoning model trained on AMD hardware that achieves competitive performance on math and coding benchmarks using under 1B active parameters. It also details Markovian RSA, a novel test-time compute method for aggregating parallel reasoning traces.

Moonshot AI has open sourced Kimi K2.6 and argues that the next frontier in test-time compute is better organization of intelligence rather than simply building bigger models.