test-time-compute

Explores whether a 35B parameter model could surpass a 1000B model by scaling its search horizon at test time, using structured process feedback rather than brute-force parameter scaling.

This paper from Stanford, Berkeley, and NVIDIA introduces LLM-as-a-Verifier, a general-purpose verification framework that uses token logits for continuous scoring. It achieves SOTA on multiple benchmarks including Terminal-Bench V2 (86.5%) and SWE-Bench Verified (78.2%), and provides fine-grained signals that can accelerate RL training.

The article examines whether the benefits of scaling test-time compute for AI models are diminishing, potentially indicating a ceiling in current scaling paradigms.

A 5M-parameter model outperforms frontier LLMs on hard logical puzzles at a fraction of the inference cost by using continuous latent space test-time compute.

Discussion of dynamic workflows for test-time compute, including their limited use cases, benefits for research experiments, and the need for better benchmarks. Mentions models like Mythos and Opus 4.8 for agent orchestration.

The post discusses the confirmed research finding that verifier-based test-time compute scaling dominates verifier-free methods, with practical examples like Apodex showing gains from separate verification processes. It argues that building independent verifiers is a key path for future AI capability improvements.

The article discusses the rise of 'loops' in AI agentic systems, where agents continuously prompt other agents to perform tasks, as a major step beyond simple agent use. Boris Cherny of Claude Code endorses this approach at Meta's @Scale conference.

This paper introduces LoopCoder-v2, a 7B code model that benefits most from a single rethinking loop; additional loops degrade performance, challenging the assumption that more test-time compute always helps.

An analysis of the economics and performance impact of AI reasoning models, showing that enabling reasoning can improve accuracy by 10-20% but costs 5-10x more tokens, and discussing different reasoning types and their applications.

A deep retrospective on the four-year evolution of reasoning models from CoT in 2022 to o1/R1 in 2024, pointing out that the true watershed is not the emergence of reasoning ability, but the conversion of reasoning into a billable, schedulable resource.

This paper systematically studies how inference-time compute (token budgets, context compaction, repeated submissions) affects frontier LLM performance on challenging benchmarks, demonstrating that scores are protocol-dependent and advocating for evaluations that report capability as a function of inference compute.

This article describes a scaffold that scales test-time compute on Qwen-3.6-27B and Gemma-4-31B using iterative corrections and branch exploration to surpass Claude Mythos in code optimization. It includes a paper link and GitHub repository.

Highlights OpenAI researcher Noam Brown's argument: the true ceiling of LLM capabilities is far higher than current benchmarks show, due to insufficient test-time compute, and stronger models benefit more from additional computation. This poses a serious challenge for AI safety evaluation, as many dangerous capabilities may only emerge under long time and high compute budgets.

This article argues that LLM benchmark performance is increasingly a function of test-time compute, and that current evaluation methods fail to capture capability improvements when controlling for inference budget. It advocates for plotting performance vs. tokens, cost, or time, and discusses implications for safety evaluations.

This paper proposes consequence-aware test-time compute allocation, routing higher-stakes software engineering tasks to larger compute budgets based on predicted failure cost rather than just difficulty. Evaluated on SWE-bench Lite and Multi-SWE-bench mini, the approach reduces cost-weighted loss by 22–33% compared to difficulty-aware routing.

This paper introduces a prefix-level trajectory evaluation protocol to distinguish harmful overthinking from verbose but harmless overthinking in large reasoning models, showing that continued reasoning after reaching the correct answer can destabilize performance. The authors find that early stopping improves accuracy by up to 21% on multimodal benchmarks, and identify logical drift and visual reinterpretation as key causes of correctness deviations.

FineVerify is a self-verification framework for agentic search that decomposes questions into sub-questions, verifies sampled candidates, and selects the best one, achieving substantial accuracy improvements over baselines on multiple benchmarks, including enabling GPT-5-mini to surpass GPT-5 on BrowseComp-Plus.

A critique arguing that training LLMs on human-generated data limits their ability to discover novel solutions via test-time compute, and that true AGI requires models that can explore hypothesis spaces more broadly, similar to AlphaZero.

The paper 'Generative Recursive Reasoning' introduces a method that scales test-time compute by sampling multiple latent reasoning trajectories in parallel, enabling the model to explore diverse hypotheses and avoid deterministic collapse. This approach improves performance on tasks such as Sudoku, ARC AGI, N Queens, and graph coloring, and can also generate valid Sudoku boards and MNIST digits.

Equilibrium Reasoners (EqR) introduce a novel framework for scalable reasoning by learning task-conditioned attractors in latent dynamical systems, achieving over 99% accuracy on Sudoku-Extreme by unrolling up to 40,000 layers.