test-time-compute

This paper introduces a benchmark for revision propagation in conversationally generated artifacts using LLMs and evaluates cost-effective test-time compute methods, showing that parallel sampling with selection improves accuracy.

This paper presents a specialization pipeline for post-training language models to achieve gold-medal performance in coding competitions, demonstrating top scores on IOI benchmarks using techniques like supervised fine-tuning and reinforcement learning.

The paper proposes CASE, a dynamic selection combiner using a decodability criterion to predict when hidden-state selection outperforms majority voting in large language models, enhancing reliability on difficult questions.

The article summarizes eight inference-time techniques to improve LLM reasoning, based on research from Google, OpenAI, and Anthropic, with tradeoffs and practical notes.

The paper 'Thought-Level Beam Search for Reasoning' introduces Gambit, a method that actively reallocates test-time compute during generation by pruning low-scoring reasoning traces and branching from strong prefixes, and it has been accepted to COLM 2026.

This paper introduces an exam-style evaluation to study how reasoning models allocate a shared test-time compute budget across multiple questions. It finds that models fail to strategically ration compute, instead prioritizing questions by presentation order and ignoring value or difficulty.

An observation that Claude Opus 5's max thinking leads to performance degradation on ~20-30% of benchmarks compared to xhigh, contrary to the expectation that more test-time compute improves performance.

Explores whether a 35B parameter model could surpass a 1000B model by scaling its search horizon at test time, using structured process feedback rather than brute-force parameter scaling.

This paper from Stanford, Berkeley, and NVIDIA introduces LLM-as-a-Verifier, a general-purpose verification framework that uses token logits for continuous scoring. It achieves SOTA on multiple benchmarks including Terminal-Bench V2 (86.5%) and SWE-Bench Verified (78.2%), and provides fine-grained signals that can accelerate RL training.

The article examines whether the benefits of scaling test-time compute for AI models are diminishing, potentially indicating a ceiling in current scaling paradigms.

A 5M-parameter model outperforms frontier LLMs on hard logical puzzles at a fraction of the inference cost by using continuous latent space test-time compute.

Discussion of dynamic workflows for test-time compute, including their limited use cases, benefits for research experiments, and the need for better benchmarks. Mentions models like Mythos and Opus 4.8 for agent orchestration.

The post discusses the confirmed research finding that verifier-based test-time compute scaling dominates verifier-free methods, with practical examples like Apodex showing gains from separate verification processes. It argues that building independent verifiers is a key path for future AI capability improvements.

The article discusses the rise of 'loops' in AI agentic systems, where agents continuously prompt other agents to perform tasks, as a major step beyond simple agent use. Boris Cherny of Claude Code endorses this approach at Meta's @Scale conference.

This paper introduces LoopCoder-v2, a 7B code model that benefits most from a single rethinking loop; additional loops degrade performance, challenging the assumption that more test-time compute always helps.

An analysis of the economics and performance impact of AI reasoning models, showing that enabling reasoning can improve accuracy by 10-20% but costs 5-10x more tokens, and discussing different reasoning types and their applications.

A deep retrospective on the four-year evolution of reasoning models from CoT in 2022 to o1/R1 in 2024, pointing out that the true watershed is not the emergence of reasoning ability, but the conversion of reasoning into a billable, schedulable resource.

This paper systematically studies how inference-time compute (token budgets, context compaction, repeated submissions) affects frontier LLM performance on challenging benchmarks, demonstrating that scores are protocol-dependent and advocating for evaluations that report capability as a function of inference compute.

This article describes a scaffold that scales test-time compute on Qwen-3.6-27B and Gemma-4-31B using iterative corrections and branch exploration to surpass Claude Mythos in code optimization. It includes a paper link and GitHub repository.

Highlights OpenAI researcher Noam Brown's argument: the true ceiling of LLM capabilities is far higher than current benchmarks show, due to insufficient test-time compute, and stronger models benefit more from additional computation. This poses a serious challenge for AI safety evaluation, as many dangerous capabilities may only emerge under long time and high compute budgets.