When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
Summary
This paper introduces Elo-per-token analysis to study how LLM agents allocate test-time compute, revealing that agents initially outperform independent sampling but slow down over time, with parallel sessions offering performance gains.
View Cached Full Text
Cached at: 09/15/26, 06:40 AM
Paper page - When Agents Slow Down: Understanding LLM Agents’ Test-Time Strategies via Elo-per-token Analysis
Source: https://huggingface.co/papers/2609.15309
Abstract
Elo-per-token analysis reveals that LLM agents initially scale faster than independent sampling but eventually slow, while parallel short sessions improve performance over single long runs.
Large language model (LLM) agents allocatetest-time computeadaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We studyopen-ended tasksthat provide continuous scores for intermediate submissions, making progress observable throughout long trajectories. We proposeElo-per-token analysis, which tracks the best solution found at each token budget and uses aBradley-Terry modelto aggregate within-task orderings into Elo ratings across tasks with different score scales. We apply it to four general-purpose agents on four open-ended benchmarks, with sessions of up to 100M tokens, and to three feedback-driven LLM optimization harnesses in controlled single-task interventions.Independent samplingprovides a theoretically characterized reference, for which Elo grows linearly with log compute. Against this reference, agents can initially convert tokens into Elo faster thanindependent sampling, but their marginal gains diminish and eventually fall below the reference. In contrast, the strongest historical human contestants improve superlinearly over contest time on sharedAtCoder Heuristic Contesttasks, providing evidence of continual learning and substantial headroom after agents slow down. We define thescaling inflection pointas the per-session budget where marginal Elo gains match the independent-sampling reference. Using this point as the per-session budget, we split 100M tokens across parallel sessions on FrontierCS Polyomino Packing, gaining +264 Elo over one long session and +355 over ten short sessions.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.15309 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.15309 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.15309 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics
The paper characterizes the resource and performance dynamics of LLM-based AI agents across tasks like question answering and coding, revealing bottlenecks and proposing optimizations that improve latency by up to 5.4×.
@polynoamial: https://x.com/polynoamial/status/2064210146558136827
This article argues that LLM benchmark performance is increasingly a function of test-time compute, and that current evaluation methods fail to capture capability improvements when controlling for inference budget. It advocates for plotting performance vs. tokens, cost, or time, and discusses implications for safety evaluations.
@rohanpaul_ai: New Stanford paper argues that, under equal reasoning budgets, one LLM usually solves multi-hop problems better than ma…
A new Stanford paper shows that under equal reasoning token budgets, single LLMs typically outperform multi-agent systems on multi-hop reasoning tasks, with gains from multi-agent setups often stemming from additional compute rather than architectural superiority. The paper uses the Data Processing Inequality to explain why information loss in handoffs harms multi-agent performance, and identifies context quality as the key factor where multi-agent systems can provide benefits.
AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?
AgentStream introduces a unified framework to evaluate self-evolving LLM agents under streaming task scenarios, showing that self-evolution reliability varies across scenarios and is gated by model capability.
LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling
This paper introduces AutoTTS, an environment-driven framework that automates the discovery of test-time scaling strategies for LLMs by formulating it as controller synthesis. It demonstrates improved accuracy-cost tradeoffs on mathematical reasoning benchmarks with minimal computational overhead.