@omarsar0: Finally a good paper testing whether self-reflection loops are worth it. Setup: Seven methods, open models at 1.5B, 3B …
Summary
A paper finds that self-reflection loops like Self-Refine and Reflexion do not beat repeated sampling at equal token cost across open models from 1.5B to 7B, with all 18 self-inspection comparisons negative.
View Cached Full Text
Cached at: 08/05/26, 12:17 AM
Finally a good paper testing whether self-reflection loops are worth it.
Setup: Seven methods, open models at 1.5B, 3B and 7B, two math benchmarks, 150 questions each.
Every generated token counted, including the ones spent on critiques, reflections, debate turns and checking. Each method compared against repeated sampling at its own measured cost, paired by question, with bootstrap intervals and multiplicity correction.
All 36 comparisons came back with no reliable win for any method. Ten are reliably worse, and every one of those is a method where the model inspects its own output. All 18 self-inspection comparisons are negative.
Self-Refine and a forced Reflexion sit 3.6 to 10.1 points below the baseline at 7B. Reflexion as published never triggered its own retry on the smallest model. It judged itself correct every time and quietly became a single chain of thought.
Worth knowing before you add another critique step to your agent loop.
Paper: https://arxiv.org/abs/2607.28576
Track more trending AI papers in our academy: https://academy.dair.ai
Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B
Source: https://arxiv.org/abs/2607.28576 Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Code, Data, Media
Code, Data and Media Associated with this Article
Demos
Demos
Related Papers
Recommenders and Search Tools
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv’s community?Learn more about arXivLabs.
Similar Articles
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
A comprehensive survey of 1,250 papers (2024–2026) on recursive self-improvement in AI, proposing a taxonomy distinguishing bounded self-refinement from open-ended recursive self-improvement, and analyzing the evaluator design space and failure modes.
@rohanpaul_ai: Self-improving AI is only as real as the signal it was tested on to see if it worked. Sorting 1,250 papers reveals a si…
Analysis of 1,250 papers on recursive self-improvement in AI reveals that the evaluator signal is the critical bottleneck. Models improve reliably only with strong, trustable signals like proof checkers, while weak signals cause loops to collapse or reinforce errors.
Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
A new verifier-free breadth-depth refinement framework improves LLM reasoning at test time by sampling multiple rollouts, iteratively refining each via self-critique, and aggregating with majority voting. It consistently outperforms greedy decoding, majority voting, and verifier-based selection across several math benchmarks and open-weight models.
Why self-reflection ReAct loops fail on long-horizon tasks, and the AgentOS verification architecture we built to fix it.
Explains why self-reflection ReAct loops fail on long-horizon tasks and introduces the AgentOS verification architecture as a solution.
Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation
The paper introduces Reflection-Enhanced Self-Distillation (Resd), a framework that transforms failure feedback into corrective supervision for LLMs, enabling efficient learning from rare successes. It outperforms standard self-distillation baselines and achieves faster early improvement than GRPO with fewer samples.