@jakevin7: Sharing a god-tier review prompt methodology. The LLM self-correction survey 'When Can LLMs Actually Correct Their Own Mistakes?' concludes that without reliable external feedback such as test results or tool outputs, a model relying only on self-reflection often cannot steadily correct errors...
Summary
Shared a prompt methodology based on LLM self-correction research, emphasizing that self-checking is limited without external feedback, and recommending progressively enhanced prompting strategies such as adversarial review.
View Cached Full Text
Cached at: 08/06/26, 12:42 PM
Sharing a top-tier review prompt methodology.
The survey on LLM self-correction, When Can LLMs Actually Correct Their Own Mistakes?, concludes that without reliable external feedback such as test results or tool outputs, models relying solely on their own reflection often struggle to reliably correct errors. In other words, simply asking the model to “check itself” has very limited effectiveness.
The following three prompts tend to be progressively more effective:
- “Review it.”
- “Perform an adversarial review.”
- “Focus on checking the XX error-prone aspects, and conduct an adversarial review.”
Essentially, the human acts as a guide, providing the Agent with additional information + methodology. The adversarial review is the methodology for the Review, and “XX error-prone aspects” is the additional information. Both of these give the Agent new attention directions and extra computational budget.
Similar Articles
When Should LLMs Trust Their Own Revisions? A Risk-Aware Study of Intrinsic Self-Correction
This arXiv paper studies intrinsic self-correction across 29 open-weight LLMs on BoolQ, GSM8K, and Corr2Cause, finding that revision can flip previously correct answers into errors and that self-correction should be treated as a selective revision policy rather than a uniformly beneficial second pass.
LLM-as-an-Improver: Turning Verification into Better Candidates
This paper introduces LLM-as-an-Improver, a method that uses verification feedback to generate improved candidate solutions for LLMs, enhancing performance beyond initial candidate pools.
PAPER2LLM++: Continual Self-Evolution of LLMs from Research Papers
Paper2LLM++ introduces a framework that turns the growing literature on LLM failure modes into a stream of supervision for continual self-improvement, verifying whether reported limitations still exist and integrating updates via a try-evaluate-commit procedure that preserves prior gains across sequential research papers.
More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review
This paper evaluates LLM systems for pre-submission peer review, demonstrating that broad generation can recover most historical review issues but compressing them into a short report is challenging, with implications for AI-assisted research feedback.
What it took to get an LLM judge from garbage to 27 of 30 agreement with humans (live, at a conference)
作者在 AI Engineer World's Fair 上的 evals 演讲总结,记录了如何将一个 LLM 评审器从完全不可用提升到与人类判断 30 次中有 27 次一致:包括改用点式评分避免偏好偏差、校准上下文、更换模型以及应对谄媚倾向等步骤。