This position paper proposes a credit system for ML conferences to incentivize quality reviewing by awarding points for good behavior and allowing redemption for perks.
“Maybe the real AGI was the friends we made along the way” is a sentiment that always hits me, and conferences are the places where I reunite with old friends and meet new ones. However, when it comes to the submission/review experience, it might not be much of an exaggeration to say that almost everyone has many unpleasant experiences to share. So I wrote a position paper to discuss this. I argue that current conference organizers lack proper tools to instill accountability and incentives for reviewers/authors/ACs/SACs… The result is that undesired behaviors (e.g., lack of engagement) often go unchecked, while good behaviors are rarely rewarded and therefore don’t happen (honestly, when was the last time you witnessed any constructive internal discussion among reviewers/ACs?). And this won’t change by writing nice words in Reviewer Guidelines or issuing a few desk rejections. I propose a CREDIT SYSTEM where community members earn points by “doing good” — e.g., reviewing a paper would get you +1, being outstanding gets you +3. Then, members can spend points to redeem perks ranging from traditional ones already adopted in current ML conferences (e.g., free registration) to new ones, such as requesting an additional reviewer to sort through a muddy situation. Such a system could also support explorative ideas like: - Refundable submission fees: say 10 points per submission, which are then refunded regardless of acceptance, unless the submission is uniformly voted to be unready / ultra-low quality. - Mobilizing non-author reviewers: non-author reviewers don’t have the bandwidth issue of wearing both the author and reviewer hats and are not influenced by their own submissions. and many more... My proposed system is far from perfect, but I’d like to think it takes a step toward a better conference review mechanism. I am also glad to see the position paper track becoming a welcoming platform for researchers to hash out their proposals and build toward a better future (see other review-related position papers below.) For a topic that affects literally everyone at ICML, I am eager to hear your thoughts.
A researcher questions why the ML community doesn't limit submissions per author to manage review quality, citing successful practices in other fields like Security and Computer Architecture.
This position paper argues that machine learning research should prioritize ideas over benchmarks and theoretical guarantees, proposing an 'Ideas First' framework that values behavioral signatures and tailored experiments to promote equity and scientific understanding.
A researcher discusses their ICML 2026 paper review experience where a reviewer increased their score during rebuttal but then decreased it again, expressing concern about rejection prospects.
This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.