An independent researcher's study finds that a single LLM misses about half of code-review defects, while using multiple models from different providers significantly improves coverage, with the biggest gain from adding a second model. The paper seeks feedback and arXiv endorsement.
tl;dr I'm an independent researcher and this is my first paper. I spent the last couple of months measuring whether a single LLM is actually good enough to review code on its own, or whether you need a few different ones. I sense through anecdotal observation that I was getting significant returns by using a mixed set of LLM for parallel code reviews. I always output the details of every code review from each individual reviewer and I also document which are legitimate findings and which are not. That combination of data provided me with what I needed to perform the analysis. Short version: one model misses a lot. Full paper is here: [https://doi.org/10.5281/zenodo.20519584](https://doi.org/10.5281/zenodo.20519584) I'd really appreciate people picking apart the methodology, and if anyone here can endorse on arxiv, I'm trying to get this posted to [cs.SE](http://cs.SE) and could use a hand. The setup: a software team ran every code review through 2 to 4 different LLMs separately, then a human went through and reconciled all the findings into one list of what was actually wrong. I used that as the answer key and scored how many of the real, confirmed defects each model caught. 18 code artifacts, 154 confirmed defects, 8 model versions across 5 providers. What I found: * No single model got above about 64% recall on the confirmed defects, and a typical one caught roughly half. * Over half of the defects (56.5%) were caught by only one of the models. They mostly weren't finding the same bugs (median overlap was about 0.37 Jaccard). * Adding providers one at a time, coverage went 33.6% with one, 57.1% with two, 74.6% with three, 88.7% with four. The biggest single gain is just adding a second model from a different provider. The practical version: don't lean on one model for code review. Run two or three different ones independently, have a human reconcile the results and check them against the actual source, and expect somewhere around half to two thirds for any single model. What I'm hoping for: 1. Feedback on the method and the stats (recall with Wilson intervals, the Jaccard overlap, the coverage curve). Tell me what's weak. 2. An arxiv endorsement. As a first-time submitter I need one already-published author (3+ cs.\* papers in the last 5 years) to endorse me for cs.SE. Takes about two minutes, and you're not vouching for the paper, just that I'm a real person. If you're open to it, comment or DM and I'll send my code privately. Happy to let you read the paper first.
The paper investigates using cross-model disagreement in LLMs to target expert effort in revising codebooks for large-scale text annotation, showing that rationale labeling outperforms other feedback methods.
This study evaluates two multimodal LLMs as peer reviewers for ICLR 2026 submissions, finding high scoring calibration but low error detection, with author identity having no effect and figures reducing error detection.
This paper evaluates LLM systems for pre-submission peer review, demonstrating that broad generation can recover most historical review issues but compressing them into a short report is challenging, with implications for AI-assisted research feedback.
A research paper introducing Chain-of-Models (CoM), an automated pipeline where a second LLM audits a first model's reasoning trace to correct cognitive biases. It finds that auditor effectiveness depends on model family and bias type, and proposes a bias-specific auditor selection rule that improves judgment accuracy.
This paper proposes a two-stage sampling design where LLM evaluations are used to augment, rather than replace, human ratings, and provides guidance on determining sample sizes for human and LLM reviews using a doubly robust estimator from missing data literature.