Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Summary
This paper introduces the Precise Debugging Benchmark (PDB), a framework that evaluates LLMs on precise fault localization rather than just test pass rates. Results show frontier models like GPT-4.1-Codex and DeepSeek-V3.2-Thinking pass 76%+ of unit tests but achieve less than 45% edit precision, revealing a critical gap between code regeneration and true debugging.
View Cached Full Text
Cached at: 04/21/26, 07:20 AM
Paper page - Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Source: https://huggingface.co/papers/2604.17338
Abstract
Frontier LLMs demonstrate high test pass rates but poor precision in debugging tasks, indicating a gap between functional correctness and precise fault localization.
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce thePrecise Debugging Benchmark(PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verifiedatomic bugsand composing them into multi-bug programs. We define two novel metrics,edit-level precisionandbug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative andagentic debuggingstrategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
View arXiv pageView PDFProject pageGitHub3Add to collection
Get this paper in your agent:
hf papers read 2604\.17338
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.17338 in a model README.md to link it from this page.
Datasets citing this paper3
#### Precise-Debugging-Benchmarking/PDB-Multi Viewer• Updatedabout 4 hours ago • 256 • 42 #### Precise-Debugging-Benchmarking/PDB-Single-Hard Viewer• Updatedabout 4 hours ago • 5.75k • 38 #### Precise-Debugging-Benchmarking/PDB-Single Viewer• Updatedabout 4 hours ago • 7.59k • 32
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.17338 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls
This paper introduces ParamBench, a difficulty-graded benchmark for LLM tool-call parameter generation, and proposes probe-guided training methods (PBT and PGR) that improve exact-match accuracy from 19.7% to 59.6%.
Open-source models are closing the coding gap with GPT/Claude/Gemini ~1.5x faster than the frontier is advancing, and on decontaminated benchmarks a 27B model already beats Claude Opus 4.8 [live dashboard + analysis]
A live dashboard and statistical analysis shows open-source coding models are closing the gap with closed models at 1.5x the rate, with a 27B model already surpassing Claude Opus on decontaminated benchmarks. Tool-call reliability remains the main bottleneck.
Predictive Data Debugging: Reveal and Shape What Your Model Learns, Before You Train (11 minute read)
This research introduces a method using interpretability to predict which behaviors DPO will amplify or suppress from a preference dataset before training, enabling data debugging to prevent undesired effects. The technique achieves R²=0.9 prediction accuracy and is integrated into Goodfire's Silico platform.
Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
This paper introduces FinED-Bench, the first public benchmark for financial error detection in documents, covering nine real-world scenarios with over 900 documents from 2025. It evaluates current LLMs like GPT-4o and Qwen3-14B, showing they still struggle with high-complexity cases, while fine-tuning can improve weaker models.
PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX
PTXBench is introduced as a benchmark to evaluate and adapt large language models for optimizing GPU kernels using architecture-specific PTX, showing uneven performance and fine-tuning insights.