Paper page - Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Source: https://huggingface.co/papers/2604.17338

Abstract

Frontier LLMs demonstrate high test pass rates but poor precision in debugging tasks, indicating a gap between functional correctness and precise fault localization.

Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce thePrecise Debugging Benchmark(PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verifiedatomic bugsand composing them into multi-bug programs. We define two novel metrics,edit-level precisionandbug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative andagentic debuggingstrategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2604\.17338

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.17338 in a model README.md to link it from this page.

Datasets citing this paper3

#### Precise-Debugging-Benchmarking/PDB-Multi Viewer• Updatedabout 4 hours ago • 256 • 42 #### Precise-Debugging-Benchmarking/PDB-Single-Hard Viewer• Updatedabout 4 hours ago • 5.75k • 38 #### Precise-Debugging-Benchmarking/PDB-Single Viewer• Updatedabout 4 hours ago • 7.59k • 32

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.17338 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.