REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming
Summary
REFORGE is a novel pipeline for benchmarking LLMs' reverse engineering capabilities, specifically for decompiled binary function naming, that addresses reliability issues in binary-to-source alignment under compiler optimization.
View Cached Full Text
Cached at: 07/13/26, 07:54 AM
# REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming Source: [https://arxiv.org/abs/2607.07738](https://arxiv.org/abs/2607.07738) [View PDF](https://arxiv.org/pdf/2607.07738) > Abstract:Large language models \(LLMs\) are increasingly applied to reverse\-engineering tasks, and recent threat\-intelligence reporting shows them operating inside live offensive\-security workflows\. Claims about their capability, however, outpace our ability to measure it\. Existing benchmarks for LLM\-assisted binary analysis treat the construction of function\-level ground truth as a solved pre\-processing step and report accuracy without disclosing how many functions were reliably evaluable\. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary\-to\-source alignment under compiler optimization\. This paper presents Reforge, a provenance\-tracked pipeline that constructs function\-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight\-gate confidence funnel with three\-tier stratification\. On a controlled micro\-benchmark, high\-confidence yield falls from 87\.2% to 65\.9% across optimization levels, and unpaired comparisons overstate optimization\-induced performance decay through survivorship bias\. A proof\-of\-concept evaluation of seven contemporary LLMs on function naming demonstrates the substrate and motivates uncertainty\-aware benchmarking practice ## Submission history From: Andreas U\. Schmidt \[[view email](https://arxiv.org/show-email/e3f81807/2607.07738)\] **\[v1\]**Tue, 7 Jul 2026 23:17:03 UTC \(351 KB\)
Similar Articles
ProgramBench Vetted: Reverse Engineering from a Runnable Binary
ProgramBench Vetted is a benchmark with 50 tasks that evaluate AI agents on reconstructing programs from runnable binaries, designed to study long-context coordination and enable dense reward for reinforcement learning.
Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation
Decompile-Bench is a large-scale open-source dataset providing two million binary-source function pairs for improving LLM-based binary decompilation, along with a benchmark (Decompile-Bench-Eval) for evaluation. Fine-tuning with it yields a 20% improvement in re-executability over previous benchmarks.
REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark
RepoExec is a new benchmark for evaluating repository-level code generation focused on executability, functional correctness via automated test generation, and cross-file dependency integration. Experiments reveal that pretrained LLMs beat instruction-tuned ones on correctness, while fine-tuned models better leverage provided code dependencies.
Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination
Introduces HERO, an LLM-based program optimizer that overcomes the weakest-link effect by generating and recombining heterogeneous atomic edits, achieving faster convergence and higher scores across algorithmic, game, agentic, and robotic domains.
Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
This paper introduces AlgoREval, a benchmark of 599 problems covering 77 classical algorithms across 14 domains, 7 languages, and 4 graph-input representations, to measure LLMs' ability to reproduce well-known algorithms as 'parametric code retrieval' rather than true synthesis. Across 15 models (7B–34B), accuracy varies widely by language and input format, and the authors caution against deploying AI-generated algorithmic code without systematic validation.