REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming
Summary
REFORGE is a novel pipeline for benchmarking LLMs' reverse engineering capabilities, specifically for decompiled binary function naming, that addresses reliability issues in binary-to-source alignment under compiler optimization.
View Cached Full Text
Cached at: 07/13/26, 07:54 AM
# REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming Source: [https://arxiv.org/abs/2607.07738](https://arxiv.org/abs/2607.07738) [View PDF](https://arxiv.org/pdf/2607.07738) > Abstract:Large language models \(LLMs\) are increasingly applied to reverse\-engineering tasks, and recent threat\-intelligence reporting shows them operating inside live offensive\-security workflows\. Claims about their capability, however, outpace our ability to measure it\. Existing benchmarks for LLM\-assisted binary analysis treat the construction of function\-level ground truth as a solved pre\-processing step and report accuracy without disclosing how many functions were reliably evaluable\. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary\-to\-source alignment under compiler optimization\. This paper presents Reforge, a provenance\-tracked pipeline that constructs function\-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight\-gate confidence funnel with three\-tier stratification\. On a controlled micro\-benchmark, high\-confidence yield falls from 87\.2% to 65\.9% across optimization levels, and unpaired comparisons overstate optimization\-induced performance decay through survivorship bias\. A proof\-of\-concept evaluation of seven contemporary LLMs on function naming demonstrates the substrate and motivates uncertainty\-aware benchmarking practice ## Submission history From: Andreas U\. Schmidt \[[view email](https://arxiv.org/show-email/e3f81807/2607.07738)\] **\[v1\]**Tue, 7 Jul 2026 23:17:03 UTC \(351 KB\)
Similar Articles
Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks
Researchers introduce BEHEMOTH benchmark and CluE cluster-based prompt optimization to enable LLMs to extract and retain heterogeneous memory across diverse tasks, achieving 9% gains over prior self-evolving frameworks.
We use LLMs to analyze every file in your codebase. Everyone told us this was a stupid idea because of cost but it wasnt.
A benchmark study demonstrates that using LLMs to analyze entire codebases is cost-effective, identifying DeepSeek V4 Flash as the optimal default model due to its low cost and comparable accuracy to premium options like Claude Opus.
ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis
ReaComp compiles LLM reasoning traces into reusable symbolic program synthesizers that achieve strong accuracy on program synthesis benchmarks while eliminating LLM calls at test time, significantly reducing computational cost.
LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning
Introduces LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier LLMs on structured linear algebra computation across matrix dimensions, revealing that LLM mathematical failure is structurally constrained and transitions from execution errors to computational abandonment at 4x4 scale.
ReFlect: An Effective Harness System for Complex Long-Horizon LLM Reasoning
This paper introduces ReFlect, a training-free harness system that wraps LLMs with deterministic error detection and recovery logic to improve performance on complex, long-horizon reasoning tasks.