在剧透与猜测之间：避免编码基准中的过度规范

# Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Source: [https://arxiv.org/html/2610.22760](https://arxiv.org/html/2610.22760)
Yuanzhe Liu††thanks:Work was done while the first author interned at IBM\.Bastian LipkaAffiliation:IBMEmail:[may13@rpi\.edu](mailto:)Avraham ShinnarAffiliation:IBMEmail:[Lipka\.Bastian@ibm\.com](mailto:)Jatin GanhotraAffiliation:IBMEmail:[shinnar@us\.ibm\.com](mailto:)Yao MaAffiliation:Rensselaer Polytechnic InstituteEmail:[jatinganhotra@us\.ibm\.com](mailto:)Martin HirzelAffiliation:IBMEmail:[hirzel@us\.ibm\.com](mailto:)

###### Abstract

Benchmarks drive AI progress, but are hampered by task under\-specification and over\-specification\. Task under\-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking\. Task over\-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure\. Since future\-looking benchmarks should be difficult, over\-specification reduces a benchmark’s shelflife of community adoption\. Repository\-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over\- and under\-specification\. For instance, SWE\-bench Pro, FEA\-bench, Doc2Feat\-bench, and Multi\-SWE\-bench all tackle under\-specification by augmenting task statements with hints derived from ground truth\. We show that leaking ground\-truth information into the task statement causes over\-specification in some of these benchmarks\. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork\. Based on these results, we recommend techniques for deriving parsimonious hints\. We will release our hints so the community can obtain more reliable signals from popular benchmarks\.

## 1Introduction

Repository\-level coding benchmarks have contributed to great advancements in coding agents by evaluating them via executing held\-out tests against their proposed patches\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22760#bib.bib2)\)\. For these evaluations to measure implementation ability, the task description and available repositories have to communicate the requirements enforced by the tests\. Achieving this alignment has proven difficult\. OpenAI introduced SWE\-bench Verified to reduce underspecification and unfair testing\([OpenAI, 2024](https://arxiv.org/html/2610.22760#bib.bib11)\)\. It later recommended SWE\-bench Pro after finding test flaws and contamination in Verified\([OpenAI, 2026c](https://arxiv.org/html/2610.22760#bib.bib12)\), then withdrew that recommendation after finding underspecified prompts and overly restrictive tests in Pro\([OpenAI, 2026b](https://arxiv.org/html/2610.22760#bib.bib13)\)\. This churn is evidence for both the importance and difficulty of this problem\. One source of misalignment is a test that requires a particular new function, class, file, or attribute name that the agent cannot infer from the task description or repository\. The problem is especially relevant to feature\-addition benchmarks\([Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3);[Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4)\), where the new names may be entirely missing in the existing repository\. An agent can then implement the requested behavior correctly yet fail solely because it chose a different name\. Recovering these unstated interface choices introduces*guesswork*into an evaluation intended to measure software implementation ability\.

To reduce this guesswork, several benchmarks augment task descriptions with information derived from reference implementations and test patches\([Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4);[Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3);[Deng et al\., 2025b](https://arxiv.org/html/2610.22760#bib.bib5);[Badertdinov et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib6)\)\. We refer to these additions as*hints*\. Figure[2](https://arxiv.org/html/2610.22760#S1.F2)provides an overview of how hints are generated and used in repository\-level coding benchmarks\. An agent receives a task descriptiondissued\_\{\\mathrm\{issue\}\}and existing code and tests\(cold,told\)\(c\_\{\\mathrm\{old\}\},t\_\{\\mathrm\{old\}\}\), and produces codecgenc\_\{\\mathrm\{gen\}\}evaluated against held\-out teststnewt\_\{\\mathrm\{new\}\}\. A hint generator additionally accesses the reference codecnewc\_\{\\mathrm\{new\}\}and held\-out tests to producedhintsd\_\{\\mathrm\{hints\}\}, an extra input to the agent\. Hints can expose details that an agent needs to satisfy the evaluation, but the reference implementation also contains design decisions that the benchmark may intend the agent to make independently\. Disclosing these decisions introduces*spoilers*, i\.e\., information that supplies part of the implementation work being evaluated\. Prior work documents solution leakage through issue reports and comments as a major problem for SWE\-bench\([Aleithan et al\., 2024](https://arxiv.org/html/2610.22760#bib.bib18)\)\. Therefore, identifier hint construction faces a tension between*under\-specification*, which leaves agents guessing about hidden constraints, and*over\-specification*, which leaks parts of the solution\. Higher solve rates alone cannot distinguish between resolving guesswork and supplying spoilers\. Our goal is to study identifier hint constructions that clarify the interface required by the tests while leaving implementation choices to the agent\.

Figure 1:To hint \(perchance spoilers\) or not to hint \(guesswork\): that is the question for repository\-level coding benchmarks\.
Figure 2:DefUse hints generation pipeline\.

#### A design space for hints\.

We consider hint construction through two design decisions:*identifier selection*and*identifier projection*\. Selection determines*which*program entities are disclosed, such as file changes, functions edits, etc\. Identifier projection asks*how much*is revealed about each selected entity\. For instance, if a function identifier is selected, then its name, its signature, its location, or its semantics descriptions are all valid information to be considered being revealed\. Existing hint schemes combine different choices along both dimensions\([Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4);[Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3);[Deng et al\., 2025b](https://arxiv.org/html/2610.22760#bib.bib5)\), making it difficult to attribute performance differences to either choice when comparing complete schemes\. Separating selection from projection allows us to examine these choices through controlled comparisons\.

Guided by this distinction, this paper proposesDefUsefor identifier selection andDeclarationfor identifier projection\. DefUse targets identifiers through which the implementation and its tests interact\. Declaration reveals interface attributes while omitting semantic descriptions and implementation guidance\. Together, these policies aim to reduce*guesswork*about the required identifiers while limiting*spoilers*about how to implement it\.

We compare selection and projection policies across five repository\-level coding benchmarks spanning Python and Java\. We complement these comparisons with hint attribute ablations and an LLM\-based assessment on SWE\-bench Pro, examining both task resolution and disclosure of reference\-solution information\. Our findings indicate that performance gains can accompany greater disclosure of implementation choices, motivating evaluation beyond solve rates alone\. Together, these findings motivate combining DefUse selection with Declaration projection to reduce*guesswork*about required interfaces while limit*spoilers*about implementation choices\.

The source code of our hint generation pipeline and the resulting hints are available open\-source:[https://github\.com/IBM/swe\-hints/](https://github.com/IBM/swe-hints/)\.

#### Contributions\.

This paper makes three contributions:

- •It characterizes identifier hints through selection and projection, providing a common framework for comparing existing benchmark policies\.
- •It introduces DefUse and Declaration, two policies designed to expose test\-facing interfaces while limiting additional reference\-derived information\.
- •It reports controlled comparisons and attribute ablations across five benchmarks and three language models, examining how hint construction affects measured task resolution\. Additionally, it adopts an LLM judge to assess information necessity, solution disclosure, and behavioral guidance\.

To the extent that benchmarks drive AI progress, we hope that finding the right balance between spoilers and guesswork in coding benchmarks will ultimately contribute to more reliable coding assistants\.

## 2Related Work

Table 1:What each hint scheme discloses\. DefUse is ours improved from Doc2Feat\. Doc2Feat means the identifier hints extraction used in the Doc2Feat benchmark\. FEA\-Bench presents two hint selection: brief and detailed\. In SWE Bench Pro, the authors assign ”Requirements” as the essential hints, and make ”Interface” optional\.#### Benchmark validity\.

Audits of SWE\-bench and SWE\-bench Pro document underspecified tasks and tests enforcing unstated implementation details\([OpenAI, 2024](https://arxiv.org/html/2610.22760#bib.bib11);[OpenAI, 2026b](https://arxiv.org/html/2610.22760#bib.bib13);[OpenAI, 2026c](https://arxiv.org/html/2610.22760#bib.bib12)\)\. SWE\-Bench\+ identifies solutions disclosed in issue reports or comments, alongside inadequate tests\([Aleithan et al\., 2024](https://arxiv.org/html/2610.22760#bib.bib18)\)\. Differential testing and manual analysis further show that passing benchmark tests does not guarantee patch correctness\([Wang et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib16)\)\. Evidence of memorization raises concerns about training\-data contamination\([Liang et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib10)\), which LiveCodeBench and SWE\-rebench seek to mitigate by collecting new tasks\([Jain et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib8);[Badertdinov et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib6)\)\. Our study complements these audits by examining how reference\-derived hints affect the information available to agents at evaluation time\.

#### Task specification and hints\.

[Khatib et al\. \(2026\)](https://arxiv.org/html/2610.22760#bib.bib1)examine how bug\-report content affects repair success, including controlled ablations of benchmark\-added requirements and interfaces\.[Ganhotra \(2026\)](https://arxiv.org/html/2610.22760#bib.bib7)identifies*hidden naming contracts*, where tests require reference\-solution identifiers absent from the issue and base repository\. Benchmarks address this ambiguity with reference\-derived names, declarations, or requirements\([Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4);[Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3);[Deng et al\., 2025b](https://arxiv.org/html/2610.22760#bib.bib5)\)\. Within agent workflows, SpecRover infers code intent\([Ruan et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib14)\), REAgent refines requirements\([Kuang et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib9)\), and HULA incorporates developer feedback into plans and code\([Takerngsaksiri et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib15)\)\. Interactive approaches address missing requirements through clarification: Ambig\-SWE evaluates agents’ ability to detect underspecification and elicit missing information\([Vijayvargiya et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib19)\)\. LHAW systematically removes task information and measures its effects through agent trials\([Pu et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib20)\)\. Our study separates*identifier selection*from*identifier projection*to examine which benchmark disclosures reduce*guesswork*about required interfaces and which supply*spoilers*about implementation choices\.

## 3Methodology

We characterize identifier hints through two decisions:*Identifier Selection*, which determines which program entities are disclosed, and*Identifier Projection*, which determines which attributes of each selected entity are revealed\. Table[1](https://arxiv.org/html/2610.22760#S2.T1)organizes existing hint schemes along these dimensions and situates our proposed selection policy, DefUse, within this design space\. This decomposition supports controlled comparisons of selection policies \(under a fixed projection\) and projection policies \(under a fixed selection\)\.

### 3\.1Identifier Selection: DefUse

Various identifier selection pipelines exist in the current repo\-level coding benchmarks\. The SWE\-Bench Pro authors employ human engineers to construct requirements and interface, while FEA\-Bench includes all new components present in the new files between commits\([Deng et al\., 2025b](https://arxiv.org/html/2610.22760#bib.bib5);[Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3)\)\. The Doc2Feat pipeline first extracts identifiers that appear after new code patches and new test patches, then selects identifiers from the intersection of those extracted entities between code and test patches\([Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4)\)\. Although Doc2Feat has the strictest identifier selection across available pipelines, occurrence in both changes does not distinguish where an identifier is defined from where it is used, leading to including irrelevant information such as field accesses and general imports\.

Thus, we propose DefUse, where the selected identifiers are defined and used\. Figure[2](https://arxiv.org/html/2610.22760#S1.F2)demonstrates DefUse pictorially\. If first uses the difference \(aka\. the patch\) between old and new to find newly introduced names:cnew−coldc\_\{\\textrm\{new\}\}\-c\_\{\\textrm\{old\}\}andtnew−toldt\_\{\\textrm\{new\}\}\-t\_\{\\textrm\{old\}\}\. Then, DefUse keeps only those names that carry a communication between code and tests, in one of two directions:

1. 1\.def​\(cnew−cold\)∩use​\(tnew−told\)\\textrm\{def\}\(c\_\{\\textrm\{new\}\}\-c\_\{\\textrm\{old\}\}\)\\cap\\textrm\{use\}\(t\_\{\\textrm\{new\}\}\-t\_\{\\textrm\{old\}\}\): The code defines the name and the test uses it;∪\\cup
2. 2\.def​\(tnew−told\)∩use​\(cnew−cold\)\\textrm\{def\}\(t\_\{\\textrm\{new\}\}\-t\_\{\\textrm\{old\}\}\)\\cap\\textrm\{use\}\(c\_\{\\textrm\{new\}\}\-c\_\{\\textrm\{old\}\}\): The test defines the name and the code uses it\.

These are the names through which the solution and its tests communicate\. The held\-out tests have already committed to them, and the agent cannot recover them from the issue alone\.

#### Why defined and used?

The criterion follows from what a held\-out test can actually observe\. A test rejects a functionally correct patch only through the identifiers it references in that patch: the functions it imports and calls, the classes it instantiates, the attributes and return structures it inspects\. This yields a necessity argument and a sufficiency argument for DefUse\.

#### Necessity\.

If the test patch reads an identifier that the code patch defines \(scenario 1\), then the identifier’s name is an unforced design choice of the reference author, yet the test treats it as a hard constraint\. Every such identifier is a potential spurious failure: an agent can implement the required behaviour perfectly and still fail on an exact\-match comparison against an arbitrary label\. These identifiers must therefore be disclosed, or the benchmark scores guesswork rather than capability\. The symmetric direction \(scenario 2\) is a contract in the same sense: the test constructs an input the code must consume, and the code\-side name is equally unrecoverable and equally load\-bearing\.

#### Sufficiency\.

Conversely, an identifier that is not both defined in one patch and used in the other never crosses the code–test boundary\. The agent may name it, place it, or eliminate it freely without affecting any test outcome\. Disclosing it repairs no ambiguity, where it potentially leaks the reference author’s internal design decisions, which is precisely the spoilers we aim to avoid\. DefUse is thus the minimal selection that removes name\-mismatch artifacts\. Every selected identifier can cause a false failure if hidden, and no discarded identifier can\.

The goal of identifier hints is to address ambiguity in the current coding tasks\. We aim to provide agents as few hints as possible to avoid unnecessary information leakage because exposing too much information will make the problems lose the functionality of evaluating coding agents’ true capabilities\. DefUse follows this principle; it discloses the full interface contract and nothing else\.

### 3\.2Identifier Projection: Declaration

Declaration reveals, for each selected identifier, what its declaration line states and nothing more: its name; its kind \(function, class, variable, or file\); the path of the file that declares it; whether that file is newly added; and, for callables, its signature\. The principle is that a hint should reproduce what a colleague could read off a header or a stub, namely identity, kind, location, and invocation, and stop where the implementation begins\. For example, for the SWE\-bench Pro task that adds locale detection to Ansible, the Declaration entry for the new function reads

> ``` ### ‘get_best_parsable_locale‘ (function) -- ‘lib/ansible/module_utils/common/locale.py‘ [new file] Signature: ‘get_best_parsable_locale(module, preferences=None)‘ ```

which fixes the name the hidden tests import, where it must live, and how it is called, and leaves everything else to the agent\.

Declaration deliberately omits the semantic attributes that richer schemes attach to each entity: a one\-line summary of what the entity does and a paragraph describing how\. For the same function, the full hint goes on to name the shell command to run \(locale \-a\), the locales to prefer \(C\.utf8,en\_US\.utf8\), and the exception to raise, and the complete hint for this task runs to thirteen times the length of the Declaration one\. These descriptions are produced by a language model that is shown the reference patch, so they are solution\-derived by construction; the open question is how much of the solution survives into the text\. We answer it with a language\-model judge in Section[4\.6](https://arxiv.org/html/2610.22760#S4.SS6): the structural attributes carry no solution content, whereas the semantic ones describe behaviour and mechanism\. The ablation in Section[4\.5](https://arxiv.org/html/2610.22760#S4.SS5)shows that structural attributes deliver a resolve\-rate gain at no disclosure, while the further gain from the semantic ones is bought with disclosed solution content\. Declaration is therefore the largest projection we can justify as removing guesswork without supplying spoilers\.

## 4Experiments

Our experiments examine how hint construction affects task resolution and what information it reveals about the reference solution\. We first isolate the effects of identifier selection and projection by varying one while holding the other fixed \(RQ1 & RQ2\)\. Then, we examine the attributes underlying these policies through ablations and an LLM\-based assessment \(RQ3 & RQ4\)\. Together, these experiments assess the rationale for combining DefUse selection with Declaration projection\. We also repeat the selection and projection comparisons on Java to validate whether the patterns observed in Python extend to a second language\.

- •RQ1\(selection, Section[4\.3](https://arxiv.org/html/2610.22760#S4.SS3)\): For which identifiers should a hint expose information?
- •RQ2\(projection, Section[4\.4](https://arxiv.org/html/2610.22760#S4.SS4)\): What information should a hint expose for each identifier?
- •RQ3\(ablation, Section[4\.5](https://arxiv.org/html/2610.22760#S4.SS5)\): Which attributes of a hint carry its effect on task resolution?
- •RQ4\(judge, Section[4\.6](https://arxiv.org/html/2610.22760#S4.SS6)\): Which attributes of a hint reveal the reference solution?

### 4\.1Implementation Details

Our pipeline extracts, for every instance, the identifiers introduced or touched by the reference patch and the test patch together with their attributes, and renders a hint from a selection rule and a projection\. To validate the extraction we reimplemented the published hint pipelines of Doc2Feat\-bench\([Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4)\)and FEA\-Bench\([Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3)\)and reproduced their released hints, with minor bug fixes\. We also implemented our version of the SWE\-Bench Pro hints pipeline by first extracting desired interfaces, and then using Claude Haiku 4\.5\([Anthropic, 2025](https://arxiv.org/html/2610.22760#bib.bib24)\)to generate requirements and the interface descriptions\. Details available in Appendix[A](https://arxiv.org/html/2610.22760#A1)\.

### 4\.2Settings

#### Benchmarks, evaluation subsets, and selection statistics\.

We use four Python benchmarks: Doc2Feat\-bench\([Deng et al\., 2025a](https://arxiv.org/html/2610.22760#bib.bib4)\), FEA\-Bench\([Li et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib3)\), SWE\-bench Pro \(denoted as Pro in tables\)\([Deng et al\., 2025b](https://arxiv.org/html/2610.22760#bib.bib5)\), and SWE\-rebench\([Badertdinov et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib6)\)\. We additionally evaluate Java tasks from Multi\-SWE\-bench \(denoted as MSB Java\)\([Zan et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib17)\)\. For the selection comparisons \(RQ1\), we retain instances for which both the Doc2Feat and FEA\-Bench Brief selection policies produce non\-empty identifier sets\. All selection policies are evaluated on this shared subset, comprising3131,4040,131131, and2626instances from the four Python benchmarks, respectively, and4040Java instances\. For the projection comparisons \(RQ2\), we retain instances for which DefUse produces a non\-empty identifier set\. These subsets contain2121,4444,6868, and1414instances from the four Python benchmarks, respectively, and2929Java instances\. The attribute ablations \(RQ3\) and LLM\-based assessment \(RQ4\) focus on the same 68\-instance SWE\-bench Pro subset\. Table[2](https://arxiv.org/html/2610.22760#S4.T2)summarizes the number of identifiers disclosed by each selection policy\. DefUse discloses the fewest identifiers per instance across all five benchmarks, while Pro discloses the most\.

Table 2:Number of identifiers disclosed per selection method, counted on the same instances as the selection tables under the Declaration projection\.*Total*is the number of identifiers over all instances and*Mean*the number per instance\. DefUse is empty on the instances whose tests reference no new identifier: it is non\-empty on 20/31, 44/142, 66/131, 9/26 and 29/40 instances, with 3\.2, 1\.8, 2\.4, 2\.0 and 6\.7 identifiers per non\-empty instance\. The other selections are non\-empty on every instance\.
#### Models and agent\.

We run three language models through the same harness: DeepSeek V4 Flash \(0431\), MiniMax M2\.5, and GPT 5\.6 Luna\([Xu et al\., 2026](https://arxiv.org/html/2610.22760#bib.bib26);[MiniMax, 2026](https://arxiv.org/html/2610.22760#bib.bib25);[OpenAI, 2026a](https://arxiv.org/html/2610.22760#bib.bib27)\)\. All models use temperature1\.01\.0, with maximum reasoning effort for DeepSeek and MiniMax and high reasoning effort for GPT 5\.6 Luna\. We use mini\-SWE\-agent\([Yang et al\., 2024](https://arxiv.org/html/2610.22760#bib.bib23)\), which issues one shell command per turn, with a limit of 250 steps per trajectory\. Due to budget constraints, we run one trajectory per instance for each configuration\.

#### Hint attributes\.

We parse each patch to extract the structural attributes, namely name, kind, path, new\-file status and signature\. The two semantic attributes mirror the human\-written fields of SWE\-bench Pro:*weak semantics*corresponds to the description under itsinterface, and*strong semantics*to itsrequirements\. Both are generated by Claude Haiku 4\.5\([Anthropic, 2025](https://arxiv.org/html/2610.22760#bib.bib24)\)given the reference patch, and both appear only in the Pro projection\.

#### Evaluation and metrics\.

We evaluate each benchmark using its official evaluator in Docker containers\. An instance counts as resolved only if all of its fail\-to\-pass and pass\-to\-pass tests pass, and we report the resolve rate, the percentage of resolved instances\.

### 4\.3Identifier Hint Selection

RQ1 examines which identifiers a hint should disclose\. We compare four selection policies under two fixed projections: Declaration \(Table[3](https://arxiv.org/html/2610.22760#S4.T3)\), which exposes structural interface attributes, and Name Only \(Appendix[B](https://arxiv.org/html/2610.22760#A2), Table[9](https://arxiv.org/html/2610.22760#A2.T9)\), which exposes only identifier names\. Because DefUse restricts selection to identifiers through which the implementation and tests communicate \(Section[3\.1](https://arxiv.org/html/2610.22760#S3.SS1)\), these comparisons test whether disclosing additional identifiers improves task resolution\.

The selection policies differ substantially in how many identifiers they disclose \(Table[2](https://arxiv.org/html/2610.22760#S4.T2)\)\. On SWE\-bench Pro, DefUse reveals an average of 1\.2 identifiers per instance, compared with 4\.8 for Doc2Feat, 7\.7 for FEA, and 17\.2 for Pro\. For GPT 5\.6 Luna, this substantial difference in disclosure corresponds to a relatively small difference in resolution: DefUse resolves 91 of 131 instances \(69\.5%\), compared with 92–95 \(70\.2–72\.5%\) for the broader selections\. Thus, for this model and benchmark, the narrower selection substantially reduces disclosure with only a small decrease in observed resolution\.

The performance cost of narrower disclosure is larger for the open\-source models onSWE\-bench Pro\. DeepSeek resolves 17 instances with DefUse, compared with 26–29 for the broader selection policies, while MiniMax resolves 20, compared with 21–29\. These gains reach 12 and 9 additional instances, respectively, compared with at most 4 for GPT 5\.6 Luna\. However, the benefit of additional identifiers varies across benchmarks\. On FEA\-Bench, DeepSeek resolves 16 instances with DefUse, compared with 13–14 for the broader policies, while MiniMax resolves 14, compared with 13–15\. On Multi\-SWE\-bench Java, DeepSeek resolves 13 of 40u instances with DefUse under Declaration, compared with 12–15 for the other policies \(Table[5](https://arxiv.org/html/2610.22760#S4.T5)\)\. Doc2Feat\-bench remains challenging for both open\-source models, with DeepSeek resolving 0–2 instances and MiniMax resolving one under every policy\. Together, these results indicate that the benefit of broader disclosure depends on both the model and the benchmark\.

Broader selection can therefore improve measured resolution, but its usefulness to an agent does not establish that the additional identifiers are necessary to specify the interface required by the tests\. Our motivation for DefUse is to expose those test\-facing identifiers while limiting disclosure of reference\-specific implementation choices\.

Table 3:Model performance across benchmarks and selection methods\. We choose Declaration projection as the fixed projection given different identifiers\.Table 4:DeepSeek V4 Flash performance on Multi\-SWE\-bench Java across selection policies under Declaration projection\.Table 5:Hint attribute ablation with DeepSeek V4 Flash on SWE\-bench Pro \(68 instances\) under DefUse selection\.

### 4\.4Identifier Hint Projection

RQ2 examines what information a hint should disclose about each selected identifier\. We hold selection fixed at DefUse and compare five projections against an empty baseline \(Table[6](https://arxiv.org/html/2610.22760#S4.T6)\)\. This comparison examines how structural interface attributes and additional semantic descriptions affect task resolution\.

The results show that structural interface information can help agents resolve tasks\. On SWE\-bench Pro, Declaration increases resolve rates over the empty baseline from 10\.3% to 29\.4% for DeepSeek, from 2\.9% to 14\.7% for MiniMax, and from 55\.9% to 73\.5% for GPT 5\.6 Luna\. It also improves over Name Only for all three models on this benchmark, suggesting that attributes beyond identifier names contribute useful interface information\. These improvements occur without the semantic descriptions included in the Pro projection\. The Java results provide supporting evidence: Declaration increases DeepSeek’s resolved instances from 5 to 10 out of 29 \(Table[7](https://arxiv.org/html/2610.22760#S4.T7)\)\.

The effect of adding semantic descriptions varies across models\. On SWE\-bench Pro, MiniMax resolves 16 instances with Pro, compared with 10 under Declaration\. DeepSeek resolves 19 with Pro and 20 with Declaration, while GPT 5\.6 Luna resolves 39 and 50, respectively\. Thus, additional reference\-derived descriptions can improve resolution, but their benefit is not consistent across models\. These comparisons alone do not establish whether the descriptions help by clarifying requirements or by supplying implementation guidance\. We also interpret small differences cautiously, given the single trajectory per instance and the limited sizes of some evaluation subsets\.

Together, these observations support Declaration as a reasonable projection for identifier hints: it provides interface information that can improve task resolution while omitting semantic descriptions derived from the reference solution\. The attribute ablations and LLM\-based assessment \(Sections[4\.5](https://arxiv.org/html/2610.22760#S4.SS5)and[4\.6](https://arxiv.org/html/2610.22760#S4.SS6)\) further examine the contributions of individual attributes and the solution information they disclose\.

Table 6:Model performance across benchmarks and projection methods\. We choose DefUse selection as the fixed selection given different projections\.*Declaration\+Doc*adds a one\-sentence description of each identifier to Declaration\.Table 7:Model performance on Multi\-SWE\-bench Java across projection methods\. We choose DefUse selection as the fixed selection given different projections\.*Declaration\+Doc*adds a one\-sentence description of each identifier to Declaration\. Model: DeepSeek V4 Flash\.
### 4\.5Hint Attribute Ablation

RQ3 examines how different hint attributes contribute to task resolution\. We fix selection to DefUse and evaluate DeepSeek V4 Flash on the SWE\-bench Pro subset \(Table[5](https://arxiv.org/html/2610.22760#S4.T5)\)\. The rows up to*Declaration*successively add structural attributes\.*Name Only and Semantics*combines names with the two semantic descriptions, while*All Hints*combines Declaration with both descriptions\.

Names improve resolution over the empty baseline, and Declaration provides a further gain\. Adding entity type alone to names yields no observed improvement, suggesting that structural attributes differ in their contribution\. Semantic descriptions produce larger gains in these runs: combining them with Declaration increases resolution from 29\.4% to 72\.1%\. The combination also outperforms names with semantic descriptions, suggesting that structural and semantic information can provide complementary benefits\.

These results distinguish the usefulness of hint attributes from their appropriateness for evaluation\. Interface attributes can help agents satisfy the test\-facing contract, while semantic descriptions may also supply implementation guidance\. Section[4\.6](https://arxiv.org/html/2610.22760#S4.SS6)examines this distinction\. Each configuration uses a single trajectory per instance, and the recorded variation across runs limits attribution of the observed differences solely to hint content\. The semantic configurations also use a different rendering from Pro in RQ2, so their absolute rates should not be directly compared with Table[6](https://arxiv.org/html/2610.22760#S4.T6)\.

### 4\.6Hint Leakage Judge

RQ4 examines what information hint attributes disclose, complementing the task\-resolution experiments\. Following prior work on LLM\-based evaluation\([Zheng et al\., 2023](https://arxiv.org/html/2610.22760#bib.bib21);[Liu et al\., 2023](https://arxiv.org/html/2610.22760#bib.bib22)\), we use GPT 5\.6 Luna with reasoning effort medium to assess attributes on SWE\-bench Pro\. The rubric scores necessityNN\(unrecoverability without the reference solution\), disclosureDD\(solution content\), and guidanceGG\(direction of the agent’s actions\)\. We summarize these ordinal scores using the heuristic hint value marginHVM=N−max⁡\(D,G\)\\mathrm\{HVM\}=N\-\\max\(D,G\)\. The rubric and assessment procedure appear in Appendix[C](https://arxiv.org/html/2610.22760#A3)\.

Table[8](https://arxiv.org/html/2610.22760#S4.T8)shows similar average necessity scores for structural and semantic attributes, but greater disclosure and guidance for semantic descriptions\. Names and signatures receive the highest margins, whereas the semantic attributes have margins near zero\. Strong semantics receives higher disclosure and guidance scores than weak semantics, consistent with descriptions that convey implementation mechanisms\.

Together with the projection comparison and attribute ablation \(Sections[4\.4](https://arxiv.org/html/2610.22760#S4.SS4)and[4\.5](https://arxiv.org/html/2610.22760#S4.SS5)\), this assessment supports Declaration as a reasonable projection for identifier hints\. The experiments show that structural interface attributes can improve task resolution without semantic descriptions, while the judge associates those descriptions with greater solution disclosure and implementation guidance\. Declaration therefore provides a principled boundary for hint construction\. It clarifies test\-facing interfaces while limiting additional reference\-derived information that may supply part of the implementation being evaluated\.

Table 8:Hint attribute scores on SWE\-bench Pro under DefUse selection\.NN: necessity;DD: disclosure;GG: guidance;HVM=N−max⁡\(D,G\)\\mathrm\{HVM\}=N\-\\max\(D,G\)\.∗marks margins indistinguishable from zero\.HVM\\mathrm\{HVM\}: Hint Value Margin\. Judge details appear in Appendix[C](https://arxiv.org/html/2610.22760#A3)\.AttributenNNDDGGHVMStructural \(interface\-level\)Signature312\.450\.000\.00\+2\.45\+2\.45Name1601\.880\.000\.11\+1\.76\+1\.76Entity type1601\.540\.000\.00\+1\.54\+1\.54Is new file580\.620\.000\.00\+0\.62\+0\.62Path581\.190\.001\.00\+0\.19\+0\.19∗*group mean*4671\.560\.000\.16\+1\.39\\mathbf\{\+1\.39\}Semantic \(behaviour\-level\)Weak semantics1601\.191\.090\.29\+0\.03\+0\.03∗Strong semantics1601\.821\.811\.68−0\.13\-0\.13∗*group mean*3201\.501\.450\.99−0\.05\\mathbf\{\-0\.05\}

## 5Conclusions and Limitations

We study identifier hints through selection and projection, showing that hint policies affect both measured task resolution and the reference\-derived information available to agents\. Our findings motivate combining DefUse selection with Declaration projection to clarify test\-facing interfaces while limiting implementation guidance\. More broadly, benchmark designers should assess hints by what they disclose as well as the performance gains they produce\.

#### Limitations\.

Budget constraints limited each reported configuration to one trajectory per instance, making small differences sensitive to stochastic variation\. The recorded experiments represent approximately $2,300 in inference value, including unbilled MiniMax usage valued at public API rates\. Small benchmark subsets and a single agent harness limit generalizability, while serving variation and rubric\-dependent LLM judgments introduce additional uncertainty\.

## References

- Aleithanet al\.\(2024\)R\. Aleithan, H\. Xue, M\. M\. Mohajer, E\. Nnorom, G\. Uddin, and S\. WangSWE\-Bench\+: enhanced coding benchmark for LLMs\.arXiv preprint arXiv:2410\.06992\.Cited by:[§1](https://arxiv.org/html/2610.22760#S1.p2.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2025\)AnthropicIntroducing Claude Haiku 4\.5\.External Links:[Link](https://www.anthropic.com/news/claude-haiku-4-5)Cited by:[§4\.1](https://arxiv.org/html/2610.22760#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px3.p1.1)\.
- Badertdinovet al\.\(2026\)I\. Badertdinov, A\. Golubev, M\. Nekrashevich, A\. Shevtsov, S\. Karasik, A\. Andriushchenko, M\. Trofimova, D\. Litvintseva, and B\. YangelSWE\-rebench: an automated pipeline for task collection and decontaminated evaluation of software engineering agents\.Advances in Neural Information Processing Systems38\.Cited by:[§1](https://arxiv.org/html/2610.22760#S1.p2.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px1.p1.1)\.
- Denget al\.\(2025a\)L\. Deng, Z\. Jiang, J\. Cao, M\. Pradel, and Z\. LiuNocode\-bench: a benchmark for evaluating natural language\-driven feature addition\.arXiv preprint arXiv:2507\.18130\.Cited by:[§1](https://arxiv.org/html/2610.22760#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22760#S1.p1.1),[§1](https://arxiv.org/html/2610.22760#S1.p2.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.22760#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.22760#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px1.p1.1)\.
- Denget al\.\(2025b\)X\. Deng, J\. Da, E\. Pan, Y\. Y\. He, C\. Ide, K\. Garg, N\. Lauffer, A\. Park, N\. Pasari, C\. Rane,et al\.SWE\-bench Pro: can AI agents solve long\-horizon software engineering tasks?\.arXiv preprint arXiv:2509\.16941\.Cited by:[§1](https://arxiv.org/html/2610.22760#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22760#S1.p2.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.22760#S3.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px1.p1.1)\.
- Ganhotra \(2026\)J\. GanhotraHidden naming contracts in SWE\-agent benchmarks\.External Links:[Link](https://jatinganhotra.dev/blog/swe-agents/2026/04/05/hidden-naming-contracts-in-swe-agent-benchmarks.html)Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 58791–58831\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world Github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§1](https://arxiv.org/html/2610.22760#S1.p1.1)\.
- Khatibet al\.\(2026\)L\. Khatib, N\. S\. Mathews, M\. Nagappan, P\. Nie, and T\. ZimmermannWhat makes a good bug report for an ai agent?\.arXiv preprint arXiv:2607\.07593\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Kuanget al\.\(2026\)S\. Kuang, Z\. Tian, K\. Lin, C\. Tao, S\. Wang, H\. Bai, L\. Shang, and J\. ChenREAgent: requirement\-driven llm agents for software issue resolution\.arXiv preprint arXiv:2604\.06861\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)W\. Li, X\. Zhang, Z\. Guo, S\. Mao, W\. Luo, G\. Peng, Y\. Huang, H\. Wang, and S\. LiFEA\-Bench: a benchmark for evaluating repository\-level code generation for feature implementation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17160–17176\.Cited by:[§1](https://arxiv.org/html/2610.22760#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22760#S1.p1.1),[§1](https://arxiv.org/html/2610.22760#S1.p2.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.22760#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.22760#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px1.p1.1)\.
- Lianget al\.\(2026\)S\. Liang, S\. Garg, and R\. Z\. MoghaddamThe SWE\-bench illusion: when state\-of\-the\-art llms remember instead of reason\.InProceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice,pp\. 395–405\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§4\.6](https://arxiv.org/html/2610.22760#S4.SS6.p1.1)\.
- MiniMax \(2026\)MiniMaxMiniMax M2\.5: built for real\-world productivity\.External Links:[Link](https://www.minimax.io/news/minimax-m25)Cited by:[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px2.p1.1)\.
- OpenAI \(2024\)OpenAIIntroducing SWE\-bench Verified\.External Links:[Link](https://openai.com/index/introducing-swe-bench-verified/)Cited by:[§1](https://arxiv.org/html/2610.22760#S1.p1.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.6 Luna: Frontier Intelligence that scales with your ambition\.External Links:[Link](https://openai.com/index/gpt-5-6/)Cited by:[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px2.p1.1)\.
- OpenAI \(2026b\)OpenAISeparating signal from noise in coding evaluations\.External Links:[Link](https://openai.com/index/separating-signal-from-noise-coding-evaluations/)Cited by:[§1](https://arxiv.org/html/2610.22760#S1.p1.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2026c\)OpenAIWhy SWE\-bench verified no longer measures frontier coding capabilities\.External Links:[Link](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/)Cited by:[§1](https://arxiv.org/html/2610.22760#S1.p1.1),[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- Puet al\.\(2026\)G\. Pu, M\. S\. Lee, U\. M\. Sehwag, D\. J\. Lee, B\. Zhu, Y\. Maurya, M\. Raghavendra, Y\. Xue, and S\. M\. DentonLHAW: controllable underspecification for long\-horizon tasks\.arXiv preprint arXiv:2602\.10525\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Ruanet al\.\(2025\)H\. Ruan, Y\. Zhang, and A\. RoychoudhurySpecRover: code intent extraction via llms\.In2025 IEEE/ACM 47th International Conference on Software Engineering \(ICSE\),pp\. 963–974\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Takerngsaksiriet al\.\(2025\)W\. Takerngsaksiri, J\. Pasuksmit, P\. Thongtanunam, C\. Tantithamthavorn, R\. Zhang, F\. Jiang, J\. Li, E\. Cook, K\. Chen, and M\. WuHuman\-in\-the\-loop software development agents\.In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice \(ICSE\-SEIP\),pp\. 342–352\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Vijayvargiyaet al\.\(2026\)S\. Vijayvargiya, X\. Zhou, A\. Yerukola, M\. Sap, and G\. NeubigAmbig\-SWE: interactive agents to overcome underspecificity in software engineering\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 146716–146737\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)Y\. Wang, M\. Pradel, and Z\. LiuAre “solved issues” in SWE\-bench really solved correctly? an empirical study\.InProceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering,pp\. 169–181\.Cited by:[§2](https://arxiv.org/html/2610.22760#S2.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2026\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px2.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSwe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px2.p1.1)\.
- Zanet al\.\(2026\)D\. Zan, Z\. Huang, W\. Liu, H\. Chen, S\. Xin, L\. Zhang, Q\. Liu, L\. Aoyan, L\. Chen, X\. Zhong,et al\.Multi\-SWE\-bench: a multilingual benchmark for issue resolving\.Advances in Neural Information Processing Systems38\.Cited by:[§4\.2](https://arxiv.org/html/2610.22760#S4.SS2.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. ZhouQwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§A\.1](https://arxiv.org/html/2610.22760#A1.SS1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§4\.6](https://arxiv.org/html/2610.22760#S4.SS6.p1.1)\.

## Appendix AImplementation Details: Baselines

### A\.1Selection

For both Doc2Feat and FEA Bench Brief hint generation, we adapted the original code so that the pipeline can extract hints on various benchmarks\. For SWE Bench Pro hint generation, since they employ human engineers to build the hints, it’s impossible to completely replicate Pro’s pipeline\. Thus, we use a LLM to approximate human engineers in terms of providing detailed hint text\. We first find the different identifiers between old code and new cold patches, extract the information about identifiers, and then have Claude Haiku 4\.5 to generate interface descriptions and requirements with few shot prompts\. Comparing the generated interface descriptions and requirements with original ones in python subset of SWE Bench Pro \(266 instances\), the generated ones achieved consine similarity of0\.89780\.8978and0\.84050\.8405respectively, using Qwen3\-0\.6\-Embedding\([Zhang et al\., 2025](https://arxiv.org/html/2610.22760#bib.bib28)\)as the embedding model\.

### A\.2Projection

To implement controllable projection, we first have a complete pipeline that extracts all possible identifiers from both the code patches and the test patches\. For each identifier, we extract relevant information by checking its abstract syntax tree to get the information not just about the identifier name\. Then, we select DefUse as our selection by performing the communication between code and test patches, and the we adopt Claude Haiku 4\.5 to enrich the semantics, where weak semantics corresponds to SWE Bench Pro interface descriptions, and strong semantics corresponds to requirements\. Then:

- •Doc2Feat: the name and entity type of each identifier, rendered as a flat list; no path, signature or description\.
- •Declaration: name, entity type, file path, a new\-file flag and the signature; class members are nested under their class\. No natural\-language text\.
- •Declaration\+Doc: Declaration plus the weak semantics, a one\-sentence \(at most 20 words\) description of what the entity is and does\. It corresponds to the docstring that FEA\-Bench’s detailed hints add, without their code; FEA\-Bench’s brief hints carry no description and coincide with Declaration\.
- •Pro: Declaration\+Doc plus the strong semantics, a two\-to\-four sentence paragraph on behaviour, edge cases and test\-implied constraints\. Both descriptions are written by Claude Haiku 4\.5 from the reference patch, so the LLM acts as the extractor that SWE\-bench Pro’s human annotators were\.

## Appendix BIdentifier Selection under Name Only Projection

We repeat the identifier selection comparison with the projection fixed at Name Only\. Each hint reveals only the names of the selected identifiers, allowing us to examine selection policies without disclosing additional attributes\. Tables[9](https://arxiv.org/html/2610.22760#A2.T9)and[10](https://arxiv.org/html/2610.22760#A2.T10)report the Python and Java results, respectively\.

The effect of selection varies across models and benchmarks even when hints contain only names\. On SWE\-bench Pro, Pro and FEA outperform DefUse for all three models, whereas DefUse leads the hint selections for DeepSeek on FEA\-Bench\. The Java results provide a further example: DefUse and Pro each resolve 9 of 40 instances, compared with 6 for Doc2Feat and FEA\. Thus, broader selection can help, but disclosing additional names does not consistently improve resolution\. As in the Declaration comparison, small differences should be interpreted cautiously given the limited evaluation subsets and single trajectory per instance\.

Table 9:Model performance across benchmarks and selection methods\. We choose NameOnly Bench projection as the fixed projection given different identifiers\.Table 10:DeepSeek V4 Flash performance on Multi\-SWE\-bench Java across selection policies under Name Only projection\.
## Appendix CHint Leakage Judge Details

#### Rubric\.

Each attribute is assessed on three ordinal scales\. NecessityNNmeasures how recoverable its value is without the reference solution: 0 if stated in the issue, 1 if findable in the repository, 2 if constrained by convention, and 3 if freely chosen by the reference author\. DisclosureDDmeasures solution content: 0 for identity and addressing, 1 for the observable contract, 2 for an implementation mechanism, and 3 for a transcription\. GuidanceGGmeasures direction of the agent’s actions: 0 for declarative information, 1 for orientation, 2 for procedural guidance, and 3 for prescriptive instructions\. Disclosure and guidance are distinct: a file path can orient the agent without describing an implementation\. The heuristic marginHVM=N−max⁡\(D,G\)\\mathrm\{HVM\}=N\-\\max\(D,G\)combines these judgments; it is not a calibrated measure of information quantity\.

#### Assessment procedure\.

We assess 787 identifier–attribute pairs across 160 identifiers selected by DefUse on 68 SWE\-bench Pro instances\. Necessity is assessed using the issue without showing the reference or test patches; disclosure and guidance are assessed with both patches available\. Each call presents all attributes of an identifier and specifies one target attribute, so the necessity judgment is conditional on the other visible attributes\. The judge is GPT 5\.6 Luna with medium reasoning effort\. It returns a category label, an evidence span, and a rationale\. We map labels to integers and take the median of three samples at temperature 1\. We estimate uncertainty using instance\-clustered bootstrap and measure consistency using three\-sample unanimity\.

#### Validation and limitations\.

We check repository membership for a subset of identifier names usinggit grepat the base commit\. When supplied with this evidence, the judge reproduces the repository\-membership distinction for all 31 identifiers checked, compared with 18 without it\. This check validates a limited aspect of the necessity assessment; it does not validate the full rubric\. Agreement also varies across attributes: three\-sample unanimity is 94% for signatures but 41–42% for semantic descriptions\. These differences indicate greater judgment uncertainty for semantic content and should be considered alongside the aggregate scores\.

#### Rubrics and prompts\.

The three blocks below reproduce, verbatim from the judge implementation, the rubric and the prompt skeleton behind each column of Table[8](https://arxiv.org/html/2610.22760#S4.T8)\. Necessity is one call per \(instance, identifier, attribute\) triple that sees the issue but neither patch; disclosure and guidance are rated together in a second call that also sees the reference and test patches\. Angle\-bracketed fields are filled per call\. The judge returns category labels only; the integers in the table are assigned in code \(REDUNDANT→\\to0, …,LOAD\-BEARING→\\to3, and likewise for the other two scales\)\. All three calls share the system prompt shown at the top of the first block\.

\[system\]

Youareanexactingresearchassistantevaluatinghintqualityforasoftware

engineeringbenchmark\.Youapplyarubricliterallyandconsistently\.Youdonot

rewardvaluesforbeingdetailedoruseful;youclassifythembythestated

criteriaonly\.YoualwaysanswerwithasingleJSONobjectandnothingelse\.

\[user\]

RateNECESSITY:couldacompetentengineerobtainthisattribute’svalueWITHOUT

seeingthereferencesolution?

0REDUNDANT\-thevalueisstatedintheproblemstatementitself\.

1SEARCH\-SAVING\-notstated,butanengineerwiththerepositoryatitscurrent

commitwouldreachitthroughnormalinvestigation\(theentity

alreadyexists;grep/readingfindsit\)\.Thehintsaveseffort,

notinformation\.

2DISAMBIGUATING\-notstatedandnotalreadypresent,butstronglyconstrainedby

repositoryconvention\(e\.g\.anewtestfile’slocationfollows

anobviousexistinglayout\)\.Anengineerwouldlikelyguessit\.

3LOAD\-BEARING\-encodesafreechoicethesolution’sauthormadethatisnot

recoverablefromtheproblemstatementortheexistingcodebase

\(e\.g\.thenameofafunctionthatdoesnotyetexist\)\.Without

thehintanengineerwouldpicksomethingelse\.

JudgeONLYthetargetattribute’svalue\.Ignorewhetherthevalueseemsimportant;

necessityisaboutRECOVERABILITY,notimportance\.

\#\#Measuredrepositoryfact\(presentonlywhenthetargetis‘name‘\)

Theidentifier‘<name\>‘appearsin<k\>file\(s\)inthisrepositoryatthecommit

theagentstartsfrom\.

<k\>0:”Anengineercouldthereforefinditbysearchingtheexistingcode\.”

k=0:”Itdoesnotexistanywhereinthetreeyet,soitcannotbefoundby

searching;anyengineerwouldhavetochooseaname\.”\>

\#\#Repository

<owner/repo\>

\#\#Problemstatement\(thisisALLtheagentistold\)

<issuetext,first6000characters\>

\#\#Theidentifier,forcontext

name:<…\>

entity\_type:<…\>

path:<…\><<<TARGET\(oneattributeismarked\)

is\_new\_file:<…\>

signature:<…\>

weak\_semantics:<…\>

strong\_semantics:<…\>

\#\#Thevalueyouarerating

attribute:‘<targetattribute\>‘

value:“‘

<thetargetattribute’svalue,alone\>

“‘

YouareratingTHETEXTABOVE\-nottheentityitrefersto,andnottheother

attributeslistedforcontext\.Askwhatthistext,byitself,tellsareader\.

\#\#Yourtask

RateNECESSITY\.

RespondwithexactlythisJSONobject:

\{”label”:”<REDUNDANT\|SEARCH\-SAVING\|DISAMBIGUATING\|LOAD\-BEARING\>”,

”evidence”:”<ashortverbatimquotefromtheproblemstatementthatjustifies

yourrating,ortheemptystringifthestatementsaysnothingrelevant\>”,

”rationale”:”<onesentence\>”,

”confidence”:<0\.0\-1\.0\>\}

RateDISCLOSURE:howmuchofthereferencesolution’sCONTENTdoesthisvaluecarry?

0INTERFACE\-identityandaddressingonly:whatitiscalled,whereitlives,

howitisinvoked\.Thisisthelinethebenchmarkitself

publishestoagents,soitisnotleakage\.

1CONTRACT\-observablebehaviour:whatitdoes,whatitsinputsandoutputs

mean\.Aspecificationsomeonecouldimplementmanyways\.

2MECHANISM\-howitdoesit:controlflow,algorithm,orderingofoperations,

orthenamesofspecifichelpers/APIsitcallsinternally\.

3TRANSCRIPTION\-restatesthesolutioncloselyenoughthatthecodecouldbe

transliteratedfromthetextwithlittleindependentthought\.

Exampleof2:”overridesadd\_jobtocheckthehostnameagainstallowedhostsusing

fnmatchpatterns”\-namesthemethodandthematchinglibrary\.

Exampleof1:”Asetofexactkeynamesthatshouldnotbesanitizedevenifthey

containsensitivesubstrings”\-statesthecontractwithoutthemechanism\.

RateBEHAVIOURALGUIDANCE:howmuchdoesthisvalueDIRECTtheagent’sactions,as

opposedtomerelystatingafactitmustactupon?

0DECLARATIVE\-statesthatsomethingexistsorwhatitiscalled\.Noaction

isimplied\.

1ORIENTING\-narrowswheretolookorwhattotouch,withoutsayingwhattodo\.

2PROCEDURAL\-impliesasequenceofstepsoraspecificapproachtotake\.

3PRESCRIPTIVE\-readsasaninstructiontheagentcanfollowdirectly;thetext

couldsubstituteforaplan\.

ThisisINDEPENDENTofdisclosure\.Alinerangedisclosesalot\(exactlywherethe

changegoes\)butguidesnothing\(itdoesnotsaywhattodo\)\.Analgorithm

descriptiondoesboth\.Scorethemseparately\.

——————————————————————————

\[usermessageoftheD/Gcall:theDISCLOSURErubric,thentheGUIDANCErubric,

thenthefollowing\]

\#\#Problemstatement

<issuetext,first4000characters\>

\#\#Referencesolution\(goldpatch\)

“‘diff

<goldpatch,first14000characters\>

“‘

\#\#Testpatch\(defineswhatmustpass\)

“‘diff

<testpatch,first6000characters\>

“‘

\#\#Theidentifier,forcontext

<allattributesoftheidentifier,targetmarked”<<<TARGET”,asabove\>

\#\#Thevalueyouarerating

<thetargetattribute’svalue,alone,asabove\>

\#\#Yourtask

RateDISCLOSUREandBEHAVIOURALGUIDANCEforthevalueabove,bycomparingit

againstthereferencesolution\.

RespondwithexactlythisJSONobject:

\{”disclosure\_label”:”<INTERFACE\|CONTRACT\|MECHANISM\|TRANSCRIPTION\>”,

”guidance\_label”:”<DECLARATIVE\|ORIENTING\|PROCEDURAL\|PRESCRIPTIVE\>”,

”evidence”:”<thespanofthetargetvaluethatcarriesthemostsolutioncontent,

verbatim\>”,

”rationale”:”<onesentence\>”,

”confidence”:<0\.0\-1\.0\>\}