Good measurement work on whether retrieved agent skills actually help.

They report that agent skills that lift your aggregate score can be hurting every task they touch.

The usual way of checking compares tasks where a skill was retrieved against tasks where none was.

Those are different tasks, so the comparison mixes the effect of retrieval with the effect of which tasks trigger it.

The fix presented in the paper is a matched comparison. Retrieval-Invoked Actual-Use Effect runs the same task twice, once with skills enabled and once disabled, and counts only tasks where the agent actually retrieved something.

Across 17 LLMs on coding and math, models frequently show positive aggregate retrieval lift alongside a negative same-task effect.

On MBPP+, several models that look beneficial system-wide are hurting themselves on exactly the tasks where retrieval fired.

Anyone maintaining a skills directory can run this against their own stack today.

Paper: https://arxiv.org/abs/2609.00549

Chat with Paper: https://academy.dair.ai/papers/skill-following-evaluating-actual-skill-use-in-retrieval-enabled-llm-agents-2609.00549…

Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents

Source: https://arxiv.org/html/2609.00549

Abstract

Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability—which we formalize asSkill Following(SF)—we introduce theRetrieval-Invoked Actual-Use Effect(RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.

††footnotetext:Corresponding authors.Figure 1:Aggregate retrieval lift and RAE on MBPP+. The x-axis compares skill-enabled accuracy between retrieval-returned and retrieval-skipped task sets, whereas the y-axis compares paired skill-enabled and skill-disabled outcomes on the same retrieval-returned tasks. Shaded off-diagonal quadrants indicate sign disagreement; labels (a)–(c) mark the cases discussed in the text.## 1Introduction

Large language model (LLM) agents are increasingly deployed within external procedural environments, utilizing tools, memory, and skill libraries to navigate and solve complex tasksSchick et al. (2023);Wu et al. (2024);Shen et al. (2024);Shi et al. (2024). As these models transition from standalone text predictors to retrieval-enabled, interactive agents, the evaluation paradigms used to assess their efficacy must evolve correspondingly. However, measuring the true utility of external skills remains fundamentally challenging. Existing evaluation protocols typically answer one of two broad, system-level questions: whether providing an agent with access to a skill environment improves average performance across an entire benchmark, or whether tasks where skills were retrieved yield higher accuracy than tasks where they were skipped. While these aggregate metrics provide a high-level summary of system behavior, they leave a critical, narrower question systematically unanswered:when an agent actually chooses to retrieve and inject a skill into its context, does that specific external knowledge genuinely improve the outcome of that precise task?

We argue that aggregate benchmark averages inherently obscure the true causal utility of skill retrieval, often creating a misleading illusion of capability. Whole-benchmark metrics dilute the impact of external tools by heavily weighting tasks where no skills were ever fetched. More problematically, aggregate retrieved-vs-skipped comparisons suffer from severe task-selection bias; they compare two self-selected subsets of tasks that may fundamentally differ in inherent difficulty, prompt complexity, or skill-library relevance. Because autonomous retrieval is merely the preamble of tool use—requiring the LLM to subsequently receive, interpret, logically ground, and synthesize the returned content into a valid final answer—evaluating this complete pipeline demands a more rigorous methodology. We formalize this multi-step, retrieval-to-answer cognitive process asSkill Following(SF). To accurately evaluate SF without the confounding variables of task self-selection, the measurement must be explicitly conditioned on the actual occurrence of retrieval, utilizing a strictly paired experimental design.

To clearly illustrate the severity of this evaluation gap, Figure1contrasts the conventional aggregate retrieval lift against our proposed same-task metric on the MBPP+ benchmark. The x-axis represents the aggregate lift—the standard approach that suggests many modern LLMs, such as Gemini-2.5-Flash-lite and DeepSeek-V3.2, achieve substantial performance gains when skills are retrieved.

However, when we isolate the exact subset of tasks where these models successfully retrieved a skill and compare their outcomes strictly against their own skill-disabled baselines on those identical tasks (the y-axis), a starkly different reality emerges. The shaded quadrants denote regions of criticalsign disagreement: models positioned here exhibit a positive aggregate lift that statistically masks a negative actual-use effect. This paradox reveals that what appears to be a systemic improvement is often just a byproduct of the model selectively retrieving skills on inherently easier tasks, while the actual integration of those skills frequently disrupts the model’s reasoning, ultimately rescuing fewer outcomes than it harms.

To correct this methodological blind spot and isolate the actual-use capability of LLMs, we introduce theRetrieval-Invoked Actual-Use Effect(RAE). RAE is a protocol-conditional, same-task outcome metric computed exclusively on the tasks where retrieval was actively invoked and a skill was successfully returned in the skill-enabled execution. By mandating a paired skill-enabled versus skill-disabled comparison, RAE strips away population-level confounding factors and directly measures whether the retrieval-invoked SF chain operates as a genuine cognitive aid or a distracting bottleneck.

In summary, this paper makes the following primary contributions:

• •Identifying the Actual-Use Gap in Agent Evaluation:We conceptualize skill integration as the fullSkill Following(SF) chain and mathematically demonstrate that standard aggregate metrics fail to isolate the true task-level impact of retrieved skills due to selection bias.
• •Introducing the RAE Metric:We propose the Retrieval-Invoked Actual-Use Effect (RAE), establishing a rigorous, paired-evaluation protocol that conditions outcome measurement specifically on active retrieval events.
• •Exposing Pervasive Metric Reversals across LLMs:Across an extensive evaluation of 17 diverse LLMs spanning coding and mathematical reasoning domains, we reveal that aggregate conclusions frequently reverse under RAE. Crucially, our comprehensive diagnostic analyses prove that mere context exposure to retrieved skills is vastly insufficient for successful Skill Following, challenging current assumptions about LLM tool-use capabilities.

2Related Work

LLM agents are increasingly evaluated not merely as standalone text predictors, but as complex systems integrating external tools, memory, and retrieval mechanisms. Benchmarks such as AgentBenchLiu et al. (2024), AgentBoardMa et al. (2024), WebArenaZhou et al. (2024), OSWorldXie et al. (2024), TheAgentCompanyXu et al. (2026), andτ\tau-benchYao et al. (2024)measure end-to-end agent behavior in interactive or environment-grounded settings. While these benchmarks provide crucial system-level evaluations, their primary metrics focus on overall task success or step-wise process diagnostics. Consequently, they do not isolate whether the injection of retrieved external content directly changed the final outcome of that exact task via paired comparisons.

Benchmarks specifically targeting external skill access provide a closer point of comparison. Datasets such as SkillsBenchLi et al. (2026), SWE-Skills-BenchHan et al. (2026), and SkillLearnBenchZhong et al. (2026)employ paired designs to contrast skill-enabled conditions against no-skill or human-skill baselines. Although this paired approach effectively controls for task identity, their reported effects are still predominantly aggregated over the entire evaluation set. In contrast, RAE precisely isolates the actual-use component of the Skill Following chain by conditioning the paired comparison strictly on the subset of tasks where retrieval was actively invoked and content was returned.

A complementary line of research investigates autonomous retrieval, memory, and skill reuse within agents, including systems like VoyagerWang et al. (2023), Agent KBTang et al. (2025), ExpeLZhao et al. (2024), ReflexionShinn et al. (2023), and CRITICGou et al. (2024). While these frameworks preserve and demonstrate agent-initiated use of external artifacts, they typically do not systematically pair each retrieval-invoked execution with a corresponding skill-disabled baseline to measure the net performance shift. Furthermore, process-level benchmarks such as BFCLPatil et al. (2025), RAGASEs et al. (2024), and IFEvalZhou et al. (2023)diagnose specific sub-stages like tool calling, retrieval quality, or grounding. RAE is fully complementary to these diagnostics: rather than replacing stage-level metrics, it evaluates the end-to-end impact of the retrieval-invoked chain, strictly asking whether the complete process ultimately rescued more same-task outcomes than it harmed.

3Skill Following

In this section, we establish the formal framework of paired outcome metrics required to rigorously evaluate the task-level efficacy of Skill Following (SF). Conceptualizing SF as the complete retrieval-to-answer cognitive chain introduced in Section1, we define these metrics through strictly paired comparisons between skill-enabled and skill-disabled executions on identical tasks.

MetricConditioning setPaired?ComparisonAggregate retrieval liftSE retrieved vs. SE non-retrievedNoMean accuracy difference between two task subsets within the skill-enabled executionOAESallS_{\mathrm{all}}YesSE–SD outcome difference over all evaluated tasksRAE (ours)ScallS_{\mathrm{call}}YesSE–SD outcome difference over retrieval-invoked tasksTable 1:Structural taxonomy of aggregate retrieval lift, OAE, and RAE. OAE and RAE are paired, same-task quantities, whereas aggregate retrieval lift compares self-selected task subsets within the skill-enabled execution.### 3.1Paired Execution Design

To construct a controlled comparison, each taskttwithin the evaluation corpus undergoes two distinct execution paths. Theskill-enabled(SE) execution grants the agent full access to the skill library and retrieval interface, whereas theskill-disabled(SD) baseline restricts this access. Letyt+,yt−∈{0,1}y^{+}_{t},y^{-}_{t}\in\{0,1\}denote the binary correctness outcomes of the SE and SD executions, respectively. The paired outcomes(yt+,yt−)∈{0,1}2(y^{+}_{t},y^{-}_{t})\in\{0,1\}^{2}map each task into one of four distinct transition categories (Table2). This strictly paired design anchors the evaluation to fixed task identities, inherently neutralizing the confounding effects of cross-population task difficulty.

SE correctSE wrongSD correctconcordant passharmful flip(c)(c)SD wronghelpful flip(b)(b)concordant failTable 2:Outcome matrix for paired skill-enabled (SE) and skill-disabled (SD) executions.bbdenotes the number of tasks that failed in SD but succeeded in SE, whereasccdenotes the converse.

3.2Overall and Retrieval-Invoked Effects

Because autonomous retrieval is a selective process, the SE execution will successfully fetch skills for certain tasks but not others. Consequently, we delineate two distinct paired metrics: theOverall Skill-Access Effect(OAE), computed over the entire evaluation corpus, and theRetrieval-Invoked Actual-Use Effect(RAE), strictly conditioned on tasks where retrieval was actively invoked and a skill was returned.

Overall skill-access effect.

LetSallS_{\mathrm{all}}denote the full evaluated task set. Equation1defines OAE as the macroscopic, paired outcome difference across the entire benchmark:

OAE=Δall\displaystyle\mathrm{OAE}=\Delta_{\mathrm{all}}=1|Sall|​∑t∈Sall(yt+−yt−)\displaystyle=\frac{1}{|S_{\mathrm{all}}|}\sum_{t\in S_{\mathrm{all}}}(y_{t}^{+}-y_{t}^{-})(1)=allb−allcnpairs,\displaystyle=\frac{\mathrm{all}_{b}-\mathrm{all}_{c}}{n_{\mathrm{pairs}}},wherenpairs=|Sall|n_{\mathrm{pairs}}=|S_{\mathrm{all}}|, andallb\mathrm{all}_{b}andallc\mathrm{all}_{c}represent the total helpful and harmful flips over the full dataset.

OAE quantifies the systemic impact of exposing the agent to a skill environment. It computes the average outcome shift across all paired executions, deliberately encompassing tasks where the SE execution bypassed retrieval entirely. While OAE accurately captures the global utility of skill access, it fundamentally dilutes the signal of actual skill usage.

Retrieval-invoked actual-use effect

To distill the genuine impact of active skill utilization, we strictly condition our measurement on the subset of tasks where the SE execution invoked retrieval and successfully acquired at least one skill. LetScallS_{\mathrm{call}}be this retrieval-invoked subset, with sizencond=|Scall|n_{\mathrm{cond}}=|S_{\mathrm{call}}|. Equation2defines RAE as the paired outcome difference isolated to this critical subset:

RAE=Δcond\displaystyle\mathrm{RAE}=\Delta_{\mathrm{cond}}=1|Scall|​∑t∈Scall(yt+−yt−)\displaystyle=\frac{1}{|S_{\mathrm{call}}|}\sum_{t\in S_{\mathrm{call}}}(y_{t}^{+}-y_{t}^{-})(2)=condb−condcncond.\displaystyle=\frac{\mathrm{cond}_{b}-\mathrm{cond}_{c}}{n_{\mathrm{cond}}}. Crucially, unlike aggregate retrieval lift—which merely contrasts self-selected retrieved and non-retrieved populations within the SE condition—RAE is an unconfounded, same-task metric. It directly addresses whether the retrieval-invoked segment of the SF chain successfully improved outcomes relative to the mathematically matched SD baselines. Table1delineates the structural distinctions between aggregate retrieval lift, OAE, and RAE.

Relationship between OAE and RAE

RAE operates as a conditional sub-component of the broader skill-access effect. LetSskip=Sall∖ScallS_{\mathrm{skip}}=S_{\mathrm{all}}\setminus S_{\mathrm{call}}denote the subset of tasks where retrieval was circumvented, with sizenskip=|Sskip|n_{\mathrm{skip}}=|S_{\mathrm{skip}}|. We define the paired effect over this uninvoked complement as:

Δskip=skipb−skipcnskip,\Delta_{\mathrm{skip}}=\frac{\mathrm{skip}_{b}-\mathrm{skip}_{c}}{n_{\mathrm{skip}}},(3)whereskipb=allb−condb\mathrm{skip}_{b}=\mathrm{all}_{b}-\mathrm{cond}_{b}andskipc=allc−condc\mathrm{skip}_{c}=\mathrm{all}_{c}-\mathrm{cond}_{c}. Combining the retrieval-invoked and skipped components yields a full decomposition of the benchmark effect:

OAE=ncondnpairs​RAE+nskipnpairs​Δskip.\mathrm{OAE}=\frac{n_{\mathrm{cond}}}{n_{\mathrm{pairs}}}\mathrm{RAE}+\frac{n_{\mathrm{skip}}}{n_{\mathrm{pairs}}}\Delta_{\mathrm{skip}}.(4) Equation4demonstrates that RAE does not supersede OAE; rather, it isolates the critical mechanism of actual usage. OAE answers the overarching system-design question, whereas RAE targets the precise actual-use inquiry. RAE is reported for each evaluated model–skill-pool–retrieval-policy configuration, whose retrieval-returned subset is system-specific. We therefore report RAE alongside its granular helpful and harmful transition counts; comprehensive details regarding statistical tests, confidence intervals, and edge-case handling are provided in AppendixA.

4Experiment Setup

Our experimental design is structured to address four primary objectives. First, we investigate whether aggregate retrieval lift and RAE yield conflicting conclusions under an identical skill-enabled protocol. Second, we analyze whether aggregate retrieval lift is confounded by retrieval-induced task selection. Third, we validate the consistency of our findings across multiple task partitions, a secondary coding benchmark, and an entirely different domain (mathematical reasoning). Finally, we conduct transition-level diagnostics and skill-content controls to rigorously bound the interpretation of negative RAE outcomes.

4.1Models and Datasets

We evaluate our proposed measurement framework across a comprehensive panel of 17 diverse LLMs. This panel spans various parameter scales and includes both closed-source APIs and open-weight models to ensure broad generalizability (see AppendixKfor complete model details).

For our primary analysis, we sample 80 tasks from the MBPP+ benchmarkLiu et al. (2023)using seed 42. To verify cross-benchmark consistency, we evaluate the models on HumanEval+Chen et al. (2021). Furthermore, to confirm that the observed phenomena are not strictly domain-dependent, we perform a cross-domain replication using 80-task partitions of Math500Hendrycks et al. (2021);Lightman et al. (2024).

4.2Skill Environment and Retrieval Mechanism

All analyses operate on fixed, procedural skill libraries tailored to the respective domains. TheCoding skill poolcomprises 9 procedural coding skills used for both MBPP+ and HumanEval+. TheMath skill poolcomprises 8 procedural skills used for the Math500 evaluation. AppendixJsummarizes both skill pools, and AppendixLreports the prompts and tool schema.

Each skill is formatted as a structured Markdown file (SKILL.md), containing lightweight frontmatter (name, description, signature) and a detailed body (application conditions, code snippets, invariants, and self-tests). The retrieval system indexes the skill descriptions and Markdown bodies using BM25Robertson and Zaragoza (2009). When an agent autonomously invokes thesearch_skills(query)function, the full texts of the top-3 retrieved skills are systematically appended to its working context. Agents are permitted up to three search invocations per task.

4.3Paired Execution Protocol

To compute the same-task metrics detailed in §3, each taskttundergoes two strictly controlled, parallel executions. In the SE condition, the LLM receives a system prompt that explicitly defines and grants access to thesearch_skills(query)tool. In the SD baseline condition, the model receives an identical prompt, but with the tool definition entirely removed. Both execution paths share the exact same task prompt, generation seed, and decoding configuration, ensuring that any outcome variance is strictly attributable to the skill access and subsequent retrieval chain.

4.4Diagnostic Controls and Annotation

To interpret the mechanistic drivers of negative RAE, we design two supplementary diagnostic setups for the strongest coding reversal models (DeepSeek-V3.2 and Gemini-2.5-Flash-lite).

Skill-Content Controls.

We run identical evaluations on MBPP+ under five perturbed environments:Normal(the standard skill library),Schema-Empty(tool is available but returns no usable skills),Filler-Dummy(matched non-informative text),Random-skills(retrieval returns structurally valid but completely irrelevant skills), andCorrupted(retrieved content is intentionally misleading or mathematically inconsistent).

Adherence Annotation

For transition-level diagnosis, a GPT-5.5 annotator annotates the retrieval-invoked executions. The annotator evaluates the task prompt, retrieval query, retrieved skill excerpts, and the final SE answer to assign a primary behavioral label:appropriate adherence, ignored/independent, misapplied/overapplied, format/interface failure,orunclear. The annotator is blinded to both the SD answers and the underlying correctness labels; the resulting post-hoc labels provide descriptive diagnostics of the observed SE traces, remain separate from the RAE computation, and are evaluated through a stratified human audit reported in AppendixG.1.

5Results

In this section, we investigate whether aggregate retrieval lift and our proposed RAE yield consistent evaluations of LLM tool-use capabilities. Through a comprehensive analysis across coding and mathematical benchmarks, we demonstrate that standard aggregate metrics frequently present a misleading illusion of performance. We structurally unpack this discrepancy by diagnosing task-selection bias, cross-domain consistency, and the precise mechanistic failures that occur once external content enters the model’s context.

5.1Do Aggregate Metrics and RAE Agree?

We first establish the prevalence of metric disagreement. Table3reports the coding-domain results, contrasting aggregate retrieval lift with RAE.

Sign Disagreements Across Model and Benchmark

Sign reversals—where models exhibit a positive aggregate lift but a negative actual-use effect (RAE)—are alarmingly common. Across MBPP+, Table3shows repeated sign disagreements among reportable model configurations.

On HumanEval+, 3 out of 13 models exhibit the same paradox. For instance, Gemini-2.5-Flash-lite consistently shows positive aggregate lift across all three MBPP+ partitions, yet its RAE remains strictly negative. These findings highlight a critical vulnerability in current evaluation standards: seemingly beneficial skill access at the benchmark level frequently masks actual performance degradation on the specific tasks where skills are deployed.

ModelMBPP+ s42MBPP+ s43MBPP+ s44HumanEval+ s42Qwen3-8B−26.4-26.4/−1.5-1.5−0.5-0.5/−9.2-9.2−26.6-26.6/−1.6-1.6−10.5-10.5/+5.7+5.7Qwen3.5-9B−11.8-11.8/+0.0+0.0+26.1+26.1/−3.2-3.2−36.4-36.4/−4.1-4.1−5.4-5.4/+7.2+7.2Gemini-2.5-Flash-lite+6.2+6.2/−15.6-15.6+16.2+16.2/−15.9-15.9+4.4+4.4/−22.2-22.2−0.1-0.1/−21.1-21.1Mistral-S-24B−27.2-27.2/−22.5-22.5+10.8+10.8/−9.0-9.0−37.9-37.9/−45.5-45.5−13.9-13.9/−38.1-38.1GPT-4o-mini−24.0-24.0/−5.5-5.5+11.4+11.4/−3.4-3.4−23.2-23.2/−8.2-8.2−2.5-2.5/−1.4-1.4Claude-3.5-H−5.0-5.0/+2.5+2.5−1.0-1.0/−2.1-2.1−6.6-6.6/−3.8-3.8−2.7-2.7/−7.7-7.7Claude-4.5-H+1.4+1.4/+1.4+1.4+21.5+21.5/+0.0+0.0+48.0+48.0/+2.7+2.7+4.5+4.5/+0.0+0.0Qwen3-32B−16.9-16.9/+1.8+1.8−27.9-27.9/−6.0-6.0−12.4-12.4/−8.5-8.5−11.2-11.2/−30.1-30.1GLM4-32B−25.1-25.1/+0.0+0.0+1.8+1.8/−1.4-1.4−13.9-13.9/−6.8-6.8−2.8-2.8/−5.6-5.6Command-R−6.2-6.2/−14.5-14.5−36.1-36.1/−24.1-24.1−33.3-33.3/−26.7-26.7−5.8-5.8/+0.0+0.0Llama-3.3-70B+69.8+69.8/+4.2+4.2+44.2+44.2/−2.0-2.0+60.0+60.0/+7.5+7.5+61.5+61.5/+7.1+7.1Qwen3-235B−0.1-0.1/+7.0+7.0−21.1-21.1/+0.0+0.0−27.1-27.1/+3.7+3.7−7.0-7.0/+15.5+15.5DeepSeek-V3.2+31.6+31.6/−9.1-9.1−11.5-11.5/−15.4-15.4+46.2+46.2/−23.1-23.1−17.1-17.1/−23.0-23.0Table 3:Main coding-domain aggregate-vs-RAE comparison. Each cell reports aggregate retrieval lift / RAE in percentage points. Bold entries indicate opposite signs between the unpaired retrieved-vs-skipped aggregate and the retrieval-invoked same-task metric. The table includes cells satisfying the reporting filter; the full 17-model panel, including non-reportable cells, is provided in AppendixB.Figure 2:Autonomous retrieval induces severe task-selection bias. The x-axis reports the skill-disabled selection gap (Δselect\Delta_{\mathrm{select}}) between tasks that did and did not return a skill. The y-axis reports aggregate retrieval lift within the skill-enabled execution. Each point represents a reportable model cell, colored by the sign of its RAE. The prevalent nonzero selection gaps mathematically demonstrate that aggregate retrieval lift compares inherently mismatched task populations, failing to isolate the true causal effect of skill usage.

5.2What Drives the Divergence Between Aggregate Lift and Actual Use?

To understandwhyaggregate retrieval lift diverges so sharply from RAE, we must analyze the task populations being compared. Aggregate lift evaluates retrieved versus non-retrieved tasks within the skill-enabled execution. However, these subsets are not randomized; they are self-selected by the LLM’s own retrieval policy.

To quantify this underlying discrepancy, we introduce the skill-disabled selection gap (Δselect\Delta_{\mathrm{select}}). LetSretS_{\mathrm{ret}}andSnoretS_{\mathrm{noret}}denote the subsets of tasks that did and did not return a skill, respectively. We measure their inherent difficulty using the paired SD baseline:

Δselect=Acc−​(Sret)−Acc−​(Snoret)\Delta_{\mathrm{select}}=\mathrm{Acc}^{-}(S_{\mathrm{ret}})-\mathrm{Acc}^{-}(S_{\mathrm{noret}})(5)

Selection Bias Confounds Aggregate Lift

As illustrated in Figure2, autonomous retrieval induces substantial task-selection differences (Δselect≠0\Delta_{\mathrm{select}}\neq 0). This provides the mechanistic explanation for the metric disagreement: aggregate retrieval lift effectively compares two distinct, self-selected task populations that already differ in baseline difficulty or prompt complexity. RAE, by contrast, neutralizes this bias by strictly comparing the same retrieval-invoked tasks against their paired SD executions.

5.3Does the Evaluation Discrepancy Persist Beyond Benchmarks?

To confirm that this paradox is not an artifact of a single benchmark, we extend our analysis to alternative datasets and domains.

Cross-Benchmark Consistency

The aggregate-vs-RAE disagreement is not confined to MBPP+. On HumanEval+, 3 out of 13 reportable models show sign disagreement between aggregate retrieval lift and RAE. Unlike the dominant MBPP+ reversal emphasized in the introduction, these HumanEval+ disagreements occur in the opposite direction: aggregate retrieval lift is negative while RAE is positive for Qwen3-8B, Qwen3.5-9B, and Qwen3-235B. This indicates that aggregate retrieved-vs-skipped comparisons can misestimate retrieval-invoked actual use in either direction, either overstating or understating the effect of SF. AppendixDreports descriptive MBPP+–HumanEval+ correlation analyses, which we use only as diagnostic context rather than as benchmark-invariant model rankings.

Figure 3:Math500 cross-domain results. Comparing aggregate retrieval lift, OAE, and RAE reveals that the metric disagreement generalizes beyond coding. Models with highly positive aggregate lift collapse into severely negative actual-use effects (RAE).

Cross-Domain Results on Math500

Figure3shows that the disagreement between aggregate retrieval lift and RAE also appears in the mathematical reasoning domain. On Math500, Llama-3.3-70B obtains a+14.2+14.2pp aggregate lift but a−39.4-39.4pp RAE under the paired retrieval-invoked comparison. Gemini-2.5-Flash-lite shows a similar pattern (+13.2+13.2pp aggregate lift vs.−11.0-11.0pp RAE). These results indicate that the aggregate-vs-RAE disagreement is not limited to coding benchmarks. Full Math500 results are reported in AppendixC.

5.4Does Higher Retrieval Coverage Translate to Better Downstream Integration?

A prevailing assumption in tool-use evaluation is that higher retrieval coverage naturally translates to improved downstream performance. We examine this assumption by mapping retrieval coverage against RAE (Figure4). The results indicate that several models with high retrieval rates nevertheless yield negative actual-use effects. To unpack this discrepancy, the corresponding stage-level diagnostic definitions and values are detailed in AppendixE.

Retrieval is not Following

If mere exposure to external skills were sufficient, models with higher retrieval coverage would consistently show positive RAE. Instead, Figure4indicates that several high-coverage models still exhibit negative RAE. As detailed in the stage-diagnostic analysis (AppendixE), there is a notable gap between fetching a skill and successfully applying it: even when models retrieve skills at a high rate, their structural adherence to the provided content and effective success rates remain substantially lower. This disconnect highlights the distinction betweenexposureandactual use. Invoking a search function and receiving returned text does not guarantee that the LLM will correctly interpret, align, and incorporate that content into a valid final answer.

Figure 4:Retrieval coverage versus RAE on MBPP+ seed 42. The horizontal line marks zero RAE. Multiple high-retrieval models exhibit negative RAE, proving that successful retrieval invocation (exposure) does not reliably translate to successful SF (utilization).Control conditionDeepSeek-V3.2Gemini-2.5-Flash-liteNormal−15.9-15.9−17.9-17.9Schema-Empty––Filler-Dummy−10.2-10.2−5.0-5.0Random-skills−15.1-15.1−15.4-15.4Corrupted−9.1-9.1−17.2-17.2Table 4:Skill-content controls reporting pooled RAE in percentage points. Schema-Empty returns no usable skill, rendering RAE mathematically undefined (ncond=0n_{\mathrm{cond}}=0). Crucially, RAE remains strictly negative across all conditions whereanycontent is returned.

5.5Mechanistic Failures: How and Where Does the SF Chain Break Down?

To diagnose how SF breaks down, we analyzeharmful transitions—cases where the SD execution succeeds, but the SE execution retrieves a skill and subsequently fails. Using a GPT-5.5 annotator, we annotated 45 harmful transitions for DeepSeek-V3.2 and 40 for Gemini-2.5-Flash-lite across three MBPP+ seeds.

Ignored or Independent Use of Retrieved Skills

The most frequent failure mode isignored/independent. In these cases, the SE execution retrieves a skill, but the final answer shows no clear evidence that the returned content was incorporated into the solution. DeepSeek-V3.2 also showsformat/interface failures, where tool traces or interface artifacts appear in the final output. These diagnostics indicate that retrieval invocation alone is not a sufficient measure of SF: after retrieval, the model must still determine whether the returned content is relevant, integrate it into the task solution, and preserve the required output format. AppendixFprovides the full harmful-transition taxonomy, and AppendixGreports the adherence-label distributions and label-conditioned RAE analysis.

5.6Is Negative RAE Solely an Artifact of Defective Retrieved Content?

Finally, we test whether negative RAE can be attributed solely to the quality of the retrieved skill text. We run skill-content controls for the two strongest coding reversal models, DeepSeek-V3.2 and Gemini-2.5-Flash-lite, replacing the normal skill returns with empty, filler, random, or corrupted returned-content conditions. Table4summarizes the pooled RAE values, while AppendixHreports the corresponding raw transition counts, retrieval coverage, OAE, and aggregate retrieval lift for each control condition.

Returned Content Alone Does Not Determine RAE

The content-control results indicate that negative RAE is not solely an artifact of poorly written skill content. For both DeepSeek-V3.2 and Gemini-2.5-Flash-lite, RAE remains negative across all non-empty returned-content conditions, including Normal skills, Random-skills returns, Corrupted content, and Filler-Dummy text. Thus, a structured skill artifact does not by itself guarantee a positive actual-use effect. RAE instead reflects the full retrieval-conditioned execution chain, including retrieval invocation, content injection, relevance assessment, skill integration, and final-answer formatting. These controls therefore bound, rather than fully identify, the source of negative RAE.

6Conclusion

As LLMs increasingly operate within external skill environments, standard evaluation metrics—which rely on whole-benchmark averages or self-selected task subsets—create a misleading illusion of tool-use proficiency. To isolate the true causal impact of retrieved knowledge, we introduced the RAE. By strictly pairing skill-enabled and skill-disabled executions on the exact tasks where retrieval occurs, RAE neutralizes task-selection bias and directly measures the efficacy of the entire SF chain. Our evaluations across coding and mathematical domains reveal a pervasive paradox: models frequently exhibit positive aggregate retrieval lift while suffering a negative RAE. Apparent benchmark-level gains often mask actual performance degradation on the specific tasks where skills were actively deployed. By decoupling mere context exposure from actual cognitive integration, RAE provides a necessary methodological lens. It ensures that as agents become more autonomous, their tool-use mechanisms are rigorously evaluated as genuine cognitive aids rather than deceptive liabilities.

Limitations

Our experiments cover a focused evaluation regime: single-pass coding and mathematical tasks under a fixed retrieval interface and fixed procedural skill libraries. The results should therefore not be read as establishing the same prevalence or magnitude for long-horizon agents, executable code libraries, language-centric skill settings, natural-language annotator-based tasks, or systems with iterative memory and planning. RAE is also bounded in its statistical and causal interpretation. BecauseScallS_{\mathrm{call}}is selected by the skill-enabled execution itself, RAE is not an unbiased causal effect over a pre-treatment task population, nor a model-level skill-use ability score. It is a protocol-conditional paired outcome signal: under a fixed deployment protocol, when the agent entered the retrieval-returned portion of the SF chain, did the chain help or harm the same tasks? Finer causal decomposition would require additional ablations such as oracle retrieval, metadata-only retrieval, explicit full-skill fetch, multiple skill-pool constructions, and manually adjudicated causal labels.

Ethical Considerations

This work does not involve human subjects, private user data, or demographic profiling; all experiments are conducted on coding and mathematical benchmarks. The main ethical relevance of the study is evaluation reliability. Skill- or tool-augmented agents may be presented as improved systems when aggregate metrics hide cases where retrieval-invoked skill use harms the same tasks it was meant to help. Such overstatement can lead to misleading deployment claims and weaker auditability of agent systems. RAE is intended to make this failure mode more visible, but it should not be interpreted as a safety guarantee or as a complete causal explanation of failures. Positive or negative RAE values remain specific to the evaluated protocol, model, retrieval policy, skill library, and benchmark, and should be reported alongside complementary diagnostics and reproducibility artifacts. AI assistant tools were used to assist with coding, script debugging, result organization, and language polishing; all experimental design decisions, analyses, and final manuscript content were reviewed and controlled by the authors.

Acknowledgments

This work was supported by the Korea Internet & Security Agency (KISA) grant funded by the Korea government (PIPC) (No. RS-2026-25526342, Development of Technologies for Preventing Sensitive Information Inference and Risk Assessment in Foundation Model Operations). This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2026-25483747). This research was further supported by the Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency, funded by the Ministry of Culture, Sports and Tourism in 2026 (Project Name: Develop AI Agent Technology to Connect Knowledge through Public Cultural Facility-Based Discussion and Communication, Project Number: RS-2026-25520645).

References

• Chenet al.(2021)M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman,et al.Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374.Cited by:§4.1.
• Comaniciet al.(2025)G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen,et al.Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261.Cited by:Appendix K.
• Eset al.(2024)S. Es, J. James, L. E. Anke, and S. SchockaertRagas: automated evaluation of retrieval augmented generation.InProceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations,pp. 150–158.Cited by:§2.
• Gouet al.(2024)Z. Gou, Z. Shao, Y. Gong, Y. Yang, N. Duan, W. Chen,et al.Critic: large language models can self-correct with tool-interactive critiquing.InInternational Conference on Learning Representations,Vol.2024,pp. 57734–57811.Cited by:§2.
• Grattafioriet al.(2024)A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan,et al.The llama 3 herd of models.arXiv preprint arXiv:2407.21783.Cited by:Appendix K.
• Hanet al.(2026)T. Han, Y. Zhang, W. Song, C. Fang, Z. Chen, Y. Sun, and L. HuSWE-skills-bench: do agent skills actually help in real-world software engineering?.arXiv preprint arXiv:2603.15401.Cited by:§2.
• Hendryckset al.(2021)D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. SteinhardtMeasuring mathematical problem solving with the math dataset.NeurIPS.Cited by:§4.1.
• Liet al.(2026)X. Li, W. Chen, Y. Liu, S. Zheng, X. Chen, Y. He, Y. Li, B. You, H. Shen, J. Sun,et al.SkillsBench: benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670.Cited by:§2.
• Lightmanet al.(2024)H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. CobbeLet’s verify step by step.InInternational Conference on Learning Representations,Vol.2024,pp. 39578–39601.Cited by:§4.1.
• Liuet al.(2025)A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong,et al.Deepseek-v3. 2: pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556.Cited by:Appendix K.
• Liuet al.(2023)J. Liu, C. S. Xia, Y. Wang, and L. ZhangIs your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation.InThirty-seventh Conference on Neural Information Processing Systems,External Links:LinkCited by:§4.1.
• Liuet al.(2024)X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang,et al.Agentbench: evaluating llms as agents.InInternational Conference on Learning Representations,Vol.2024,pp. 52989–53046.Cited by:§2.
• Maet al.(2024)C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong, and J. HeAgentboard: an analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems37,pp. 74325–74362.Cited by:§2.
• Patilet al.(2025)S. G. Patil, H. Mao, F. Yan, C. C. Ji, V. Suresh, I. Stoica, and J. E. GonzalezThe berkeley function calling leaderboard (bfcl): from tool use to agentic evaluation of large language models.InForty-second International Conference on Machine Learning,Cited by:§2.
• Qwen Team (2026)Qwen TeamQwen3.5: towards native multimodal agents.External Links:LinkCited by:Appendix K.
• Robertson and Zaragoza (2009)S. Robertson and H. ZaragozaThe probabilistic relevance framework: bm25 and beyond.Vol.4,Now Publishers Inc.Cited by:§4.2.
• Schicket al.(2023)T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. ScialomToolformer: language models can teach themselves to use tools.External Links:2302.04761,LinkCited by:§1.
• Shenet al.(2024)W. Shen, C. Li, H. Chen, M. Yan, X. Quan, H. Chen, J. Zhang, and F. HuangSmall llms are weak tool learners: a multi-llm agent.External Links:2401.07324,LinkCited by:§1.
• Shiet al.(2024)Z. Shi, S. Gao, X. Chen, Y. Feng, L. Yan, H. Shi, D. Yin, P. Ren, S. Verberne, and Z. RenLearning to use tools via cooperative and interactive agents.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp. 10642–10657.Cited by:§1.
• Shinnet al.(2023)N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. YaoReflexion: language agents with verbal reinforcement learning.Advances in neural information processing systems36,pp. 8634–8652.Cited by:§2.
• Tanget al.(2025)X. Tang, T. Qin, T. Peng, Z. Zhou, D. Shao, T. Du, X. Wei, P. Xia, F. Wu, H. Zhu,et al.Agent kb: leveraging cross-domain experience for agentic problem solving.arXiv preprint arXiv:2507.06229.Cited by:§2.
• Team (2024)Q. TeamQwen2.5: a party of foundation models.External Links:LinkCited by:Appendix K.
• Wanget al.(2023)G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. AnandkumarVoyager: an open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291.Cited by:§2.
• Wuet al.(2024)S. Wu, S. Zhao, Q. Huang, K. Huang, M. Yasunaga, K. Cao, V. N. Ioannidis, K. Subbian, J. Leskovec, and J. ZouAvaTaR: optimizing llm agents for tool usage via contrastive reasoning.External Links:2406.11200,LinkCited by:§1.
• Xieet al.(2024)T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei,et al.Osworld: benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems37,pp. 52040–52094.Cited by:§2.
• Xuet al.(2026)F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Wang, X. Zhou, Z. Guo, M. Cao,et al.Theagentcompany: benchmarking llm agents on consequential real world tasks.Advances in Neural Information Processing Systems38.Cited by:§2.
• Yanget al.(2025)A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:Appendix K.
• Yaoet al.(2024)S. Yao, N. Shinn, P. Razavi, and K. Narasimhanτ\tau-Bench: a benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045.Cited by:§2.
• Zhaoet al.(2024)A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. HuangExpel: llm agents are experiential learners.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.38,pp. 19632–19642.Cited by:§2.
• Zhonget al.(2026)S. Zhong, Y. Lu, J. Ning, Y. Wan, L. Feng, Y. Ao, L. F. Ribeiro, M. Dreyer, S. Ammirati, and C. XiongSkillLearnBench: benchmarking continual learning methods for agent skill generation on real-world tasks.arXiv preprint arXiv:2604.20087.Cited by:§2.
• Zhouet al.(2023)J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. HouInstruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911.Cited by:§2.
• Zhouet al.(2024)S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried,et al.Webarena: a realistic web environment for building autonomous agents.InInternational Conference on Learning Representations,Vol.2024,pp. 15585–15606.Cited by:§2.

Appendix AMetric Computation and Reporting Filters

All cell-level metrics are computed from pooled raw counts rather than from means of per-run rates. A cell is defined by a fixed model, benchmark, partition seed, skill pool or control condition, and evaluation protocol. When config-identical reruns exist for the same cell, we first sum the underlying counts and then compute the reported rate.

For RAE, we aggregate raw counts across config-identical runs:

Bcond\displaystyle B_{\mathrm{cond}}=∑icondb,i,\displaystyle=\sum_{i}\mathrm{cond}_{b,i},Ccond\displaystyle C_{\mathrm{cond}}=∑icondc,i,\displaystyle=\sum_{i}\mathrm{cond}_{c,i},(6)Ncond\displaystyle N_{\mathrm{cond}}=∑incond,i.\displaystyle=\sum_{i}n_{\mathrm{cond},i}. We report

RAE=Δcond=100×Bcond−CcondNcond.\displaystyle\mathrm{RAE}=\Delta_{\mathrm{cond}}=100\times\frac{B_{\mathrm{cond}}-C_{\mathrm{cond}}}{N_{\mathrm{cond}}}.(7) HereBcondB_{\mathrm{cond}}counts helpful transitions, where the skill-disabled execution fails and the skill-enabled execution succeeds, andCcondC_{\mathrm{cond}}counts harmful transitions, where the skill-disabled execution succeeds and the skill-enabled execution fails, restricted to tasks where retrieval was invoked and at least one skill was returned in the skill-enabled execution.

For any task subsetSS, the paired binary outcomes are(yt+,yt−)∈{0,1}2(y_{t}^{+},y_{t}^{-})\in\{0,1\}^{2}, where 1 denotes a correct final answer and 0 denotes an incorrect final answer. The four paired-outcome cells can be written as

yt+=1yt+=0yt−=1acyt−=0bd\begin{array}[]{c|cc}&y_{t}^{+}=1&y_{t}^{+}=0\\ \hline\cr y_{t}^{-}=1&a&c\\ y_{t}^{-}=0&b&d\end{array}wherea,b,c,da,b,c,dare task counts. The discordant cells arebbandcc:bbcounts helpful transitions,(yt+,yt−)=(1,0)(y_{t}^{+},y_{t}^{-})=(1,0), andcccounts harmful transitions,(yt+,yt−)=(0,1)(y_{t}^{+},y_{t}^{-})=(0,1).

The paired outcome difference overSSis

1|S|​∑t∈S(yt+−yt−).\displaystyle\frac{1}{|S|}\sum_{t\in S}(y_{t}^{+}-y_{t}^{-}).(8)Concordant pairs contribute zero, while helpful and harmful transitions contribute+1+1and−1-1, respectively. Therefore,

∑t∈S(yt+−yt−)=b−c.\displaystyle\sum_{t\in S}(y_{t}^{+}-y_{t}^{-})=b-c.(9) Thus, the paired same-task difference over any subsetSSis(b−c)/|S|(b-c)/|S|.

For OAE, we aggregate

Ball\displaystyle B_{\mathrm{all}}=∑iallb,i,\displaystyle=\sum_{i}\mathrm{all}_{b,i},Call\displaystyle C_{\mathrm{all}}=∑iallc,i,\displaystyle=\sum_{i}\mathrm{all}_{c,i},(10)Nall\displaystyle N_{\mathrm{all}}=∑inpairs,i.\displaystyle=\sum_{i}n_{\mathrm{pairs},i}.We report

OAE=Δall=100×Ball−CallNall.\mathrm{OAE}=\Delta_{\mathrm{all}}=100\times\frac{B_{\mathrm{all}}-C_{\mathrm{all}}}{N_{\mathrm{all}}}.(11) The decomposition into retrieval-invoked and skipped subsets follows fromSall=Scall∪SskipS_{\mathrm{all}}=S_{\mathrm{call}}\cup S_{\mathrm{skip}}:

OAE=NcondNall​RAE+NskipNall​Δskip,\mathrm{OAE}=\frac{N_{\mathrm{cond}}}{N_{\mathrm{all}}}\mathrm{RAE}+\frac{N_{\mathrm{skip}}}{N_{\mathrm{all}}}\Delta_{\mathrm{skip}},(12)where

Δskip\displaystyle\Delta_{\mathrm{skip}}=100×Bskip−CskipNskip,\displaystyle=100\times\frac{B_{\mathrm{skip}}-C_{\mathrm{skip}}}{N_{\mathrm{skip}}},(13)Bskip\displaystyle B_{\mathrm{skip}}=Ball−Bcond,\displaystyle=B_{\mathrm{all}}-B_{\mathrm{cond}},Cskip\displaystyle C_{\mathrm{skip}}=Call−Ccond.\displaystyle=C_{\mathrm{all}}-C_{\mathrm{cond}}. WhenNskip=0N_{\mathrm{skip}}=0, the skipped-subset term is omitted.

Unlike RAE, this quantity averages the paired skill-enabled/skill-disabled difference over the entire evaluated task set, including tasks where the agent never retrieved a skill.

For aggregate retrieval lift, letRiR_{i}andR¯i\bar{R}_{i}denote the retrieved and non-retrieved task subsets in runii. We aggregate

Rcorr\displaystyle R_{\mathrm{corr}}=∑icorrect+​(Ri),\displaystyle=\sum_{i}\mathrm{correct}^{+}(R_{i}),RN\displaystyle R_{N}=∑i|Ri|,\displaystyle=\sum_{i}|R_{i}|,(14)Scorr\displaystyle S_{\mathrm{corr}}=∑icorrect+​(R¯i),\displaystyle=\sum_{i}\mathrm{correct}^{+}(\bar{R}_{i}),SN\displaystyle S_{N}=∑i|R¯i|.\displaystyle=\sum_{i}|\bar{R}_{i}|.When both denominators are nonzero, we report

Δagg=100×(RcorrRN−ScorrSN).\Delta_{\mathrm{agg}}=100\times\left(\frac{R_{\mathrm{corr}}}{R_{N}}-\frac{S_{\mathrm{corr}}}{S_{N}}\right).(15) ThresholdMBPP+ s42MBPP+ s43MBPP+ s44HumanEval+ s42ncond≥5n_{\mathrm{cond}}\geq 55/135/136/136/133/133/133/133/13ncond≥10n_{\mathrm{cond}}\geq 105/135/136/136/133/133/133/133/13ncond≥15n_{\mathrm{cond}}\geq 155/135/136/136/133/133/133/123/12ncond≥20n_{\mathrm{cond}}\geq 205/135/136/136/133/133/133/113/11Table 5:Sensitivity of aggregate-vs-RAE sign-disagreement counts to the retrieval-invoked reporting threshold. Each cell reports sign-disagreement cells divided by reportable cells. The main text usesncond≥10n_{\mathrm{cond}}\geq 10.If eitherRN=0R_{N}=0orSN=0S_{N}=0, aggregate retrieval lift is undefined because one of the two skill-enabled task populations is empty. This occurs, for example, when the agent invokes retrieval on every task.

ModelMBPP+ s42MBPP+ s43MBPP+ s44HumanEval+ s42Sign disagree / cellsQwen2.5-7B–––––Qwen3-8B−26.4-26.4/−1.5-1.5−0.5-0.5/−9.2-9.2−26.6-26.6/−1.6-1.6−10.5-10.5/+5.7+5.71/41/4Qwen3.5-9B−11.8-11.8/+0.0+0.0+26.1+26.1/−3.2-3.2−36.4-36.4/−4.1-4.1−5.4-5.4/+7.2+7.22/42/4Llama-3.1-8B–––––Mistral-Nemo–––––Gemini-2.5-Flash-lite+6.2+6.2/−15.6-15.6+16.2+16.2/−15.9-15.9+4.4+4.4/−22.2-22.2−0.1-0.1/−21.1-21.13/43/4Gemini-2.0-Flash–––––Mistral-S-24B−27.2-27.2/−22.5-22.5+10.8+10.8/−9.0-9.0−37.9-37.9/−45.5-45.5−13.9-13.9/−38.1-38.11/41/4GPT-4o-mini−24.0-24.0/−5.5-5.5+11.4+11.4/−3.4-3.4−23.2-23.2/−8.2-8.2−2.5-2.5/−1.4-1.41/41/4Claude-3.5-H−5.0-5.0/+2.5+2.5−1.0-1.0/−2.1-2.1−6.6-6.6/−3.8-3.8−2.7-2.7/−7.7-7.71/41/4Claude-4.5-H+1.4+1.4/+1.4+1.4+21.5+21.5/+0.0+0.0+48.0+48.0/+2.7+2.7+4.5+4.5/+0.0+0.00/40/4Qwen3-32B−16.9-16.9/+1.8+1.8−27.9-27.9/−6.0-6.0−12.4-12.4/−8.5-8.5−11.2-11.2/−30.1-30.11/41/4GLM4-32B−25.1-25.1/+0.0+0.0+1.8+1.8/−1.4-1.4−13.9-13.9/−6.8-6.8−2.8-2.8/−5.6-5.61/41/4Command-R−6.2-6.2/−14.5-14.5−36.1-36.1/−24.1-24.1−33.3-33.3/−26.7-26.7−5.8-5.8/+0.0+0.00/40/4Llama-3.3-70B+69.8+69.8/+4.2+4.2+44.2+44.2/−2.0-2.0+60.0+60.0/+7.5+7.5+61.5+61.5/+7.1+7.11/41/4Qwen3-235B−0.1-0.1/+7.0+7.0−21.1-21.1/+0.0+0.0−27.1-27.1/+3.7+3.7−7.0-7.0/+15.5+15.53/43/4DeepSeek-V3.2+31.6+31.6/−9.1-9.1−11.5-11.5/−15.4-15.4+46.2+46.2/−23.1-23.1−17.1-17.1/−23.0-23.02/42/4Sign disagree / both defined𝟓/𝟏𝟑\mathbf{5/13}𝟔/𝟏𝟑\mathbf{6/13}𝟑/𝟏𝟑\mathbf{3/13}𝟑/𝟏𝟑\mathbf{3/13}Reporting cells1313131313131313

Table 6:Full 17-model coding-domain sign-disagreement panel. Each cell reports aggregate retrieval lift / RAE in percentage points. Bold entries indicate opposite signs between the two metrics. Dashes indicate non-reportable cells, typically because the retrieval-invoked subset is empty, below the reporting threshold, or aggregate retrieval lift is undefined. Non-reportable cells are retained for transparency but excluded from sign-disagreement denominators.For main sign-disagreement counts, a cell is included only whenNcond≥10N_{\mathrm{cond}}\geq 10and both RAE and aggregate retrieval lift are defined. Cells below this retrieval-invocation threshold are omitted from prevalence counts because the conditional same-task estimate is too sparse. Cells withNcond≥10N_{\mathrm{cond}}\geq 10but undefined aggregate retrieval lift retain their RAE value, but they are not counted in aggregate-vs-RAE sign-disagreement denominators.

A sign disagreement is counted when the aggregate retrieval lift and RAE have opposite signs:

Δagg⋅Δcond<0\Delta_{\mathrm{agg}}\cdot\Delta_{\mathrm{cond}}<0(16) This criterion is metric-level. It should not be confused with task-level helpful and harmful transitions, which are the paired skill-enabled/skill-disabled outcome events aggregated by RAE.

We usencond≥10n_{\mathrm{cond}}\geq 10as the main reporting cutoff. Table5shows that the aggregate-vs-RAE sign-disagreement counts remain stable under alternative cutoffs of 5, 15, and 20.

ModelAgg. liftOAERAEbc𝒏𝐜𝐨𝐧𝐝\boldsymbol{n_{\mathrm{cond}}}Qwen2.5-7B–−65.8-65.8–000Qwen3-8B+0.0+0.0−10.0-10.0−20.0-20.01415Qwen3.5-9B+0.7+0.7−4.2-4.2−7.1-7.10114Llama-3.1-8B+5.8+5.8−10.4-10.4−9.3-9.31631162Mistral-Nemo−25.6-25.6−5.0-5.0+0.0+0.0002Gemini-2.5-Flash-lite+13.2+13.2−18.3-18.3−11.0-11.0416109Gemini-2.0-Flash−7.3-7.3−1.7-1.7+0.0+0.0118Mistral-S-24B−64.7-64.7−1.7-1.7−20.0-20.0015GPT-4o-mini+31.4+31.4+1.7+1.7+0.0+0.0001Claude-3.5-H−24.7-24.7−6.2-6.2−6.5-6.50231Claude-4.5-H−4.8-4.8−1.7-1.7−1.1-1.14588Qwen3-32B−7.5-7.5−9.6-9.6−9.7-9.7719124GLM4-32B+16.6+16.6−18.3-18.3−14.4-14.4936187Command-R+0.8+0.8−10.4-10.4−5.5-5.51119145Llama-3.3-70B+14.2+14.2−42.1-42.1−39.4-39.434299Qwen3-235B−3.2-3.2+2.1+2.1+0.0+0.00024DeepSeek-V3.2−8.7-8.7−8.3-8.3−11.0-11.0522154Table 7:Full Math500 cross-domain results. Values are in percentage points except forbb,cc, andncondn_{\mathrm{cond}}. Aggregate retrieval lift is undefined when there are no retrieved tasks or no non-retrieved tasks for comparison. Rows with very smallncondn_{\mathrm{cond}}are reported for completeness but are interpreted as sparse conditional estimates.### A.1Representative Reversal Statistics

Table8reports paired transition counts and uncertainty for representative coding-domain reversal cells. Confidence intervals use 20,000 task-level paired multinomial bootstrap resamples, andpp-values are from exact McNemar tests over the discordant counts.

ModelBenchmark𝒃\boldsymbol{b}𝒄\boldsymbol{c}𝒏𝐜𝐨𝐧𝐝\boldsymbol{n_{\mathrm{cond}}}RAE95% CIExact McNemarppMistral-S-24BMBPP+ (partition 3)13166−45.5-45.5[−57.6,−33.3][-57.6,-33.3]×10−81.54\!\times\!10^{-8}DeepSeek-V3.2MBPP+ (partition 3)11978−23.1-23.1[−33.3,−12.8][-33.3,-12.8]×10−54.01\!\times\!10^{-5}Gemini-2.5-Flash-liteHumanEval+1938−21.1-21.1[−36.8,−7.9][-36.8,-7.9]0.02150.0215Qwen3-235BHumanEval+13271+15.5+15.5[+5.6,+25.4][+5.6,+25.4]0.007390.00739Table 8:Representative coding-domain reversal cells. RAE and confidence intervals are in percentage points.bbcounts SD-wrong→\rightarrowSE-correct transitions,cccounts SD-correct→\rightarrowSE-wrong transitions, andncondn_{\mathrm{cond}}is the retrieval-returned paired subset size.

Appendix BFull Coding-Domain Results

Table6reports the full 17-model coding-domain panel underlying Table3. The main table includes reportable cells only, whereas this appendix table retains all evaluated models for transparency. Dashes indicate non-reportable cells, typically because the retrieval-invoked subset is empty, below the reporting threshold, or one of the aggregate retrieval lift denominators is undefined. These cells are retained in the full panel but excluded from sign-disagreement denominators.

Appendix CMath500 Cross-Domain Results

Table7reports the full Math500 cross-domain replication. The table includes aggregate retrieval lift, OAE, and RAE, together with the retrieval-invoked transition counts used to compute RAE.

MetricnRaw PearsonrrPartialrrSingle-control rangeAggregate retrieval lift13+0.746+0.746+0.763+0.763+0.745+0.745–+0.755+0.755OAE13+0.700+0.700+0.830+0.830+0.732+0.732–+0.825+0.825RAE13+0.634+0.634+0.776+0.776+0.633+0.633–+0.747+0.747Table 9:Descriptive cross-benchmark correlations between MBPP+ and HumanEval+. The partialrrcolumn controls for both MBPP+ and HumanEval+ skill-disabled accuracies. The single-control range reports the minimum and maximum partial correlations obtained when controlling for only one benchmark’s skill-disabled accuracy. These values are reported as diagnostic context, not as evidence that RAE is the most cross-benchmark-stable metric.Excluded modelMBPP+ncondn_{\mathrm{cond}}HE+ncondn_{\mathrm{cond}}ReasonGemini-2.0-Flash00no retrieval-invoked conditional subsetLlama-3.1-8B00no retrieval-invoked conditional subsetMistral-Nemo00no retrieval-invoked conditional subsetQwen2.5-7B00no retrieval-invoked conditional subsetTable 10:Models excluded from the filtered cross-benchmark correlation analysis. The reporting filter requiresncond≥10n_{\mathrm{cond}}\geq 10on both MBPP+ and HumanEval+.ModelSeedInvocationRetrievalAdherenceEffectiveRAEDeepSeek-V3.242100.0100.0100.0100.018.818.811.211.2−9.1-9.1Gemini-2.5-Flash-lite4282.582.582.582.513.613.610.610.6−15.6-15.6DeepSeek-V3.243100.0100.0100.0100.015.015.011.211.2−15.4-15.4Gemini-2.5-Flash-lite4382.582.582.582.516.716.710.610.6−15.9-15.9DeepSeek-V3.24497.597.597.597.519.219.212.812.8−23.1-23.1Gemini-2.5-Flash-lite4478.878.878.878.817.517.512.712.7−22.2-22.2Table 11:Stage-diagnostic values for the two anchor reversal models across MBPP+ seeds. Stage values are percentages. RAE is reported in percentage points. The adherence and effective columns are diagnostic proxies from the skill-enabled logs, whereas RAE is computed from paired skill-enabled/skill-disabled outcomes.

Appendix DCross-Benchmark Correlation Details

Table9reports a descriptive cross-benchmark correlation analysis comparing MBPP+ seed 42 against HumanEval+ seed 42 under the same coding skill pool.

We include models with sufficient retrieval-invocation coverage on both benchmarks, requiringncond≥10n_{\mathrm{cond}}\geq 10for both MBPP+ and HumanEval+. This yields 13 models. Gemini-2.0-Flash, Llama-3.1-8B, Mistral-Nemo, and Qwen2.5-7B are excluded from this filtered correlation because at least one benchmark has fewer than 10 retrieval-invoked tasks. Table10lists these exclusions and the corresponding retrieval-invoked subset sizes.

Skill-disabled accuracy is used as the baseline ability control. For each benchmark cell, skill-disabled accuracy is reconstructed from the paired identity

Acc−=Acc+−OAE/100.\mathrm{Acc}^{-}=\mathrm{Acc}^{+}-\mathrm{OAE}/100.(17)Partial correlations are computed by residualizing the MBPP+ and HumanEval+ metric values against the specified skill-disabled accuracy controls and then correlating the residuals.

In this rerun, aggregate retrieval lift has the highest raw cross-benchmark correlation, and OAE has the highest skill-disabled-accuracy-controlled partial correlation. We therefore use these correlations only to bound interpretation: RAE is a protocol-conditional actual-use signal over the agent’s retrieval-invoked subset, not a benchmark-invariant ranking of model skill-use ability.

Appendix EStage-Diagnostic Definitions and Values

The stage diagnostics in this appendix are descriptive proxies computed from the skill-enabled execution logs and paired outcome counts. They are not used to define RAE. Their purpose is to support the main-text observation that high invocation or retrieval coverage does not imply a positive same-task actual-use outcome.

We use the following stage proxies.Invocationis the percentage of tasks where the agent called the skill-search interface at least once.Retrievalis the percentage of tasks where at least one skill was returned.Adherenceis a lexical/structural proxy indicating whether at least one retrieved-skill identifier or code-relevant pattern appears in the submitted answer.Effectiveis the percentage of retrieval-invoked tasks where this adherence proxy is present and the final answer is correct. Table11reports the resulting stage-diagnostic values for the two anchor reversal models across MBPP+ seeds. The final column reports RAE, the paired same-task outcome metric over the retrieval-invoked subset.

ModelPrimary labelnShareDeepSeek-V3.2appropriate adherence613.3%DeepSeek-V3.2format/interface failure715.6%DeepSeek-V3.2ignored/independent2964.4%DeepSeek-V3.2misapplied/overapplied36.7%Gemini-2.5-Flash-liteappropriate adherence410.0%Gemini-2.5-Flash-liteformat/interface failure12.5%Gemini-2.5-Flash-liteignored/independent2972.5%Gemini-2.5-Flash-litemisapplied/overapplied512.5%Gemini-2.5-Flash-liteunclear12.5%Table 12:Full harmful-transition primary-label summary before the category compression used in the main text. The unit is a skill-disabled-correct/ skill-enabled-wrong retrieval-invoked execution.Failure modeDefinitionAppropriate adherenceThe skill-enabled answer visibly uses relevant retrieved-skill content or patterns in a plausible way.Ignored/independentThe skill-enabled answer retrieves a skill but shows no clear uptake of the returned content.Misapplied/overappliedThe skill-enabled answer uses retrieved-skill content but applies it in a way that changes or overconstrains the task solution.Format/interface failureThe skill-enabled answer fails at the output or benchmark-interface level, such as leaking tool traces, omitting the required function, or returning malformed code.UnclearThe annotator cannot assign a stable narrower behavioral label from the available evidence.Table 13:Primary failure-mode definitions used for harmful-transition taxonomy.Modelbbccncondn_{\mathrm{cond}}DeepSeek-V3.2845233Gemini-2.5-Flash-lite540196Table 14:Annotation-count reconciliation.bbandccare helpful and harmful retrieval-invoked transitions; counts match the RAE computation.ModelTransitionPrimary labelShareDeepSeek-V3.2Helpfulappropriate adherence50.0%DeepSeek-V3.2Helpfulignored/independent50.0%DeepSeek-V3.2Harmfulappropriate adherence13.3%DeepSeek-V3.2Harmfulformat/interface failure15.6%DeepSeek-V3.2Harmfulignored/independent64.4%DeepSeek-V3.2Harmfulmisapplied/overapplied6.7%Gemini-2.5-Flash-liteHelpfulignored/independent100.0%Gemini-2.5-Flash-liteHarmfulappropriate adherence10.0%Gemini-2.5-Flash-liteHarmfulformat/interface failure2.5%Gemini-2.5-Flash-liteHarmfulignored/independent72.5%Gemini-2.5-Flash-liteHarmfulmisapplied/overapplied12.5%Gemini-2.5-Flash-liteHarmfulunclear2.5%Table 15:GPT-5.5 adherence annotation distribution over discordant retrieval-invoked transitions. The annotator is blinded to the skill-disabled answer and correctness labels.ModelCollapsed labelbc𝒏\boldsymbol{n}RAEDeepSeek-V3.2appropriate4689−2.2-2.2DeepSeek-V3.2no clear431129−20.9-20.9DeepSeek-V3.2problematic0815−53.3-53.3Gemini-2.5-Flash-liteappropriate0426−15.4-15.4Gemini-2.5-Flash-liteno clear531152−17.1-17.1Gemini-2.5-Flash-liteproblematic0518−27.8-27.8Table 16:Label-conditioned RAE values using collapsed GPT-5.5 adherence labels. Values are in percentage points. These diagnostic subsets do not redefine RAE; they describe which behavioral labels account for helpful and harmful transitions in the retrieval- invoked representative results.ModelConditionb/c𝒏𝐜𝐨𝐧𝐝\boldsymbol{n_{\mathrm{cond}}}RAEOAECoverageAgg. liftDeepSeek-V3.2Normal8/458/45233−15.9-15.9−15.8-15.897.197.1+21.2+21.2DeepSeek-V3.2Schema-Empty0/00/00–−10.0-10.00.00.0–DeepSeek-V3.2Filler-Dummy7/307/30226−10.2-10.2−10.8-10.894.294.2+28.0+28.0DeepSeek-V3.2Random-skills5/415/41238−15.1-15.1−15.4-15.499.299.2+7.1+7.1DeepSeek-V3.2Corrupted8/298/29230−9.1-9.1−9.6-9.695.895.8+43.5+43.5Gemini-2.5-Flash-liteNormal5/405/40196−17.9-17.9−17.9-17.981.781.7+7.8+7.8Gemini-2.5-Flash-liteSchema-Empty0/00/00–−15.4-15.40.00.0–Gemini-2.5-Flash-liteFiller-Dummy5/145/14181−5.0-5.0−7.1-7.175.475.4+8.1+8.1Gemini-2.5-Flash-liteRandom-skills2/322/32195−15.4-15.4−17.9-17.981.281.2+14.0+14.0Gemini-2.5-Flash-liteCorrupted5/365/36180−17.2-17.2−17.5-17.575.075.0+1.1+1.1Table 17:Detailed skill-content control results pooled over MBPP+ seeds 42–44. Values for RAE, OAE, coverage, and aggregate retrieval lift are in percentage points. Coverage is the percentage of tasks where retrieval returned at least one skill. Aggregate retrieval lift is undefined when retrieved or non-retrieved task populations are empty.

Appendix FTransition-Level Annotation Details

Table12reports the full harmful-transition taxonomy computed over skill-disabled-correct/skill-enabled-wrong retrieval-invoked executions from the DeepSeek-V3.2 and Gemini-2.5-Flash-lite coding-skill runs. The unit is a log-level paired execution, not a unique task. The anchor set includes MBPP+ partition seeds 42, 43, and 44 for each model, so the counts are used as qualitative diagnostic evidence rather than as an independent estimate of task-frequency prevalence.

Each harmful transition receives one primary failure-mode label. When multiple symptoms are present, the primary label is assigned to the most proximate visible cause of the submitted answer failure. Table13defines the labels. Table12reports the full label distribution before the category compression used in the main-text taxonomy.

Appendix GAdherence Annotation Details

We annotate the pooled DeepSeek-V3.2 and Gemini-2.5-Flash-lite retrieval-invoked representative results with a GPT-5.5 annotator. The annotator receives the task prompt, retrieval query, retrieved skill names and excerpts, and the skill-enabled answer. It does not receive the skill-disabled answer or the paired correctness labels. The annotation is therefore used as diagnostic evidence about skill uptake behavior, not as part of the definition of RAE.

Each skill-enabled answer receives one primary behavioral label: appropriate adherence, ignored/independent, misapplied/overapplied, format/interface failure, or unclear. For Figure5, we also collapse these into three groups: appropriate adherence, no clear adherence, and problematic adherence. The collapsed labels are used only for visualization. Table14reconciles the annotated logs with the transition counts used in RAE. Table15reports the GPT-5.5 label distribution over discordant transitions, and Table16reports RAE after grouping retrieval-invoked executions by collapsed adherence labels.

G.1Human Audit of Annotation Reliability

We evaluate annotation reliability on a stratified sample of 50 of the 85 harmful transitions. Nine non-author annotators provide 150 labels while blinded to the GPT-5.5 label, the SD answer, the paired correctness labels, and one another’s annotations. Table18summarizes the main agreement results.

The following is an English translation of the full Korean instructions provided to the human annotators.

Human Annotator InstructionsHuman audit labeling guide (16--17 cases per person; 20--35 minutes) Purpose. This audit tests whether the behavioral labels assigned by an LLM (GPT-5.5) are reliable. Relabel your cases independently; we will later compute agreement with the LLM labels. The LLM labels are hidden. Do not consult other annotators. Each row contains one coding-task record: - task_prompt: the problem given to the model - retrieval_queries: the model’s skill-search queries - skills_retrieved / skill_text_excerpt: the skill documents actually returned to the model - on_answer: the model’s final answer Read the problem, retrieved skills, and final answer, in that order. Ask: "How did this answer handle the retrieved skills?" Enter exactly one label in human_primary_label: - adhered_appropriate: visibly and appropriately uses retrieved-skill content for the task - misapplied_or_overapplied: uses a retrieved skill incorrectly or unnecessarily - ignored_or_independent: shows no trace of using the retrieved skill and solves independently - format_or_interface_failure: fails in output or interface format, such as omitting the required function or returning prose instead of code - unclear: cannot be assigned to the four labels above from the available evidence Rules. 1. Do not judge task correctness. All cases are skill-enabled executions that failed; label only how the answer handled the retrieved skill. 2. If uncertain, choose unclear rather than forcing a narrower label. 3. Work alone and only on your assigned sheet. Notes in human_notes are optional. Enter confidence in human_confidence_1_to_5, from 1 (guess) to 5 (certain). Low confidence is natural for unclear cases. Submission. Fill human_primary_label and human_confidence_1_to_5 in your assigned audit_annotator_A#.csv file, save it, and return it.

Audit statisticResultFive-way Krippendorff’sα\alpha0.3770.377Five-way human-majority–GPT agreement52.4%52.4\%(22/4222/42)Cases without a strict five-way majority8/508/50Binaryengaged/not engagedhuman–human agreement84%84\%Binaryengaged/not engagedhuman-majority–GPT agreement84%84\%(41/4941/49)Binary Krippendorff’sα\alpha0.4010.401Human / GPTappropriate adherencerate9.3%9.3\%/10.0%10.0\%Table 18:Human-audit agreement for the GPT-5.5 annotations. The five labels areappropriate adherence,ignored/independent,misapplied/overapplied,format/interface failure, andunclear. For the binary analysis,appropriate adherenceandmisapplied/overappliedare grouped asengaged, whileignored/independent,format/interface failure, andunclearare grouped asnot engaged.Agreement is modest under the original five-way taxonomy, indicating that the fine-grained category boundaries are interpretive. The binaryengaged/not engageddistinction is more stable, with84%84\%human–human and human-majority–GPT agreement. We therefore use the labels as descriptive post-hoc diagnostics and limit our conclusion to the low prevalence ofappropriate adherenceamong the audited harmful transitions.

(a)GPT-5.5 adherence labels over helpful and harmful transitions. (b)Label-conditioned RAE over retrieval-invoked executions.

Figure 5:Adherence-conditioned diagnosis of retrieval-invoked outcome changes in the pooled DeepSeek-V3.2 and Gemini-2.5-Flash-lite runs. Panel (a) shows how helpful and harmful transitions are distributed across GPT-5.5 behavioral labels. Panel (b) reports RAE after grouping executions by collapsed adherence labels.

Appendix HSkill-Content Control Details

Table17reports the pooled raw counts underlying Table4. The three Normal cells use distinct 80-task partitions, whereas the control replicates reuse the same 80-task partition; pooled arm values are therefore descriptive. On shared tasks, the Gemini Normal–Filler RAE differences are−7.7-7.7,−14.0-14.0, and−3.8-3.8pp across the three control replicates, with only the second interval excluding zero. DeepSeek Normal–Filler differences are+4.1+4.1,+4.1+4.1, and0.00.0pp. These heterogeneous comparisons do not identify a single content mechanism. Schema-Empty hasncond=0n_{\mathrm{cond}}=0because the interface returns no usable skill, so RAE is undefined while OAE remains defined over the whole benchmark.

Appendix IRetrieval-Configuration Sensitivity

We rerun three representative models on the same 80 MBPP+ tasks with retrieval depthsk∈{1,3,5}k\in\{1,3,5\}and with a mechanically merged single-skill library. All runs use one generic agent role, temperature0.70.7, maximum output length 8,192, reasoning disabled, and at most three agent turns. Confidence intervals use 20,000 paired multinomial bootstrap resamples;pp-values are within-configuration exact McNemar tests.

ModelConfiguration𝒃\boldsymbol{b}𝒄\boldsymbol{c}𝒏𝐜𝐨𝐧𝐝\boldsymbol{n_{\mathrm{cond}}}CoverageRAE95% CIExactppGemini-2.5-FLk=1k=1096581.281.2−13.8-13.8[−23.1,−6.2][-23.1,-6.2]0.003910.00391k=3k=3(paper)2126480.080.0−15.6-15.6[−26.6,−4.7][-26.6,-4.7]0.01290.0129k=5k=5197087.587.5−11.4-11.4[−20.0,−2.9][-20.0,-2.9]0.02150.0215merged-1022328.728.7−8.7-8.7[−21.7,+0.0][-21.7,+0.0]0.5000.500DeepSeek-V3.2k=1k=10197796.296.2−24.7-24.7[−35.1,−15.6][-35.1,-15.6]×10−63.81\!\times\!10^{-6}k=3k=3(paper)3107796.296.2−9.1-9.1[−18.2,+0.0][-18.2,+0.0]0.09230.0923k=5k=54117796.296.2−9.1-9.1[−19.5,+0.0][-19.5,+0.0]0.1180.118merged-1467188.888.8−2.8-2.8[−11.3,+5.6][-11.3,+5.6]0.7540.754Claude-4.5-Hk=1k=1127290.090.0−1.4-1.4[−6.9,+2.8][-6.9,+2.8]1.0001.000k=3k=3(paper)327087.587.5+1.4+1.4[−4.3,+7.1][-4.3,+7.1]1.0001.000k=5k=5447087.587.5+0.0+0.0[−8.6,+7.1][-8.6,+7.1]1.0001.000merged-1031923.823.8−15.8-15.8[−31.6,+0.0][-31.6,+0.0]0.2500.250Table 19:Retrieval-depth and merged-library sensitivity. Coverage is the percentage of tasks for which at least one skill was returned; RAE and confidence intervals are in percentage points. The merged-1 intervention changes both skill granularity and retrieval coverage, so it is not interpreted as a pure granularity effect.Acrossk=1,3,5k=1,3,5, Gemini remains negative with significant within-configuration tests, DeepSeek remains negative with significance only atk=1k=1, and Claude remains near zero. The repeated direction for Gemini and DeepSeek rules out the defaultk=3k=3choice as the sole explanation, while the merged-1 results remain jointly affected by lower coverage.

Appendix JSkill Pool Details

Tables20and21summarize the fixed procedural skill pools used in the coding and mathematical evaluations. Skills are stored as Markdown files following the AgentSkill-style structure described in §4.2. The retrieval index uses the skill descriptions and Markdown bodies, and the top retrieved skill texts are appended to the agent context when retrieval is invoked.

Coding skillSummarybinary-search-boundaryFind a threshold in a sorted or monotonic search space.counting-hashCount occurrences with a hash map for frequency, duplicate, or anagram-style tasks.dfs-iterativeTraverse graphs iteratively for reachability, connected components, or cycle-safe search.edge-case-guardAdd defensive checks for empty,None, singleton, and boundary inputs.knapsack-01-dpUse one-dimensional bottom-up dynamic programming for 0/1 knapsack-style constraints.prefix-sumBuild prefix sums for constant-time contiguous range queries.rotate-leftRotate a sequence left by a specified offset with wrap-around behavior.sliding-window-longestFind the longest contiguous window satisfying a monotonic predicate.two-pointer-pair-sumFind pair or closest-sum patterns in sorted or sortable sequences.Table 20:Coding skill pool used for MBPP+ and HumanEval+. The pool contains 9 procedural skills.The Math skill pool is summarized in Table21.

Math skillSummaryalgebraic-substitutionSubstitute known values into an expression and simplify.case-analysisSplit a problem into mutually exclusive cases such as sign, parity, range, or absolute-value branches.check-by-substitutionVerify a derived answer by substituting it back into the original equations or constraints.coordinate-distanceCompute distance, midpoint, slope, or angle from coordinate representations.factor-then-solveFactor polynomial equations and solve by setting factors to zero.modular-arithmeticWork with remainders, divisibility, cyclic patterns, and large-power computations.sequence-formulaeApply standard closed forms for arithmetic, geometric, and telescoping sequences.systematic-enumerationCount or list candidates by traversing a bounded finite search space.Table 21:Math skill pool used for the Math500 cross-domain replication. The pool contains 8 procedural skills.

Appendix KModel Panel

Table22lists the model panel used in our experiments. The coding panel consists of the 17 models (Claude 3.5 Haiku111https://www-cdn.anthropic.com/c7822cdc35ad788ec87e14b3a9d45010f1f86c38.pdf, Claude Haiku 4.5222https://www.anthropic.com/news/claude-haiku-4-5, Command R333https://huggingface.co/CohereLabs/c4ai-command-r-v01, DeepSeek V3.2, Gemini 2.0 Flash444https://ai.google.dev/gemini-api/docs/models/gemini-2.0-flash, Gemini 2.5 Flash-Lite, Llama 3.1 8B Instruct, Llama 3.3 70B Instruct, Mistral NeMo555https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407, Mistral Small 3.2 24B Instruct666https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503, GPT-4o mini777https://openai.com/ko-KR/index/gpt-4o-mini-advancing-cost-efficient-intelligence, Qwen2.5 7B Instruct, Qwen3 235B A22B 2507, Qwen3 32B, Qwen3 8B, Qwen3.5 9B, GLM4-32B888https://huggingface.co/zai-org/GLM-4-32B-0414)Liu et al. (2025);Comanici et al. (2025);Grattafiori et al. (2024);Yang et al. (2025);Qwen Team (2026);Team (2024)evaluated on MBPP+ and HumanEval+. The Math500 cross-domain replication uses the same 17-model panel, with reportability varying according to retrieval-invoked coverage. Model identifiers correspond to the API-facing model names used in the experiment logs.

Paper aliasModel identifierProvider / familyAccess typeClaude-3.5-Hanthropic/claude-3.5-haikuAnthropic ClaudeClosed/APIClaude-4.5-Hanthropic/claude-haiku-4.5Anthropic ClaudeClosed/APICommand-Rcohere/command-r-08-2024Cohere Command-RClosed/APIDeepSeek-V3.2deepseek/deepseek-v3.2DeepSeekOpen-weight familyGemini-2.0-Flashgoogle/gemini-2.0-flash-001Google GeminiClosed/APIGemini-2.5-Flash-litegoogle/Gemini-2.5-Flash-liteGoogle GeminiClosed/APILlama-3.1-8Bmeta-llama/llama-3.1-8b-instructMeta LlamaOpen-weight familyLlama-3.3-70Bmeta-llama/llama-3.3-70b-instructMeta LlamaOpen-weight familyMistral-Nemomistralai/mistral-nemoMistral AIOpen-weight familyMistral-S-24Bmistralai/mistral-small-3.2-24b-instructMistral AIOpen-weight familyGPT-4o-miniopenai/gpt-4o-miniOpenAI GPTClosed/APIQwen2.5-7Bqwen/qwen-2.5-7b-instructQwenOpen-weight familyQwen3-235Bqwen/qwen3-235b-a22b-2507QwenOpen-weight familyQwen3-32Bqwen/qwen3-32bQwenOpen-weight familyQwen3-8Bqwen/qwen3-8bQwenOpen-weight familyQwen3.5-9Bqwen/qwen3.5-9bQwenOpen-weight familyGLM4-32Bz-ai/glm-4-32bZ-AI GLMOpen-weight familyTable 22:Model panel used in the experiments. Access type is used only to describe the diversity of the evaluated panel; all models were accessed through the same OpenAI-compatible experiment interface.

Appendix LAgent Prompts and Tool Schema

This appendix reports the prompts and tool schema used in the paired SE and SD executions and in the adherence annotation. The SE and SD prompts are kept identical except for the availability of thesearch_skillsinterface. The annotator prompt is used only for post hoc diagnostic labeling and is not used in the computation of RAE.

Execution settings.

Generation uses temperature0.70.7, a maximum output length of 8,192 tokens, provider-side reasoning disabled, and at most three sequentialsearch_skillscalls. Each valid call returns the top-3 BM25 matches with their full skill text. Invalid arguments return no skill and generation continues; when several calls are emitted together, only the first is executed. No manual context truncation is applied. Transient requests are retried up to three times, and unrecoverable provider or context-length failures produce an empty prediction scored as incorrect.

Answer extraction and correctness.

For coding tasks, the evaluator extracts the first Python-fenced block, falls back to the first generic fenced block, and otherwise evaluates the raw response. The extracted program is executed against the EvalPlus extended test suite and is correct only when every test passes. Math answers are evaluated withmath-verify, followed by boxed-answer parsing and normalized exact matching.

SE Agent PromptYou are a helpful assistant. You have a ‘search_skills‘ tool that searches a skill library. Use it if you think it could help solve the task. Implement the requested function with the exact signature. Reply inside a ‘‘‘python block — only the function, no test code.

SD Agent PromptYou are a helpful assistant. Implement the requested function with the exact signature. Reply inside a ‘‘‘python block — only the function, no test code.

GPT-5.5 Annotator PromptYou annotate whether an LLM answer substantively follows retrieved skill content. Do not infer from task correctness; no correctness labels are provided. Return one compact JSON object only.

1SEARCH_SKILLS_TOOL={

2“type“:“function”,

3“function“:{

4“name“:“search_skills”,

5“description“:(

6“Searchtheskilllibraryforreusableproblem-solvingartifacts.“

7“Returnsutilityfunctions(callablehelpersyoucanusedirectly),“

8“proceduretemplates(algorithmskeletonsyoucanspecialize),and“

9“guardchecklists(defensivecheckstoaddatfunctionentry).“

10),

11“parameters“:{

12“type“:“object”,

13“properties“:{

14“query“:{

15“type“:“string”,

16“description“:“Natural-languagequerydescribingthesubproblemorpatternyouneed.”,

17},

18“types“:{

19“type“:“array”,

20“items“:{“type”:“string”,“enum”:[“utility”,“procedure”,“guard”]},

21“description“:(

22“Optionalfilter.Omitforalltypes.Use“

23“[’utility’,’procedure’]toskipdefensive-onlyskills.“

24),

25},

26},

27“required“:[“query”],

28},

29},

30}

Listing 1:Search skill tool schema