Cool paper from Apple. Most evaluation of tool-calling agents happens after the trajectory is over. By then the wrong call has already shipped. This new paper moves evaluation into the execution loop. A specialized reviewer agent inspects each provisional tool call before it executes. If something is off, it injects feedback and the primary agent revises. To quantify the tradeoff between corrections and new mistakes, they introduce Helpfulness-Harmfulness metrics. Helpfulness measures the percentage of base errors fixed; harmfulness measures correct calls degraded by the reviewer. Results on BFCL: +5.5% on irrelevance detection (84.9% to 90.4%), +1.6% on relevance, all with no retraining of the base agent. On τ²-Bench multi-turn: +7.1% (48.7% to 55.8%). Reasoning-model reviewers get a 3:1 benefit-to-risk ratio vs. 2.1:1 for GPT-4o. Adding GEPA prompt optimization stacks another +1.5–2.8%. Why does it matter? You can keep the base tool-calling agent frozen and still ship measurable accuracy gains by improving only the reviewer. Model selection and prompt optimization on the reviewer become real, separable production levers. Paper: https://arxiv.org/abs/2604.27233 Learn to build effective AI agents in our academy: https://academy.dair.ai

Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents

Source: https://arxiv.org/html/2604.27233

Abstract

Tool-calling agents are evaluated on tool selection, parameter accuracy, and scope recognition, yet LLM trajectory assessments remain inherentlypost-hoc. Disconnected from the active execution loop, such assessments identify errors that are usually addressed through prompt-tuning or retraining, and fundamentally cannot course-correct the agent in real time. To close this gap, we move evaluation into the execution loop atinference time: a specialized reviewer agent evaluates provisional tool callsprior toexecution, shifting the paradigm from post-hoc recovery to proactive evaluation and error mitigation.

In practice, this architecture establishes a clear separation of concerns between the primary execution agent and a secondary review agent. As with any multi-agent system, the reviewer can introduce new errors while correcting others, yet no prior work to our knowledge has systematically measured this tradeoff. To quantify this tradeoff, we introduceHelpfulness-Harmfulness metrics: helpfulness measures the percentage of base agent errors that feedback corrects; harmfulness measures the percentage of correct responses that feedback degrades. These metrics directly inform reviewer design by revealing whether a given model or prompt provides net positive value.

We evaluate our approach on BFCL (single-turn) andτ2\tau^{2}-Bench (multi-turn stateful scenarios), achieving +5.5% on irrelevance detection and +7.1% on multi-turn tasks. Our metrics reveal that reviewer model choice is critical: the reasoning model o3-mini achieves a 3:1 benefit-to-risk ratio versus 2.1:1 for GPT-4o. Automated prompt optimization via GEPA provides an additional +1.5–2.8%. Together, these results demonstrate a core advantage of separating execution and review: the reviewer can be systematically improved through model selection and prompt optimization, without retraining the base agent.

Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents

Anh Ta Junjie Zhu Shahin ShayandehApple{atta, jason.zhu, shn}@apple.com

1Introduction

“What’s the weather in New York City?”Userget_weather(“NYC”, unit=“C”)Tool-Calling AgentError detected:For US cities, temperature should use Fahrenheit by default. Also, use full city name instead of abbreviation.Reviewer AgentReview Loop 1get_weather(“New York City”, unit=“F”)(Revised)Correct.Tool call is properly formatted with appropriate units. Full city name used correctly.Review Loop 2Execute Tool Callquery sentprovisional tool callfeedback injectedprovisional tool callfeedback injected

Figure 1:Example trajectory with inference-time feedback. The feedback agent (o3-mini) evaluates provisional tool calls from the tool-calling agent (GPT-4o) before execution. Loop 1: feedback provided. Loop 2: revised call approved.Large language models are increasingly deployed as agents that interact with external tools and APIs. These tool-calling agents face systematic challenges: selecting the correct tool, constructing calls with appropriate arguments, and recognizing when no tool can address a requestPatilet al.(2023,2024); Kokaneet al.(2025).

Two main classes of strategies address these challenges.Training-based approacheslike GRPOTanget al.(2024)require substantial compute and slow deployment.Inference-time approacheslike Self-RefineMadaanet al.(2023)and ReflexionShinnet al.(2023)enable self-correction without training, but require complex infrastructure and context management when agents must simultaneously generate and reflect on tool calls.

Both strategies face a fundamentalstate recovery problem. When an agent executes an incorrect action (such as deleting an alarm instead of updating it), self-correction requires maintaining the previous state in context. This becomes prohibitively expensive in complex execution environments and multi-turn scenarios, where the space of alternative trajectories grows exponentially. Without unreasonably large context windows (limited by model capacity and context budget), agents cannot reliably recover from destructive errors.

To address the native challenges of tool-calling agents and the state recovery problem, we proposeinference-time feedbackusing a simple, configurable duo-agent architecture: a specialized reviewer agent evaluates provisional tool calls before execution and either provides feedback to the tool-calling agent or uses a selection strategy to choose among candidates. An example trajectory with the reviewer model is shown in Figure1. The key proposal is simple separation of concerns, which yields strong benefits. First, it requires only one additional agent (configurable by model and review strategy) rather than complex infrastructure changes. Second, by reviewing calls before execution, it helps mitigate destructive errors rather than attempting recovery, reducing the state recovery problem. The tool-calling agent requires no retraining or rearchitecture and seamlessly adopts feedback from the reviewer.

However, introducing a reviewer brings trade-offs: feedback can mitigate errors but can also break valid responses. To quantify this, we introduceHelpfulness-Harmfulness metricsquantifying how often feedback corrects errors versus introduces new ones. Reviewer quality can be improved through model capacity or prompt optimization. Latency overhead can be reduced through distillation.

To find the optimal feedback agent configuration, we explore multiple review strategies (progressive feedback, best-of-N selection, and best-of-N grading) and address reviewer failures through automated prompt optimization (APO). APO optimizes only the reviewer’s prompt (the main agent’s prompt remains unchanged), automatically refining it by observing cases where the reviewer made incorrect judgments.

For the reviewer agent, we compare non-reasoning (GPT-4o) and reasoning models to assess the impact of reasoning on review quality. We use o3-mini for initial experiments, then GPT-5 mini (adopted upon release) for APO experiments to leverage the latest reasoning capability. We chose mini variants to balance reasoning capability with cost efficiency. The main tool-calling agent remains GPT-4o throughout.

Key Results

We evaluate on two benchmarks: BFCL (Berkeley Function-Calling Leaderboard)Patilet al.(2024)for single-turn function calling andτ2\tau^{2}-BenchBarreset al.(2025)for multi-turn stateful scenarios. Our best configuration achieves +5.5% on irrelevance detection (84.9%→\rightarrow90.4%), +1.6% on the relevance suite (90.9%→\rightarrow92.5%) on BFCL (Table5), and +7.1% onτ2\tau^{2}-Bench (48.7%→\rightarrow55.8%; Table8). Using our Helpfulness-Harmfulness metrics, we find that reasoning models (o3-mini) outperform standard language models as reviewers, achieving a 3:1 benefit-to-risk ratio (36.8% helpfulness, 11.7% harmfulness; Figure4).

Reasoning model comparison and APO are evaluated on BFCL only; extending these toτ2\tau^{2}-Bench remains future work.

This approach provides practical benefits for tool-calling systems: it requires no retraining or infrastructure modifications, supports rapid iteration on reviewer strategies through automated optimization, and offers tunable accuracy-latency trade-offs for different application requirements. The modular architecture enables organizations to enhance agent reliability incrementally while leaving existing tool-calling pipelines unchanged.

Key Contributions

In summary, our work makes the following contributions:

1. 1.Inference-time feedback mechanismimproving tool-calling performance without training, achieving +5.5% on irrelevance detection (BFCL) and +7.1% on multi-turn scenarios (τ2\tau^{2}-Bench).
2. 2.Helpfulness-Harmfulness metricsquantifying benefit-risk tradeoffs of feedback interventions, showing reasoning models achieve 3:1 ratios versus standard language models (BFCL).
3. 3.Automated reviewer prompt optimizationsystematically discovering effective review strategies via GEPAAgrawalet al.(2025), achieving +1.5% (relevance) and +2.8% (irrelevance) on BFCL (Table5).

2Method

Tool-CallingAgentReviewerAgentExecutionEnvironmentQueryResponseProvisionalTool CallFeedbackExecuteTool CallResultFigure 2:Feedback Architecture. The reviewer agent reviews provisional tool calls before execution. If errors are detected, feedback is provided for revision. This loop continues until approval or maximum iterations (N) is reached.### 2.1Multi-Agent Architecture

We evaluate three collaboration mechanisms between a tool-calling agent and a reviewer agent:

Progressive Feedback:The feedback agent iteratively reviews the tool-calling agent’s responses (Figure2). If errors are found, feedback is injected as a system message and the tool-calling agent generates a revised response. This continues for up to N review loops or until no errors are detected. We denote this asrN(e.g.,r2for up to 2 review loops).

Best-of-N Selection (Selector):The tool-calling agent generates N candidate responses with varying temperatures (0.3 to 1.0). The selector agent evaluates all candidates and selects the best one. This single-shot selection is denoted assN(e.g.,s5for 5 candidates).

Best-of-N Grading (Grader):Similar to selection, but the grader agent assigns explicit numeric scores (0.0-1.0) to each candidate with rationales. The highest-scored candidate is selected. Denoted asgN(e.g.,g5).

We use a systematic naming convention for all mechanisms when reporting the evaluation results. For example,4o-r2-5-mini-v3-gepaindicates GPT-4o base model, progressive feedback with up to 2 feedback loops (r2), GPT-5 mini feedback model (5-mini), GEPA prompt version 3. See AppendixA.3for concrete examples of how each mechanism operates.

2.2Reviewer Prompt Optimization

Manual prompt engineering for reviewer agents is labor-intensive and may miss subtle failure patterns. Inspired by GEPAAgrawalet al.(2025), which iteratively improves prompts based on execution feedback, we automatically refine reviewer prompts by observing cases where the reviewer made incorrect judgments. Starting from manually-refined v2 prompts for BFCL, we iteratively improve them using a reasoning model for reflection. Applying this optimization toτ2\tau^{2}-Bench prompts remains future work. Details and results appear in Section4.4.3.

3Experiment Setup

3.1Benchmarks

We evaluate on two benchmarks: BFCL (Berkeley Function Calling Leaderboard) for single-turn tool calling andτ2\tau^{2}-Bench for multi-turn stateful scenarios.

3.1.1BFCL

Single-turn, stateless tool calling. We evaluate on Non-Live (BFCL V1, curated) and Live (BFCL V2, community-contributed) categoriesPatilet al.(2024). Categories includesimple,multiple,parallel, andparallel_multiple(combined parallel and sequential; hardest), which together form therelevance suite. Theirrelevancecategory tests detecting when no tool is relevant.

3.1.2τ2\tau^{2}-Bench

Multi-turn, stateful tool calling with domain-specific policies across three domains (airline, retail, telecom). Agents must maintain conversational context, verify state preconditions, and handle benchmark-specific constraints.

3.2Models

All experiments use GPT-4o (gpt-4o-2024-11-20 snapshot) as the base tool-calling agent with temperature=0 and seed=42 for reproducibility. Initial experiments use o3-mini as the reasoning model reviewer; APO experiments use GPT-5 mini (adopted upon release to leverage the latest reasoning capability). Both reasoning models use reasoning_effort=medium.

4Results & Analysis

We organize our evaluation around three research questions:

• •RQ1 (Effectiveness & Error):What is the effectiveness of inference-time feedback for tool-calling agents, and what are the associated error correction trade-offs?
• •RQ2 (Design & Optimization):How do feedback mechanism design, reviewer model selection, and automated optimization affect reviewer agent performance?
• •RQ3 (Latency & Deployment):What are the latency overhead and deployment trade-offs of inference-time feedback across different application scenarios?

We answer these questions using both standard benchmark metrics and additional metrics that quantify error correction versus error introduction (Section4.1). We then address RQ1 by evaluating effectiveness on BFCL andτ2\tau^{2}-Bench (Section4.2), RQ2 by comparing reviewer models, feedback mechanisms, and automated prompt optimization (Section4.4), and RQ3 by analyzing latency overhead and deployment trade-offs (Section4.5).

4.1Evaluation Metrics

We evaluate using each benchmark’s default metrics: per-category accuracy for BFCL (simple, multiple, parallel, parallel_multiple, irrelevance) and relevance suite (unweighted average of the first four), and per-domain pass rate forτ2\tau^{2}-Bench (airline, retail, telecom). To complement these standard metrics, we introduce three metrics that quantify reviewer quality and error correction trade-offs:

• •Helpfulness:Percentage of test cases where base agent is wrong AND reviewer agent corrects it.
• •Harmfulness:Percentage of test cases where base agent is right AND reviewer introduces error.
• •Benefit-to-Risk Ratio:Helpfulness ÷ Harmfulness.

These metrics reveal whether feedback provides net positive value, showing not just final accuracy but how the reviewer affects correct and incorrect responses.

4.2Effectiveness & Error Trade-offs

RQ1: What is the effectiveness of inference-time feedback for tool-calling agents, and what are the associated error correction trade-offs?

4.2.1BFCL Evaluation

We first evaluate inference-time feedback on BFCL to establish baseline effectiveness on single-turn, stateless tool calling. Results are averaged across experimental sessions.

Initial Results

With minimal v1 prompts and GPT-4o as reviewer, we improved irrelevance detection but yielded only marginal gains on other categories.

Table 1:Initial Reviewer Impact (4o-r5-4o-v1).

Root Cause Analysis

The reviewer incorrectly flagged valid tool calls as “incomplete,” expecting execution results. However, as shown in Figure2, the reviewer evaluates provisional tool calls before execution, so no results exist yet. Analysis showed 23% of cases had redundant iterations where the reviewer demanded elaboration it should not expect. Figure3illustrates a typical example of this mismatch.

{“tool_calls”: [{“name”: “get_weather”, “arguments”: {“city”: “Seattle”}}]}Tool AgentSent for review“Response lacks follow-up explanation for the user about what the weather information means.”Reviewer Agent

Figure 3:Reviewer misunderstanding example. The reviewer expects user-facing dialogue, but BFCL only evaluates tool call accuracy.

Addressing Over-Skepticism

We added an explicit guideline addressing the over-skepticism failure mode:

[CRITICAL] Tool-only responses are complete.Do not mark tool-only responses as incomplete for lacking user-facing answers, follow-up explanations, or final results. Tool calls are standalone steps. Focus on whether the actual tool calls are correct.

The explicit guidance reduced redundant review loops from 23% to 8%, improving efficiency. However, v2 prompts reduced harmfulness but did not fully solve the irrelevance detection challenge. This motivated comparing different review models.

Table 2:Helpfulness vs. Harmfulness on BFCL Non-Live. Results from dedicated experiment measuring reviewer quality (see Figure4). Rel. = relevance suite, Irrel. = irrelevance, Help. = helpfulness, Harm. = harmfulness.Table2compares o3-mini and GPT-4o as reviewer models with manually-engineered prompts (v1, v2; see AppendixA.4for full prompts). The o3-mini configuration achieves the best performance: 91.8% on the relevance suite and 91.0% on irrelevance detection. Beyond accuracy, the helpfulness and harmfulness metrics reveal critical differences in reviewer quality. The o3-mini configuration corrects 36.8% of base agent errors while introducing errors in only 11.7% of previously correct responses, achieving a 3.1:1 benefit-to-risk ratio (Figure4). A detailed analysis of model comparison is presented in Section 5.2.

Figure 4:Helpfulness vs. Harmfulness trade-off on BFCL Non-Live. o3-mini achieves the best balance (3.1:1 ratio).

4.3τ2\tau^{2}-Bench Evaluation

We extend evaluation toτ2\tau^{2}-Bench to test whether feedback mechanisms generalize to multi-turn, stateful scenarios with domain-specific policies. Results are averaged over 3 benchmark runs.

Initial Results

Table3shows performance across three domains. The best configuration (4o-r5-4o-v1) achieves +7.1% average improvement over baseline (48.7% → 55.8%).

Table 3:Performance onτ2\tau^{2}-Bench across all mechanisms. Progressive Feedback achieves the highest average (55.8%). See Table8for additional details.

Error analysis

We analyzed failures to understand where feedback helps and hurts. Table4shows distribution.

Table 4:Failure Mode Distribution onτ2\tau^{2}-BenchFor example, in an airline booking domain, the policy requires checking seat availability before booking. Base agent error: directly callsbook_flight()without callingcheck_availability()first. Reviewer agent catches: “Response violates state precondition: seat availability must be checked.”

The reviewer effectively catches policy violations (-13%) but introduces new “over-verbalization” errors (+17%). The over-skepticism problem observed in BFCL recurs here: the reviewer flags tool-only responses in contexts whereτ2\tau^{2}-Bench expects mixed responses. Comparing mechanisms, Progressive Feedback substantially outperforms Best-of-N approaches (+3–8% on average), with selection actually underperforming the baseline on some domains (Table3).

Cross-benchmark generalization:Applying BFCL-optimized prompts (v2-bfcl) directly toτ2\tau^{2}-Bench introduces errors, highlighting the domain mismatch between single-turn function calling and multi-turn stateful agents. The different evaluation criteria and interaction patterns require benchmark-specific prompt adaptation.

Addressing Domain Constraints

τ2\tau^{2}-specific prompts (v2-tau; see AppendixA.4) emphasize context awareness (marked CRITICAL), state preconditions, and benchmark-specific constraints. However, baseline reviewer prompts (v1) sometimes outperform domain-tuned reviewer prompts on average across domains (55.8% vs. 52.9%; Table3), suggesting that manually-engineered instructions may not generalize across all tasks within a domain. This opens an opportunity for automated prompt optimization onτ2\tau^{2}-Bench as future work.

4.4Design & Optimization

RQ2: How do feedback mechanism design, reviewer model selection, and automated optimization affect reviewer agent performance?

4.4.1Model Comparison

On the BFCL benchmark, we compare o3-mini and GPT-4o as reviewer models with manually-engineered prompts (v1, v2; see AppendixA.4for full prompts) for initial experiments, then GPT-5 mini (its successor, as the more up-to-date reasoning model) for automated reviewer prompt optimization using GEPA (v3-gepa).

The reasoning model’s advantage manifests in two key areas. First, o3-mini outperforms GPT-4o on irrelevance detection by +0.6% (91.0% vs. 90.4%), a category that requires determining whether any available tool can address the request. Second, o3-mini exhibits lower harmfulness (11.7% vs. 12.9%; Table2), making fewer false positive errors. The reasoning model’s systematic verification reduces over-correction, leading to more reliable feedback.

Irrelevance detection:o3-mini outperforms GPT-4o by +0.6% on irrelevance (91.0% vs. 90.4% for v2 prompts; Table2). This category requires determining whether any available tool can address the request.

Error introduction rates:o3-mini exhibits lower harmfulness (11.7% vs. 12.9% for GPT-4o v2; Table2), making fewer false positive errors. The reasoning model’s systematic verification reduces over-correction.

Key insight:The goal is maximizing helpfulness while minimizing harmfulness (Figure4). The o3-mini configuration achieves the best balance: 36.8% of base agent errors corrected with 11.7% new errors introduced, achieving a 3.1:1 benefit-to-risk ratio. This demonstrates that reasoning models provide better feedback quality than standard language models.

4.4.2Mechanism Comparison

We evaluate three feedback mechanisms on BFCL to understand their strengths: Progressive Feedback (iterative refinement with feedback), Best-of-N Selection (choosing the best response from multiple candidates), and Best-of-N Grading (scoring and selecting from multiple candidates). Appendix Table6provides complete results across all mechanisms and categories.

Reasoning model advantage:As shown in Table2, o3-mini achieves the best helpfulness-to-harmfulness ratio (3.1:1) compared to GPT-4o (2.7:1 for v2 prompts). This suggests that reasoning models provide more reliable feedback with fewer false corrections.

Mechanism effectiveness:Progressive Feedback outperforms Best-of-N approaches on irrelevance detection by +4–5%, while Best-of-N shows only marginal gains on relevance. Progressive Feedback’s advantage lies in explicitly identifying and correcting errors through targeted feedback, particularly for irrelevance where the model must determine no tool can address the request.

4.4.3Automatic Context Optimization

Having established effectiveness on BFCL and demonstrated generalization onτ2\tau^{2}-Bench, we now systematize improvements through automated reviewer prompt optimization. We apply GEPA to BFCL reviewer prompts; extending this toτ2\tau^{2}-Bench remains future work.

GEPA Algorithm and Results

GEPA (Genetic-Pareto prompt evolution) addresses limitations of manual prompt engineering. The algorithm starts with manually-engineered reviewer prompts (v2), collects failure cases, uses an LLM to reflect and propose improvements, and iterates until convergence. This produces v3 prompts approximately 4.5×\timeslonger (1,599 vs. 358 tokens)111Measured using OpenAI’s tokenizer (https://platform.openai.com/tokenizer) with the GPT-5.x & O1/3 option.with detailed error criteria, edge case handling, and error checklists (AppendixA.4).

Table 5:GEPA Optimization on BFCL Non-Live (Progressive Feedback). v2 = manual, v3-gepa = GEPA-optimized. Rel. = relevance suite, Irrel. = irrelevance.The GEPA-optimized prompts show improvements across categories, with particularly strong gains on the parallel_multiple category (+2.1%; Table6). This category involves orchestrating multiple parallel tool calls with correct argument passing, where APO-optimized feedback catches mismatches and missing calls, demonstrating automated optimization discovers strategies difficult to anticipate manually.

4.5Latency & Deployment Trade-offs

RQ3: What are the latency overhead and deployment trade-offs of inference-time feedback across application scenarios?

Inference-time feedback introduces computational overhead that varies by task type. We analyze latency measurements from our experiments to characterize the cost-accuracy trade-off.

Latency Analysis:

We measure latency impact on both BFCL (single-turn function calling) andτ2\tau^{2}-Bench (multi-turn agent episodes). On BFCL, the feedback mechanism (r5-4o) increases average latency from 1.27s to 7.87s, a6.2× multiplier. This increase occurs because the baseline is a single inference call, so reviewer overhead dominates. Onτ2\tau^{2}-Bench, the same mechanism increases average episode duration from 158.7s to 384.3s, a2.4× multiplier. The lower relative impact reflects the multi-turn nature: reviewer overhead is amortized across approximately 40 turns per episode. Figure5shows the latency distributions.

Figure 5:Latency distributions. Top: BFCL per-item latency (log scale). Bottom:τ2\tau^{2}-Bench per-episode duration. Blue = baseline, coral = with reviewer. Dashed lines = means.

Reviewer Call Patterns:

On BFCL, feedback averages 1.33 reviewer calls per item. Onτ2\tau^{2}-Bench, the mechanism averages 0.96 reviewer calls per turn. The per-turn call count is lower forτ2\tau^{2}-Bench likely due to stateful nature reducing ambiguity in later turns.

Application-Specific Deployment:

The latency-accuracy trade-off allows flexible deployment strategies:

• •Single-turn, high-volume applications:The 6.2× latency overhead may be prohibitive for real-time use cases. Consider baseline deployment or selective feedback on uncertain cases.
• •Multi-turn agents:The 2.4× overhead is more acceptable when amortized across conversation turns. Feedback mechanisms are viable for complex, accuracy-critical workflows.
• •API-intensive workflows:Feedback may achieve ROI-positive gains by preventing spurious API calls that incur downstream costs.

5Related Work

Tool-Calling Benchmarks.BFCLPatilet al.(2024,2023)evaluates single-turn function calling across categories. ToolSandboxLuet al.(2025)introduces stateful, conversational evaluation.τ2\tau^{2}-Bench tests multi-turn tool calling with domain-specific policies. Our work is the first to systematically compare feedback mechanisms across both stateless (BFCL) and stateful (τ2\tau^{2}-Bench) benchmarks.

Agent Feedback and Self-Refinement.Self-RefineMadaanet al.(2023)and ReflexionShinnet al.(2023)use self-feedback for iterative refinement. Our work differs by using a specialized reviewer agent instead of self-feedback, demonstrating that external feedback from reasoning models outperforms self-correction.

Prompt Optimization.MIPROv2Opsahl-Onget al.(2024)uses Bayesian optimization for instructions and few-shot examples. GEPAAgrawalet al.(2025)uses genetic-Pareto evolution with LLM reflection. We apply GEPA to reviewer agent prompts, showing improvements over manual engineering.

Training-Based Approaches.GRPOTanget al.(2024)fine-tunes via weight-space RL but requires thousands of rollouts. Recent reasoning models (o1, o3)OpenAI (2024)demonstrate strong verification capabilities. Knowledge distillationHintonet al.(2015); Hoet al.(2023); Fuet al.(2023)provides a foundation for future work on distilling reviewer agents. Our inference-time approach provides immediate gains without training.

6Conclusion

We introduce Reinforced Agent, an inference-time feedback mechanism for tool-calling agents. Evaluation on BFCL establishes baseline effectiveness and identifies over-skepticism as the primary reviewer error mode. Evaluation onτ2\tau^{2}-Bench demonstrates generalization to multi-turn scenarios. Our Helpfulness-Harmfulness metrics show reasoning models achieve favorable benefit-to-risk ratios as reviewers. Automated prompt optimization via GEPA systematizes reviewer improvement over manual engineering. Latency overhead is most viable for complex, accuracy-critical workflows where it amortizes across turns, and can be further mitigated by distilling the reviewer into a smaller, faster model (e.g., a lightweight reward model or classifier) suitable for local or on-device deployment. This work establishes a practical path from benchmarking to optimization to deployment, with potential for distillation into reward models for reinforcement learning.

7Limitations

Our work has several limitations.

Base model scope.We evaluate only GPT-4o as the base tool-calling agent and compare two reviewer models (GPT-4o and o3-mini). While the modular architecture imposes no constraints on the base model (the reviewer operates on tool-call outputs regardless of which model generates them), generalization to open-source models (e.g., Llama, Mistral) and smaller proprietary models remains empirically unvalidated.

Analysis scope.GEPA-based prompt optimization and Helpfulness-Harmfulness metrics were applied only to BFCL; extending these to multi-turn benchmarks likeτ2\tau^{2}-Bench requires adapting for partial-credit scoring and multi-turn error propagation. Additionally, generic prompts (v1) sometimes outperform domain-specific prompts (v2-tau) onτ2\tau^{2}-Bench, suggesting manual tuning may not generalize without automated optimization.

8Ethics Statement

To the best of our knowledge, all results published in this paper are accurate. All data sources are publicly available benchmarks and are cited accordingly. No human subjects, private user data, or personally identifiable information were used in this work.

Acknowledgments

We thank Jaechan Lee for contributions to the exploratory experiments in this study during his internship at Apple. We are grateful to Sylvia Xu and Nishant Kanakia for detailed feedback on multiple drafts of this paper. We also thank Anatoly Adamov, Alex Braunstein, and Amar Subramanya for their continued support of this research.

References

• L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab (2025)GEPA: reflective prompt evolution can outperform reinforcement learning.External Links:2507.19457,LinkCited by:item 3,§2.2,§5.
• V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan (2025)τ2\tau^{2}-Bench: evaluating conversational agents in a dual-control environment.External Links:2506.07982,LinkCited by:§1.
• Y. Fu, H. Peng, L. Ou, A. Sabharwal, and T. Khot (2023)Specializing smaller language models towards multi-step reasoning.External Links:2301.12726,LinkCited by:§5.
• G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network.External Links:1503.02531,LinkCited by:§5.
• N. Ho, L. Schmid, and S. Yun (2023)Large language models are reasoning teachers.External Links:2212.10071,LinkCited by:§5.
• S. Kokane, M. Zhu, T. Awalgaonkar, J. Zhang, T. Hoang, A. Prabhakar, Z. Liu, T. Lan, L. Yang, J. Tan, R. Murthy, W. Yao, Z. Liu, J. C. Niebles, H. Wang, S. Heinecke, C. Xiong, and S. Savarese (2025)ToolScan: a benchmark for characterizing errors in tool-use llms.External Links:2411.13547,LinkCited by:§1.
• J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, F. Bai, S. Ma, S. Ma, M. Li, G. Yin, Z. Wang, and R. Pang (2025)ToolSandbox: a stateful, conversational, interactive evaluation benchmark for llm tool use capabilities.External Links:2408.04682,LinkCited by:§5.
• A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023)Self-refine: iterative refinement with self-feedback.External Links:2303.17651,LinkCited by:§1,§5.
• OpenAI (2024)Learning to reason with llms.Note:Accessed: 2025-02-01External Links:LinkCited by:§5.
• K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab (2024)Optimizing instructions and demonstrations for multi-stage language model programs.External Links:2406.11695,LinkCited by:§5.
• S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2024)Berkeley function calling leaderboard.Note:https://gorilla.cs.berkeley.edu/leaderboard.htmlCited by:§1,§1,§3.1.1,§5.
• S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2023)Gorilla: large language model connected with massive apis.External Links:2305.15334,LinkCited by:§1,§5.
• N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023)Reflexion: language agents with verbal reinforcement learning.External Links:2303.11366,LinkCited by:§1,§5.
• Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Á. Pires, and B. Piot (2024)Generalized preference optimization: a unified approach to offline alignment.External Links:2402.05749,LinkCited by:§1,§5.

Appendix AComplete Experimental Results

A.1BFCL Results

Table6presents complete results on the Berkeley Function Calling Leaderboard (BFCL) across all configurations tested. We report both Non-Live (BFCL V1, curated) and Live (BFCL V2, community-contributed) categories.222BFCL V1 contains curated benchmark tasks; V2 adds community-contributed real-world data. Seehttps://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboardfor dataset details.

Table 6:Complete BFCL Non-Live Results. All scores are percentages.APO: evaluates GEPA-optimized prompts using GPT-5 mini. Agent naming:{base}-{mechanism}{N}-{feedback_model}-{prompt_version}.Table 7:BFCL Live Results. Live categories are community-contributed and generally more challenging than Non-Live. Parallel categories omitted due to limited Live data.

A.2τ2\tau^{2}-Bench Results

Table8presents complete results onτ2\tau^{2}-Bench across all configurations tested.

Table 8:Completeτ2\tau^{2}-Bench Results. All scores are percentages. Best results per column in bold.

A.3Feedback Mechanism Examples

This section illustrates how each feedback mechanism operates with concrete examples using a weather query task.

A.3.1Progressive Feedback (r2): Iterative Refinement

Progressive feedback allows up to 2 review loops. The process continues until the feedback agent accepts the response or the maximum loops are reached.

Example execution:

User Query:“What’s the weather in New York City?”

Loop 1:

• •GPT-4o generates:get_weather(location=“NYC”, temp_unit=“celsius”)
• •o3-mini reviews:“Error: For US cities, temperature should use Fahrenheit by default.”

Loop 2:

• •GPT-4o revises with feedback:get_weather(location=“New York”, temp_unit=“fahrenheit”)
• •o3-mini reviews:“Correct. Tool call is properly formatted with appropriate units.”
• •Result:Accepted. Process terminates after 2 loops.

A.3.2Best-of-N Grading (g5): Scored Selection

The grader mechanism generates 5 candidate responses at different temperatures and assigns explicit scores to each.

Example execution:

User Query:“What’s the weather in New York City?”

Generated Candidates (temperatures 0.3–1.0):

1. 1.get_weather(location=“New York”)
2. 2.get_weather(location=“NYC”, temp_unit=“C”)
3. 3.get_weather(location=“New York”, temp_unit=“fahrenheit”)
4. 4.get_weather(“New York”)
5. 5.get_weather(location=“NY”)

o3-mini scores:

• •Candidate 1: 0.8 (missing temperature unit but acceptable)
• •Candidate 2: 0.3 (uses abbreviation “C” instead of full unit name)
• •Candidate 3:0.9(complete, proper formatting, correct unit)
• •Candidate 4: 0.6 (missing keyword argument)
• •Candidate 5: 0.7 (abbreviation “NY” less precise than full name)

Result:Selects Candidate 3 with highest score 0.9.

A.3.3Best-of-N Selection (s5): Direct Selection

The selector mechanism operates similarly to grading but picks the best candidate without explicit scoring.

Example execution:

User Query:“What’s the weather in New York City?”

The tool-calling agent generates 5 candidates (same as grading example above).

o3-mini evaluates and selects:

“Candidate 3 is best: it uses the full city name, includes proper keyword arguments, and specifies the appropriate temperature unit for US locations.”

Result:Selects Candidate 3 directly based on qualitative evaluation.

A.4Reviewer Prompt Examples

We present key versions of reviewer prompts showing the evolution from simple instructions to GEPA-optimized policies. Selector and grader prompts minimally adapt these with their own output sections.

A.4.1v1: Baseline Prompt

You are evaluating an assistant’s response for correctness, considering the full conversational context. ## Output Evaluate the assistant’s response candidate. Output your evaluation in the following format: {output_start_tag} { reasoning: string, (Detailed evaluation reasoning) message: string, (Brief explanation of why the response is correct or incorrect) error: boolean (Whether response is erroneous or contextually inappropriate) } {output_end_tag} Your response must be a valid JSON object and must be wrapped between {output_start_tag} and {output_end_tag} tags.

A.4.2v1-1: With Critical Guideline

Adds the key insight discovered through error analysis:

You are evaluating an assistant’s response for correctness, considering the full conversational context. [CRITICAL] Tool-only responses are complete.DO NOT MARK TOOL-ONLY RESPONSES AS INCOMPLETE ON THE BASIS OF LACKING *USER-FACING ANSWER*, *FOLLOW-UP EXPLANATION*, OR *FINAL RESULTS PRESENTATION* IN THE SAME RESPONSE. Tool call is a standalone step. Marking correct tool-calling responses as incomplete for these reasons is wrong. Focus instead on whether their actual tool calls are correct. [Output section same as v1]

A.4.3v2-bfcl: Manually Optimized for BFCL

The baseline manually-engineered prompt (358 tokens).

You are evaluating an assistant’s response for correctness, considering the full conversational context. ## Criteria ### Request Fulfillment - Is the response necessary and reasonably sufficient given the conversational context and the user request? ### Tool Call Correctness *(if response invokes tool calls)* - Are the selected tools appropriate? - Any syntax errors relative to the tool doc? Any type errors (e.g., float vs. integer)? - Are argument assignments correct? -Accept sensible defaults: If the parameter values are not explicitly specified in the user request, using default arguments is acceptable. ### Other Guidelines -Don’t be pedantic: Take a charitable interpretation of the assistant’s response. Critique logical failures, NOT surface-level features like tone, style, or minor inefficiencies. -No external facts: Base judgments solely on the conversation, tool doc, and other available information. -Binary solvability: Tasks are designed to be either reasonably solvable or unsolvable. -Unsolvability: If there is no way to reasonably address the request, recognize this as an unsolvable task rather than an assistant error. [Output section same as v1]

A.4.4v2-tau: Manually Optimized forτ2\tau^{2}-Bench

You are evaluating an assistant’s response for correctness, considering the full conversational context. ## Evaluation Criteria ### Context Awareness (CRITICAL) - Does the response logically follow given the entire conversational context? - If the task explicitly states constraints, policies, or rules for assistant actions, does the response apply them correctly? - If the task explicitly states preconditions for assistant actions, does the response verify them before executing actions? ### Request Fulfillment (CRITICAL) - Does the response make a sensible progress towards the complete fulfillment of the user’s last request? - Warning: The response might represent one step in an ongoing multi-step request handling. Partial completion is not erroneous if it represents reasonable logical progress. ### Logical Consistency - If the assistant communicates tool execution outcomes: Do they match what the tools actually returned? - If the assistant quotes its policy: Does it match what the policy actually states? ### Tool Call Correctness *(if response includes tool calls)* - Are the selected tools appropriate? - Any syntax errors relative to the tool documentation? - Are argument assignments correct? [Additional Guidelines and Output section follow]

A.4.5v3-2025-09-16-bfcl-gepa: GEPA-Optimized

The most comprehensive version (1,599 tokens). Key sections:

You are evaluating an assistant’s response for correctness, considering the full conversational context and the available tool documentation. Judge technical and logical correctness of tool use and task coverage; do not critique tone or style. [CRITICAL] Tool-only responses are complete. DO NOT MARK TOOL-ONLY RESPONSES AS INCOMPLETE ON THE BASIS OF LACKING *USER-FACING ANSWER*, *FOLLOW-UP EXPLANATION*, OR *FINAL RESULTS PRESENTATION* IN THE SAME RESPONSE. Review policy (apply all): - Evidence scope: Base judgments solely on the conversation and the provided tool documentation. Do not fact-check with outside knowledge. - Tool selection, necessity, and relevance: Approve a tool call only if the tool directly matches the user’s intent and can produce the requested target output. - Directness and parsimony: Prefer the minimal number of tool calls that directly yield the requested outputs. - Completeness across multi-step or multi-item requests: If the user asks for multiple actions, all such calls must be present with appropriate arguments. - Argument fidelity: Values must faithfully reflect the user’s constraints. Prefer canonical/minimal forms. - Thresholds and inequalities: Unless the tool documentation explicitly specifies strict inequality behavior, treat threshold-like parameters as inclusive bounds. - Units and scales: For rates/percentages, prefer normalized proportions (e.g., 0.03 for 3%) unless the tool explicitly requires 0–100. - Error criteria (mark error=true when any apply): Wrong or unnecessary tool used; missing required parameters; values that contradict user constraints; mis-scaled units; fabricating missing required values. [Full prompt available in supplementary materials]

A.5Latency Analysis

We analyze latency overhead of the feedback mechanism across both benchmarks. Table9presents summary statistics.

Table 9:Latency Statistics. BFCL measures per-item latency (seconds).τ2\tau^{2}-Bench measures per-episode duration (seconds).Key Observations.On BFCL (single-turn function calling), the feedback mechanism increases average latency by 6.2×. This substantial increase occurs because the baseline is a single inference call, so reviewer overhead dominates. Onτ2\tau^{2}-Bench (multi-turn agent episodes), the mechanism increases average duration by 2.4×. The lower relative impact reflects the multi-turn nature: reviewer overhead is amortized across approximately 40 turns per episode. Figure5shows the latency distributions.