CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants

# CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants
Source: [https://arxiv.org/html/2607.22635](https://arxiv.org/html/2607.22635)
###### Abstract

Target\-oriented dialogue systems have demonstrated strong capabilities in completing user goals through interactive conversations\. However, existing studies are primarily designed for single, explicit goal completion, while phone call assistants face a proxy setting that requires coordinating the device owner’s explicit preset goal with the caller’s implicit and dynamic goal\. We introduceCallBench, a Chinese benchmark for evaluating dual\-goal coordination in phone call assistants\.CallBenchcontains 50,000 complete multi\-turn phone call dialogues across six scenarios: takeout, delivery, taxi, work, life, and harassment\. It covers regular presets, emergent presets, and no\-preset cases, and includes diverse relations between owner\-side and caller\-side goals, such as alignment, complementarity, irrelevance, and conflict\. We further design a preset\-aware turn\-level evaluation protocol covering semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety\. Experiments on representative dialogue methods show that existing approaches still struggle with this task, highlighting the need for phone call assistants that can make reliable turn\-level decisions between two independent goals under proxy constraints\.

![Refer to caption](https://arxiv.org/html/2607.22635v1/x1.png)

Figure 1:A regular\-preset example fromCallBench\.## Introduction

Target\-oriented dialogue systems aim to guide multi\-turn interactions toward predefined goals, such as recommendation, persuasion, or task completion\(Lewiset al\.[2017](https://arxiv.org/html/2607.22635#bib.bib16); Wanget al\.[2019](https://arxiv.org/html/2607.22635#bib.bib18); Liuet al\.[2021](https://arxiv.org/html/2607.22635#bib.bib17)\)\. Existing studies have achieved strong progress in goal planning, response generation, and multi\-turn decision\-making\(Denget al\.[2023a](https://arxiv.org/html/2607.22635#bib.bib19)\)\. However, most of them assume a relatively simple setting: the system directly interacts with the user whose goal should be completed\. This single\-goal assumption does not fully capture the proxy setting of phone call assistants\.

A phone call assistant answers incoming calls on behalf of the device owner\. Therefore, it must coordinate two different sides of goals: the owner’s explicit preset and the caller’s implicit, dynamic goal\. These two goals may be aligned, complementary, irrelevant, or conflicting\. For example, the owner may preset the instruction “leave the takeout at the door”, while the caller reports that the food was spilled\. Blindly advancing the preset in such cases may lead to an irrelevant or even harmful response\. Thus, the key challenge is not merely completing a target, but deciding when the owner\-side goal should be advanced, delayed, or suspended according to the evolving caller\-side situation\.

This proxy setting also introduces strict safety boundaries\. The assistant must not reveal the owner’s private information, fabricate unsupported facts, claim access to the physical world, or make decisions on behalf of the owner\. In practice, the assistant can easily drift from reasonable assistance to unauthorized decision\-making, such as agreeing to changes, making commitments, or refusing requests without the owner’s confirmation\. Therefore, safety is not an additional requirement after goal completion, but a fundamental condition for deploying phone call assistants\.

Existing dialogue benchmarks are insufficient for evaluating these abilities\. Traditional task\-oriented dialogue datasets mainly focus on intent recognition, dialogue state tracking, API calls, and task success under user\-centered goals\(Budzianowskiet al\.[2018](https://arxiv.org/html/2607.22635#bib.bib1); Rastogiet al\.[2020](https://arxiv.org/html/2607.22635#bib.bib2)\)\. Target\-oriented dialogue benchmarks emphasize goal advancement, but usually assume a single stable target\(Lewiset al\.[2017](https://arxiv.org/html/2607.22635#bib.bib16); Wanget al\.[2019](https://arxiv.org/html/2607.22635#bib.bib18); Liuet al\.[2021](https://arxiv.org/html/2607.22635#bib.bib17); Denget al\.[2023a](https://arxiv.org/html/2607.22635#bib.bib19)\)\. Recent LLM\-based dialogue and agent benchmarks evaluate general response quality, tool use, or policy compliance\(Zhenget al\.[2023](https://arxiv.org/html/2607.22635#bib.bib15); Yaoet al\.[2024](https://arxiv.org/html/2607.22635#bib.bib8); Liet al\.[2023](https://arxiv.org/html/2607.22635#bib.bib22)\), yet they rarely consider a proxy assistant that must mediate between two independent persons under strict safety constraints\. As a result, we still lack a benchmark for evaluating dual\-goal coordination in phone call assistants\.

To address this gap, we introduceCallBench, a Chinese benchmark for evaluatingDual\-Goal Coordinationin phone call assistants\. Although real call assistants operate through voice interfaces, most practical systems follow an ASR–text processing–TTS pipeline\. We therefore focus on the text\-based dialogue processing stage, which allows us to isolate decision\-making and response\-generation abilities from speech recognition and synthesis errors\.CallBenchcontains 50,000 complete multi\-turn phone call dialogues across six scenarios: takeout, delivery, taxi, work, life, and harassment\. The benchmark covers regular presets, emergent presets, and no\-preset cases, and includes diverse relations between owner\-side and caller\-side goals\. Figure[1](https://arxiv.org/html/2607.22635#S0.F1)shows a regular\-preset takeout example inCallBench, where the assistant must first handle a caller\-side exception before returning to the owner\-side preset\.

We further design a preset\-aware turn\-level evaluation protocol for phone call assistants\. Instead of relying only on task success or surface\-level response quality, our evaluation measures semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety\. This allowsCallBenchto reveal whether an assistant can make appropriate local decisions under dual\-goal conditions, such as conveying a preset at the right time, handling caller\-side exceptions, avoiding premature or repeated preset delivery, and staying within proxy\-specific safety boundaries\.

Our main contributions are as follows:

- •We formulate the call assistant task as a turn\-level decision process, dynamically coordinating the owner’s explicit targets and the caller’s implicit intentions under strict safety constraints\.
- •We constructCallBench, a Chinese multi\-scenario benchmark containing 50,000 complete multi\-turn phone call dialogues across six scenarios, three preset cases, and diverse dual\-goal relations\.
- •We deploy a fine\-grained evaluation framework to assess model capabilities across multiple aspects\.
- •We evaluate representative task\- and target\-oriented dialogue frameworks, quantifying critical bottlenecks in dual\-goal trade\-offs, exception handling, and boundary control to guide future research\.

## Related Work

### Task\- and Target\-oriented Dialogue Benchmarks

Task\-oriented dialogue benchmarks have been widely used to evaluate dialogue state tracking, intent recognition, and task completion\. Representative datasets such as MultiWOZ\(Budzianowskiet al\.[2018](https://arxiv.org/html/2607.22635#bib.bib1)\), SGD\(Rastogiet al\.[2020](https://arxiv.org/html/2607.22635#bib.bib2)\), and CrossWOZ\(Zhuet al\.[2020](https://arxiv.org/html/2607.22635#bib.bib3)\)focus on multi\-domain slot filling and API\-oriented task execution\. Other datasets further introduce more realistic interaction settings, including transactional dialogues in Taskmaster\(Byrneet al\.[2019](https://arxiv.org/html/2607.22635#bib.bib4)\)and AirDialogue\(Weiet al\.[2018](https://arxiv.org/html/2607.22635#bib.bib5)\), implicit preference elicitation in Decision\-Oriented Dialogue\(Linet al\.[2024](https://arxiv.org/html/2607.22635#bib.bib6)\), and negotiation in CaSiNo\(Chawlaet al\.[2021](https://arxiv.org/html/2607.22635#bib.bib7)\)\. These benchmarks have advanced the evaluation of multi\-turn dialogue systems under explicit or partially implicit user goals\.

However, most of these benchmarks assume that the system directly serves the current interactor\. The system’s role is usually to understand and complete the user’s own task, rather than act as a proxy for another person\. This assumption does not match phone call assistants, where the assistant must coordinate the device owner’s explicit preset with the caller’s dynamic implicit goal\. Existing benchmarks provide limited coverage of cases where the two goals are irrelevant, complementary, or conflicting, and they rarely treat proxy\-specific constraints such as privacy protection and unauthorized decision\-making as central evaluation criteria\.

### Policy\-Constrained Agentic Dialogue Evaluation

Recent LLM\-agent benchmarks extend dialogue evaluation beyond text generation to tool use, policy compliance, and multi\-turn decision\-making\(Schicket al\.[2023](https://arxiv.org/html/2607.22635#bib.bib13); Liuet al\.[2024](https://arxiv.org/html/2607.22635#bib.bib14)\)\. For example,τ\\tau\-bench\(Yaoet al\.[2024](https://arxiv.org/html/2607.22635#bib.bib8)\)andτ2\\tau^\{2\}\-bench\(Barreset al\.[2025](https://arxiv.org/html/2607.22635#bib.bib9)\)evaluate agents in customer\-service environments with policy documents and controlled interactions\. ABCD\(Chenet al\.[2021](https://arxiv.org/html/2607.22635#bib.bib10)\)aligns customer requests with institutional policies and action sequences, while ToolTalk\(Farn and Shin[2023](https://arxiv.org/html/2607.22635#bib.bib11)\)studies safety issues in tool\-use dialogues, especially under irreversible actions\. Related work such as PAChat\(Fuet al\.[2025](https://arxiv.org/html/2607.22635#bib.bib12)\)further explores multi\-speaker and personalized communication settings\.

These benchmarks cover important aspects of policy following, action correctness, and communication structure, but they still differ from the proxy setting of phone call assistants\. Their constraints are usually defined by institutions or tools, while the interaction remains largely between a user and a service system\. In contrast, a phone call assistant mediates between two independent natural persons: the device owner and the caller\. It must decide when to advance, delay, or suspend the owner’s preset while respecting the caller’s current need and strict safety boundaries\.CallBenchtargets this gap by evaluating dual\-goal coordination under phone\-call\-specific constraints, including goal conflict handling, preset timing, privacy protection, and decision\-boundary compliance\.

## Problem Definition

Current real\-time call assistant systems are commonly built as a pipeline consisting of ASR, text processing, and TTS\. This paper focuses on the text processing stage\. We assume that the system has already obtained the textual form of the caller’s utterance and needs to generate the next response of the call assistant\. Speech recognition, speech synthesis, endpoint detection, and other speech\-side issues are beyond the scope of this work\. Under this setting, we define the task ofDual\-Goal Coordinationfor real\-time call assistants\.

Given the dialogue history at turntt:

Ht=\{\(u1,a1\),\(u2,a2\),…,\(ut−1,at−1\)\},H\_\{t\}=\\\{\(u\_\{1\},a\_\{1\}\),\(u\_\{2\},a\_\{2\}\),\\ldots,\(u\_\{t\-1\},a\_\{t\-1\}\)\\\},whereuiu\_\{i\}denotes the caller’s utterance at turnii, andaia\_\{i\}denotes the assistant’s response at turnii\. The current input includes the caller’s latest utteranceutu\_\{t\}, a scene labels∈𝒮s\\in\\mathcal\{S\}, and the owner’s preset pool𝒫o\\mathcal\{P\}^\{o\}, where:

𝒮=\{takeout,delivery,taxi,work,life,harassment\}\.\\mathcal\{S\}=\\\{\\text\{takeout\},\\text\{delivery\},\\text\{taxi\},\\text\{work\},\\text\{life\},\\text\{harassment\}\\\}\.
The owner’s preset pool𝒫o\\mathcal\{P\}^\{o\}contains goal\-related information predefined by the device owner\. To align with practical phone call assistant systems under deployment, we model𝒫o\\mathcal\{P\}^\{o\}as a structured preset interface rather than an arbitrary set of instructions\. Each preset can be represented as:

pio=\(qio,rio\),p\_\{i\}^\{o\}=\(q\_\{i\}^\{o\},r\_\{i\}^\{o\}\),whereqioq\_\{i\}^\{o\}denotes the trigger expression or trigger condition, andrior\_\{i\}^\{o\}denotes the corresponding owner\-specified response\. Each preset slot may also be left empty\.

The preset pool contains two types of slots: regular preset slots and emergent preset slots\. Regular preset slots correspond to default owner instructions for high\-frequency service calls\. In our setting, regular preset slots are only defined fortakeout,delivery, andtaxi\. Each of these scenes has at most one regular preset slot\. For a regular preset, the trigger expressionqioq\_\{i\}^\{o\}is predefined by the system, and the owner only specifies the responserior\_\{i\}^\{o\}\.

A regular preset is available whenever the current service scene has a non\-empty regular slot\. If the caller’s utterance matches the default trigger query, the assistant may directly convey the owner\-specified response\. If the utterance does not yet match the default trigger query, the assistant should not treat the preset as irrelevant\. Instead, it may ask for missing information, confirm the caller’s situation, or guide the conversation toward a state where the preset can be naturally conveyed\. However, if the caller raises an exception or a conflicting request, the assistant should delay or suspend the preset rather than force it\.

Emergent preset slots correspond to user\-defined conditional instructions\. For an emergent preset, both the trigger expressionqioq\_\{i\}^\{o\}and the responserior\_\{i\}^\{o\}are specified by the owner\. Unlike regular presets, emergent presets may be activated in any of the six scenarios, but only when the current caller utterance matches the user\-defined trigger expression\.

At turntt, the assistant retrieves the owner\-side explicit goal available in the current context:

gto=RetrievePreset​\(Ht,ut,s,𝒫o\)\.g\_\{t\}^\{o\}=\\textsc\{RetrievePreset\}\(H\_\{t\},u\_\{t\},s,\\mathcal\{P\}^\{o\}\)\.The retrieval process checks the preset pool according to the dialogue history, caller utterance, and current scene\. Fortakeout,delivery, andtaxi, the assistant first checks the regular preset slot associated with the current scene\. If the slot is empty, the assistant then checks whether any emergent preset is triggered by the current utterance\. Forwork,life, andharassment, the assistant only checks whether the current caller utterance matches any user\-defined emergent trigger condition\. If no preset is available, thengto=∅g\_\{t\}^\{o\}=\\varnothing\.

This setting gives rise to three preset cases\. Aregular presetcase occurs when the current service scene has a non\-empty regular preset slot\. Anemergent presetcase occurs when a user\-defined trigger condition is matched by the caller’s utterance\. Ano\-presetcase occurs when no owner\-side preset is available or activated in the current turn\. Since every regular slot and emergent slot can be empty, no\-preset cases may appear in any scenario\.

In contrast to the owner\-side explicit goal, the caller’s goal at the current turn is denoted asgtcg\_\{t\}^\{c\}\. This goal is usually not explicitly provided before the call, but needs to be understood from the dialogue history, current utterance, and scene information\. The caller\-side implicit goal may be aligned with, complementary to, irrelevant to, or conflicting with the current owner\-side goal\.

This setting gives rise to a turn\-level dual\-goal coordination problem in real\-time call assistants\. At each turn, the system needs to determine the relationship between the current owner\-side explicit goalgtog\_\{t\}^\{o\}and the caller\-side implicit goalgtcg\_\{t\}^\{c\}, and generate the assistant response:

at=f​\(Ht,ut,s,gto,gtc;𝒞\),a\_\{t\}=f\(H\_\{t\},u\_\{t\},s,g\_\{t\}^\{o\},g\_\{t\}^\{c\};\\mathcal\{C\}\),where𝒞=\{Cpriv,Cdec,Cfact,Cphys\}\\mathcal\{C\}=\\\{C\_\{\\text\{priv\}\},C\_\{\\text\{dec\}\},C\_\{\\text\{fact\}\},C\_\{\\text\{phys\}\}\\\}denotes the set of safety boundary constraints that the call assistant must follow\. Specifically,CprivC\_\{\\text\{priv\}\}denotes the privacy protection constraint, requiring the system not to disclose the owner’s personal information, location, contact information, or other sensitive content;CdecC\_\{\\text\{dec\}\}denotes the decision\-boundary constraint, requiring the system not to make commitments, agreements, refusals, or choices on behalf of the owner;CfactC\_\{\\text\{fact\}\}denotes the factuality constraint, requiring the system not to fabricate information that is not provided in the dialogue context or the owner’s presets; andCphysC\_\{\\text\{phys\}\}denotes the non\-embodiment constraint, requiring the system not to present itself as an embodied agent with direct sensory access or physical agency in the real world, such as claiming that it can see an object, check a location, or physically handle an item\. The responseata\_\{t\}should advance or complete the owner\-side goal at an appropriate time, while also properly addressing the caller’s current request\. When the two goals conflict, when the current information is insufficient, or when the conversation is ready to end, the system should adjust its response according to the current context rather than mechanically pushing the original owner\-side goal\.

Under this definition, a call assistant should be evaluated not only by the naturalness and fluency of its responses, but also by its ability to correctly understand the caller’s current request, activate and advance the owner’s preset goal at the right time, handle deviation or conflict between the two goals, and consistently obey the safety boundaries\.

## Benchmark Construction

Real phone calls often contain private information and are difficult to release as public evaluation data\. To enable controlled and reproducible evaluation, we constructCallBench, a Chinese benchmark forDual\-Goal Coordinationin phone call assistants\. In total,CallBenchcontains 50,000 multi\-turn Chinese phone call dialogues across six incoming\-call scenarios, covering different preset cases, caller\-side goals, owner\-side goals, and dual\-goal relations\.

### Scenario and Preset Configuration

CallBenchcovers the six incoming\-call scenarios defined above, spanning short service calls and more open\-ended interpersonal calls\. Intakeout,delivery, andtaxi, calls are usually short and revolve around a concrete service task, such as delivering an order, placing a package, or coordinating a ride\. Inwork,life, andharassment, caller intents are more diverse, requiring the assistant to retain useful information, respond cautiously, or terminate the call safely\.

For each scenario, we define possible caller roles, call purposes, and conversation states\. Intakeout,delivery, andtaxi, callers may report normal arrival, address ambiguity, item damage, failed delivery, or ask for confirmation\. Inworkandlife, callers may ask for the owner, leave a message, provide new information, or request a decision\. Inharassment, callers may engage in telemarketing, repeatedly ask irrelevant questions, try to obtain private information, or pressure the assistant into making unauthorized commitments\.

Following the preset cases defined in the problem definition,CallBenchincludes regular presets, emergent presets, and no\-preset instances\. Regular presets are included only intakeout,delivery, andtaxi, where stable owner instructions commonly exist\. Emergent presets and no\-preset instances may appear in any of the six scenarios\. This design allows the benchmark to cover both scene\-level default instructions and turn\-level conditional owner goals, while also testing cases where no owner\-side goal is available\.

### Dual\-Goal Relation Design

A central property of the benchmark is that each dialogue instance contains both an owner\-side explicit goal and a caller\-side implicit goal\. The owner\-side goal is represented by the preset pool, while the caller\-side goal is reflected through the caller’s role, motivation, and utterances\. We explicitly construct different relations between the two goals, including alignment, complementarity, irrelevance, and conflict\.

In aligned cases, the caller\-side goal already supports advancing the owner\-side goal, such as a delivery person reporting arrival under a leave\-at\-door instruction\. In complementary cases, the caller\-side goal does not directly trigger the preset, but can help establish the missing condition for advancing it\. The assistant should respond to the caller\-side situation and guide the dialogue toward a state where the owner preset can be naturally conveyed\. In irrelevant cases, the caller\-side goal is unrelated to the owner preset, so forcing the preset into the turn would be unnatural or premature\. In conflict cases, the caller\-side goal blocks or contradicts advancement of the owner\-side goal, such as reporting a spilled order, a damaged package, or an inability to complete the service as originally planned\.

This design allows the benchmark to evaluate not only whether the owner\-side goal can be completed, but also whether it is advanced timely\. In many cases, a desirable response should delay, suspend, or ignore the owner preset when context makes direct goal execution inappropriate\.

### Dialogue Construction

Table 1:Statistics ofCallBench\. We report the number of dialogues and the average turns per dialogue\.We construct complete multi\-turn phone call dialogues through a structured role\-play generation process\. For each dialogue, we first prepare a scenario description that specifies the scene, caller role, caller\-side goal, preset case, owner target response, trigger query if available, and possible abnormal conditions\. The caller\-side goal describes why the caller makes the call and what the caller wants to achieve in the conversation\. For example, in the takeout scene, the caller\-side goal may be to report that the food was spilled and ask how to handle the order\.

We use Doubao\-Seed\-1\.6\(ByteDance Seed[2025](https://arxiv.org/html/2607.22635#bib.bib23)\)to implement two role\-playing agents, one acting as the caller and the other acting as the phone call assistant\. The caller agent is given the caller role, caller\-side goal, and abnormal conditions, and is instructed to behave like a real caller in a phone conversation\. The assistant agent is given the dialogue history, scene label, owner preset information, trigger condition, and safety constraints, and is asked to respond as a phone call assistant answering on behalf of the owner\.

Since the caller\-side goal and the owner\-side preset are designed from two different perspectives, they are not always naturally aligned\. Within each scene, we combine different caller\-side goals with different owner presets, so that the generated calls can cover diverse dual\-goal relations\.

Under each configuration, the two role\-playing agents generate multiple complete calls with the temperature set to1\.01\.0\. In this way, the goal relation is reflected through the actual dialogue flow rather than only assigned as a static label\. During generation, we encourage short spoken utterances and avoid written expressions, so that the dialogues better resemble real phone calls\. After a dialogue is generated, we build the owner preset pool for this dialogue according to its preset case\. In addition, we also randomly insert silence turns where the caller utterance is marked as ”caller no response” to reflect irregularities in real phone calls\.

CallBenchstores each generated sample as a complete multi\-turn dialogue\. During evaluation, assistant responses are assessed turn by turn within the full dialogue context\. For each assistant response, the evaluator considers the preceding dialogue history, the current caller utterance, the scene label, and the owner preset pool\. This setting preserves the temporal context of the phone call while allowing fine\-grained assessment of goal timing, caller\-side issue handling, and natural conversation closing\.

### Quality Control

We apply quality control throughout the dialogue construction process\. The automatic checking modules are implemented with Doubao\-Seed\-2\.0\-lite\(ByteDance Seed[2026](https://arxiv.org/html/2607.22635#bib.bib24)\)\. At each assistant turn, the generation pipeline tracks whether the caller utterance matches the trigger query, whether the owner\-side target has already been conveyed, and whether advancing the preset would conflict with the caller’s current request or the scene condition\. These checks help avoid premature preset delivery, repeated conveyance, and inappropriate goal advancement in conflict cases\.

Candidate assistant responses are validated before being added to the dialogue history\. A response is rejected if it fails to follow the latest caller utterance, ignores the previous context, repeats information unnecessarily, or sounds unlike a phone conversation\. We also filter responses that violate the safety boundaries of phone call assistants, including unauthorized decisions on behalf of the owner, fabricated information, privacy leakage, and claims of direct access to the physical world\. For example, the assistant should not claim that it can see a parcel, inspect food damage, identify a vehicle, or handle an item in the real world\.

If a generated assistant response fails these checks, it is regenerated\. Dialogues with invalid role behavior, inconsistent goal relations, severe safety violations, or unnatural conversation flow are removed\. In addition, we manually inspect 10% of the generated dialogues at the dialogue level using the same criteria\. A dialogue receives a score of 1 only when no issue is found, 0\.5 when it contains minor imperfections that do not affect the main goal or safety, and 0 when it contains severe errors or violates core safety rules\. The inspected subset achieves an average score of 0\.93\.

### Data Statistics

CallBenchcontains 50,000 complete multi\-turn Chinese phone call dialogues across six high\-level scenarios and we split it into training, validation, and test sets with a ratio of 8:1:1\. Each dialogue is associated with a scene label, caller\-side goal, preset case and the owner preset pool\.

The dataset is approximately balanced across scenarios\. Fortakeout,delivery, andtaxi, regular\-preset, emergent\-preset, and no\-preset instances are distributed with an approximate ratio of 3:1:1\. Forwork,life, andharassment, where no regular preset slots are defined, the data is approximately balanced between emergent\-preset and no\-preset instances\. Detailed information is listed in Table[1](https://arxiv.org/html/2607.22635#Sx4.T1)\.

## Evaluation Protocol

Table 2:Main results onCallBenchwith Doubao\-Seed\-1\.6 as the backbone\. Higher is better\. Best results arebolded, and the second best ones areunderlined\.### Turn\-level Multi\-dimensional Evaluation

CallBenchevaluates phone call assistants at the turn level\. For each assistant turn, the evaluator receives the dialogue historyHtH\_\{t\}, current caller utteranceutu\_\{t\}, scene labelss, preset metadata, and the assistant response\. The goal is to judge whether the response appropriately handles the caller\-side need, the owner\-side preset, and the safety boundaries of phone call assistants\. Since preset timing is context\-dependent and may allow multiple acceptable responses, we do not define a single gold action for each turn\. For example, when the caller reports an exception, the assistant may either ask for more details or state that the issue will be relayed to the owner, as long as it does not force the preset prematurely or make unauthorized decisions\. We therefore adopt a rubric\-based automatic judge for scalable evaluation\. We use Qwen3\.7\-Max\(Qwen Team[2026b](https://arxiv.org/html/2607.22635#bib.bib27)\)as the judge, scoring each applicable dimension as0,0\.50\.5, or11, indicating severe issue, minor issue and no issue\. The rubric covers semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety\.

### Evaluation Dimensions

The evaluation covers both general response quality and dual\-goal coordination ability\.

Semantic Understandingmeasures whether the assistant correctly understands the current caller utterance and provides a logically relevant response\. A response receives a low score when it answers the wrong question, ignores the caller’s request, or misunderstands the current situation\.

Context Usemeasures whether the assistant correctly uses the dialogue history\. This dimension is important in multi\-turn calls, where the assistant must avoid asking repeated questions, contradicting previous information, or forgetting that the owner\-side goal has already been conveyed\.

Active Guidancemeasures whether the assistant guides the conversation at the right time while coordinating the owner\-side preset with the caller\-side goal\. In regular\-preset cases, the assistant should guide the caller toward a state where the preset can be naturally conveyed, but should not ignore caller\-side exceptions or constraints\. After the preset is conveyed, it should stop unnecessary guidance and move toward confirmation or closing\. In emergent\-preset cases, the assistant should not induce the caller to say the trigger condition, but should respond naturally to the current caller\-side goal\. For no\-preset cases, this dimension is not applied\.

Response Qualityevaluates the linguistic and conversational quality of the response\. The response should be natural, concise, colloquial, and suitable for a phone call\. It should not be overly written, unnecessarily verbose, repetitive, mechanically templated, or grammatically awkward\.

Preset Compliancemeasures whether the assistant follows the owner\-side preset at the appropriate time\. For regular presets, the assistant should convey the owner target response when the dialogue context is ready, but should not force the preset when the caller reports an exception or conflict\. For emergent presets, the assistant should convey the target response only when the current caller utterance matches the trigger query\. If the trigger is not matched, premature delivery of the preset is penalized\. For no\-preset cases, this dimension is not applied\.

Dialogue Rhythmevaluates whether the assistant manages the progression and closing of the call appropriately, with assessment criteria varying across distinct scenarios\. The assistant should respond to the current caller utterance in a meaningful way, close the call only after the key information has been exchanged and the caller has no further requirement, and avoid unnecessarily prolonging the conversation after the caller\-side issue has been handled and the owner\-side goal has been completed\. For silence turns, the assistant should handle the lack of caller input reasonably, such as checking whether the caller is still available\.

Safetyevaluates whether the assistant obeys the safety constraint set𝒞\\mathcal\{C\}defined in the problem definition\. This includes avoiding privacy leakage, unauthorized decision\-making, unsupported factual claims, and false claims of real\-world access\. Unlike other dimensions, safety is scored only as0or11\. Any violation of𝒞\\mathcal\{C\}results in a safety score of0\.

### Score Aggregation

For each turn, the judge outputs both an overall score and scores for the applicable dimensions\. The overall score is assigned on a three\-level scale:11indicates that the response has no major issue,0\.50\.5indicates that it contains minor or partial issues, and0indicates a severe error or a violation of core requirements\. Dimensions that are not applicable to the current preset case are excluded\. We report both the overall score and the scores of individual dimensions\.

## Experiments

Table 3:Overall scores on regular\-preset cases\. LLM denotes the automatic evaluation score, while Human denotes the score from human evaluation on sampled instances\. Best results arebolded, and the second best ones areunderlined\.### Experimental Settings

We evaluate different task\-oriented dialogue\(Zhanget al\.[2020](https://arxiv.org/html/2607.22635#bib.bib25)\)methods on the test split ofCallBench, including Direct Prompting \(DP\), ReAct\(Yaoet al\.[2022](https://arxiv.org/html/2607.22635#bib.bib28)\), SimpleTOD\(Hosseini\-Aslet al\.[2020](https://arxiv.org/html/2607.22635#bib.bib32)\), DivTOD\(Zenget al\.[2024](https://arxiv.org/html/2607.22635#bib.bib34)\), AutoTOD\(Xuet al\.[2024](https://arxiv.org/html/2607.22635#bib.bib29)\), ProCoT\(Denget al\.[2023b](https://arxiv.org/html/2607.22635#bib.bib30)\), EnPL\(Zhenget al\.[2024](https://arxiv.org/html/2607.22635#bib.bib33)\), and ChatSOP\(Liet al\.[2025](https://arxiv.org/html/2607.22635#bib.bib31)\)\. We do not train or fine\-tune any baseline onCallBench; all methods are evaluated in an inference\-only setting\. For methods that originally involve training, fine\-tuning, or learned modules, we replace training\-dependent components with prompt\-based adaptation using the same observable input\. This keeps the comparison focused on how different dialogue\-control paradigms coordinate the two goals, rather than on additional supervised training\. Each system receives the same input, including dialogue history, current caller utterance, scene label, safety constraints, and the owner preset associated with the current instance, which isolates response\-level dual\-goal coordination from upstream preset selection\. Unless otherwise specified, all experiments use Doubao\-Seed\-1\.6 with temperature 0 as the backbone model\. We additionally conduct experiments with Qwen3\.5\-27B\(Qwen Team[2026a](https://arxiv.org/html/2607.22635#bib.bib26)\)to examine backbone robustness, with results reported in Appendix A\.

### Main Results

Table[2](https://arxiv.org/html/2607.22635#Sx5.T2)reports the main results onCallBench\. The results show that existing task\- and target\-oriented dialogue methods do not transfer directly to phone call assistants\. Although these methods are designed for structured task completion or goal progression, they generally underperform ReAct on the overall score, and some even fall behind the simple direct\-prompting baseline\. This suggests that current dialogue methods still lack a comprehensive solution for phone call assistants: they may perform well on certain aspects, but struggle to balance goal coordination, response quality, dialogue rhythm, and safety simultaneously\. These results demonstrate the value ofCallBench, whose multi\-dimensional evaluation reveals weaknesses that would be overlooked by overall scores alone\.

### Results for Regular Preset

![Refer to caption](https://arxiv.org/html/2607.22635v1/x2.png)Figure 2:Error distribution of the evaluated methods\. Preset non\-comp\. stands for preset non\-compliance\.Table[3](https://arxiv.org/html/2607.22635#Sx6.T3)presents the results on regular\-preset cases across the three service scenarios\. These cases most directly test dual\-goal coordination, since the owner\-side preset remains available throughout the call while the caller\-side goal may evolve, remain incomplete, or conflict with it\. We report both LLM\-judge scores on the full turn\-level subset and human ratings on sampled turn\-level instances\. For human evaluation, we sample 250 turn\-level instances from each scenario and ask annotators to follow the same scoring rubric as the LLM judge, assigning11,0\.50\.5, or0according to whether the response has no issue, minor issues, or severe errors\. ReAct achieves the strongest average performance under both evaluation settings, while methods with higher LLM\-judge scores generally also receive higher human ratings\. To examine the relation between automatic and human evaluation, we compute rank correlation over the 24 method\-scenario pairs in Table[3](https://arxiv.org/html/2607.22635#Sx6.T3)\. The automatic and human scores show a positive association, with Kendall’sτb=0\.764\\tau\_\{b\}=0\.764\. This result is used as a sanity check rather than a direct inter\-annotator agreement measure, since the scores are aggregated and the human ratings are collected on sampled turn\-level instances\. At the same time, the absolute scores remain far from saturated, indicating that existing methods still struggle with coordination between two independent goals\.

### Error Analysis

Figure[2](https://arxiv.org/html/2607.22635#Sx6.F2)reports method\-wise error rates for the six most frequent diagnostic error types, selected by their method\-averaged rates across all evaluated methods\. The most common error is safety issue, with an average rate of 10\.63%, suggesting that existing assistants still struggle to maintain proxy boundaries when responding on behalf of the owner\. In addition, dialogue\-level errors are also frequent: repetitive reply and improper closing reach 6\.68% and 6\.57% on average, respectively, indicating that many assistants still have difficulty controlling dialogue rhythm in phone calls, either repeating similar expressions or ending the call at an inappropriate time\. Meanwhile, preset\-related errors remain prominent\. Over\-following the preset reaches 6\.18% on average, while repeated preset delivery reaches 5\.23%\. These errors capture different failures in owner\-side goal handling: applying the preset despite an unsuitable caller\-side situation, or unnecessarily repeating it after delivery\. Irrelevant answers account for 6\.20% on average, showing that assistants may produce locally plausible but contextually inappropriate responses when caller\-side requests deviate from the preset flow\. These results suggest that existing methods fail less because of surface\-level fluency and more because of turn\-level decision\-making under dual\-goal conditions, especially safety boundaries, dialogue rhythm, and preset timing under noisy, evolving caller\-side contexts\.

## Conclusion

We presentCallBench, a Chinese benchmark for dual\-goal coordination in phone call assistants\. The benchmark targets a realistic proxy setting where the assistant must answer calls for the device owner, coordinate the owner’s explicit target with the caller’s implicit goal, and obey strict safety boundaries\. We formalize structured owner preset pools with regular, emergent, and no\-preset cases, and construct 50,000 multi\-turn dialogues across six incoming\-call scenarios\. We propose a turn\-level evaluation protocol measuring semantic understanding, context use, active guidance, response quality, preset compliance, dialogue rhythm, and safety\. Our evaluation shows existing task\-oriented dialogue methods still struggle, suggesting that effective phone call assistants must reason about two independent goals under strict proxy constraints and make reliable decisions throughout the call\. We hopeCallBenchwill support future research on reliable call assistants in realistic settings\.

## References

- V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan \(2025\)τ2\\tau^\{2\}\-bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. Gasic \(2018\)Multiwoz\-a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 5016–5026\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1),[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.
- B\. Byrne, K\. Krishnamoorthi, C\. Sankar, A\. Neelakantan, D\. Duckworth, S\. Yavuz, B\. Goodrich, A\. Dubey, A\. Cedilnik, and K\. Kim \(2019\)Taskmaster\-1: toward a realistic and diverse dialog dataset\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 4516–4525\.Cited by:[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.
- ByteDance Seed \(2025\)Seed1\.6\.Note:https://seed\.bytedance\.com/en/seed1\_6Cited by:[Dialogue Construction](https://arxiv.org/html/2607.22635#Sx4.SSx3.p2.1)\.
- ByteDance Seed \(2026\)Seed2\.0\.Note:https://seed\.bytedance\.com/en/seed2Cited by:[Quality Control](https://arxiv.org/html/2607.22635#Sx4.SSx4.p1.1)\.
- K\. Chawla, J\. Ramirez, R\. Clever, G\. Lucas, J\. May, and J\. Gratch \(2021\)Casino: a corpus of campsite negotiation dialogues for automatic negotiation systems\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3167–3185\.Cited by:[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.
- D\. Chen, H\. Chen, Y\. Yang, A\. Lin, and Z\. Yu \(2021\)Action\-based conversations dataset: a corpus for building more in\-depth task\-oriented dialogue systems\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3002–3017\.Cited by:[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- Y\. Deng, W\. Lei, W\. Lam, and T\. Chua \(2023a\)A survey on proactive dialogue systems: problems, methods, and prospects\.arXiv preprint arXiv:2305\.02750\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1)\.
- Y\. Deng, L\. Liao, L\. Chen, H\. Wang, W\. Lei, and T\. Chua \(2023b\)Prompting and evaluating large language models for proactive dialogues: clarification, target\-guided, and non\-collaboration\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10602–10621\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- N\. Farn and R\. Shin \(2023\)Tooltalk: evaluating tool\-usage in a conversational setting\.arXiv preprint arXiv:2311\.10775\.Cited by:[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- D\. Fu, X\. Cheng, L\. Li, X\. Yang, L\. Yang, and T\. Jin \(2025\)PACHAT: persona\-aware speech assistant for multi\-party dialogue\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 29313–29330\.Cited by:[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- E\. Hosseini\-Asl, B\. McCann, C\. Wu, S\. Yavuz, and R\. Socher \(2020\)A simple language model for task\-oriented dialogue\.Advances in neural information processing systems33,pp\. 20179–20191\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- M\. Lewis, D\. Yarats, Y\. Dauphin, D\. Parikh, and D\. Batra \(2017\)Deal or no deal? end\-to\-end learning of negotiation dialogues\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,pp\. 2443–2453\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1)\.
- M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li \(2023\)Api\-bank: a comprehensive benchmark for tool\-augmented llms\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 3102–3116\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1)\.
- Z\. Li, J\. Peng, Y\. Wang, Y\. Cao, T\. Shen, M\. Zhang, L\. Su, S\. Wu, Y\. Wu, Y\. Wang,et al\.\(2025\)ChatSOP: an sop\-guided mcts planning framework for controllable llm dialogue agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17637–17659\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- J\. Lin, N\. Tomlin, J\. Andreas, and J\. Eisner \(2024\)Decision\-oriented dialogue for human\-ai collaboration\.Transactions of the Association for Computational Linguistics12,pp\. 892–911\.Cited by:[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2024\)Agentbench: evaluating llms as agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 52989–53046\.Cited by:[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- Z\. Liu, H\. Wang, Z\. Niu, H\. Wu, and W\. Che \(2021\)DuRecDial 2\.0: a bilingual parallel corpus for conversational recommendation\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 4335–4347\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1)\.
- Qwen Team \(2026a\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- Qwen Team \(2026b\)Qwen3\.7: the agent frontier\.External Links:[Link](https://qwen.ai/blog?id=qwen3.7)Cited by:[Turn\-level Multi\-dimensional Evaluation](https://arxiv.org/html/2607.22635#Sx5.SSx1.p1.6)\.
- A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. Khaitan \(2020\)Towards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 8689–8696\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1),[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- X\. Wang, W\. Shi, R\. Kim, Y\. Oh, S\. Yang, J\. Zhang, and Z\. Yu \(2019\)Persuasion for good: towards a personalized persuasive dialogue system for social good\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 5635–5649\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1)\.
- W\. Wei, Q\. Le, A\. Dai, and J\. Li \(2018\)Airdialogue: an environment for goal\-oriented dialogue research\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 3844–3854\.Cited by:[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.
- H\. Xu, X\. Mao, P\. Yang, F\. Sun, and H\. Huang \(2024\)Rethinking task\-oriented dialogue systems: from complex modularity to zero\-shot autonomous agent\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2748–2763\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1),[Policy\-Constrained Agentic Dialogue Evaluation](https://arxiv.org/html/2607.22635#Sx2.SSx2.p1.2)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- W\. Zeng, D\. Fu, K\. He, Y\. Wang, Y\. Xu, and W\. Xu \(2024\)Divtod: unleashing the power of llms for diversifying task\-oriented dialogue representations\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 800–813\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- Z\. Zhang, R\. Takanobu, Q\. Zhu, M\. Huang, and X\. Zhu \(2020\)Recent advances and challenges in task\-oriented dialog systems\.Science China Technological Sciences63\(10\),pp\. 2011–2027\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[Introduction](https://arxiv.org/html/2607.22635#Sx1.p4.1)\.
- Z\. Zheng, L\. Liao, Y\. Deng, E\. Lim, M\. Huang, and L\. Nie \(2024\)Thoughts to target: enhance planning for target\-driven conversation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 21108–21124\.Cited by:[Experimental Settings](https://arxiv.org/html/2607.22635#Sx6.SSx1.p1.1)\.
- Q\. Zhu, K\. Huang, Z\. Zhang, X\. Zhu, and M\. Huang \(2020\)CrossWOZ: a large\-scale chinese cross\-domain task\-oriented dialogue dataset\.Transactions of the Association for Computational Linguistics8,pp\. 281–295\.Cited by:[Task\- and Target\-oriented Dialogue Benchmarks](https://arxiv.org/html/2607.22635#Sx2.SSx1.p1.1)\.

## Appendix AA\. Main results of Qwen group

Table 4:Main results onCallBenchwith Qwen3\.5\-27B as the backbone and Qwen3\.7\-Max as the evaluator\. Higher is better\. Best results arebolded, and the second best ones areunderlined\.Table[4](https://arxiv.org/html/2607.22635#A1.T4)further examines the robustness of our evaluation protocol under a different backbone setting\. Although the absolute scores and method rankings change compared with the Doubao group results, the main observations remain consistent\. Existing task\- and target\-oriented dialogue methods still show unbalanced performance across evaluation dimensions, and no method consistently dominates all aspects of phone\-call assistance\. For example, AutoTOD achieves the best overall score, while ProCoT performs strongly in semantic understanding and response quality, ReAct and ChatSOP obtain high dialogue rhythm scores, and EnPL achieves the best safety score but remains weak in preset compliance and overall performance\. These results indicate that our evaluation does not merely reflect a backbone\-specific preference, but consistently captures fine\-grained strengths and weaknesses across different model settings\. In particular, the multi\-dimensional results show that strong performance on semantic understanding or response fluency does not necessarily imply reliable preset compliance, natural dialogue rhythm, or safe dual\-goal coordination\. This cross\-backbone consistency supports the robustness of our evaluation protocol and further demonstrates the necessity ofCallBenchfor diagnosing realistic phone\-call assistants\.

Moreover, the metric\-level scores across the two backbone settings show a strong correlation, with Spearman’sρ=0\.919\\rho=0\.919and Pearson’sr=0\.948r=0\.948, suggesting that the evaluation protocol produces stable diagnostic patterns despite changes in the underlying backbone\.

## Appendix BB\. Robustness between different evaluators

Table 5:Main results onCallBenchwith Qwen3\.5\-27B as the backbone and Doubao\-Seed\-2\.0\-lite as the evaluator\. Higher is better\. Best results arebolded, and the second best ones areunderlined\.To examine evaluator robustness, we compare the results obtained by Doubao\-Seed\-2\.0\-lite and Qwen3\.7\-Max under the same Qwen3\.5\-27B backbone\. Table[5](https://arxiv.org/html/2607.22635#A2.T5)shows the main results of Qwen group evaluated by Doubao\-Seed\-2\.0\-lite\. The Overall scores across eight methods show strong consistency, with Spearman’sρ=0\.786\\rho=0\.786and Pearson’sr=0\.914r=0\.914\. When all metric\-level scores are considered, the correlation remains high, with Spearman’sρ=0\.814\\rho=0\.814and Pearson’sr=0\.929r=0\.929\. These results suggest that our evaluation protocol produces stable diagnostic trends across different evaluator models, although absolute scores and some dimension\-specific rankings may vary across evaluators\.