From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scale

# From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scale
Source: [https://arxiv.org/html/2609.05758](https://arxiv.org/html/2609.05758)
Peng WangChuan ShiYufeng ZhangYing LyuAffiliation:Wanmeng RenRobert XueClaire Na ChengYashar MehdadAffiliation:Airbnb, Inc\.Affiliation:\{mia\.zhao, peng\.wang, chuan\.shi, wayne\.zhang, ying\.lyu,Affiliation:emma\.ren, robert\.xue, claire\.cheng, yashar\.mehdad\}@airbnb\.com

###### Abstract

Conversational assistants can blend retrieval, action selection, escalation, and wording in one model path, or separate those roles\. We report a production migration of a conversational assistant for customer support at a large accommodation marketplace \(millions of conversations per month, 11 languages, 10\-second P90\)\. Dynamic Response \(DR\) replaces a single Qwen3\-235B\-A22B blended responder with a bounded ReAct orchestrator over typed tools plus a smaller generator that writes from a backend\-validated context contract\. Because the migration also changed prompts, alignment, and serving, we attribute each effect to its cause and claim as architecture effects only those measured on identical replayed turns: typed entity selection moves the reservation selector to a precision\-first operating point \(precision 8\.3% to 89\.1% at recall 75\.2% to 67\.3%\), and typed action IDs with a membership check remove observed structured\-action hallucination \(2\.14% to 0\.0%\)\. A low\-ramp A/B test reproduces the replay escalation reductions: hard\-escalation responses fall 5\.60% to 3\.08% and soft\-escalation responses 9\.56% to 2\.49% while production handoff volume holds roughly steady, and self\-solve is directional \(\+5\.1 points, 95% CI\[−2,\+12\]\[\-2,\+12\]\)\. Serving optimizations cut orchestrator P90 from 3\.87s to 2\.24s on a GPU footprint reduced by roughly one\-third, and self\-hosting reduces estimated annual model\-serving cost by more than an order of magnitude \(Appendix[F](https://arxiv.org/html/2609.05758#A6)\)\.

## 1Introduction

Conversational assistants built on LLMs increasingly act as first\-line customer\-support agents\. In our setting, the assistant answers policy questions, selects structured cancellation and refund flows, asks clarifying questions, and routes to humans when automation should stop\. Each turn must fit a 10\-second P90 end\-to\-end budget, including translation across 11 languages\. The central systems problem is that support turns require both*decision making*and*wording*\. A single blended responder can retrieve context, select entities or actions, decide whether to clarify or escalate, and write the reply, but these responsibilities fail differently\. Wrong entity cards, unsupported action language, or premature human handoff become hard to localize when routing and prose are produced in one generation path\. Dynamic Response \(DR\) separates those roles\. A large orchestrator plans, calls typed tools, and owns routing, action, and escalation decisions; a smaller generator writes the user\-facing reply from a backend\-validated context contract\. We focus on systems design; scoring\-instrument construction, offline\-to\-online calibration, and launch\-decision methodology are companion evaluation work\. Because the migration changed architecture, prompts, alignment, and serving together, we attribute each reported effect to its cause and claim as architecture effects only those measured on identical replayed turns \(Table[2](https://arxiv.org/html/2609.05758#S6.T2)\); the remaining results are system\-migration outcomes\. We make four contributions\. Unlike a RAG workflow with an API wrapper, DR structures and logs intermediate decisions, constrains generation to orchestrator decisions and retrieved evidence, and uses role\-specific model sizes\. The transferable principle is to turn high\-consequence decisions into explicit, backend\-validated typed tools and keep wording downstream and constrained\. Concretely: \(1\)production decomposition, a typed context contract that separates orchestration from final prose; \(2\) atool\-mediated runtimewith bounded ReAct planning, typed schemas, boundary guardrails, and conservative fallback; \(3\) aserving recipe, a self\-hosted open\-weight MoE on vLLM with tensor\-parallel serving, FlashInfer, EAGLE\-3 speculative decoding, and a latency\-fit layout; and \(4\)operational lessonson context handoff, tool false positives, soft escalation, and open\-weight cost and control trade\-offs\.

## 2Background and Problem Setting

### 2\.1Task

When a guest or host starts support, the assistant must understand the request, retrieve user state \(reservation, listing, payment, account\), retrieve policy and help\-center evidence, and choose among structured action cards, clarification, informational answers, and human routing\. Many turns require cross\-source reasoning: a cancellation may depend on reservation dates, listing policy, payment state, and regional rules\. The system must also support translation, avoid unsupported actions, keep references traceable, and preserve human handoff; these constraints make architecture as important as base\-model choice\.

### 2\.2Monolithic Blending Baseline

The predecessor used the same Qwen3\-235B\-A22B\-class model as a monolithic responder\. Backend services preassembled candidate help or policy snippets, user\-state variables, and renderable structured\-action candidates\. A single blended model path then chose the response strategy, decided whether to present an action or escalate, and wrote the user\-facing reply; backend rules validated renderability, blocked unsupported outputs, and provided conservative fallback\. This design was simple and avoided a second model call, but it entangled*what to do*with*what to say*: wrong actions, unsupported handoff language, or wrong entity references all appeared in one generation trace, hard to assign to retrieval, routing, or wording\. Adding tools also enlarged the blended prompt, since every capability had to be exposed to the path that produced final prose\. DR keeps the same large\-model reasoning capacity for planning while making routing, tool use, and final wording separately observable and controllable\. A stronger monolithic system with constrained decoding or structured JSON output would enforce syntactic validity, but routing and prose would still share one generation trace that cannot be individually localized, gated, or rolled back, and role\-specific model sizing would remain unavailable\.

### 2\.3Related Work

DR combines ideas from RAG, tool\-using agents, and LLM serving\. RAG\([Lewis et al\., 2020](https://arxiv.org/html/2609.05758#bib.bib5)\)grounds generation in retrieved evidence, but single\-pass RAG\-style support systems can still couple evidence selection, action choice, escalation, and final wording\. Building on chain\-of\-thought prompting\([Wei et al\., 2022](https://arxiv.org/html/2609.05758#bib.bib4)\), ReAct\([Yao et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib1)\)interleaves reasoning and acting, which maps naturally to support resolution as tool calls plus observations, and tool\-augmented LLMs\([Schick et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib2);[Qin et al\., 2024](https://arxiv.org/html/2609.05758#bib.bib3)\)study how models select and use external tools\. Serving efficiency also matters because orchestration must fit a production latency budget: mixture\-of\-experts models\([Shazeer et al\., 2017](https://arxiv.org/html/2609.05758#bib.bib18)\), grouped\-query attention\([Ainslie et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib17)\), vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib12)\), and speculative decoding\([Leviathan et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib13);[Chen et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib31)\)make large routing models practical to serve\. Agent frameworks such as AutoGen\([Wu et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib25)\), LangGraph\([LangChain, 2024](https://arxiv.org/html/2609.05758#bib.bib26)\), Smolagents\([Hugging Face, 2025](https://arxiv.org/html/2609.05758#bib.bib27)\), and CrewAI\([CrewAI, Inc\., 2024](https://arxiv.org/html/2609.05758#bib.bib28)\)provide useful orchestration abstractions, but our deployment required backend\-enforced control over routing, guardrail placement, logging, context handoff, and fallback behavior\. Closest to our setting are recent LLM customer\-support agents: benchmarks for resolving real e\-commerce issues\([Wang et al\., 2025](https://arxiv.org/html/2609.05758#bib.bib21)\)and for asking clarifying questions in conversational product search\([Ye et al\., 2025a](https://arxiv.org/html/2609.05758#bib.bib22)\), and deployed systems built around continuous\-improvement data flywheels\([Zhao et al\., 2025](https://arxiv.org/html/2609.05758#bib.bib23)\)and incremental summarization with agent feedback\([Wu et al\., 2025](https://arxiv.org/html/2609.05758#bib.bib24)\)\. These target task performance, data loops, and assistance features, and typically evaluate a single end\-to\-end policy\. DR instead turns high\-consequence decisions into typed, backend\-validated tool calls and reports the migration with explicit effect attribution on identical replayed turns; the evaluation methodology behind our release gates is companion work\.

## 3Dynamic Response Architecture

### 3\.1Design Principles and Runtime

DR follows three design rules\.Separate routing from wording:the orchestrator chooses tools, actions, and escalation; the generator writes the reply without overriding those decisions\.Retrieve on demand:the orchestrator requests only the context the turn needs and passes a curated state to the generator\.Make control flow observable:each orchestration step is logged as a structured event, so failures can be localized and guardrails attached at the right boundary\. Figure[1](https://arxiv.org/html/2609.05758#S3.F1)shows the runtime\. After input safety checks and translation, the orchestrator enters a bounded ReAct loop capped at three steps\. It may call retrieval tools, selector tools, the escalation tool, or response generation; tool observations are appended to orchestration state\. Once response generation is selected, the generator receives the typed context contract and writes the final reply\. Post\-generation checks validate formatting, references, unsupported actions, and safety\-sensitive output before delivery\.

User message \(\+\+translation\)Pre\-guardrail \(input safety\)Orchestrator\(Qwen3\-235B\-A22B\)bounded ReAct loop \(≤3\\leq 3steps\)tools: USV / KB / SA retrieval, selectors, escalationGenerator\(Qwen3\-Next\-80B\-A3B\)writes user\-facing replyPost\-guardrail \(output safety, format, risk\)Response / Structured actionobserveFigure 1:DR runtime\. The large orchestrator plans and calls tools in a bounded ReAct loop; the smaller generator writes from the typed contract\. Guardrails wrap input, tool use, and output\.
### 3\.2Model Roles

The orchestrator is a fine\-tuned open\-weight MoE model, Qwen3\-235B\-A22B\([Qwen Team, 2025](https://arxiv.org/html/2609.05758#bib.bib19)\), with 235B total and 22B active parameters\. It is the same large\-model class used by the monolithic baseline, but DR restricts it to structured tool\-call decisions, terminal actions, and fallback conditions rather than user\-facing prose\. The generator is Qwen3\-Next\-80B\-A3B, selected because final responses are longer and must fit the latency budget\. It writes from the contract in Section[4](https://arxiv.org/html/2609.05758#S4)and cannot select a new action, call retrieval, or initiate escalation\.

### 3\.3Tool Registry

The orchestrator uses a typed registry \(full schema in Appendix[C](https://arxiv.org/html/2609.05758#A3), Table[13](https://arxiv.org/html/2609.05758#A3.T13)\)\. Each entry defines schema, eligibility, logging fields, and guardrail hooks\. User\-facing tools change the customer experience; internal tools retrieve context or constrain the action space\. Tool outputs are small typed objects when possible, letting the backend validate IDs, reason codes, and source membership before generation\.

### 3\.4Two\-Phase Deployment

DR shipped in two stages, both retaining fallback to the previous path to separate migration risks\. Stage 1 used one orchestrator call over pre\-fetched context, preserving much of the backend while validating structured routing; unlike Monolithic, it invoked a separate generator instead of writing final prose\. Stage 2 introduced the full tool\-mediated loop, letting the orchestrator decide which retrieval tools to call, whether entity selection is needed, whether a structured action applies, and when generation should run\. The architecture\-local wins reported below come from Stage 2’s typed decisions; Stage 1 mainly validated structured routing and the handoff, and a per\-stage ablation is out of scope\.

## 4Context Contract, Guardrails, and Fallback

### 4\.1Context Contract

The handoff between orchestrator and generator is a product interface, not a prompt detail\. The forward contract passes task state, eligible tools, retrieved sources, user\-state variables, policy constraints, locale, and prior observations\. The return contract contains response text, source references, an optional orchestrator\-selected action\-card reference, and rendering fields; full fields are in Appendix[A](https://arxiv.org/html/2609.05758#A1)\. Two constraints are central\. First, the generator may reference only tools, sources, and user\-state keys present in the contract, and post\-generation checks enforce this closure\. Second, the handoff includes the orchestration state needed for coherence, not only cited snippets: cited\-only handoff was shorter but lost context such as why an action was unavailable\.

### 4\.2Guardrails and Fallback

DR places guardrails before orchestration, inside the ReAct loop, and after generation\. Pre\-checks classify input safety and eligibility; mid\-stream checks validate tool schema, eligibility, and consistency with prior state; post\-checks validate references, action\-card IDs, format, and risk\-sensitive phrasing\. If the orchestrator exceeds the iteration limit, emits an invalid tool call, or receives inconsistent observations, DR falls back to conservative generation over available context\. Fallback cannot invent actions or bypass safety routing; every fallback logs its trigger, separating infrastructure failures \(timeout, malformed tool results\) from policy failures \(no eligible action, ambiguous state\), which require different owners\.

## 5Model and Serving Choices

### 5\.1Open\-Weight Serving

The orchestrator and generator are self\-hosted on vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib12)\)\. Open weights let us align models to policy with full\-parameter preference optimization \(Appendix[E](https://arxiv.org/html/2609.05758#A5)\), keep pre\-production and production settings identical, tune serving to our traffic shape, and run large shadow tests; reward calibration to production self\-solve is out of scope\. At our volume and decoding settings, self\-hosting reduced estimated annual model\-serving cost by more than an order of magnitude relative to a proprietary\-API baseline; these are internal order\-of\-magnitude estimates whose basis, scope, and exclusions are detailed in Appendix[F](https://arxiv.org/html/2609.05758#A6)\.

### 5\.2Inference\-Engine Tuning

The orchestrator runs deterministic decoding on H100 GPUs, where latency rather than throughput was the binding constraint\. Switching from tensor\-plus\-expert parallelism to tensor\-parallel\-only serving\([Shoeybi et al\., 2019](https://arxiv.org/html/2609.05758#bib.bib16)\), enabling asynchronous scheduling, and using FlashInfer kernels\([Ye et al\., 2025b](https://arxiv.org/html/2609.05758#bib.bib15)\)reduced P90 from 3\.87s to 2\.24s while shrinking the GPU footprint by roughly one\-third\. Orchestrator\-only serving latency across operating points is reported in Appendix[D](https://arxiv.org/html/2609.05758#A4)\(Table[20](https://arxiv.org/html/2609.05758#A4.T20)\), and the end\-to\-end latency distribution by stage is reported in Table[1](https://arxiv.org/html/2609.05758#S5.T1)\. EAGLE\-3 speculative decoding\([Li et al\., 2025](https://arxiv.org/html/2609.05758#bib.bib14)\)is being validated as a serving\-only optimization toward a further\-reduced operating point, and orchestrator and generator capacity are co\-located behind the stable contract; details are in Appendix[D](https://arxiv.org/html/2609.05758#A4)\.

Table 1:End\-to\-end latency distribution by stage \(seconds\), DR arm on live traffic over the stabilized tail 2026\-03\-02 to 2026\-03\-05 \(n=4,155n=4\{,\}155turns\)\. Percentiles are computed over per\-turn stage sums, not as sums of per\-stage percentiles, so the stage rows need not add to the end\-to\-end row; the unattributed residual is at most 0\.39s at P99\. The orchestration stage spans the full bounded ReAct loop, including tool executions and up to three model calls, so it exceeds the single\-call orchestrator serving latency in Table[20](https://arxiv.org/html/2609.05758#A4.T20)\. End\-to\-end P90 \(6\.539s\) is within the 10s release gate, and orchestration is the dominant stage \(about 58% of the median turn\)\.

## 6Controlled Comparison and Launch Process

We evaluate DR as an architecture migration, with evidence from zero\-exposure shadow replay, a low\-ramp online A/B experiment, and targeted human and risk review, all under fixed production release checks\.

#### What we attribute to architecture\.

The migration changed the architecture, generator placement, prompt and tool schemas, alignment, and serving stack\. The high\-level decision\-model class remained Qwen3\-235B\-A22B\-class in both systems: Monolithic used it as a blended final responder, while DR uses it as the orchestrator\. We therefore separate*architecture\-local*effects, which correspond to typed selector, action, and escalation decisions, from*model and alignment*and serving effects, which would also benefit a single\-call system\. Table[2](https://arxiv.org/html/2609.05758#S6.T2)states this mapping\. We do not claim a pure architecture\-only ablation; the end\-to\-end comparison is a production migration, with component replay localizing the main behavior changes\.

Table 2:Attribution of the main gains\. Architecture\-local effects are measured on identical replayed turns; model, alignment, and serving effects would also benefit a single\-call system\.*Core correctness \(alignment effect\)*is the self\-hosting\-plus\-alignment gain on one instrument \(0\.825 pre\-alignment baseline to 0\.93 aligned model\); it is a different measurement from*shadow\-window logic correctness*in Table[3](https://arxiv.org/html/2609.05758#S6.T3), where both arms already use the aligned self\-hosted models\.
#### Evaluation scope\.

Table[8](https://arxiv.org/html/2609.05758#A2.T8)\(Appendix[B](https://arxiv.org/html/2609.05758#A2)\) maps evidence to claims: replay localizes component failures on identical turns, human and risk review gates broadening, and A/B measures real\-user outcomes under identical eligibility\. We claim only architecture\-local behavior, release\-gated response quality, low\-ramp online outcomes, and serving cost and latency; evaluator construction, judge certification, and offline\-to\-online calibration are companion work\. Fixed launch gates were: bad\-response rate≤1%\\leq 1\\%, correctness≥90%\\geq 90\\%, groundedness≥75%\\geq 75\\%, structured\-action FPP≤7%\\leq 7\\%, structured\-action recall≥60%\\geq 60\\%, and end\-to\-end P90≤10\\leq 10s\. Correctness and groundedness in Table[3](https://arxiv.org/html/2609.05758#S6.T3)come from fixed production release\-check judges\([Zheng et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib20)\)on a calibrated≈\\approx1k\-turn set\. Each judge is certified against expert\-adjudicated human labels before use \(the problem\-solution judge reaches ICC 0\.88 and AUC 0\.84 against a 0\.75 human baseline\)\. These scores interpret the architecture comparison and are not treated as powered significance tests\.

### 6\.1Shadow Replay

We replayed production requests through both systems while serving only Monolithic responses\. Fixed user behavior plus logged inputs, tool decisions, and outputs lets us localize failures to orchestration, tool selection, or final wording\.

#### Response quality and risk\.

Table[3](https://arxiv.org/html/2609.05758#S6.T3)shows that DR remains within production release thresholds while shifting the error profile\. Orchestrator\-decided metrics \(selector, escalation, and reference checks\) and generator\-realized metrics \(correctness, groundedness, style\) are reported in separate columns, so each win or regression is attributed to the role that produced it\. The main architecture wins are zero observed structured\-action hallucinations and lower user\-role confusion; the main regressions are generator\-side USV attribution, style and tone, and repetition\. USV errors mix strict citation mismatch with genuine gap filling, so we are moving from per\-sentence to per\-key attribution to separate those cases \(Appendix[B](https://arxiv.org/html/2609.05758#A2)\)\. Style and repetition regressions are treated as generator headroom rather than evidence against the split, because upstream decision quality improves \(Appendices[C\.7](https://arxiv.org/html/2609.05758#A3.SS7)and[C\.6](https://arxiv.org/html/2609.05758#A3.SS6)\)\. The structured\-action hallucination claim is scoped to structured\-action references only, and the 0\.0% is the delivered rate: the generator can reference only actions present in the contract, the backend independently blocks any spurious action ID before delivery, and blocked attempts are logged separately\.

Table 3:Turn\-level quality on a fixed shadow window \(n≈1,000n\\approx 1\{,\}000turns\)\.DR Orch\.reports metrics decided by the orchestrator \(reference checks, selector, escalation\);DR Gen\.reports metrics realized in the generator’s final wording; “n/a” marks a non\-applicable role\. Arrows give the preferred direction; bold marks the better arm\.†\\dagger*Shadow\-window logic correctness*compares final wording on identical replayed turns and is a different instrument from*core correctness \(alignment effect\)*in Table[2](https://arxiv.org/html/2609.05758#S6.T2)\.
#### Entity selection\.

The selector shifts to a precision\-first operating point \(full precision and recall in Table[4](https://arxiv.org/html/2609.05758#S6.T4)\)\. Monolithic favored recall and surfaced many wrong entity cards; DR makes entity choice an explicit typed orchestrator decision\. The design goal of the precision\-first point is that a selector miss should not surface a wrong entity card, which would be misinformation\. On the shadow slice where the Monolithic arm surfaced an entity but the DR selector returned null \(n=6,945n=6\{,\}945turns, drawn from the full replay traffic\), that goal largely holds\. Here*wrong entity*counts a delivered response that references an incorrect specific entity — a stricter event than the spurious\-card false positives that dominate Monolithic’s precision in Table[4](https://arxiv.org/html/2609.05758#S6.T4): DR references a wrong entity in only 1\.01% of these turns \(Monolithic 0\.20% on the same turns\), so a miss rarely produces entity\-level misinformation\. The cost is elsewhere in resolution quality\. On this slice DR gives an entity\-free answer judged correct in 47\.75% of turns, asks a clarifying question in 1\.43%, and escalates in 3\.87%; the remaining turns give entity\-free answers not judged fully correct\. This slice is a known weakness: picker correctness here \(about 48%\) is well below the selector’s overall operating point, and raising miss\-slice resolution is ongoing work\. Precision is also enforced structurally: the backend rejects any rendered card whose ID the orchestrator did not select\.

Table 4:Selector precision and recall \(%, approximately 1k shadow samples\)\.
#### Structured\-action selection\.

DR exposes the precision and recall trade\-off in the orchestrator prompt \(Appendix[B](https://arxiv.org/html/2609.05758#A2), Table[10](https://arxiv.org/html/2609.05758#A2.T10)\)\. We distinguish the F1 frontier from the shipped point: one configuration reaches Monolithic\-level F1 but exceeds the false\-positive gate, while the deployed configuration sacrifices a small amount of F1 to satisfy the false\-positive and recall gates\. Thus the shipped claim is gate compliance; the parity claim refers only to the achievable frontier\.

#### Escalation behavior\.

Treating escalation as a tool gives a clearer control point than allowing handoff phrases to appear in final prose\. DR reduces avoidable soft escalation, while replay shows that nearly all Monolithic hard escalations remain hard\-escalated, so the reduction is not simply suppressed routing\. Per\-risk\-category risk prevalence on the same replayed traffic is reported in Table[5](https://arxiv.org/html/2609.05758#S6.T5), so the escalation reduction can be checked against risk exposure rather than taken on faith\.

### 6\.2Human Review and Risk Monitoring

Pre\-broadening review covered 1,173 DR conversations and 2,335 dual\-labeled judgments from 28 annotators\. One development snapshot passed correctness and groundedness checks but failed an internal risk gate, blocking broadening\. We include this as release\-process evidence, not as a launch\-ready result\. Error modes appear in Appendix[B](https://arxiv.org/html/2609.05758#A2)\. We also compared Monolithic and DR on false promise, medical\-records, tax, ID\-verification, and refund risks\. Review found no statistically reliable increase, though this is monitoring evidence rather than proof of equivalence \(Table[5](https://arxiv.org/html/2609.05758#S6.T5)\)\. In internal red\-team exercises\([Perez et al\., 2022](https://arxiv.org/html/2609.05758#bib.bib32);[Ganguli et al\., 2022](https://arxiv.org/html/2609.05758#bib.bib33)\), DR introduced no qualitatively new attack surface: of the issues identified in manual red\-teaming of DR, all that were retested \(24 of 24, across 123 attempts by six testers\) reproduced at least once on the Monolithic production system, and in automated adversarial template scans DR’s pass rate matched or exceeded Monolithic’s on all three primary templates after iterative remediation\. Findings follow standard remediation\.

Table 5:Per\-risk\-category risk prevalence on identical shadow traffic \(Monolithicn=2,795n\{=\}2\{,\}795turns; DRn=2,318n\{=\}2\{,\}318responses: prevalence is computed among responses because a subset of replayed turns routed to typed non\-response actions such as entity pickers, which would otherwise bias the comparison downward\)\. Flags come from five risk classifiers calibrated against subject\-matter\-expert labels, built as a three\-stage hierarchical pipeline \(category filter, sub\-skill filter, focused per\-pattern scoring with policy\-encoded prompts\)\. No category shows a statistically significant difference between the systems \(all 95% CIs overlap\)\. ID\-verification and other safety\-sensitive contact reasons are routed deterministically to humans, independent of the model’s decision \(Section[6\.2](https://arxiv.org/html/2609.05758#S6.SS2)\)\.
### 6\.3Online A/B \(Low Ramp\)

We then ran a low\-ramp online A/B test under identical eligibility, randomized at the user level \(sessions appear later only as an analysis unit, in the session\-level results of Table[9](https://arxiv.org/html/2609.05758#A2.T9), not as the randomization unit\), and report the 5% ramp stage at its final day \(2026\-03\-05; 1,328 control and 1,346 treatment users enrolled\)\. Escalation rates are turn\-level response rates over all turns from enrolled users, with the same definitions in both arms: a hard\-escalation response routes the user to a human agent after a user\-initiated request \(excluding guardrail\-triggered routing\), and a soft\-escalation response offers to connect the user to a human\. The escalation reductions observed in replay reproduce online: hard escalation falls from 5\.60% to 3\.08% \(95% CI on the change\[−4\.1,−0\.9\]\[\-4\.1,\-0\.9\]points\) and soft escalation from 9\.56% to 2\.49% \(95% CI\[−8\.1,−5\.8\]\[\-8\.1,\-5\.8\]points\)\. The production human\-handoff rate on the same cohort held roughly steady \(8\.56% to 8\.20%\), so the reduction reflects fewer escalation\-framed responses, not fewer human handoffs — itself evidence against under\-escalation\. Self\-solve moves directionally \(\+5\.1 points, 95% CI\[−2,\+12\]\[\-2,\+12\]\) but is not significant at this ramp; contact\-rate effects are likewise directional\. At the fuller ramp the session\-level self\-solve gain is smaller but well powered; we treat the low\-ramp estimate as directional and escalation behavior, not self\-solve, as the supported online result\. Per\-language and per\-tier splits are in Appendix[B](https://arxiv.org/html/2609.05758#A2), Table[9](https://arxiv.org/html/2609.05758#A2.T9)\(different windows and denominators; the wider window reproduces the reductions, e\.g\., English soft escalation 9\.43% to 2\.03%\)\.

## 7Production Rollout and Operations

DR rolled out in stages: zero\-exposure replay while the previous system served responses; low\-traffic canary with immediate fallback; expansion by region and language; and retained rollback to the previous system during migration\. Because the orchestrator, generator, tool calls, and post\-generation checks are logged separately, operators can localize failures to a boundary and roll back only the affected component\. Operations follow a weekly release train: tool or policy changes update the registry and prompt contracts first, and model updates stage behind the same interface, so generator swaps do not change the orchestrator and prompt updates do not change the backend renderer\. Training and monitoring data is de\-identified and excludes opt\-out users\.

## 8Lessons Learned

#### The handoff is the architecture\.

Decomposition only works when the boundary between orchestrator and generator is explicit\. Early prototypes treated the handoff as a prompt string, making failures hard to localize; the typed contract made the boundary inspectable and gave guardrails concrete objects to validate\.

#### High\-impact decisions should be explicit tools\.

Escalation, entity selection, and action\-card rendering are too consequential to leave to final prose: a single\-call model can say “an agent can help” without making a routing decision, or surface a wrong entity card when clarification would be safer\. DR makes these orchestrator tool decisions; the generator may explain the selected path but cannot create a new escalation or action through wording alone\.

#### Role separation changes the optimization target\.

Because the generator no longer performs global planning, it can be smaller, provided the contract carries the evidence needed for grounded wording and prevents action changes\. Open weights made the split practical: self\-hosting let us tune parallelism, kernels, speculative decoding, and placement while keeping the previous system as a large\-scale fallback during migration\.

## 9Conclusion

We presented Dynamic Response, a two\-model agentic architecture for production conversational assistants: a large orchestrator plans and calls tools, and a smaller generator writes the final reply\. Self\-hosted open\-weight MoE models and serving optimizations fit a 10\-second P90 multilingual budget with observable, modular control flow\. The added complexity pays off when the handoff contract, guardrails, and serving stack are first\-class parts of the architecture\.

## Limitations

DR was built for one support domain with mature tools, stable policy sources, and a human\-support fallback\. Decomposition is most likely to pay off when high\-consequence typed decisions must be logged, validated, and rolled back, when the latency budget can absorb a second model call, and when tools and policy are mature; it helps less when tools are sparse, when policy changes faster than the contract and registry can be maintained, or when wrong decisions carry little risk\. It also has a real operational price: a second serving stack and release train, and version coordination across the tool registry, orchestrator prompts, and backend validators\. The second model call adds operational complexity even though serving optimizations keep latency within budget\. Cost and latency figures reflect our hardware, traffic, and decoding settings\.

#### Reproducibility\.

The paper releases the reusable interfaces \(the context\-contract schemas and invariants, the tool\-registry schema, and the release\-evidence framework\); the shadow sets, judge prompts, and labeled data are proprietary and cannot be released\. This paper reports release\-process evidence needed to interpret the architecture, but omits the companion methodology used to construct evaluator roles, certify LLM judges, calibrate offline scores to online outcomes, or select prompt and model variants\. The online A/B evidence is low\-ramp migration evidence; self\-solve and contact effects are directional at the low ramp, while the fuller\-ramp session\-level self\-solve gain is smaller but powered \(Table[9](https://arxiv.org/html/2609.05758#A2.T9)\)\. Because DR was evaluated as a production migration, the comparison should not be read as a pure causal ablation of architecture alone\.

## Ethical Considerations

The system processes customer\-support conversations\. Data used for training, debugging, and monitoring is de\-identified, opt\-out users are excluded before training use, and log access is restricted; human annotators reviewed only de\-identified conversations\. Pre\-, mid\-, and post\-generation guardrails reduce harmful, discriminatory, privacy\-violating, or policy\-violating outputs\. Human escalation remains available, and safety\-sensitive topics can be routed automatically\. We also monitor under\-escalation because reducing escalations is only desirable when automation remains appropriate; escalation is a typed tool with reason codes, so under\-escalation is monitorable per risk category and per language in production\. Table[5](https://arxiv.org/html/2609.05758#S6.T5)reports per\-category risk prevalence, Table[9](https://arxiv.org/html/2609.05758#A2.T9)per\-language escalation, and Appendix[B](https://arxiv.org/html/2609.05758#A2)the layered controls that bound under\-escalation\. Together with the steady production handoff rate \(Section[6\.3](https://arxiv.org/html/2609.05758#S6.SS3)\), these breakdowns let a reader check that the escalation reduction does not come from under\-escalation\. Cost and infrastructure figures are illustrative estimates, not audited financial statements or actuals\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stańczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2306.13649)Cited by:[§C\.6](https://arxiv.org/html/2609.05758#A3.SS6.p1.1)\.
- Ainslieet al\.\(2023\)J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. SanghaiGQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2305.13245)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Chenet al\.\(2023\)C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. JumperAccelerating large language model decoding with speculative sampling\.External Links:2302\.01318,[Link](https://arxiv.org/abs/2302.01318)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- CrewAI, Inc\. \(2024\)CrewAI, Inc\.CrewAI: framework for orchestrating role\-playing, autonomous ai agents\.Note:[https://github\.com/crewAIInc/crewAI](https://github.com/crewAIInc/crewAI)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Ganguliet al\.\(2022\)D\. Ganguli, L\. Lovitt, J\. Kernion, A\. Askell, Y\. Bai, S\. Kadavath, B\. Mann, E\. Perez,et al\.Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned\.External Links:2209\.07858,[Link](https://arxiv.org/abs/2209.07858)Cited by:[§6\.2](https://arxiv.org/html/2609.05758#S6.SS2.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.Note:NeurIPS Deep Learning Workshop, 2014External Links:1503\.02531Cited by:[§C\.6](https://arxiv.org/html/2609.05758#A3.SS6.p1.1)\.
- Honget al\.\(2024\)J\. Hong, N\. Lee, and J\. ThorneORPO: monolithic preference optimization without reference model\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 11170–11189\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.626),[Link](https://aclanthology.org/2024.emnlp-main.626/)Cited by:[§E\.1](https://arxiv.org/html/2609.05758#A5.SS1.p1.1)\.
- Hugging Face \(2025\)Hugging Facesmolagents: a barebones library for agents\.Note:[https://github\.com/huggingface/smolagents](https://github.com/huggingface/smolagents)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles \(SOSP\),External Links:[Link](https://arxiv.org/abs/2309.06180)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1),[§5\.1](https://arxiv.org/html/2609.05758#S5.SS1.p1.1)\.
- LangChain \(2024\)LangChainLangGraph\.Note:[https://github\.com/langchain\-ai/langgraph](https://github.com/langchain-ai/langgraph)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Leviathanet al\.\(2023\)Y\. Leviathan, M\. Kalman, and Y\. MatiasFast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning \(ICML\),pp\. 19274–19286\.External Links:[Link](https://arxiv.org/abs/2211.17192)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.11401)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Liet al\.\(2025\)Y\. Li, F\. Wei, C\. Zhang, and H\. ZhangEAGLE\-3: scaling up inference acceleration of large language models via training\-time test\.External Links:2503\.01840,[Link](https://arxiv.org/abs/2503.01840)Cited by:[§5\.2](https://arxiv.org/html/2609.05758#S5.SS2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2203.02155)Cited by:[§E\.1](https://arxiv.org/html/2609.05758#A5.SS1.p1.1)\.
- Perezet al\.\(2022\)E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. IrvingRed teaming language models with language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2202.03286)Cited by:[§6\.2](https://arxiv.org/html/2609.05758#S6.SS2.p1.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin,et al\.ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2307.16789)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.2](https://arxiv.org/html/2609.05758#S3.SS2.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2305.18290)Cited by:[§E\.1](https://arxiv.org/html/2609.05758#A5.SS1.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2302.04761)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[§E\.1](https://arxiv.org/html/2609.05758#A5.SS1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§E\.1](https://arxiv.org/html/2609.05758#A5.SS1.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1701.06538)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Shoeybiet al\.\(2019\)M\. Shoeybi, M\. Patwary, R\. Puri, P\. LeGresley, J\. Casper, and B\. CatanzaroMegatron\-LM: training multi\-billion parameter language models using model parallelism\.External Links:1909\.08053,[Link](https://arxiv.org/abs/1909.08053)Cited by:[§5\.2](https://arxiv.org/html/2609.05758#S5.SS2.p1.1)\.
- Wanget al\.\(2025\)H\. Wang, X\. Peng, H\. Cheng, Y\. Huang, M\. Gong, C\. Yang, Y\. Liu, and J\. LinECom\-Bench: can LLM agent resolve real\-world E\-commerce customer support issues?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,Suzhou, China,pp\. 276–284\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.19),[Link](https://aclanthology.org/2025.emnlp-industry.19/)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2201.11903)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Wuet al\.\(2023\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.External Links:2308\.08155,[Link](https://arxiv.org/abs/2308.08155)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Wuet al\.\(2025\)Y\. Wu, C\. Zhao, Y\. Cao, X\. Xu, Y\. Mehdad, M\. Ji, and C\. N\. ChengIncremental summarization for customer support via progressive note\-taking and agent feedback\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,Suzhou, China,pp\. 2000–2015\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.140),[Link](https://aclanthology.org/2025.emnlp-industry.140/)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Yeet al\.\(2025a\)J\. Ye, Y\. Jiang, X\. Wang, Y\. Li, Y\. Li, P\. Xie, and F\. HuangProductAgent: benchmarking conversational product search agent with asking clarification questions\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,Suzhou, China,pp\. 383–398\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.25),[Link](https://aclanthology.org/2025.emnlp-industry.25/)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Yeet al\.\(2025b\)Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. CezeFlashInfer: efficient and customizable attention engine for LLM inference serving\.InProceedings of the 8th Conference on Machine Learning and Systems \(MLSys\),External Links:[Link](https://arxiv.org/abs/2501.01005)Cited by:[§5\.2](https://arxiv.org/html/2609.05758#S5.SS2.p1.1)\.
- Zhaoet al\.\(2025\)C\. Zhao, T\. Zhang, H\. Su, Y\. Zhang, S\. Su, M\. Xu, Y\. Liu, W\. Han, J\. Werner, C\. N\. Cheng, and Y\. MehdadAgent\-in\-the\-loop: a data flywheel for continuous improvement in LLM\-based customer support\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,Suzhou, China,pp\. 1919–1930\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.135),[Link](https://aclanthology.org/2025.emnlp-industry.135/)Cited by:[§2\.3](https://arxiv.org/html/2609.05758#S2.SS3.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.External Links:[Link](https://arxiv.org/abs/2507.18071)Cited by:[§E\.1](https://arxiv.org/html/2609.05758#A5.SS1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§6](https://arxiv.org/html/2609.05758#S6.SS0.SSS0.Px2.p1.1)\.

## Appendix AContext\-Contract Specification

The forward contract restricts what the generator can see and reference; the return contract restricts what it can send back to the backend\. The tables use the full page width because field descriptions are part of the interface specification\. The contract enforces four invariants\. First, reference closure: every article ID, structured action, and user\-state key in the return object must be a member of the forward object\. Second, role separation: the generator may explain an action but may not choose a different action path\. Third, replayability: the forward object plus tool observations is sufficient to reconstruct why the user\-facing response was allowed\. Fourth, localization safety: locale is carried as data, while translation and rendering stay in backend services rather than being delegated to free\-form model text\. These invariants are checked by backend code, not by trusting the model to follow natural\-language instructions alone\.

Table 6:Forward context contract: fields the orchestrator passes to the generator\.Table 7:Return context contract: fields the generator returns to the backend\.
## Appendix BAdditional Error and Component Analysis

This appendix expands process evidence from Section[6](https://arxiv.org/html/2609.05758#S6); example conversations are synthetic\. Table[8](https://arxiv.org/html/2609.05758#A2.T8)summarizes which instrument supports which claim; Table[2](https://arxiv.org/html/2609.05758#S6.T2)\(main text\) attributes each main gain to its cause\.

Table 8:Release evidence for the architecture comparison\. Instrument construction, judge certification, and offline\-to\-online calibration are not contributions\.Table 9:Per\-language\-tier hard and soft escalation: turn\-level response classifications on evaluated turns over 2026\-03\-05 to 2026\-03\-12, the widest window in which both arms served non\-shadow traffic\.Δ\\Deltais treatment minus control in points\. The Overall rows additionally include a small number of turns not attributed to a listed language tier \(17 control, 40 treatment\), so they slightly exceed the sum of the English and Tier 1 rows\. English and Tier 1 confidence intervals overlap on both metrics, indicating no language\-tier interaction\. Within Tier 1, both single\-language soft\-escalation reductions are significant, while the single\-language hard\-escalation deltas are directional at these sample sizes \(their CIs cross zero\); ES turn counts are derived as Tier 1 minus FR, since Portuguese carried no assistant traffic in this window, and the ES/FR confidence intervals are computed with the same unpooled two\-proportion method as the tier rows\. Tier 2 \(DE, IT, RU, JA, KO, TH, ZH\) was routed to the legacy bot during the experiment and carried no assistant traffic in either arm; the 11 languages in the abstract describe the deployed platform, not this experiment’s eligible population\. Session\-level results over the full experiment window \(2026\-02\-24 to 2026\-04\-14;n≈139n\\approx 139K engaged sessions per arm, a different denominator\) show the same consistency: self\-solve\+1\.77\+1\.77points for English and\+1\.80\+1\.80for Tier 1, combined escalation−1\.68\-1\.68and−1\.77\-1\.77\(allp<10−8p<10^\{\-8\}\), and end\-to\-end P90 latency improves from 9\.43s to 7\.34s for English and 10\.08s to 8\.25s for Tier 1, though Tier 1 remains slower than English in both arms\. Tier\-level differences are not driven by language routing: the language\-identification guardrail exceeds 99% weighted accuracy on human\-labeled production data\.#### Per\-category under\-escalation controls\.

Under\-escalation is bounded by layered controls rather than by orchestrator discretion alone\. First, safety\-sensitive content is detected by dedicated per\-category guardrail models \(safety, jailbreak, inappropriate content\) and routed deterministically to humans, independent of the orchestrator decision\. The safety detector is evaluated offline against human labels and operated at a deliberately recall\-first point that trades false positives for coverage of policy\-relevant safety content, so detector misses — the per\-category under\-escalation channel — are the minority case by design\. Jailbreak and inappropriate\-content guardrails run on the same independent routing path and are threshold\-calibrated on production traffic against judge\-labeled samples\. The detectors’ operating points are security\-sensitive and not disclosed\. Second, the user can request human escalation at any turn, and user\-initiated handoff volume held roughly steady across the migration \(Section[6\.3](https://arxiv.org/html/2609.05758#S6.SS3)\), so the response\-level escalation reduction did not suppress user\-initiated handoffs\. Third, a post\-interaction outreach program reviews flagged conversations and follows up after the fact, so a missed live escalation is not a terminal failure\. Production prevalence of these categories is low \(well under 0\.1% of turns for all but general safety topics\), so per\-category under\-escalation rates carry wide confidence intervals and are monitored operationally rather than estimated from the shadow window\.

#### Structured\-action selection\.

Table[10](https://arxiv.org/html/2609.05758#A2.T10)reports precision, recall, F1, and false\-positive proportion across routing\-prompt iterations\. DR reaches F1 parity while making the operating point explicit\.

Table 10:Structured\-action precision, recall, F1, and false\-positive proportion \(%\)\. DR reaches F1 parity; the routing prompt sets the operating point\.
#### Human\-review error taxonomy\.

Table 11:Top error modes from human review\. Shares use different denominators and are not a mutually exclusive partition\.
#### Soft\-escalation buckets\.

Of the Monolithic soft escalations that DR resolves without escalation \(approximately 3% of total traffic\), Table[12](https://arxiv.org/html/2609.05758#A2.T12)shows the scenario mix used for policy review\.

Table 12:Scenario mix of soft escalations resolved by DR without escalation\. “Other” is the residual after the named buckets\.
#### Generator user\-state\-variable attribution\.

The elevated generator\-side USV attribution error in Table[3](https://arxiv.org/html/2609.05758#S6.T3)comes from strict attribution and genuine gap filling\. In the first case, a reply may correctly summarize multiple variables but cite only one key; in the second, the generator states a value absent from the contract\. We are moving from per\-sentence to per\-key attribution to separate these cases\. Gap filling is also structurally bounded: post\-generation checks reject any user\-state key outside the contract, so the residual exposure is wrong values for legitimate keys, which per\-key attribution measures directly\.

#### Illustrative failures\.

Ambiguous requests such as “I need to change my trip” can trigger the wrong policy instead of a clarifying question\. Safety\-sensitive check\-in and access issues can also require escalation even when a help article exists\. Both cases are handled as routing\-policy updates rather than generator\-only fixes\.

## Appendix CComponent Design and Optimization Details

This appendix details design choices behind DR\. It reports operational component checks, not a new measurement methodology; scoring\-instrument construction and production\-outcome validation remain out of scope\. The intent is to document the interfaces and choices a reader would need to reproduce the decomposition: what the planner can choose, what context it receives, how routing prompts were isolated, and why the smaller generator was acceptable\. Throughout,Ldenotes Qwen3\-235B\-A22B for orchestration andSdenotes Qwen3\-Next\-80B\-A3B for generation\.

### C\.1Architectural decomposition

Monolithic fixed much of the turn flow inside a single blended model path over preassembled context\. DR instead lets the orchestrator choose response generation, escalation, reservation and listing selectors, structured\-action retrieval, help\-content retrieval, and user\-state\-variable retrieval\. The generator receives the selected path through the contract and cannot choose a different action\. This exposes separately tunable surfaces: tool schemas, selector context, routing prompts, generator prompts, fallback behavior, and model placement\. The registry is versioned with the prompts: when a tool schema changes, both the orchestrator prompt and backend validator are advanced together, then shadowed before user exposure\. Table[13](https://arxiv.org/html/2609.05758#A3.T13)lists the full registry referenced in Section[3](https://arxiv.org/html/2609.05758#S3)\.

Table 13:Orchestrator tool registry\. Each tool has a typed schema, eligibility constraints, and logging fields\.
### C\.2Selector context implementation

The selector input contains the set of candidate reservation or listing IDs, whether an ID is currently active in the conversation, whether it came from the user message, profile state, itinerary state, or a previous tool observation, and a short natural\-language mention span when one exists\. The selector can return a concrete ID, anulldecision when no entity is needed, or a clarification request when multiple entities remain plausible\. This schema moved entity choice out of final prose: the generator no longer has to infer a reservation from wording, and the backend can reject any rendered card whose ID was not selected by the orchestrator\.

### C\.3Planner model selection

The orchestrator must produce short but high\-impact routing decisions\. We compared candidate planners on action\-card selection against the same human\-labeled set \(Table[14](https://arxiv.org/html/2609.05758#A3.T14)\)\. Among untuned candidate planners, GPT\-4o had the strongest initial F1, butLwas already deployed, self\-hosted, and adaptable in house; we selectedLand closed the gap through routing\-prompt iteration\.

Table 14:Planner model comparison for action\-card selection against the same human\-labeled set\. FPP denotes false\-positive proportion\.
### C\.4Action\-card routing prompt

After selectingL, we tuned the action\-card routing prompt\. The first DR prompt raised recall but over\-fired; adding selector context raised recall further; later instructions reduced over\-firing while preserving most recall \(Table[15](https://arxiv.org/html/2609.05758#A3.T15)\)\.

Table 15:Action\-card routing against the human golden label, planner onL\. Later prompt iterations explored the F1 and false\-positive trade\-off; Appendix[B](https://arxiv.org/html/2609.05758#A2)identifies the gate\-compliant deployed point\.
### C\.5Help\-content routing

The help\-content path uses a stage\-aware primary\-source rule: “why” or “can I” queries prefer policy or overview articles, while “how” or “what do I do” queries prefer procedural articles\. Encoding this at routing time keeps source choice separate from wording and improves labeled help\-content selection \(Table[16](https://arxiv.org/html/2609.05758#A3.T16)\)\. The generator then explains the selected source rather than searching again, which prevents late\-stage source drift\.

Table 16:Help\-content selection against ground\-truth labels on approximately 800 samples\. F1 is macro\-averaged per\-class F1, so it does not equal the harmonic mean of the aggregate precision and recall columns\.
### C\.6Generator model selection

The generator must leave latency headroom for orchestration, retrieval, guardrails, and translation\. We compared the largerL, the smallerS, and frontier APIs on a production\-shaped synthetic set \(Table[17](https://arxiv.org/html/2609.05758#A3.T17)\)\. Claude Opus 4 was infeasible on latency in this setup\. We choseSbecause it met release thresholds at lower P90 and about half the serving footprint ofL\. Selecting a smaller model for the narrower generation role is consistent with model compression and on\-policy distillation results\([Hinton et al\., 2015](https://arxiv.org/html/2609.05758#bib.bib10);[Agarwal et al\., 2024](https://arxiv.org/html/2609.05758#bib.bib11)\); here we use an existing open\-weight model aligned to policy \(Appendix[E](https://arxiv.org/html/2609.05758#A5)\) rather than distilling our own\.

Table 17:Generator candidates on a production\-shaped synthetic set: serving footprint \(relative to the deployed configuration\), model\-only P90 latency, and release\-check scores\. Latency is measured under model\-selection settings and is not directly comparable to the production generation\-stage latency in Table[1](https://arxiv.org/html/2609.05758#S5.T1)\.
### C\.7Generator prompt and residual headroom

WithSselected, we compared a reduced\-context per\-solution prompt with a merged full\-context prompt\. The reduced prompt is attractive because it passes only the action\-card, help\-content, or contextual\-variable slice selected by the orchestrator\. The merged prompt is operationally simpler: it always passes the same schema and lets the generator decide how much of the allowed context to mention\. The core quality scores were close, so we chose the merged prompt because one schema removes per\-path branching, shortens replies, and slightly improves style, tone, and repetition behavior \(Table[18](https://arxiv.org/html/2609.05758#A3.T18)\)\.

Table 18:Generator prompting regimes onSover a shared shadow\-traffic set of approximately 455 samples\. Scores are in\[0,1\]\[0,1\]except word count; lower is better for repeated instruction\.Table[19](https://arxiv.org/html/2609.05758#A3.T19)shows the residual gap betweenSand largerL\. We treat it as future generator headroom, not evidence against the split, because the architecture gains come from better upstream decisions\.

Table 19:Deployed generatorSversus larger in\-houseLon a shared shadow\-traffic set\.Δ\\DeltaisSminusL\.

## Appendix DServing Model Detail

#### Speculative decoding and co\-location\.

EAGLE\-3 speculative decoding is being validated as a serving\-only optimization for a further\-reduced footprint target \(Table[20](https://arxiv.org/html/2609.05758#A4.T20)\)\. The draft model fuses low, middle, and high verifier\-layer hidden states to predict the next three tokens; under standard speculative verification, accepted outputs preserve the verifier distribution\. We train the draft on production\-shaped traces, warm\-start from a public Qwen3\-235B speculator checkpoint, and validate on shadow traffic so the optimization changes serving rather than behavior\. We co\-locate orchestrator and generator capacity so the two share one availability envelope; future generator swaps leave the orchestrator interface unchanged because the contract is stable\. Table[20](https://arxiv.org/html/2609.05758#A4.T20)reports orchestrator\-only serving latency across operating points, expanding the headline figures in Section[5](https://arxiv.org/html/2609.05758#S5)\. Footprints are stated relative to the pre\-optimization baseline\.

Table 20:Orchestrator serving latency in seconds and GPU\-footprint reduction relative to the pre\-optimization baseline\. Opt\. 1 is the current production operating point; Opt\. 2 is the rollout target for speculative decoding; Opt\. 3 is a candidate operating point that trades slightly higher latency for a further\-reduced footprint\.Table[21](https://arxiv.org/html/2609.05758#A4.T21)summarizes orchestrator candidates from a serving and controllability perspective: active parameters, latency under our stack, and whether the model allows in\-house adaptation\. We selected Qwen3\-235B\-A22B not because it was the highest\-quality model on every slice, but because it gave the best joint operating point: low served latency in our stack, self\-hosted control, and a path to future in\-house adaptation without changing the tool interface\.

Table 21:Orchestrator model options from a serving and controllability perspective\. Latency is measured on the model\-selection benchmark under internal settings and is not directly comparable to the production orchestrator latency in Table[20](https://arxiv.org/html/2609.05758#A4.T20)\. Proprietary models are API\-served, while Qwen3 is self\-hosted\.

## Appendix EAlignment Data and Reward Abstraction

This appendix gives a minimal description of the alignment stage used after model selection\. We intentionally keep implementation details abstract because this paper focuses on the deployed orchestrator and generator interface, not on a full evaluation or reward\-calibration study\.

### E\.1Alignment data

We construct alignment examples from de\-identified production and simulated support cases after removing opt\-out traffic\. Cases are sampled to cover both successful and unsuccessful interactions\. For each input, the current policy produces multiple candidate responses under the same conversation state, retrieved context, and available actions\. This creates within\-input comparisons that isolate response\-policy behavior while holding the orchestrator inputs fixed\. The resulting candidates are scored by task rewards and converted into the training format required by the chosen post\-training method, following the RLHF lineage\([Ouyang et al\., 2022](https://arxiv.org/html/2609.05758#bib.bib6);[Schulman et al\., 2017](https://arxiv.org/html/2609.05758#bib.bib29)\)\. Pairwise methods use higher\-scoring and lower\-scoring responses as preference pairs\([Rafailov et al\., 2023](https://arxiv.org/html/2609.05758#bib.bib7);[Hong et al\., 2024](https://arxiv.org/html/2609.05758#bib.bib8)\), while relative\-reward methods use the candidate set to compute group\-level advantages\([Shao et al\., 2024](https://arxiv.org/html/2609.05758#bib.bib30);[Zheng et al\., 2025](https://arxiv.org/html/2609.05758#bib.bib9)\)\.

### E\.2Reward abstraction

We use two reward families\. A quality\-centered reward scores whether the response is correct, grounded, clear, and compliant with the prompt\. A reference\-aware reward additionally checks whether the response covers a likely solution and uses supporting knowledge\-base or action\-card evidence consistently\. The second family is intended to discourage safe but under\-resolving answers, such as generic policy summaries or clarification\-only responses when enough evidence exists to provide a conditional next step\. To reduce leakage between training and evaluation, reward scoring is separated from final offline evaluation\. Before serving, the selected checkpoint is frozen and evaluated through the same deployment checks as other candidate updates\. The serving interface is unchanged: the orchestrator still supplies conversation state, retrieved context, and available actions, while only the generator policy is replaced\.

## Appendix FServing\-Cost Estimate

For illustration only, we estimate annual model\-serving cost at our volume and decoding settings at roughly $6M for the proprietary\-API baseline versus roughly $200K self\-hosted\. Both are order\-of\-magnitude internal estimates, not audited financials or actuals\. We state the basis so readers can rescale to their own setting\.

#### Proprietary\-API baseline\.

Estimated from annual generated\-plus\-prompt token volume across orchestration and generation at production traffic, and the blended list price of the proprietary API considered\. For rescaling: the orchestrator processes roughly 8,000 prompt and 200 completion tokens per model call \(up to three calls per turn\), and annual turn volume is on the order of10710^\{7\}\.

#### Self\-hosted\.

Estimated as amortized hardware plus power for the production footprint \(the orchestrator at its current production operating point plus co\-located generator capacity\), at our measured utilization over a standard amortization period\. It*excludes*engineering and on\-call labor, model\-development cost, and shared\-platform overhead; including loaded labor would raise the self\-hosted figure but not by enough to close an order\-of\-magnitude gap\.

#### Scope of the estimate\.

The comparison is therefore self\-hosted hardware cost against API list price\. The takeaway we stand behind is a greater\-than\-tenfold reduction in*serving*cost at our scale, not a precise ratio\. The figures move with traffic, hardware generation, utilization, and API pricing\.