Natural-Language Policies to Executable Decisions: An Interpretable Large Language Model Framework

# Natural-Language Policies to Executable Decisions: An Interpretable Large Language Model Framework
Source: [https://arxiv.org/html/2608.26124](https://arxiv.org/html/2608.26124)
Ziqiang Zhang,Jing Ma,Zilong Wang,Jiayuan Chen,Yi Qiao,Yu He, Wei Zhang,Dai Cheng,Xiaoyu Shen Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo zqzhang@idt\.eitech\.edu\.cn xyshen@eitech\.edu\.cn

###### Abstract

Pricing automation in large\-scale tourism is challenging because travel orders are highly unstructured, while pricing policies are complex, rapidly evolving, and inherently open\-ended\. Traditional rule engines are brittle and costly to maintain, whereas unconstrained LLM agents lack the reliability and auditability required for financial decisions\. We present a production\-grade LLM\-powered pricing system with a strict decision boundary: LLMs perform structured extraction and bounded policy/path selection, while all numeric pricing, including total\-price computation, is executed deterministically\. Policies are compiled into interpretable condition trees, enabling open\-ended support for new clauses and evolving rules without code changes, while exposing auditable artifacts for human\-in\-the\-loop control\. Periodic fine\-tuning on logged traces further improves tree induction and path matching\. Deployed at a municipal state\-owned tourism enterprise across 7 scenic sites and 12 business categories with 1,500\+ operators and 1,000\+ active policies, the system processed 3,960 orders in six months, reduced the order management team from 15–20 to 3, and cut per\-order handling time from∼\\sim10 minutes to<2<2minutes\.

Natural\-Language Policies to Executable Decisions: An Interpretable Large Language Model Framework

Ziqiang Zhang, Jing Ma, Zilong Wang, Jiayuan Chen, Yi Qiao, Yu He,Wei Zhang,Dai Cheng,Xiaoyu Shen††thanks:Corresponding author\.Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbozqzhang@idt\.eitech\.edu\.cn xyshen@eitech\.edu\.cn

## 1Introduction

Accurate and timely processing of travel orders is the operational core of modern tourism services\. A typical order describes a group’s itinerary, including scenic spots, travel dates, agent name, etc, which must be transformed into structured facts and matched with complex repository of pricing policiesKaushiket al\.\([2017](https://arxiv.org/html/2608.26124#bib.bib15)\)\. While automating this workflow is essential for operational scalability, it remains remarkably difficult in practice due to the unstructured nature of travel requests and the intricate, ever\-evolving pricing policiesZhouet al\.\([2025](https://arxiv.org/html/2608.26124#bib.bib9)\); Liuet al\.\([2025](https://arxiv.org/html/2608.26124#bib.bib16)\)\.

In real\-world production settings, travel orders exhibit extreme heterogeneity\. Requests range from semi\-structured digital forms to casual instant\-messaging text or even handwritten notices\. Critical information is often implicit or ambiguous: scenic spots may be referred to by informal abbreviations, and travel dates frequently rely on background context rather than explicit mention\. These factors transform simple data parsing into a high\-order information\-understanding task that requires reasoning well beyond surface\-level patterns\(Xuet al\.,[2019](https://arxiv.org/html/2608.26124#bib.bib8); Mathewet al\.,[2020](https://arxiv.org/html/2608.26124#bib.bib10)\)\.

This input complexity is compounded by the fundamentally*open\-ended*nature of pricing policies\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.26124#bib.bib12)\)\. Determining the applicability of a policy requires reasoning over a vast array of interacting conditions, such as seasonal windows, group size thresholds, and bespoke contractual clauses, distributed across heterogeneous policy documents\. Because new policies routinely introduce novel conditions that were never anticipated at system design time, traditional software\-based solutions that rely on manually encoding logic into structured databases and rule engines\(Desmondet al\.,[2022](https://arxiv.org/html/2608.26124#bib.bib17)\)are inherently brittle\. Each policy update often necessitates synchronized changes to backend schemas, rule code, and user interfaces, creating prohibitive maintenance costs and making city\-scale deployment unscalable\.

Recent advances in large language models \(LLMs\) offer a flexible alternative for reasoning over natural languageSuet al\.\([2022](https://arxiv.org/html/2608.26124#bib.bib5)\); Achiamet al\.\([2023](https://arxiv.org/html/2608.26124#bib.bib4)\); Liuet al\.\([2024](https://arxiv.org/html/2608.26124#bib.bib3)\); Xuet al\.\([2025](https://arxiv.org/html/2608.26124#bib.bib6)\); Dinget al\.\([2026](https://arxiv.org/html/2608.26124#bib.bib7)\)\. In principle, LLMs can parse informal orders, interpret policy text, and perform cross\-document reasoning without rigid schemas\. In practice, however, fully autonomous LLMs are ill\-suited for high\-stakes applications\. They often lack stability, provide limited interpretability, and offer no clear mechanism for non\-technical users to inspect or correct intermediate reasoning steps, which is an essential requirement for pricing and auditability\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.26124#bib.bib11)\)\.

In this work, we present a production\-grade, LLM\-powered pricing system that reconciles the linguistic flexibility of large language models with the strict reliability requirements of real\-world financial operations\. Our central insight is that LLMs should not replace pricing systems, but instead operate within a carefully defined and interpretable decision boundary\. The system is designed around two core principles: \(1\)Open\-ended condition support: Instead of hard\-coding logic, we leverage LLMs to match order\-side facts against policy\-side condition trees\. This design naturally supports an evolving rule space, enabling the system to handle arbitrary ad\-hoc rules and seasonal exceptions without code changes or model retraining\(Gaoet al\.,[2022](https://arxiv.org/html/2608.26124#bib.bib13); Chenet al\.,[2022](https://arxiv.org/html/2608.26124#bib.bib14)\)\. \(2\)End\-to\-end interpretability: The system automatically extracts logic from policy documents and organizes it into explicitcondition trees\. These trees preserve the semantics of natural language while exposing their logical structure, allowing tourism managers to review and maintain rules without programming knowledgeXionget al\.\([2024](https://arxiv.org/html/2608.26124#bib.bib1)\); Wanget al\.\([2025](https://arxiv.org/html/2608.26124#bib.bib2)\)\. Crucially, all numeric computation is handled by a deterministic engine\. LLMs are restricted to structured extraction and discrete decision selection\. This separation provides the reliability of traditional software systems while retaining the reasoning flexibility of modern language models\.

We deployed the system at a municipal, state\-owned tourism enterprise operating a city\-wide platform spanning seven major scenic areas and twelve business categories, with 1,500\+ operators\. Over the first six months after rollout, the system processed 3,960 orders, reduced the order\-management team from 15–20 to 3, and cut per\-order handling time from∼\\sim10 minutes to<2<2minutes\. The deployment externalizes pricing governance into auditable artifacts—OrderFacts, policy\-induced condition trees, and logged decision traces—making disagreements actionable via bounded human confirmation and overrides\. These traces enable systematic error diagnosis and iterative improvements, without changing numeric execution\. An internal survey reports improved communication/efficiency \(96\.92%\) and positive feedback on usability and analyzable rules \(76\.92%\)\.

While our deployment focuses on tourism, the challenges of open\-ended policy logic, informal inputs, strict correctness requirements, and human\-centered governance are common across many domains such as insurance underwriting, compliance checking, and contract execution\. By eliminating rigid rule engineering while preserving human agency, our system offers a practical path toward scalable, trustworthy AI in policy\-driven industries\.

## 2Background

#### The Pricing Lifecycle in Tourism

Operational pricing in modern tourism follows a recurring tripartite lifecycle: \(i\)policy onboarding, \(ii\)order intake and quotation, and \(iii\)settlement and verification\. The process begins when business managers publish pricing policies—documents \(often PDFs or spreadsheets\) that define the rules for various customer segments and travel seasons\. Once published, tour coordinators receive travel orders and must match them against the active policy repository to generate a quote\. Finally, the calculated price must be verified against the original policy to ensure auditability and financial compliance\.

#### Open\-Ended Complexity of Pricing Policies

A core challenge in this domain is that pricing conditions are fundamentallyopen\-ended\. Policies are defined over*resources*\(atomic sellable units like a cable car ride\) and*products*\(bundles of resources\)\. However, the conditions governing their price, ranging from specific age brackets and group size thresholds to complex seasonal overlaps and ad\-hoc contractual clauses, are virtually unlimited\. New policies frequently introduce entirely novel logic that was not anticipated during initial system design\. This evolving complexity means that “pricing logic” is not a static set of parameters, but a growing library of natural\-language rules that must be interpreted in context\.

#### Informality and Mismatch in Travel Orders

The intake side of the lifecycle is equally challenging due to the extreme informality of travel orders\. In our deployment, 97% of orders \(based on 2025H2 production logs\) are received as instant\-messaging screenshots rather than structured digital forms\. These requests mix travel dates, destinations, and casual notes without a fixed template\. Furthermore, a “resource mismatch” is common: the itinerary described in the initial text often deviates from the final executed plan\. Nearly half of our production cases require manual edits to the item list before a quotation can be finalized, necessitating a system that can tolerate incomplete descriptions and mid\-process human correction\.

#### The Inadaptability of Traditional Rule Engines

Traditional software solutions are ill\-equipped for this environment because they rely on rigid, pre\-defined schemas\. Since pricing conditions are unlimited, adding a new policy often requires more than just data entry; it necessitates a coordinated update of the*database structure*\(to store new attributes\), the*rule logic*\(to handle new predicates\), and the*input UI*\(to allow users to select these new options\)\. This “hard\-coding” cycle creates a massive maintenance burden, as software developers must constantly translate natural\-language nuances into executable code\. For large tourism sites with hundreds of evolving policies, the delay and cost of these manual system updates make traditional rule engines fundamentally unscalable\.

#### Implications for System Design

These constraints dictate that a production system cannot rely on a “closed\-world” assumption\. Instead, the system must: \(i\) decouple policy logic from the underlying software schema to handle an open\-ended condition space, and \(ii\) externalize its reasoning into*interpretable intermediate artifacts*that preserve natural\-language semantics and treat human intervention as a first\-class operation\.

## 3Methodology

We design an auditable pricing pipeline that operationalizes informal travel orders and natural\-language pricing policies under strict governance constraints\. The central design principle is a*strict LLMs decision boundary*: LLMs are restricted to structured information extraction and discrete decision selection, while all numeric calculations are performed by a deterministic engine\. This boundary structurally eliminates numeric hallucination, makes remaining uncertainty explicit, and enables rapid human verification\.

![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/flow_overview.png)Figure 1:System overview\. The policy parsing module induces interpretable condition trees from pricing documents, and the price calculation module routes orders to candidate trees, verifies applicability, and computes final prices with deterministic execution and human\-in\-the\-loop overrides\.We provide the executable tree schema \(Table[1](https://arxiv.org/html/2608.26124#S3.T1)\), the validator suite \(Table[2](https://arxiv.org/html/2608.26124#S3.T2)\), a production\-faithful tree instance \(Fig\.[6](https://arxiv.org/html/2608.26124#S3.F6)\), and end\-to\-end UI traces that externalize L1–L3 confirmations and overrides \(Figs\.[9](https://arxiv.org/html/2608.26124#S3.F9)–[11](https://arxiv.org/html/2608.26124#S3.F11)\)\. Before publication, every induced tree must pass an explicit validator suite \(Table[2](https://arxiv.org/html/2608.26124#S3.T2)\), and any failure is blocked from entering execution and routed to manual correction; moreover, all tree management is presented to operators through a dedicated UI \(Fig\.[5](https://arxiv.org/html/2608.26124#S3.F5)\) for friendly oversight\. The system is decomposed into two major components: a*policy onboarding module*\(stages*A0–A3*\) and a*price calculation module*\(stages*B0–D*\)\. To ensure reliability, we incorporate a tiered human\-in\-the\-loop override process \(*L1–L3*\) that allows operators to intervene at distinct levels of the reasoning chain\. These identifiers correspond to the architectural components illustrated in Figure[1](https://arxiv.org/html/2608.26124#S3.F1)\.

### 3\.1Policy Onboarding and Tree Induction

The policy processing module is the "offline" stage where policy documents are transformed into*condition trees*\. A condition tree acts as the sole executable interface for pricing a specific product\. In this structure, each internal node represents a natural\-language applicability condition, while each root\-to\-leaf path corresponds to exactly one*price specification*\. This*one\-path–one\-price\-spec*invariant guarantees that once a path is selected, the deterministic calculation\. The induction of these trees follows a four\-step pipeline:

![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/ui_catalog_R.png)Figure 2:Catalog management UI defining the executable resource spaceRR\(A1\)\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/ui_policy_parse.png)Figure 3:Policy parsing/editing UI for inducing and reviewing condition trees before publication \(A2–A3\)\.A0: Text extraction\.We first convert policy documents into text via OCR or table extraction, producing a normalized textual input\.

A1: Identify resources\.We maintain a predefined resource catalogRRthat serves as the canonical space of system\-recognized resources\. Each resource represents an operator\-defined atomic capability that can be referenced by downstream execution components\. In addition to defining individual resources, the catalog also specifies the product \(bundle\) configurations that can be constructed from these resources\. As a result,RRconstrains both the allowableresource\_idsand the feasible product combinations used in subsequent filtering steps \(B1\)\. During policy interpretation, the LLM performs catalog grounding, mapping natural language policy descriptions into a set of resources drawn strictly fromRR\. Because the grounding process operates over this predefined catalog, it prevents out\-of\-catalog entities from entering the executable interface\. In our system architecture \(Fig\.[2](https://arxiv.org/html/2608.26124#S3.F2)\), the catalogRRis treated as a first\-class operator artifact and serves as the only executable namespace for resources\. To ensure consistency of the catalog, uniqueness constraints are enforced during the catalog maintenance stage\. At the resource level, each resource entry is uniquely identified by its name attribute in the database schema\. Newly introduced resources are also subject to manual review to avoid semantic duplication or naming conflicts\. At the product level, each product represents a predefined bundle of resources stored in the catalog, and duplicate bundles are disallowed\. During catalog maintenance, the system checks whether a candidate product definition duplicates an existing bundle\. Given a list of resource names describing the candidate bundle, the system performs an exact matching query over the existing product–resource mappings in the database\. The query verifies that the total number of resources matches and that the occurrence count of each resource is identical, while also ensuring that no additional resources are present\. If such a product already exists, the candidate definition is rejected to prevent duplicate bundle entries\. These constraints ensure that the catalog remains consistent and unambiguous before it is used by the LLM grounding process\. Consequently, all LLM operations rely exclusively on operator\-defined and prevalidated catalog data, ensuring deterministic and reliable execution\.

A2: Extract nodes\.For each grounded entity, the LLM extracts three types of information that align with the structure illustrated in Fig\.[6](https://arxiv.org/html/2608.26124#S3.F6): \(i\) policy metadata, such as customer scope and validity windows \(the outer metadata layer used for deterministic routing\); \(ii\) the natural\-language conditions that define the logic of the policy \(the inner arbitrarily nested condition tree branching into resource\-/product\-specific subtrees\); and \(iii\) the leaf\-level*price specifications*\. Crucially, the LLM treats numeric values as static fields to be stored, rather than performing any calculations at this stage\. Fig\.[6](https://arxiv.org/html/2608.26124#S3.F6)shows a production\-faithful example of this structure, which motivates our choice of a UI\-editable tree representation rather than hard\-coded rule schemas\.

A3: Assemble & validate\.We assemble extracted clauses into an executable condition tree while enforcing the*one\-path–one\-price\-spec*invariant: each root\-to\-leaf path corresponds to exactly one leaf\-level*PriceSpec*, ensuring deterministic execution once a path is selected\. To make trees*auditable*and*UI\-editable*in production, each node follows a typed schema that separates \(i\) structural fields \(id,children,isLeafNode\), \(ii\) semantic payload \(fieldName,fieldValue\), and \(iii\) leaf\-only pricing fields \(price,unit\), plus constraint/UI metadata such aslimitValueandexpanded\(Table[1](https://arxiv.org/html/2608.26124#S3.T1)\)\. In particular,limitValueencodes applicability constraints as a two\-level boolean form \(and/orover groups, each group over atomic predicates\), which is expressive enough for our policy space while remaining easy to validate and render consistently\.

Before publication, every induced tree must pass an explicit validator suite \(Table[2](https://arxiv.org/html/2608.26124#S3.T2)\) that checks schema completeness, unit compatibility, numeric sanity, catalog resolvability, constraint\-format compliance, and cross\-path conflict detection under the same metadata window\. Any failure blocks the tree from entering execution and routes it to operator review\. Operators inspect and edit the induced structure in an onboarding console \(Fig\.[3](https://arxiv.org/html/2608.26124#S3.F3)\), where the nested condition hierarchy and leaf PriceSpecs are displayed in a tree form, making extraction errors observable and correctable prior to release\. Only validated trees are published to the*Tree Store*\.

FieldTypeDescriptionidintUnique node identifierfieldNamestrNode category \(e\.g\., product type, condition\)fieldValuestrCategory value \(product name or condition clause\)childrenarrayChild nodes; empty array for leavesisLeafNodeboolWhether this node is a leafLeaf\-only fields \(pricing specification\)pricenumberUnit priceunitstr/nullBilling unit \(e\.g\., per person, per group\)Constraint and UI fieldslimitValueobject/nullStructured applicability constraintslimitstrReserved for future useexpandedboolUI expansion state \(default: true\)Table 1:Full schema of condition\-tree nodes\. Fields are grouped by function: identification and structure \(top\), leaf pricing specification \(middle\), and constraints/UI metadata \(bottom\)\.ValidatorPurposeschema\_completeAll required fields present; tree is well\-formedunit\_compatBilling units are consistent and compatible across sibling leavesnumeric\_sanityPrices are non\-negative and within plausible boundsconflict\_detectNo contradictory rules under overlapping metadata \(scope, validity\)service\_nameResource/service names resolve to valid catalog entriesrestrict\_formatlimitValuestructure conforms to the two\-level boolean schemaTable 2:Validators applied during policy onboarding\. A tree is published only if all validators pass\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/OrderFacts.png)Figure 4:OrderFacts panel for order intake\. Extracted structured fields are visible and editable before downstream bundle enumeration and policy matching \(B0\)\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/price_policy_ori.png)Figure 5:Policy management UI for validity windows and status, which serve as hard constraints for metadata\-based routing \(B2–B3\)\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/price_policy_demo.png)Figure 6:A production\-faithful induced policy structure with an outer metadata layer and an arbitrarily nested condition tree\.
### 3\.2Order Parsing and Price Calculation

![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/B1_v3.png)Figure 7:B1: Candidate bundle enumeration\.Given atomic resource mentions and a catalog of sellable bundles, we deterministically enumerate all feasible bundle decompositions that exactly exhaust the requested resources for downstream tree routing and path selection\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/demo_v6.png)Figure 8:Toy walkthrough of tree\-guided pricing\.The LLM extracts*OrderFacts*\(B0\)\. B1–B3 deterministically route the order to a*bounded*policy tree via metadata filtering\. The LLM then performs*discrete*root\-to\-leaf path selection \(C\) and returns a leaf\-level*PriceSpec*, while the deterministic engine computes the final amount from validated quantities \(D\)\. A production\-faithful condition tree with explicit metadata constraints is shown in Fig\.[6](https://arxiv.org/html/2608.26124#S3.F6)\.Once policies are onboarded as condition trees, the price calculation module processes incoming orders through the following stages:

B0: Order Parsing and Fact ExtractionIncoming orders \(typically screenshots\) are parsed by an LLM into*OrderFacts*\(*B0*\)\. This structured record contains non\-resource fields, such as travel dates, group size, and customer identity, and grounded resource mentions\.OrderFacts are surfaced in the operator console as an editable, structured form to correct missing/ambiguous fields before downstream routing and execution \(Fig\.[4](https://arxiv.org/html/2608.26124#S3.F4)\)\. This editability is essential because many screenshot orders are under\-specified or later revised: operators can correct missing dates/team size and reconcile resource mentions before deterministic routing and execution, turning ambiguity into an explicit, auditable intervention rather than a silent failure\.

B1: Candidate Bundle EnumerationSince orders may mention individual resources that are sold as*products*\(bundles\), the system must determine the best way to group these resources\. We use a deterministic DFS backtracking algorithm to enumerate all feasible bundle combinations that exactly exhaust the requested resources; Figure[7](https://arxiv.org/html/2608.26124#S3.F7)gives a toy illustration\.

B2–B3: Retrieval and RoutingFor each candidate bundle/product enumerated in B1, we retrieve its associated condition trees from the*Tree Store*and deterministically filter/prune them using policy metadata constraints, including: \(i\) overlap between the order date range and the policy validity window, and \(ii\) match between the order\-side customer scope and the policy applicability scope\. After obtaining a bounded set of viable trees per candidate bundle/product, we further route candidates deterministically by a predefined pricing schema \(e\.g\.,single\-itemvs\.bundlestructures\) to select the appropriate matching policy trees\. In production, routing metadata \(validity window, customer scope, and policy status\) is governed as a first\-class artifact: policies are versioned, activated/deactivated, and audited through an operator console \(Fig\.[5](https://arxiv.org/html/2608.26124#S3.F5)\)\. This ensures that deterministic routing only considers policies that are currently effective and approved, preventing stale or unofficial rules from entering execution\.

C–D: Path Selection and ExecutionAs shown in Figure[8](https://arxiv.org/html/2608.26124#S3.F8), in the final reasoning step, an LLM analyzes the filtered trees and selects the root\-to\-leaf path whose conditions are satisfied by the*OrderFacts*\(*C*\); in production, the same selection operates over policy trees with an explicit metadata layer and richer nested subtrees \(Fig\.[6](https://arxiv.org/html/2608.26124#S3.F6)\)\. To ensure transparency, the model provides evidence pointers to specific spans in the order text\. Once a path is confirmed, the deterministic engine executes the final pricing \(*D*\) by applying the quantity normalization and units defined in the selected*PriceSpec*\. The PriceResult view externalizes the selected candidate tree \(L2\) and matched path/leaf \(L1\) together with resource\-level breakdown and recomputable totals, enabling auditable confirmation and bounded overrides within the exposed candidate set \(Fig\.[11](https://arxiv.org/html/2608.26124#S3.F11)\)\.

### 3\.3Governance and Self\-Evolving

#### Override Protocol

To maintain zero\-tolerance for numeric errors, we expose the system’s reasoning through three override levels: \(1\)*L1 \(Path Override\):*The operator selects a different branch within a fixed tree, correcting LLM errors of natural\-language conditions \(Fig\.[11](https://arxiv.org/html/2608.26124#S3.F11)\)\. \(2\)*L2 \(Tree Override\):*The operator switches between valid policy trees, typically used to resolve overlapping policy conflicts \(Fig\.[11](https://arxiv.org/html/2608.26124#S3.F11)\)\. \(3\)*L3 \(Composition/Data Override\):*The operator corrects upstream inputs, either by editing the resource list \(*L3\-a*\) to fix extraction or itinerary mismatch \(Fig\.[9](https://arxiv.org/html/2608.26124#S3.F9)\), or by switching the bundle decomposition/product \(*L3\-b*\) when composition differs from the actual itinerary \(Fig\.[10](https://arxiv.org/html/2608.26124#S3.F10)\)\.

![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/ui_resource_edit.png)Figure 9:Add/remove resource UI \(*L3\-a resource edit*\)\. Operators reconcile extracted resources with the actually executed itinerary before pricing\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/ui_switch_bundle.png)Figure 10:Bundle/product switching UI \(*L3\-b composition override*\)\. Operators switch among feasible bundle decompositions enumerated in B1\.![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/ui_price_result.png)Figure 11:PriceResult view\. Each item externalizes the selected tree \(*L2*\) and matched path/leaf \(*L1*\) with recomputable breakdown; operators can switch the candidate tree \(L2\) or candidate path \(L1\) within the exposed set\.
#### Failure Modes and Data Flywheel

Production failures are categorized into three types:*under\-specification*\(incomplete order info\),*selection errors*\(incorrect path choice\), and*fatal numeric errors*\. As shown in Figure[12](https://arxiv.org/html/2608.26124#S4.F12), our architecture is designed to make the first two types recoverable via the*L1–L3*protocols, while the third is prevented by our strict decision boundary\. Finally, we maintain a*production data flywheel*by logging raw model outputs alongside human\-confirmed overrides\. This allows for iterative improvement of extraction and selection prompts without risking the stability of the deterministic execution logic\.

## 4Experiment

We deploy our system on a city\-level state\-owned tourism enterprise\. This enterprise is responsible for managing seven major scenic areas across the city, encompassing 12 diverse tourism business categories across land, water, and air dimensions \(including accommodation, dining, car rental, attraction visits, shopping, entertainment, tour guide services, performances, cruise ships, conferencing, cable cars, drones, etc\.\), with over 1,500 operational staff, complex organizational structure, and diverse role distributions\. We evaluate our system on production logs from 2025H2 \(Jul–Dec 2025\)\. The dataset comprises 3,960 orders, of which 3,842 \(97\.0%\) are instant\-messaging screenshots processed via OCR and 118 \(3\.0%\) are Word uploads\.

![Refer to caption](https://arxiv.org/html/2608.26124v1/latex/figs/flywheel.png)Figure 12:Failure\-mode taxonomy and data flywheel\.We focus on screenshot orders as the dominant deployment modality\. Table[3](https://arxiv.org/html/2608.26124#S4.T3)summarizes order structural complexity: over 70% of orders involve multiple resources, and 25% contain bundle products, motivating the deterministic composition enumeration in stage B1\.

CategoryCount%Single\-resource orders1,17829\.7Multi\-resource orders2,78270\.32 resources1,85846\.93\+ resources92423\.4Orders with bundle products99025\.0Table 3:Order structural complexity \(2025H2,n=3,960n=3\{,\}960\)\.#### Evaluation Protocol

We organize evaluation as a decision\-boundary funnel that progressively isolates each decision level, so that downstream metrics reflect only the residual decision\. In Track 1, we quantify operator workload on all 3,842 screenshot orders\. We distinguish two mutually exclusive L3 interventions: L3\-a \(resource edit\) addresses mismatch between the extracted resource set and the actually executed itinerary, while L3\-b \(bundle/product switch\) addresses composition mismatch when the resource set is already consistent\. After removing all L3 cases, we measure L2 \(tree switching\) on the remaining orders\. In Track 2, we isolate within\-tree path selection \(L1\) and pricing correctness\. From the 1,785 orders that survive L2 and L3 filtering, we further restrict to an information\-consistent subset where critical pricing fields \(travel dates and team size\) are consistent with logged outcomes, yielding 1,349 orders covering 2,598 matched product instances\. All correctness metrics are reported on this subset\. We report three metrics aligned with production requirements\. First, workload is measured by the frequency of L3\-a, L3\-b, and L2 interventions across the full screenshot set\. Second, unit\-price accuracy is the fraction of product instances whose predicted unit price matches any logged ground\-truth price under the same order\-product key\. Third, the fatal numeric error rate captures any output with invalid units, aggregation mismatch, or prices unsupported by a verified leaf specification\.

#### Results

Table[4](https://arxiv.org/html/2608.26124#S4.T4)presents the complete decision\-boundary funnel\.

Decision boundaryCountRateTrack 1: Production workload \(n=3,842n\{=\}3\{,\}842\)L3\-a resource edits1,84247\.9%L3\-b bundle/product switch1082\.8%No L3 intervention1,892—L2 tree switching1075\.7%No L2/L3 intervention1,785—Track 2: Correctness \(n=2,598n\{=\}2\{,\}598product inst\.\)Unit correctness2,598 / 2,598100%Unit\-price accuracy \(L1\)2,214 / 2,59885\.2%Fatal numeric error0 / 2,5980%Table 4:Decision\-boundary funnel\. Track 1 reports operator workload on the full screenshot set; Track 2 reports correctness on the information\-consistent subset where only L1 remains\.On the full 3,842 screenshot orders, 1,842 \(47\.9%\) require L3\-a resource edits and 108 \(2\.8%\) require L3\-b bundle/product switching\. The high L3\-a rate reflects a known operational reality: coordinators frequently omit or revise resources after submission, so this is not a system error but a characteristic of the domain workflow\. Among the 1,892 orders requiring no composition\-level intervention, only 107 \(5\.7%\) require L2 tree switching, indicating that metadata\-based routing resolves the vast majority of policy\-selection decisions automatically\. On the 1,349 information\-consistent orders \(2,598 product instances\), the system achieves zero fatal numeric errors, confirming that the strict LLM decision boundary—where the model selects discrete paths rather than generating prices—structurally eliminates numeric hallucination\. Unit\-price accuracy reaches 85\.2%; the remaining 14\.8% are L1 path\-selection errors that operators can recover via a single UI click rather than requiring any change to the execution layer\.

#### Operational Impact

Table[5](https://arxiv.org/html/2608.26124#S4.T5)summarizes key deployment metrics collected over one year of production operation at a tourism site comprising 19\+ scenic spots and over 1,000 annual pricing policies\.

MetricBeforeAfterPolicy onboarding time∼\{\\sim\}20 min∼\{\\sim\}1 minOrder processing timemanual∼\{\\sim\}2 minFatal numeric errors—0%Developer involvementrequirednot requiredTable 5:Operational impact before and after deployment\.The most significant operational gain is the elimination of developer involvement in policy onboarding\. Previously, each new pricing policy required an expert developer to translate natural\-language conditions into executable rules, taking approximately twenty minutes per policy\. With our system, non\-technical tourism managers can review and publish condition trees directly, reducing onboarding time to roughly one minute and eliminating the communication overhead between business and engineering teams\. Order processing time is also reduced to about three minutes on average, covering the full pipeline from OCR extraction through pricing execution and operator confirmation\.

## 5Discussion

The central design choice of our system is confining the LLM to discrete selection tasks—resource grounding, condition extraction, and path selection—while delegating all numeric computation to deterministic execution from validated leaf specifications\. This strict decision boundary yields two concrete benefits observed in production\. First, the zero fatal numeric error rate across 2,598 product instances demonstrates that preventing the LLM from generating prices structurally eliminates numeric hallucination\. Second, all remaining errors are discrete and recoverable: operators correct path\-selection mistakes via L1–L3 overrides without touching the execution layer, making the system’s failure modes transparent and manageable\.

The condition trees serve not merely as post\-hoc explanations but as the primary editing interface for non\-technical operators\. This design is validated by the low L2 override rate \(5\.7%\): once a tree is correctly constructed during onboarding, metadata\-based routing reliably selects it, and operators need only inspect and occasionally adjust the within\-tree condition path\. The interpretability\-as\-interface principle also simplifies the data flywheel—disagreements between model outputs and operator\-confirmed results can be directly attributed to specific decision boundaries, enabling targeted iteration rather than opaque end\-to\-end retraining\.

The 14\.8% unit\-price mismatch in Track 2 is concentrated in two categories\. The first is under\-specification induced by open\-ended eligibility conditions: a group may simultaneously satisfy multiple policy paths, and some discounted clauses require extra evidence that is typically unavailable in the order text \(e\.g\., military ID, local employment/eligibility certificates\)\. As a result, the correct unit price is not uniquely identifiable fromOrderFactsalone; the system may select a verifiable default path while the logged outcome reflects additional offline information, leading to an apparent mismatch\. The second is OCR noise from screenshot orders, where garbled text leads to incorrectOrderFactsextraction\. Both are addressable without architectural changes: the former by making eligibility evidence explicit during onboarding and surfacing such paths as “needs additional proof” for operator confirmation, and the latter through improved OCR preprocessing or encouraging structured order input formats\.

While our evaluation is conducted in tourism pricing, the framework’s core components—condition tree induction, deterministic candidate enumeration, and evidence\-backed discrete selection—are domain\-agnostic\. Industries with evolving natural\-language policies and zero\-tolerance pricing requirements, such as insurance underwriting, regulatory compliance, and contract execution, can adopt the same architecture by replacing the resource catalog and product dictionary with domain\-specific counterparts\.

## 6Conclusion

We present a production LLM framework that converts natural\-language pricing policies into executable, interpretable condition trees and processes informal travel orders through a deterministic pricing pipeline\. By strictly separating LLM\-based discrete selection from deterministic numeric execution, the system achieves zero fatal pricing errors across thousands of production orders, while reducing policy onboarding time from twenty minutes to one minute without developer involvement\. In a six\-month deployment at a municipal, state\-owned tourism enterprise, the system processed 3,960 orders and delivered substantial operational gains, demonstrating that LLMs can be safely integrated into high\-stakes decision workflows when paired with structured, auditable reasoning interfaces\. The framework generalizes to other policy\-driven domains where interpretability, reliability, and non\-expert usability are equally critical\.

## Limitations

Our current deployment relies on a manual grouping strategy that merges policies with similar customer scope and validity patterns into shared condition trees, in order to limit the number of trees per product\. As the system runs longer and policies accumulate, this grouping may degrade, increasing matching ambiguity and operator burden\. Developing automatic tree merging and splitting strategies is a natural direction for future work\.

Nearly half of production orders \(47\.9%\) require L3\-a resource edits, reflecting the domain\-specific gap between the resources stated in the order and the itinerary actually executed\. While the system surfaces this mismatch for operator correction, reducing this rate—through richer order input formats or multi\-turn clarification with coordinators—would further lower the human workload\.

Our correctness metrics in Track 2 are computed on an information\-consistent subset of 1,349 orders where composition and tree selection are pre\-resolved\. End\-to\-end accuracy on the full order set, including undecidable cases where the input is genuinely ambiguous, remains difficult to evaluate without additional ground\-truth annotations\. Furthermore, all experiments are conducted at a single tourism deployment site\. Although the architecture is designed to be domain\-agnostic, empirical validation in other policy\-driven domains such as insurance or regulatory compliance is needed to confirm generalizability\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p4.1)\.
- C\. Agarwal, S\. H\. Tanneru, and H\. Lakkaraju \(2024\)Faithfulness vs\. plausibility: on the \(un\)reliability of explanations from large language models\.ArXivabs/2402\.04614\.External Links:[Link](https://api.semanticscholar.org/CorpusID:267523276)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p4.1)\.
- W\. Chen, X\. Ma, X\. Wang, and W\. W\. Cohen \(2022\)Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks\.Trans\. Mach\. Learn\. Res\.2023\.External Links:[Link](https://api.semanticscholar.org/CorpusID:253801709)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p5.1)\.
- M\. Desmond, E\. Duesterwald, V\. Isahagian, and V\. Muthusamy \(2022\)A no\-code low\-code paradigm for authoring business automations using natural language\.ArXivabs/2207\.10648\.External Links:[Link](https://api.semanticscholar.org/CorpusID:250917225)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p3.1)\.
- L\. Ding, A\. Zhao, F\. Ye, Z\. Chen, and X\. Shen \(2026\)From llms to lrms: rethinking pruning for reasoning\-centric models\.arXiv preprint arXiv:2601\.18091\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p4.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2022\)PAL: program\-aided language models\.ArXivabs/2211\.10435\.External Links:[Link](https://api.semanticscholar.org/CorpusID:253708270)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p5.1)\.
- D\. Hendrycks, C\. Burns, A\. Chen, and S\. Ball \(2021\)CUAD: an expert\-annotated nlp dataset for legal contract review\.ArXivabs/2103\.06268\.External Links:[Link](https://api.semanticscholar.org/CorpusID:232170369)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p3.1)\.
- D\. Kaushik, S\. Gupta, C\. Raju, R\. A\. Dias, and S\. Ghosh \(2017\)Making travel smarter: extracting travel information from email itineraries using named entity recognition\.InRecent Advances in Natural Language Processing,External Links:[Link](https://api.semanticscholar.org/CorpusID:29894232)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p1.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p4.1)\.
- Y\. Liu, D\. Zhu, Z\. Al\-Khalili, D\. Cheng, Y\. Chen, D\. Klakow, W\. Zhang, and X\. Shen \(2025\)PricingLogic: evaluating llms reasoning on complex tourism pricing tasks\.ArXivabs/2510\.12409\.External Links:[Link](https://api.semanticscholar.org/CorpusID:282064405)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p1.1)\.
- M\. Mathew, D\. Karatzas, R\. Manmatha, and C\. V\. Jawahar \(2020\)DocVQA: a dataset for vqa on document images\.2021 IEEE Winter Conference on Applications of Computer Vision \(WACV\),pp\. 2199–2208\.External Links:[Link](https://api.semanticscholar.org/CorpusID:220280200)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p2.1)\.
- H\. Su, X\. Zhou, H\. Yu, X\. Shen, Y\. Chen, Z\. Zhu, Y\. Yu, and J\. Zhou \(2022\)Welm: a well\-read pre\-trained language model for chinese\.arXiv preprint arXiv:2209\.10372\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p4.1)\.
- Y\. Wang, Y\. Tian, X\. Shen, G\. Zhang, J\. Sun, H\. Zhang, R\. Xu, and F\. Zhao \(2025\)Fault2Flow: an alphaevolve\-optimized human\-in\-the\-loop multi\-agent system for fault\-to\-workflow automation\.arXiv preprint arXiv:2511\.12916\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p5.1)\.
- S\. Xiong, Y\. Ihlamur, F\. Alican, and A\. O\. Yin \(2024\)Gptree: towards explainable decision\-making via llm\-powered decision trees\.arXiv preprint arXiv:2411\.08257\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p5.1)\.
- F\. Xu, Q\. Hao, Z\. Zong, J\. Wang, Y\. Zhang, J\. Wang, X\. Lan, J\. Gong, T\. Ouyang, F\. Meng,et al\.\(2025\)Towards large reasoning models: a survey of reinforced reasoning with large language models\.arXiv preprint arXiv:2501\.09686\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p4.1)\.
- Y\. Xu, M\. Li, L\. Cui, S\. Huang, F\. Wei, and M\. Zhou \(2019\)LayoutLM: pre\-training of text and layout for document image understanding\.Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining\.External Links:[Link](https://api.semanticscholar.org/CorpusID:209515395)Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p2.1)\.
- R\. Zhou, W\. Hua, L\. Pan, S\. Cheng, X\. Wu, E\. Yu, and W\. Y\. Wang \(2025\)Rulearena: a benchmark for rule\-guided reasoning with llms in real\-world scenarios\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 550–572\.Cited by:[§1](https://arxiv.org/html/2608.26124#S1.p1.1)\.