TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs

# TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs
Source: [https://arxiv.org/html/2608.00640](https://arxiv.org/html/2608.00640)
Jin Zhang1,2, Linyu Li3, Weili Jiang4, Yuqing Cai1, Yutong Liu1Guanquecairang1, Yongbin Yu1, Jingye Cai1, Nyima Tashi2, Gadeng Luosang21University of Electronic Science and Technology of China2Tibet University3Peking University4Southwest Jiaotong University

###### Abstract

Large language models are increasingly viewed as a potential means of mitigating global health inequities, yet their outputs often reflect dominant high\-resource medical traditions and provide limited coverage of traditional medical knowledge systems\. Tibetan medicine, one of the world’s four major traditional medical systems, has an independent and highly structured theoretical framework\. When models lack grounded understanding of Tibetan medicine, they may fall back on dominant epistemic systems and distort the native knowledge structure during reasoning\. However, quantitative tools for evaluating cultural bias in Tibetan medicine remain largely absent\. To address this gap, we introduce TreeProbe, the first cultural\-bias benchmark organized around the native Tree of Medicine framework in Tibetan medicine\. It contains 4,719 expert\-adjudicated items covering 467 diseases and 10 subtasks along the three roots\. Experiments on representative LLMs show that current models remain limited in native Tibetan medical contexts and exhibit systematic external ontology drift\. Further analysis reveals that models diverge in whether they drift toward biomedical or TCM reasoning, shaped by pretraining data composition and surface resemblance between TCM and Tibetan medicine\. TreeProbe provides a diagnostic benchmark for developing medical AI systems that are both linguistically inclusive and epistemically fair\. Code and data are available in an anonymous repository at[https://anonymous\.4open\.science/r/TreeProbe/](https://anonymous.4open.science/r/TreeProbe/)\.

TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs

Jin Zhang1,2, Linyu Li3, Weili Jiang4, Yuqing Cai1, Yutong Liu1Guanquecairang1, Yongbin Yu1, Jingye Cai1, Nyima Tashi2, Gadeng Luosang21University of Electronic Science and Technology of China2Tibet University3Peking University4Southwest Jiaotong University

## 1Introduction

The World Health Organization \(WHO\) has repeatedly highlighted that more than half of the global population still lacks adequate access to essential health services\(World Health Organization,[2023](https://arxiv.org/html/2608.00640#bib.bib19)\), while also advocating for a stronger integration of traditional medicine into healthcare systems\(World Health Organization,[2025](https://arxiv.org/html/2608.00640#bib.bib21)\)\. Against this backdrop, traditional medicine should not be viewed as a peripheral supplement outside formal healthcare systems, but rather as a critical source of everyday care, health knowledge, and social trust for many underserved populations\(World Health Organization,[2024](https://arxiv.org/html/2608.00640#bib.bib20)\)\. Tibetan medicine, recognized alongside Traditional Chinese Medicine, ancient Indian medicine, and ancient Arabic medicine as one of the world’s four major traditional medical systems, has long served cross\-regional populations across China, Mongolia, Bhutan, Nepal, India, and surrounding regions\. Over centuries of development, it has formed an independent and internally coherent medical epistemology\(Zhaoet al\.,[2019](https://arxiv.org/html/2608.00640#bib.bib22)\)\.

Large language models \(LLMs\) have recently been regarded as a promising strategy for mitigating healthcare inequality\(Chenet al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib26); Rodriguezet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib27)\)\. However, existing studies suggest that contemporary LLMs are predominantly trained on Western\-centric or high\-resource corpora\(Taoet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib28); Ochienget al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib29)\), causing them to underrepresent localized cultural contexts\(Nguyenet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib24); Chiuet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib25)\)and region\-specific medical knowledge\.This issue becomes particularly pronounced in Tibetan medicine due to its highly distinctive theoretical framework\. When LLMs lack grounded understanding of Tibetan medical knowledge, they tend to default to dominant external epistemological systems, most notably biomedicine or Traditional Chinese Medicine \(TCM\)\. Such behavior distorts the native reasoning structure of Tibetan medicine during inference and further reinforces existing hierarchies of knowledge and cognition, marginalizing already underrepresented medical traditions\(Mohamedet al\.,[2020](https://arxiv.org/html/2608.00640#bib.bib23)\)\. Unchecked cultural bias may therefore exacerbate healthcare inequities, transforming LLMs into mechanisms through which dominant medical paradigms overwrite or replace vulnerable knowledge systems\(Pfohlet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib30)\)\. Addressing this challenge requires rigorous evaluation frameworks capable of quantifying and constraining cultural bias\. Standardized quantitative evaluation enables abstract biases to be transformed into measurable, comparable, and traceable indicators, providing a foundation for model diagnosis, bias attribution, and alignment\(Beanet al\.,[2026](https://arxiv.org/html/2608.00640#bib.bib31)\)\. Nevertheless, systematic evaluation of cultural bias in Tibetan medicine remains largely absent\.

To address this gap, we adopt the native structure of Tibetan medicine, the Medical Tree, as our evaluation architecture, mapping its three roots — Physiopathology, Diagnosis, and Therapy — onto evaluation dimensions\. This self\-contained system organizes, along its three roots, a complete account of bodily structure, disease mechanisms, and clinical intervention within Tibetan medicine\. Building upon this structure, we introduce TreeProbe, a benchmark specifically designed to evaluate cultural bias in Tibetan medicine\. Through TreeProbe, we pursue two complementary goals: \(i\) objectively assessing whether state\-of\-the\-art LLMs can understand and reason within the native epistemological context of Tibetan medicine, and \(ii\) systematically characterizing the mechanisms through which cultural bias emerges across medical ontologies, including the sources and directions of bias\. In summary, our contributions are threefold:

\(1\) We introduce TreeProbe, the first cultural\-bias benchmark grounded in the native Medical Tree of Tibetan medicine, comprising 4,719 expert\-adjudicated items over 467 diseases, three roots, and 10 subtasks\.

\(2\) We propose a reusable construction paradigm for low\-resource traditional medicine, using structured knowledge units and operators to derive items, with cross\-system distractors authored by domain physicians from ontology\-neutral clinical vignettes\.

\(3\) We systematically evaluate representative LLMs on TreeProbe, characterizing the practical capabilities of existing models in low\-resource traditional medical settings and revealing concrete sources of cultural bias\.

## 2Related Work

Medical LLM benchmarks primarily evaluate clinical knowledge, factuality, safety, and patient\-facing interaction quality\(Singhalet al\.,[2023](https://arxiv.org/html/2608.00640#bib.bib36); Zhanget al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib37); Wanget al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib38)\), but they largely take biomedicine as the default medical frame\. Existing evaluations of traditional medicine, mostly measure in\-system task performance\(Chenget al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib39); Yueet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib42)\), without asking whether a model preserves the ontology of the target system when competing medical paradigms are also available\. Work on cultural bias has shown that LLMs often reflect high\-resource languages and dominant cultural assumptions, falling back to Western norms in cross\-regional or cross\-value settings\(Naouset al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib32); Myunget al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib17); Adilazuardaet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib40); AlKhamissiet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib41)\); however, these evaluations typically concern general cultural knowledge, social norms, or values rather than expert knowledge systems\. TreeProbe fills this gap by grounding cultural\-bias evaluation in Tibetan medicine\.

## 3TreeProbe Benchmark Structure

![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/main.png)Figure 1:TreeProbe construction pipeline\.TreeProbe is organized along the Medical Tree, the native ontological architecture of Tibetan medicine codified in the*Four Tantras*, which arranges medical knowledge into three roots—physiopathology, diagnosis, and therapy—spanning the full chain of clinical reasoning\. We instantiate ten subtasks under these three roots, covering both multiple\-choice questions \(MCQs\) and generative question answering \(QA\)\.

##### Physiopathology\.

This root targets the theoretical foundation of Tibetan medicine, probing both the recognition of core concepts and the reconstruction of native causal reasoning chains\. It comprises Knowledge Probing \(KP\) and Pathological Reasoning \(PR\)\.

##### Diagnosis\.

In Tibetan medicine, diagnosis centers on the condition of the three humors:Rlung,Tripa, andBeken\. Clinical signs are not treated as isolated symptoms, but are interpreted as evidence of humoral imbalance and then linked to a disease judgment within the Tibetan medical framework\. This root therefore examines whether models can follow the diagnostic sequence from clinical evidence to three\-humor assessment and disease identification\. It includes Three\-Humor Reasoning \(THR\), Disease Diagnosis \(DD\), and Complex Diagnostic Reasoning \(CDR\)\.

##### Therapy\.

Tibetan therapeutic principles aim to restore humoral balance through the coordinated use of four intervention categories—diet, lifestyle, medication, and external therapy\. This root evaluates adherence to these principles within each category and the ability to integrate them into a coherent plan\. It comprises Dietary Intervention \(DI\), Lifestyle Regulation \(LR\), Medication Prescription \(MP\), External Therapy \(ET\), and the integrative Comprehensive Treatment Plan \(CTP\)\.

## 4Dataset Construction

TreeProbe is constructed in four stages\(Figure[1](https://arxiv.org/html/2608.00640#S3.F1)\): extracting atomic knowledge units from authoritative Tibetan medical sources, deriving items with a fixed operator family, eliciting cross\-system distractors from domain experts, and adjudicating each item under a label\-blind protocol\.

### 4\.1Operational Definitions of Option Categories

Each MCQ contains four mutually exclusive option categories and is retained only when their boundaries are defensible under the Tibetan medical ontology\. The Faithful Answer \(FA\) is the evidence\-supported answer under Tibetan medical reasoning\. An Intra\-Tibetan Distractor \(ITD\) uses related Tibetan medical terms or adjacent concepts but mismatches the relation required by the stem, capturing Tibetan\-internal reasoning failures\. A TCM\-drift Distractor \(TCD\) is plausible under TCM but incorrect under Tibetan theory, testing substitution with the neighboring TCM ontology\. A Biomedical\-drift Distractor \(BID\) aligns with biomedical classification or practice but is incorrect in the Tibetan context, testing default biomedical reasoning\.

### 4\.2Stage I: Evidence\-Linked Knowledge Unit Extraction

The source corpus is the*Four Tantras*, foundational to Tibetan medical theory and clinical practice, with the national standard "Classification and Codes of Diseases of Tibetan Medicine" \(GB/T 46946–2025\) serving as the disease anchor\. Extraction is carried out by eight native Tibetan\-speaking annotators with at least a bachelor’s degree in Tibetan medicine, after training and qualification testing\. A fully instantiated knowledge unit is shown in Appendix Table[5](https://arxiv.org/html/2608.00640#A2.T5)\.

Each knowledge unit is a quadrupleu=⟨s,e,d,𝐚⟩u=\\langle s,e,d,\\mathbf\{a\}\\rangle, wheresspoints to the source passage,eeis the set of precise evidence spans,ddis the GB/T 46946\-2025 disease code, and𝐚\\mathbf\{a\}is a structured attribute vector\. All Tibetan medical terms appearing ineeand𝐚\\mathbf\{a\}are normalized during annotation\.

The attribute vector𝐚\\mathbf\{a\}contains 17 slots covering the full Tibetan medical trajectory, including etiology, pathogenesis, diagnostic evidence, therapeutic principles, and treatment modalities\. This partition follows the native clinical knowledge structure of Tibetan medicine\. Slots not covered by the source are explicitly recorded as empty, preventing operators from fabricating unsupported content\.

Three licensed Tibetan medicine physicians adjudicate each candidate unit against its evidence spansee, discarding units with insufficient evidence or ambiguous interpretation\. After filtering, 467 of 506 candidate disease units are retained\.

### 4\.3Stage II: Operator\-Guided Item Instantiation and Drafting

#### 4\.3\.1Operator\-Driven Item Derivation

We treat each knowledge unituuas a structured object queried by a family of probing operators, where each operator declares its input and output over the attribute vector𝐚\\mathbf\{a\}asop:⟨𝒫req,𝒫opt⟩→τ\\operatorname\{op\}:\\langle\\mathcal\{P\}\_\{\\mathrm\{req\}\},\\,\\mathcal\{P\}\_\{\\mathrm\{opt\}\}\\rangle\\rightarrow\\tau, where𝒫req\\mathcal\{P\}\_\{\\text\{req\}\}and𝒫opt\\mathcal\{P\}\_\{\\text\{opt\}\}are the required and optional premise slots \(disjoint\), andτ\\tauis the target slot\. Two properties are enforced by design: \(i\) an item exposes the slots declared by its operator, so difficulty arises from reasoning rather than information leakage; and \(ii\) the target serves as a single semantic focus in each item, so distractors attach to a common locus\. An operator is instantiated only when the target slotτ\\tauand all required premise slots in𝒫req\\mathcal\{P\}\_\{\\text\{req\}\}are explicitly supported by evidence spans; otherwise the\(u,op\)\(u,\\text\{op\}\)is skipped\. Optional slots are sampled only from evidence\-supported non\-empty slots\.

Given a knowledge unituuand operatorop, the realized premise set of an item is𝒫=𝒫req∪S\\mathcal\{P\}=\\mathcal\{P\}\_\{\\text\{req\}\}\\cup S, where each slot in𝒫opt\\mathcal\{P\}\_\{\\text\{opt\}\}is independently exposed with probabilityp=0\.5p=0\.5to formSS\. This design mirrors the partial observability of real clinical settings and yields premise\-combination variants of the same operator across the 467 knowledge units, preventing models from exploiting fixed templates\. TreeProbe defines ten subtasks across Physiopathology, Diagnosis, and Therapy roots; each subtask corresponds to one operator \(see Appendix[A](https://arxiv.org/html/2608.00640#A1)\)\.

#### 4\.3\.2LLM Drafting

For each \(knowledge unituu, operatorop\) pair, the item instantiation procedure in §[4\.3\.1](https://arxiv.org/html/2608.00640#S4.SS3.SSS1)*first*samples the realized premise set𝒫\\mathcal\{P\}and targetτ\\tau; GPT\-5\.4*then*drafts the stem and the Faithful Answer \(FA\) for MCQ items, or the reference answer for QA items, over this fixed𝒫\\mathcal\{P\}andτ\\tau, following the prompt in Appendix[E](https://arxiv.org/html/2608.00640#A5)\. The prompt exposes only the slot values in𝒫\\mathcal\{P\}together with the corresponding Tibetan evidence spanee, and forbids introducing content beyondee\.

Annotators then perform two checks per item againsteeand𝒫\\mathcal\{P\}: \(i\) factual entailment—drafts introducing content not entailed byeeare discarded; and \(ii\) Tibetan linguistic quality—terminology, syntax, and case\-marking are revised to conform to written Tibetan medical convention\. The knowledge content and linguistic quality of every retained item are determined by human curation\.

### 4\.4Stage III: Cross\-Ontology Distractor Authoring by Experts

The three non\-faithful options of each MCQ are authored by human experts; the LLM is excluded to prevent the self\-preference bias that would arise if the same model drafted both the Faithful Answer and the distractors\. The three distractor types are produced by separately recruited expert groups\.

##### ITD\.

A Tibetan medicine expert, after viewing knowledge unituu, the stem, and the Faithful Answer, constructs an option phrased in Tibetan medical terminology that violates the relation required by the operator\. The expert substitutes the target slot value with an adjacent value in the slot \(e\.g\., a neighboring three\-humor state\) or borrows a value from a non\-target slot \(e\.g\., presenting a tongue\-diagnosis cue in place of a pulse\-diagnosis cue\)\.

##### TCD and BID\.

Authoring proceeds in three steps\. \(i\) A Tibetan medicine expert transcribes each stem into an*ontology\-neutral clinical vignette*in Chinese, removing Tibetan\-specific systemic terms and retaining only clinical content identifiable across medical ontologies\. \(ii\) Three licensed TCM physicians and three licensed biomedical physicians independently read the Chinese vignette and provide the most likely answer under their respective ontologies, with no exchange between the two groups and no access to the Faithful Answer\. \(iii\) The annotators in §[4\.2](https://arxiv.org/html/2608.00640#S4.SS2)translate the TCD and BID answers into Tibetan and return the translations to the original authors for fidelity verification against the Chinese source\.

### 4\.5Stage IV: Filtering, Adjudication, and Dataset Statistics

#### 4\.5\.1Data Filtering

##### \(i\) Ontological separability\.

Items in which the Faithful Answer and any distractor are jointly valid under the Tibetan medical ontology are discarded\.

##### \(ii\) Cross\-category exclusivity\.

TCD and BID options are checked for category leakage—options simultaneously valid under both ontologies—to preserve clean ontological boundaries among the four option categories\.

##### \(iii\) Surface shortcuts\.

Items whose answers can be inferred from option length, lexical overlap with the stem, or register differences are discarded\(Sugawaraet al\.,[2018](https://arxiv.org/html/2608.00640#bib.bib9); Parrishet al\.,[2022](https://arxiv.org/html/2608.00640#bib.bib8)\); the stem is additionally checked for unintended entailment of the target value, in which case the item is dropped\.

##### \(iv\) Clinical safety\.

Items lacking contextual qualification that could be misread as executable clinical advice or that may induce high\-risk interventions are removed\.

#### 4\.5\.2Expert Adjudication

Items that pass filtering are adjudicated by a panel of five independent experts: three Tibetan medicine physicians, one TCM physician, and one biomedical physician\. For MCQ items, experts perform two label\-blind annotations: \(i\) selecting the Faithful Answer, and \(ii\) assigning a category label \(ITD / TCD / BID\) to each non\-faithful option\. For generative QA items, experts assess whether the reference answer is fully supported by the evidence spans, preserves Tibetan medical reasoning, and contains no unsupported TCM or biomedical substitution\. An item is included in the final dataset only when at least four of the five experts agree on the corresponding adjudication criteria\.

The resulting TreeProbe benchmark contains 4,719 items: 3,289 MCQs and 1,430 generative QA instances, spanning ten subtasks \(Table[1](https://arxiv.org/html/2608.00640#S4.T1)\)\.

RootSubtaskFormat\# ItemsAvg\. Len\.PhysiopathologyKPMCQ49316\.6PRQA45051\.3DiagnosisTHRMCQ490107\.7DDMCQ70156\.0CDRQA49032\.2TherapyDIMCQ49183\.6LRMCQ498102\.3MPMCQ26040\.2ETMCQ356127\.5CTPQA49020\.4Total4,71960\.6Table 1:Dataset statistics after expert adjudication\. Avg\. Len\. denotes average stem length in tokens\.

## 5Experimental Setup

### 5\.1Model Selection

We evaluate six representative LLMs on TreeProbe, covering frontier models from both international and Chinese providers, with parameter scales ranging from lightweight to several hundred billion\. The evaluated models include GPT\-5\.4\(Achiamet al\.,[2023](https://arxiv.org/html/2608.00640#bib.bib10)\), Claude Sonnet 4\.6\(Anthropic,[2026](https://arxiv.org/html/2608.00640#bib.bib15)\), and Gemini 3\.1 Flash\-Lite\(Teamet al\.,[2023](https://arxiv.org/html/2608.00640#bib.bib12)\)from international providers, alongside GLM\-4\.7\(Zenget al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib11)\), DeepSeek\-V3\.2\-Chat\(Liuet al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib13)\), and Qwen3\.5\-397B\-A17B\(Yanget al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib14)\)from Chinese providers\.

All models are evaluated under zero\-shot and one\-shot conditions; the one\-shot exemplar is fixed per subtask and drawn from a held\-out pool disjoint from the evaluation set\. MCQ option order is randomized to mitigate position bias\. Prompts use English while task content remains in Tibetan; prompt templates are provided in Appendix[F](https://arxiv.org/html/2608.00640#A6)\.

### 5\.2Quantitative Evaluation

For MCQs, we report accuracy as the primary metric and log each incorrect response by the selected distractor category for later bias attribution\. For open\-ended QA, GPT\-5\.4 scores each response on a 1–100 scale using a unified five\-dimensional rubric: Accuracy, factual consistency with authoritative Tibetan medical sources; Helpfulness, clinical and educational utility; Linguistic Quality, Tibetan grammar, terminology, and writing conventions; Ontology Fidelity, consistency with Tibetan medical reasoning without TCM or biomedical substitution; and Completeness, coverage of the required answer points\.

### 5\.3Cultural Bias Metrics

We divide MCQ errors into two categories: cross\-ontology drift toward TCM or biomedicine, and intra\-Tibetan confusion\. The former reflects cultural bias, whereas the latter reflects gaps in domain knowledge\. To quantify the magnitude and direction of drift, we define two log\-odds metrics\. LetNTCDN\_\{\\text\{TCD\}\},NBIDN\_\{\\text\{BID\}\},NITDN\_\{\\text\{ITD\}\}, andNFAN\_\{\\text\{FA\}\}denote the counts of TCD, BID, ITD, and FA selections by modelMMon sliceDD, respectively\.

Ontology Drift Log\-Odds \(ODLO\)measures the overall departure from the Tibetan ontology; the denominator represents intra\-Tibetan distractor, withODLO\>0\\text\{ODLO\}\>0indicating drift toward external systems:

ODLO​\(M,D\)=log⁡NTCD\+NBID\+αNITD\+α,\\text\{ODLO\}\(M,D\)=\\log\\frac\{N\_\{\\text\{TCD\}\}\+N\_\{\\text\{BID\}\}\+\\alpha\}\{N\_\{\\text\{ITD\}\}\+\\alpha\},\(1\)
Biomedical–TCM Drift Log\-Odds \(BTD\-LO\)characterizes the direction of external drift, withBTD\-LO\>0\\text\{BTD\-LO\}\>0indicating bias toward biomedicine andBTD\-LO<0\\text\{BTD\-LO\}<0toward TCM:

BTD\-LO​\(M,D\)=log⁡NBID\+αNTCD\+α\.\\text\{BTD\-LO\}\(M,D\)=\\log\\frac\{N\_\{\\text\{BID\}\}\+\\alpha\}\{N\_\{\\text\{TCD\}\}\+\\alpha\}\.\(2\)α=0\.5\\alpha=0\.5is a smoothing constant added to prevent logarithmic singularities at zero counts\.

## 6Results and Analysis

### 6\.1How well do LLMs understand Tibetan medicine?

Table 2:MCQ accuracy \(%\) on TreeProbe under zero\-shot and one\-shot prompting\. Bold marks the best score per subtask within each setting; underline marks the second\-best score\. Avg denotes the unweighted \(macro\) mean across subtasks\.##### MCQ results\.

Frontier LLMs remain limited on Tibetan medical tasks\(Table[2](https://arxiv.org/html/2608.00640#S6.T2)\)\. The strongest zero\-shot model, Qwen3\.5\-397B\-A17B, averages only 60\.55%, and Claude Sonnet 4\.6 only 40\.61%\. Performance is also clearly uneven across the three roots\. At the aggregate level, therapy subtasks are easier than diagnosis and physiopathology\. We interpret this gap in terms of uneven transfer, where therapy items are more amenable to partial transfer through procedural templates and surface vocabulary shared with biomedical and TCM corpora, whereas diagnosis and physiopathology depend more heavily on Tibetan medicine’s own ontology, conceptual system, and diagnostic reasoning, which external medical knowledge does not cover\(Petrovet al\.,[2023](https://arxiv.org/html/2608.00640#bib.bib16); Myunget al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib17)\)\.

A single in\-context demonstration helps most models, but the gains are uneven\. GPT\-5\.4 improves the most, while Claude regresses on three subtasks\. Consistent with prior findings, in\-context demonstrations may align models with the format, but they do not necessarily correct the ontology gap in Tibetan medical reasoning\(Minet al\.,[2022](https://arxiv.org/html/2608.00640#bib.bib18)\)\.

![Refer to caption](https://arxiv.org/html/2608.00640v1/x1.png)Figure 2:Open\-ended QA scores on the three generation subtasks \(PR, CDR, CTP\) under zero\-shot and one\-shot prompting\. Scores are LLM\-judge ratings in\[0,1\]\[0,1\]averaged across items\.
##### QA results\.

QA performance is substantially lower than MCQ performance \(Figure[2](https://arxiv.org/html/2608.00640#S6.F2)\): the strongest zero\-shot model averages only 0\.47 \(Gemini 3\.1 Flash\-Lite\), while the strongest one\-shot model reaches 0\.55 \(GPT\-5\.4\)\. PR is consistently the hardest subtask across models\. Decomposing model performance across the five evaluation dimensions reveals a pronounced imbalance, as shown in Appendix[C](https://arxiv.org/html/2608.00640#A3): Linguistic Quality reaches 0\.83–0\.89, while the four content dimensions—Accuracy, Helpfulness, Ontology Fidelity, and Completeness—cluster between 0\.20 and 0\.50; except for Claude, the gap exceeds 0\.50 for every model\. Models can generate fluent and grammatical Tibetan, yet the content remains often underspecified, inaccurate, and insufficiently ontology\-faithful\. One\-shot prompting yields modest gains on the content dimensions, while Linguistic Quality remains largely unchanged\.

### 6\.2Where do model errors drift?

![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/bias_scatter.png)Figure 3:Per\-model MCQ bias patterns\. ODLO \(x\-axis\) measures ontology drift away from Tibetan medicine; BTD\-LO \(y\-axis\) measures the relative tilt toward biomedical drift \(positive\) or TCM drift \(negative\)\.We restrict bias analysis to MCQ items\. Figure[3](https://arxiv.org/html/2608.00640#S6.F3)plots every model in the ODLO–BTD\-LO plane\. Two structural patterns stand out\. First, every point sits in the right half \(ODLO\>0\.48\\text\{ODLO\}\>0\.48\), and the “Tibetan confusion” quadrants are empty: errors consistently take the form of ontology drift toward an external medical system\. Second, models separate along the BTD\-LO axis\. Claude Sonnet 4\.6 and GPT\-5\.4 sit above zero \(biomedical\-leaning\); GLM\-4\.7, DeepSeek\-Chat, and Gemini sit below it \(TCM\-leaning\); Qwen3\.5 lies near the axis\. This separation stems from two sources\. The first is differences in the cultural composition and coverage of pretraining corpora\(Naouset al\.,[2024](https://arxiv.org/html/2608.00640#bib.bib32)\)\. The second is an asymmetry in surface semantic similarity between the two external systems: Tibetan and Chinese medical concepts exhibit substantial surface semantic similarity, so that TCM categories often look superficially plausible as renderings of Tibetan ones, even when the underlying clinical logic differs\. Biomedicine offers no comparable surface alignment with Tibetan medical vocabulary\. This asymmetric similarity lowers the cost of TCM\-style substitution and explains why TCM drift appears even in models without disproportionate TCM exposure \(Gemini in particular\)\.

Two models occupy the extremes\. Claude has the largest ontology drift \(ODLO=0\.91\\text\{ODLO\}=0\.91\) and the strongest biomedical lean \(BTD\-LO=\+0\.17\\text\{BTD\-LO\}=\+0\.17\); these patterns indicate its low MCQ accuracy stems from systematic biomedical reframing\. GLM\-4\.7 carries an unusually strong TCM lean \(BTD\-LO=−0\.52\\text\{BTD\-LO\}=\-0\.52\), more than twice that of any other model\. One\-shot prompting consistently pulls models toward the origin without flipping their direction, dampening the cultural component of the error while leaving the underlying drift intact\. A model\-wise breakdown is provided in Appendix[D](https://arxiv.org/html/2608.00640#A4)\.

### 6\.3Do models drift beyond knowledge gaps?

![Refer to caption](https://arxiv.org/html/2608.00640v1/x2.png)\(a\)Probe 1\.
![Refer to caption](https://arxiv.org/html/2608.00640v1/x3.png)\(b\)Probe 2\.

Figure 4:Choice distributions for two controlled diagnostic tests on items each model correctly answered\.Probe 1\(left\): items offer internally correct answers from Tibetan medicine, TCM, and biomedicine for the same disease\.Probe 2\(right\): the option in each model’s drift direction becomes a clinically wrong answer \(hatched segments\)\.Table 3:Meta\-evaluation of GPT\-5\.4 as QA judge\. H–H: pairwise agreement among three Tibetan medicine experts; GPT–H: GPT\-5\.4 vs\. expert consensus\. Spearman’sρ\\rho: rank\-level alignment; QWK: Quadratic Weighted Kappa over ordinal scores\.Δ\\Delta: H–H minus GPT–H\. Bold/underline mark smallest/second\-smallest absolute gaps\.To examine whether the drift of model errors toward external medical systems reflects a template\-like preference for external paradigms, we design two controlled diagnostic tests\. Both tests are restricted to items that each model answered correctly in the original MCQ setting, and are sampled only from four subtasks \(DI, LR, MP, and ET\)\. Within this candidate set, we retain only items for which Tibetan, TCM, and biomedical expert groups confirmed comparable treatment options across the three systems\. Candidate items without consensus on cross\-system comparability are excluded\.

The first controlled diagnostic test examines whether models still depart from the Tibetan context after demonstrating access to the Tibetan answer \(Figure[4\(a\)](https://arxiv.org/html/2608.00640#S6.F4.sf1)\)\. Each item presents three system\-specific correct answers for the same disease: one from Tibetan medicine, one from TCM, and one from biomedicine\. Because the prompt contains clear Tibetan medical cultural cues and the model has already answered these items correctly in the original MCQ setting, the Tibetan\-answer retention rate should approach 100%; any external choice indicates a departure from the target ontology under an explicit Tibetan medical context\. However, all models except DeepSeek\-Chat and Claude Sonnet 4\.6 retain Tibetan answers at 72–85%; the two exceptions fall below this range\. Specifically, the external choices of GLM\-4\.7, Gemini 3\.1 Flash\-Lite, and Qwen3\.5\-397B\-A17B shift mainly to TCM \(13\.8%, 14\.7%, and 15\.2%, respectively\), whereas DeepSeek\-Chat and Claude Sonnet 4\.6 split their external choices roughly evenly between TCM and biomedical answers \(26\.7%/23\.0% and 21\.0%/20\.9%, respectively\)\. DeepSeek\-Chat shows the strongest departure: it selects the Tibetan answer in only 50% of zero\-shot cases and is the only model that moves further away from the Tibetan context under one\-shot prompting \(44%\)\.

The second controlled diagnostic test further examines whether models continue to follow their preferred external paradigm when that paradigm is made incorrect \(Figure[4\(b\)](https://arxiv.org/html/2608.00640#S6.F4.sf2)\)\. We replace each model’s preferred external option according to its drift direction with a clinically incorrect answer from the same system, drawn from the original MCQ distractor pool\. The two bias directions differ in their persistence\. For biomedical\-leaning models, even after the biomedical option is replaced with a clinically incorrect answer from the same system, GPT\-5\.4 and Claude Sonnet 4\.6 still select this incorrect option in 5\.8%/3\.8% and 10\.8%/6\.0% of cases, respectively \(Biomedical \(wrong\)\)\. The pattern is more pronounced for TCM\-leaning models: Gemini 3\.1 Flash\-Lite, GLM\-4\.7, and Qwen3\.5\-397B\-A17B still select the TCM option after it has been made incorrect in fewer than 12% of cases \(TCM\(Wrong\)\), while DeepSeek\-Chat reaches 40\.5% under zero\-shot and 33\.9% under one\-shot prompting, exceeding its rate of choosing the correct Tibetan answer \(24\.0%\)\. This suggests that some models are not primarily comparing system\-internal correctness, but instead continue to follow their external preference under multi\-ontology competition; when an external system is more easily confounded with Tibetan medicine at the surface level, such incorrect substitutions become harder to detect\.

### 6\.4Reliability of the LLM Judge

To validate GPT\-5\.4 as an auxiliary scorer for generative QA, we conduct a meta\-evaluation on 882 model responses evenly sampled from the three QA subtasks, with three Tibetan medicine experts independently scoring each response using the five\-dimensional rubric\. We compare Human–Human agreement, measured as pairwise expert agreement, and GPT–Human agreement, measured against the expert consensus, using Spearman’sρ\\rhofor rank\-level alignment and Quadratic Weighted Kappa \(QWK\) for ordinal score agreement\.

Rubric\-based evaluation of Tibetan medical QA involves non\-trivial subjectivity, and expert judgments are not expected to be perfectly identical\(Aroraet al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib35)\)\. As shown in Table[3](https://arxiv.org/html/2608.00640#S6.T3), GPT–Human agreement is close to Human–Human agreement on Accuracy, Helpfulness, Linguistic Quality, and Completeness, indicating that GPT\-5\.4 captures expert\-aligned aggregate trends on general\-quality dimensions\. Ontology Fidelity shows a larger gap, reflecting the difficulty of judging fine\-grained fidelity to Tibetan medical ontology; nevertheless, its GPT–Human agreement remains in a similar moderate range to expert agreement, suggesting that GPT\-5\.4 still provides a useful aggregate\-level scoring signal\. We therefore use GPT\-5\.4 as an auxiliary scorer for aggregate generative QA evaluation, rather than as a substitute for medical ground\-truth adjudication or expert judgment\(Zhenget al\.,[2023](https://arxiv.org/html/2608.00640#bib.bib33); Tanet al\.,[2025](https://arxiv.org/html/2608.00640#bib.bib34)\)\.

## 7Conclusion

We introduce TreeProbe, the first Tibetan\-medicine cultural\-bias benchmark, with 4,719 expert\-adjudicated items across 10 subtasks along three roots\. Experiments show current frontier LLMs’ errors drift systematically toward external medical ontologies, with substantial model variation in drift direction and rigidity\.

## Limitations

TreeProbe is grounded in authoritative Tibetan medical sources and the native Tree of Medicine framework\. As the first benchmark of this kind, its current coverage focuses on disease\-centered knowledge units that can be traced to source evidence and expert adjudication\. Since Tibetan medical knowledge may vary across regional practices, lineages, and clinical interpretations, future work can extend the benchmark to broader textual traditions and multimodal diagnostic evidence such as pulse and urine diagnosis\.

TreeProbe focuses on cross\-ontology drift from Tibetan medicine toward TCM or biomedicine, a central form of bias in Tibetan medical LLM evaluation, rather than an exhaustive account of all possible biases\. Future work can present the same items in Tibetan, Chinese, and English to test whether the language channel affects the direction or magnitude of drift, further disentangling ontology\-level bias from language\-mediated activation effects\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§5\.1](https://arxiv.org/html/2608.00640#S5.SS1.p1.1)\.
- M\. F\. Adilazuarda, S\. Mukherjee, P\. Lavania, S\. S\. Singh, A\. F\. Aji, J\. O’Neill, A\. Modi, and M\. Choudhury \(2024\)Towards measuring and modeling “culture” in llms: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15763–15784\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- Investigating cultural alignment of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12404–12422\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- Anthropic \(2026\)Introducing Claude Sonnet 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-6](https://www.anthropic.com/news/claude-sonnet-4-6)Accessed: 2026\-05\-22Cited by:[§5\.1](https://arxiv.org/html/2608.00640#S5.SS1.p1.1)\.
- R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel,et al\.\(2025\)Healthbench: evaluating large language models towards improved human health\.arXiv preprint arXiv:2505\.08775\.Cited by:[§6\.4](https://arxiv.org/html/2608.00640#S6.SS4.p2.1)\.
- A\. M\. Bean, R\. O\. Kearns, A\. Romanou, F\. S\. Hafner, H\. Mayne, J\. Batzner, N\. Foroutan Eghlidi, C\. Schmitz, K\. Korgul, H\. Batra,et al\.\(2026\)Measuring what matters: construct validity in large language model benchmarks\.Advances in Neural Information Processing Systems38\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- H\. Chen, D\. Zeng, Y\. Qin, Z\. Fan, F\. N\. Y\. Ci, D\. C\. Klonoff, J\. S\. Ji, S\. Zhang, K\. N\. Amissah\-Arthur, M\. M\. J\. de Tavárez,et al\.\(2025\)Large language models and global health equity: a roadmap for equitable adoption in lmics\.The Lancet Regional Health–Western Pacific63\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- Z\. Cheng, Y\. Lu, H\. Ye, Z\. Liu, M\. Wang, J\. Liu, Z\. Li, W\. Fan, Y\. Guo, R\. Fu,et al\.\(2025\)TCM\-eval: an expert\-level dynamic and extensible benchmark for traditional chinese medicine\.arXiv preprint arXiv:2511\.07148\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- Y\. Y\. Chiu, L\. Jiang, B\. Y\. Lin, C\. Y\. Park, S\. S\. Li, S\. Ravi, M\. Bhatia, M\. Antoniak, Y\. Tsvetkov, V\. Shwartz,et al\.\(2024\)CulturalBench: a robust, diverse and challenging benchmark on measuring \(the lack of\) cultural knowledge of llms\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§5\.1](https://arxiv.org/html/2608.00640#S5.SS1.p1.1)\.
- S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. Zettlemoyer \(2022\)Rethinking the role of demonstrations: what makes in\-context learning work?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 11048–11064\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.759/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.759)Cited by:[§6\.1](https://arxiv.org/html/2608.00640#S6.SS1.SSS0.Px1.p2.1)\.
- S\. Mohamed, M\. Png, and W\. Isaac \(2020\)Decolonial ai: decolonial theory as sociotechnical foresight in artificial intelligence\.Philosophy & Technology33\(4\),pp\. 659–684\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- J\. Myung, N\. Lee, Y\. Zhou, J\. Jin, R\. A\. Putri, D\. Antypas, H\. Borkakoty, E\. Kim, C\. Perez\-Almendros, A\. A\. Ayele,et al\.\(2024\)Blend: a benchmark for llms on everyday knowledge in diverse cultures and languages\.Advances in Neural Information Processing Systems37,pp\. 78104–78146\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1),[§6\.1](https://arxiv.org/html/2608.00640#S6.SS1.SSS0.Px1.p1.1)\.
- T\. Naous, M\. J\. Ryan, A\. Ritter, and W\. Xu \(2024\)Having beer after prayer? measuring cultural bias in large language models\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 16366–16393\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1),[§6\.2](https://arxiv.org/html/2608.00640#S6.SS2.p1.1)\.
- T\. Nguyen, S\. Razniewski, and G\. Weikum \(2024\)Cultural commonsense knowledge for intercultural dialogues\.InProceedings of the 33rd ACM International Conference on Information and Knowledge Management,pp\. 1774–1784\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- M\. Ochieng, V\. Gumma, S\. Sitaram, J\. Wang, V\. Chaudhary, K\. Ronen, K\. Bali, and J\. O’Neill \(2025\)Beyond metrics: evaluating llms effectiveness in culturally nuanced, low\-resource real\-world scenarios\.InProceedings of the Sixth Workshop on African Natural Language Processing \(AfricaNLP 2025\),pp\. 230–247\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- A\. Parrish, A\. Chen, N\. Nangia, V\. Padmakumar, J\. Phang, J\. Thompson, P\. M\. Htut, and S\. Bowman \(2022\)BBQ: a hand\-built bias benchmark for question answering\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 2086–2105\.Cited by:[§4\.5\.1](https://arxiv.org/html/2608.00640#S4.SS5.SSS1.Px3.p1.1)\.
- A\. Petrov, E\. La Malfa, P\. Torr, and A\. Bibi \(2023\)Language model tokenizers introduce unfairness between languages\.Advances in neural information processing systems36,pp\. 36963–36990\.Cited by:[§6\.1](https://arxiv.org/html/2608.00640#S6.SS1.SSS0.Px1.p1.1)\.
- S\. R\. Pfohl, H\. Cole\-Lewis, R\. Sayres, D\. Neal, M\. Asiedu, A\. Dieng, N\. Tomasev, Q\. M\. Rashid, S\. Azizi, N\. Rostamzadeh,et al\.\(2024\)A toolbox for surfacing health equity harms and biases in large language models\.Nature Medicine30\(12\),pp\. 3590–3600\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- J\. A\. Rodriguez, E\. Alsentzer, and D\. W\. Bates \(2024\)Leveraging large language models to foster equity in healthcare\.Journal of the American Medical Informatics Association31\(9\),pp\. 2147–2150\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- S\. Sugawara, K\. Inui, S\. Sekine, and A\. Aizawa \(2018\)What makes reading comprehension questions easier?\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 4208–4219\.Cited by:[§4\.5\.1](https://arxiv.org/html/2608.00640#S4.SS5.SSS1.Px3.p1.1)\.
- S\. Tan, S\. Zhuang, K\. Montgomery, W\. Tang, A\. Cuadron, C\. Wang, R\. Popa, and I\. Stoica \(2025\)Judgebench: a benchmark for evaluating llm\-based judges\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 63277–63303\.Cited by:[§6\.4](https://arxiv.org/html/2608.00640#S6.SS4.p2.1)\.
- Y\. Tao, O\. Viberg, R\. S\. Baker, and R\. F\. Kizilcec \(2024\)Cultural bias and cultural alignment of large language models\.PNAS nexus3\(9\),pp\. pgae346\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p2.1)\.
- G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§5\.1](https://arxiv.org/html/2608.00640#S5.SS1.p1.1)\.
- S\. Wang, Z\. Tang, H\. Yang, Q\. Gong, T\. Gu, H\. Ma, Y\. Wang, W\. Sun, Z\. Lian, K\. Mao,et al\.\(2025\)A novel evaluation benchmark for medical llms illuminating safety and effectiveness in clinical domains\.npj Digital Medicine\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- World Health Organization \(2023\)Billions left behind on the path to universal health coverage\.External Links:[Link](https://www.who.int/news/item/18-09-2023-billions-left-behind-on-the-path-to-universal-health-coverage)Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p1.1)\.
- World Health Organization \(2024\)Draft global traditional medicine strategy \(2025–2034\)\.Report by the Director\-GeneralTechnical ReportEB156/16,World Health Organization\.External Links:[Link](https://apps.who.int/gb/ebwha/pdf_files/EB156/B156_16-en.pdf)Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p1.1)\.
- World Health Organization \(2025\)Traditional, complementary and integrative medicine\.Note:[https://www\.who\.int/health\-topics/traditional\-complementary\-and\-integrative\-medicine](https://www.who.int/health-topics/traditional-complementary-and-integrative-medicine)Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1](https://arxiv.org/html/2608.00640#S5.SS1.p1.1)\.
- W\. Yue, X\. Wang, W\. Zhu, M\. Guan, H\. Zheng, P\. Wang, C\. Sun, and X\. Ma \(2024\)Tcmbench: a comprehensive benchmark for evaluating large language models in traditional chinese medicine\.arXiv preprint arXiv:2406\.01126\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang,et al\.\(2025\)Glm\-4\.5: agentic, reasoning, and coding \(arc\) foundation models\.arXiv preprint arXiv:2508\.06471\.Cited by:[§5\.1](https://arxiv.org/html/2608.00640#S5.SS1.p1.1)\.
- M\. Zhang, Y\. Shen, Z\. Li, H\. Sha, B\. Hu, Y\. Wang, C\. Huang, S\. Liu, J\. Tong, C\. Jiang,et al\.\(2025\)LLMEval\-med: a real\-world clinical benchmark for medical llms with physician validation\.arXiv preprint arXiv:2506\.04078\.Cited by:[§2](https://arxiv.org/html/2608.00640#S2.p1.1)\.
- M\. Zhao, K\. Wang, R\. Gu, and S\. Zhong \(2019\)A comparative study on shared\-use medicines in tibetan and chinese medicine\.Journal of ethnobiology and ethnomedicine15\(1\),pp\. 43\.Cited by:[§1](https://arxiv.org/html/2608.00640#S1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§6\.4](https://arxiv.org/html/2608.00640#S6.SS4.p2.1)\.

## Appendix ATask Operators and Subtask Instantiations

Table 4:Probing operators and subtasks in TreeProbe\. Each subtask is defined by an operatorop:⟨𝒫req,𝒫opt⟩→τ\\text\{op\}:\\langle\\mathcal\{P\}\_\{\\text\{req\}\},\\mathcal\{P\}\_\{\\text\{opt\}\}\\rangle\\to\\tau, where𝒫req\\mathcal\{P\}\_\{\\text\{req\}\}slots are always exposed and𝒫opt\\mathcal\{P\}\_\{\\text\{opt\}\}slots are independently exposed with probabilityp=0\.5p=0\.5\. The symbol∧\\wedgeindicates joint occurrence;∨\\veeindicates one\-of\-the\-listed\.Table[4](https://arxiv.org/html/2608.00640#A1.T4)summarizes the operator design used to instantiate TreeProbe items\. Each subtask is defined by a targetτ\\tau, a set of required premises𝒫req\\mathcal\{P\}\_\{\\mathrm\{req\}\}, and optional premises𝒫opt\\mathcal\{P\}\_\{\\mathrm\{opt\}\}\. Required premises are always exposed to ensure that each item is answerable from evidence\-supported slots, while optional premises introduce controlled variation without changing the target capability\. The operators are organized along the three roots of Tibetan medicine: physiopathology, diagnosis, and therapy\. This design allows MCQ and QA items to be derived from the same structured knowledge units while preserving task\-specific reasoning requirements, such as probing a single clinical slot, inferring three\-humor imbalance from diagnostic evidence, or constructing a comprehensive treatment plan\.

## Appendix BInstantiated Knowledge Unit Example

Table 5:A fully instantiated knowledge unit example from the TreeProbe dataset\. Multi\-valued slots are listed with each value bracketed\. Slots not covered by the source are recorded as empty\.Table[5](https://arxiv.org/html/2608.00640#A2.T5)shows a fully instantiated knowledge unit in TreeProbe\. Each unit is organized as a slot–value structure anchored to a disease code and Tibetan disease name, and covers the clinical trajectory from etiology and pathogenesis to diagnostic evidence and treatment\. Diagnostic slots include chief complaint, tongue, urine, pulse, and auxiliary diagnosis, while therapeutic slots specify the treatment principle and interventions across diet, lifestyle, medication, external therapy, and consolidation therapy\. This structure makes each item traceable to evidence\-supported Tibetan medical knowledge and allows downstream operators to instantiate MCQ and QA items only from populated slots, while unsupported slots are explicitly left empty\.

## Appendix CDimension\-Level Analysis of Generative QA

![Refer to caption](https://arxiv.org/html/2608.00640v1/x4.png)\(a\)Zero\-shot\.
![Refer to caption](https://arxiv.org/html/2608.00640v1/x5.png)\(b\)One\-shot\.

Figure 5:Per\-model decomposition of QA performance across five evaluation dimensions: Accuracy, Helpfulness, Linguistic Quality, Ontology Fidelity, and Completeness\.Figure[5](https://arxiv.org/html/2608.00640#A3.F5)further show that the QA gap is not a uniform degradation across dimensions, but a decoupling between linguistic realization and medical reasoning\. In both zero\-shot and one\-shot settings, Linguistic Quality forms a consistently high outer contour for most models, whereas Accuracy, Helpfulness, Ontology Fidelity, and Completeness remain compressed toward the center\. This indicates that fluent Tibetan generation is a weak proxy for Tibetan medical competence: models can produce well\-formed Tibetan text while failing to supply accurate, complete, or ontology\-faithful medical content\. The model\-level shapes also reveal different failure profiles\. For models such as Qwen3\.5\-397B\-A17B and DeepSeek\-Chat, high linguistic scores coexist with much lower content scores, suggesting a surface\-fluency bottleneck rather than a purely language\-generation bottleneck\. Claude Sonnet 4\.6 is an exception, showing weaker performance even on Linguistic Quality, which suggests that its QA failures involve both Tibetan realization and medical reasoning\. One\-shot prompting slightly expands the content\-related dimensions for several models, especially GPT\-5\.4 and Gemini 3\.1 Flash\-Lite, but the overall geometry remains stable: the linguistic contour stays separated from the medical\-content contours\. This reinforces our conclusion that in\-context demonstrations improve task adaptation only partially and do not close the underlying ontology\-level reasoning gap\.

## Appendix DModel\-wise Drift Geometry

![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/single/p1.png)
![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/single/p5.png)
![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/single/p2.png)
![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/single/p4.png)
![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/single/p3.png)
![Refer to caption](https://arxiv.org/html/2608.00640v1/acl-style-files-master/figure/single/p6.png)

Figure 6:Per\-model drift geometry across the three roots of the Medical Tree\. Each panel plots ODLO \(xx\-axis\) against BTD\-LO \(yy\-axis\) for Physiopathology \(blue\), Diagnosis \(green\), and Therapy \(orange\) under zero\-shot \(circle\) and one\-shot \(square\) conditions\. Top row: GPT\-5\.4, Gemini 3\.1 Flash\-Lite, Claude Sonnet 4\.6\. Bottom row: GLM\-4\.7, DeepSeek\-Chat, Qwen3\.5\-397B\-A17B\.Figure[3](https://arxiv.org/html/2608.00640#S6.F3)gives a more fine\-grained diagnosis of MCQ errors by placing each model, root, and prompting condition in the ODLO–BTD\-LO plane\. The most salient pattern is that points are concentrated in the right half of the plane, while the Tibetan\-confusion quadrants are largely empty\. This indicates that model failures are rarely dominated by confusion among Tibetan\-internal alternatives; instead, once models leave the faithful answer, their errors tend to move toward an external medical ontology\. This supports the main premise of TreeProbe: low accuracy on Tibetan medicine is not merely a generic domain\-knowledge deficit, but often takes the form of culturally structured ontology drift\.

The vertical separation further shows that this drift is not monolithic\. Models differ in whether their external errors lean toward biomedical or TCM reasoning\. Claude Sonnet 4\.6 and GPT\-5\.4 more often occupy the biomedical\-leaning region, whereas GLM\-4\.7, DeepSeek\-Chat, and Gemini 3\.1 Flash\-Lite show stronger TCM\-oriented drift; Qwen3\.5\-397B\-A17B stays closer to the boundary between the two\. This model\-level separation is consistent with differences in the cultural and medical composition of pretraining corpora, but the root\-level structure reveals an additional mechanism\. Therapy points are repeatedly pulled toward the TCM\-bias region, while diagnosis and physiopathology more often move upward toward biomedical or mixed drift\. This suggests that external drift is activated by the reasoning function required by the item: therapeutic tasks are more vulnerable to TCM\-style substitution because Tibetan and Chinese medicine share surface\-level therapeutic vocabulary, whereas diagnostic and physiopathological tasks more readily trigger biomedical reframing\.

The zero\-shot/one\-shot comparison suggests that demonstrations dampen but do not eliminate this failure mode\. In several models, one\-shot points move closer to the origin, indicating a reduction in the cultural component of the error\. However, their quadrant membership and relative vertical ordering remain largely stable: biomedical\-leaning models do not become TCM\-leaning, and TCM\-leaning models are not pulled back into Tibetan\-internal confusion\. This shows that in\-context examples can adjust the strength of drift, but they do not reliably change the underlying ontology selected by the model\. In this sense, Figure[3](https://arxiv.org/html/2608.00640#S6.F3)complements the accuracy and QA analyses: prompting improves task adaptation at the margin, but the deeper problem is the model’s tendency to resolve Tibetan medical uncertainty by substituting a more available external medical system\.

## Appendix ELLM Drafting Prompt

Table 6:Prompt template used by the LLM drafter to produce a Tibetan item stem and reference answer, grounded in the provided evidence span\.Table[6](https://arxiv.org/html/2608.00640#A5.T6)shows the prompt template used for constrained LLM drafting\. The drafter is given the subtask, task type, disease information, premise slots, target slot, target value, and the corresponding Tibetan evidence span\. It is instructed to write the item stem only from the exposed premise slots, without revealing or implying the target value, and to generate either the Faithful Answer for MCQ or the reference answer for QA\. The output is restricted to Tibetan and formatted as JSON\. This design keeps the LLM’s role limited to linguistic and structural drafting: all content must be entailed by the provided evidence, while unsupported knowledge or additional medical assumptions are explicitly prohibited\.

## Appendix FEvaluation Prompts

Table 7:Zero\-shot prompt template for MCQ items\.Table 8:One\-shot prompt template for MCQ items\.Table 9:Zero\-shot prompt template for QA items\.Table 10:One\-shot prompt template for QA items\.Table 11:Compressed scoring prompt template used for GPT\-5\.4 evaluation of open\-ended QA items\.Tables[7](https://arxiv.org/html/2608.00640#A6.T7)–[11](https://arxiv.org/html/2608.00640#A6.T11)document the prompt templates used for model evaluation and GPT\-5\.4\-based scoring\. Tables[7](https://arxiv.org/html/2608.00640#A6.T7)and[8](https://arxiv.org/html/2608.00640#A6.T8)show the zero\-shot and one\-shot MCQ prompts, both of which constrain models to return only a single answer letter\. Tables[9](https://arxiv.org/html/2608.00640#A6.T9)and[10](https://arxiv.org/html/2608.00640#A6.T10)present the zero\-shot and one\-shot QA prompts, which require models to answer in Tibetan\. Table[11](https://arxiv.org/html/2608.00640#A6.T11)gives the compressed scoring prompt used by GPT\-5\.4 to evaluate open\-ended QA responses\. The evaluator scores each response along five dimensions \(Accuracy, Helpfulness, Linguistic Quality, Ontology Fidelity, and Completeness\) and assigns a dominant error type\. These templates standardize input format, output constraints, and scoring criteria across models, ensuring that differences in performance reflect model behavior rather than prompt inconsistency\.

## Appendix GEthical Considerations

TreeProbe is a research benchmark for evaluating cultural bias and ontology\-faithful reasoning in LLMs, not a clinical decision\-support system\. It should not be used for diagnosis, treatment planning, or medical advice, and benchmark scores should not be interpreted as evidence of clinical deployability\. Any real\-world use of Tibetan medical knowledge should remain under the judgment of qualified Tibetan medicine practitioners and clinical professionals\.

The benchmark is constructed from authoritative Tibetan medical sources and expert adjudication\. Multiple rounds of expert review are used to reduce risks of medical and cultural misrepresentation\. At the same time, TreeProbe should be understood as a diagnostic evaluation resource rather than an exhaustive representation of Tibetan medicine; it may not cover all regional practices, lineages, or clinical interpretations\.

Potential misuse includes training on the benchmark, overfitting to its format, or using scores to claim general clinical competence\. We encourage its use for bias diagnosis and model evaluation, not as a source of medical advice or as a substitute for expert assessment\.