真实性信号在语码混合中是否持续有效?探究Hinglish幻觉检测的隐藏状态

arXiv cs.CL 论文

摘要

本研究探究了基于单语数据训练的幻觉检测探针在Hinglish上的迁移效果,结果显示检测性能稳健,且Hindi和Hinglish中的幻觉率高于英语。

arXiv:2609.22138v1 Announce Type: new Abstract: Hidden-state hallucination probing - training a linear classifier on an LLM's internal activations to detect whether a generated answer is faithful to the input - is an active area of 2026 research, with recent work reporting 0.90-1.00 AUROC across several benchmarks and languages. However, none of this work has tested probes on code-mixed input, despite the fact that a huge population of chatbot users write in Hindi-English code-mixed text ("Hinglish"). We address this gap directly: does a hallucination probe trained on clean-language hidden states transfer to Hinglish, or does the signal degrade under code-mixing? We construct a 5,674-item Hindi/English/Hinglish QA benchmark, generate and label 17,022 model responses across three open-weight 7-8B LLMs (Qwen2.5-7B, Mistral-7B, Llama-3.1-8B), extract per-layer hidden states at two token positions, and train linear and MLP probes for in-distribution detection and cross-lingual transfer. We find that the hallucination signal survives code-mixing well: transfer AUROC ranges from 0.88 to 0.99, with gaps of mostly under 0.05 AUROC relative to in-distribution performance, and that Hindi-trained probes transfer to Hinglish more reliably than English-trained probes. As an independent, practically motivated finding, all three models hallucinate substantially more on Hindi and Hinglish than on English for matched facts. We release our code and synthetic Hinglish QA dataset to support further work on code-mixed hallucination detection.
查看原文
查看缓存全文

缓存时间: 2026/09/22 09:04

# Does the Truthfulness Signal Survive Code-Mixing? Probing Hidden States for Hallucination Detection in Hinglish
Source: [https://arxiv.org/html/2609.22138](https://arxiv.org/html/2609.22138)
###### Abstract

Hidden\-state hallucination probing \- training a linear classifier on an LLM’s internal activations to detect whether a generated answer is faithful to the input \- is an active area of 2026 research, with recent work reporting 0\.90–1\.00 AUROC across several benchmarks and languages\. However, none of this work has tested probes on code\-mixed input, despite the fact that a huge population of chatbot users write in Hindi–English code\-mixed text \(“Hinglish”\)\. We address this gap directly: does a hallucination probe trained on clean\-language hidden states transfer to Hinglish, or does the signal degrade under code\-mixing? We construct a 5,674\-item Hindi/English/ Hinglish QA benchmark, generate and label 17,022 model responses across three open\-weight 7–8B LLMs \(Qwen2\.5\-7B, Mistral\-7B, Llama\-3\.1\-8B\), extract per\-layer hidden states at two token positions, and train linear and MLP probes for in\-distribution detection and cross\-lingual transfer\. We find that the hallucination signal survives code\-mixing well: transfer AUROC ranges from 0\.88 to 0\.99, with gaps of mostly under 0\.05 AUROC relative to in\-distribution performance, and that Hindi\-trained probes transfer to Hinglish more reliably than English\-trained probes\. As an independent, practically motivated finding, all three models hallucinate substantially more on Hindi and Hinglish than on English for matched facts\. We release our code and synthetic Hinglish QA dataset to support further work on code\-mixed hallucination detection\.

## 1Introduction

Large language models \(LLMs\) are increasingly deployed as conversational assistants for users who do not write in clean, monolingual English\. A particularly large such population types in*Hinglish*\- Hindi and English mixed within the same sentence or utterance \- when interacting with chatbots, search assistants, and customer\-service agents across South Asia\. Despite this, essentially all published work on detecting when an LLM is hallucinating by probing its internal hidden states has been evaluated on clean, single\-language text\.

Hidden\-state probing for hallucination detection \- training a lightweight classifier \(typically linear\) on an LLM’s per\-layer activations to predict whether a generated answer is faithful or hallucinated \- has emerged as one of the most effective and lightweight detection strategies available\. Four recent studies illustrate the strength of this approach:[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.22138#bib.bib3)track how each layer contributes to the residual stream update rather than probing static activations, and use this signal to detect hallucinations with substantially fewer parameters than prior probes;[Mrykhin and Malykh \(2026\)](https://arxiv.org/html/2609.22138#bib.bib2)augment standard linear probes with a small number of learnable prompt embeddings and show consistent gains over the vanilla\-linear baseline, along with promising pre\-generation and cross\-model transfer;[Alvi et al\. \(2026\)](https://arxiv.org/html/2609.22138#bib.bib1)extend hidden\-state probing to a multilingual setting for the first time, reporting robust cross\-lingual generalization of a stacked probing framework across French, Bangla, and Amharic; and[Aiersilan \(2026\)](https://arxiv.org/html/2609.22138#bib.bib4)show that even under aggressive 4\-bit quantization, a single linear probe on a mid\-network layer recovers 0\.904–1\.000 AUROC across three 7–8B open\-weight model families, with MLP probes rarely improving on the linear baseline by more than 0\.01 AUROC\. Taken together, this body of work establishes that truthfulness is linearly decodable from hidden states, that the signal is robust to quantization, and \- per[Alvi et al\. \(2026\)](https://arxiv.org/html/2609.22138#bib.bib1)\- that it can transfer across typologically distinct*languages*\. What remains untested is whether the signal survives*within\-sentence*code\-mixing, where the model must track truthfulness across a single input that switches languages at the word or phrase level, a qualitatively different and arguably harder generalization test than transfer between monolingual languages\.

This gap is not a narrow technical oversight\. Code\-mixed input is the default register of everyday typed communication for hundreds of millions of Hindi–English bilingual speakers, and any hallucination detector intended for real\-world deployment in this population will encounter Hinglish constantly\. If the truthfulness signal that hidden\-state probes rely on is disrupted by code\-mixing \- for instance because code\-mixed tokens sit outside the distribution the base LLM was most heavily trained on, or because switching languages mid\-sentence perturbs the residual stream in ways that are entangled with \(rather than orthogonal to\) the truthfulness direction \- then probes validated only on clean text could fail silently exactly where they are needed most\.

We study two research questions:

1. RQ1\(Primary\) Does a hallucination probe trained on clean\-language hidden states transfer to Hinglish, or does the truthfulness signal degrade under code\-mixing?
2. RQ2\(Secondary\) Do models hallucinate more on Hinglish than on clean Hindi or English for the same underlying facts?

To answer these questions, we construct a QA benchmark spanning clean Hindi, clean English, and synthetic Hindi–English code\-mixed \(Hinglish\) items derived from the same underlying facts, generate answers from three open\-weight 7–8B instruction\-tuned LLMs \(Qwen2\.5\-7B, Mistral\-7B, and Llama\-3\.1\-8B\) in all three language conditions, label each answer as faithful or hallucinated, extract per\-layer hidden states at two token positions, and train linear and MLP probes for both in\-distribution detection and train\-on\-clean/test\-on\-Hinglish transfer\.

We find that the hallucination signal survives code\-mixing well: transfer AUROC ranges from 0\.88 to 0\.99 across models and probing positions, with the gap to in\-distribution performance under 0\.05 AUROC in most conditions, and that probes trained on clean Hindi transfer to Hinglish more reliably than probes trained on clean English\. As a secondary, independently motivating finding, all three models are substantially more likely to hallucinate when answering in Hindi or Hinglish than in English on matched questions\.

Our contributions are:

- •The first evaluation of hidden\-state hallucination probing under within\-sentence code\-mixed input, addressing a gap left open by all four prior probing studies we survey\.
- •Evidence that the truthfulness signal transfers to Hinglish with only modest degradation, and that this holds across three model families and two probing positions\.
- •Evidence that Hindi\-trained probes transfer to Hinglish better than English\-trained probes, which we discuss as consistent with a shared\-script/shared\-vocabulary hypothesis\.
- •Release of our code and synthetic Hinglish QA dataset to support future work validating these findings on naturally\-occurring code\-mixed corpora\.

## 2Related Work

#### Hidden\-state hallucination detection\.

A growing line of work shows that an LLM’s internal activations encode a recoverable signal about whether its own output is truthful\.[Azaria and Mitchell \(2023\)](https://arxiv.org/html/2609.22138#bib.bib12)were among the first to demonstrate that a simple classifier trained on hidden states can predict the truthfulness of a statement more reliably than the model’s own stated confidence\.[Orgad et al\. \(2024\)](https://arxiv.org/html/2609.22138#bib.bib13)extend this analysis, showing that truthfulness information is concentrated in specific token positions rather than spread uniformly across a sequence, and that probes trained on one dataset transfer poorly to datasets requiring different underlying skills \- a generalization concern closely related to, but distinct from, the cross\-lingual and code\-mixing transfer question we study here\. Building on this foundation, four recent 2026\-era papers push hidden\-state probing further along different axes\.[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.22138#bib.bib3)introduce the ICR Score, a metric that captures each transformer module’s*contribution*to the residual stream update rather than the static hidden state itself, and show this dynamic signal outperforms static\-representation probes with far fewer parameters\.[Mrykhin and Malykh \(2026\)](https://arxiv.org/html/2609.22138#bib.bib2)propose Prompt Embedding Probes, which augment a frozen LLM’s input with a small number of learnable prompt embeddings before extracting hidden states, improving in\-distribution detection and remaining effective in pre\-generation and cross\-model transfer settings, though the authors note that cross\-*dataset*transfer remains difficult\.[Aiersilan \(2026\)](https://arxiv.org/html/2609.22138#bib.bib4)run a systematic comparison of linear probes, MLP probes, and several sampling\-based detectors \(INSIDE EigenScore, self\-consistency, attention entropy\) across three 7–8B open\-weight models under 4\-bit quantization, finding that a single linear probe on a mid\-network layer \(blocks 13–18 of 32 for Llama and Mistral; 19–25 of 28 for Qwen\) achieves 0\.904–1\.000 AUROC and that MLP probes rarely improve on this by more than 0\.01 AUROC \- a finding we also test in our own linear\-vs\-MLP comparison \(Section[4](https://arxiv.org/html/2609.22138#S4)\)\.[Alvi et al\. \(2026\)](https://arxiv.org/html/2609.22138#bib.bib1)take the most direct step toward multilinguality, introducing a stacking framework that probes full hidden\-state trajectories across layers and evaluating cross\-lingual generalization from English to French, Bangla, and Amharic, reporting strong transfer across this typologically diverse set\.

Across all four of these 2026 papers, and the earlier foundational work they build on, one gap is consistent: every evaluation is conducted on*monolingual*input, whether the language is English, French, Bangla, Amharic, or another single language per example\. None test whether the truthfulness signal survives*within\-sentence code\-mixing*, where a single input alternates between two languages at the word or phrase level\. Cross\-lingual transfer \(e\.g\., English\-trained probe evaluated on Bangla text\) and code\-mixing robustness are related but distinct generalization challenges: the former asks whether a direction learned in one language’s representational geometry recovers in another language’s geometry, while the latter asks whether that direction remains coherent when both geometries are interleaved within the same forward pass\. Our work is, to our knowledge, the first to test hidden\-state hallucination probes under this second condition\.

#### Code\-mixing and Hinglish NLP\.

Code\-mixing has long been studied from a theoretical linguistics perspective; the Matrix Language Frame \(MLF\) model of[Myers\-Scotton \(1993\)](https://arxiv.org/html/2609.22138#bib.bib7)formalizes intrasentential code\-switching in terms of a dominant*matrix language*that supplies the grammatical frame into which content words from an*embedded language*are inserted, a framework we adopt for our synthetic Hinglish construction \(Section[3](https://arxiv.org/html/2609.22138#S3)\)\. On the NLP benchmarking side, LinCE\([Aguilar et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib5)\)and GLUECoS\([Khanuja et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib6)\)established centralized benchmarks for evaluating code\-switched NLP systems on tasks such as language identification, part\-of\-speech tagging, named entity recognition, and sentiment analysis across several code\-mixed language pairs, including Hindi–English\. These benchmarks establish that code\-mixing is a first\-class, well\-studied NLP phenomenon with dedicated evaluation infrastructure \- infrastructure that, to date, has not been connected to the hidden\-state hallucination\-probing literature reviewed above\. We were unable to obtain access to naturally\-occurring code\-mixed data from these or comparable sources during data collection \(Section[Limitations](https://arxiv.org/html/2609.22138#Sx1)\), and instead construct synthetic Hinglish; we see validating our findings on naturally\-occurring corpora such as LinCE or GLUECoS as the most direct and important piece of future work\.

#### Multilingual hallucination and factuality evaluation\.

More broadly, a separate line of work evaluates LLM factuality and hallucination rates across languages without probing internal representations, typically via multilingual QA or fact\-verification benchmarks built on resources such as XQuAD\([Artetxe et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib8)\)and IndicQA\([Doddapaneni et al\., 2023](https://arxiv.org/html/2609.22138#bib.bib9)\), both of which we draw on for our clean\-language data \(Section[3](https://arxiv.org/html/2609.22138#S3)\)\. This literature generally finds that LLMs are less factually reliable in lower\-resource languages than in English, which is consistent with, and provides context for, our own secondary finding \(RQ2\) that all three models we evaluate hallucinate far more often in Hindi and Hinglish than in English on matched questions\.

## 3Method

### 3\.1Data construction

We construct a 5,674\-item question\-answering benchmark spanning three language conditions derived from comparable underlying facts: clean Hindi, clean English, and synthetic Hindi–English code\-mixed \(Hinglish\)\.

#### Clean Hindi and English\.

We combine XQuAD\-hi\([Artetxe et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib8)\)and IndicQA\-hi\([Doddapaneni et al\., 2023](https://arxiv.org/html/2609.22138#bib.bib9)\)for clean Hindi context\-question\-answer triples, and XQuAD\-en for a matched clean\-English condition, yielding 3,432 clean QA pairs \(2,242 Hindi, 1,190 English\)\. We filter IndicQA’s SQuAD\-2\.0\-style unanswerable questions, which are returned as an empty\-string answer rather than an empty answer list, to ensure every retained item has a genuine gold answer\.

#### Synthetic Hinglish\.

We were unable to obtain naturally\-occurring Hindi–English code\-mixed QA data during data collection \(see Section[Limitations](https://arxiv.org/html/2609.22138#Sx1)\); the Hindi–English configurations of LinCE\([Aguilar et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib5)\)were unreachable via their external hosting infrastructure at the time of this work, and no comparable GLUECoS\([Khanuja et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib6)\)mirror was accessible\. We therefore construct synthetic Hinglish following the Matrix Language Frame \(MLF\) model of[Myers\-Scotton \(1993\)](https://arxiv.org/html/2609.22138#bib.bib7): Hindi is retained as the*matrix language*supplying the grammatical frame, while a controlled fraction of*content*words \- nouns, verbs, and other open\-class items \- are substituted with their English translations drawn from the MUSE Hindi–English bilingual dictionary\([Conneau et al\., 2018](https://arxiv.org/html/2609.22138#bib.bib11)\)\. Function words \(pronouns, postpositions, copulas, conjunctions, and other closed\-class items\) are explicitly excluded from substitution, since code\-mixing that alters function words produces text that is linguistically unnatural and not representative of how Hindi–English bilinguals actually code\-mix\. We apply this procedure to the 2,242 clean\-Hindi items, targeting a 40% content\-word substitution rate per context and per question; the rate actually achieved varies by item depending on MUSE dictionary coverage\. This yields 2,242 synthetic Hinglish items, for a combined dataset of 5,674 rows across all three conditions\.

### 3\.2Hallucination labeling

For each of three open\-weight instruction\-tuned LLMs \- Qwen2\.5\-7B\-Instruct, Mistral\-7B\-Instruct\-v0\.3, and Llama\-3\.1\-8B\-Instruct\([AI@Meta, 2024](https://arxiv.org/html/2609.22138#bib.bib14)\)\- we generate an answer to every item in all three language conditions, using each model’s native chat template with a system prompt instructing short, direct, context\-grounded answers, and greedy decoding\. This yields5,674×3=17,0225\{,\}674\\times 3=17\{,\}022model responses\. We label each response as*faithful*or*hallucinated*via token\-level F1 against the gold answer\([Rajpurkar et al\., 2016](https://arxiv.org/html/2609.22138#bib.bib10)\), the standard metric used in prior hidden\-state probing work\([Aiersilan, 2026](https://arxiv.org/html/2609.22138#bib.bib4)\), with faithful defined as F1≥0\.5\\geq 0\.5\.

We note that early pilot runs using a raw string\-concatenation prompt rather than each model’s chat template produced degenerate, document\-continuation style generations for all three instruction\-tuned models \- an artifact of prompting an instruct model outside its trained input format rather than a genuine capability failure\. We verified this on a small held\-out sample before the full labeling run and confirmed that chat\-template prompting resolves it; we flag this explicitly, since silently mislabeling such artifacts as hallucination would inflate hallucination rates in a way unrelated to our research questions\.

### 3\.3Hidden\-state extraction

For each labeled \(context, question, generated\-answer\) triple, we perform a single teacher\-forced forward pass over the concatenation of the chat\-templated prompt and the model’s own generated answer, and extract the hidden state at every transformer layer \(plus the embedding layer\) at two token positions: the*last\-prompt\-token*position, immediately before generation begins, and the*last\-answer\-token*position, at the end of the generated answer\. We test both positions because prior work differs on which is more informative: pre\-generation probing tests whether the model already encodes a signal that it may be about to hallucinate, while post\-generation probing tests whether the completed answer’s representation encodes its own faithfulness\. Hidden states are stored in half precision to limit storage overhead\. This yields, per model, one 29\-layer \(Qwen2\.5\-7B\) or 33\-layer \(Mistral\-7B, Llama\-3\.1\-8B\) hidden\-state vector at each of the two positions for every generated response \- 5,674 responses for Qwen2\.5\-7B and Mistral\-7B, and 5,544 for Llama\-3\.1\-8B after excluding rows lost to an occasional out\-of\-memory condition during single\-row extraction \(∼\\sim2% of rows; see Section[Limitations](https://arxiv.org/html/2609.22138#Sx1)\)\.

### 3\.4Probing setup

For each model, language condition, token position, and layer, we train a linear probe \- logistic regression with balanced class weighting and standard feature scaling, since unscaled raw hidden\-state features are poorly conditioned for L\-BFGS optimization and converge extremely slowly without it \- to predict the faithful/hallucinated label from the corresponding hidden\-state vector\. We report three sets of results:

- •In\-distribution detection: 3\-fold stratified cross\-validated AUROC within each \(model, language, position, layer\) combination, reported as a layer\-wise curve per model and position \(Figure[1](https://arxiv.org/html/2609.22138#S4.F1)\)\.
- •Cross\-lingual transfer to Hinglish: a probe is trained on all available clean\-language data \(Hindi or English, separately\) at a given layer and position, then evaluated on the held\-out Hinglish condition at the same layer and position\. This experiment directly answers RQ1\.
- •Linear\-vs\-MLP comparison: at the best\-performing layer identified by the in\-distribution experiment, we additionally train a single\-hidden\-layer \(64\-unit\) MLP probe under the same cross\-validation protocol, to test whether a linear decision boundary is sufficient or whether nonlinear probing recovers meaningfully more signal, following the comparison reported by[Aiersilan \(2026\)](https://arxiv.org/html/2609.22138#bib.bib4)\.

Conditions with fewer than 10 examples in the minority class are excluded from in\-distribution evaluation, and conditions with fewer than 5 examples in either class are excluded from transfer evaluation, since AUROC estimated on very few minority\-class examples is unstable\. As we report in Section[4](https://arxiv.org/html/2609.22138#S4), class imbalance is substantial for Hindi and Hinglish in particular, since all three models hallucinate far more frequently in these conditions than in English; we return to the implications of this imbalance for interpreting the resulting AUROC values in Section[5](https://arxiv.org/html/2609.22138#S5)\.

## 4Experiments and Results

### 4\.1In\-distribution detection

Figure[1](https://arxiv.org/html/2609.22138#S4.F1)shows layer\-wise AUROC for all three models, both token positions, and all three language conditions\. Two patterns are consistent across every model and position\. First, AUROC rises sharply from the embedding layer and plateaus within the first few transformer blocks, consistent with prior findings that truthfulness becomes linearly decodable early and remains so through most of the network\([Aiersilan, 2026](https://arxiv.org/html/2609.22138#bib.bib4)\)\. Second, and more strikingly for our purposes,Hindi and Hinglish probes substantially outperform English probes in\-distribution across all three models and both positions: at the best layer, English AUROC tops out at 0\.77–0\.84, while Hindi and Hinglish reach 0\.89–0\.99 \(full best\-layer values in Table[3](https://arxiv.org/html/2609.22138#S4.T3)\)\. We discuss this pattern, and the class\-imbalance caveat relevant to interpreting it, in Section[5](https://arxiv.org/html/2609.22138#S5)\.

![Refer to caption](https://arxiv.org/html/2609.22138v1/layerwise_auroc_curves.png)Figure 1:Layer\-wise in\-distribution AUROC for each model \(rows\) and token position \(columns\), by language condition\. Dashed line marks chance level \(AUROC = 0\.5\)\.
### 4\.2Cross\-lingual transfer to Hinglish

Table[1](https://arxiv.org/html/2609.22138#S4.T1)reports our central result: for each model and position, we train a probe on clean Hindi or clean English at the best\-performing layer for that training condition, and evaluate it on held\-out Hinglish\. Transfer AUROC ranges from 0\.876 \(Qwen, English\-trained, prompt position\) to 0\.991 \(Mistral, Hindi\-trained, answer position\) \- consistently well above chance and, in most conditions, close to what a probe trained directly on Hinglish achieves in\-distribution\.

ModelTrainLayerAUROClast\-answer\-token positionLlama\-3\.1en190\.924Llama\-3\.1hi150\.963Mistralen150\.939Mistralhi190\.991Qwen2\.5en270\.934Qwen2\.5hi160\.964last\-prompt\-token positionLlama\-3\.1en120\.898Llama\-3\.1hi140\.953Mistralen140\.917Mistralhi20\.939Qwen2\.5en150\.876Qwen2\.5hi140\.943Table 1:Cross\-lingual transfer to Hinglish: probes trained on clean Hindi \(hi\) or clean English \(en\) at the best\-performing layer for that training language, evaluated on held\-out Hinglish\. In every one of the six model×\\timesposition combinations, the Hindi\-trained probe transfers better than the English\-trained probe\.Table[2](https://arxiv.org/html/2609.22138#S4.T2)makes the comparison to in\-distribution performance explicit\. The gap between transfer AUROC and in\-distribution Hinglish AUROC \(at each model’s own best in\-distribution layer and position\) is under 0\.05 AUROC in 10 of 12 conditions, and is*negative*\- i\.e\., transfer*exceeds*in\-distribution performance \- in all six Hindi\-trained conditions except two, where it is nonetheless close to zero\. The largest gap we observe is 0\.086 AUROC \(Qwen2\.5, English\-trained, prompt position\)\. Across every one of the six model×\\timesposition pairs,the Hindi\-trained probe transfers more reliably to Hinglish than the English\-trained probe, with a mean gap of−0\.017\-0\.017for Hindi\-trained probes versus\+0\.043\+0\.043for English\-trained probes\.

Table 2:Gap = transfer AUROC−\-in\-distribution Hinglish AUROC \(at the model’s own best in\-distribution layer/position\)\. Negative values mean transfer*outperforms*in\-distribution detection\. Hindi\-trained probes \(gap column,hirows\) are closer to zero or negative in 5 of 6 cases; English\-trained probes are positive \(i\.e\., a real degradation\) in 5 of 6 cases\.Taken together, these two tables directly answer RQ1:the truthfulness signal survives code\-mixing well\. We find no evidence of the signal collapsing or becoming unreliable under Hinglish input for any of the three model families or two probing positions we test\.

### 4\.3Linear vs\. MLP probes

Table[3](https://arxiv.org/html/2609.22138#S4.T3)compares linear and MLP probes at each model’s best\-performing layer per language and position\. Consistent with[Aiersilan \(2026\)](https://arxiv.org/html/2609.22138#bib.bib4), MLP probes rarely improve substantially on the linear baseline, and in several low\-resource conditions \(particularly Hinglish and Hindi at the prompt position for Mistral\) the MLP*underperforms*the linear probe by a wide margin, most likely due to overfitting on the smaller, highly imbalanced minority class discussed in Section[3\.4](https://arxiv.org/html/2609.22138#S3.SS4)\. We take this as further support for using linear probes as the primary, most reliable method, with MLP probes serving only as a secondary sanity check\.

Table 3:Linear vs\. MLP probe AUROC at each condition’s best in\-distribution layer \(prompt position shown; answer position follows the same pattern and is reported in full in Table[4](https://arxiv.org/html/2609.22138#A1.T4), Appendix[A\.4](https://arxiv.org/html/2609.22138#A1.SS4)\)\. “cm” denotes Hinglish\. The MLP’s sharp drop for Mistral/Hinglish illustrates overfitting under class imbalance rather than a genuine linear\-probe ceiling\.
### 4\.4Faithful\-rate comparison across languages

As an independent, practically motivating finding addressing RQ2, Figure[2](https://arxiv.org/html/2609.22138#S4.F2)shows the fraction of faithful \(non\-hallucinated\) answers per model and language\. All three models are dramatically less faithful in Hindi and Hinglish than in English: Llama\-3\.1 drops from 86\.4% faithful in English to 24\.2% in Hindi and 24\.4% in Hinglish; Mistral drops from 60\.9% to under 1% in both Hindi and Hinglish; and Qwen2\.5 drops from 81\.5% to 3\.3% \(Hindi\) and 5\.9% \(Hinglish\)\. Hindi and Hinglish faithful rates are close to each other for every model, suggesting that code\-mixing itself does not substantially change how often these models hallucinate relative to clean Hindi \- the large faithfulness gap is primarily a Hindi\-vs\-English gap, not a code\-mixing\-specific effect\.

![Refer to caption](https://arxiv.org/html/2609.22138v1/faithful_rate_comparison.png)Figure 2:Faithful \(non\-hallucinated\) answer rate by model and language condition\. All three models hallucinate far more often in Hindi and Hinglish than in English\.

## 5Discussion

#### Why does the signal survive code\-mixing?

We did not expect, a priori, that transfer to Hinglish would be this robust\. One plausible explanation is that the truthfulness direction identified by these probes is encoded relatively early and is tied more to task\-level semantic content \(whether the stated fact matches the context\) than to surface tokenization or script\. Since our synthetic Hinglish retains the Hindi grammatical frame and only swaps content words, the underlying semantic content the model must track for truthfulness is largely preserved even as surface form changes \- which may explain why the signal is more robust to code\-mixing than one might initially expect from a purely surface\-level generalization argument\.

#### Why do Hindi\-trained probes transfer better than English\-trained probes?

Across all six model×\\timesposition pairs, Hindi\-trained probes transfer to Hinglish more reliably than English\-trained probes \(Table[1](https://arxiv.org/html/2609.22138#S4.T1), Table[2](https://arxiv.org/html/2609.22138#S4.T2)\)\. We hypothesize this reflects a shared\-script and shared\-grammatical\-frame effect: our synthetic Hinglish retains Hindi as its matrix language, so a probe trained on clean Hindi representations is being asked to generalize to text that shares grammatical structure and \(for the majority, non\-substituted portion of each sentence\) surface form with its training distribution, whereas an English\-trained probe must generalize across a much larger representational shift\. We present this as a hypothesis motivated by our construction procedure rather than a claim we independently verify; testing it directly \- for instance by varying the matrix language or the substitution rate \- is a natural direction for follow\-up work\.

#### Practical implication\.

For practitioners building hallucination detectors intended for real\-world, code\-mixed traffic, our central finding is reassuring: a detector validated only on clean\-language data is not obviously broken when it encounters Hinglish input, at least for the in\-sentence code\-mixing pattern and language pair we study\. This does not mean such a detector should be deployed without further validation \(see Limitations below\), but it does mean the code\-mixing gap identified in Section[2](https://arxiv.org/html/2609.22138#S2)is, on the evidence here, not the silent failure mode one might have feared\.

#### Interpreting the Hindi/Hinglish AUROC advantage\.

The in\-distribution results in Section[4\.1](https://arxiv.org/html/2609.22138#S4.SS1)show Hindi and Hinglish probes reaching substantially higher AUROC than English probes\. Part of this advantage is likely genuine \- the faithful\-rate results in Section[4\.4](https://arxiv.org/html/2609.22138#S4.SS4)show these models are far less capable in Hindi and Hinglish, and a larger, more separable gap between confident\-and\- correct and confident\-and\-wrong internal states is a plausible consequence of operating further outside a model’s core competence\. However, we caution that severe class imbalance \(as low as 17 faithful examples out of 2,242 for Mistral/Hindi\) makes AUROC estimates in these conditions noisier and potentially more optimistic than the equivalent estimates for the more balanced English condition, even with class\-weighted training and stratified cross\-validation\. We do not believe this fully explains the pattern \- it is consistent across three independently trained models and two independent token positions \- but we flag it as a factor future work should control for, for instance by evaluating on class\-balanced subsamples or reporting confidence intervals over resampled folds\.

## Limitations

#### Synthetic rather than natural code\-mixing\.

Our Hinglish data is synthetically constructed via matrix\-language\-frame word substitution \(Section[3\.1](https://arxiv.org/html/2609.22138#S3.SS1)\), not drawn from naturally\-occurring code\-mixed text\. We attempted to obtain natural Hindi–English code\-mixed QA data from LinCE\([Aguilar et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib5)\)and GLUECoS\([Khanuja et al\., 2020](https://arxiv.org/html/2609.22138#bib.bib6)\)during data collection but were unable to access either at the time of this work\. Naturally\-occurring code\-mixing exhibits patterns \- inconsistent switch\-points, borrowed discourse markers, transliteration variation, speaker\-specific mixing styles \- that our controlled substitution procedure does not capture\. Our central finding \(the signal survives code\-mixing\) should therefore be read as evidence about a controlled, linguistically\-motivated approximation of Hinglish, and validating it on naturally\-occurring code\-mixed corpora is, in our view, the single most important piece of follow\-up work\.

#### Single language pair\.

We study only Hindi–English code\-mixing\. Whether these findings generalize to other code\-mixed language pairs, with different scripts, typological distances, or sociolinguistic mixing patterns, is untested\.

#### Class imbalance\.

As discussed in Section[5](https://arxiv.org/html/2609.22138#S5), Hindi and Hinglish faithful/hallucinated labels are severely imbalanced for two of our three models \(Mistral and Qwen2\.5\), which affects the reliability of AUROC estimates in those conditions despite our use of class\-weighted training, stratified cross\-validation, and minimum class\-count thresholds for excluding underpowered conditions\.

#### Data loss in extraction\.

Hidden\-state extraction for Llama\-3\.1\-8B completed for 5,544 of 5,674 rows \(∼\\sim97\.7%\), with the remainder excluded due to an occasional out\-of\-memory condition during single\-row forward\-pass extraction\. We do not expect this small, effectively random subset of missing rows to bias our results, but note it for completeness\.

#### Token\-F1 as a labeling proxy\.

We label hallucination via token\-F1 against a single gold answer, following standard practice in prior hidden\-state probing work\([Aiersilan, 2026](https://arxiv.org/html/2609.22138#bib.bib4)\)\. This is a proxy for genuine faithfulness, not a human\-verified ground truth: it can penalize correct answers phrased differently from the reference, and cannot detect hallucinated content in answers that happen to contain the correct key term\. We expect this labeling noise to affect all three language conditions similarly rather than to selectively bias the Hindi\-vs\-Hinglish comparison central to our claims, but it remains a limitation of the evaluation protocol we inherit from prior work rather than one we resolve\.

## Ethics Statement

This work aims to improve hallucination detection for a large, currently underserved population of Hindi–English bilingual LLM users, and we believe better detection tools for this population have clear positive potential\. At the same time, we emphasize that hidden\-state probes of the kind we study are not perfect classifiers and should not be treated as a sole or final safety mechanism in any deployed system; a probe reporting high confidence that an answer is faithful does not guarantee correctness, and the risks of over\-trusting such a signal, particularly in high\-stakes applications, should be weighed carefully\. Our synthetic Hinglish dataset is derived entirely from existing, publicly available QA benchmarks \(XQuAD, IndicQA\) and a publicly available bilingual dictionary \(MUSE\); it does not involve human subjects or personally identifiable data, and we release it to support, not substitute for, further validation on naturally\-occurring code\-mixed data\.

## 6Conclusion

We present the first evaluation of hidden\-state hallucination probing under within\-sentence code\-mixed input, addressing a gap left open by every prior study of this probing paradigm we are aware of\. Across three open\-weight 7–8B LLMs, two token positions, and a controlled synthetic Hindi–English Hinglish benchmark, we find that the truthfulness signal these probes rely on survives code\-mixing well: cross\-lingual transfer AUROC to Hinglish ranges from 0\.876 to 0\.991, with a gap to in\-distribution performance under 0\.05 AUROC in most conditions, and probes trained on clean Hindi transfer more reliably than probes trained on clean English across every model and position we test\. As an independent finding, all three models we evaluate hallucinate substantially more often in Hindi and Hinglish than in English on matched questions, underscoring the practical importance of reliable hallucination detection for this language setting\. We release our code and synthetic Hinglish QA benchmark, and identify validation on naturally\-occurring code\-mixed corpora as the clearest direction for future work\.

## References

- Aguilaret al\.\(2020\)G\. Aguilar, S\. Kar, and T\. SolorioLinCE: a centralized benchmark for linguistic code\-switching evaluation\.InProceedings of the Twelfth Language Resources and Evaluation Conference,Marseille, France,pp\. 1803–1813\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2609.22138#S3.SS1.SSS0.Px2.p1.1),[Synthetic rather than natural code\-mixing\.](https://arxiv.org/html/2609.22138#Sx1.SS0.SSS0.Px1.p1.1)\.
- AI@Meta \(2024\)AI@MetaThe Llama 3 herd of models\.InarXiv preprint arXiv:2407\.21783,Cited by:[§3\.2](https://arxiv.org/html/2609.22138#S3.SS2.p1.1)\.
- Aiersilan \(2026\)A\. AiersilanHallucination is linearly decodable from mid\-layer hidden states in quantized LLMs\.arXiv preprint arXiv:2606\.02628\.Cited by:[§1](https://arxiv.org/html/2609.22138#S1.p2.1),[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2609.22138#S3.I1.i3.p1.1),[§3\.2](https://arxiv.org/html/2609.22138#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.22138#S4.SS1.p1.1),[§4\.3](https://arxiv.org/html/2609.22138#S4.SS3.p1.1),[Token\-F1 as a labeling proxy\.](https://arxiv.org/html/2609.22138#Sx1.SS0.SSS0.Px5.p1.1)\.
- Alviet al\.\(2026\)R\. Alvi, N\. L\. Sayeedi, and Md\. F\. A\. SayeediMultiHaluDet: multilingual hallucination detection via LLM hidden state probing\.InProceedings of the Workshop on Multilingual and Equitable Language Models \(MELLM 2026\),Cited by:[§1](https://arxiv.org/html/2609.22138#S1.p2.1),[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px1.p1.1)\.
- Artetxeet al\.\(2020\)M\. Artetxe, S\. Ruder, and D\. YogatamaOn the cross\-lingual transferability of monolingual representations\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2609.22138#S3.SS1.SSS0.Px1.p1.1)\.
- Azaria and Mitchell \(2023\)A\. Azaria and T\. MitchellThe internal state of an LLM knows when it’s lying\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 967–976\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px1.p1.1)\.
- Conneauet al\.\(2018\)A\. Conneau, G\. Lample, M\. Ranzato, L\. Denoyer, and H\. JégouWord translation without parallel data\.arXiv preprint arXiv:1710\.04087\.Cited by:[§3\.1](https://arxiv.org/html/2609.22138#S3.SS1.SSS0.Px2.p1.1)\.
- Doddapaneniet al\.\(2023\)S\. Doddapaneni, R\. Aralikatte, G\. Ramesh, S\. Goyal, M\. M\. Khapra, A\. Kunchukuttan, and P\. KumarTowards leaving no Indic language behind: building monolingual corpora, benchmark and models for Indic languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2609.22138#S3.SS1.SSS0.Px1.p1.1)\.
- Khanujaet al\.\(2020\)S\. Khanuja, S\. Dandapat, A\. Srinivasan, S\. Sitaram, and M\. ChoudhuryGLUECoS: an evaluation benchmark for code\-switched NLP\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 3575–3585\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2609.22138#S3.SS1.SSS0.Px2.p1.1),[Synthetic rather than natural code\-mixing\.](https://arxiv.org/html/2609.22138#Sx1.SS0.SSS0.Px1.p1.1)\.
- Mrykhin and Malykh \(2026\)Z\. Mrykhin and V\. MalykhPrompt embedding probes \(PEP\): hallucination detection in LLMs from hidden states\.arXiv preprint arXiv:2608\.08024\.Cited by:[§1](https://arxiv.org/html/2609.22138#S1.p2.1),[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px1.p1.1)\.
- Myers\-Scotton \(1993\)C\. Myers\-ScottonDuelling languages: grammatical structure in codeswitching\.Clarendon Press,Oxford\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2609.22138#S3.SS1.SSS0.Px2.p1.1)\.
- Orgadet al\.\(2024\)H\. Orgad, M\. Toker, Z\. Gekhman, R\. Reichart, I\. Szpektor, H\. Kotek, and Y\. BelinkovLLMs know more than they show: on the intrinsic representation of LLM hallucinations\.arXiv preprint arXiv:2410\.02707\.Cited by:[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px1.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100,000\+ questions for machine comprehension of text\.arXiv preprint arXiv:1606\.05250\.Cited by:[§3\.2](https://arxiv.org/html/2609.22138#S3.SS2.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, X\. Hu, H\. Zhang, J\. Zhang, and X\. WanICR probe: tracking hidden state dynamics for reliable hallucination detection in LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 17986–18002\.Cited by:[§1](https://arxiv.org/html/2609.22138#S1.p2.1),[§2](https://arxiv.org/html/2609.22138#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAppendix

### A\.1Prompt template

We use the following system prompt, combined with each model’s native chat template, for all generation:“You are a helpful assistant that answers questions based only on the given context\. Give a short, direct answer only \- no explanation, no extra text, no additional questions\.”The user turn contains the context, question, and an explicitAnswer:cue\. Decoding is greedy \(temperature\-independent\) for all models and all conditions\.

### A\.2Compute and environment

Data generation and hidden\-state extraction were run on an NVIDIA RTX A6000 \(48GB\) and, for one model, on Kaggle’s dual T4 \(16GB×\\times2\) instances using 4\-bit quantization\. Probe training and analysis are CPU\-only and complete in minutes given the extracted hidden states\. All three LLMs were run in their publicly released instruction\-tuned form with no additional fine\-tuning\.

### A\.3Full per\-layer results

Complete per\-layer AUROC values for all \(model, language, position\) triples, underlying Figure[1](https://arxiv.org/html/2609.22138#S4.F1), along with the full transfer results for both token positions underlying Table[1](https://arxiv.org/html/2609.22138#S4.T1)and Table[2](https://arxiv.org/html/2609.22138#S4.T2), are released with our code and data\.

### A\.4Full linear\-vs\-MLP comparison

Table[4](https://arxiv.org/html/2609.22138#A1.T4)reports the complete linear\-vs\-MLP comparison for all three models, all three language conditions, and both token positions \(Table[3](https://arxiv.org/html/2609.22138#S4.T3)in the main text shows only the prompt\-position subset\)\. The answer\-position results follow the same overall pattern: MLP probes rarely improve meaningfully on the linear baseline, and the largest MLP underperformance again occurs for Mistral on Hinglish, though the drop is smaller at the answer position than at the prompt position\.

Table 4:Full linear\-vs\-MLP probe AUROC comparison at each condition’s best in\-distribution layer, both token positions\. “cm” denotes Hinglish\.

相似文章

检测性差距:大语言模型幻觉检测中隐藏的异质性

arXiv cs.CL

本文揭示了幻觉检测中的“检测性差距”:幻觉可分为高一致性(Ghost)与低一致性(Flickering)两种模式,二者之间存在 0.35–0.46 的 AUC 差距。该差距在四个模型与三个事实问答数据集上持续存在,即使在固定模式归属并采用更严格的基于轨迹的测试后依然如此。作者指出,聚合层面的检测指标掩盖了模型相关的异质性,并呼吁采用基于模式条件的评估方法。

PARALLAX: 区分真实幻觉检测与基准构建伪影

arXiv cs.CL

本文揭示了LLM幻觉检测领域报道的许多进展实际上源于基准构建伪影,其中真实答案被嵌入到提示中,使得简单的文本相似度基线方法能够获得近乎完美的分数。通过大规模受控评估,作者证明,在适当控制下,大多数方法的表现接近随机水平,除了对上层隐藏状态的监督探针(如SAPLMA)以及他们提出的DRIFT。

长文本幻觉检测的健全性检验

arXiv cs.CL

本文介绍了一种受控不变性方法以及两种测试(Force 和 Remove),旨在确定大语言模型(LLM)幻觉检测器是依赖于推理过程还是最终答案的特征。研究提出了 TRACT,这是一种基于词汇特征的轻量级评分器,证明了其在不依赖答案层面线索的情况下仍能保持鲁棒的性能。