CamSpokenNLI:用于喀麦隆皮钦英语与喀麦隆法英混合语跨变体迁移评估的试点资源

arXiv cs.CL 论文

摘要

本文介绍了 CamSpokenNLI,这是一个涵盖喀麦隆皮钦英语和喀麦隆法英混合语、包含 228 个条目的试点口语自然语言推理(NLI)数据集,并评估了 XLM-RoBERTa 的跨语言变体迁移能力和 Whisper 的转录表现。结果显示,准确率不高,评估结果对对应样本泄漏和仅使用假设的基线较为敏感,且零样本自动语音识别(ASR)表现不佳,凸显了针对这些语言变体调整评估方法和语音识别系统的必要性。

arXiv:2611.17573v1 Announce Type: new Abstract: Cameroon Pidgin English (a West African Pidgin English variety) and Camfranglais are widely used in Cameroon, yet to our knowledge, neither has a dedicated spoken Natural Language Inference (NLI) resource. We introduce CamSpokenNLI, a pilot dataset of 114 scenario-matched spoken NLI items in each variety (228 in total), recorded by one native speaker from Buea. Each item pairs premise audio and text with a hypothesis, an NLI label, and English glosses. The author assigned labels, so the dataset is an exploratory resource rather than a mature benchmark. The NLI experiments use the reference text; the audio is evaluated separately. XLM-RoBERTa baselines with five-fold item-level cross-validation reach 44.7% on Pidgin and 46.2% on Camfranglais (chance 33.3%). Still, neither is statistically distinguishable from hypothesis-only input, so hypothesis wording may account for much of the accuracy. Under disjoint-item cross-variety transfer, where a test item's counterpart is never seen in training, accuracy is 40.1% (Pidgin to Camfranglais) and 45.6% (Camfranglais to Pidgin). A counterpart-leak audit raises this to 60-64%, illustrating how counterpart leakage can overstate transfer. Zero-shot Whisper transcription of the recordings performs poorly: the best word error rates are 66.6% for Pidgin and 45.5% for Camfranglais, and with automatic language identification, Whisper Large-v3 labels 93 of 114 Pidgin recordings as Yoruba or Swahili. Together, the results suggest that such resources need disjoint-item evaluation and hypothesis-only controls, and that speech recognition must be adapted to these varieties before spoken NLI can be tested end to end.
查看原文
查看缓存全文

缓存时间: 2026/11/16 08:19

# CamSpokenNLI: A Pilot Resource for Cross-Variety Transfer Evaluation in Cameroon Pidgin English and Camfranglais
Source: [https://arxiv.org/abs/2611.17573](https://arxiv.org/abs/2611.17573)
[View PDF](https://arxiv.org/pdf/2611.17573)

> Abstract:Cameroon Pidgin English \(a West African Pidgin English variety\) and Camfranglais are widely used in Cameroon, yet to our knowledge, neither has a dedicated spoken Natural Language Inference \(NLI\) resource\. We introduce CamSpokenNLI, a pilot dataset of 114 scenario\-matched spoken NLI items in each variety \(228 in total\), recorded by one native speaker from Buea\. Each item pairs premise audio and text with a hypothesis, an NLI label, and English glosses\. The author assigned labels, so the dataset is an exploratory resource rather than a mature benchmark\. The NLI experiments use the reference text; the audio is evaluated separately\. XLM\-RoBERTa baselines with five\-fold item\-level cross\-validation reach 44\.7% on Pidgin and 46\.2% on Camfranglais \(chance 33\.3%\)\. Still, neither is statistically distinguishable from hypothesis\-only input, so hypothesis wording may account for much of the accuracy\. Under disjoint\-item cross\-variety transfer, where a test item's counterpart is never seen in training, accuracy is 40\.1% \(Pidgin to Camfranglais\) and 45\.6% \(Camfranglais to Pidgin\)\. A counterpart\-leak audit raises this to 60\-64%, illustrating how counterpart leakage can overstate transfer\. Zero\-shot Whisper transcription of the recordings performs poorly: the best word error rates are 66\.6% for Pidgin and 45\.5% for Camfranglais, and with automatic language identification, Whisper Large\-v3 labels 93 of 114 Pidgin recordings as Yoruba or Swahili\. Together, the results suggest that such resources need disjoint\-item evaluation and hypothesis\-only controls, and that speech recognition must be adapted to these varieties before spoken NLI can be tested end to end\.

## Submission history

From: Tiani Pekins Ebika \[[view email](https://arxiv.org/show-email/a968429e/2611.17573)\] **\[v1\]**Thu, 12 Nov 2026 22:51:31 UTC \(95 KB\)

相似文章

非洲语言NLI评估的样本量缩放

arXiv cs.CL

本文利用AfriXNLI基准测试,研究标注数据大小对16种非洲语言自然语言推理性能的影响。结果表明,缩放行为对语言敏感且通常非单调,挑战了常见的单调改进假设,并强调了需要为特定语言创建数据集以及更强的多语言策略。

衡量跨语言理解差距:证据语言如何影响语言模型的理解

arXiv cs.CL

本文引入了跨语言理解差距(CLCG)指标,用于衡量当内容以非英语语言呈现时,LLM响应质量的下降程度。通过对18种语言和多个模型的评估,研究发现性能显著下降,尤其是在低资源语言上,这对以英语为中心的能力迁移假设提出了质疑。