CamSpokenNLI:用于喀麦隆皮钦英语与喀麦隆法英混合语跨变体迁移评估的试点资源
摘要
本文介绍了 CamSpokenNLI,这是一个涵盖喀麦隆皮钦英语和喀麦隆法英混合语、包含 228 个条目的试点口语自然语言推理(NLI)数据集,并评估了 XLM-RoBERTa 的跨语言变体迁移能力和 Whisper 的转录表现。结果显示,准确率不高,评估结果对对应样本泄漏和仅使用假设的基线较为敏感,且零样本自动语音识别(ASR)表现不佳,凸显了针对这些语言变体调整评估方法和语音识别系统的必要性。
查看缓存全文
缓存时间: 2026/11/16 08:19
# CamSpokenNLI: A Pilot Resource for Cross-Variety Transfer Evaluation in Cameroon Pidgin English and Camfranglais Source: [https://arxiv.org/abs/2611.17573](https://arxiv.org/abs/2611.17573) [View PDF](https://arxiv.org/pdf/2611.17573) > Abstract:Cameroon Pidgin English \(a West African Pidgin English variety\) and Camfranglais are widely used in Cameroon, yet to our knowledge, neither has a dedicated spoken Natural Language Inference \(NLI\) resource\. We introduce CamSpokenNLI, a pilot dataset of 114 scenario\-matched spoken NLI items in each variety \(228 in total\), recorded by one native speaker from Buea\. Each item pairs premise audio and text with a hypothesis, an NLI label, and English glosses\. The author assigned labels, so the dataset is an exploratory resource rather than a mature benchmark\. The NLI experiments use the reference text; the audio is evaluated separately\. XLM\-RoBERTa baselines with five\-fold item\-level cross\-validation reach 44\.7% on Pidgin and 46\.2% on Camfranglais \(chance 33\.3%\)\. Still, neither is statistically distinguishable from hypothesis\-only input, so hypothesis wording may account for much of the accuracy\. Under disjoint\-item cross\-variety transfer, where a test item's counterpart is never seen in training, accuracy is 40\.1% \(Pidgin to Camfranglais\) and 45\.6% \(Camfranglais to Pidgin\)\. A counterpart\-leak audit raises this to 60\-64%, illustrating how counterpart leakage can overstate transfer\. Zero\-shot Whisper transcription of the recordings performs poorly: the best word error rates are 66\.6% for Pidgin and 45\.5% for Camfranglais, and with automatic language identification, Whisper Large\-v3 labels 93 of 114 Pidgin recordings as Yoruba or Swahili\. Together, the results suggest that such resources need disjoint\-item evaluation and hypothesis\-only controls, and that speech recognition must be adapted to these varieties before spoken NLI can be tested end to end\. ## Submission history From: Tiani Pekins Ebika \[[view email](https://arxiv.org/show-email/a968429e/2611.17573)\] **\[v1\]**Thu, 12 Nov 2026 22:51:31 UTC \(95 KB\)
相似文章
非洲语言NLI评估的样本量缩放
本文利用AfriXNLI基准测试,研究标注数据大小对16种非洲语言自然语言推理性能的影响。结果表明,缩放行为对语言敏感且通常非单调,挑战了常见的单调改进假设,并强调了需要为特定语言创建数据集以及更强的多语言策略。
从单语到多语:评估Mamba在南非语言中的ASR性能
本文评估了Mamba状态空间模型在七种南非语言上的ASR性能,发现其在资源更少的情况下达到了与Conformer相当的准确率,并探讨了多语训练策略和低资源场景。
语言特异性与领域多样性:Bangla医疗命名实体识别中的Transformer基准测试
本文评测了Transformer模型在Bangla医疗命名实体识别任务上的表现,结果显示,经过微调的XLM-RoBERTa模型取得了最优性能,超越了语言特定模型,突显了在资源稀缺场景下领域多样性的重要性。
衡量跨语言理解差距:证据语言如何影响语言模型的理解
本文引入了跨语言理解差距(CLCG)指标,用于衡量当内容以非英语语言呈现时,LLM响应质量的下降程度。通过对18种语言和多个模型的评估,研究发现性能显著下降,尤其是在低资源语言上,这对以英语为中心的能力迁移假设提出了质疑。
哪些语言最适合迁移到瓦尔皮里语?一项基于相似度的低资源语音识别研究
本文研究了低资源自动语音识别(ASR)中跨语言迁移的问题,针对瓦尔皮里语提出了一种结合声学与语言特征的相似度框架,以选择最优源语言。实验表明,像阿萨姆语和印地语这样声学相似的语言能显著降低词错误率和字符错误率。