LLMs for automatic annotation of Mandarin narrative transcripts
Summary
This paper evaluates LLMs for automatically annotating narrative macrostructure in spoken Mandarin, finding that the best model achieves near-human reliability while reducing annotation time by 65%, though performance degrades on semantically complex or lexically diverse narratives.
View Cached Full Text
Cached at: 05/19/26, 06:38 AM
# LLMs for automatic annotation of Mandarin narrative transcripts Source: [https://arxiv.org/abs/2605.17205](https://arxiv.org/abs/2605.17205) [View PDF](https://arxiv.org/pdf/2605.17205) > Abstract:Linguistic annotation of transcribed speech is essential for research in language acquisition, language disorders, and sociolinguistics, yet remains labor\-intensive and time\-consuming\. While Large Language Models \(LLMs\) have shown promise in automating annotation tasks, their ability to handle complex discourse\-level annotation in non\-English languages remains understudied\. This study evaluates whether LLMs can reliably annotate narrative macrostructure\-the hierarchical organization of story grammar elements\-in spoken Mandarin, using the Multilingual Assessment Instrument for Narratives \(MAIN\) as a testbed\. We compared four LLMs against trained human annotators on narratives produced by children, young adults, and older adults\. The best\-performing model achieved agreement with human raters \(k=\.794\) approaching human\-human reliability levels \(k=\.872\) while reducing annotation time by 65%, whereas the locally deployable lightweight model performed substantially worse\. Annotation difficulty varied systematically by macrostructure element type, with categories requiring subtle semantic differentiation posing persistent challenges\. Furthermore, model reliability decreased on young adult narratives, which exhibited greater lexical variation, semantic ambiguity, and multi\-element integration within single utterances\. These findings suggest that LLMs can effectively support discourse\-level annotation in non\-English spoken corpora, while highlighting the continued need for human oversight in semantically complex tasks\. Our prompt templates are open sourced for future use\. ## Submission history From: Qingwen Zhao \[[view email](https://arxiv.org/show-email/bd09e29a/2605.17205)\] **\[v1\]**Sun, 17 May 2026 00:37:25 UTC \(582 KB\)
Similar Articles
Exploring the Capability Boundaries of LLMs in Mastering Chinese Chouxiang Language
This paper introduces Mouse, a specialized benchmark for evaluating LLMs on Chinese Chouxiang Language tasks across six NLP domains, revealing that current state-of-the-art models have significant limitations with this subcultural internet language despite performing well on contextual understanding tasks.
Challenges in annotations by humans and LLMs: A case study of evaluative language
This paper compares human linguists in training, a trained linguist, and LLMs on annotating evaluative language using Appraisal theory, finding that LLMs achieve strong performance and can assist in complex annotation tasks.
Observational Equivalence of LLM and Human Annotation
This paper demonstrates that large language models (LLMs) achieve annotation quality equivalent to human experts, suggesting the main challenge is minimizing ambiguity in coding rules rather than choosing between human and machine coders.
Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts
This paper proposes a hybrid framework combining LLM summarization, a linguistically grounded Expert Index, and STAC classification to generate causal graphs from narrative texts, outperforming GPT-4o and Claude 3.5 on benchmark stories.
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
This paper evaluates how well LLMs can replace human annotators for machine translation quality assessment across MQM and ESA schemes, 70 language pairs, and WMT23/WMT25 data, finding that both humans and LLMs remain unreliable in many settings and proposing human-LLM collaborative annotation pipelines.