MixLID: Token-Level Language Identification in Mixed-Language Text

Hugging Face Daily Papers Papers
language-identification multilingual code-switching token-classification nlp open-source

Summary

论文提出MixLID，一个支持111种语言-文字组合的token级混合语言识别模型，采用两阶段训练（先单语再混合语言数据），在单语和混合语言识别基准上均超越现有基线，模型已在Hugging Face开源。

Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at https://huggingface.co/nlpai-lab/MixLID.
Original Article
View Cached Full Text

Cached at: 10/14/26, 02:52 PM

Paper page - MixLID: Token-Level Language Identification in Mixed-Language Text

Source: https://huggingface.co/papers/2610.16601

Abstract

Processingmultilingualtextrequiresbothidentifyingabroadrangeoflanguagesanddistinguishingthelanguageofeachtokenininputscontainingmultiplelanguages.WeintroduceMixLID,atoken-levellanguageidentificationmodelsupporting111language-scriptcombinations.MixLIDfollowsatwo-stagetrainingprocess:itfirstlearnstodistinguishindividuallanguagesusingthelanguagelabelsofmonolingualexamples,thencontinuestrainingonmixed-languagedata.Thesecondstageusesconcatenatedexamplesfromexistingcode-switchingdataalongsideaugmentedexamplescreatedthroughwordsubstitutionbasedoncorrespondencesbetweenEnglishsourcewordsandloanwords.Formonolingualidentification,MixLIDoutperformsallbaselinesinbothaccuracyandmacroF1onfourbenchmarkswithinthelanguagecoveragesharedwitheachbaseline.Formixed-languageidentification,itoutperformsthebaselinesinidentifyingthelanguageofindividualtokensoneveryevaluatedlanguagepair.MixLIDachieveshighperformanceacrossabroadrangeoflanguagesinbothmonolingualandmixed-languagesettings.Themodelispubliclyavailableathttps://huggingface.co/nlpai-lab/MixLID.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.16601

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### nlpai-lab/MixLID Token Classification• 0.1B• Updatedabout 3 hours ago • 54 • 18

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.16601 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Data Mixing for Large Language Models Pretraining: A Survey and Outlook

arXiv cs.CL

This paper presents a comprehensive survey of data mixing methods for LLM pretraining, formalizing the problem as bilevel optimization and introducing a taxonomy that distinguishes static (rule-based and learning-based) from dynamic (adaptive and externally guided) mixing approaches. The authors analyze trade-offs, identify cross-cutting challenges, and outline future research directions including finer-grained domain partitioning and pipeline-aware designs.