Dango: A Strictly L1-Only Large Language Model for Studying Second Language Acquisition

# A Strictly L1-Only Large Language Model for Studying Second Language Acquisition
Source: [https://arxiv.org/html/2606.19170](https://arxiv.org/html/2606.19170)
Shiho Matta1Yin Jou Huang1Fei Cheng1Takashi Kodama2 Hirokazu Kiyomaru2Yugo Murawaki1 1Kyoto University2NII\-LLMC \{matta,huang,feicheng,murawaki\}@nlp\.ist\.i\.kyoto\-u\.ac\.jp \{tkodama,kiyomaru\}@nii\.ac\.jp

###### Abstract

We introduceDango, a 1\.8B\-parameter large language model designed for controlled studies of L1→L2 \(Japanese→English\) transfer in second language acquisition \(SLA\)\. While previous studies have explored SLA in language models, they have predominantly relied on smaller or non\-decoder models, limiting their ability to generate open\-ended text and reducing their suitability as practical L2 simulators\. We identify a key challenge when scaling models to this size: L2 contamination within the "monolingual" pretraining corpus used for L1 acquisition\. To address this, we propose a filtering method to reduce premature exposure to English while preserving realistic, minimal exposure\. We then fine\-tune the model on LLM\-generated L2\-learning lessons to simulate the L2 acquisition process\. Our evaluations confirm that Dango develops human\-like L2 production patterns, outperforming both unfiltered and standard multilingual baselines\. We release the model, data, and code to facilitate reproducible computational SLA research and learner\-facing applications\.111We will soon release the model, data, and code\.

Dango: A Strictly L1\-Only Large Language Model for Studying Second Language Acquisition

Shiho Matta1Yin Jou Huang1Fei Cheng1Takashi Kodama2Hirokazu Kiyomaru2Yugo Murawaki11Kyoto University2NII\-LLMC\{matta,huang,feicheng,murawaki\}@nlp\.ist\.i\.kyoto\-u\.ac\.jp\{tkodama,kiyomaru\}@nii\.ac\.jp

## 1Introduction

Second language acquisition \(SLA\) refers to the learning of a second language \(L2\) after a first language \(L1\) has already been acquired natively\. In humans, the influence of L1 can facilitate L2 learning in some cases and interfere with it in others, a phenomenon widely known as the language transfer effectLado \([1957](https://arxiv.org/html/2606.19170#bib.bib26)\); Odlin \([1989](https://arxiv.org/html/2606.19170#bib.bib27)\)\. This may result in overuse, avoidance, and misuse patterns of certain linguistic features in a speaker’s L2 productionJarvis and Pavlenko \([2008](https://arxiv.org/html/2606.19170#bib.bib44)\)\. For example, because Japanese lacks a grammatical article system, Japanese\-speaking learners of English often show difficulty with English article useMurakami and Alexopoulou \([2015](https://arxiv.org/html/2606.19170#bib.bib39)\)\. This can lead to article\-omission errors such as “I went to library” instead of “I went to the library\.”

![Refer to caption](https://arxiv.org/html/2606.19170v1/x1.png)Figure 1:English contamination examples: severe case of translation pairs versus minor, typical English exposure for Japanese speakers\.As transformer\-based language models \(LMs\) have recently shown strong performance on many linguistic tasksLaiet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib25)\), many studies have investigated whether they also exhibit such human\-like transfer effectsObaet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib5)\); Aoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\); Yadavalliet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib10)\); Constantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)\. These studies train LMs from scratch first on L1 and then on L2, thereby mimicking the order in which humans learn languages\. Transfer effects are then evaluated, for example, by testing the model’s grammatical judgments on the L2 and comparing them with human tendenciesYadavalliet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib10)\)\.

Although these studies provide valuable insights into transfer effects in LMs, their use as practical simulators of L2 speakers remains limited\. This is largely due to limitations of architecture and scale\. Prior work has used either encoder\-only modelsObaet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib5)\); Yadavalliet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib10)\); Constantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)or small decoder\-only models, such as the 137M\-parameter GPT\-2 inConstantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)\. As a result, these models are less suitable for robust open\-ended or interactive generation, and for potential applications such as L2 learner supportLyuet al\.\([2024](https://arxiv.org/html/2606.19170#bib.bib15)\)and teacher training with virtual L2 speakersVasmatzoglou and Ní Chiaráin \([2020](https://arxiv.org/html/2606.19170#bib.bib14)\)\.

To equip future SLA research with a more modern and capable model, and to enable broader practical applications, we releaseDango, a Llama\-2\-like decoder\-only transformer trained in an L1→L2 manner at a much larger scale\. Dango has 1\.8B parameters and is pretrained on far more data than prior work, with up to 100B L1\-stage tokens\.

To train an LLM at this size, web\-crawled text is the only viable data sourceHoffmannet al\.\([2022](https://arxiv.org/html/2606.19170#bib.bib18)\)\. However, the Japanese web corpus we used for L1 acquisition contained substantial accidental English exposure\. We call this problemL2 contamination\. Although Japanese speakers naturally see English through loanwords, proper nouns, and code\-switchingHyde \([2002](https://arxiv.org/html/2606.19170#bib.bib32)\); Stanlaw \([2004](https://arxiv.org/html/2606.19170#bib.bib16)\), systematic acquisition of English is relatively rare outside classroom\-like educational settingsBarrs \([2020](https://arxiv.org/html/2606.19170#bib.bib33)\)\. It is therefore natural to reshape the corpus into a state closer to a typical Japanese speaker’s environment: removing structured English content, such as in the worst case, translation pairs, while retaining minimal English such as brand names and simple phrases \(Figure[1](https://arxiv.org/html/2606.19170#S1.F1)\)\. We designed a filtering pipeline that removes excessive English exposure, reducing the corpus size by 30% \(Figure[2](https://arxiv.org/html/2606.19170#S1.F2)\)\.

In the L2 acquisition stage, we fine\-tune Dango on LLM\-synthesized textbook\-like lessons\. The lessons follow a gradual difficulty progression, as they are based on an English seed word drawn from a word list with difficulty labels\. The lessons are anchored in both Japanese and English, containing \(i\) a Japanese explanation of the seed word, \(ii\) an example English sentence using the word, paired with a Japanese translation, and \(iii\) a brief Japanese commentary highlighting relevant grammar or usage\. In this way, Dango is exposed to English while keeping instruction and metalinguistic reasoning in its L1, which mirrors common textbook learning practiceNakayama \([2022](https://arxiv.org/html/2606.19170#bib.bib34)\)\.

Our experiments show that Dango acquires English generation abilities, including the capacity to produce long English sentences absent from its strictly filtered pretraining data\. Evaluations further show that, among all methods, Dango’s English outputs most closely resemble those of human Japanese learners of English, outperforming unfiltered and multilingual baselines and matching prompted GPT\-5\.5\. These results provide evidence of successful language transfer\. We release Dango’s code and checkpoints to support controlled SLA research and practical applications\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x2.png)Figure 2:An overview of our proposed methodology\.
## 2Related Work

### 2\.1Second Language Acquisition \(SLA\) with Language Models \(LMs\)

##### Modeling SLA by Training in an L1→L2 Setup\.

Previous studies have trained LMs on L1 and L2 sequentially to model human L2 acquisition\.Obaet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib5)\)trained 18M\-parameter encoder\-only models and analyzed how different L1 pretraining languages shape the trajectory and ultimate success of L2 grammatical generalization\.Yadavalliet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib10)\)also trained 110M\-parameter encoder\-only models, but aimed to isolate positive and negative cross\-linguistic transfer\.Aoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\)trained 11M\-parameter \(our estimation\) GPT\-2 models as L2 English speakers, investigating transfer effects and showing that LM surprisal helps predict L2 English readers’ word\-by\-word reading times\.Constantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)trained 125M\-parameter encoder\-only and 137M\-parameter decoder\-only models, but their focus was on recreating the critical period effect observed in humans rather than directly examining transfer effects\.

##### Training Data\.

Training data is central to modeling SLA\. Some work has attempted to create a more human\-like setting by using data derived from human speech\. For example,Yadavalliet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib10)\)used child\-directed speech, andConstantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)partially used spoken\-language data\. However, these data sources are limited in size, andConstantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)had to include web text to obtain sufficient data and cross\-linguistic coverage\. On the other hand,Obaet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib5)\)andAoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\)relied on CC\-100Conneauet al\.\([2020](https://arxiv.org/html/2606.19170#bib.bib36)\); Wenzeket al\.\([2020](https://arxiv.org/html/2606.19170#bib.bib35)\)web\-crawled data for L1 training\.

##### Evaluating Transfer Effect\.

Obaet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib5)\),Aoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\),Yadavalliet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib10)\), andConstantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)used BLiMPWarstadtet al\.\([2020](https://arxiv.org/html/2606.19170#bib.bib7)\)to measure the L2 \(English\) grammatical generalization\. Other probes include perplexityAoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\); Constantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\), model surprisalAoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\)and knowledge\-based multiple\-choice tasksConstantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\)\.

However, because these probes rely entirely on the predefined inputs within existing benchmarks, they do not truly test a model’s generation ability\. To address this limitation, we evaluate the transfer effect in Dango’s English text production using the automatic LLM\-as\-a\-judge framework introduced byGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\)\. We directly annotate linguistic features in the model’s output to compare their distribution with authentic human L2 productions\.

### 2\.2Controlling Knowledge Exposure during Pretraining

##### LLMs with Knowledge Cutoff\.

Recent efforts highlight the necessity of strictly constrained pretraining environments to isolate specific phenomena from the massive data contamination inherent in standard LLMs\. For instance, TimeCapsule LLM is trained exclusively on texts from constrained historical periods to accurately emulate the vocabulary and worldview of that era without the intrusion of modern concepts\.222[https://github\.com/haykgrigo3/TimeCapsuleLLM](https://github.com/haykgrigo3/TimeCapsuleLLM)Similarly, History LLMs are trained from scratch using strict temporal knowledge cutoffs to completely prevent the leakage of future knowledge\.333[https://github\.com/DGoettlich/history\-llms](https://github.com/DGoettlich/history-llms)These projects demonstrate that true behavioral simulation requires a verifiable, uncontaminated training baseline\.

##### L2 Contamination in Web Corpora\.

L2 contamination matters in controlled studies of second\-language acquisition and transfer effects\. Because language identification is imperfect, corpora intended to be non\-English often still contain English dataCaswellet al\.\([2020](https://arxiv.org/html/2606.19170#bib.bib37)\)\. In controlled experiments, such accidental exposure can be problematic because it may introduce unintended cross\-linguistic transferBlevins and Zettlemoyer \([2022](https://arxiv.org/html/2606.19170#bib.bib38)\)\. This issue may also apply to prior work that used web corpora for trainingObaet al\.\([2023](https://arxiv.org/html/2606.19170#bib.bib5)\); Aoyama and Schneider \([2024](https://arxiv.org/html/2606.19170#bib.bib6)\); Constantinescuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib30)\), although any resulting failures may have been too subtle to detect at their scale\. To our knowledge, our work is the first to directly address the L2 contamination problem in computational SLA\.

## 3Methodology

The training of Dango consists of two stages \(Figure[2](https://arxiv.org/html/2606.19170#S1.F2)\)\. First, we model L1 \(Japanese\) acquisition by pretraining a decoder\-only language model on a strictly filtered Japanese corpus \(§[3\.1](https://arxiv.org/html/2606.19170#S3.SS1)\), which yields theDango\-pretrainedmodel\. Second, we model L2 \(English\) acquisition by fine\-tuning the pretrained model on LLM\-generated L2 learning data, yieldingDango\(§[3\.2](https://arxiv.org/html/2606.19170#S3.SS2)\)\.

### 3\.1L1 Acquisition

We conducted filtering to remove non\-Japanese exposures in a Japanese corpus and trained a decoder\-only transformer model on the resulting data\.

#### 3\.1\.1Corpus for L1 Acquisition and Filtering

To bootstrap our study from an established baseline, we adopted llm\-jp\-corpus\-v3, the training corpus released by the llm\-jp project\.444[https://gitlab\.llm\-jp\.nii\.ac\.jp/datasets/llm\-jp\-corpus\-v3/\-/tree/main](https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-corpus-v3/-/tree/main)The full corpus is multilingual, containing Japanese, English, Chinese, Korean, and code data\. In this work, we used only the Japanese portion, which is mainly composed of Japanese Common Crawl data555[https://commoncrawl\.org](https://commoncrawl.org/)\.and PDF data from the National Diet Library’s Web Archiving Project \(WARP\)\.666National Diet Library \(Japan\)\. NDL Web Archiving Project \(WARP\),[https://warp\.ndl\.go\.jp/](https://warp.ndl.go.jp/)\.Nevertheless, we observed that this subset still included a non\-negligible amount of English text\.

We built a filtering pipeline to limit exposure to English grammar and long, structured English sentences that could confound L1→L2 transferBarrs \([2020](https://arxiv.org/html/2606.19170#bib.bib33)\); Blevins and Zettlemoyer \([2022](https://arxiv.org/html/2606.19170#bib.bib38)\), while retaining benign English signals common in Japanese text, such as everyday words, phrases, and proper nounsHyde \([2002](https://arxiv.org/html/2606.19170#bib.bib32)\); Stanlaw \([2004](https://arxiv.org/html/2606.19170#bib.bib16)\)\.

The pipeline combines a whitelist filter with blocklist filters applied at both the document and line levels\. The whitelist filter defines the character inventory allowed for Japanese\-centric text, including Japanese scripts, common CJK characters, punctuation, symbols, and other frequent marks\. Documents with more than 1‰ out\-of\-inventory characters are discarded, which removes most text in non\-CJK and non\-Latin scripts, such as Russian and Arabic\. Next, blocklist filters remove lines separated by‘‘\\n’’that contain excessive English or Chinese characters\. For example, the English filter removes a line if it contains more than four consecutive English words, more than 20 Latin letters, or a Japanese\-to\-English character ratio below 40%\. If more than 5% of a document’s lines are removed, the entire document is discarded to avoid damaged context\. Further implementation details are provided in Appendix[A](https://arxiv.org/html/2606.19170#A1)\.

#### 3\.1\.2Pretraining an L1\-Only LLM

We pretrained a 1\.8B\-parameter Llama 2–style decoder\-only Transformer from scratch on the corpus described in §[3\.1\.1](https://arxiv.org/html/2606.19170#S3.SS1.SSS1)\. To our knowledge, this is the largest model trained from scratch for studying second language acquisition\. Pretraining details are provided in Appendix[C](https://arxiv.org/html/2606.19170#A3)\.

We adopted the pretrained multilingual tokenizer from the released llm\-jp\-3 model, which uses a vocabulary of 99,584 tokens covering both Japanese and English\. We acknowledge that this gives the model access to L2 subword units from initialization\. We leave experiments with more human\-like tokenization assumptions to future work\.777[https://huggingface\.co/llm\-jp/llm\-jp\-3\-1\.8b](https://huggingface.co/llm-jp/llm-jp-3-1.8b)

#### 3\.1\.3Linguistic Proficiency Assessments\.

We monitor the model’s linguistic proficiency during L1 pretraining to assess whether it effectively acquires Japanese and whether unwanted language contamination occurs after filtering\.

##### Assessing Japanese Proficiency\.

We assess Japanese proficiency along two dimensions: grammatical knowledge and general Japanese\-language ability\. For grammatical knowledge, we use JBLiMPSomeya and Oseki \([2023](https://arxiv.org/html/2606.19170#bib.bib12)\), a Japanese benchmark in the style of BLiMPWarstadtet al\.\([2020](https://arxiv.org/html/2606.19170#bib.bib7)\)\. Models are evaluated on minimal pairs that differ in a targeted grammatical property; an item is correct if the model assigns higher probability to the grammatical sentence\. Performance is reported as accuracy, with chance at 50%\.

For general Japanese\-language ability, we use llm\-jp\-eval\-v1\.4\.1 ,888[https://github\.com/llm\-jp/scripts/tree/main/evaluation/installers/llm\-jp\-eval\-v1\.4\.1](https://github.com/llm-jp/scripts/tree/main/evaluation/installers/llm-jp-eval-v1.4.1)a few\-shot benchmarkBrownet al\.\([2020](https://arxiv.org/html/2606.19170#bib.bib20)\)covering diverse tasks such as natural language inference and multiple\-choice question answering\. Programming and mathematical tasks are excluded\.

##### Assessing English Leakage\.

We use the English section of MultiBLiMPJumeletet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib8)\), another BLiMP style benchmark\.

##### Assessing Other Leakage\.

We assess Russian and Chinese proficiency using the Russian part in MultiBLiMP and ZhoBLiMPLiuet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib43)\)\.

### 3\.2L2 Acquisition

To introduce Dango to English, we synthesized English learning data using an off\-the\-shelf multilingual LLM and fine\-tuned the model\.

#### 3\.2\.1Synthesizing L2\-Learning Data with LLM

We generate parallel, lesson\-style data for English L2 learning using a strong multilingual LLM\. Each example is centered on a seed word from the CEFR\-J vocabulary listTono \([2019](https://arxiv.org/html/2606.19170#bib.bib24)\), a CEFR\-basedCouncil of Europe \([2001](https://arxiv.org/html/2606.19170#bib.bib22)\)inventory tailored to Japanese learners\. Since the original CEFR\-J annotations cover only A1–B2, we use an externally annotated extension to include words up to C2\. We refer to the combined inventory as CEFR\-J throughout the paper\.999[https://github\.com/openlanguageprofiles/olp\-en\-cefrj/tree/master](https://github.com/openlanguageprofiles/olp-en-cefrj/tree/master), C1/C2 labels are from the Octanove Vocabulary Profile \(CC BY\-SA 4\.0\)\.

As shown in Figure[11](https://arxiv.org/html/2606.19170#A6.F11), we prompt the LLM to generate: \(i\) a detailed explanation of the seed word, \(ii\) an example sentence containing the word and its Japanese translation, and \(iii\) an analysis of salient usage patterns in the example\. Since the prompt controls sentence difficulty, average sentence length and complexity increases with the CEFR\-J level of the seed word \(Figure[14](https://arxiv.org/html/2606.19170#A6.F14)\)\. Both the lexical explanation and usage analysis are written in Japanese, reflecting textbook\-style English learning for Japanese speakers and encouraging Japanese\-to\-English transfer\.

The above setup yields theEnglish–Japanese\(Eng–Jpn\) setting, where the model generates an English sentence followed by its Japanese translation\. We also create aJapanese–English\(Jpn–Eng\) variant by simply swapping the English and Japanese sentences in the data\.

#### 3\.2\.2Fine\-tuning on L2 Learning Data

We simulate L2 acquisition by fine\-tuning models on L2 learning data\. Each training example follows a prompt format in which the user asks for an explanation of a word and the assistant provides the corresponding L2 learning response \(Figure[12](https://arxiv.org/html/2606.19170#A6.F12)\)\. By combining data from different CEFR levels, we train models that mimic L2 learners at specific proficiency stages \(see §[5\.1](https://arxiv.org/html/2606.19170#S5.SS1.SSS0.Px1)\)\.

#### 3\.2\.3Assessing L2 Acquisition Progress

We evaluate L2 acquisition progress with BLEU scores on Eng–Jpn and Jpn–Eng translation\. We use the translation component of a held\-out subset of the L2 learning data for in\-distribution testing, and Jpn–Eng parallel sentences from Tatoeba for out\-of\-distribution evaluation\.101010[https://tatoeba\.org](https://tatoeba.org/)\. Created by volunteer contributors from the Tatoeba community\. Text data licensed under CC BY 2\.0 FR\. Downloaded 2025\-12\-05\.

#### 3\.2\.4L1–L2 Transfer Assessment

To directly quantify L1–L2 transfer effects, we adopt an LLM\-as\-a\-judge evaluation protocol followingGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\)to analyze the patterns in the English productions and compare that to human performance\. We prompt a judge LLM to extract instances of seven English linguistic feature categories \(excludingSpeech Acts\) from model outputs and to assess their contextual correctness:Number Agreement\(local number concord inside noun phrases\),Tense Agreement\(tense choices consistent with temporal cues\),Subject–Verb Agreement\(verb inflection matching the subject\),Modal Verbs and Expressions,Quantifiers and Numerals,Noun–Verb Collocations\(conventional verb–noun pairings supporting fluent lexical selection\), andReference Word\(referential devices such as pronouns/demonstratives that maintain discourse continuity\)\.

As shown in Figure[16](https://arxiv.org/html/2606.19170#A10.F16), the judge identifies each instance of a target feature in a translation output and returns \(i\) the exact text span and \(ii\) a binary label indicating whether the usage is correct in context\. From these analysis outputs, we derive for each category \(a\) aUsage Frequency \(UF\)distribution \(how often the feature appears\) and \(b\) anError Rate \(ER\)distribution \(how often detected instances are incorrect\)\.

Finally, we examine whether Dango produces English texts that share UF and ER distributions similarly to a human English learner\. We apply the same llm\-as\-a\-judge method to extract feature distributions from the ICNALE Written EssaysIshikawa \([2013](https://arxiv.org/html/2606.19170#bib.bib41),[2023](https://arxiv.org/html/2606.19170#bib.bib42)\)corpus\. This corpus contains topic\-centric essays written by English learners of various L1 backgrounds, including Japanese\. We use Jensen–Shannon divergence \(JSD\) to capture distributional shift in UF, and Bray\-Curtis Dissimilarity \(BCD\) in addition to JSD in ER to better reflect raw error count shifts \(Appendix[J\.1](https://arxiv.org/html/2606.19170#A10.SS1)\)\. We also provide an extended analysis comparing model L2 production against ICNALE Spoken DialoguesIshikawa \([2019](https://arxiv.org/html/2606.19170#bib.bib17)\)in Appendix[J\.2](https://arxiv.org/html/2606.19170#A10.SS2)\.

## 4Experiments and Results for L1 \(Japanese\) Acquisition

In this section, we describe the experimental setup and evaluation results for pretraining Dango\-pretrained, the L1\-only LLM\. For brevity, we may omit the “\-pretrained” suffix when referring to the pretrained models in this section\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x3.png)\(a\)Japanese\(llm\-jp\-eval\-v1\.4\.1\)
![Refer to caption](https://arxiv.org/html/2606.19170v1/x4.png)\(b\)Japanese\(JBLiMP\)
![Refer to caption](https://arxiv.org/html/2606.19170v1/x5.png)\(c\)English\(MultiBLiMP\)
![Refer to caption](https://arxiv.org/html/2606.19170v1/x6.png)\(d\)Russian\(MultiBLiMP\)
![Refer to caption](https://arxiv.org/html/2606.19170v1/x7.png)\(e\)Chinese\(ZhoBLiMP\)

Figure 3:Linguistic proficiency assessments across the L1\-pretraining stage\.### 4\.1Experimental Settings

We detail the data preparation and training method for Dango\-pretrained and the baselines\.

##### Training\-based Baselines\.

For training\-based baselines, we trained anunfilteredmodel and used thellm\-jp\-3models as multilingual baselines\. The unfiltered model was trained on the Japanese portion of llm\-jp\-corpus\-v3 without the filtering described in §[3\.1\.1](https://arxiv.org/html/2606.19170#S3.SS1.SSS1)\. We obtained the llm\-jp\-3\-pretrained checkpoints by contacting the llm\-jp team directly\. These models have the same size, architecture, and training method as Dango; they differ only in the training data used\.

##### L1 Corpus Filtering and Statistics\.

Detailed filtering parameters are provided in Appendix[A](https://arxiv.org/html/2606.19170#A1)\. Filtering removed 30% of documents \(Table[9](https://arxiv.org/html/2606.19170#A10.T9)\); the remaining documents were further stripped of English\-heavy lines through line\-level filtering\. Figures[6](https://arxiv.org/html/2606.19170#A2.F6)and[7](https://arxiv.org/html/2606.19170#A2.F7)show document\- and phrase\-level contamination removed by the English filter\. However, short English phrases, such as common daily words and proper nouns, may remain \(Figure[8](https://arxiv.org/html/2606.19170#A2.F8)\)\. Thus, Dango can still learn weak word\- or phrase\-level English associations, but not the ability to generate long, coherent English sentences after L1 pretraining, as shown in Appendix[E](https://arxiv.org/html/2606.19170#A5)\.

##### L1 Pretraining Details\.

Dango\-pretrained was trained on 100B tokens from the filtered corpus\. Compute\-optimal scaling suggests that a 1\.8B\-parameter model is optimally trained on roughly 35B tokensHoffmannet al\.\([2022](https://arxiv.org/html/2606.19170#bib.bib18)\); however, we trained for about 3×\\timesmore tokens because its Japanese generalization continued improving beyond 35B tokens \(Figure[3\(a\)](https://arxiv.org/html/2606.19170#S4.F3.sf1)\)\. Additional pretraining details are provided in Appendix[C](https://arxiv.org/html/2606.19170#A3)\.

### 4\.2Linguistic Proficiency Assessments

We track the linguistic proficiency of the pretrained models over a 10B\-token training interval\.111111In Figure[3](https://arxiv.org/html/2606.19170#S4.F3), the leftmost points on the plot correspond to models trained on 1B tokens\.

##### Japanese Proficiency\.

On llm\-jp\-eval \(Figure[3\(a\)](https://arxiv.org/html/2606.19170#S4.F3.sf1)\), Dango performs slightly below llm\-jp\-3 as training approaches 100B tokens \(\-0\.029\)\. A category breakdown is provided in Appendix[D](https://arxiv.org/html/2606.19170#A4)\.

On JBLiMP \(Figure[3\(b\)](https://arxiv.org/html/2606.19170#S4.F3.sf2)\), Dango outperforms both baselines in early stage \(1B and 10B tokens\), likely because it is trained on nearly pure Japanese data\. As training continues, the gap closes and all models reach comparable performance\. Overall, Dango attains Japanese ability roughly on par with the unfiltered and llm\-jp\-3 models\.

##### English Proficiency\.

On the English portion of MultiBLiMP \(Figure[3\(c\)](https://arxiv.org/html/2606.19170#S4.F3.sf3)\), llm\-jp\-3 saturates after only 10B tokens, likely because roughly half of its training tokens are English\. In contrast, Dango remains near 0\.65 even after 100B tokens\. The unfiltered model indicates that naive training on a “Japanese” corpus can still yield strong English proficiency, reaching 0\.917\.

These results suggest that filtering prevents Dango’s English ability from scaling with additional pretraining\. The case study in Appendix[E](https://arxiv.org/html/2606.19170#A5)further shows that Dango’s English remains largely n\-gram\-like, likely because the filtering rule removes sequences of more than four consecutive English words\.

##### Other L2 Leakage\.

As an additional check for L2 leakage, we evaluate Russian and Chinese \(Figures[3\(d\)](https://arxiv.org/html/2606.19170#S4.F3.sf4)and[3\(e\)](https://arxiv.org/html/2606.19170#S4.F3.sf5)\) to assess the effectiveness of the whitelist and Chinese filter\. All three models perform poorly, but Dango shows the lowest proficiency in both languages\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/v3-1.8b_jpn-eng_test_BLEU_quadratic_table.png)
![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/llm-jp-3-100B_jpn-eng_test_BLEU_quadratic_table.png)

Figure 4:Dango and llm\-jp\-3 Japanese\-to\-English translation performance when trained and evaluated on different levels of data\. The stair\-shaped black line separates seen levels \(lower left\) from unseen levels \(upper right\)\.

## 5Experiments and Results for L2 \(English\) Acquisition

### 5\.1Experimental Settings

##### Synthesizing L2 Learning Data\.

We used GPT\-5\.2 to synthesize 9\.9K textbook\-style L2\-learning examples from the CEFR\-J word list, generating one example per target word\. The dataset comprises 8,938 training, 494 validation, and 501 test examples\. See Appendices[F\.1](https://arxiv.org/html/2606.19170#A6.SS1)and[F\.3](https://arxiv.org/html/2606.19170#A6.SS3)for prompt design and additional synthesis statistics\.

Before fine\-tuning on the L2 learning data, we first trained the model on Ichikara, a Japanese instruction\-following dataset\.121212Visit[this link](https://liat-aip.sakura.ne.jp/wp/llm%e3%81%ae%e3%81%9f%e3%82%81%e3%81%ae%e6%97%a5%e6%9c%ac%e8%aa%9e%e3%82%a4%e3%83%b3%e3%82%b9%e3%83%88%e3%83%a9%e3%82%af%e3%82%b7%e3%83%a7%e3%83%b3%e3%83%87%e3%83%bc%e3%82%bf%e4%bd%9c%e6%88%90/llm%e3%81%ae%e3%81%9f%e3%82%81%e3%81%ae%e6%97%a5%e6%9c%ac%e8%aa%9e%e3%82%a4%e3%83%b3%e3%82%b9%e3%83%88%e3%83%a9%e3%82%af%e3%82%b7%e3%83%a7%e3%83%b3%e3%83%87%e3%83%bc%e3%82%bf-%e5%85%ac%e9%96%8b/)for the Ichikara dataset \(CC\-BY\-NC\-SA\)\.This step was intended to strengthen instruction\-following behavior in the model’s native language and to enable the case studies in §[4\.2](https://arxiv.org/html/2606.19170#S4.SS2)\(Appendix[E](https://arxiv.org/html/2606.19170#A5)\)\.

##### Fine\-tuning on L2 Learning Data\.

We trained separate models for six CEFR\-J proficiency levels under two direction settings: English–Japanese \(Eng–Jpn\) and Japanese–English \(Jpn–Eng\)\. For a given target level, we used cumulative data from all levels up to and including that level \(e\.g\., B2 uses A1\+A2\+B1\+B2\)\. See Appendix[F\.2](https://arxiv.org/html/2606.19170#A6.SS2)and[G](https://arxiv.org/html/2606.19170#A7)for additional fine\-tuning details\.

### 5\.2L2 Proficiency Development Assessment

Figure[4](https://arxiv.org/html/2606.19170#S4.F4)shows Jpn–Eng translation performance across difficulty levels for Dango and llm\-jp\-3 trained on different mixtures of difficulties\. Results for the Eng–Jpn direction are shown in Appendix[H](https://arxiv.org/html/2606.19170#A8)\. For Tatoeba, we randomly sampled 1,000 sentence pairs from the downloaded dataset\. We provide a human analysis of whether the BLEU score captures meaningful L2 development in Appendix[I](https://arxiv.org/html/2606.19170#A9)\.

#### 5\.2\.1Japanese–English Translation

##### General observations\.

When trained and evaluated on the same difficulty level, for example A1, Dango produces understandable English translations and reaches a BLEU score of 31\.7\. At this level, llm\-jp\-3 outperforms Dango with a \+37\.2 BLEU advantage\.

Performance improves as the training and evaluation settings expand\. When trained on all levels from A1 to C2 and evaluated on C2, Dango reaches 21\.6 BLEU\. Qualitatively, it often preserves the core meaning in long sentences, but with frequent lexical and grammatical errors \(see Appendix[I\.1](https://arxiv.org/html/2606.19170#A9.SS1)\)\. Overall, llm\-jp\-3 consistently outperforms Dango, likely due to its multilingual pretraining\.

Both models can generalize to unseen levels and Tatoeba data to some extent\. For example, when trained on A1–B2 data, Dango generalizes to C1 \(21\.2\) and C2 \(15\.8\)\. Being multilingual, llm\-jp\-3 generalizes better, achieving 27\.9 and 25\.2\.

##### Training on more difficult data improves performance at easier levels\.

Training with more difficult data consistently enhances performance on easier test sets\. For instance, when evaluated on B1, Dango sees a \+12\.5 BLEU gain when training expands from A1–B1 to A1–C2, whereas llm\-jp\-3 gains only \+5\.7 BLEU\. This demonstrates that Dango learns effectively from the L2\-learning data\.

ModelJSD ofUFJSD ofERBCD ofERTraining\-based methodsDango0\.0104![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0110![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.2467![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Unfiltered0\.01760\.0304![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.2502![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)llm\-jp\-30\.02490\.04870\.6982Prompting\-based methodsllm\-jp\-3\.1\-instruct0\.03460\.06580\.8523GPT\-4o0\.0028![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.03740\.5295GPT\-5\.50\.0084![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0191![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1925![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Table 1:LLM output linguistic feature distribution divergences to human Written Essay produced by Japanese learners\. Lower is more similar\.Human L1JSD ofUFJSD ofERBCD ofERJapanese0\.0104![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0110![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.2467![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Korean0\.0111![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0094![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1078![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Mandarin0\.0140![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0161![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.3509Thai0\.01550\.02210\.1272![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Table 2:Linguistic feature divergences between Dango and humans with different L1 backgrounds\.

### 5\.3Transfer Effect Assessment

##### Baselines\.

In addition to the training\-based methods \(Dango, unfiltered, and llm\-jp\-3\), we adopt the prompting\-based L2 speaker simulation method fromGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\), where an LLM is instructed to generate L2 English text conditioned on prompts containing a specified L1 background and L1\-dependent linguistic traits\. We evaluate three LLMs in this setting: llm\-jp\-3\.1\-instruct,131313We used the 1\.8B model:[https://huggingface\.co/llm\-jp/llm\-jp\-3\.1\-1\.8b\-instruct4](https://huggingface.co/llm-jp/llm-jp-3.1-1.8b-instruct4)GPT\-4o,141414[https://openai\.com/index/hello\-gpt\-4o/](https://openai.com/index/hello-gpt-4o/)and GPT\-5\.5\.151515[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)For GPT\-4o and GPT\-5\.5, we follow the original prompt design, modifying it to output English translations \(Figure[21](https://arxiv.org/html/2606.19170#A10.F21)and[23](https://arxiv.org/html/2606.19170#A10.F23)\); for llm\-jp\-3\.1\-instruct, we use a simplified role\-playing and error\-injection prompt \(Figure[24](https://arxiv.org/html/2606.19170#A10.F24)\) due to its smaller size and shorter context limit\.

Both approaches offer distinct advantages\. Prompting\-based methods are highly accessible, utilizing off\-the\-shelf LLMs without the need for additional training\. Conversely, training\-based models allow us to trace the transfer effect directly back to the L1→L2 training process itself\. This isolates the learning mechanism from the confounding effects of an LLM simply “role\-playing” to mask its actual proficiency, providing a stronger foundation for future controlled studies and interpretability research\.

As described in §[3\.2\.4](https://arxiv.org/html/2606.19170#S3.SS2.SSS4), we first construct UF and ER distributions for human and model English productions\. We then compute JSD and BCD for human–model pairs to identify which models best match human transfer patterns\. Here, we report results on ICNALE Written Essays; Appendix[J\.2](https://arxiv.org/html/2606.19170#A10.SS2)presents the corresponding analysis on ICNALE Spoken Dialogues, confirming the robustness of our findings\. For trained baselines, we use models trained on A1–B2 data, since the ICNALE learners are below B2 level\.

##### Focusing on JSD of UF\.

Table[1](https://arxiv.org/html/2606.19170#S5.T1)shows that, among trained baselines, Dango is closer to Japanese L1 speakers than the unfiltered model or llm\-jp\-3 in terms of UF divergence\. In other words, Dango has more human\-like overuse and avoidance patterns over the seven feature categories\.

Compared with prompting baselines, GPT\-4o and GPT\-5\.5 show UF distributions closer to humans, slightly outperforming Dango\. In contrast, llm\-jp\-3\.1\-instruct remains the most distant, likely due to its small size and limited exposure to such role\-play prompts during training\. It generally failed to output systematic English grammar errors, even when prompted to do so\.

Table[2](https://arxiv.org/html/2606.19170#S5.T2)shows Dango’s UF divergence from human English learners across L1 backgrounds\. The pattern broadly aligns with the degree of structural overlap with Japanese \(Table[8](https://arxiv.org/html/2606.19170#A10.T8)\): Dango is most similar to Japanese learners, followed by Korean, likely due to shared SOV word order and similar particle systems\. Mandarin shows greater divergence, while Thai diverges the most, reflecting its limited structural similarity to Japanese\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/error_rate_bcd_view_we.png)Figure 5:Error Rates \(raw values\) of human and LLMs\.
##### Focusing on JSD of ER\.

In Table[1](https://arxiv.org/html/2606.19170#S5.T1), Dango achieves the lowest ER JSD among all methods, indicating that its error distribution is closest to that of human learners\. Compared with non\-Japanese L1 learners \(Table[2](https://arxiv.org/html/2606.19170#S5.T2)\), Dango is closest to Korean speakers, followed by Japanese speakers\. We attribute this to the close linguistic distance between Japanese and Korean, and interpret it as evidence that Dango captures L1\-induced transfer effects\.

##### Focusing on BCD of ER\.

Dango achieves the second\-lowest ER BCD in Table[1](https://arxiv.org/html/2606.19170#S5.T1)\. GPT\-5\.5 obtains the lowest BCD and remains balanced across the other metrics, suggesting that it also models Japanese–English transfer effects well\. In contrast, the other multilingual baselines show large BCD values because their absolute error rates differ substantially from human learners \(Figure[5](https://arxiv.org/html/2606.19170#S5.F5)\): llm\-jp\-3 and llm\-jp\-3\.1\-instruct rarely make grammar errors, whereas GPT\-4o produces too many\.

Overall, these results suggest that Dango’s English productions broadly align with a Japanese L1 speaker profile\. Together with the additional analysis in Appendix[J\.2](https://arxiv.org/html/2606.19170#A10.SS2), they show that filtering L2 exposure in the Japanese corpus improves L2 speaker modeling, as the unfiltered baseline performs worse than Dango on all measured metrics\.

### 5\.4Analysis on Dango’s English Production

We present a case study on Dango’s English production in Figure[20](https://arxiv.org/html/2606.19170#A10.F20), with a focus on its error patterns\. We observe that Dango makes many grammatical and phrasing errors typical of Japanese speakers, such as preposition, article form, and tense errors\. We also observed that it sometimes mistranslates Japanese words literally into English, mirroring a strategy L2 speakers often use when they lack the vocabulary for a specific conceptFærch and Kasper \([1983](https://arxiv.org/html/2606.19170#bib.bib45)\)\.

## 6Conclusions

We presentedDango, a 1\.8B\-parameter Japanese L1\-only decoder LLM trained on a strictly filtered corpus to study L1→L2 transfer\. By fine\-tuning on LLM\-generated English\-learning lessons, Dango exhibits learner\-like transfer patterns closer to Japanese L2 speakers than other trained baselines, on par with prompted GPT\-5\.5\. We release Dango to advance controlled SLA studies and facilitate practical learner\-facing applications\.

## Limitations

##### Tokenizer and vocabulary\.

Dango uses a pretrained multilingual tokenizer with substantial English subword coverage\. This choice improves training stability and enables direct comparison to llm\-jp\-3, but it also means Dango does not fully learn its vocabulary from scratch under an L1\-only constraint\. Importantly, however, the token embeddings themselves are learned from scratch during pretraining\. Future work could explore tokenizer choices and dynamic vocabulary expansion schemes during L2 exposure to better approximate staged, human\-like acquisition\.

##### Single model and single L1→L2 setting\.

We study a single model scale \(1\.8B parameters\) due to the high cost of pretraining, and we focus on one transfer direction \(Japanese→\\rightarrowEnglish\) because English is a common and practically important L2\. Future work can extend this framework to larger model scales and additional L1→\\rightarrowL2 pairs spanning a wider range of linguistic distances to better characterize the generality of our findings\.

## AI Assistance in This Work

We utilized AI\-based tools primarily to refine the writing style and improve the grammatical accuracy of this manuscript\. Additionally, we employed AI assistance for data visualization and minor coding support\. The authors carefully reviewed and verified all AI\-assisted content, figures, and code, and we take full responsibility for the final manuscript and its outcomes\.

## References

- Modeling nonnative sentence processing with L2 language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4927–4940\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.283/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.283)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px3.p1.1),[§2\.2](https://arxiv.org/html/2606.19170#S2.SS2.SSS0.Px2.p1.1)\.
- K\. Barrs \(2020\)Learning from the linguistic landscape: a project\-based learning approach to investigating english in japan\.Electronic Journal of Foreign Language Teaching17\.External Links:[Document](https://dx.doi.org/10.56040/ktbr1711),[Link](https://e-flt.nus.edu.sg/v17s12020/barrs.pdf)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p5.1),[§3\.1\.1](https://arxiv.org/html/2606.19170#S3.SS1.SSS1.p2.1)\.
- T\. Blevins and L\. Zettlemoyer \(2022\)Language contamination helps explains the cross\-lingual capabilities of English pretrained models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 3563–3574\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.233/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.233)Cited by:[§2\.2](https://arxiv.org/html/2606.19170#S2.SS2.SSS0.Px2.p1.1),[§3\.1\.1](https://arxiv.org/html/2606.19170#S3.SS1.SSS1.p2.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.External Links:2005\.14165,[Link](https://arxiv.org/abs/2005.14165)Cited by:[§3\.1\.3](https://arxiv.org/html/2606.19170#S3.SS1.SSS3.Px1.p2.1)\.
- I\. Caswell, T\. Breiner, D\. van Esch, and A\. Bapna \(2020\)Language ID in the wild: unexpected challenges on the path to a thousand\-language web text corpus\.InProceedings of the 28th International Conference on Computational Linguistics,D\. Scott, N\. Bel, and C\. Zong \(Eds\.\),Barcelona, Spain \(Online\),pp\. 6588–6608\.External Links:[Link](https://aclanthology.org/2020.coling-main.579/),[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.579)Cited by:[§2\.2](https://arxiv.org/html/2606.19170#S2.SS2.SSS0.Px2.p1.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px2.p1.1)\.
- I\. Constantinescu, T\. Pimentel, R\. Cotterell, and A\. Warstadt \(2025\)Investigating critical period effects in language acquisition through neural language models\.Transactions of the Association for Computational Linguistics13,pp\. 96–120\.External Links:[Link](https://aclanthology.org/2025.tacl-1.5/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00725)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p2.1),[§1](https://arxiv.org/html/2606.19170#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px3.p1.1),[§2\.2](https://arxiv.org/html/2606.19170#S2.SS2.SSS0.Px2.p1.1)\.
- Council of Europe \(2001\)Common european framework of reference for languages: learning, teaching, assessment\.Cambridge University Press,Cambridge\.External Links:[Link](https://rm.coe.int/1680459f97)Cited by:[§3\.2\.1](https://arxiv.org/html/2606.19170#S3.SS2.SSS1.p1.1)\.
- C\. Færch and G\. Kasper \(Eds\.\) \(1983\)Strategies in interlanguage communication\.Longman Publishing Group,London; New York\.External Links:ISBN 0582553733Cited by:[§5\.4](https://arxiv.org/html/2606.19170#S5.SS4.p1.1)\.
- R\. Gao, X\. Wu, T\. Kuribayashi, M\. Ye, S\. Qi, C\. Roever, Y\. Liu, Z\. Yuan, and J\. H\. Lau \(2025\)Can LLMs simulate L2\-English dialogue? an information\-theoretic analysis of L1\-dependent biases\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 4355–4379\.External Links:[Link](https://aclanthology.org/2025.acl-long.219/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.219),ISBN 979\-8\-89176\-251\-0Cited by:[Figure 15](https://arxiv.org/html/2606.19170#A10.F15),[Figure 21](https://arxiv.org/html/2606.19170#A10.F21),[Figure 23](https://arxiv.org/html/2606.19170#A10.F23),[Appendix J](https://arxiv.org/html/2606.19170#A10.SS0.SSS0.Px1.p1.1),[Appendix J](https://arxiv.org/html/2606.19170#A10.SS0.SSS0.Px1.p2.1),[Appendix J](https://arxiv.org/html/2606.19170#A10.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px3.p2.1),[§3\.2\.4](https://arxiv.org/html/2606.19170#S3.SS2.SSS4.p1.1),[§5\.3](https://arxiv.org/html/2606.19170#S5.SS3.SSS0.Px1.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre \(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.19170#S4.SS1.SSS0.Px3.p1.1)\.
- B\. Hyde \(2002\)Japan’s emblematic english\.English Today18,pp\. 12–16\.External Links:[Document](https://dx.doi.org/10.1017/s0266078402003024)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p5.1),[§3\.1\.1](https://arxiv.org/html/2606.19170#S3.SS1.SSS1.p2.1)\.
- S\. Ishikawa \(2013\)The icnale and sophisticated contrastive interlanguage analysis of asian learners of english\.Learner Corpus Studies in Asia and the World1,pp\. 91–118\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.24546/81006678),[Link](https://da.lib.kobe-u.ac.jp/da/kernel/81006678/)Cited by:[§3\.2\.4](https://arxiv.org/html/2606.19170#S3.SS2.SSS4.p3.1)\.
- S\. Ishikawa \(2019\)The icnale spoken dialogue: a new dataset for the study of asian learners’ performance in l2 english interviews\.ENGLISH TEACHING74,pp\. 153–177\.External Links:[Document](https://dx.doi.org/10.15858/engtea.74.4.201912.153)Cited by:[§J\.2](https://arxiv.org/html/2606.19170#A10.SS2.p1.1),[§3\.2\.4](https://arxiv.org/html/2606.19170#S3.SS2.SSS4.p3.1)\.
- S\. Ishikawa \(2023\)The icnale guide: an introduction to a learner corpus study on asian learners’ l2 english\.1 edition,Routledge\.External Links:[Document](https://dx.doi.org/10.4324/9781003252528),ISBN 9781032180250Cited by:[§3\.2\.4](https://arxiv.org/html/2606.19170#S3.SS2.SSS4.p3.1)\.
- S\. Jarvis and A\. Pavlenko \(2008\)Crosslinguistic influence in language and cognition\.Routledge\.Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p1.1)\.
- J\. Jumelet, L\. Weissweiler, J\. Nivre, and A\. Bisazza \(2025\)MultiBLiMP 1\.0: a massively multilingual benchmark of linguistic minimal pairs\.External Links:2504\.02768,[Link](https://arxiv.org/abs/2504.02768)Cited by:[§3\.1\.3](https://arxiv.org/html/2606.19170#S3.SS1.SSS3.Px2.p1.1)\.
- R\. Lado \(1957\)Linguistics across cultures: applied linguistics for language teachers\.University of Michigan Press,Ann Arbor\.Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p1.1)\.
- V\. D\. Lai, N\. Ngo, A\. Pouran Ben Veyseh, H\. Man, F\. Dernoncourt, T\. Bui, and T\. H\. Nguyen \(2023\)ChatGPT beyond English: towards a comprehensive evaluation of large language models in multilingual learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13171–13189\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.878/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.878)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p2.1)\.
- Y\. Liu, Y\. Shen, H\. Zhu, L\. Xu, Z\. Qian, S\. Song, K\. Zhang, J\. Tang, P\. Zhang, B\. Yang, R\. Wang, and H\. Hu \(2025\)A systematic assessment of language models with linguistic minimal pairs in chinese\.External Links:2411\.06096,[Link](https://arxiv.org/abs/2411.06096)Cited by:[§3\.1\.3](https://arxiv.org/html/2606.19170#S3.SS1.SSS3.Px3.p1.1)\.
- B\. Lyu, C\. Lai, and J\. Guo \(2024\)Effectiveness of chatbots in improving language learning: a meta‐analysis of comparative studies\.International Journal of Applied Linguistics\.External Links:[Document](https://dx.doi.org/10.1111/ijal.12668)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p3.1)\.
- A\. Murakami and T\. Alexopoulou \(2015\)L1 influence on the acquisition order of english grammatical morphemes\.Studies in Second Language Acquisition38,pp\. 365–401\.External Links:[Document](https://dx.doi.org/10.1017/s0272263115000352)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p1.1)\.
- S\. Nakayama \(2022\)A close examination of vocabulary in japanese efl textbooks\.InJALT2021 Postconference Publication: Reflections and New Perspectives,P\. Ferguson and R\. Derrah \(Eds\.\),pp\. 209–216\.External Links:[Document](https://dx.doi.org/10.37546/JALTPCP2021-24),[Link](https://doi.org/10.37546/JALTPCP2021-24)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p6.1)\.
- M\. Oba, T\. Kuribayashi, H\. Ouchi, and T\. Watanabe \(2023\)Second language acquisition of neural language models\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13557–13572\.External Links:[Link](https://aclanthology.org/2023.findings-acl.856/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.856)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p2.1),[§1](https://arxiv.org/html/2606.19170#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px3.p1.1),[§2\.2](https://arxiv.org/html/2606.19170#S2.SS2.SSS0.Px2.p1.1)\.
- T\. Odlin \(1989\)Language transfer: cross\-linguistic influence in language learning\.Cambridge University Press,Cambridge\.External Links:ISBN 9780521378093Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p1.1)\.
- T\. Someya and Y\. Oseki \(2023\)JBLiMP: Japanese benchmark of linguistic minimal pairs\.InFindings of the Association for Computational Linguistics: EACL 2023,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 1581–1594\.External Links:[Link](https://aclanthology.org/2023.findings-eacl.117/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-eacl.117)Cited by:[§3\.1\.3](https://arxiv.org/html/2606.19170#S3.SS1.SSS3.Px1.p1.1)\.
- J\. Stanlaw \(2004\)Japanese english: language and culture contact\.Hong Kong University Press\.External Links:ISBN 9789622095717,[Link](http://www.jstor.org/stable/j.ctt2jc5qm)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p5.1),[§3\.1\.1](https://arxiv.org/html/2606.19170#S3.SS1.SSS1.p2.1)\.
- Y\. Tono \(2019\)Coming full circle—from cefr to cefr\-j and back\.CEFR Journal—Research and Practice1\.External Links:[Document](https://dx.doi.org/10.37546/JALTSIG.CEFR1-1),[Link](https://cefrjapan.net/images/PDF/CEFRJournal/CEFRJournal-1_D_2019May.pdf)Cited by:[§3\.2\.1](https://arxiv.org/html/2606.19170#S3.SS2.SSS1.p1.1)\.
- A\. Vasmatzoglou and N\. Ní Chiaráin \(2020\)The development of an online game\-based simulation for the training of english language teachers in virtual environments\.InCALL for Widening Participation: Short Papers from EUROCALL 2020,K\. Frederiksen, S\. Larsen, L\. Bradley, and S\. Thouësny \(Eds\.\),pp\. 334–341\.External Links:ISBN 978\-2\-490057\-81\-8,[Document](https://dx.doi.org/10.14705/rpnet.2020.48.1210),[Link](https://doi.org/10.14705/rpnet.2020.48.1210)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p3.1)\.
- A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. Bowman \(2020\)BLiMP: the benchmark of linguistic minimal pairs for English\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.External Links:[Link](https://aclanthology.org/2020.tacl-1.25/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00321)Cited by:[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px3.p1.1),[§3\.1\.3](https://arxiv.org/html/2606.19170#S3.SS1.SSS3.Px1.p1.1)\.
- G\. Wenzek, M\. Lachaux, A\. Conneau, V\. Chaudhary, F\. Guzmán, A\. Joulin, and E\. Grave \(2020\)CCNet: extracting high quality monolingual datasets from web crawl data\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 4003–4012\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.494/),ISBN 979\-10\-95546\-34\-4Cited by:[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px2.p1.1)\.
- A\. Yadavalli, A\. Yadavalli, and V\. Tobin \(2023\)SLABERT talk pretty one day: modeling second language acquisition with BERT\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 11763–11777\.External Links:[Link](https://aclanthology.org/2023.acl-long.657/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.657)Cited by:[§1](https://arxiv.org/html/2606.19170#S1.p2.1),[§1](https://arxiv.org/html/2606.19170#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2606.19170#S2.SS1.SSS0.Px3.p1.1)\.

## Appendix AFiltering Pipeline Design and Statistics

Our filtering pipeline for constructing a high\-quality Japanese pretraining corpus consists of three stages: \(1\) a Unicode whitelist filter, \(2\) an English \(Latin\-script\) blocklist filter, and \(3\) a Chinese blocklist filter\. Table[10](https://arxiv.org/html/2606.19170#A10.T10)summarizes the parameter settings used in each stage\.

##### Unicode whitelist filter\.

This filter drops a document if the fraction of characters outside a predefined Unicode whitelist exceeds a threshold\. The whitelist includes Japanese kana, CJK ideographs, punctuation, emojis, and ASCII characters \(which may include Latin letters\)\. CJK ideographs contains Chinese characters used by Chinese, Japanese \(as Kanji\), and Korean \(as Hanja\)\. Therefore, Korean Hanguls does not pass the whitelist filter\. Chinese characters and English \(Latin\) characters may pass this stage, and are handled by the subsequent blocklist filters\. For other languages, if a document contains, e\.g\., Cyrillic, characters totaling more than 0\.1% of all characters, the entire document is removed from the corpus\.

##### English \(Latin\) filter\.

This filter operates at the line level \(lines are split by “\\n”\)\. It removes lines that contain excessive Latin\-script content, and drops the entire document if too many lines are removed \(to avoid broken context\)\. A line is removed if any of the following conditions hold: \(i\) it contains more than 20 Latin characters; \(ii\) the ratio of Latin to non\-Latin characters exceeds 40%; or \(iii\) it contains more than four consecutive Latin words separated by spaces\. Lines containing fewer than eight Latin characters are ignored, as we consider such cases harmless\. Finally, if more than 5% of lines in a document are removed, we drop the document\.

##### Chinese filter\.

To reduce Chinese text \(primarily Simplified Chinese\) in the corpus, we construct a character\-level blocklist of ideographs that are highly indicative of Chinese\-only usage\. The blocklist is derived from the Unihan BMP list\.161616https://unicode\.org/charts/unihan\.htmlIf any character from this blocklist appears in a line, the line is removed\. If more than 0\.1% of lines are removed from a document, the document is dropped\.

##### Filtering outcome\.

Applying the full pipeline removes approximately 30% of documents from the original corpus, highlighting the need for strict filtering when building a Japanese L1 pretraining dataset\. After filtering, the corpus’s total token count shrank from 592B to 376B\.

## Appendix BFiltered Documents: Case Study

The English \(Latin\-script\) blocklist filter rejects most cases of severe English contamination in the Japanese corpus, including document\-level parallel data \(Figure[6](https://arxiv.org/html/2606.19170#A2.F6)\) and phrase\-level parallel data \(Figure[7](https://arxiv.org/html/2606.19170#A2.F7)\)\. In these examples, lines \(separated by “\\n”\) shown in red are flagged by the filter\. Because the fraction of flagged lines exceeds 5% in each document, the entire document is then removed\.

At the same time, the pipeline permits limited English that a typical Japanese speaker may encounter or use in everyday life\. For instance, in Figure[8](https://arxiv.org/html/2606.19170#A2.F8), common short tokens such as “hit” and “ok,” as well as proper nouns such as “Access” and “Excel,” do not trigger the blocklist\. Short phrases \(e\.g\., “stay at home order,” “Reasons I Love You”\) are also allowed; these represent an upper bound on the length of consecutive English segments that Dango is likely to have encountered during training\.

\(\.\.\.\.\.\.\)若年女性の流行TrendsJapan is the land of trends\. Nowhere else do trends arise, spread and\\ndie with such speed\. The reasons for this are simple: affluent youth, merciless\\nadvertising, high population density and an insatiable appetite for ’the\\n\(\.\.\.\.\.\.\)流行日本は流行り廃りの激しい国だ。こんなに速いスピードで流行が生まれ、広がり、そして廃れていく国は他にない。そうなった理由は単純だ。小金を持った若年層、容赦ない宣伝、高い人口密度、そして新し物好きの国民性。\\n\(\.\.\.\.\.\.\)Figure 6:Contamination: document\-level parallel data\.\(\.\.\.\.\.\.\)左ひだり left \(side\)\\n折おれて turn \(toward\)\\n血塔けっとう Bloody Tower\\n門もん gate\\n入はいる enter\\n今いま now; the present\\n昔むかし long ago; ancient times\\n薔薇しょうびの乱らん War of the Roses \(15th century struggle for the throne of England\)\\n\(\.\.\.\.\.\.\)Figure 7:Contamination: word/phrase\-level parallel data\.アクセス数:0 hit \(累計:15524 hit\) 逆アクセス数:0 hit \(累計:6371 hit\)\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-応募資格 IT関連実務経験が少しでもあればok！Access、Excelマクロ経験ある方は尚歓迎です！／ ◆ブランク3年程度でもご相談ください◆\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-ハワイでは、3月22日に州政府より緊急事態宣言が発令され、翌日午後4時より4月30日まで不要不急の外出を禁止する “stay at home order”が発令されました。この法令は、生活に不可欠なサービス・商品を提供している機関を除き、原則すべての従業員は在宅勤務を行わなければならない、というものでした。\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-今でも年に2回は海外へ行っているという西野カナさん。2016年の新曲「あなたの好きなところ」は、52枚のトランプに好きな人の好きなところを書いてプレゼントするラブレター"52 Reasons I Love You" に着想を得て生まれた曲だそう。Figure 8:English appearances that are allowed by the filtering pipeline\.
## Appendix CL1 Model Pretraining Details

We reproduced the customized Megatron\-LM pretraining framework developed by llm\-jp\.171717[https://github\.com/llm\-jp/scripts/tree/main/pretrain/scripts](https://github.com/llm-jp/scripts/tree/main/pretrain/scripts)Most hyperparameters, such as model architecture and optimization settings, were kept identical to those of the official llm\-jp\-3 models \(Table[3](https://arxiv.org/html/2606.19170#A3.T3)\), except that we increased the global batch size from 512 to 1024\. We set the learning\-rate schedule to a maximum of 376B tokens to match the size of the L1\-stage Japanese corpus, but trained for only 100B tokens\. In contrast, the final released llm\-jp\-3 model is scheduled for 2\.1T tokens in total\. Thus, our model and the corresponding intermediate llm\-jp\-3 checkpoint represent partial progress along their respective full training trajectories\. Pretraining was conducted on 32 NVIDIA H200 GPUs \(4 nodes × 8 GPUs\) for approximately 26 hours\.

HyperparameterValueModel ArchitectureLayers24Hidden Size2048FFN Hidden Size7168Attention Heads16Sequence Length4096Positional EmbeddingsRoPEActivation FunctionSwiGLUNormalizationRMSNorm\(ϵ=1×10−5\\epsilon=1\\times 10^\{\-5\}\)BiasDisabledOptimizationOptimizerAdam\(β1,β2=0\.9,0\.95\\beta\_\{1\},\\beta\_\{2\}=0\.9,0\.95\)Peak Learning Rate3×10−43\\times 10^\{\-4\}Min Learning Rate3×10−53\\times 10^\{\-5\}LR SchedulerCosineWarmup Steps180Weight Decay0\.1Gradient Clipping1\.0Global Batch Size1024Precisionbf16Flash\-attentionTrueZ\-Loss Strength1×10−41\\times 10^\{\-4\}Table 3:Model architecture and training hyperparameters\.
## Appendix DCategory Breakdown of Tasks in llm\-jp\-eval

We exclude Code Generation \(CG\) and Mathematical Reasoning \(MR\) from the evaluation suite because they are not directly related to our goal\. We also exclude Machine Translation \(MT; Japanese–English\) because we intend to focus on evaluating Japanese proficiency using llm\-jp\-eval; we leave English evaluation to MultiBLiMP\.

In Figure[9](https://arxiv.org/html/2606.19170#A4.F9), we find that models show mixed performance across task categories, with each model outperforming the other in different categories\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x8.png)Figure 9:Task category breakdown of llm\-jp\-eval performance\.
## Appendix ECase Study on Dango’s English Competency After L1 Acquisition

After completing L1 pretraining, we fine\-tuned Dango\-pretrained on the Ichikara instruction\-following dataset, which includes a small number of Japanese–English translation examples\. We then manually inspected the models’ English generation \(translation\) behavior\.

As shown in the upper block of Figure[10](https://arxiv.org/html/2606.19170#A5.F10), for a relatively simple Japanese–English translation prompt, Dango occasionally captures local lexical correspondences \(e\.g\., mapping “犬” to “dog”\), but its output is typically unfaithful and ungrammatical at the sentence level\. In contrast, llm\-jp\-3 captures more nuances and the grammar is correct\.

For a longer sentence \(bottom block\), Dango fails to preserve the meaning of the Japanese source and instead generates largely unrelated and repetitive English text\. Nevertheless, parts of the output contain locally well\-formed fragments \(e\.g\., “you are not”, “I love you”\), suggesting that the four\-consecutive\-English\-word filter encourages learning short\-range English regularities \(n\-gram\-level patterns\) rather than maintaining global sentence semantics\. llm\-jp\-3 also fails to translate the sentence faithfully in this case, but its output remains grammatically coherent and stylistically natural\.

User:「私は可愛い犬がいるカフェに行きました。」を英語で書いてください。\(translation: Write "I went to a cafe where there were cute dogs\." in English\. \)Dango\-pretrained\-Ichikara:「私は可愛い犬がいるカフェに行きました。」を英語に翻訳すると、以下のようになります。I am beautiful dog\.\(translation: If you translate "\.\.\.\.\.\." to English, it becomes \\n I am beautiful dog\. \)llm\-jp\-3\-pretrained\-Ichikara:I’m looking for a cute dog restaurant\.

User:「彼女は、プレッシャーの中でも落ち着いていられる彼の能力を感心して話した。」を英語で書いてください。\(translation: Write "She spoke admiringly of his ability to stay calm under pressure\." in English\. \)Dango\-pretrained\-Ichikara:""He is a hot day to meet you are not happen again’s hot day to meet you are not happen again’s hot day to meet you are not been happen\. I love you, I love you\."のように書くと良いでしょう。\(translation: You can write it as "\.\.\.\.\.\." \)llm\-jp\-3\-pretrained\-Ichikara:「彼女は、プレッシャーの中でも落ち着いていられる彼の能力を感心して話した。」を英語に翻訳すると、以下のようになります。I enjoyed listening to the story of the day because the conflict was resolved in a healthy way\.\(translation: If you translate "\.\.\.\.\.\." to English, it becomes \\n \.\.\.\.\.\. \)

Figure 10:Case study of English production after completing L1 pretraining and Ichikara instruction\-tuning\.
## Appendix FL2 Acquisition Data Synthesis Details

### F\.1Prompt for L2 Learning Data Generation

Figure[11](https://arxiv.org/html/2606.19170#A6.F11)showcases the prompt used to generate textbook\-style L2 learning data with GPT\-5\.2\. The deep orange block highlights the parallel sentence pair, which is used in evaluation \(Figure[13](https://arxiv.org/html/2606.19170#A6.F13)\)\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x9.png)Figure 11:Prompt used to generate textbook\-style L2 learning data\.
### F\.2Prompt for L2 Acquisition Fine\-tuning

We show the prompt format used to fine\-tune the models on the L2 learning data\. We modified the user prompt and fine\-tuned the models to learn GPT\-5\.2’s outputs; the assistant message is identical to that in Figure[11](https://arxiv.org/html/2606.19170#A6.F11)\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x10.png)Figure 12:Prompt format used to train Dango and llm\-jp\-3 on the textbook\-style L2 learning data\.
### F\.3L2 Learning Data Length Statistics

In Figure[14](https://arxiv.org/html/2606.19170#A6.F14), we report the average sentence length \(in words\) in the test set of our LLM\-generated L2 learning data\. For comparison, we also report the sentence lengths of Dango and llm\-jp\-3’s English outputs from the Japanese–English translation task\. For each CEFR\-J level \(e\.g\., B2\), outputs are sampled from models trained on data up to that level \(e\.g\., A1\+A2\+B1\+B2\)\.

### F\.4Prompt for Evaluating Models on Translation Task

We show the prompt format used to evaluate the translation task in Figure[13](https://arxiv.org/html/2606.19170#A6.F13)\. We place the source sentence in the assistant message rather than in the user prompt, and the model continues generating after “Translate this into Japanese/English:” \(highlighted in yellow\)\. We adopt this format because it more closely matches what models see during L2 acquisition: the example sentence and its translation appear together in the assistant message rather than in the user prompt\. Empirically, we find that this format yields more stable translation outputs\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x11.png)Figure 13:Prompt format used to evaluate the models on the translation task\. The model continues generating after the source sentence appears in the assistant message, only producing the highlighted yellow part\.![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/sentence_length_distribution.png)Figure 14:Sentence length distribution of the LLM\-generated L2 learning data\.

## Appendix GL2 Acquisition Fine\-tuning Details

We fine\-tuned the LLMs using Axolotl\.181818[https://github\.com/axolotl\-ai\-cloud/axolotl/](https://github.com/axolotl-ai-cloud/axolotl/)All experiments were conducted on a single GPU without early stopping\. The fine\-tuning hyperparameters are shown in Table[4](https://arxiv.org/html/2606.19170#A7.T4)\.

HyperparameterValueTraining SetupSequence Length4096Sample PackingFalseNum Epochs3Micro Batch Size4Gradient Accumulation Steps4OptimizationOptimizerAdamWLearning Rate2×10−42\\times 10^\{\-4\}LR SchedulerCosineWarmup Ratio0\.1Weight Decay0\.1Table 4:Fine\-tuning hyperparameters\.
## Appendix HL2 Development Assessment: English–Japanese Translation

We observe similar trends in the Eng–Jpn direction in Figure[22](https://arxiv.org/html/2606.19170#A10.F22)\. Because Dango is native in Japanese, it reaches strong performance on easier levels more quickly than in the Jpn–Eng direction\. BLEU saturates earlier, and the performance gap relative to llm\-jp\-3 is smaller than in Japanese–English translation\. However, Dango still struggles with generalization to unseen difficulty levels compared to llm\-jp\-3, especially when evaluated on harder data than it has seen during training\.

## Appendix IL2 Development Assessment: Does BLEU Capture Proficiency Development?

We conduct a human evaluation to examine whether the observed BLEU improvement reflects meaningful gains in linguistic development, rather than only changes in an automatic metric\.

To assess the validity of BLEU as a proxy, we perform a blinded human evaluation across two Dango checkpoints trained on different proficiency ranges\. Specifically, we compare a checkpoint trained on A1–B1 data with one trained on A1–B2 data, and evaluate both on B2\-level test examples\. This setting corresponds to “Dango, trained on A1\+A2\+B1 / A1\+A2\+B1\+B2, tested on B2” in Figure[4](https://arxiv.org/html/2606.19170#S4.F4)\. Since B2 is out\-of\-distribution for the A1–B1 checkpoint, this comparison probes the seen/unseen proficiency boundary and provides an informative test of whether additional proficiency\-level training improves adequacy\.

A human annotator \(one of the author\) with C2\-level English proficiency, who is a non\-native speaker, blindly \(not knowing which model produced the output\) rated each model output on a 1–5 correctness scale, with the GPT\-5\.2 generated translation as gold reference\. A score of 5 indicates a fully correct translation with no omissions or unrelated content, 3 indicates that at least one key meaning unit is missing or mistranslated, and 1 indicates that the output is completely incorrect or unrelated\.

ModelBLEUAvg\. human correctness\(1–5\)Dango \[A1–B1\]17\.92\.76Dango \[A1–B2\]27\.83\.04Table 5:Human evaluation results on the B2 test set \(n=50n=50\)\.As shown in Table[5](https://arxiv.org/html/2606.19170#A9.T5), the checkpoint with the higher BLEU score also receives a higher average human correctness score\. This suggests that, in this setting, BLEU tracks meaningful improvements in translation adequacy, although it should still be interpreted as an automatic proxy rather than a direct measure of L2 proficiency development\.

### I\.1Case Study on Error Pattern and Proficiency Development

We present a case study examining Dango’s error patterns and proficiency development\. As shown in Figure[17](https://arxiv.org/html/2606.19170#A10.F17), Dango exhibits human\-like L2 errors, such as article omission and non\-native phrasing\. As it is exposed to more training data, it successfully learns to avoid these mistakes\.

## Appendix JL2 Human\-like Linguistic Feature Assessment Details

We adopted the framework ofGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\)to extract linguistic features and measure their usage frequency and error rate in model translations and human English productions\. Following their setup, we used GPT\-4o as the annotator with temperature set to 0, and parsed each instance once\.

##### Prompt Designs\.

SinceGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\)originally designed prompts for dialogue annotation, we adapted their prompts for single\-sentence model outputs by instructing the LLM to annotate one sentence at a time\.

For human English productions, we further adapted the prompts for written essays in the main experiment\. Each essay was split into 5\-sentence chunks, and sentences with fewer than 4 words were skipped\. For the extended analysis on ICNALE Spoken Dialogues \(Appendix[J\.2](https://arxiv.org/html/2606.19170#A10.SS2)\), we used the original dialogue\-based prompt design\. The full prompt set followsGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\)and their repository\.191919[https://github\.com/RenaGao/LLMPirorknowledge/tree/main](https://github.com/RenaGao/LLMPirorknowledge/tree/main)

We show the prompt designs used for single\-sentence model outputs\. Figure[15](https://arxiv.org/html/2606.19170#A10.F15)presents the system prompt with general instructions for linguistic feature extraction, while Figure[16](https://arxiv.org/html/2606.19170#A10.F16)gives the user prompt for parsing theNumber Agreementcategory\. All prompts used in our experiments will be released in the open\-source repository for the camera\-ready version\.

![Refer to caption](https://arxiv.org/html/2606.19170v1/x12.png)Figure 15:System prompt for linguistic feature annotation\. We build on the framework ofGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\); text in red indicates our modifications\.![Refer to caption](https://arxiv.org/html/2606.19170v1/x13.png)Figure 16:User prompt for annotatingNumbers Agreement\.
### J\.1L2 Linguistic Feature Distribution Construction and JSD/BCD Calculation

The construction of linguistic feature profiles follows the same procedure for both usage frequency \(UF\) and error rate \(ER\)\. For each model or human learner group, we first aggregate the judge annotations across the seven linguistic feature categories\. For UF, we count how often each feature category appears \(even if the usage is wrong\)\. For ER, we compute the proportion of incorrect instances when they appear within each feature category\.

For Jensen–Shannon divergence \(JSD\), both UF and ER vectors are normalized into probability distributions before comparison\. That is, given a nonnegative feature vector𝐱=\(x1,…,xK\)\\mathbf\{x\}=\(x\_\{1\},\\ldots,x\_\{K\}\)overKKfeature categories, we normalize it as

pi=xi∑j=1Kxj\.p\_\{i\}=\\frac\{x\_\{i\}\}\{\\sum\_\{j=1\}^\{K\}x\_\{j\}\}\.We then compute JSD between two normalized distributions𝐩\\mathbf\{p\}and𝐪\\mathbf\{q\}:

JSD​\(𝐩,𝐪\)\\displaystyle\\mathrm\{JSD\}\(\\mathbf\{p\},\\mathbf\{q\}\)=12​KL​\(𝐩∥𝐦\)\+12​KL​\(𝐪∥𝐦\),\\displaystyle=\\frac\{1\}\{2\}\\mathrm\{KL\}\(\\mathbf\{p\}\\parallel\\mathbf\{m\}\)\+\\frac\{1\}\{2\}\\mathrm\{KL\}\(\\mathbf\{q\}\\parallel\\mathbf\{m\}\),𝐦\\displaystyle\\mathbf\{m\}=12​\(𝐩\+𝐪\)\.\\displaystyle=\\frac\{1\}\{2\}\(\\mathbf\{p\}\+\\mathbf\{q\}\)\.
In addition to JSD, we use Bray–Curtis dissimilarity \(BCD\) for ER comparison\. Unlike JSD, BCD is computed directly on the unnormalized ER vectors and therefore preserves differences in the raw magnitude of category\-wise error tendencies\. Given two nonnegative ER vectors𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}, BCD is defined as

BCD​\(𝐱,𝐲\)=∑i=1K\|xi−yi\|∑i=1K\(xi\+yi\)\.\\mathrm\{BCD\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\frac\{\\sum\_\{i=1\}^\{K\}\|x\_\{i\}\-y\_\{i\}\|\}\{\\sum\_\{i=1\}^\{K\}\(x\_\{i\}\+y\_\{i\}\)\}\.A BCD value of 0 indicates identical ER profiles, while larger values indicate greater dissimilarity\. Thus, JSD captures relative distributional shifts of UF and ER across feature categories, whereas BCD complements it by measuring the direct discrepancy between unnormalized ER profiles\.

### J\.2Human\-like L2 Linguistic Feature Assessment: Additional Analysis with Human Spoken Dialogues

We further evaluate transfer effects using another source of human English production, ICNALE Spoken DialoguesIshikawa \([2019](https://arxiv.org/html/2606.19170#bib.bib17)\)\.

Overall, the results show a trend similar to that in[5\.3](https://arxiv.org/html/2606.19170#S5.SS3)\. In Table[6](https://arxiv.org/html/2606.19170#A10.T6), Dango ranks first in modeling human transfer effects for UF JSD, third for ER JSD, and second for ER BCD\. GPT\-5\.5 also remains a strong prompting\-based baseline\.

Table[7](https://arxiv.org/html/2606.19170#A10.T7)shows that Dango’s English production largely aligns with that of Japanese learners\. English produced by Korean speakers is again closest to Dango’s production, likely because Japanese and Korean share similar word order, grammatical structures, and particle systems\. We also find that the JSDs, especially ER JSD, roughly reflect the structural similarity of these languages from Japanese\.

ModelJSD ofUFJSD ofERBCD ofERTraining\-based methodsDango0\.0100![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0214![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1491![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Unfiltered0\.0113![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.05540\.4184llm\-jp\-30\.01530\.05930\.7847Prompting\-based methodsllm\-jp\-3\.1\-instruct0\.01790\.07430\.8973GPT\-4o0\.01400\.0104![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.3761![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)GPT\-5\.50\.0104![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0186![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1123![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Table 6:LLM output linguistic feature distribution divergences to human Spoken Dialogue produced by Japanese learners\. Lower is more similar\.Human L1JSD ofUFJSD ofERBCD ofERJapanese0\.0100![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.0214![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1491![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Korean0\.01360\.0180![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1496![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Urdu0\.0123![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.02570\.1778Mandarin0\.01420\.0253![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.1430![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)Cantonese0\.01310\.04360\.2140Thai0\.01620\.03030\.1818Malay0\.0112![[Uncaptioned image]](https://arxiv.org/html/2606.19170v1/all-twemojis.pdf)0\.04040\.1863Table 7:Linguistic feature divergences between Dango and humans with different L1 backgrounds\. Human English production is from ICNALE Spoken Dialogue\.Japanese source: 最終的な決定をする前に、私たちはリスクを評価する必要があります。GT: We need to assess the risks before we make a final decision\.Dango \[A1\-\-B1\] on B2:We need to review risk before we decide the final decision\.Correctness: 4/5\(article omission: ‘‘review the risk’’; non\-native phrasing: ‘‘decide a decision’’\)Dango \[A1\-\-B2\] on B2:We need to consider the risk before we make a final decision\.Correctness: 5/5\(comprehensible\)Figure 17:Case study of Dango’s English production quality across training levels\.![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/frequency_distribution_bar_js_divergence_we.png)Figure 18:Usage Frequency distribution of human and LLMs for calculating JSD\.![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/error_rate_distribution_jsd_view_we.png)Figure 19:Error Rate distribution of human and LLMs for calculating JSD\.Japanese source: 私は踊ることが好きです。Model: I like dance\.GT: I like dancing\.\(gerund omission: ‘‘like dance’’ instead of like dancing’’\)Japanese source: 私は夕方にテレビを見ます。Model: I watch TV at evening\.GT: I watch TV in the evening\.\(preposition error: ‘‘at evening’’ instead of ‘‘in the evening’’\)Japanese source: 彼女はみんなを助けるので、天使のような人です。Model: She is a angel who helps people\.GT: She is an angel because she helps everyone\.\(article form error: ‘‘a angel’’ instead of an angel’’\)Japanese source: このテレビ番組は勉強になります。Model: This TV shows is studying\.GT: This TV show is educational\.\(non\-native phrasing: literal mistranslation of the Japanese idiom ‘‘benkyou ni naru’’; unnecessary plural: ‘‘This TV shows’’\)Japanese source: その犬は私の足を攻撃しました。Model: The dog attack my leg\.GT: The dog attacked my leg\.\(tense omission: ‘‘attack’’ instead of ‘‘attacked’’\)Figure 20:Case study of grammatical and phrasing errors in Dango’s English production\.![Refer to caption](https://arxiv.org/html/2606.19170v1/x14.png)Figure 21:L1 knowledge injection user prompt for L2 role\-playing\. The original prompt is fromGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\); text in red indicates our modifications\.![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/v3-1.8b_eng-jpn_test_BLEU_quadratic_table.png)
![Refer to caption](https://arxiv.org/html/2606.19170v1/figures/llm-jp-3-100B_eng-jpn_test_BLEU_quadratic_table.png)

Figure 22:Dango and llm\-jp\-3 when trained and evaluated on different levels of Japanese\-English data, showing L2 development progress\. The stair\-shaped black line separates seen levels \(lower left\) from unseen levels \(upper right\)\.LanguageFamilyOrderRelation to Japanese \(Distance\)JapaneseJaponicSOVBaselineKoreanKoreanicSOVClosest: Similar syntax & particle systemUrduIndo\-AryanSOVClose: Similar SOV structureMandarinSino\-TibetanSVODistant: Different word order, partial shared scriptCantoneseSino\-TibetanSVODistant: Different word order, partial shared scriptThaiKra\-DaiSVOFurthest: SVO \+ Adjectives follow nounsMalayAustronesianSVOFurthest: SVO \+ Adjectives follow nounsTable 8:Languages ranked by structural similarity to Japanese\.![Refer to caption](https://arxiv.org/html/2606.19170v1/x15.png)Figure 23:L1 knowledge injection system prompt for L2 role\-playing\. The original prompt is fromGaoet al\.\([2025](https://arxiv.org/html/2606.19170#bib.bib4)\); text in red indicates our modifications\.![Refer to caption](https://arxiv.org/html/2606.19170v1/x16.png)Figure 24:Our simplified L1 knowledge\-injection prompt for L2 role\-playing, designed for llm\-jp\-3\.1\-instruct\. System prompt was not used for llm\-jp\-3\.1\-instruct\. The prompt on the left was used in the experiment, while the prompt on the right is its English translation for reference\.Subcategoryby whitelistby Englishby ChineseTotal Dropped\# docs after filteringCommonCrawl6\.89%23\.00%0\.17%29\.96%295,309,943Kaken13\.13%54\.67%0\.00%67\.80%2,228,758WARP HTML2\.61%13\.03%0\.07%15\.70%1,659,736WARP PDF14\.10%8\.61%0\.18%22\.84%35,528,902Wikipedia7\.35%23\.82%0\.22%31\.32%1,364,529Overall7\.67%21\.65%0\.17%29\.39%336,091,868Table 9:Statistics of the Japanese corpus after filtering\.FilterParameterValueUnicode WhitelistAllowed ScriptsJapanese \(Kana\), CJK Ideographs,ASCII, Emoji, PunctuationDocument Rejection Threshold\>0\.1%\>0\.1\\%non\-whitelisted charactersEnglish \(Latin\)Ratio Threshold\>0\.40\>0\.40\(Latin chars / Total chars\)Max Consecutive Latin Words\>4\>4Max Latin Length\>20\>20charactersMin Latin Length \(Ignored\)<8<8charactersDocument Rejection Threshold\>5\.0%\>5\.0\\%violated linesChineseBlocklist SourceUnihan BMPDocument Rejection Threshold\>0\.1%\>0\.1\\%violated linesTable 10:Filtering parameters and thresholds\.