A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

# A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books
Source: [https://arxiv.org/html/2607.22376](https://arxiv.org/html/2607.22376)
Varun Ghat Ravikumar1Sina Ahmadi2Lena Jäger2Rico Sennrich2 1Department of Informatics, University of Zurich 2Department of Computational Linguistics, University of Zurich \{varunghat\.ravikumar, sina\.ahmadi, lenaann\.jaeger, rico\.sennrich\}@uzh\.ch

###### Abstract

Most endangered languages lack the parallel data required for machine translation, despite the existence of descriptive grammar books\. We introduce a pipeline that uses large language models to extract grammatical rules, example sentences, and lexicons from grammar books and generate synthetic parallel corpora for fine\-tuning—rather than feeding grammar content into prompts at inference time, as in prior work\. Validated on three typologically diverse low\-resource languages—Kalamang \(Papuan\), Tuatschin \(Romance\), and Mandan \(Siouan\)—we show that fine\-tuning on synthetic data improves over seed\-data baselines in 75% of configurations for Kalamang and 59% for Tuatschin, with best\-case ChrF\+\+ gains of \+8\.8, \+5\.3, and \+3\.3 respectively\. Through a systematic factorial study across 96 configurations varying target part\-of\-speech, retrieval granularity, and sample volume, we identify which factor combinations drive gains and where they break down\. Our results demonstrate that static linguistic documentation can be repurposed for machine translation fine\-tuning, offering a practical path towards translation tools for severely under\-resourced languages\.

![[Uncaptioned image]](https://arxiv.org/html/2607.22376v1/images/github-logo.png)[varunghat/GrammarMT](https://github.com/varunghat/GrammarMT)

## 1Introduction

Nearly half of the world’s 7,000\+ languages are endangered, and many may vanish within the next century if preservation efforts do not accelerate\(UNESCO,[2022](https://arxiv.org/html/2607.22376#bib.bib1)\)\. While high\-resource languages benefit from large\-scale monolingual and parallel corpora, most endangered and minority languages lack these basic training data\(Haddowet al\.,[2022](https://arxiv.org/html/2607.22376#bib.bib2)\)\. Despite limited digital presence, many endangered languages possess descriptive grammar books compiled through linguistic field workNordhoff and Hammarström \([2011](https://arxiv.org/html/2607.22376#bib.bib85)\), yet such data remains largely unexploited in natural language processing \(NLP\)\.

![Refer to caption](https://arxiv.org/html/2607.22376v1/x1.png)Figure 1:Overview of our methodology\. Stage 1 extracts linguistic information \(parallel sentences, dictionaries, rules\)\. Stage 2 shows the combinatorial experimental configuration\. Stage 3 generates synthetic variants via rule\-guided lexical substitution \(Tuatschin example shown\)\.Grammar books document grammatical rules, parallel sentence examples, and lexicons, offering structured descriptions of morphology, syntax, and vocabulary\(Leech,[2015](https://arxiv.org/html/2607.22376#bib.bib5)\)\.Tanzeret al\.\([2024](https://arxiv.org/html/2607.22376#bib.bib6)\)show that prompting LLMs with grammar book content improves translation over zero\-shot baselines\. However,Aycocket al\.\([2025](https://arxiv.org/html/2607.22376#bib.bib17)\)observe that LLMs primarily benefit from embedded parallel sentences rather than raw grammatical descriptions\. In contrast,Zhanget al\.\([2025](https://arxiv.org/html/2607.22376#bib.bib21)\)show that when grammatical rules are manually codified into structured representations, they do improve both translation quality and rule retrieval\. Despite the growing interest in leveraging grammar books and reasoning capabilities of large language models \(LLMs\) for low\-resourced NLP\(Hus and Anastasopoulos,[2024b](https://arxiv.org/html/2607.22376#bib.bib81); Spencer and Kongborrirak,[2025](https://arxiv.org/html/2607.22376#bib.bib80),inter alia\), the impact of the distinction between narrative descriptions and codified rules has not been thoroughly examined\. This motivates our central question:*whether the structured content of grammar books can be operationalized computationally, and use explicit grammatical instructions to generate novel sentences\.*

We introduce a semi\-automated framework for synthetic data generation from grammar books, requiring only minimal supervision in the form of∼\\sim200 annotated lines per language to train the sentence classifier\. Automating this process is necessary because manual rule codification requires expert linguistic effort that does not scale to the thousands of languages with existing grammars\. Our pipeline, illustrated in Figure[1](https://arxiv.org/html/2607.22376#S1.F1), extracts parallel sentences, word\-lists, and grammatical rules from unstructured PDF text\. These components are then used to generate synthetic parallel corpora by replacing words in extracted sentence templates while applying retrieved grammatical rules to maintain morphological and syntactic correctness\. We evaluate the generated corpora by fine\-tuning Gemini\-2\.5\-flash on the synthetic data and measuring translation performance on held\-out test sets from three typologically diverse and severely low\-resource languages: Kalamang \(agglutinative, Papuan\), Tuatschin \(analytic, Romance\), and Mandan \(polysynthetic, Siouan\)\. We selected Gemini\-2\.5\-flash for its fine\-tuning API availability and low inference cost\. Through ablation studies, we analyze the impact of retrieval granularity \(rule\-level vs\. section\-level\), target part\-of\-speech, and generation volume on translation quality\.

## 2Related Work

Machine translation \(MT\) performance relies on large\-scale parallel corpora\. For low\-resource languages, massively multilingual models such as NLLB\(NLLB Teamet al\.,[2022](https://arxiv.org/html/2607.22376#bib.bib28)\)apply cross\-lingual transfer from high\-resource languages, but performance degrades significantly for languages absent from pre\-training data\(Ahmadiet al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib82)\)\. Back\-translation\(Sennrichet al\.,[2016](https://arxiv.org/html/2607.22376#bib.bib44)\)generates synthetic source sentences from monolingual target text but requires an initial translation model of sufficient quality, which is often unavailable for extremely low\-resource languages\.Frontull and Moser \([2024](https://arxiv.org/html/2607.22376#bib.bib64)\)find that combining neural models with explicit rule\-based constraints helps preserve structural integrity when parallel data is scarce, motivating the use of grammatical rules in our pipeline\.

A separate line of work has explored using descriptive grammar books to support translation for languages without web data\.Tanzeret al\.\([2024](https://arxiv.org/html/2607.22376#bib.bib6)\)show that LLMs can perform translation when prompted with entire grammar books, thoughAycocket al\.\([2025](https://arxiv.org/html/2607.22376#bib.bib17)\)find that models primarily learn from embedded parallel examples rather than descriptions\.Hus and Anastasopoulos \([2024a](https://arxiv.org/html/2607.22376#bib.bib7)\)corroborate this finding across 16 languages, observing that grammar book prompting yields only modest gains at high computational cost due to token\-heavy context injection\. Other work has pursued structured retrieval to reduce context injection costs\(Guoet al\.,[2024](https://arxiv.org/html/2607.22376#bib.bib9); Zebazeet al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib10)\), whileZhanget al\.\([2024](https://arxiv.org/html/2607.22376#bib.bib18)\)chain morphological analyzers and dictionaries, requiring pre\-existing tools rarely available for low\-resource languages\. Most recently,Zhanget al\.\([2025](https://arxiv.org/html/2607.22376#bib.bib21)\)convert grammatical rules into executable code to guide translation, achieving strong results but relying on manual expert annotation\.

Complementary to direct translation, synthetic data generation provides an alternative augmentation strategy\. Prior work has explored lexical substitution scored by language models\(Fadaeeet al\.,[2017](https://arxiv.org/html/2607.22376#bib.bib84)\), learning joint source\-target probabilities\(Dinget al\.,[2020](https://arxiv.org/html/2607.22376#bib.bib54)\), and combining morphological features with dictionary entries\(Alamet al\.,[2024](https://arxiv.org/html/2607.22376#bib.bib83)\)\.Anikinaet al\.\([2025](https://arxiv.org/html/2607.22376#bib.bib12)\)find that combining target\-language demonstrations with LLM\-based revision narrows the gap with gold\-standard data to within 5%\. At the grammar\-guided end,Lucaset al\.\([2024](https://arxiv.org/html/2607.22376#bib.bib19)\)manually construct formal grammars to generate parallel corpora for Guarani, demonstrating the value of linguistic constraints but requiring expert effort\. Conversely,de Gibertet al\.\([2025](https://arxiv.org/html/2607.22376#bib.bib13)\)generate massive synthetic corpora without grammar constraints, achieving scale but lacking morphological precision\.

Current approaches face a bottleneck: they either inject entire grammar books into context windows at high computational cost with limited utilization of grammatical content\(Aycocket al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib17)\), or rely on manual expert annotation of grammatical rules\(Zhanget al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib21)\)\. Furthermore, retrieval\-augmented approaches\(Guoet al\.,[2024](https://arxiv.org/html/2607.22376#bib.bib9); Zebazeet al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib10)\)operate during inference, which does not address training data scarcity for fine\-tuned models\. Existing augmentation methods that do generate training data rely on monolingual LM probabilities\(Fadaeeet al\.,[2017](https://arxiv.org/html/2607.22376#bib.bib84)\)or unconstrained LLM generation\(de Gibertet al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib13)\)\. No existing method automatically parses grammar books with minimal human intervention, specifically to create synthetic training data for machine translation models\. Our work fills this gap\.

## 3Methodology

### 3\.1Linguistic Information Extraction

Descriptive grammar books contain parallel sentence examples in heterogeneous formats\. A typical example consists visually of one or more source\-gloss line pairs where each source line is immediately followed by its word\-by\-word morpheme annotation, concluded by an English translation\. The number of source\-gloss line pairs per example varies with sentence length, and these components are not always visually delimited\. We extract these triplets using a two\-step process:

- •First convert PDF text into individual text lines ordered top\-to\-bottom by their vertical coordinates\. A single sentence may span multiple lines; this stage operates on raw lines, not linguistic units\.
- •Then fine\-tune a BERT111bert\-base\-multilingual\-casedclassifier\(Devlinet al\.,[2019](https://arxiv.org/html/2607.22376#bib.bib68)\)on∼200\\sim 200manually annotated examples per language to identify source, gloss, and translation lines\.222Classifier hyperparameters: learning rate =2×10−52\\times 10^\{\-5\}, 5 epochs, batch size = 16\.Consecutive lines assigned the same label are then grouped to reconstruct complete source sentences, glosses, and translations\. We apply rule\-based post\-corrections exploiting the expected triplet structure to fix occasional misclassifications\.

The classifier takes each candidate line with a context window of±\\pm3 surrounding lines\. Input features include both textual context and geometric properties \(bounding box coordinates, line width, aspect ratio\)\. After extraction, we parse English translations with spaCy \(en\_core\_web\_sm\)\(Honnibalet al\.,[2020](https://arxiv.org/html/2607.22376#bib.bib69)\)to obtain morphosyntactic metadata \(e\.g\., tense, number, part\-of\-speech tags\)\.

#### Section and Dictionary Extraction\.

To isolate grammatical content, we implement a heuristic algorithm that detects section headers based on font size statistics, identifying the top\-3 distinct sizes as hierarchical levels\. Non\-grammatical sections \(e\.g\., “Acknowledgments”, “History of the community”\) are filtered using embedding\-based semantic similarity: we compute cosine similarity between section headers and a taxonomy of target categories \(e\.g\., “Morphology”, “Syntax”\) usingall\-MiniLM\-L6\-v2sentence embeddings\(Sentence\-Transformers Team,[2025](https://arxiv.org/html/2607.22376#bib.bib59)\)\. We retain only sections whose similarity scores exceed the median score across all candidate sections\. Additionally, word lists are semi\-automatically extracted from glossary sections identified via embedding\-based keyword search \(“word\-list,” “dictionary”\)\. We de\-duplicate entries and annotate them with part\-of\-speech tags using spaCy\.

ParallelMorphSent\.WordLanguageDictSent\.RulesTTRLenLenKalamang2,5587613990\.366\.34\.5Tuatschin1,7889811440\.2411\.63\.8Mandan4599215190\.644\.07\.0Table 1:Extracted information and corpus statistics based on type\-token ratio \(TTR\), lengths measured in words and characters\. Mandan’s high TTR \(0\.64\) and word length \(7\.0\) reflect on its polysynthetic morphology\.
#### Rule Extraction and Codification\.

To convert narrative grammatical descriptions into codified rules, we use a two\-stage extraction process with Gemini\-2\.5\-flash\(Google DeepMind,[2023](https://arxiv.org/html/2607.22376#bib.bib58)\)\. First, sections exceeding 500 words are semantically chunked by detecting topic shifts via sentence embedding similarity\. We set this threshold to balance chunk coherence with sufficient context for rule extraction\. Gemini is then prompted to extract rules conforming to the UniMorph annotation schema\(Batsurenet al\.,[2022](https://arxiv.org/html/2607.22376#bib.bib53)\)\. Each rule is a structured YAML object containing: target part\-of\-speech, affix type \(e\.g\., suffix, clitic\), UniMorph feature\-value pairs \(e\.g\.,CASE:ACC\), and context dependencies \(e\.g\., “applies to consonant\-final stems”\)\.

Second, extracted rules are de\-duplicated and converted into deterministic pseudo\-code functions \(ApplyRule\(STEM, POS\)\) to ensure unambiguous application during synthesis\. This codification step consolidates multiple descriptions of the same rule into a single canonical representation\. The results of this stage are summarized in Table[1](https://arxiv.org/html/2607.22376#S3.T1)\.

### 3\.2Synthetic Corpus Construction

Relying on the extracted information, we synthetically generate sentences via constrained lexical substitution considering three factors illustrated in Figure[2](https://arxiv.org/html/2607.22376#S3.F2):

Target POS\(4 categories\)RetrievalGranularity\(2 levels\)Samplevolume,kk\(4 levels\)Extracted Parallel Sentences \(∼\{\\sim\}400\)Filter byreplaceableword oftarget POSNounVerbAdjectiveAdverbRuleSectionRuleSectionRuleSectionRuleSectionk∈\{5,10,15,20\}k\\in\\\{5,10,15,20\\\}sentences per combination4×2×4=324\\times 2\\times 4=32configurations per language×\\times3 languages = 96 experiments

Figure 2:Experimental design\. The seed corpus is filtered by target part\-of\-speech to select sentences containing a replaceable word of that category\. Each subset is varied along retrieval granularity and sample volumekk\. All 32 configurations are evaluated independently per language\.#### Factor 1: Part\-of\-Speech \(POS\)\.

For each parallel sentence, we identify a replaceable content word belonging to an open POS class, i\.e\. noun, verb, adjective, or adverb, using greedy string matching against the dictionary, disambiguated via alignment with spaCy\-parsed English glosses\. Sentences with no dictionary match are excluded from that POS configuration\. We then retrieve replacement words from the dictionary subset matching the target POS, ranked by cosine similarity betweenall\-MiniLM\-L6\-v2embeddings of English definitions\. POS tags are assigned by running spaCy on each dictionary entry’s English definition in isolation; for single\-word definitions this amounts to tagging without sentential context, which introduces occasional mis\-tagging\. \(See Appendix[A](https://arxiv.org/html/2607.22376#A1)for details\)\.

#### Factor 2: Retrieval Granularity\.

For each substitution, we retrieve relevant grammatical rules using a weighted combination of embedding\-based semantic similarity \(all\-MiniLM\-L6\-v2\(Sentence\-Transformers Team,[2025](https://arxiv.org/html/2607.22376#bib.bib59)\)\) and structured feature alignment:

Sfinal=0\.7⋅Ssem\+0\.3⋅SstructS\_\{\\text\{final\}\}=0\.7\\cdot S\_\{\\text\{sem\}\}\+0\.3\\cdot S\_\{\\text\{struct\}\}\(1\)whereSsemS\_\{\\text\{sem\}\}is the cosine similarity between the query \(source sentence plus morphological gloss\) and rule embeddings, andSstructS\_\{\\text\{struct\}\}is the proportion of exact POS and UniMorph feature matches between the query and the rule\. The weighting \(0\.7/0\.3\) prioritizes semantic relevance and is a fixed hyperparameter that we do not optimize\. We hold it constant across all languages\. We retrieve the top\-5 scoring rules and prompt Gemini\-2\.5\-flash to generate the synthetic sentence by applying the retrieved rules to the replacement word within the original sentence context\. We test two retrieval granularities: \(a\) codified rules with their textual descriptions, and \(b\) full source sections from the grammar book\. The former provides compact, structured input; the latter preserves phrasal context and parallel examples that may aid generation\. To control computational costs, we limit the parallel sentence examples present in the section to a maximum of two per section\.

#### Factor 3: Sample Volume \(kk\)\.

Each seed sentence can produce multiple synthetic variants by substituting different replacement words\. We generate 20 replacements per seed sentence, then construct training sets of sizek∈\{5,10,15,20\}k\\in\\\{5,10,15,20\\\}by taking the subsets with top\-kkcandidate words ranked by embedding\-based semantic similarity with the word to be replaced \(calculated using cosine similarity of embeddings usingall\-MiniLM\-L6\-v2\(Sentence\-Transformers Team,[2025](https://arxiv.org/html/2607.22376#bib.bib59)\)\) \. The smaller training sets are therefore nested subsets of the larger ones\. Atk=20k\{=\}20with 400 seed sentences, this targets 8,000 synthetic pairs per POS and granularity combination; atk=5k\{=\}5, approximately 2,000\. After filtering malformed outputs \(4\.2% average failure rate\), training sets range from approximately 1,900 to 7,660 sentences\.

The combination of these three factors in a fully\-crossed factorial design yields an extensive number of samples in 32 configurations per language and enables the examination of the impact of each of the manipulated factors\.

### 3\.3Selected Languages

We validate the pipeline on three low\-resource languages selected to represent diverse typological profiles and distinct language families\.Kalamang\(Papuan\) is an endangered language from the Karas Islands in Indonesia, characterized by agglutinative morphology\(Visser,[2022](https://arxiv.org/html/2607.22376#bib.bib25)\)\.Tuatschin\(Romansh\) is a dialect of Sursilvan Romansh spoken in Switzerland that exhibits an analytic structure, with source data consisting primarily of narrative text\(Maurer\-Cecchini,[2021](https://arxiv.org/html/2607.22376#bib.bib27)\)\.Mandan\(Siouan\) is a critically endangered polysynthetic language from Fort Berthold, USA\(Kasak,[2024](https://arxiv.org/html/2607.22376#bib.bib26)\), featuring complex morphology with a high verb\-to\-noun rule ratio\. Due to the lack of a glossary in the source grammar, the Mandan lexicon was extracted from the externalComparative Siouan Dictionary\(Rankinet al\.,[2015](https://arxiv.org/html/2607.22376#bib.bib67)\)\.

## 4Experiments

### 4\.1Experimental Setup

#### Baselines\.

We compare synthetic data augmentation against five baselines\. First, we prompt three foundation models \(GPT\-4\.1, GPT\-5\.2, Gemini\-2\.5\-flash\) in a zero\-shot setting without language\-specific fine\-tuning\. Second, we use NLLB\-200\(NLLB Teamet al\.,[2022](https://arxiv.org/html/2607.22376#bib.bib28)\), a massively multilingual translation model, with typologically or geographically related proxy languages \(Indonesian for Kalamang, Romansh for Tuatschin, Dakota for Mandan\)\. Third, we define Seed\-Data Fine\-Tuning \(SDFT\) as our primary control by fine\-tuning Gemini\-2\.5\-flash exclusively on the 750–900 authentic parallel sentences extracted from each grammar book\. This baseline isolates the contribution of synthetic augmentation relative to the limited available gold\-standard data\. Note that prior work on grammar book translation\(Tanzeret al\.,[2024](https://arxiv.org/html/2607.22376#bib.bib6); Aycocket al\.,[2025](https://arxiv.org/html/2607.22376#bib.bib17)\)uses different test sets and evaluation splits, so their reported scores are not directly comparable to ours\.

#### Fine\-tuning\.

We fine\-tune Gemini\-2\.5\-flash on the generated synthetic corpora for 3 epochs with a LoRA adapter rank of 1 to mitigate overfitting\.333We use the Gemini fine\-tuning API with default settings except for adapter size = 1\.Test sets comprise 500 held\-out original sentences per language, extracted directly from the grammar book; the remaining∼\{\\sim\}800 sentences serve both as the SDFT training set and as the seed pool for synthetic generation\. From this pool, we select 400 seed sentences per configuration\. We train separate models for each of the 32 configurations per language \(Figure[2](https://arxiv.org/html/2607.22376#S3.F2)\)\.

#### Metrics\.

We evaluate translation quality using BLEU\(Papineniet al\.,[2002](https://arxiv.org/html/2607.22376#bib.bib36)\), ChrF and ChrF\+\+\(Popović,[2015](https://arxiv.org/html/2607.22376#bib.bib39),[2017](https://arxiv.org/html/2607.22376#bib.bib40)\)\. We report all three throughout\. While BLEU relies on n\-grams, ChrF\+\+ provides more stable character\-level discrimination at low performance levels\. We therefore rely on ChrF\+\+ as our primary metric\.

### 4\.2Baselines vs Synthetic Augmentation

Table[2](https://arxiv.org/html/2607.22376#S4.T2)presents translation quality across all baselines and the mean performance of our synthetic augmentation across 32 configurations\. Zero\-shot approaches yield low absolute scores across all languages\. For Kalamang, GPT\-5\.2 achieves the highest zero\-shot ChrF\+\+ of12\.0312\.03, followed by GPT\-4\.1 at11\.3411\.34and Gemini\-2\.5\-flash at11\.2711\.27\. Tuatschin shows substantially higher baseline scores, likely due to related Romansh data in pre\-training corpora, with GPT\-4\.1 reaching22\.6422\.64ChrF\+\+\. Mandan yields the lowest scores, with all zero\-shot models below11\.0011\.00ChrF\+\+ and BLEU under1\.01\.0, consistent with its high morphological complexity and absence from pre\-training data\.

The NLLB\-200 model underperforms all three LLMs across all languages, even with typologically motivated proxy languages, consistent with the observation that multilingual transfer provides limited benefit in extremely low\-resource settings\.

In\-context learning \(ICL\) with grammar book content shows a striking language\-dependent pattern\. For Kalamang, ICL with full sections achieves27\.7627\.76ChrF\+\+, outperforming all other methods including our best synthetic configuration \(21\.4821\.48\)\. For Tuatschin and Mandan, however, ICL underperforms both zero\-shot models and SDFT, suggesting that grammar book context at inference time is most effective when the model has no pre\-existing knowledge of the language, or needs to be adapted to a specific dialect\.

SDFT shows ChrF\+\+ scores of12\.6812\.68for Kalamang,26\.3926\.39for Tuatschin, and13\.0913\.09for Mandan, and serves as the primary control for evaluating synthetic data contributions\. Note that SDFT already substantially outperforms all zero\-shot and ICL baselines for Tuatschin and Mandan; improvements over SDFT therefore reflect the added value of synthetic augmentation beyond what real parallel data alone provides\. Averaged across all configurations, synthetic augmentation outperforms SDFT on all metrics for Kalamang \(ChrF\+\+ 14\.71 vs\. 12\.68\)\. For Tuatschin and Mandan, the mean falls below SDFT on surface metrics\.

ModelBLEUChrFChrF\+\+Language:Kalamang→\\rightarrowEnglishNLLB1\.1210\.699\.23Gemini \(zero\-shot\)0\.7712\.6011\.27GPT\-4\.1 \(zero\-shot\)1\.6513\.0111\.34GPT\-5\.2 \(zero\-shot\)1\.8913\.2512\.03ICL Gemini \(rule\)3\.7726\.9125\.32ICL Gemini \(section\)5\.7029\.2927\.76SDFT1\.4814\.0312\.68Ours \(mean±\\pmSE\)2\.09±\\pm0\.2515\.65±\\pm0\.7814\.71±\\pm0\.75Ours \(best config\)4\.6122\.4721\.48Language:Tuatschin→\\rightarrowEnglishNLLB7\.9213\.0712\.20Gemini \(zero\-shot\)8\.3223\.3722\.28GPT\-4\.1 \(zero\-shot\)12\.8123\.4822\.64GPT\-5\.2 \(zero\-shot\)10\.3920\.4719\.58ICL Gemini \(rule\)7\.8222\.6421\.49ICL Gemini \(section\)9\.0724\.5923\.48SDFT15\.3127\.2926\.39Ours \(mean±\\pmSE\)14\.90±\\pm1\.0626\.00±\\pm1\.0925\.44±\\pm1\.05Ours \(best config\)22\.2132\.3331\.68Language:Mandan→\\rightarrowEnglishNLLB0\.6311\.379\.79Gemini \(zero\-shot\)0\.8311\.9910\.47GPT\-4\.1 \(zero\-shot\)0\.8712\.1610\.94GPT\-5\.2 \(zero\-shot\)0\.7912\.2010\.82ICL Gemini \(rule\)0\.1711\.719\.75ICL Gemini \(section\)0\.7613\.3411\.24SDFT0\.8315\.0213\.09Ours \(mean±\\pmSE\)0\.93±\\pm0\.1311\.55±\\pm0\.6510\.44±\\pm0\.58Ours \(best config\)2\.8517\.8616\.40

Table 2:Baseline and synthetic augmentation results\. Methods above the dashed line operate at inference time; methods below fine\-tune on extracted or synthetic data\. ICL prompts Gemini with extracted rules or full grammar book sections\. SDFT fine\-tunes on extracted parallel sentences only\. “Ours \(mean\)” reports the mean±\\pmSE across all 32 configurations; “Ours \(best\)” reports the single best configuration\.Boldindicates best performance per language\.
### 4\.3Fine\-tuning

Fine\-tuning on synthetically\-augmented corpora shows substantial variation across configurations and languages\. Averaged across all 32 configurations per language \(see Appendix[B](https://arxiv.org/html/2607.22376#A2)for the full table\), Kalamang shows consistent improvement over SDFT, \(mean ChrF\+\+ 14\.71±\\pm0\.75 SE, 75% of configurations above baseline\)\. Tuatschin improves atk≥10k\\geq 10but is pulled down atk=5k\{=\}5\(mean 25\.44±\\pm1\.05, 59% above\)\. Mandan rarely exceeds baseline on surface metrics \(mean 10\.44±\\pm0\.58, 16% above\)\.

We now report best\-configuration results to characterize the upper bound of pipeline performance\. The three languages exhibit distinct patterns\. Kalamang benefits most from synthetic augmentation, with the best configuration \(section\-level retrieval, adjective\-numeral replacements,k=20k\{=\}20\) withΔ=\+8\.8\\Delta=\+8\.8\. Tuatschin achieves comparable improvements under rule\-level retrieval with verb replacements atk=20k\{=\}20\(Δ\\Delta=\+5\.28\+5\.28\), though its mean across configurations falls slightly below SDFT\. Mandan proves most resistant to augmentation: 84% of configurations underperform SDFT on ChrF\+\+\.

Qualitative analysis shows improvements in specific linguistic phenomena\. For Kalamang, synthetic training enables correct reordering of adjective\-noun constructions from source SOV patterns to English SVO word order\. In Mandan, fine\-tuned models produce more accurate lexical choices, correctly identifying specific named entities \(e\.g\., “Cornsilk”\) where baseline models produce generic pronouns\. For Tuatschin, models successfully learn to interpret complex auxiliary verb constructions \(e\.g\., passive voice withvegnir\), though occasional errors arise from literal translation of etymologically transparent compound words \(e\.g\., rendering “June” as “weeder” due to the stemzarclarmeaning “to weed”\)\.

![Refer to caption](https://arxiv.org/html/2607.22376v1/x2.png)Figure 3:Distribution of translation quality across all 32 configurations per language, grouped by sample volumekk\. Red dashed lines mark the SDFT baseline\. Kalamang exceeds the baseline at allkkvalues in most configurations\. Tuatschin fails atk=5k\{=\}5but recovers at higher volumes\. Mandan rarely exceeds the baseline, with performance peaking atk=15k\{=\}15before degrading sharply atk=20k\{=\}20\.

## 5Analysis

Having conditioned the synthetic data generation on the three factors, we can analyze how each of those individually affects translation quality\.

### 5\.1Impact of Part\-of\-Speech

Figure[4](https://arxiv.org/html/2607.22376#S5.F4)shows the distribution of performance by target POS and retrieval granularity\. Averaged across all configurations, Kalamang shows positive mean gains for all POS categories, with nouns producing the most consistent improvement \(Δ\\Delta=\+2\.98±\\pm1\.06 ChrF\+\+, 75% above SDFT\)\. For Tuatschin, verbs \(Δ\\Delta=\+1\.25±\\pm0\.73\) and adverbs \(Δ\\Delta=\+0\.09±\\pm2\.34\) are the only categories that reliably improve, while adjective\-numeral configurations show high variance \(Δ\\Delta=−\-4\.01±\\pm2\.71\)\. Mandan shows negative mean deltas across all POS categories, with nouns performing least poorly \(Δ\\Delta=−\-1\.27±\\pm0\.80\)\.

The largest single\-configuration improvement is Kalamang’s adjective\-numeral combination at section\-levelk=20k\{=\}20\(Δ\\Delta=\+8\.80 ChrF\+\+\), though the mean for adjective\-numeral configurations is more modest \(Δ\\Delta=\+2\.05±\\pm1\.54\)\. Mandan’s adjective\-focused configurations are most prone to failure, with only 12% exceeding the baseline\.

### 5\.2Impact of Retrieval Granularity

The relative effectiveness of rule\-level versus section\-level retrieval varies by grammar book, as shown in Figure[4](https://arxiv.org/html/2607.22376#S5.F4)\. For Kalamang, section\-level retrieval produces higher mean gains overall \(Δ\\Delta=\+3\.55±\\pm0\.87, 88% above SDFT\) compared to rule\-level \(Δ\\Delta=\+0\.52±\\pm1\.12, 62% above\), with particularly strong advantages for verbs and adjectives\. Rule\-level retrieval is more effective only for nouns\. For Tuatschin and Mandan, the two granularities perform similarly on average\. Since we test one grammar book per language, we cannot separate the effect of the language from the effect of how the grammar is written \(e\.g\., whether some sections provide more explicit rule descriptions than others\)\. Full per\-POS breakdowns are reported in Appendix[B](https://arxiv.org/html/2607.22376#A2)\.

### 5\.3Impact of Sample Volume

The effect of increasingkkdepends on both language and configuration, as shown in Figure[3](https://arxiv.org/html/2607.22376#S4.F3)\. For Kalamang, mean performance is above baseline at all sample volumes, with the largest gains atk=10k\{=\}10\(Δ\\Delta=\+3\.64\) andk=20k\{=\}20\(Δ\\Delta=\+3\.45\)\. For Tuatschin,k=5k\{=\}5produces catastrophic failures \(Δ\\Delta=−\-7\.13, only 12% above SDFT\), but performance recovers atk≥10k\\geq 10and peaks atk=15k\{=\}15\(Δ\\Delta=\+2\.15, 100% above\)\. Since smaller training sets are nested subsets of larger ones, the general upward trend fromk=5k\{=\}5tok=15k\{=\}15is partly expected\.

Mandan exhibits non\-monotonic behavior, withk=15k\{=\}15performing least poorly \(Δ\\Delta=−\-0\.76, 38% above\) andk=20k\{=\}20degrading sharply \(Δ\\Delta=−\-3\.71, 0% above\)\. Inspection of thek=20k\{=\}20outputs shows that the model begins generating grammatical annotations rather than fluent translations, indicating that excessive synthetic data can degrade fine\-tuning performance\.

### 5\.4Interpretation of Results

![Refer to caption](https://arxiv.org/html/2607.22376v1/x3.png)Figure 4:Translation quality by target POS and retrieval granularity, aggregated across all sample volumes\. For Kalamang, section\-level retrieval produces higher medians across most POS categories\. For Tuatschin, adjective configurations show high variance while verbs are most stable\. Mandan rarely exceeds the SDFT baseline \(red dashed lines\) regardless of POS or granularityThe ablation results reveal that synthetic data effectiveness depends on the interaction between language structure and retrieval granularity\.

No single POS category is universally optimal: nouns produce the most consistent gains for Kalamang, verbs for Tuatschin, and no category reliably improves Mandan\. Some experimental configurations outperforming SDFT in Tuatschin and Mandan suggests that more comprehensive rule\-based generation strategies, beyond single\-word lexical substitution, could yield further gains\.

Mandan’s sharp degradation atk=20k\{=\}20illustrates a failure mode: when synthetic volume is excessive, the model learns the surface form of training data \(interlinear glosses\) rather than its function \(translation\)\. Inspection of these outputs shows structurally valid but semantically incoherent sentences produced by incompatible feature combinations \(e\.g\., combining a transitive verb suffix with an intransitive stem\), indicating that rule retrieval lacks semantic co\-occurrence constraints\.

We identify three recurring error types across languages\. First, models sometimes produce literal translations of compound words instead of recognizing them as expressions\. Second, polysynthetic constructions occasionally fragment into unconnected pieces\. Third, because synthetic data draws primarily from high\-frequency dictionary entries, rare morphological features remain underrepresented\.

## 6Discussion and Conclusion

We have presented the first semi\-automated pipeline for extracting linguistic information from grammar books and synthesizing parallel corpora via rule\-guided lexical substitution\. Our pipeline operates end\-to\-end on unstructured PDFs with minimal human intervention\. Validated on three typologically diverse languages, the approach achieves a mean ChrF\+\+ improvement of \+2\.0 for Kalamang across all 32 configurations \(75% above baseline\), with best\-case gains of \+8\.8 \(Kalamang\), \+5\.3 \(Tuatschin\), and \+3\.3 \(Mandan\)\. Performance is highly configuration\-dependent: Tuatschin improves reliably atk≥10k\\geq 10but not atk=5k\{=\}5, while Mandan rarely exceeds baseline\. The ablation studies further show that configuration factors interact in ways that affect both the magnitude and reliability of gains\.

From a practical standpoint, the pipeline requires only a grammar book PDF as input and produces a synthetic parallel corpus\. This makes it applicable to any language with existing grammatical documentation\. With over 2,000 endangered languages possessing grammar books but lacking parallel corpora\(Hammarströmet al\.,[2018](https://arxiv.org/html/2607.22376#bib.bib20)\), automated grammar\-to\-corpus pipelines offer a practical path toward building MT systems to languages that currently have none\. Moreover, because the pipeline produces standalone parallel corpora, the generated sentences are not tied to any specific model architecture\. They can equally be used to train smaller, locally deployable MT or NLP systems that do not depend on commercial LLM APIs\. It does not replace community\-driven language work, but it can provide baseline translation tools that support documentation, education and revitalization efforts\.

## Limitations

Several limitations constrain the generalizability of these results\. Most importantly, we rely entirely on automatic metrics; the absence of native\-speaker evaluation means we cannot distinguish genuine translation competence from test\-set overfitting\. Since we test one grammar book per language, observed configuration preferences may reflect properties of how each grammar is written, such as section structure, example density, and rule explicitness, rather than properties of the language itself\. The pipeline has been validated on three languages and its performance on tonal systems, non\-concatenative morphology, or non\-standard grammar book formats remains untested\.

Our lexical substitution mechanism lacks semantic co\-occurrence constraints, producing sentences that are syntactically valid but sometimes semantically implausible\. POS disambiguation is a further source of noise\. At the dictionary level, each entry’s English definition is tagged by spaCy in isolation; for single\-word definitions this amounts to tagging without context\. The pipeline currently depends on English\-specific NLP tools at multiple stages\. This means it supports source\-to\-English translation directly, but applying it to non\-English target languages \(e\.g\., Kalamang\-to\-Indonesian\) would require replacing these components with target\-language equivalents, which may not exist for many low\-resource settings\.

Future work should address these gaps in three directions\. First, semantic plausibility filtering, for instance using cross\-lingual entailment models to discard incoherent synthetic sentences before training, could reduce the noise that degrades Mandan at high sample volumes\. Second, extending the pipeline to the reverse direction \(English to low\-resource language\), validated with native speakers, is necessary for language revitalization applications\. Third, replacing the LLM\-based extraction stage with smaller fine\-tuned models would reduce costs and make the pipeline economically viable at scale\.

## Ethical Considerations

All grammar books used in this study are published open\-access under Creative Commons licenses through Language Science Press\. For Kalamang,Tanzeret al\.\([2024](https://arxiv.org/html/2607.22376#bib.bib6)\)report that the author obtained community consent for computational use\. We have not independently contacted the speech communities for Tuatschin or Mandan\. Our pipeline processes only published linguistic descriptions and does not collect new data from speakers\. We do not release translation systems, as output quality is not yet sufficient for deployment without community validation\.

## Declaration on Generative AI

During the preparation of this work, the authors used Gemini and ChatGPT for grammar and spelling check\. In addition, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content\.

## Acknowledgments

This research relies entirely on the foundational work of field linguists and the communities they document\. We are deeply indebted to Eline Visser, Ryan Kasak, and Philippe Maurer\-Cecchini, whose descriptive grammars of Kalamang, Mandan, and Tuatschin served as the basis of this study\. The grammar books used in this study are published open\-access under Creative Commons licenses through Language Science Press\. We also extend our respect to the speakers of these languages and hope that this work proves beneficial to them\. Sina Ahmadi gratefully thanks the support of the UZH Grant \(reference number 269093\)\.

## References

- S\. Ahmadi, R\. Sennrich, E\. Karami, A\. Marani, P\. Fekrazad, G\. A\. Baghban, H\. Hadi, S\. Heidari, M\. Dogan, P\. Asadi,et al\.\(2025\)PARME: parallel corpora for low\-resourced Middle Eastern languages\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 30032–30053\.Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p1.1)\.
- A morphologically\-aware dictionary\-based data augmentation technique for machine translation of under\-represented languages\.arXiv preprint arXiv:2402\.01939\.Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p3.1)\.
- T\. Anikina, J\. Cegin, J\. Simko, and S\. Ostermann \(2025\)A rigorous evaluation of LLM data generation strategies for low\-resource languages\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8293–8314\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.418/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.418),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p3.1)\.
- S\. Aycock, D\. Stap, D\. Wu, C\. Monz, and K\. Simaan \(2025\)Can LLMs really learn to translate a low\-resource language from one grammar book?\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 12334–12357\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/20f44da80080d76bbc35bca0027f14e6-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p4.1),[§4\.1](https://arxiv.org/html/2607.22376#S4.SS1.SSS0.Px1.p1.1)\.
- K\. Batsuren, O\. Goldman, S\. Khalifa, N\. Habash,et al\.\(2022\)UniMorph 4\.0: universal morphology\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,Marseille, France,pp\. 840–855\.External Links:[Link](https://aclanthology.org/2022.lrec-1.89)Cited by:[§3\.1](https://arxiv.org/html/2607.22376#S3.SS1.SSS0.Px2.p1.1)\.
- O\. de Gibert, J\. Attieh, T\. Vahtola, M\. Aulamo, Z\. Li, R\. Vázquez, T\. Hu, and J\. Tiedemann \(2025\)Scaling low\-resource MT via synthetic data generation with LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 27674–27692\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1408/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1408),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p3.1),[§2](https://arxiv.org/html/2607.22376#S2.p4.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[2nd item](https://arxiv.org/html/2607.22376#S3.I1.i2.p1.1)\.
- B\. Ding, L\. Liu, L\. Bing, C\. Kruengkrai, T\. H\. Nguyen, S\. Joty, L\. Si, and C\. Miao \(2020\)DAGA: data augmentation with a generation approach for low\-resource tagging tasks\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6045–6057\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.488/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.488)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p3.1)\.
- M\. Fadaee, A\. Bisazza, and C\. Monz \(2017\)Data augmentation for low\-resource neural machine translation\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 567–573\.External Links:[Link](https://aclanthology.org/P17-2090/),[Document](https://dx.doi.org/10.18653/v1/P17-2090)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p3.1),[§2](https://arxiv.org/html/2607.22376#S2.p4.1)\.
- S\. Frontull and G\. Moser \(2024\)Rule\-based, neural and LLM back\-translation: comparative insights from a variant of Ladin\.InProceedings of the Seventh Workshop on Technologies for Machine Translation of Low\-Resource Languages \(LoResMT 2024\),A\. Kr\. Ojha, C\. Liu, E\. Vylomova, F\. Pirinen, J\. Abbott, J\. Washington, N\. Oco, V\. Malykh, V\. Logacheva, and X\. Zhao \(Eds\.\),Bangkok, Thailand,pp\. 128–138\.External Links:[Link](https://aclanthology.org/2024.loresmt-1.13/),[Document](https://dx.doi.org/10.18653/v1/2024.loresmt-1.13)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p1.1)\.
- Google DeepMind \(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.External Links:[Link](https://arxiv.org/abs/2312.11805)Cited by:[§3\.1](https://arxiv.org/html/2607.22376#S3.SS1.SSS0.Px2.p1.1)\.
- P\. Guo, Y\. Ren, Y\. Hu, Y\. Li, J\. Zhang, X\. Zhang, and H\. Huang \(2024\)Teaching large language models to translate on low\-resource languages with textbook prompting\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 15685–15697\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1362/)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p4.1)\.
- B\. Haddow, R\. Bawden, A\. V\. Miceli Barone, J\. Helcl, and A\. Birch \(2022\)Survey of low\-resource machine translation\.Computational Linguistics48\(3\),pp\. 673–732\.External Links:[Link](https://aclanthology.org/2022.cl-3.6/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00446)Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p1.1)\.
- H\. Hammarström, T\. Castermans, R\. Forkel, K\. Verbeek, M\. A\. Westenberg, and B\. Speckmann \(2018\)Simultaneous visualization of language endangerment and language description\.Language Documentation & Conservation12,pp\. 359–392\.Cited by:[§6](https://arxiv.org/html/2607.22376#S6.p2.1)\.
- M\. Honnibal, I\. Montani, S\. Van Landeghem, and A\. Boyd \(2020\)spaCy: industrial\-strength natural language processing in Python\.Note:[https://spacy\.io](https://spacy.io/)Software available from spacy\.ioExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.1212303)Cited by:[§3\.1](https://arxiv.org/html/2607.22376#S3.SS1.p3.1)\.
- J\. Hus and A\. Anastasopoulos \(2024a\)Back to school: translation using grammar books\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 20207–20219\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1127/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1127)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p2.1)\.
- J\. Hus and A\. Anastasopoulos \(2024b\)Back to school: translation using grammar books\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 20207–20219\.Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p2.1)\.
- R\. Kasak \(2024\)A grammar of Mandan\.Comprehensive Grammar Library,Language Science Press,Berlin\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.14227513)Cited by:[§3\.3](https://arxiv.org/html/2607.22376#S3.SS3.p1.1)\.
- G\. Leech \(2015\)Descriptive grammar\.InThe Cambridge Handbook of English Corpus Linguistics,Cambridge Handbooks in Language and Linguistics,pp\. 146–160\.Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p2.1)\.
- A\. Lucas, A\. Baladón, V\. Pardiñas, M\. Agüero\-Torales, S\. Góngora, and L\. Chiruzzo \(2024\)Grammar\-based data augmentation for low\-resource languages: the case of Guarani\-Spanish neural machine translation\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 6385–6397\.External Links:[Link](https://aclanthology.org/2024.naacl-long.354/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.354)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p3.1)\.
- P\. Maurer\-Cecchini \(2021\)A grammar of Tuatschin\.Comprehensive Grammar Library,Language Science Press,Berlin\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.5137647)Cited by:[§3\.3](https://arxiv.org/html/2607.22376#S3.SS3.p1.1)\.
- NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi,et al\.\(2022\)No language left behind: scaling human\-centered machine translation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 7791–7807\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.532),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.532)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.22376#S4.SS1.SSS0.Px1.p1.1)\.
- S\. Nordhoff and H\. Hammarström \(2011\)Glottolog/Langdoc: defining dialects, languages, and language families as collections of resources\.InProceedings of the First International Workshop on Linked Science 2011, Bonn, Germany, October 24, 2011,T\. Kauppinen, L\. C\. Pouchard, and C\. Keßler \(Eds\.\),CEUR Workshop Proceedings, Vol\.783\.External Links:[Link](https://ceur-ws.org/Vol-783/paper7.pdf)Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)BLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting on Association for Computational Linguistics,ACL ’02,USA,pp\. 311–318\.External Links:[Link](https://doi.org/10.3115/1073083.1073135),[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[§4\.1](https://arxiv.org/html/2607.22376#S4.SS1.SSS0.Px3.p1.1)\.
- M\. Popović \(2015\)ChrF: character n\-gram F\-score for automatic MT evaluation\.InProceedings of the Tenth Workshop on Statistical Machine Translation,O\. Bojar, R\. Chatterjee, C\. Federmann, B\. Haddow, C\. Hokamp, M\. Huck, V\. Logacheva, and P\. Pecina \(Eds\.\),Lisbon, Portugal,pp\. 392–395\.External Links:[Link](https://aclanthology.org/W15-3049/),[Document](https://dx.doi.org/10.18653/v1/W15-3049)Cited by:[§4\.1](https://arxiv.org/html/2607.22376#S4.SS1.SSS0.Px3.p1.1)\.
- M\. Popović \(2017\)ChrF\+\+: words helping character n\-grams\.InProceedings of the Second Conference on Machine Translation,Copenhagen, Denmark,pp\. 612–618\.External Links:[Document](https://dx.doi.org/10.18653/v1/W17-4770),[Link](https://aclanthology.org/W17-4770/)Cited by:[§4\.1](https://arxiv.org/html/2607.22376#S4.SS1.SSS0.Px3.p1.1)\.
- R\. L\. Rankin, R\. T\. Carter, A\. W\. Jones, J\. E\. Koontz, D\. S\. Rood, and I\. Hartmann \(Eds\.\) \(2015\)Comparative siouan dictionary\.Max Planck Institute for Evolutionary Anthropology,Leipzig\.External Links:[Link](https://csd.clld.org/)Cited by:[§3\.3](https://arxiv.org/html/2607.22376#S3.SS3.p1.1)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Improving neural machine translation models with monolingual data\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 86–96\.External Links:[Link](https://aclanthology.org/P16-1009/),[Document](https://dx.doi.org/10.18653/v1/P16-1009)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p1.1)\.
- Sentence\-Transformers Team \(2025\)All\-minilm\-l6\-v2: sentence embedding model\.Note:[https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)Accessed: 2025\-11\-08Cited by:[§3\.1](https://arxiv.org/html/2607.22376#S3.SS1.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2607.22376#S3.SS2.SSS0.Px2.p1.3),[§3\.2](https://arxiv.org/html/2607.22376#S3.SS2.SSS0.Px3.p1.4)\.
- P\. T\. Spencer and N\. Kongborrirak \(2025\)Can LLMs help create grammar?: automating grammar creation for endangered languages with in\-context learning\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 10214–10227\.Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p2.1)\.
- G\. Tanzer, M\. Suzgun, E\. Visser, D\. Jurafsky, and L\. Melas\-Kyriazi \(2024\)A benchmark for learning to translate a new language from one grammar book\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 18955–18985\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/52d63f9e4b81f866bf69fb3c834aad47-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.22376#S4.SS1.SSS0.Px1.p1.1),[Ethical Considerations](https://arxiv.org/html/2607.22376#Sx2.p1.1)\.
- UNESCO \(2022\)A digital future for indigenous languages: insights from partnerships forum\.Note:Accessed: 2025\-05\-06External Links:[Link](https://www.unesco.org/en/articles/digital-future-indigenous-languages-insights-partnerships-forum)Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p1.1)\.
- E\. Visser \(2022\)A grammar of Kalamang\.Comprehensive Grammar Library,Language Science Press,Berlin\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.6499927)Cited by:[§3\.3](https://arxiv.org/html/2607.22376#S3.SS3.p1.1)\.
- A\. R\. Zebaze, B\. Sagot, and R\. Bawden \(2025\)Compositional translation: a novel LLM\-based approach for low\-resource machine translation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 22328–22357\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1216/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1216),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p4.1)\.
- C\. Zhang, J\. Lin, X\. Liu, Z\. Zhang, and Y\. Feng \(2025\)Read it in two steps: translating extremely low\-resource languages with code\-augmented grammar books\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3977–3997\.External Links:[Link](https://aclanthology.org/2025.acl-long.202/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.202),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.22376#S1.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p2.1),[§2](https://arxiv.org/html/2607.22376#S2.p4.1)\.
- C\. Zhang, X\. Liu, J\. Lin, and Y\. Feng \(2024\)Teaching large language models an unseen language on the fly\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 8783–8800\.External Links:[Link](https://aclanthology.org/2024.findings-acl.519/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.519)Cited by:[§2](https://arxiv.org/html/2607.22376#S2.p2.1)\.

## Appendix ALinguistic Information Extraction

Each dictionary entry may contain multiple English definitions \(e\.g\.,yap: Kalamang entry glossed as both “black potato” and “to divide”\)\. We assign POS tags by running spaCy on each definition independently\. When an entry has multiple definitions with conflicting POS tags, we retain all tags and allow the entry to appear in multiple POS\-specific candidate pools\. For multi\-word definitions, spaCy tags the syntactic head word; for single\-word definitions, the tag is assigned without sentential context, which can produce errors for ambiguous words \(e\.g\., “light” tagged as adjective rather than noun\)\.

During candidate retrieval, we match against the POS of the target word being replaced\. If a dictionary entry appears under multiple POS tags, it is only retrieved when its tag matches the target POS for that configuration\.

Morphosyntactic Feature Extraction\.For each dictionary entry, spaCy processes the English definition and identifies the syntactic head via dependency parsing\. The head’s POS tag \(head\_pos\) determines which substitution pool the entry is assigned to: a word is placed in a POS category if at least 50% of its senses share that head POS\. For parallel sentences, spaCy extracts per\-token features and aggregates sentence\-level properties used for rule selection \(Table[3](https://arxiv.org/html/2607.22376#A1.T3)\)\. Additionally,Caseis extracted from the morphological features of the target word being replaced and used for case\-specific rule filtering \(e\.g\., locative, accusative\)\.

StageFeatureUsageDict\. filteringhead\_posAssigns entries to POS poolsRule selectiontenseSelects tense\-related rulesnumberSelects agreement rulesgenderSelects gender\-related rulesCaseSelects case\-specific rulesTable 3:Morphosyntactic features actively used in the pipeline\. All features are extracted by spaCy from English translations\.Dictionary Processing Examples\.Table[4](https://arxiv.org/html/2607.22376#A1.T4)illustrates how spaCy processes dictionary entries\. For single\-word definitions \(e\.g\.,alar→\\rightarrow“fish”\), the token is tagged directly as the head\. For multi\-word definitions \(e\.g\.,emnem→\\rightarrow“old woman”\), spaCy identifies the syntactic head \(woman, NOUN\) via dependency parsing; the head POS determines which substitution pool the entry is assigned to\.

Lang\.EntryDefinitionHeadPOSMorph\.Single\-word definitionsKLMalarfishfishNOUNNum=SgTUAauawaterwaterNOUNNum=SgMulti\-word definitionsKLMemnemold womanwomanNOUNNum=SgTUAcavaglhorsehorseNOUNNum=Sg

Table 4:spaCy analysis of representative dictionary entries\.Headindicates the syntactic head used for POS assignment\. Single\-word entries are tagged in isolation; multi\-word entries are parsed for head identification\.
## Appendix BDetailed Results by Language

This section provides experimental results for all configurations tested across the three target languages\.

Table[5](https://arxiv.org/html/2607.22376#A2.T5)shows that Kalamang exhibits a split preference for retrieval granularity, with section\-level contexts yielding the best overall performance for adjective\-numeral replacements atk=20k\{=\}20\(ChrF\+\+ = 21\.48,Δ\\Delta= \+8\.8\)\. Rule\-level retrieval performed best for noun replacements atk=15k\{=\}15\(ChrF\+\+ = 19\.47,Δ\\Delta= \+6\.79\)\.

Language:Kalamang→\\rightarrowEnglishMetrics \(Value \(Δ\\Delta\)\)POSGranularitySamples \(k\)BLEUChrFChrF\+\+ADJ\+NUMrule52\.075​\(\+0\.595\)2\.075\(\+0\.595\)14\.943 \(\+0\.918\)13\.666 \(\+0\.991\)100\.915​\(−0\.565\)0\.915\(\-0\.565\)13\.956 \(\-0\.069\)13\.125 \(\+0\.45\)150\.213​\(−1\.267\)0\.213\(\-1\.267\)8\.098 \(\-5\.927\)7\.325 \(\-5\.35\)200\.712​\(−0\.768\)0\.712\(\-0\.768\)12\.818 \(\-1\.207\)12\.106 \(\-0\.569\)section51\.293​\(−0\.187\)1\.293\(\-0\.187\)15\.095 \(\+1\.07\)14\.04 \(\+1\.365\)102\.246​\(\+0\.766\)2\.246\(\+0\.766\)18\.992 \(\+4\.967\)17\.803 \(\+5\.128\)153\.337​\(\+1\.857\)3\.337\(\+1\.857\)19\.196 \(\+5\.171\)18\.271 \(\+5\.596\)204\.614​\(\+3\.134\)4\.614\(\+3\.134\)22\.466 \(\+8\.441\)21\.475 \(\+8\.8\)ADVrule51\.487​\(\+0\.007\)1\.487\(\+0\.007\)14\.523 \(\+0\.498\)13\.338 \(\+0\.663\)103\.128​\(\+1\.648\)3\.128\(\+1\.648\)18\.178 \(\+4\.153\)17\.286 \(\+4\.611\)153\.641​\(\+2\.161\)3\.641\(\+2\.161\)19\.777 \(\+5\.752\)18\.786 \(\+6\.111\)200\.623​\(−0\.857\)0\.623\(\-0\.857\)7\.601 \(\-6\.424\)7\.282 \(\-5\.393\)section51\.507​\(\+0\.027\)1\.507\(\+0\.027\)14\.66 \(\+0\.635\)13\.46 \(\+0\.785\)103\.252​\(\+1\.772\)3\.252\(\+1\.772\)18\.041 \(\+4\.016\)16\.952 \(\+4\.277\)150\.419​\(−1\.061\)0\.419\(\-1\.061\)8\.982 \(\-5\.043\)8\.332 \(\-4\.343\)204\.505​\(\+3\.025\)4\.505\(\+3\.025\)19\.419 \(\+5\.394\)18\.28 \(\+5\.605\)NOUNrule50\.694​\(−0\.786\)0\.694\(\-0\.786\)12\.984 \(\-1\.041\)11\.892 \(\-0\.783\)102\.032​\(\+0\.552\)2\.032\(\+0\.552\)18\.546 \(\+4\.521\)17\.343 \(\+4\.668\)154\.891 \(\+3\.411\)20\.524 \(\+6\.499\)19\.467 \(\+6\.792\)202\.739​\(\+1\.259\)2\.739\(\+1\.259\)18\.223 \(\+4\.198\)17\.353 \(\+4\.678\)section52\.267​\(\+0\.787\)2\.267\(\+0\.787\)15\.939 \(\+1\.914\)14\.538 \(\+1\.863\)102\.897​\(\+1\.417\)2\.897\(\+1\.417\)18\.729 \(\+4\.704\)17\.55 \(\+4\.875\)150\.484​\(−0\.996\)0\.484\(\-0\.996\)11\.177 \(\-2\.848\)10\.816 \(\-1\.859\)201\.635​\(\+0\.155\)1\.635\(\+0\.155\)17\.052 \(\+3\.027\)16\.276 \(\+3\.601\)VERBrule50\.268​\(−1\.212\)0\.268\(\-1\.212\)8\.538 \(\-5\.487\)7\.835 \(\-4\.84\)101\.251​\(−0\.229\)1\.251\(\-0\.229\)14\.747 \(\+0\.722\)13\.838 \(\+1\.163\)150\.215​\(−1\.265\)0\.215\(\-1\.265\)5\.034 \(\-8\.991\)4\.822 \(\-7\.853\)201\.87​\(\+0\.39\)1\.87\(\+0\.39\)16\.101 \(\+2\.076\)15\.591 \(\+2\.916\)section51\.311​\(−0\.169\)1\.311\(\-0\.169\)16\.319 \(\+2\.294\)14\.779 \(\+2\.104\)102\.725​\(\+1\.245\)2\.725\(\+1\.245\)17\.66 \(\+3\.635\)16\.613 \(\+3\.938\)153\.601​\(\+2\.121\)3\.601\(\+2\.121\)20\.785 \(\+6\.76\)19\.729 \(\+7\.054\)203\.987​\(\+2\.507\)3\.987\(\+2\.507\)21\.612 \(\+7\.587\)20\.63 \(\+7\.955\)Table 5:Translation performance on Kalamang test sets across all experimental configurations\. Results are organized by target POS, retrieval granularity, and sample volume \(kk\)\. Values in parentheses show absolute improvement \(Δ\\Delta\) over SDFT baseline\. Bold values indicate best performance per metric\.Table[6](https://arxiv.org/html/2607.22376#A2.T6)shows Tuatschin’s preference for rule\-level retrieval, particularly for verb replacements\. The best configuration achieved ChrF\+\+ = 31\.68 \(Δ\\Delta= \+5\.28\) using rule\-level retrieval with verbs atk=20k\{=\}20\.

Language:Tuatschin→\\rightarrowEnglishMetrics \(Value \(Δ\\Delta\)\)POSGranularitySamples \(k\)BLEUChrFChrF\+\+ADJ\+NUMrule50\.505​\(−14\.804\)0\.505\(\-14\.804\)5\.08 \(\-22\.211\)4\.875 \(\-21\.522\)107\.418​\(−7\.891\)7\.418\(\-7\.891\)22\.349 \(\-4\.942\)21\.909 \(\-4\.488\)1519\.29​\(\+3\.981\)19\.29\(\+3\.981\)28\.993 \(\+1\.702\)28\.305 \(\+1\.908\)207\.2​\(−8\.109\)7\.2\(\-8\.109\)22\.139 \(\-5\.152\)21\.705 \(\-4\.692\)section59\.834​\(−5\.475\)9\.834\(\-5\.475\)23\.248 \(\-4\.043\)22\.59 \(\-3\.807\)1018\.382​\(\+3\.073\)18\.382\(\+3\.073\)28\.584 \(\+1\.293\)27\.928 \(\+1\.531\)1518\.617​\(\+3\.308\)18\.617\(\+3\.308\)29\.06 \(\+1\.769\)28\.382 \(\+1\.985\)209\.764​\(−5\.545\)9\.764\(\-5\.545\)24\.063 \(\-3\.228\)23\.437 \(\-2\.96\)ADVrule514\.096​\(−1\.213\)14\.096\(\-1\.213\)26\.904 \(\-0\.387\)26\.196 \(\-0\.201\)1019\.478​\(\+4\.169\)19\.478\(\+4\.169\)29\.881 \(\+2\.59\)29\.278 \(\+2\.881\)1520\.075​\(\+4\.766\)20\.075\(\+4\.766\)30\.46 \(\+3\.169\)29\.851 \(\+3\.454\)2020\.029​\(\+4\.72\)20\.029\(\+4\.72\)30\.395 \(\+3\.104\)29\.665 \(\+3\.268\)section51\.831​\(−13\.478\)1\.831\(\-13\.478\)10\.87 \(\-16\.421\)10\.392 \(\-16\.005\)1018\.994​\(\+3\.685\)18\.994\(\+3\.685\)28\.783 \(\+1\.492\)28\.073 \(\+1\.676\)1518\.675​\(\+3\.366\)18\.675\(\+3\.366\)29\.96 \(\+2\.669\)29\.27 \(\+2\.873\)2019\.023​\(\+3\.714\)19\.023\(\+3\.714\)29\.819 \(\+2\.528\)29\.142 \(\+2\.745\)NOUNrule512\.485​\(−2\.824\)12\.485\(\-2\.824\)24\.925 \(\-2\.366\)24\.198 \(\-2\.199\)1010\.267​\(−5\.042\)10\.267\(\-5\.042\)24\.759 \(\-2\.532\)24\.252 \(\-2\.145\)1519\.346​\(\+4\.037\)19\.346\(\+4\.037\)29\.901 \(\+2\.61\)29\.281 \(\+2\.884\)208\.568​\(−6\.741\)8\.568\(\-6\.741\)23\.521 \(\-3\.77\)23\.058 \(\-3\.339\)section54\.27​\(−11\.039\)4\.27\(\-11\.039\)11\.705 \(\-15\.586\)12\.855 \(\-13\.542\)1020\.545​\(\+5\.236\)20\.545\(\+5\.236\)30\.709 \(\+3\.418\)30\.072 \(\+3\.675\)1519\.763​\(\+4\.454\)19\.763\(\+4\.454\)28\.314 \(\+1\.023\)27\.714 \(\+1\.317\)2021\.048​\(\+5\.739\)21\.048\(\+5\.739\)31\.226 \(\+3\.935\)30\.605 \(\+4\.208\)VERBrule516\.698​\(\+1\.389\)16\.698\(\+1\.389\)26\.544 \(\-0\.747\)25\.863 \(\-0\.534\)1020\.847​\(\+5\.538\)20\.847\(\+5\.538\)29\.657 \(\+2\.366\)29\.122 \(\+2\.725\)1516\.65​\(\+1\.341\)16\.65\(\+1\.341\)27\.924 \(\+0\.633\)27\.275 \(\+0\.878\)2022\.209 \(\+6\.9\)32\.334 \(\+5\.043\)31\.677 \(\+5\.28\)section518\.002​\(\+2\.693\)18\.002\(\+2\.693\)27\.774 \(\+0\.483\)27\.183 \(\+0\.786\)1013\.832​\(−1\.477\)13\.832\(\-1\.477\)27\.51 \(\+0\.219\)26\.666 \(\+0\.269\)1516\.981​\(\+1\.672\)16\.981\(\+1\.672\)28\.814 \(\+1\.523\)28\.266 \(\+1\.869\)2012\.176​\(−3\.133\)12\.176\(\-3\.133\)25\.727 \(\-1\.564\)25\.136 \(\-1\.261\)Table 6:Translation performance on Tuatschin test sets across all experimental configurations\. Results are organized by target POS, retrieval granularity, and sample volume \(kk\)\. Values in parentheses show absolute improvement \(Δ\\Delta\) over SDFT baseline\. Bold values indicate best performance per metric\.Table[7](https://arxiv.org/html/2607.22376#A2.T7)illustrates Mandan’s high performance variability and strong preference for rule\-level retrieval\. The best configuration used rule\-level retrieval with adverbs atk=15k\{=\}15\(ChrF\+\+ = 16\.40,Δ\\Delta= \+3\.31\), though performance degraded sharply atk=20k\{=\}20for most POS categories\.

Language:Mandan→\\rightarrowEnglishMetrics \(Value \(Δ\\Delta\)\)POSGranularitySamples \(k\)BLEUChrFChrF\+\+ADJ\+NUMrule50\.746​\(−0\.081\)0\.746\(\-0\.081\)12\.453 \(\-2\.569\)11\.075 \(\-2\.014\)100\.074​\(−0\.753\)0\.074\(\-0\.753\)3\.499 \(\-11\.523\)3\.16 \(\-9\.929\)152\.42​\(\+1\.593\)2\.42\(\+1\.593\)16\.387 \(\+1\.365\)14\.819 \(\+1\.73\)200\.324​\(−0\.503\)0\.324\(\-0\.503\)8\.659 \(\-6\.363\)7\.889 \(\-5\.2\)section51\.34​\(\+0\.513\)1\.34\(\+0\.513\)13\.942 \(\-1\.08\)12\.296 \(\-0\.793\)100\.62​\(−0\.207\)0\.62\(\-0\.207\)11\.693 \(\-3\.329\)10\.579 \(\-2\.51\)150\.885​\(\+0\.058\)0\.885\(\+0\.058\)12\.933 \(\-2\.089\)11\.753 \(\-1\.336\)200\.083​\(−0\.744\)0\.083\(\-0\.744\)2\.053 \(\-12\.969\)2\.033 \(\-11\.056\)ADVrule50\.307​\(−0\.52\)0\.307\(\-0\.52\)10\.593 \(\-4\.429\)9\.387 \(\-3\.702\)101\.223​\(\+0\.396\)1\.223\(\+0\.396\)15\.031 \(\+0\.009\)13\.576 \(\+0\.487\)152\.847 \(\+2\.02\)17\.856 \(\+2\.834\)16\.395 \(\+3\.306\)200\.425​\(−0\.402\)0\.425\(\-0\.402\)9\.853 \(\-5\.169\)9\.094 \(\-3\.995\)section50\.363​\(−0\.464\)0\.363\(\-0\.464\)11\.842 \(\-3\.18\)10\.57 \(\-2\.519\)100\.259​\(−0\.568\)0\.259\(\-0\.568\)7\.409 \(\-7\.613\)6\.767 \(\-6\.322\)150\.299​\(−0\.528\)0\.299\(\-0\.528\)6\.69 \(\-8\.332\)6\.186 \(\-6\.903\)201\.144​\(\+0\.317\)1\.144\(\+0\.317\)14\.028 \(\-0\.994\)12\.86 \(\-0\.229\)NOUNrule50\.22​\(−0\.607\)0\.22\(\-0\.607\)8\.032 \(\-6\.99\)7\.152 \(\-5\.937\)100\.835​\(\+0\.008\)0\.835\(\+0\.008\)13\.281 \(\-1\.741\)11\.958 \(\-1\.131\)152\.806​\(\+1\.979\)2\.806\(\+1\.979\)16\.529 \(\+1\.507\)15\.237 \(\+2\.148\)200\.727​\(−0\.1\)0\.727\(\-0\.1\)11\.776 \(\-3\.246\)10\.764 \(\-2\.325\)section50\.704​\(−0\.123\)0\.704\(\-0\.123\)13\.86 \(\-1\.162\)12\.129 \(\-0\.96\)101\.585​\(\+0\.758\)1\.585\(\+0\.758\)14\.221 \(\-0\.801\)12\.642 \(\-0\.447\)151\.602​\(\+0\.775\)1\.602\(\+0\.775\)13\.741 \(\-1\.281\)12\.46 \(\-0\.629\)201\.037​\(\+0\.21\)1\.037\(\+0\.21\)13\.372 \(\-1\.65\)12\.18 \(\-0\.909\)VERBrule50\.109​\(−0\.718\)0\.109\(\-0\.718\)5\.382 \(\-9\.64\)4\.733 \(\-8\.356\)100\.665​\(−0\.162\)0\.665\(\-0\.162\)11\.489 \(\-3\.533\)10\.369 \(\-2\.72\)150\.76​\(−0\.067\)0\.76\(\-0\.067\)12\.133 \(\-2\.889\)10\.907 \(\-2\.182\)201\.024​\(\+0\.197\)1\.024\(\+0\.197\)12\.607 \(\-2\.415\)11\.519 \(\-1\.57\)section50\.893​\(\+0\.066\)0\.893\(\+0\.066\)11\.815 \(\-3\.207\)10\.515 \(\-2\.574\)102\.132​\(\+1\.305\)2\.132\(\+1\.305\)14\.979 \(\-0\.043\)13\.557 \(\+0\.468\)150\.759​\(−0\.068\)0\.759\(\-0\.068\)11\.996 \(\-3\.026\)10\.846 \(\-2\.243\)200\.534​\(−0\.293\)0\.534\(\-0\.293\)9\.361 \(\-5\.661\)8\.692 \(\-4\.397\)Table 7:Translation performance on Mandan test sets across all experimental configurations\. Results are organized by target POS, retrieval granularity, and sample volume \(kk\)\. Values in parentheses show absolute improvement \(Δ\\Delta\) over SDFT baseline\. Bold values indicate best performance per metric\.
## Appendix CQualitative results

#### Kalamang\.

Table[10](https://arxiv.org/html/2607.22376#A4.T10)presents eight translation examples from the best\-performing Kalamang model\. The outputs show that synthetic training enabled correct adjective\-noun reordering from source to English \(e\.g\., example 1\)\. The dominant failure mode is hallucination under lexical uncertainty: when the model encounters unknown vocabulary or missing arguments, it repeats salient words \(example 2\) or inserts entirely unrelated content \(examples 3 and 5\)\.

#### Tuatschin\.

Table[11](https://arxiv.org/html/2607.22376#A4.T11)presents eight translation examples from the best\-performing Tuatschin model\. The outputs demonstrate robust handling of complex auxiliary constructions: example 1 correctly interprets the passive formed withvegnir\(“to come”\) rather than producing a literal translation of movement, and example 2 is nearly fluent\. A recurring failure mode is the literal translation of etymologically transparent compounds: in example 3, the model renderszarcladur\(“June”, derived fromzarclar, “to weed”\) as “weeder” instead of recognizing it as a fixed calendar term\. Example 5 exhibits task confusion, where the model outputs a grammatical analysis of the verb \(“1st Person Singular Past Tense”\) rather than a fluent translation\.

#### Mandan\.

Table[12](https://arxiv.org/html/2607.22376#A4.T12)presents three translation examples from the best\-performing Mandan model\. Mandan proved the most difficult target, and the outputs reflect this: example 3 shows the model hallucinating an entirely unrelated narrative scene rather than translating the input\. Despite these failures, the synthetic training data taught the model specific named entities: in example 1, the model correctly identifiesPaxirúukeas “Cornsilk” where the reference translation uses only a generic pronoun\. Example 2 illustrates a failure induced by noise in the parallel sentence data, where interlinear gloss tags \(1sg\.poss\-pro\) appear directly in the English output instead of fluent text\.

## Appendix DData Synthesis Examples

This appendix illustrates the synthesis pipeline for each language’s best\-performing configuration\. For each example, we show the original sentence, the replacement target, and a sample of generated outputs\. Table[8](https://arxiv.org/html/2607.22376#A4.T8)presents representative outputs; Table[9](https://arxiv.org/html/2607.22376#A4.T9)shows the candidate rules provided to the LLM for one example\.

Kalamang → English:ADJ, section\-level,k=20k\{=\}20Original:mu pas sem=ten=at koupTranslation:“They hugged the scared woman\.”tayuonmu pas sem=tayuon=at koup→\\rightarrowThey hugged the not\-good woman\.besmu pas sem=bes=at koup→\\rightarrowThey hugged the good woman\.siktaktakmu pas sem=siktaktak=at koup→\\rightarrowThey hugged the slow woman\.Tuatschin → English:VERB, rule\-level,k=20k\{=\}20Original:Nossadùna lèva dá li gjuven préjr\.\.\.Translation:“The Holy Virgin wanted to give…”lèva…lèva dá li gjuven préjr…→\\rightarrow…wanted to give the young priest…saprèndev…saprèndev ajn dá li gjuven…→\\rightarrow…took herself to give the young priest…vulajèv…vulajèv dá li gjuven préjr…→\\rightarrow…wanted to give the young priest…Mandan → English:ADV, rule\-level,k=15k\{=\}15rąh́šit pxinaate\\tipaencodingPš→\\rightarrowHe sneezed in back of\.wįrį\- w\\textglotstopahka→\\rightarrowI \(acting on you\) sneeze\.Table 8:Representative synthesis outputs for each language’s best\-performing configuration\. Bold marks the substituted element\.\#AffixDescription1=atOBJ case clitic on NOUN2=koLOC case on demonstrative ADJ3=kaLAV case on interrogative ADJ4=saetSuperlative degree on ADJ5=kaALL case on demonstrative ADJTable 9:Candidate rules provided to the LLM for the Kalamang ADJ synthesis example above\. Rules are retrieved via the scoring function \(Eq\. 1\) and formatted as structured YAML\.SourceReferenceModel Output\[ror wa\] me tabusikThis tree is short\.That tree is small\.Kapal cicauna konA small ship\.One is small, one is small\.som\-kon oe\-te kon~konGave each person one, or gave each person one fruit\.\(The child\) has one hundred teeth, or \(the child\) has new one hundred teeth\.ma hukat= bon miaHe is coming with the net\.He/she hits the bottom net\.ma rap= et eba gier\-un iriskapWhen he laughs, his teeth are white\.He/she/it hits the whatsit, so his/her/its forehead is blue\.wa= nan im karuok weinun\[In\] this \[picture there are\] three bananas too\.This is like a twenty net\.an tok tumun\-an=at boubou=ninI haven’t yet bathed my child\.I haven’t softened my child yet\.gier\-ca me me tenThat tooth of yours is bad\.All the lobsters are dead\.Table 10:Translation examples for Kalamang → English \(best model: ADJ\-NUM, section\-level,k=20k\{=\}20\)\.SourceReferenceModel OutputLa mùma è vegnida dada in bi schénghètgMother was given a beautiful present\.\*The mother received a nice little present\.Quaj taur è bjals, ábar gròns sè ’l bétgThis bull is beautiful, but big he is not\.This bull is beautiful, but he is not big\.La scùla finéva … al matg … zarcladurSchool ended in May … June\.School ended … in May … weeder\.1920 \[f4\] vas schòn a paj\[that’s what it takes you to\] go on foot\.1920 \[f4\] you are already paying\.L’ autar dé va ju gju la lubiantscha dadI should go and make an appointment with those up there…The original verb form is “gju”, identified as 1st Person Singular Past Tense…Ju sùn dada gjù séla fatschaI fell down on my face\.I am given down faces\.Las tgautschas fòn bétg agl ùmTrousers do not make a man\.My legs don’t make the man\.Las nòtízjas sa ju bétg danùndar als gjanitursI don’t know where my parents had the news from\.The news I don’t know where my parents had them from\.Table 11:Translation examples for Tuatschin → English \(best model: VERB, rule\-level,k=20k\{=\}20\)\.SourceReferenceModel OutputPaxirúukeena miní hįįhereHe was drinking water\.Cornsilk made him drink water\.Mí’shak máa’ąk íwasek pshíiwahaani…My land…1sg\.poss\-proland…Kináatani máapte kaní’ni kasími réehoomako’shAll around, there were very thick tall trees\.He got up again, climbed up the river bank, and he set off traveling\.Table 12:Translation examples for Mandan → English \(best model: ADJ\-NUM, rule\-level,k=15k\{=\}15\)\.
## Appendix EPrompts

This section contains all the prompt templates used\. The rule extraction prompt is given in Figure[5](https://arxiv.org/html/2607.22376#A5.F5)\. The output of these rules were cleaned, deduplicated and codified using the prompt in Figure[6](https://arxiv.org/html/2607.22376#A5.F6)\.

You are an expert descriptive linguist and data structurer\. Your core task is toextract morphological and syntactic rulesfrom a grammar book paragraph for a low\-resource language\. These rules will be used by a downstream AI agent to synthesize valid sentences\.
Selection Criteria
Extract a rule IF AND ONLY IF:
1\. It involves anovert surface change\(suffix, prefix, clitic, particle, or mutation\)\.
2\. It isproductive\(applies generally to a class of words, not one specific exception\)\.
3\. It has a cleargrammatical function\(Case, Tense, Mood, Aspect, Agreement\)\.
Do NOT extract: purely phonological rules, typological trivia, examples, or footnotes\.
Output Schema \(YAML\)
category: surface\_rule
description:<POS, morpheme, and grammatical function\>
target\_pos:<NOUN \| VERB \| ADJ \| PRON\>
affix\_type:<SUFFIX \| PREFIX \| CLITIC \| PARTICLE \| \.\.\.\>
morpheme:<literal string, e\.g\., "\-at", "ko="\>
application\_string:<e\.g\., STEM \+ "\-an"\>
unimorph\_feature:<e\.g\., CASE, TENSE, AGR, NUM\>
unimorph\_value:<e\.g\., ACC, PST, PL, 3SG\>
context\_dependency:<phonological/agreement environment, or N/A\>
semantic\_trigger:<when to apply this rule\>
Output Handling
If no rule is found: output category: N/A and description: "No applicable rule found\."
Task
Process the following paragraph: \{input\_paragraph\}Figure 5:Prompt template for grammar rule extraction\. The model outputs one YAML block per extracted surface rule, or a fixedN/Ablock if no applicable rule is found\. The\{input\_paragraph\}placeholder is filled with a single paragraph from the parsed grammar book\.You are a computational linguist\.
Task: Translate the single input rule into a concise pseudo\-code function, ensuring the output YAML is perfectly parsable\.
Output Schema \(YAML\)
canonical\_description:<single sentence synthesizing
meaning, usage, and all conditions\>
lrl\_code:<pseudo\-code function applying the rule\>
Codification Rules
1\. Function MUST be named ApplyRule\(STEM, POS\)\.
2\. Affixation must use the application\_string field\.
3\. Implement all conditions with IF/ELSE IF checks
\(e\.g\., IF STEM\_ENDS\_IN\(’x’\), IF POS == ’Y’\)\.
4\. If conditions are not met, MUST return the original
STEM\.
Input Rule \(YAML\)
Process this rule\.
\-\-\-
\{input\_rule\}
\-\-\-Figure 6:Prompt template for rule codification\. Each extracted surface rule \(in YAML\) is converted into acanonical\_descriptionand a deterministicApplyRule\(STEM, POS\)pseudo\-code function\. The codified rules are indexed for semantic retrieval and serve as the grammar representation in therule\-granularity generation condition\.The following template \(Figure[7](https://arxiv.org/html/2607.22376#A5.F7)\) is used across all model configurations\. Forrulegranularity,\{rules\_data\}contains individual morphological rules \(e\.g\., clitic attachment patterns\)\. Forsectiongranularity, it contains full grammar book sections\.

You are a professional linguist for \{lang\}\.
Task: Replace the \{pos\} in the sentence using provided rules\. Output must be concise\.
Constraints
\-\- Deduce existing clitics by comparing the original sentence to the provided stem\.
\-\- Allomorphy: Apply the correct variant based on the new stem’s ending\.
\-\- You MUST cite Rule \#s for all changes\.
Inputs
\-\- \{granularity\_statement\}: \{rules\_data\}
\-\- Sentence: \{sentence\_text\}
\-\- Translation: \{translation\}
\-\- Target: Replace \{stem\_to\_replace\} \(\{original\_gloss\_features\}\) with \{word\} \("\{word\_translation\}"\) \[POS: \{pos\}\]
Step\-by\-step reasoning
1\. Context: \[Target word\] \+ \[Deduced clitics\]
2\. Rule Selection: \[Rule \#s\] \+ \[Justification\]
3\. Transformation: \[New stem\] \+ \[Applied changes\]
4\. Final Sentence: \[\{lang\} result\]
5\. Translation: \[English result\]
Output format
final\_sentence: "\.\.\."
english\_translation: "\.\.\."Figure 7:Prompt template for grammar\-aware synthetic generation\. Placeholders \(in braces\) are filled per language and configuration\. Thegranularity\_statementfield contains either codified rules or full grammar book sections depending on the retrieval granularity condition\.The prompt for the ICL baseline is found in Figure[8](https://arxiv.org/html/2607.22376#A5.F8)\. The prompt in Figure[9](https://arxiv.org/html/2607.22376#A5.F9)was used for finetuning, zero\-shot inference and final inference from the finetuned \(including SDFT\) models\.

You are a professional linguist for \{lang\}\.
Task: Translate the sentence into English using the provided grammatical rules\. Output must be concise\. Use fragments, not full paragraphs\.
Constraints
\-\- Use the provided rules to interpret morphological forms in the sentence\.
\-\- Use the dictionary entries to identify known word meanings\.
\-\- Citation: You MUST cite Rule \#s used in your reasoning\.
Inputs
\-\- \{granularity\_statement\}:
\{rules\_data\}
\-\- Sentence: \{sentence\_text\}
\-\- Dictionary \(source words found in lexicon\):
\{dictionary\_entries\}
Step\-by\-step reasoning \(Direct & Brief\)
1\. Segmentation: \[Words \+ clitics/morphology identified\]\.
2\. Rule Selection: \[Rule \#s\] used \+ \[1\-phrase justification\]\.
3\. Word Meanings: \[Known words from dictionary\]\.
4\. Translation: \[English result\]\.
Output format
english\_translation: "\.\.\."Figure 8:Prompt template for the grammar\-aware ICL translation baseline\. Unlike the generation prompt \(Figure[7](https://arxiv.org/html/2607.22376#A5.F7)\), there is no target word to substitute; instead the model translates the full source sentence directly\. Thegranularity\_statementfield selects between codified rules \(rulecondition\) and raw grammar book paragraphs \(sectioncondition\)\. Dictionary entries are retrieved by tokenising the source sentence and looking up sub\-morphemes in the bilingual lexicon\.User turn
Translate to english from \{language\_name\}: \{source\_sentence\}
Model turn\(training target only\)
\{english\_translation\}Figure 9:Prompt template used for Gemini fine\-tuning and baseline batch inference\. Each example is formatted as a two\-turn conversation: the user provides the source sentence prefixed with a language identifier, and the model response is the English translation\. At inference time only the user turn is sent; the model turn is omitted\.