Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

# Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation
Source: [https://arxiv.org/html/2607.26286](https://arxiv.org/html/2607.26286)
###### Abstract

Large language models \(LLMs\) are increasingly used as general\-purpose translation systems, but their behavior is usually evaluated under a single prompt shape: translate one source sentence into one target language\. In practice, users may ask for one target language, for several related languages at once, or for translations conditioned on examples\. This paper studies prompt scope and demonstration selection as experimental variables for local LLM machine translation\. We evaluate English\-to\-Romance and English\-to\-Germanic translation on the full FLORES devtest split for nine official European Union languages\. We compare three local instruction\-tuned LLMs,llama3\.2:3b,mistral:latest, andqwen2\.5:14b, against dedicated MT baselines from OPUS\-MT and NLLB\-200\. We test zero\-shot prompting andk=5k=5few\-shot prompting with random, lexical\-similarity, and embedding\-similarity demonstration selection\. We also compare single\-target prompts with JSON\-formatted family\-scope prompts that request all languages in a family at once\. Results show that dedicated MT systems remain strongest overall, especially for Germanic languages\. Few\-shot prompting helpsmistral:latestandqwen2\.5:14b, but hurtsllama3\.2:3b; embedding retrieval is best on average for the stronger LLMs, but its advantage over random and lexical examples is modest\. Family\-scope prompting is feasible for stronger local LLMs but exposes structured\-output failures in smaller models\. These findings motivate evaluating LLM translation not only by language pair and metric, but also by prompt scope, retrieval strategy, and multi\-target compliance\.

## 1Introduction

Machine translation \(MT\) evaluation traditionally treats translation as a language\-pair task: a system is asked to translate from one source language into one target language, and the output is compared against reference translations\. This framing remains appropriate for dedicated MT systems, but it does not fully capture how large language models \(LLMs\) are used as translation tools\. LLM users can request one translation, several translations, or a set of related target languages in a single instruction\. They can also provide examples, style constraints, and structured\-output requirements\. These extra degrees of freedom make prompt design part of the translation system itself\.

This paper asks whether prompt scope should be treated as an explicit MT evaluation variable\. We focus on local LLMs that can run on consumer hardware rather than on proprietary high\-end systems\. This setting matters because local deployment is attractive for privacy, cost control, and offline use, but smaller local models may be more sensitive to prompt length, example formatting, and structured\-output requirements\. We evaluate two prompt scopes\. In the*single\-target*scope, the model translates English into one target language at a time\. In the*family\-scope*setting, the model is asked to translate the same English sentence into all languages in a related family, returning a JSON object keyed by FLORES language codes\.

We also study few\-shot example selection\. Prior work has shown that LLMs can be sensitive to in\-context examples, but it is not obvious whether semantically similar examples improve translation more than random examples, especially in a multi\-target prompt\. We therefore compare zero\-shot prompting withk=5k=5random examples, lexical\-similarity examples, and embedding\-similarity examples\. Few\-shot demonstrations are drawn from FLORES dev and evaluation is performed on FLORES devtest, avoiding leakage from the test split\.

Our empirical study covers English\-to\-Romance and English\-to\-Germanic translation for nine official EU languages: French, Spanish, Italian, Portuguese, Romanian, German, Dutch, Danish, and Swedish\. We evaluate three local instruction\-tuned LLMs and compare them against dedicated MT baselines from OPUS\-MT/MarianMT and NLLB\-200\. We report sacreBLEU BLEU, chrF\+\+, COMET, paired bootstrap confidence intervals for key comparisons, sentence\-level retrieval diagnostics, and family\-scope compliance metrics such as target coverage and complete\-output rate\.

The results show that prompt scope and demonstration strategy materially affect local LLM translation\. Dedicated MT systems remain strongest overall, but local LLMs are competitive for some Romance targets\. Few\-shot prompting helpsmistral:latestandqwen2\.5:14b, while degradingllama3\.2:3b\. Embedding\-based retrieval is best on average for stronger models, but its margin over random and lexical retrieval is smaller than the differences between models\. Family\-scope prompting is reliable for stronger LLMs but can fail severely for smaller models, especially in the Germanic setting\. The contribution of this paper is therefore not a new translation model, but an evaluation design and empirical analysis showing that prompt scope and structured\-output compliance should be reported when LLMs are used for MT\.

## 2Related Work

#### Machine translation benchmarks and metrics\.

FLORES\-200 was introduced with No Language Left Behind \(NLLB\) as a multilingual evaluation benchmark covering many translation directions and professionally translated evaluation data\[[10](https://arxiv.org/html/2607.26286#bib.bib1)\]\. We use FLORES because it provides a consistent dev/devtest split across all target languages in our study\. For automatic evaluation, BLEU remains widely reported, but score reproducibility depends on tokenization and implementation details; SacreBLEU was proposed to standardize BLEU reporting\[[13](https://arxiv.org/html/2607.26286#bib.bib2)\]\. We additionally report chrF\+\+\[[11](https://arxiv.org/html/2607.26286#bib.bib3),[12](https://arxiv.org/html/2607.26286#bib.bib4)\], which is often useful for morphologically varied languages, and COMET, a learned metric designed to better correlate with human judgments\[[15](https://arxiv.org/html/2607.26286#bib.bib5),[14](https://arxiv.org/html/2607.26286#bib.bib6)\]\.

#### LLMs for machine translation\.

Recent work has evaluated GPT\-style models as translation systems and shown that LLMs can be strong translators in high\-resource settings while still exhibiting weaknesses relative to dedicated MT systems\[[6](https://arxiv.org/html/2607.26286#bib.bib9),[19](https://arxiv.org/html/2607.26286#bib.bib12)\]\. These studies motivate treating prompting and evaluation conditions carefully\. Our work differs by focusing on local, consumer\-hardware LLMs and by making prompt scope part of the experimental design\.

#### In\-context learning and example selection\.

In\-context learning performance can depend strongly on which examples are placed in the prompt\[[3](https://arxiv.org/html/2607.26286#bib.bib10),[8](https://arxiv.org/html/2607.26286#bib.bib11)\]\. Retrieval\-based example selection has been studied for language tasks, including selecting examples similar to the test input\. For MT, however, an example can help by providing semantic analogies, terminology, output style, or simply the expected task format\. We therefore include random demonstrations as a control, lexical retrieval as a lightweight baseline, and multilingual embedding retrieval as a semantic retrieval condition\.

#### Terminology, lexical resources, and domain knowledge in MT\.

Terminology and lexical\-resource translation are relevant to our setting because they isolate a problem that also appears in prompt\-based LLM translation: translation quality often depends on whether the system receives enough semantic, lexical, or domain\-specific evidence to choose the right target expression\. Earlier work therefore provides a useful precedent for studying not only the translation model, but also the auxiliary knowledge made available to it\. McCrae et al\. combine semantic similarity with MT for cross\-lingual knowledge graph translation, while Arcan et al\. show that translating terminological expressions in knowledge bases is difficult without domain context and can benefit from adaptation or injected terminology\[[9](https://arxiv.org/html/2607.26286#bib.bib13),[2](https://arxiv.org/html/2607.26286#bib.bib14)\]\. Related work has also used multi\-way or multilingual NMT to infer dictionary entries and translate lexical resources, including closely related language settings and under\-resourced WordNet gloss translation\[[1](https://arxiv.org/html/2607.26286#bib.bib15),[4](https://arxiv.org/html/2607.26286#bib.bib16)\]\. Other studies examine whether rule\-based linguistic, terminology, and named\-entity knowledge can improve under\-resourced NMT systems\[[17](https://arxiv.org/html/2607.26286#bib.bib17),[18](https://arxiv.org/html/2607.26286#bib.bib18)\]\. We do not inject terminology dictionaries or domain resources directly; instead, we test an analogous prompt\-time question: whether retrieved demonstrations and related target languages provide useful semantic or lexical cues for local LLMs\. This connection is why these works belong in the related work, and it also motivates our empirical caution: additional knowledge may help, but its benefit must be measured against random examples and zero\-shot prompts rather than assumed\.

#### Semantic ambiguity and translation context\.

Recent multimodal MT work also argues that translation quality can depend on whether additional context helps disambiguate the source\. Hatami et al\. use semantic diversity estimates to decide when visual information is likely to improve translation\[[5](https://arxiv.org/html/2607.26286#bib.bib19)\]\. Although our experiments are text\-only, our sentence\-level retrieval analysis is motivated by a similar question: when does additional contextual evidence help, and can a similarity score predict that benefit?

#### Structured and multi\-target LLM outputs\.

LLMs are often used not only to generate text but to produce structured outputs such as JSON\. In translation, requesting multiple target languages in one response can reduce orchestration cost but creates new failure modes: omitted targets, wrong\-language outputs, prompt echoes, or malformed structures\. Our family\-scope setting evaluates this practical dimension directly by measuring both translation quality and target coverage\.

## 3Methodology

### 3\.1Task Definition

Given an English source sentencexxand a set of target languagesTT, the model must produce translationsyty\_\{t\}for eacht∈Tt\\in T\. In the single\-target setting,\|T\|=1\|T\|=1and each prompt requests exactly one translation\. In the family\-scope setting,TTis the set of languages in one family\. For Romance,T=\{French,Spanish,Italian,Portuguese,Romanian\}T=\\\{\\text\{French\},\\text\{Spanish\},\\text\{Italian\},\\text\{Portuguese\},\\text\{Romanian\}\\\}; for Germanic,T=\{German,Dutch,Danish,Swedish\}T=\\\{\\text\{German\},\\text\{Dutch\},\\text\{Danish\},\\text\{Swedish\}\\\}\. Family\-scope outputs are requested as JSON objects keyed by FLORES language codes\.

### 3\.2Prompt Conditions

We evaluate four prompting conditions for both scopes\. The zero\-shot condition gives only the translation instruction and source sentence\. The random few\-shot condition prependsk=5k=5demonstration translations selected uniformly from the demonstration pool\. The lexical retrieval condition selects the five FLORES dev examples with highest cosine similarity under bag\-of\-token source representations\. The embedding retrieval condition selects the five examples with highest cosine similarity under multilingual sentence embeddings\. All few\-shot examples are selected from FLORES dev; no evaluation examples from FLORES devtest are used as demonstrations\.

### 3\.3Single\-Target and Family\-Scope Prompts

Single\-target prompts ask for one target language and instruct the model to return only the translation\. Family\-scope prompts ask for all languages in the family in a single JSON object\. For few\-shot family prompts, each demonstration contains the English source sentence and aligned translations for all targets in the same family\. This design allows us to compare whether examples selected for the English source sentence transfer across all requested target languages\.

### 3\.4Evaluation Metrics

For translation quality, we report corpus BLEU and chrF\+\+ computed with SacreBLEU, and sentence\-level COMET means computed withUnbabel/wmt22\-comet\-da\. For paired comparisons between the best local LLM and the chrF\+\+\-best MT baseline, we use paired bootstrap resampling over matching FLORES source identifiers and report 95% confidence intervals and two\-sidedppvalues for chrF\+\+ and COMET\. For family\-scope prompting, quality metrics are computed over produced target translations\. We additionally report coverage, complete\-output count, missing\-target count, and prompt echo count\.

## 4Experimental Setup

### 4\.1Data

The main dataset is FLORES\-200\. Demonstrations are drawn from FLORES dev and evaluation is performed on the full FLORES devtest split\. The devtest split contains 1012 English source sentences\. We evaluate official EU languages from two families: Romance \(French, Spanish, Italian, Portuguese, Romanian\) and Germanic \(German, Dutch, Danish, Swedish\)\.

### 4\.2Models

The local LLMs arellama3\.2:3b,mistral:latest, andqwen2\.5:14b, run through Ollama with temperature 0\. The dedicated MT baselines arefacebook/nllb\-200\-distilled\-600M,facebook/nllb\-200\-distilled\-1\.3B, and OPUS\-MT/MarianMT models\[[10](https://arxiv.org/html/2607.26286#bib.bib1),[16](https://arxiv.org/html/2607.26286#bib.bib7),[7](https://arxiv.org/html/2607.26286#bib.bib8)\]\. The MT baselines are evaluated only in the single\-target setting, because they are designed as dedicated translation systems rather than structured multi\-target instruction followers\.

### 4\.3Reproducibility

Task files are deterministic and contain source identifiers, target language codes, references, prompt condition, prompt scope, and few\-shot metadata\. Few\-shot source audits verify that demonstrations come from FLORES dev rather than FLORES devtest\. Generation outputs are stored as JSONL files and evaluation tables are generated as CSV and TeX/Markdown summaries\. The code and generated artifacts are organized so that single\-target translation quality, family\-scope compliance, COMET scoring, bootstrap significance, and sentence\-level retrieval analysis can be reproduced independently\.

## 5Results

We report the complete FLORES devtest outcomes for two official EU language families\. Romance contains French, Spanish, Italian, Portuguese, and Romanian; Germanic contains German, Dutch, Danish, and Swedish\. For each family we evaluate \(i\) single\-target prompting, where each target language is translated separately, and \(ii\) family\-scope prompting, where the model produces all translations for the family in one JSON\-formatted response\. Single\-target prompting is the primary translation\-quality setting; family\-scope prompting is a combined translation and structured\-output compliance setting\.

All few\-shot runs usek=5k=5demonstrations retrieved from FLORES dev, while evaluation is performed on FLORES devtest\. We compare zero\-shot prompting with random demonstrations, lexical\-similarity retrieval, and embedding\-similarity retrieval\. Tables report corpus chrF\+\+, corpus BLEU, and sentence\-level COMET means\. Compact baseline and diagnostic tables use slash\-separated metric triples where noted in the caption\.

### 5\.1Romance languages

TableLABEL:tab:romance\-single\-fullgives the full single\-target Romance results for all local LLMs and prompting conditions\. The strongest local model is usuallyqwen2\.5:14b, especially for French, Spanish, Italian, and Portuguese\. Romanian is the main exception, wheremistral:latestwith lexical examples gives the strongest local chrF\+\+ score\. Few\-shot retrieval is not uniformly helpful: it consistently improvesmistral:latestandqwen2\.5:14brelative to their zero\-shot variants, but it does not improvellama3\.2:3b\.

Table 1:Romance single\-target local LLM outcomes on FLORES devtest\. Few\-shot conditions usek=5k=5examples from FLORES dev\.TargetModelConditionchrF\+\+BLEUCOMETfrallama3\.2:3bzero\-shot60\.5536\.530\.8446frallama3\.2:3brandomk=5k=560\.2136\.210\.8436frallama3\.2:3blexicalk=5k=560\.2336\.310\.8448frallama3\.2:3bembeddingk=5k=560\.1536\.120\.8438framistral:latestzero\-shot58\.0426\.760\.7992framistral:latestrandomk=5k=561\.6837\.190\.8525framistral:latestlexicalk=5k=561\.5237\.230\.8517framistral:latestembeddingk=5k=561\.3937\.380\.8539fraqwen2\.5:14bzero\-shot63\.7738\.460\.8514fraqwen2\.5:14brandomk=5k=564\.3141\.600\.8649fraqwen2\.5:14blexicalk=5k=564\.2741\.260\.8642fraqwen2\.5:14bembeddingk=5k=564\.2741\.090\.8672spallama3\.2:3bzero\-shot49\.8522\.320\.8368spallama3\.2:3brandomk=5k=549\.3621\.920\.8357spallama3\.2:3blexicalk=5k=549\.3221\.920\.8349spallama3\.2:3bembeddingk=5k=549\.7922\.330\.8390spamistral:latestzero\-shot47\.0515\.010\.7624spamistral:latestrandomk=5k=550\.5123\.660\.8415spamistral:latestlexicalk=5k=550\.5923\.810\.8427spamistral:latestembeddingk=5k=550\.7324\.020\.8448spaqwen2\.5:14bzero\-shot51\.4324\.020\.8340spaqwen2\.5:14brandomk=5k=552\.2726\.120\.8479spaqwen2\.5:14blexicalk=5k=551\.9925\.720\.8479spaqwen2\.5:14bembeddingk=5k=552\.5926\.530\.8521itallama3\.2:3bzero\-shot50\.1422\.400\.8440itallama3\.2:3brandomk=5k=549\.9621\.930\.8472itallama3\.2:3blexicalk=5k=550\.0122\.080\.8467itallama3\.2:3bembeddingk=5k=549\.9822\.470\.8452itamistral:latestzero\-shot48\.4915\.740\.7906itamistral:latestrandomk=5k=551\.1122\.920\.8514itamistral:latestlexicalk=5k=551\.3523\.670\.8559itamistral:latestembeddingk=5k=551\.7024\.130\.8537itaqwen2\.5:14bzero\-shot51\.7421\.370\.8414itaqwen2\.5:14brandomk=5k=552\.5524\.320\.8621itaqwen2\.5:14blexicalk=5k=552\.5224\.310\.8594itaqwen2\.5:14bembeddingk=5k=552\.7825\.030\.8646porllama3\.2:3bzero\-shot62\.6938\.910\.8703porllama3\.2:3brandomk=5k=561\.4437\.230\.8682porllama3\.2:3blexicalk=5k=561\.4537\.120\.8679porllama3\.2:3bembeddingk=5k=561\.4537\.010\.8696pormistral:latestzero\-shot55\.6821\.730\.7805pormistral:latestrandomk=5k=561\.6237\.090\.8698pormistral:latestlexicalk=5k=561\.7037\.410\.8692pormistral:latestembeddingk=5k=562\.1337\.910\.8728porqwen2\.5:14bzero\-shot65\.0440\.550\.8722porqwen2\.5:14brandomk=5k=565\.4442\.650\.8791porqwen2\.5:14blexicalk=5k=565\.6943\.070\.8807porqwen2\.5:14bembeddingk=5k=565\.0841\.530\.8796ronllama3\.2:3bzero\-shot51\.1224\.770\.8182ronllama3\.2:3brandomk=5k=550\.0024\.090\.8119ronllama3\.2:3blexicalk=5k=550\.2724\.060\.8127ronllama3\.2:3bembeddingk=5k=550\.2023\.990\.8168ronmistral:latestzero\-shot48\.2815\.610\.7408ronmistral:latestrandomk=5k=552\.2425\.210\.8401ronmistral:latestlexicalk=5k=553\.2527\.400\.8423ronmistral:latestembeddingk=5k=553\.1326\.680\.8456ronqwen2\.5:14bzero\-shot49\.0818\.460\.7632ronqwen2\.5:14brandomk=5k=550\.0020\.840\.7877ronqwen2\.5:14blexicalk=5k=550\.8822\.410\.7925ronqwen2\.5:14bembeddingk=5k=550\.9922\.500\.7997TableLABEL:tab:romance\-mt\-fullreports the dedicated MT baselines for the same Romance targets\. The MT systems remain strong, with the chrF\+\+\-best systems being OPUS\-MT for French and Romanian and NLLB\-1\.3B for Spanish, Italian, and Portuguese\. The local LLMs are closest to MT on Spanish and Portuguese, while Romanian remains substantially harder\.

Table 2:Romance dedicated MT baseline outcomes\. Each metric cell reports chrF\+\+/BLEU/COMET\.TargetOPUS\-MTNLLB\-600MNLLB\-1\.3BBest chrF\+\+fra67\.95/47\.81/0\.861665\.24/45\.08/0\.858667\.06/47\.48/0\.8710OPUS\-MTspa52\.83/26\.68/0\.849951\.91/26\.08/0\.849252\.97/27\.30/0\.8569NLLB\-1\.3Bita54\.75/27\.53/0\.852153\.55/26\.87/0\.859855\.40/29\.34/0\.8740NLLB\-1\.3Bpor37\.04/7\.37/0\.746865\.92/44\.92/0\.878066\.87/46\.33/0\.8873NLLB\-1\.3Bron58\.18/32\.90/0\.868656\.39/31\.90/0\.873358\.06/33\.80/0\.8908OPUS\-MTFamily\-scope Romance results are shown at the target level in TableLABEL:tab:romance\-family\-target\-fulland as compliance\-aware family averages in TableLABEL:tab:romance\-family\-full\. In this setting the model must produce all five Romance translations in one structured response\.qwen2\.5:14bgives the best average quality, whilemistral:latestandqwen2\.5:14bboth maintain high coverage\.llama3\.2:3bis less reliable: even though its zero\-shot family coverage is above 0\.91, coverage drops under few\-shot family prompting, showing that additional context can make structured multi\-target output harder for the smaller model\.

Table 3:Romance family\-scope target\-level quality for outputs produced when all Romance targets are requested together in one JSON response\.TargetModelConditionchrF\+\+BLEUCOMETfrallama3\.2:3bzero\-shot48\.3126\.010\.8075frallama3\.2:3brandomk=5k=552\.6730\.410\.8341frallama3\.2:3blexicalk=5k=553\.5030\.850\.8331frallama3\.2:3bembeddingk=5k=553\.4430\.770\.8374framistral:latestzero\-shot60\.3037\.150\.8486framistral:latestrandomk=5k=561\.9338\.790\.8584framistral:latestlexicalk=5k=561\.8438\.800\.8574framistral:latestembeddingk=5k=561\.9038\.830\.8573fraqwen2\.5:14bzero\-shot64\.1242\.470\.8730fraqwen2\.5:14brandomk=5k=564\.5442\.940\.8792fraqwen2\.5:14blexicalk=5k=564\.6643\.000\.8787fraqwen2\.5:14bembeddingk=5k=564\.7143\.040\.8789spallama3\.2:3bzero\-shot39\.7815\.400\.8175spallama3\.2:3brandomk=5k=543\.2518\.090\.8368spallama3\.2:3blexicalk=5k=543\.9218\.370\.8369spallama3\.2:3bembeddingk=5k=544\.0218\.680\.8358spamistral:latestzero\-shot49\.0421\.990\.8395spamistral:latestrandomk=5k=550\.4823\.470\.8506spamistral:latestlexicalk=5k=550\.3723\.190\.8493spamistral:latestembeddingk=5k=550\.5923\.800\.8493spaqwen2\.5:14bzero\-shot51\.8725\.120\.8636spaqwen2\.5:14brandomk=5k=552\.1925\.510\.8666spaqwen2\.5:14blexicalk=5k=552\.1425\.500\.8663spaqwen2\.5:14bembeddingk=5k=552\.0425\.440\.8661itallama3\.2:3bzero\-shot40\.2115\.610\.8259itallama3\.2:3brandomk=5k=543\.7118\.490\.8464itallama3\.2:3blexicalk=5k=544\.4318\.830\.8475itallama3\.2:3bembeddingk=5k=544\.4018\.680\.8481itamistral:latestzero\-shot49\.6321\.830\.8528itamistral:latestrandomk=5k=551\.0923\.360\.8637itamistral:latestlexicalk=5k=551\.0623\.170\.8637itamistral:latestembeddingk=5k=551\.2523\.570\.8632itaqwen2\.5:14bzero\-shot52\.6125\.160\.8763itaqwen2\.5:14brandomk=5k=552\.5724\.810\.8796itaqwen2\.5:14blexicalk=5k=552\.5925\.090\.8781itaqwen2\.5:14bembeddingk=5k=552\.9025\.290\.8802porllama3\.2:3bzero\-shot48\.5225\.300\.8459porllama3\.2:3brandomk=5k=552\.7129\.730\.8686porllama3\.2:3blexicalk=5k=553\.7930\.430\.8682porllama3\.2:3bembeddingk=5k=553\.3629\.730\.8667pormistral:latestzero\-shot58\.9934\.150\.8632pormistral:latestrandomk=5k=559\.8035\.060\.8717pormistral:latestlexicalk=5k=559\.8235\.030\.8714pormistral:latestembeddingk=5k=560\.0935\.270\.8721porqwen2\.5:14bzero\-shot64\.2241\.350\.8927porqwen2\.5:14brandomk=5k=564\.3841\.540\.8945porqwen2\.5:14blexicalk=5k=564\.3141\.420\.8946porqwen2\.5:14bembeddingk=5k=564\.6241\.800\.8952ronllama3\.2:3bzero\-shot38\.7415\.320\.7573ronllama3\.2:3brandomk=5k=542\.7618\.160\.8018ronllama3\.2:3blexicalk=5k=544\.1919\.150\.8025ronllama3\.2:3bembeddingk=5k=543\.9419\.020\.8051ronmistral:latestzero\-shot50\.5124\.290\.8214ronmistral:latestrandomk=5k=551\.8325\.730\.8350ronmistral:latestlexicalk=5k=552\.0725\.890\.8397ronmistral:latestembeddingk=5k=551\.9825\.960\.8415ronqwen2\.5:14bzero\-shot51\.4225\.290\.8207ronqwen2\.5:14brandomk=5k=551\.7525\.800\.8328ronqwen2\.5:14blexicalk=5k=551\.9925\.930\.8340ronqwen2\.5:14bembeddingk=5k=551\.9726\.000\.8375Table 4:Romance family\-scope JSON outcomes\. Quality metrics are averaged across Romance target languages\. Complete items are out of 1012 source sentences; missing targets are summed across French, Spanish, Italian, Portuguese, and Romanian\.ModelConditionchrF\+\+BLEUCOMETQuality on produced translationsllama3\.2:3bzero\-shot43\.1119\.530\.8108llama3\.2:3brandomk=5k=547\.0222\.980\.8375llama3\.2:3blexicalk=5k=547\.9723\.520\.8376llama3\.2:3bembeddingk=5k=547\.8323\.370\.8386mistral:latestzero\-shot53\.6927\.880\.8451mistral:latestrandomk=5k=555\.0229\.280\.8559mistral:latestlexicalk=5k=555\.0329\.210\.8563mistral:latestembeddingk=5k=555\.1629\.490\.8567qwen2\.5:14bzero\-shot56\.8531\.880\.8652qwen2\.5:14brandomk=5k=557\.0932\.120\.8705qwen2\.5:14blexicalk=5k=557\.1432\.190\.8703qwen2\.5:14bembeddingk=5k=557\.2532\.310\.8716ModelConditionCoverageCompleteMissingStructured\-output compliancellama3\.2:3bzero\-shot0\.914925/1012434llama3\.2:3brandomk=5k=50\.863873/1012695llama3\.2:3blexicalk=5k=50\.876887/1012625llama3\.2:3bembeddingk=5k=50\.870880/1012660mistral:latestzero\-shot0\.9941006/101230mistral:latestrandomk=5k=50\.9931005/101235mistral:latestlexicalk=5k=50\.9911003/101244mistral:latestembeddingk=5k=50\.9941006/101230qwen2\.5:14bzero\-shot0\.985997/101275qwen2\.5:14brandomk=5k=50\.986998/101270qwen2\.5:14blexicalk=5k=50\.986997/101271qwen2\.5:14bembeddingk=5k=50\.986996/101272
### 5\.2Germanic languages

TableLABEL:tab:germanic\-single\-fullgives the full single\-target Germanic results\. The same model\-dependent few\-shot pattern appears:mistral:latestandqwen2\.5:14bimprove with retrieved demonstrations, whilellama3\.2:3bgenerally degrades\. Germanic targets show larger gaps to dedicated MT systems than Romance targets, particularly for Danish and Swedish\.

Table 5:Germanic single\-target local LLM outcomes on FLORES devtest\. Each row reports chrF\+\+/BLEU/COMET for one target, model, and prompt condition\. Few\-shot conditions usek=5k=5examples from FLORES dev\.TargetModelConditionchrF\+\+BLEUCOMETdeullama3\.2:3bzero\-shot54\.3226\.580\.8299deullama3\.2:3brandomk=5k=553\.9226\.130\.8292deullama3\.2:3blexicalk=5k=553\.4725\.860\.8274deullama3\.2:3bembeddingk=5k=553\.5225\.930\.8267deumistral:latestzero\-shot50\.2815\.710\.7414deumistral:latestrandomk=5k=553\.8224\.240\.8299deumistral:latestlexicalk=5k=553\.7724\.390\.8309deumistral:latestembeddingk=5k=553\.7024\.160\.8336deuqwen2\.5:14bzero\-shot54\.9924\.580\.8195deuqwen2\.5:14brandomk=5k=555\.4926\.800\.8403deuqwen2\.5:14blexicalk=5k=555\.4827\.240\.8410deuqwen2\.5:14bembeddingk=5k=555\.9527\.950\.8422nldllama3\.2:3bzero\-shot48\.4619\.230\.8285nldllama3\.2:3brandomk=5k=548\.3619\.390\.8256nldllama3\.2:3blexicalk=5k=548\.1718\.930\.8266nldllama3\.2:3bembeddingk=5k=548\.1919\.180\.8278nldmistral:latestzero\-shot45\.3111\.130\.7446nldmistral:latestrandomk=5k=548\.6218\.330\.8350nldmistral:latestlexicalk=5k=548\.8818\.580\.8362nldmistral:latestembeddingk=5k=549\.0819\.110\.8406nldqwen2\.5:14bzero\-shot48\.4116\.410\.8145nldqwen2\.5:14brandomk=5k=549\.2718\.980\.8377nldqwen2\.5:14blexicalk=5k=549\.0818\.510\.8343nldqwen2\.5:14bembeddingk=5k=549\.3518\.940\.8416danllama3\.2:3bzero\-shot52\.1625\.620\.7889danllama3\.2:3brandomk=5k=551\.6025\.120\.7890danllama3\.2:3blexicalk=5k=551\.7725\.450\.7909danllama3\.2:3bembeddingk=5k=551\.7425\.460\.7969danmistral:latestzero\-shot49\.3015\.050\.7093danmistral:latestrandomk=5k=555\.6928\.980\.8478danmistral:latestlexicalk=5k=555\.8729\.210\.8520danmistral:latestembeddingk=5k=556\.2230\.050\.8572danqwen2\.5:14bzero\-shot49\.5317\.710\.7323danqwen2\.5:14brandomk=5k=550\.4619\.710\.7656danqwen2\.5:14blexicalk=5k=550\.7820\.020\.7668danqwen2\.5:14bembeddingk=5k=551\.3020\.890\.7739swellama3\.2:3bzero\-shot56\.0630\.250\.8369swellama3\.2:3brandomk=5k=555\.2829\.290\.8351swellama3\.2:3blexicalk=5k=555\.2629\.650\.8358swellama3\.2:3bembeddingk=5k=555\.2629\.790\.8399swemistral:latestzero\-shot50\.8016\.510\.7495swemistral:latestrandomk=5k=556\.1828\.330\.8555swemistral:latestlexicalk=5k=556\.2328\.580\.8599swemistral:latestembeddingk=5k=556\.5929\.090\.8625sweqwen2\.5:14bzero\-shot50\.2017\.850\.7625sweqwen2\.5:14brandomk=5k=552\.1821\.700\.7947sweqwen2\.5:14blexicalk=5k=552\.4722\.130\.7977sweqwen2\.5:14bembeddingk=5k=553\.3423\.490\.8077TableLABEL:tab:germanic\-mt\-fullreports the Germanic MT baselines\. OPUS\-MT is the strongest baseline by chrF\+\+ and BLEU for all four Germanic targets in this experiment, with especially large margins for Danish and Swedish\. These results show that strong dedicated MT models remain difficult to match with small local instruction\-tuned LLMs\.

Table 6:Germanic dedicated MT baseline outcomes\. Each metric cell reports chrF\+\+/BLEU/COMET\.TargetOPUS\-MTNLLB\-600MNLLB\-1\.3BBest chrF\+\+deu61\.11/36\.04/0\.846757\.38/32\.27/0\.842159\.32/35\.23/0\.8615OPUS\-MTnld53\.80/25\.01/0\.849952\.18/24\.42/0\.850953\.39/25\.54/0\.8656OPUS\-MTdan66\.25/43\.37/0\.882261\.43/38\.16/0\.878062\.89/40\.36/0\.8935OPUS\-MTswe65\.99/43\.70/0\.889660\.64/37\.01/0\.875362\.53/39\.81/0\.8935OPUS\-MTTablesLABEL:tab:germanic\-family\-target\-fullandLABEL:tab:germanic\-family\-fullshow the Germanic family\-scope results\. This is the clearest compliance stress test in the paper\.mistral:latestandqwen2\.5:14bproduce mostly complete JSON outputs, with embedding coverage of 0\.980 and 0\.972 respectively\.llama3\.2:3b, however, fails the family\-scope format frequently: its Germanic zero\-shot coverage is only 0\.343, and even embedding retrieval reaches only 0\.493\. For this model, the family\-scope result is therefore primarily a structured\-output failure, not only a translation\-quality failure\.

Table 7:Germanic family\-scope target\-level quality\. Each row reports chrF\+\+/BLEU/COMET for one target, model, and prompt condition when all Germanic targets are requested together in one JSON response\. Compliance for these same runs is summarized in TableLABEL:tab:germanic\-family\-full\.TargetModelConditionchrF\+\+BLEUCOMETdeullama3\.2:3bzero\-shot15\.451\.050\.8058deullama3\.2:3brandomk=5k=526\.847\.260\.8278deullama3\.2:3blexicalk=5k=523\.415\.250\.8224deullama3\.2:3bembeddingk=5k=526\.967\.390\.8316deumistral:latestzero\-shot51\.6724\.230\.8277deumistral:latestrandomk=5k=553\.2926\.080\.8414deumistral:latestlexicalk=5k=553\.0525\.920\.8420deumistral:latestembeddingk=5k=553\.6726\.320\.8439deuqwen2\.5:14bzero\-shot55\.2729\.040\.8608deuqwen2\.5:14brandomk=5k=555\.7329\.350\.8646deuqwen2\.5:14blexicalk=5k=555\.7029\.080\.8655deuqwen2\.5:14bembeddingk=5k=556\.1529\.820\.8652nldllama3\.2:3bzero\-shot13\.810\.780\.7943nldllama3\.2:3brandomk=5k=523\.725\.200\.8266nldllama3\.2:3blexicalk=5k=520\.903\.770\.8214nldllama3\.2:3bembeddingk=5k=523\.995\.350\.8274nldmistral:latestzero\-shot47\.0118\.200\.8309nldmistral:latestrandomk=5k=548\.0418\.970\.8428nldmistral:latestlexicalk=5k=548\.2019\.280\.8438nldmistral:latestembeddingk=5k=548\.3619\.260\.8429nldqwen2\.5:14bzero\-shot48\.8720\.610\.8558nldqwen2\.5:14brandomk=5k=548\.9720\.360\.8599nldqwen2\.5:14blexicalk=5k=549\.1620\.410\.8625nldqwen2\.5:14bembeddingk=5k=549\.2920\.380\.8621danllama3\.2:3bzero\-shot13\.520\.760\.7305danllama3\.2:3brandomk=5k=524\.806\.050\.7737danllama3\.2:3blexicalk=5k=521\.554\.440\.7707danllama3\.2:3bembeddingk=5k=524\.676\.230\.7740danmistral:latestzero\-shot52\.0125\.300\.8366danmistral:latestrandomk=5k=553\.2026\.370\.8528danmistral:latestlexicalk=5k=553\.2726\.950\.8521danmistral:latestembeddingk=5k=553\.7827\.410\.8537danqwen2\.5:14bzero\-shot52\.1326\.420\.8250danqwen2\.5:14brandomk=5k=552\.6226\.820\.8359danqwen2\.5:14blexicalk=5k=553\.1727\.230\.8374danqwen2\.5:14bembeddingk=5k=553\.5427\.740\.8359swellama3\.2:3bzero\-shot15\.581\.120\.8188swellama3\.2:3brandomk=5k=526\.737\.710\.8425swellama3\.2:3blexicalk=5k=523\.305\.530\.8353swellama3\.2:3bembeddingk=5k=526\.457\.600\.8396swemistral:latestzero\-shot53\.4326\.940\.8586swemistral:latestrandomk=5k=554\.3428\.300\.8691swemistral:latestlexicalk=5k=554\.0627\.980\.8676swemistral:latestembeddingk=5k=554\.8028\.690\.8688sweqwen2\.5:14bzero\-shot52\.9226\.830\.8429sweqwen2\.5:14brandomk=5k=553\.0626\.760\.8500sweqwen2\.5:14blexicalk=5k=553\.1726\.750\.8475sweqwen2\.5:14bembeddingk=5k=553\.5427\.170\.8462Table 8:Germanic family\-scope JSON outcomes\. Quality metrics are averaged across Germanic target languages\. Complete items are out of 1012 source sentences; missing targets are summed across German, Dutch, Danish, and Swedish\.ModelConditionchrF\+\+BLEUCOMETQuality on produced translationsllama3\.2:3bzero\-shot14\.590\.930\.7873llama3\.2:3brandomk=5k=525\.536\.550\.8177llama3\.2:3blexicalk=5k=522\.294\.750\.8124llama3\.2:3bembeddingk=5k=525\.526\.640\.8181mistral:latestzero\-shot51\.0323\.670\.8385mistral:latestrandomk=5k=552\.2224\.930\.8515mistral:latestlexicalk=5k=552\.1425\.030\.8514mistral:latestembeddingk=5k=552\.6525\.420\.8523qwen2\.5:14bzero\-shot52\.3025\.730\.8461qwen2\.5:14brandomk=5k=552\.5925\.820\.8526qwen2\.5:14blexicalk=5k=552\.8025\.870\.8532qwen2\.5:14bembeddingk=5k=553\.1326\.280\.8524ModelConditionCoverageCompleteMissingStructured\-output compliancellama3\.2:3bzero\-shot0\.343347/10122660llama3\.2:3brandomk=5k=50\.487493/10122076llama3\.2:3blexicalk=5k=50\.437442/10122278llama3\.2:3bembeddingk=5k=50\.493499/10122052mistral:latestzero\-shot0\.987999/101252mistral:latestrandomk=5k=50\.975987/1012100mistral:latestlexicalk=5k=50\.975987/1012100mistral:latestembeddingk=5k=50\.980992/101280qwen2\.5:14bzero\-shot0\.960972/1012160qwen2\.5:14brandomk=5k=50\.965977/1012140qwen2\.5:14blexicalk=5k=50\.968980/1012128qwen2\.5:14bembeddingk=5k=50\.972984/1012112
### 5\.3Few\-shot retrieval effects

TableLABEL:tab:retrieval\-deltas\-allaggregates few\-shot deltas over all nine target languages\. The main finding is that retrieval is model\-dependent\.llama3\.2:3bloses chrF\+\+ under all retrieval settings\.mistral:latestgains strongly under all retrieval settings, with embedding retrieval giving the largest average gain \(\+4\.60 chrF\+\+, \+11\.03 BLEU, \+0\.0941 COMET\)\.qwen2\.5:14balso improves across all targets, with embedding retrieval again strongest on average \(\+1\.27 chrF\+\+, \+3\.17 BLEU, \+0\.0264 COMET\)\.

Table 9:Mean few\-shot deltas over all nine target languages relative to zero\-shot\. Each cell reportsΔ\\DeltachrF\+\+/Δ\\DeltaBLEU/Δ\\DeltaCOMET\.Modelrandomlexicalembeddingllama3\.2:3b\-0\.58/\-0\.59/\-0\.0014\-0\.60/\-0\.58/\-0\.0012\-0\.56/\-0\.48/\+0\.0008mistral:latest\+4\.25/\+10\.30/\+0\.0895\+4\.44/\+10\.78/\+0\.0914\+4\.60/\+11\.03/\+0\.0941qwen2\.5:14b\+0\.87/\+2\.59/\+0\.0210\+0\.99/\+2\.81/\+0\.0215\+1\.27/\+3\.17/\+0\.0264Embedding retrieval is therefore the best average retrieval method for the two models that benefit from demonstrations\. However, the advantage over lexical and random retrieval is smaller than the difference between models\. This indicates that the ability to use demonstrations is the dominant factor; semantic retrieval helps, but it does not rescue a model that is already brittle under longer prompts\.

### 5\.4Comparison with dedicated MT systems

TableLABEL:tab:best\-llm\-vs\-mt\-significancegives paired significance tests for the best local LLM condition against the chrF\+\+\-best MT baseline per target\. The best local LLM remains below this MT baseline on chrF\+\+ for all targets\. The average chrF\+\+ gap is smaller for Romance languages \(\-2\.55\) than for Germanic languages \(\-7\.26\)\. Spanish is the closest target: the chrF\+\+ difference betweenqwen2\.5:14bwith embedding retrieval and NLLB\-1\.3B is not significant \(p=0\.520p=0\.520\)\. COMET narrows some differences: French slightly favors the local LLM over the chrF\+\+\-best MT baseline in mean COMET but not significantly, and German shows a non\-significant COMET difference despite a significant chrF\+\+ gap\.

Table 10:Paired comparison of the best local LLM condition and the chrF\+\+\-best dedicated MT baseline per target\. Deltas are local LLM minus MT baseline\.TargetBest local LLMchrF\-bestΔ\\DeltachrF95% CIpchrFp\_\{\\mathrm\{chrF\}\}Δ\\DeltaCOMETpCOMETp\_\{\\mathrm\{COMET\}\}fraqwen2\.5:14brandomOPUS\-MT\-3\.63\[\-4\.53, \-2\.97\]0\.000\+0\.00330\.200spaqwen2\.5:14bembeddingNLLB\-1\.3B\-0\.38\[\-0\.69, 0\.40\]0\.520\-0\.00490\.072itaqwen2\.5:14bembeddingNLLB\-1\.3B\-2\.62\[\-3\.03, \-1\.76\]0\.000\-0\.00940\.000porqwen2\.5:14blexicalNLLB\-1\.3B\-1\.18\[\-1\.66, \-0\.07\]0\.020\-0\.00660\.008ronmistral:latestlexicalOPUS\-MT\-4\.93\[\-5\.46, \-4\.02\]0\.000\-0\.02630\.000deuqwen2\.5:14bembeddingOPUS\-MT\-5\.16\[\-5\.72, \-4\.22\]0\.000\-0\.00460\.236nldqwen2\.5:14bembeddingOPUS\-MT\-4\.45\[\-4\.71, \-3\.52\]0\.000\-0\.00830\.004danmistral:latestembeddingOPUS\-MT\-10\.03\[\-10\.77, \-9\.18\]0\.000\-0\.02500\.000swemistral:latestembeddingOPUS\-MT\-9\.40\[\-9\.75, \-8\.21\]0\.000\-0\.02710\.000
### 5\.5Sentence\-level retrieval analysis

TableLABEL:tab:sentence\-retrieval\-fullreports sentence\-level retrieval diagnostics\. The correlation between the demonstration similarity score and the sentence\-level chrF gain is close to zero in almost all settings\. This is important: even when embedding retrieval is the best average condition, higher embedding similarity does not strongly predict that a specific sentence will improve\. Retrieval gain is therefore not simply a monotonic function of the similarity score used for selecting demonstrations\.

Table 11:Sentence\-level retrieval diagnostics\. Each cell reports meanΔ\\DeltachrF/meanΔ\\DeltaBLEU/positive\-rate/correlation between demonstration similarity score and sentence\-level chrF gain\.RomanceModelrandomlexicalembeddingllama3\.2:3b\-0\.69/\-0\.88/0\.42/\-0\.000\-0\.62/\-0\.78/0\.43/\-0\.003\-0\.64/\-0\.81/0\.43/\+0\.015mistral:latest\+3\.70/\+7\.45/0\.67/\+0\.019\+3\.80/\+7\.53/0\.68/\+0\.005\+3\.97/\+7\.73/0\.68/\+0\.025qwen2\.5:14b\+0\.44/\+0\.65/0\.42/\-0\.023\+0\.60/\+0\.89/0\.44/\-0\.006\+0\.67/\+0\.93/0\.46/\+0\.002GermanicModelrandomlexicalembeddingllama3\.2:3b\-0\.50/\-0\.44/0\.44/\+0\.004\-0\.64/\-0\.43/0\.45/\-0\.019\-0\.62/\-0\.45/0\.45/\-0\.009mistral:latest\+4\.63/\+8\.29/0\.70/\+0\.025\+4\.76/\+8\.36/0\.72/\+0\.004\+4\.86/\+8\.74/0\.72/\+0\.019qwen2\.5:14b\+0\.88/\+1\.16/0\.47/\+0\.001\+0\.99/\+1\.19/0\.49/\-0\.022\+1\.37/\+1\.65/0\.52/\+0\.023Length effects are more visible but still model\-specific\. Formistral:latest, retrieval gains are larger on longer sentences in both families\. With embedding retrieval, Romance gains increase from \+2\.53 chrF on the shortest length quartile to \+4\.92 on the longest, and Germanic gains increase from \+3\.30 to \+5\.36\. Forqwen2\.5:14b, the length pattern is weaker and sometimes reversed\. The sentence\-level analysis therefore supports a cautious interpretation: retrieved examples help when the model can use them, but neither lexical nor embedding similarity score is sufficient by itself to predict sentence\-level gains\.

### 5\.6Summary of findings

The complete outcomes support five findings\. First, dedicated MT systems remain the strongest local translation systems on FLORES devtest, especially for Germanic languages\. Second, local LLMs are competitive for some Romance targets, particularly Spanish and Portuguese, but they do not consistently match MT baselines\. Third, few\-shot prompting is beneficial formistral:latestandqwen2\.5:14b, while it hurtsllama3\.2:3b; this makes model capacity and instruction\-following ability central to few\-shot MT prompting\. Fourth, embedding retrieval is the best average retrieval method for the models that benefit from examples, but its advantage over lexical and random examples is modest\. Fifth, family\-scope prompting should be reported as both translation and compliance: larger local LLMs can produce mostly complete structured multi\-target outputs, whereas smaller models can omit required languages, especially in the Germanic family setting\.

These results motivate treating prompt scope as an experimental variable in LLM\-based MT evaluation\. Single\-target prompting measures translation quality most cleanly; family\-scope prompting tests whether a local LLM can function as a structured multi\-target translation engine; and retrieval experiments reveal whether demonstrations help because of semantic relevance or because a given model is generally capable of using in\-context examples\.

## 6Discussion and Limitations

The results show that prompt scope is not a superficial formatting choice in LLM\-based machine translation\. A single\-target prompt and a family\-scope prompt ask for different behaviours: the former tests translation into one language, while the latter also tests whether the model can coordinate several target languages and obey a structured output schema\. This distinction matters in practice\. Family\-scope prompting is attractive because it can produce several translations in one generation call, but it also introduces failure modes that standard single\-target MT evaluation would miss, including omitted languages, incomplete JSON objects, and cross\-target interference\. The Germanic family results forllama3\.2:3bmake this particularly clear: the model can produce reasonable single\-target translations, but frequently fails to return all required languages when asked for the whole family at once\.

The few\-shot results also caution against a simple “more similar examples are better” interpretation\. Embedding retrieval gives the best average gains formistral:latestandqwen2\.5:14b, but the sentence\-level correlations between retrieval similarity score and translation improvement are close to zero\. This suggests that demonstrations help through several mechanisms at once: they can clarify the task format, anchor the output style, expose target\-language conventions, and occasionally provide useful semantic parallels\. For smaller models, however, the same additional context can become a burden rather than a benefit\. Random demonstrations are therefore not merely a weak baseline; they are a necessary control for separating the effect of example relevance from the more general effect of showing the model the expected translation pattern\.

The comparison with dedicated MT systems provides an important boundary for the claims\. Local LLMs are useful and competitive in selected settings, especially for some Romance targets, but the strongest dedicated MT baselines remain better overall in chrF\+\+ and BLEU, with the largest gaps on Germanic languages\. COMET narrows some differences and occasionally gives a less severe view of the LLM outputs, but it does not overturn the main conclusion\. The contribution of this study is therefore not that small local LLMs replace dedicated MT systems\. Rather, it is that prompt scope, demonstration selection, and structured\-output compliance substantially affect how local LLM translation should be evaluated and reported\.

Several limitations remain\. The study uses English as the only source language and covers two European language families\. The patterns may differ for non\-English sources, lower\-resource languages, non\-Latin scripts, morphologically richer languages, or domain\-specific corpora\. We also evaluate a fixed set of local model checkpoints through one inference stack and with temperature set to 0; other decoding settings, context lengths, quantizations, or instruction templates may change both translation quality and output compliance\. The evaluation is automatic: BLEU, chrF\+\+, and COMET provide complementary signals, but they do not replace human adequacy and fluency assessment\. In the family\-scope setting, quality scores are computed over produced translations, so they must always be interpreted together with coverage and missing\-output counts\. A model that produces fluent translations for only part of the requested family is not equivalent to a model that reliably produces every target\.

The practical implication is that local LLM translation should be deployed with validation rather than assumed reliability\. Local inference is attractive for privacy\-sensitive settings because source text need not be sent to a remote API, but local execution does not by itself guarantee correctness\. Missing outputs, wrong\-language responses, prompt echoes, and fluent mistranslations can create real risks in legal, medical, administrative, or safety\-critical contexts\. For multi\-target translation, structured\-output checks should be treated as part of the system, not as optional post\-processing\. Users should preserve source text, model metadata, prompts, and validation logs, and should avoid relying on automatic translations as the sole basis for high\-stakes decisions\.

## 7Conclusion

This paper evaluated local LLM machine translation under two prompt scopes and three few\-shot demonstration strategies\. On FLORES English\-to\-Romance and English\-to\-Germanic translation, dedicated MT systems remain the strongest overall baselines, while local LLMs are competitive for selected Romance targets\. Few\-shot prompting helps stronger local LLMs but hurts the smallest model, and embedding\-based retrieval is best on average without being universally predictive at the sentence level\. Family\-scope prompting exposes a different capability: the ability to produce complete structured multi\-target translations\. Larger local LLMs mostly satisfy this requirement, while smaller models can fail by omitting languages\. These findings support evaluating LLM translation as a combination of translation quality, prompt scope, demonstration selection, and output compliance\.

## References

- \[1\]M\. Arcan, D\. Torregrosa, S\. Ahmadi, and J\. P\. McCrae\(2019\)Inferring translation candidates for multilingual dictionary generation with multi\-way neural machine translation\.InProceedings of the Translation Inference Across Dictionaries 2019 Shared Task,CEUR Workshop Proceedings, Vol\.2493,pp\. 13–23\.External Links:[Link](https://ceur-ws.org/Vol-2493/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px4.p1.1)\.
- \[2\]M\. Arcan, D\. Torregrosa, and P\. Buitelaar\(2017\)Translating terminological expressions in knowledge bases with neural machine translation\.arXiv preprint arXiv:1709\.02184\.External Links:[Link](https://arxiv.org/abs/1709.02184)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px4.p1.1)\.
- \[3\]T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in Neural Information Processing Systems33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px3.p1.1)\.
- \[4\]B\. R\. Chakravarthi, M\. Arcan, and J\. P\. McCrae\(2019\)WordNet gloss translation for under\-resourced languages using multilingual neural machine translation\.InProceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation,Dublin, Ireland,pp\. 1–7\.External Links:[Link](https://aclanthology.org/W19-7101/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px4.p1.1)\.
- \[5\]A\. Hatami, M\. Arcan, and P\. Buitelaar\(2024\)Enhancing translation quality by leveraging semantic diversity in multimodal machine translation\.InProceedings of the 16th Conference of the Association for Machine Translation in the Americas \(Volume 1: Research Track\),Chicago, USA,pp\. 154–166\.External Links:[Link](https://aclanthology.org/2024.amta-research.14/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px5.p1.1)\.
- \[6\]A\. Hendy, M\. Abdelrehim, A\. Sharaf, V\. Raunak, M\. Gabr, H\. Matsushita, Y\. J\. Kim, M\. Afify, and H\. H\. Awadalla\(2023\)How good are GPT models at machine translation? a comprehensive evaluation\.InarXiv preprint arXiv:2302\.09210,External Links:[Link](https://arxiv.org/abs/2302.09210)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]M\. Junczys\-Dowmunt, R\. Grundkiewicz, T\. Dwojak, H\. Hoang, K\. Heafield, T\. Neckermann, F\. Seide, U\. Germann, A\. F\. Aji, N\. Bogoychev, A\. F\. T\. Martins, and A\. Birch\(2018\)Marian: fast neural machine translation in C\+\+\.InProceedings of ACL 2018, System Demonstrations,pp\. 116–121\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-4020),[Link](https://aclanthology.org/P18-4020/)Cited by:[§4\.2](https://arxiv.org/html/2607.26286#S4.SS2.p1.1)\.
- \[8\]J\. Liu, D\. Shen, Y\. Zhang, B\. Dolan, L\. Carin, and W\. Chen\(2022\)What makes good in\-context examples for GPT\-3?\.InProceedings of Deep Learning Inside Out,pp\. 100–114\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.deelio-1.10),[Link](https://aclanthology.org/2022.deelio-1.10/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px3.p1.1)\.
- \[9\]J\. P\. McCrae, M\. Arcan, and P\. Buitelaar\(2017\)Linking knowledge graphs across languages with semantic similarity and machine translation\.InProceedings of the First Workshop on Multi\-Language Processing in a Globalising World,External Links:[Link](https://john.mccr.ae/publications)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px4.p1.1)\.
- \[10\]NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood,et al\.\(2022\)No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.External Links:[Link](https://arxiv.org/abs/2207.04672)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2607.26286#S4.SS2.p1.1)\.
- \[11\]M\. Popovic\(2015\)chrF: character n\-gram F\-score for automatic MT evaluation\.InProceedings of the Tenth Workshop on Statistical Machine Translation,Lisbon, Portugal,pp\. 392–395\.External Links:[Document](https://dx.doi.org/10.18653/v1/W15-3049),[Link](https://aclanthology.org/W15-3049/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]M\. Popovic\(2017\)chrF\+\+: words helping character n\-grams\.InProceedings of the Second Conference on Machine Translation,Copenhagen, Denmark,pp\. 612–618\.External Links:[Document](https://dx.doi.org/10.18653/v1/W17-4770),[Link](https://aclanthology.org/W17-4770/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]M\. Post\(2018\)A call for clarity in reporting BLEU scores\.InProceedings of the Third Conference on Machine Translation: Research Papers,Brussels, Belgium,pp\. 186–191\.External Links:[Document](https://dx.doi.org/10.18653/v1/W18-6319),[Link](https://aclanthology.org/W18-6319/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]R\. Rei, J\. G\. C\. de Souza, D\. Alves, C\. Zerva, A\. C\. Farinha, T\. Glushkova, A\. Lavie, and L\. Coheur\(2022\)COMET\-22: unbabel\-IST 2022 submission for the metrics shared task\.InProceedings of the Seventh Conference on Machine Translation,pp\. 578–585\.External Links:[Link](https://aclanthology.org/2022.wmt-1.52/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]R\. Rei, C\. Stewart, A\. C\. Farinha, and A\. Lavie\(2020\)COMET: a neural framework for MT evaluation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 2685–2702\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.213),[Link](https://aclanthology.org/2020.emnlp-main.213/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]J\. Tiedemann and S\. Thottingal\(2020\)The OPUS\-MT project: building open translation services for the world\.InProceedings of the 22nd Annual Conference of the European Association for Machine Translation,pp\. 479–480\.External Links:[Link](https://aclanthology.org/2020.eamt-1.61/)Cited by:[§4\.2](https://arxiv.org/html/2607.26286#S4.SS2.p1.1)\.
- \[17\]D\. Torregrosa, N\. Pasricha, M\. Masoud, B\. R\. Chakravarthi, J\. Alonso, N\. Casas, and M\. Arcan\(2019\)Leveraging rule\-based machine translation knowledge for under\-resourced neural machine translation models\.InProceedings of Machine Translation Summit XVII: Translator, Project and User Tracks,Dublin, Ireland,pp\. 125–133\.External Links:[Link](https://aclanthology.org/W19-6725/)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px4.p1.1)\.
- \[18\]D\. Torregrosa, N\. Pasricha, M\. Masoud, B\. R\. Chakravarthi, J\. Alonso, N\. Casas, and M\. Arcan\(2020\)Aspects of terminological and named entity knowledge within rule\-based machine translation models for under\-resourced neural machine translation scenarios\.arXiv preprint arXiv:2009\.13398\.External Links:[Link](https://arxiv.org/abs/2009.13398)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px4.p1.1)\.
- \[19\]B\. Zhang, B\. Haddow, and A\. Birch\(2023\)Prompting large language model for machine translation: a case study\.arXiv preprint arXiv:2301\.07069\.External Links:[Link](https://arxiv.org/abs/2301.07069)Cited by:[§2](https://arxiv.org/html/2607.26286#S2.SS0.SSS0.Px2.p1.1)\.