Last Translation Benchmark
Summary
The Last Translation Benchmark introduces a live dataset of peer-reviewed, multimodal examples designed to evaluate and break leading machine translation models, with handcrafted verification rules for reliable assessment. It addresses the saturation of current benchmarks and the unreliability of automatic metrics.
View Cached Full Text
Cached at: 09/04/26, 03:58 PM
Paper page - Last Translation Benchmark
Source: https://huggingface.co/papers/2609.04173 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
The Last Translation Benchmark introduces peer-reviewed, multimodal examples that break leading translation models alongside handcrafted verification rules for reliable, actionable evaluation.
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks formachine translationare approaching saturation. Further,automatic translation metricsare unreliable, vulnerable toreward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce theLast Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leadingmachine translationmodels. We also present a new evaluation approach: each example comes with handcraftedverification rulesdescribing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. TheLast Translation Benchmarkis a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.
View arXiv pageView PDFProject pageGitHub45Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.04173 in a model README.md to link it from this page.
Datasets citing this paper1
#### zouhar/last-translation-benchmark Viewer• Updatedabout 11 hours ago • 3.46k • 33 • 17
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.04173 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
OpenSTBench: Beyond Semantic Evaluation for Speech Translation
OpenSTBench is a unified multidimensional evaluation framework for speech translation systems that jointly assesses translation quality, speech quality, speaker preservation, emotion fidelity, and latency across both S2TT and S2ST systems in offline and streaming settings. The framework addresses the gap left by fragmented evaluation protocols and provides a reproducible benchmark for comparing heterogeneous speech translation systems.
MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts
MultiGhostBench is a multilingual benchmark for long-form LLM-generated text attribution under distribution shifts, featuring 928 books in six languages and highlighting performance degrades and no single method consistently best across settings.
Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Introduces Cultivar, a contrastive localized translation benchmark for detecting data contamination and evaluating localization robustness in multilingual translation models. It benchmarks 32 open-weight models and finds that MT-specialised models are less robust, with potential overfitting to FLORES.
PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks
Introduces PaliBench, a multi-reference benchmark for Pali-to-English translation using independent translations from multiple scholars, and a reusable methodology for creating similar benchmarks for classical languages.
Benchmarking Speech-to-Speech Translation Models
COMPASS is a unified benchmarking framework for speech-to-speech translation (S2ST) that integrates 46 metrics across eight dimensions, evaluated on 1,248 model-language configurations. It identifies complementary architecture strengths and proposes reduced metric subsets that preserve rankings while cutting evaluation time.