Studying quantization trade-offs for efficient inference deployment in machine translation
Summary
This paper studies quantization trade-offs for deploying translation models like EuroLLM and Hy-MT2 on a single GPU, showing that combining document chunking with W4A8/W8A8 quantization improves latency-throughput trade-offs while introducing a document-level evaluation to assess long-context dynamics.
View Cached Full Text
Cached at: 08/03/26, 07:36 AM
# Studying quantization trade-offs for efficient inference deployment in machine translation
Source: [https://arxiv.org/abs/2607.29397](https://arxiv.org/abs/2607.29397)
[View PDF](https://arxiv.org/pdf/2607.29397)
> Abstract:Deploying large language models in realistic server environments poses challenges, as the system needs to provide high\-quality responses with low latency\. Quantization is a common approach to reduce the memory footprint and improve inference efficiency, yet its impact on latency and throughput is rarely evaluated under controlled, orchestration\-level workloads\. In this work we study the quantization trade\-offs of two translation model families, EuroLLM \\citep\{martins2025eurollm\} and Hy\-MT2 \\citep\{zheng2026hy\} across five models ranging from 1\.7B to 22B for efficient deployment on a single A100 or H100 GPU\. We demonstrate that combining a document\-chunking strategy with W4A8 or W8A8 quantization improves the latency\-throughput Pareto\-curve under a wide range of workloads\. Furthermore, since standard machine translation \(MT\) benchmarks rely on isolated sentences and fail to capture long\-context dynamics, we introduce a document\-level evaluation from WMT24\+\+ to assess how text chunking strategies affect translation quality under quantization\. Our results reveal that standard segment\-level evaluation can fail to predict the interaction between quantization and long\-context document translation\. While Hy\-MT2 remains robust under quantization, EuroLLM shows strong sensitivity and translation quality collapses rapidly for all considered quantization formats\. Overall, our experiments show that the trade\-off between inference efficiency and translation quality depends not only on the quantization format, but also on the choice of text chunking strategy\.
## Submission history
From: Jim Zhao \[[view email](https://arxiv.org/show-email/b19a3c83/2607.29397)\] **\[v1\]**Fri, 31 Jul 2026 13:15:11 UTC \(263 KB\)Similar Articles
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
MixQuant proposes an adaptive mixed-precision quantization framework for LLMs that handles variable memory budgets by marginalizing layer distortion over random upstream configurations, outperforming existing methods across multiple models and budgets.
Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
This paper systematically evaluates component-wise quantization of small vision-language models on Jetson edge devices, finding that model architecture (MoE vs dense) significantly affects quantization sensitivity and that quantization errors are largely additive except along modality-alignment paths.
CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs
This paper demonstrates that small open-weight LLMs (<30B parameters) can achieve competitive interpretable translation quality estimation, including MQM error annotations and corrections, rivaling much larger proprietary models while preserving data privacy.
LLM Compression with Jointly Optimizing Architectural and Quantization choices
Researchers from UiT and University of Oslo propose a differentiable NAS framework that jointly optimizes architectural configurations and mixed-precision quantization for LLM compression, achieving up to 1.4× faster inference or 6% higher accuracy across seven reasoning tasks compared to sequential NAS-then-quantization baselines.
Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
This paper presents a framework for quantizing vision-language models to 2.7 bits per parameter, enabling efficient mobile deployment by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB while preserving performance on visual QA tasks.