Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment

# Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
Source: [https://arxiv.org/html/2607.08029](https://arxiv.org/html/2607.08029)
###### Abstract

The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on\-device multimodal intelligence\. However, a detailed understanding of component\-wise quantization remains a bottleneck for optimal deployment\. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX\. By separating the vision encoder, projector, and large language model backbone yields the following results: \(1\) Quantization sensitivity is governed by the structural paradigm \(MoE vs\. dense\) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; \(2\) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere–a deployment\-specific encoder\-kernel\-hardware interaction, not a SigLIP flaw; \(3\) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; \(4\) Composite quantization errors are largely additive, except along the modality\-alignment path, which is architecture\-dependent; \(5\) The intelligence\-per\-joule profile varies significantly across platforms owing to memory bandwidth constraints\.

vision\-language models, post\-training quantization, component\-wise analysis, edge deployment, energy\-efficient inference

Table 1:Comprehensive evaluation: perception, cognition, and total MME scores with percentage variations relative tocfg0\(baseline\) across all models and configurations\.ModelConfigJetson NX OrinJetson AGX OrinPerc\. \(Δ%\\Delta\\%\)Cogn\. \(Δ%\\Delta\\%\)Total \(Δ%\\Delta\\%\)Perc\. \(Δ%\\Delta\\%\)Cogn\. \(Δ%\\Delta\\%\)Total \(Δ%\\Delta\\%\)Qwen3\-VL\-2B\-Instructcfg0 \(Base\)1508\.20513\.572021\.781485\.46497\.861983\.32cfg11524\.25 \(\+1\.06%\)553\.57 \(\+7\.79%\)2077\.82 \(\+2\.77%\)1535\.48 \(\+3\.37%\)533\.57 \(\+7\.17%\)2069\.05 \(\+4\.32%\)cfg21512\.75 \(\+0\.30%\)477\.14 \(\-7\.09%\)1989\.89 \(\-1\.58%\)1501\.77 \(\+1\.10%\)503\.57 \(\+1\.15%\)2005\.34 \(\+1\.11%\)cfg31514\.92 \(\+0\.45%\)531\.43 \(\+3\.48%\)2046\.35 \(\+1\.22%\)1538\.39 \(\+3\.56%\)526\.07 \(\+5\.67%\)2064\.46 \(\+4\.09%\)cfg41505\.65 \(\-0\.17%\)467\.86 \(\-8\.90%\)1973\.51 \(\-2\.39%\)1504\.48 \(\+1\.28%\)523\.93 \(\+5\.24%\)2028\.41 \(\+2\.27%\)cfg51511\.96 \(\+0\.25%\)517\.14 \(\+0\.69%\)2029\.10 \(\+0\.36%\)1524\.09 \(\+2\.60%\)554\.64 \(\+11\.40%\)2078\.73 \(\+4\.81%\)DeepSeek\-VL2\-Tinycfg0 \(Base\)1550\.76360\.361911\.121556\.35360\.361916\.71cfg11584\.33 \(\+2\.16%\)368\.21 \(\+2\.18%\)1952\.54 \(\+2\.17%\)1589\.25 \(\+2\.11%\)372\.86 \(\+3\.47%\)1962\.10 \(\+2\.37%\)cfg21552\.34 \(\+0\.10%\)364\.64 \(\+1\.19%\)1916\.98 \(\+0\.31%\)1548\.53 \(\-0\.50%\)376\.07 \(\+4\.36%\)1924\.60 \(\+0\.41%\)cfg31582\.56 \(\+2\.05%\)370\.36 \(\+2\.78%\)1952\.92 \(\+2\.19%\)1580\.41 \(\+1\.55%\)370\.36 \(\+2\.78%\)1950\.76 \(\+1\.78%\)cfg41545\.90 \(\-0\.31%\)351\.07 \(\-2\.58%\)1896\.98 \(\-0\.74%\)1546\.17 \(\-0\.65%\)363\.57 \(\+0\.89%\)1909\.74 \(\-0\.36%\)cfg51574\.16 \(\+1\.51%\)370\.36 \(\+2\.78%\)1944\.51 \(\+1\.75%\)1584\.90 \(\+1\.83%\)368\.21 \(\+2\.18%\)1953\.11 \(\+1\.90%\)PaliGemma2\-3Bcfg0 \(Base\)1424\.94236\.071661\.011429\.53228\.571658\.10cfg11387\.46 \(\-2\.63%\)211\.43 \(\-10\.44%\)1598\.89 \(\-3\.74%\)1389\.45 \(\-2\.80%\)209\.29 \(\-8\.44%\)1598\.74 \(\-3\.58%\)cfg21425\.10 \(\+0\.01%\)238\.57 \(\+1\.06%\)1663\.68 \(\+0\.16%\)1424\.90 \(\-0\.32%\)228\.57 \(\+0\.00%\)1653\.47 \(\-0\.28%\)cfg31390\.95 \(\-2\.39%\)209\.29 \(\-11\.34%\)1600\.24 \(\-3\.66%\)1391\.08 \(\-2\.69%\)209\.29 \(\-8\.44%\)1600\.37 \(\-3\.48%\)cfg41429\.35 \(\+0\.31%\)238\.93 \(\+1\.21%\)1668\.28 \(\+0\.44%\)1420\.90 \(\-0\.60%\)236\.07 \(\+3\.28%\)1656\.97 \(\-0\.07%\)cfg51382\.68 \(\-2\.97%\)207\.14 \(\-12\.25%\)1589\.83 \(\-4\.29%\)1379\.90 \(\-3\.47%\)209\.29 \(\-8\.44%\)1589\.19 \(\-4\.16%\)LLaVA\-OV\-0\.5Bcfg0 \(Base\)1210\.76151\.071361\.831214\.02155\.711369\.73cfg1967\.52 \(\-20\.09%\)174\.29 \(\+15\.37%\)1141\.81 \(\-16\.16%\)1029\.01 \(\-15\.24%\)184\.64 \(\+18\.58%\)1213\.65 \(\-11\.39%\)cfg21209\.34 \(\-0\.12%\)151\.07 \(\+0\.00%\)1360\.41 \(\-0\.1%\)1215\.06 \(\+0\.09%\)148\.21 \(\-4\.82%\)1363\.28 \(\-0\.47%\)cfg3965\.8 \(\-20\.23%\)171\.79 \(\+13\.72%\)1137\.59 \(\-16\.47%\)1021\.05 \(\-15\.90%\)186\.43 \(\+19\.73%\)1207\.48 \(\-11\.85%\)cfg41220\.27 \(\+0\.79%\)148\.93 \(\-1\.42%\)1369\.20 \(\+0\.54%\)1212\.08 \(\-0\.16%\)153\.21 \(\-1\.61%\)1365\.30 \(\-0\.32%\)cfg5974\.79 \(\-19\.49%\)174\.29 \(\+15\.37%\)1149\.08 \(\-15\.62%\)1026\.16 \(\-15\.47%\)189\.29 \(\+21\.57%\)1215\.45 \(\-11\.26%\)Kosmos\-2\.5cfg0 \(Base\)464\.80197\.50662\.30464\.55197\.50662\.05cfg1454\.97 \(\-2\.11%\)196\.79 \(\-0\.36%\)651\.76 \(\-1\.59%\)453\.05 \(\-2\.48%\)196\.79 \(\-0\.36%\)649\.84 \(\-1\.84%\)cfg2466\.05 \(\+0\.27%\)198\.21 \(\+0\.36%\)664\.27 \(\+0\.30%\)465\.80 \(\+0\.27%\)197\.50 \(\+0\.00%\)663\.30 \(\+0\.19%\)cfg3457\.14 \(\-1\.65%\)196\.79 \(\-0\.36%\)653\.92 \(\-1\.27%\)459\.97 \(\-0\.99%\)196\.79 \(\-0\.36%\)656\.76 \(\-0\.80%\)cfg4465\.05 \(\+0\.05%\)201\.43 \(\+1\.99%\)666\.48 \(\+0\.63%\)464\.05 \(\-0\.11%\)198\.21 \(\+0\.36%\)663\.27 \(\+0\.18%\)cfg5457\.39 \(\-1\.59%\)199\.29 \(\+0\.91%\)656\.67 \(\-0\.85%\)453\.80 \(\-2\.31%\)196\.79 \(\-0\.36%\)650\.59 \(\-1\.73%\)

- Note:Each data point reflects the average ofn≥3n\\geq 3independent runs\. Refer to Appendix[D](https://arxiv.org/html/2607.08029#A4)for the detailed bar chart representing the mean score across runs\.

Table 2:Definition of configurations for component\-wise quantization sensitivity analysis\.ConfigVisionProjectorLLMDescriptioncfg0 \(Base\)FP16FP16FP16Baselinecfg1FP16FP16INT4LLM Robustnesscfg2FP16INT8FP16Proj\. Sensitivitycfg3FP16INT8INT4cfg1 \+ cfg2cfg4INT8FP16FP16Enc\. Sensitivitycfg5INT8FP16INT4cfg1 \+ cfg4## 1Introduction

Recently, large multimodal models \(LMMs\) have opened a new horizon in artificial intelligence \(AI\) research owing to their innovative reasoning capabilities, which combine visual understanding and language generation\(Liuet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib1); Sunet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib2); Hurstet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib3)\)\. Small vision\-language models \(sVLMs\) with 3 billion or less parameters are rapidly emerging as practical candidates for on\-device AI\. These models can be used in mobile, offline, privacy\-protected, and low\-power environments\(Chuet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib4); Zhouet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib5); Linet al\.,[2026](https://arxiv.org/html/2607.08029#bib.bib6)\)\. However, heterogeneous edge system\-on\-chip \(SoC\) environments such as the NVIDIA Jetson Orin series are subject to fundamental hardware constraints, including strict memory bandwidth \(NX: 102\.4 GB/s; AGX: 204\.8 GB/s\) and limited VRAM capacity \(NX: 16 GB; AGX: 64 GB\)\. In such an environment, model quantization is gaining attention as an essential compression technique to ensure practical reasoning capacity\(Kimet al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib7); Caiet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib8); Xuet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib9)\)\.

However, most existing research on the quantization of large language models \(LLMs\) and vision\-language models \(VLMs\) has generally focused on evaluating the entire model end\-to\-end in high\-performance server\-grade GPU environments\(Leeet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib10); Xiaoet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib11); Frantaret al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib12); Dettmerset al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib13)\)\. Existing studies have yet to provide a systematic basis for key deployment\-time decisions faced by engineers, including which components should be prioritized for quantization and how bit\-widths should be allocated across these components\. A VLM typically consists of three major components: a vision encoder that extracts visual features, a projector that maps these features into the embedding space of the language model, and an LLM backbone that performs the final inference\(Liuet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib1); Alayracet al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib14); Chenet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib15)\)\. Each component has different parameter sizes, model structures, and computational characteristics; thus, the sensitivity of each component to quantization should be evaluated independently\. Studies that control and quantify the marginal contribution of each component are rare\(Sharsharet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib16)\)\. Cross\-platform comparative studies, especially on heterogeneous edge SoCs, are even rarer\.

Motivated by our preliminary observations, we formulate five hypotheses and evaluate each by designing a hypothesis\-driven evaluation framework\. We evaluate five architecturally diverse sVLMs: Qwen3\-VL\-2B, DeepSeek\-VL2\-Tiny, PaliGemma2\-3B, LLaVA\-OV\-0\.5B, and Kosmos\-2\.5\. The developed configurations include single\-component ablations \(cfg1,cfg2, andcfg4\), which isolate the effect of quantizing individual components, and composite settings \(cfg3andcfg5\), which capture interactions among jointly quantized components\. We conduct all experiments on the Jetson Orin NX and Jetson AGX platforms and analyze each configuration along four axes: MME benchmark accuracy, VRAM footprint, component\-wise latency across the vision encoder, projector, and LLM time per output token \(TPOT\), and intelligence\-per\-joule \(IPJ\)\(Saad\-Falconet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib17)\)\.

We formulate the following hypotheses:

- •Scale\-Dependent Sensitivity according to Parameter Size \(H1\):INT4 quantization of the LLM backbone results in increasingly severe accuracy degradation as the model size decreases\. This effect is expected to be especially pronounced in ultra\-small sVLMs with fewer than 1B parameters, where limited representational redundancy makes the model less robust to quantization noise\.
- •Vision Encoder Architecture\-Specific Latency Anomaly \(H2\):SigLIP\-based vision encoder quantization disproportionately increases the latency relative to the variation in accuracy because of interactions between certain architectures and the quantization software stack that are specific to the hardware\.
- •Relationship between Resource Savings and Inference Efficiency \(H3\):BitsAndBytes\-based LLM INT4 quantization achieves over 50% VRAM savings, whereas TPOT consistently increases due to the dequantization overhead\.
- •Non\-Additive Interaction of Composite Quantization Errors \(H4\):The performance degradation when quantizing heterogeneous modalities is characterized by a non\-additive pattern that cannot be explained by simple arithmetic summation of the individual component errors\.
- •Stability and Transferability of Performance Rankings across Platforms \(H5\):The accuracy ranking of the models is platform\-invariant across the quantization configurations and heterogeneous edge platforms \(NX vs\. AGX\), whereas the latency and energy profiles are platform\-specific\.

The main contributions of this study are as follows:\(1\)We present the first systematic controlled experiment that independently separates the vision encoder, projector, and LLM backbone of VLMs using 3 billion parameters to measure accuracy sensitivity and resource efficiency\.\(2\)Through cross\-hardware evaluation using two heterogeneous platforms \(Jetson Orin NX and AGX\), we empirically distinguish platform\-consistent accuracy rankings from hardware\-specific latency profiles\.\(3\)We identify abnormal latency phenomena that occur when applying vision INT8 quantization in SigLIP\-based architectures, and analyze these issues from a hardware\-software stack perspective\.\(4\)By testing the hypotheses, we provide practical deployment guidelines for implementing modality\- and hardware\-aware precision allocation strategies in edge environments\.

## 2Related Works

Small Vision\-Language Models\.A sVLM is a multimodal AI system with 7 billion parameters or less\. sVLMs are designed to efficiently process and integrate visual and textual information while supporting implementation in resource\-constrained environments, such as edge and mobile devices\(Chuet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib4)\)\. A typical VLM architecture consists of three core components: a vision encoder that extracts visual features from images, a projector that maps visual representations into the embedding space of the language model, and a LLM backbone for text generation and reasoning\(Liuet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib1)\)\. These components serve distinct functions within the model and may, therefore, differ in their sensitivity to quantization\(Sharsharet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib16)\)\. Table[4](https://arxiv.org/html/2607.08029#S3.T4)summarizes the architectural configurations of the five models selected for this study\. Except for Kosmos\-2\.5, the other four models use SigLIP as the vision encoder\. Thus, SigLIP can be treated as a controlled architectural factor in the architecture\-effect analysis for RQ2, as demonstrated in Section 4\.2\.

Quantization of VLMs\.Model quantization is a widely used compression technique that reduces the memory footprint and computational overhead by representing model weights and activations in low\-precision formats\. Existing quantization methods, such as activation\-aware weight quantization \(AWQ\), generative pre\-trained transformer quantization \(GPTQ\), SmoothQuant, and BitsAndBytes\-based INT8/INT4 quantization, have demonstrated substantial efficiency gains in large\-scale neural models\(Linet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib18); Frantaret al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib12); Xiaoet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib11); Dettmerset al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib13)\)\.

These techniques have been extensively explored for LLMs\. Nevertheless, their extension to VLMs is less straightforward\. VLMs combine multiple heterogeneous components, including a vision encoder, a cross\-modal projector, and an LLM backbone\. Because these components differ in size, function, and sensitivity to numerical precision, uniform quantization may not provide the best trade\-off between efficiency and accuracy cost\(Sharsharet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib16); Xueet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib20); Liet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib19)\)\. This motivates component\-wise mixed\-precision quantization, where different bit\-widths are assigned to different VLM components based on their quantization sensitivity and deployment requirements\.

## 3Experimental Setup

### 3\.1Hardware Setup

The Jetson Orin family is an edge AI platform built on NVIDIA Ampere architecture\. It offers multiple memory configurations, such as the NX \(16 GB, LPDDR5\) and AGX \(64 GB, LPDDR5\)\. Even with the same quantization configuration, the latency profile can vary across hardware platforms depending on the level of optimization of the kernels\. INT4 quantization on certain platforms, for instance, involves dequantization overhead when using BitsAndBytes\. If native INT4 operation support is limited in the Ampere architecture of Jetson Orin, the TPOT may increase\(Xuet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib9)\)\. This explains why VRAM savings do not necessarily correspond to latency improvements, as confirmed by the data presented in Section 4\.3\.

Table 3:Hardware specifications of Jetson Orin platformsSpecificationJetson Orin NXJetson Orin AGXGPU1024\-core Ampere, 32 Tensor Cores2048\-core Ampere, 64 Tensor CoresCPU8\-core ARM CPU12\-core ARM CPUMemory16GB LPDDR564GB LPDDR5Memory BW102\.4 GB/s204\.8 GB/sTDP25 W \(MODE\_25W\)50 W \(MODE\_50W\)AI Perf\.157 TOPS275 TOPSJetpack6\.2\.16\.2\.1The Jetson Orin family comprises the Jetson Orin Nano, NX, and AGX\. The Jetson Orin Nano was excluded from the experimental set because it consistently encountered out\-of\-memory \(OOM\) failures during VLM inference\. Prior work has shown that these failures stem from the substantial peak memory overhead introduced during the quantized weight conversion stage in BitsAndBytes\(Dettmerset al\.,[2022](https://arxiv.org/html/2607.08029#bib.bib13)\)\. This emphasizes the need to provide additional memory capacity and bandwidth at runtime, beyond that required to simply load the model parameters\. Consequently, experiments were carried out only on the Jetson Orin NX and AGX platforms, as summarized in Table[3](https://arxiv.org/html/2607.08029#S3.T3)\(NVIDIA,[2026](https://arxiv.org/html/2607.08029#bib.bib21)\)\.

### 3\.2Models

This study focuses on five 3B\-parameter models supported by the VLMEvalKit framework, which is also employed in the Open VLM Leaderboard\(OpenCompass,[2026](https://arxiv.org/html/2607.08029#bib.bib22)\)\. These models were chosen based on their frequency of academic citation and prevalence in industrial applications\. The main selection criteria are architectural diversity \(SigLIP vs\. non\-SigLIP\)\(Zhaiet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib23)\)and variability in the LLM backbones, as shown in Table[4](https://arxiv.org/html/2607.08029#S3.T4)\(Baiet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib24); Wuet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib25); Steineret al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib26); Liet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib27); Lvet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib28)\)\.

Table 4:Detailed architectural specifications and parameter distributions across five multimodal models\. All percentages represent the relative share of the total model parameters\. For Kosmos\-2\.5, the output projection is tied to the input embedding, so no separate lm\_head parameters are counted\.ModelVision EncoderLanguage ModelProjectorArchParams DetailsArchParams DetailsArchParams DetailsQwen3\-VL\-2BSigLIP\-2∼\\sim306M \(12\.6%\)Qwen3∼\\sim1\.72B \(70\.6%\)MLP\-based∼\\sim100M \(4\.13%\)\-Instruct \(2\.44B\)Linear: 302M Conv3d: 1\.6M\(Dense/MoE\)lm\_head: 311M \(12\.8%\)MergerDeepSeek\-VL2SigLIP\-∼\\sim428M \(12\.7%\)DeepSeek∼\\sim2\.93B \(87\.1%\)2\-layer∼\\sim7\.53M \(0\.22%\)\-Tiny \(3\.37B\)So400mLinear: 426M ViT: 0\.8M\-MoEMoE\-Gate: 0\.9MMLPPaliGemma2\-3BSigLIP\-∼\\sim412M \(11\.4%\)Gemma 2∼\\sim2\.61B \(72\.2%\)Linear∼\\sim2\.65M \(0\.07%\)\(3\.6B\)So400mLinear: 411M Conv2d: 0\.7Mlm\_head: 592M \(16\.4%\)Proj\.LLaVA\-OV\-0\.5BSigLIP\-∼\\sim397M \(38\.6%\)Qwen\-2∼\\sim494M \(48\.0%\)2\-layer∼\\sim1\.83M \(0\.18%\)\(1\.03B\)So400mLinear: 396M Conv2d: 0\.7Mlm\_head: 136M \(13\.2%\)MLPKosmos\-2\.5Pix2Struct∼\\sim512\.8M \(37\.31%\)Decoder\-only∼\\sim846\.9M \(61\.61%\)Resampler∼\\sim14\.9M \(1\.09%\)\(1\.37B\)\-based ViTLinear: 512MTrans\.lm\_head: tiedAlgorithm 1Component Latency & VRAM Profiling for sVLM1:Quantized VLM

ℳQ=\{V,P,L\}\\mathcal\{M\}\_\{Q\}=\\\{V,P,L\\\}, Evaluation Dataset

DD, Output token length

NN
2:Accuracy Score, Peak VRAM, Latency components \(

LV,LP,LT​P​O​TL\_\{V\},L\_\{P\},L\_\{TPOT\}\)

3:

4:\# 1\. Accuracy and Peak VRAM Measurement

5:ResetCUDAMemoryStats\(\)

6:

S​c​o​r​e←VLMEvalKit\_Evaluate​\(ℳQ,D\)Score\\leftarrow\\text\{VLMEvalKit\\\_Evaluate\}\(\\mathcal\{M\}\_\{Q\},D\)
7:

P​e​a​k​\_​V​R​A​M←GetMaxMemoryAllocated\(\)Peak\\\_VRAM\\leftarrow\\text\{GetMaxMemoryAllocated\(\)\}
8:

9:\# 2\. Latency Analysis

10:

I,X←Sample​\(D\)I,X\\leftarrow\\text\{Sample\}\(D\)⊳\\trianglerightImageII, Text PromptXX

11:

12:CUDASynchronize\(\)

13:

t0←GetCurrentTime\(\)t\_\{0\}\\leftarrow\\text\{GetCurrentTime\(\)\}
14:

Ev←ℳQ\.V​\(I\)E\_\{v\}\\leftarrow\\mathcal\{M\}\_\{Q\}\.V\(I\)
15:CUDASynchronize\(\)

16:

t1←GetCurrentTime\(\)t\_\{1\}\\leftarrow\\text\{GetCurrentTime\(\)\}
17:

LV←t1−t0L\_\{V\}\\leftarrow t\_\{1\}\-t\_\{0\}⊳\\trianglerightVision Encoder Latency

18:

19:

Ep←ℳQ\.P​\(Ev\)E\_\{p\}\\leftarrow\\mathcal\{M\}\_\{Q\}\.P\(E\_\{v\}\)
20:CUDASynchronize\(\)

21:

t2←GetCurrentTime\(\)t\_\{2\}\\leftarrow\\text\{GetCurrentTime\(\)\}
22:

LP←t2−t1L\_\{P\}\\leftarrow t\_\{2\}\-t\_\{1\}⊳\\trianglerightProjector Latency

23:

24:

I​n​p​u​t​s←Concat​\(Ep,Embed​\(X\)\)Inputs\\leftarrow\\text\{Concat\}\(E\_\{p\},\\text\{Embed\}\(X\)\)
25:

ℳQ\.L\.Generate​\(I​n​p​u​t​s,max\_new\_tokens=1\)\\mathcal\{M\}\_\{Q\}\.L\.\\text\{Generate\}\(Inputs,\\text\{max\\\_new\\\_tokens\}=1\)
26:CUDASynchronize\(\)

27:

t3←GetCurrentTime\(\)t\_\{3\}\\leftarrow\\text\{GetCurrentTime\(\)\}
28:

29:

O​u​tN←ℳQ\.L\.G​\(I​n​p​u​t​s,max\_tokens=N\)Out\_\{N\}\\leftarrow\\mathcal\{M\}\_\{Q\}\.L\.\\text\{G\}\(Inputs,\\text\{max\\\_tokens\}=N\)
30:CUDASynchronize\(\)

31:

t4←GetCurrentTime\(\)t\_\{4\}\\leftarrow\\text\{GetCurrentTime\(\)\}
32:

33:

T​o​t​a​l​\_​D​e​c​o​d​e​\_​T​i​m​e←t4−t3Total\\\_Decode\\\_Time\\leftarrow t\_\{4\}\-t\_\{3\}
34:

LT​P​O​T←T​o​t​a​l​\_​D​e​c​o​d​e​\_​T​i​m​e/\(N−1\)L\_\{TPOT\}\\leftarrow Total\\\_Decode\\\_Time/\(N\-1\)⊳\\trianglerightTPOT

35:return

S​c​o​r​e,P​e​a​k​\_​V​R​A​M,LV,LP,LT​P​O​TScore,Peak\\\_VRAM,L\_\{V\},L\_\{P\},L\_\{TPOT\}

### 3\.3Datasets

For evaluation, we use the multimodal large language model evaluation \(MME\) benchmark, which is the most widely adopted benchmark for VLM models\. The MME scores each subtask by combining the ACC \(accuracy per question\) and ACC\+ \(granted when both questions for a single image are answered correctly\)\. The overall maximum score is 2,800 points, calculated as the sum of the cognition and perception scores\(Fuet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib29)\)\.

### 3\.4Quantization Methods

In this study, we applied 4\-bit \(INT4\) quantization exclusively to the LLM to reduce VRAM consumption, given that the LLM typically represents more than 70% of the total parameters in most VLMs \(Table[4](https://arxiv.org/html/2607.08029#S3.T4)\)\. Vision\-related components \(the vision encoder and projector\), which are more susceptible to degradation arising from information loss, were kept at INT8 or higher precision\. This prevents performance degradation in terms of modality alignment performance\. Based on this design, we defined six quantization configurations: three single\-component quantization ablations \(cfg1,cfg2, andcfg4\), in which two components are fixed at FP16 and one component is quantized; and two composite quantizations \(cfg3andcfg5\), in which both components are quantized simultaneously\. This design incorporates structural pairs that enable independent measurement of the limiting effects of each component and verification of the additivity in composite quantization configurations\. The detailed experimental configurations are presented in Table[2](https://arxiv.org/html/2607.08029#S0.T2)\.

### 3\.5Evaluation Metrics

We utilized the open\-source VLMEvalKit framework for accuracy evaluation\(Duanet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib30)\)\. The latency was assessed by decomposing it into three parts: vision encoding, projector, and LLM TPOT\. The VRAM consumption was reported as the peak usage \(GB\)\. The pseudocode used for reproduction is provided in Algorithm 1\.

Algorithm 1 is built to precisely pinpoint performance bottlenecks in edge environments\. The peak VRAM indicates increases in temporary memory usage introduced by the BitsAndBytes quantization model during dequantization in the inference phase\. It also reflects the actual hardware capacity\. In addition, the latency was decomposed and measured around each component \(vision encoder, projector, and LLM\) by recording the execution times before and after each component\. Notably, for the LLM, only TPOT was measured independently, while excluding the latency associated with generating the first token \(the prefill phase\)\.

## 4Experimental Results

In this section, we explore the impact of component\-wise quantization of the sVLMs on the performance and hardware profiles in edge SoC environments, such as the Jetson Orin NX/AGX, by examining five research questions \(RQs\)\.

### 4\.1RQ1: \[Scale and Architecture Sensitivity\] For different backbone architectures and model scales, how does INT4 quantization of the LLM backbone affect the change in MME accuracy?

The accuracy variations for each model \(Δ​M​M​E=c​f​gN−b​a​s​e\\Delta MME=cfg\_\{N\}\-base\) from the single\-component ablations \(cfg1: LLM INT4;cfg2: Projector INT8; andcfg4: Vision INT8\) are summarized in Table[5](https://arxiv.org/html/2607.08029#S4.T5)\. The experiments revealed a complex interplay between parameter scale, LLM architecture, and INT4 quantization sensitivity\. While the severe degradation in ultra\-small models initially appears to align with a scale\-dependent trend, a cross\-architectural comparison reveals that algorithmic structure dominates over mere parameter scale\. Consequently, the empirical evidence rejects the original hypothesis,H1\. Reflecting the interaction between architecture and scale, we reviseH1into \(H1rev\\text\{H1\}\_\{\\text\{rev\}\}\): The quantization sensitivity of sub\-3B VLMs is determined by the underlying structural paradigm \(MoE vs\. Dense\), with parameter scale acting as an aggravating factor only within a homogeneous architecture family\.

Table 5:Δ\\DeltaMME comparison across models undercfg1\.Model\\columncolorgray\!15\# LLMParams \(B\)\\columncolorgray\!15MoEΔ\\DeltaMME \(NX\)Δ\\DeltaMME \(AGX\)Qwen3\-VL\\columncolorgray\!151\.72\\columncolorgray\!15O\+56\.04\+101\.88DeepSeek\-VL2\\columncolorgray\!152\.93\\columncolorgray\!15O\+39\.50\+48\.18PaliGemma2\\columncolorgray\!152\.61\\columncolorgray\!15X\-62\.12\-59\.35LLaVA\-OV\\columncolorgray\!150\.49\\columncolorgray\!15X\-220\.02\-156\.08Key Findings:

- •Architecture Dominance Over Scale Sensitivity \(Falsification of H1\)\.Although the substantial performance decline of LLaVA\-OV \(Qwen2\-0\.5B, \-220\.02 points\) initially suggests a scale\-dependent vulnerability, this hypothesis is obscured by architectural variables\. Notably, the larger model, PaliGemma2 \(2\.61B\), shows a significant decline of 62\.12 points, while the smaller MoE\-based Qwen3 \(1\.72B\) demonstrates an enhancement in accuracy, gaining 56\.04 points\. This contradiction rejectsH1based exclusively on scale\.
- •Impact of Architectural Bifurcation on INT4 Accuracy of LLM\.The LLM architecture fundamentally dictates the impact ofcfg1on accuracy\. Qwen3 \(MoE\-based, \+56\.04\) and DeepSeek\-VL2 \(MoE\-based, \+39\.50\), achieve consistent improvements across all platforms, whereas the dense PaliGemma2 \(Gemma\-2B, \-62\.12\) and LLaVA\-OV \(Qwen2\-0\.5B, \-220\.02\) suffer pronounced performance declines\. This pattern supports a reformulatedH1rev\\text\{H1\}\_\{\\text\{rev\}\}, aligning with prior studies that identify MoE models as notably resilient to severe quantization noise\(Frantar and Alistarh,[2023](https://arxiv.org/html/2607.08029#bib.bib31)\)\.

### 4\.2RQ2: \[Architectural Latency Anomaly\] Does vision INT8 quantization have a significantly different impact on the latency of vision encoding for SigLIP\-based and non\-SigLIP architectures? Is this latency change independent of the accuracy changes?

The INT8 quantization \(cfg4\) of the vision encoder resulted in increased latency, which was largely unaffected by changes in the accuracy of the models based on the SigLIP architecture\. The latency increase ratio \(Table[6](https://arxiv.org/html/2607.08029#S4.T6)\) is calculated asr​a​t​i​o=c​f​g​4/c​f​g​0ratio=cfg4/cfg0\. This ratio was rounded to three decimal places\.

Table 6:Impact of vision INT8 quantization on vision encoding latency \(ms\)\.ModelVisionEncoderJetson NXJetson AGXΔ\\DeltaMME\(NX\)cfg0cfg4\\columncolorgray\!15Ratio↓\\downarrowcfg0cfg4\\columncolorgray\!15Ratio↓\\downarrowPG2SigLIP\-So400m311\.91,203\.5\\columncolorgray\!15×\\times3\.8698\.5459\.1\\columncolorgray\!15×\\times4\.66\+7\.3DS\-VL2SigLIP\-So400m1,423\.34,297\.3\\columncolorgray\!15×\\times3\.02903\.52,810\.7\\columncolorgray\!15×\\times3\.11\-14\.1LV\-OVSigLIP\-So400m719\.81,879\.1\\columncolorgray\!15×\\times2\.61381\.2925\.1\\columncolorgray\!15×\\times2\.43\+7\.4QW3\-VLSigLIP\-2394\.6684\.2\\columncolorgray\!15×\\times1\.73160\.6403\.3\\columncolorgray\!15×\\times2\.51\-48\.3Kosmos\-2\.5Pix2Struct11,029\.212,644\.2\\columncolorgray\!15×\\times1\.152,903\.63,479\.4\\columncolorgray\!15×\\times1\.20\+4\.2Note: Latency is measured in milliseconds \(ms\)\.Table 7:Impact of LLM INT4 quantization \(cfg1\) on VRAM usage and token generation speed \(TPOT\) relative tocfg0\(baseline\)\.ModelsVRAM \(GB\)↓\\downarrowTPOT \(ms\)↓\\downarrowcfg0cfg1\\columncolorgray\!15Reduct\.cfg0cfg1\\columncolorgray\!15Incr\.NXQwen3\-VL4\.062\.13\\columncolorgray\!15\-47\.5%111\.1173\.1\\columncolorgray\!15\+55\.8%PaliGemma25\.853\.13\\columncolorgray\!15\-46\.5%166\.0183\.3\\columncolorgray\!15\+10\.4%LLaVA\-OV1\.891\.43\\columncolorgray\!15\-24\.3%96\.5143\.9\\columncolorgray\!15\+49\.1%AGXQwen3\-VL4\.062\.13\\columncolorgray\!15\-47\.5%136\.2212\.4\\columncolorgray\!15\+55\.9%PaliGemma25\.853\.18\\columncolorgray\!15\-45\.6%137\.9182\.1\\columncolorgray\!15\+32\.0%LLaVA\-OV2\.251\.76\\columncolorgray\!15\-21\.8%90\.5141\.4\\columncolorgray\!15\+56\.3%Key Findings:

- •Sharp Increase in Vision INT8 Latency of SigLIP\-So400m\-Based Models\.Whencfg4is applied, the latency of the vision encoders of the three SigLIP\-So400m\-based models \(PaliGemma2, DeepSeek\-VL2, and LLaVA\-OV\) increases sharply \(by a factor of 2\.43\-4\.66\), compared with that \(1\.73\-2\.51\-fold\) for the SigLIP\-2\-based Qwen3\-VL\. The Kosmos\-2\.5 model with a non\-SigLIP architecture exhibits a smaller increase \(1\.15\-1\.20\-fold\)\. These results support the latency anomaly phenomenon specific to the SigLIP\-BitsAndBytes\-Ampere interaction established in H2\.
- •Independence of Accuracy Changes\.This latency increase occurs regardless of changes in accuracy\. For PaliGemma2 and LLaVA\-OV,cfg4causedΔ\\DeltaMME to increase by 7\.3 and 7\.4, respectively, indicating a slight improvement in accuracy\. However, the vision encoding latency increased by factors of 3\.86 and 2\.61, respectively\. Because there was no loss in accuracy, the intuitive judgment that vision INT8 was efficient overlooks the latency cost\. This error was confirmed through actual measurements\.
- •Hardware\-Software Stack Incompatibility\.This incompatibility arises when the BitsAndBytes INT8 kernel fails to efficiently accelerate the computational patterns of the SigLIP vision transformer on the Jetson Orin Ampere architecture\(Caiet al\.,[2023](https://arxiv.org/html/2607.08029#bib.bib8)\)\. Although the Pix2Struct\-style Kosmos\-2\.5 encoder incurs the highest absolute vision latency due to its variable\-resolution patching, the relative INT8 overhead remains small precisely because its non\-SigLIP computation pattern avoids the kernel fragmentation that BitsAndBytes induces on SigLIP encoders\. We verify this with a non\-SigLIP control under the same isolated vision\-path protocol in Appendix[E](https://arxiv.org/html/2607.08029#A5)\.

### 4\.3RQ3: \[Resource and Accuracy Trade\-offs\] When applying LLM INT4 quantization, does the VRAM reduction translate into an actual improvement in the token generation latency \(TPOT\), or is it offset by the dequantization overhead?

Although INT4 quantization of LLMs significantly lowers the VRAM usage, it also reduces the token generation speed due to the extra computational overhead for dequantization\. The formulas used to calculate the percentage change of VRAM and TPOT\(Table[7](https://arxiv.org/html/2607.08029#S4.T7)\)are as follows:

ΔX=Xcfg1−Xcfg0Xcfg0×100\(%\),X∈\{VRAM,TPOT\}\\Delta X=\\frac\{X\_\{\\text\{cfg1\}\}\-X\_\{\\text\{cfg0\}\}\}\{X\_\{\\text\{cfg0\}\}\}\\times 100\(\\%\),\\quad X\\in\\\{VRAM,TPOT\\\}
Table 8:Energy consumption \(joules\) and rate of increase owing tocfg1quantization across Jetson platforms\.ModelsNXAGXcfg0cfg1\\columncolorgray\!15Incr\.↓\\downarrowcfg0cfg1\\columncolorgray\!15Incr\.↓\\downarrowQwen3\-VL12\.3119\.04\\columncolorgray\!15\+54\.7%4\.907\.12\\columncolorgray\!15\+45\.3%PaliGemma235\.9845\.95\\columncolorgray\!15\+27\.7%13\.3116\.13\\columncolorgray\!15\+21\.2%LLaVA\-OV21\.4721\.87\\columncolorgray\!15\+1\.9%21\.5521\.86\\columncolorgray\!15\+1\.4%Note: Energy consumption is measured in Joules \(J\)\.Key Findings:

- •Trade\-off between VRAM Reduction and TPOT Increase\.LLM INT4 \(cfg1\) lowers the VRAM usage by 21\.8–47\.5% but raises the TPOT by 10\.4–56\.3% across all platforms \(Table[7](https://arxiv.org/html/2607.08029#S4.T7)\)\. These empirical results demonstrate that within the Jetson Orin Ampere architecture, the dequantization overhead introduced by BitsAndBytes outweighs the native computation speedup provided by INT4\(Xuet al\.,[2024](https://arxiv.org/html/2607.08029#bib.bib9)\)\. This provides strong evidence supporting H3 for balancing resource savings and inference efficiency\. Consequently, LLM INT4 implemented via BitsAndBytes should be viewed purely as a VRAM\-saving approach, rather than as a technique for reducing latency\.
- •Paradoxical Increase in Energy Consumption\.Despite the decrease in VRAM usage, Qwen3\-VL increased energy consumption substantially, i\.e\., by 54\.7% for NX and 45\.3% for AGX\. This increase results from the longer time required for the dequantization operations, which in turn increases power usage\. In contrast, LLaVA\-OV incurred only a slight increase in energy usage \(\+1\.4% to \+1\.9%\)\. This indicates that the computational cost of dequantization is relatively low for small\-scale LLMs \(0\.49B\)\.

### 4\.4RQ4: \[Interaction Dynamics\] Can the accuracy degradation caused by composite quantization configurations \(cfg3: Projector INT8 \+ LLM INT4 andcfg5: Vision INT8 \+ LLM INT4\) be predicted by adding the corresponding single\-component ablation results, or is a non\-additive pattern observed?

Two configuration pairs are examined to assess additivity:

- •cfg3:Δ​M​M​E​\(c​f​g​3\)​vs\.​Δ​M​M​E​\(c​f​g​1\)​\+​Δ​M​M​E​\(c​f​g​2\)\\Delta MME\(cfg3\)\\text\{ vs\. \}\\Delta MME\(cfg1\)\\text\{ \+ \}\\Delta MME\(cfg2\)
- •cfg5:Δ​M​M​E​\(c​f​g​5\)​vs\.​Δ​M​M​E​\(c​f​g​1\)​\+​Δ​M​M​E​\(c​f​g​4\)\\Delta MME\(cfg5\)\\text\{ vs\. \}\\Delta MME\(cfg1\)\\text\{ \+ \}\\Delta MME\(cfg4\)

We assess additivity separately for the two composite configurations\. Forcfg3, the observed degradation is nearly additive across all models, closely matching the linear sum of the individual component errors\. In contrast,cfg5deviates from additivity in a model\-specific manner\. These results lead us to interpret H4 in a more targeted form: composite quantization is near\-additive forcfg3, and the hypothesized non\-additivity emerges incfg5, in an architecture\-dependent manner\.

Table 9:Additivity analysis: comparison of expected sum of individual quantization impacts \(Δ\\Deltacfg1 \+Δ\\Deltacfg2\) with the actual combined impact \(Δ\\Deltacfg3\) in NX\.ModelΔ\\DeltaMME\(cfg1\)Δ\\DeltaMME\(cfg2\)\\columncolorgray\!15ExpectedSum\\columncolorgray\!15ActualΔ\\Deltacfg3Residual\(cfg3\-Exp\)Qwen3\-VL\+56\.04\-31\.89\\columncolorgray\!15\+24\.15\\columncolorgray\!15\+24\.570\.42DeepSeek\-VL2\+39\.50\+5\.86\\columncolorgray\!15\+45\.36\\columncolorgray\!15\+41\.80\-3\.56PaliGemma2\-62\.12\+2\.67\\columncolorgray\!15\-59\.45\\columncolorgray\!15\-60\.77\-1\.32LLaVA\-OV\-220\.02\-1\.42\\columncolorgray\!15\-221\.44\\columncolorgray\!15\-224\.24\+2\.80Kosmos\-2\.5\-10\.54\+1\.97\\columncolorgray\!15\-8\.57\\columncolorgray\!15\-8\.38\+0\.19Table 10:Additivity analysis: comparison of expected sum of individual quantization impacts \(Δ\\Deltacfg1 \+Δ\\Deltacfg4\) with the actual combined impact \(Δ\\Deltacfg5\) in NX\.ModelΔ\\DeltaMME\(cfg1\)Δ\\DeltaMME\(cfg4\)\\columncolorgray\!15ExpectedSum\\columncolorgray\!15ActualΔ\\Deltacfg5Residual\(cfg5\-Exp\)Qwen3\-VL\+56\.04\-48\.27\\columncolorgray\!15\+7\.77\\columncolorgray\!15\+7\.32\-0\.45DeepSeek\-VL2\+39\.50\-14\.14\\columncolorgray\!15\+25\.36\\columncolorgray\!15\+33\.39\+8\.03PaliGemma2\-62\.12\+7\.27\\columncolorgray\!15\-54\.85\\columncolorgray\!15\-71\.18\-16\.33LLaVA\-OV\-220\.02\+7\.37\\columncolorgray\!15\-212\.65\\columncolorgray\!15\-212\.75\-0\.10Kosmos\-2\.5\-10\.54\+4\.18\\columncolorgray\!15\-6\.36\\columncolorgray\!15\-5\.63\+0\.73Key Findings:

- •Overall Near\-Additivity ofcfg3\(Projector \+ LLM\)\.For all models, thecfg3residuals fall within a±4\\pm 4\-point range, demonstrating nearly additive behavior, as shown in Table[9](https://arxiv.org/html/2607.08029#S4.T9)\. This indicates minimal interference between the Projector INT8 and LLM INT4 components and confirms that their independent effects overlap linearly\.
- •Model\-Specific Non\-Additivity ofcfg5\(Vision \+ LLM\)\.The behavior ofcfg5is non\-uniform across models, showing distinct patterns for different architectures\. In PaliGemma2, an additional accuracy degradation of 16\.33 \(NX\) is observed in the residuals compared to the expected sum, whereas DeepSeek\-VL2 exhibits a super\-additive pattern \(\-8\.03\), as shown in Table[10](https://arxiv.org/html/2607.08029#S4.T10)\. These results suggest that the modality\-alignment pathways between the vision encoder and the LLM differ across architectures, underscoring the need for prior architecture\-specific evaluation forcfg5\.

### 4\.5RQ5: \[Hardware Transferability\] Is the ordering of the model accuracy across different quantization settings the same on both the Jetson Orin NX and AGX platforms? Are the latency and energy characteristics dependent on the specific platform?

The empirical results demonstrated that the accuracy rankings are strictly preserved across heterogeneous edge platforms regardless of the quantization configuration, thereby obviating the need for hardware\-specific accuracy re\-validation during deployment \(Table[11](https://arxiv.org/html/2607.08029#S4.T11)\)\.

Table 11:Ranking of evaluated models based on MME total scores across Jetson Orin NX and AGX platforms\.RankJetson Orin NX \(Score\)Jetson Orin AGX \(Score\)1Qwen3\-VL\-2B \(2021\.78\)Qwen3\-VL\-2B \(1983\.32\)2DeepSeek\-VL2 \(1911\.12\)DeepSeek\-VL2 \(1916\.71\)3PaliGemma2 \(1661\.01\)PaliGemma2 \(1658\.10\)4LLaVA\-OV\-0\.5B \(1361\.83\)LLaVA\-OV\-0\.5B \(1369\.73\)5Kosmos\-2\.5 \(662\.30\)Kosmos\-2\.5 \(662\.05\)In contrast, energy optimization requires a granular understanding of how architectural bottlenecks interact with platform\-specific bandwidth constraints\. To quantify this, we adopt the IPJ metric, calculated as the normalized MME accuracy per unit of energy consumed \(in Joules\)\(Saad\-Falconet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib17)\):

I​P​J​\(m,h\)=𝔼q​\[a​c​c​\(m,q\)\]𝔼q​\[P​\(m,h,q\)⋅τ​\(m,h,q\)\]IPJ\(m,h\)=\\frac\{\\mathbb\{E\}\_\{q\}\[acc\(m,q\)\]\}\{\\mathbb\{E\}\_\{q\}\[P\(m,h,q\)\\cdot\\tau\(m,h,q\)\]\}wherea​c​c​\(m,q\)acc\(m,q\)denotes the normalized MME score \(S​c​o​r​e/2800Score/2800\)for a given modelmmand processed queryqq\. The denominator represents the energy consumption; specifically, the average powerP​\(m,h,q\)P\(m,h,q\)is empirically measured using NVIDIA’stegrastatsutility at 50ms sampling intervals, and the total inference latencyτ​\(m,h,q\)\\tau\(m,h,q\)spans the end\-to\-end model execution, including vision encoding, projector mapping, the prefill phase, and sequential token generation\.

Table 12:Comparison of intelligence\-per\-joule \(IPJ\) usingcfg0andcfg1on Jetson Orin platforms\.Modelscfg0cfg1NXAGXNXAGXQwen3\-VL5\.8714\.453\.9010\.38DeepSeek\-VL21\.373\.030\.761\.72PaliGemma21\.654\.451\.253\.54LLaVA\-OV2\.272\.251\.861\.98Kosmos\-2\.50\.120\.370\.110\.36Note: IPJ values represent accuracy per Joule\(Saad\-Falconet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib17)\)\.Key Findings:

- •Platform\-Invariant Accuracy Ranking\.The experimental results confirmed that the MME accuracy ranking by model was completely consistent across all quantization configurations for the two platforms, Jetson NX and AGX, in the order: Qwen3\-VL\>\>DeepSeek\-VL2\>\>PaliGemma2\>\>LLaVA\-OV\>\>Kosmos\-2\.5\. These results strongly support hypothesis H5, demonstrating that the architecture of the algorithm has a greater effect on the sensitivity of the models to the quantization settings than the available hardware resources\.
- •Asymmetric Power Scaling of Power Consumption Profiles\.Even for the same quantization configuration, NX \(TDP 25W\) and AGX \(TDP 50W\) exhibit different power consumption patterns\. As shown in Table[11](https://arxiv.org/html/2607.08029#S4.T11), AGX consistently has a lower average power consumption than NX \(ratio 0\.65–0\.70\) on Qwen3\-VL\. However, on PaliGemma2, LLaVA\-OV, and Kosmos\-2\.5, AGX has a higher average power consumption than NX \(ratio 1\.02–1\.24\)\. These results suggest that the higher TDP \(50 W\) of AGX does not always lead to higher energy consumption\. For Qwen3\-VL, the memory bandwidth of AGX is twice as wide \(204\.8 GB/s\), which alleviates computational bottlenecks and results in a lower average power consumption\.
- •Platform Specificity and Deployment Strategy for Energy Efficiency \(IPJ\)\.In contrast to the accuracy of task execution, the energy efficiency is highly sensitive to specific model\-platform combinations\. Aligning with earlier work highlighting the importance of hardware advances\(Saad\-Falconet al\.,[2025](https://arxiv.org/html/2607.08029#bib.bib17)\), the present results indicate that Qwen3\-VL delivers up to2\.5×2\.5\\timeshigher IPJ on the AGX platform than on the NX, with some models surpassing a3×3\\timesgain\. Together, these results reinforce the concept that achieving peak efficiency for edge\-AI requires an integrated HW\-SW co\-design strategy, rather than relying solely on standalone model compression\.

## 5Conclusion

This study provides important insights into the interaction between the algorithm architecture and hardware deployment by systematically analyzing vision\-language models with fewer than 3 billion parameters on heterogeneous edge platforms\. The technical implications of these results are as follows:First,the architecture has a more profound impact on the quantization stability than the model size\. Hypothesis H1 predicted severe performance degradation in smaller models\. However, consistent performance improvements were observed with MoE\-based architectures such as Qwen3\-VL\. These improvements include an increase of 56\.04 in the MME on the NX platform\. This suggests that sparse activation functions are effective in localizing and mitigating the propagation of INT4 quantization noise\. This provides a clear advantage over dense backbones, such as LLaVA\-OV, which exhibited a significant performance decline of 220\.02 points\.Second,we confirmed significant hardware\-software stack friction in SigLIP\-based encoders using BitsAndBytes\. Although the accuracy remains consistent, the vision encoding latency increases sharply \(by a factor of up to 4\.66\), suggesting that standard quantization kernels \(e\.g\., BitsAndBytes\) are not optimized for the specific computational patterns of SigLIP on Ampere\-based edge SoCs\. This underscores the need to transition from standalone model compression approaches to hardware\-software co\-design strategies\.

Limitations and Future Research\.This study presents a modular framework for PTQ sensitivity analysis, but it is limited to the BitsAndBytes framework\. Future research should expand the analysis to include new hardware\-native quantization formats, such as FP8, and should evaluate the impact of weight\-activation quantization \(W8A8\) to further reduce the dequantization overhead\. Furthermore, exploring automatic precision allocation through neural architecture search \(NAS\) could enhance the modality\-specific deployment guidelines presented in this study\.

## Software

In support of open science and reproducibility, we have released an end\-to\-end profiling tool\. This tool includes a modular quantization framework\. The code and related metadata are available in the following repository:[https://anonymous\.4open\.science/r/EdgeQuant\-VLMEvalKit/README\.md](https://anonymous.4open.science/r/EdgeQuant-VLMEvalKit/README.md)

## Acknowledgements

This work was supported by ETRI grant funded by Korean government\. \[26ZK1100, Honam region strategic industries\(optical convergence, energy, AI, etc\.\) technology advancement and commercialization\]

## References

- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.Advances in neural information processing systems35,pp\. 23716–23736\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- H\. Cai, J\. Li, M\. Hu, C\. Gan, and S\. Han \(2023\)Efficientvit: lightweight multi\-scale attention for high\-resolution dense prediction\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 17302–17313\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1),[3rd item](https://arxiv.org/html/2607.08029#S4.I2.i3.p1.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, C\. He, J\. Wang, F\. Zhao, and D\. Lin \(2024\)Sharegpt4v: improving large multi\-modal models with better captions\.InEuropean Conference on Computer Vision,pp\. 370–387\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1)\.
- X\. Chu, L\. Qiao, X\. Lin, S\. Xu, Y\. Yang, Y\. Hu, F\. Wei, X\. Zhang, B\. Zhang, X\. Wei,et al\.\(2023\)Mobilevlm: a fast, strong and open vision language assistant for mobile devices\.arXiv preprint arXiv:2312\.16886\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1),[§2](https://arxiv.org/html/2607.08029#S2.p1.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)Gpt3\. int8 \(\): 8\-bit matrix multiplication for transformers at scale\.Advances in neural information processing systems35,pp\. 30318–30332\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1),[§2](https://arxiv.org/html/2607.08029#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.08029#S3.SS1.p2.1)\.
- H\. Duan, J\. Yang, Y\. Qiao, X\. Fang, L\. Chen, Y\. Liu, X\. Dong, Y\. Zang, P\. Zhang, J\. Wang,et al\.\(2024\)Vlmevalkit: an open\-source toolkit for evaluating large multi\-modality models\.InProceedings of the 32nd ACM international conference on multimedia,pp\. 11198–11201\.Cited by:[§3\.5](https://arxiv.org/html/2607.08029#S3.SS5.p1.1)\.
- E\. Frantar and D\. Alistarh \(2023\)Qmoe: practical sub\-1\-bit compression of trillion\-parameter models\.arXiv preprint arXiv:2310\.16795\.Cited by:[2nd item](https://arxiv.org/html/2607.08029#S4.I1.i2.p1.1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1),[§2](https://arxiv.org/html/2607.08029#S2.p2.1)\.
- C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun,et al\.\(2023\)Mme: a comprehensive evaluation benchmark for multimodal large language models\.arXiv preprint arXiv:2306\.13394\.Cited by:[§3\.3](https://arxiv.org/html/2607.08029#S3.SS3.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1)\.
- S\. Kim, A\. Gholami, Z\. Yao, N\. Lee, P\. Wang, A\. Nrusimha, B\. Zhai, T\. Gao, M\. W\. Mahoney, and K\. Keutzer \(2022\)Integer\-only zero\-shot quantization for efficient speech recognition\.InICASSP 2022\-2022 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 4288–4292\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1)\.
- J\. Lee, S\. Park, J\. Kwon, J\. Oh, and Y\. Kwon \(2024\)Exploring the trade\-offs: quantization methods, task difficulty, and model size in large language models from edge to giant\.arXiv preprint arXiv:2409\.11055\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1)\.
- B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, P\. Zhang, Y\. Li, Z\. Liu,et al\.\(2024\)Llava\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- S\. Li, Y\. Hu, X\. Ning, X\. Liu, K\. Hong, X\. Jia, X\. Li, Y\. Yan, P\. Ran, G\. Dai,et al\.\(2025\)Mbq: modality\-balanced quantization for large vision\-language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 4167–4177\.Cited by:[§2](https://arxiv.org/html/2607.08029#S2.p3.1)\.
- B\. Lin, Z\. Tang, Y\. Ye, J\. Huang, J\. Zhang, Y\. Pang, P\. Jin, M\. Ning, J\. Luo, and L\. Yuan \(2026\)Moe\-llava: mixture of experts for large vision\-language models\.IEEE Transactions on Multimedia\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§2](https://arxiv.org/html/2607.08029#S2.p2.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1),[§1](https://arxiv.org/html/2607.08029#S1.p2.1),[§2](https://arxiv.org/html/2607.08029#S2.p1.1)\.
- T\. Lv, Y\. Huang, J\. Chen, Y\. Zhao, Y\. Jia, L\. Cui, S\. Ma, Y\. Chang, S\. Huang, W\. Wang, L\. Dong, W\. Luo, S\. Wu, G\. Wang, C\. Zhang, and F\. Wei \(2024\)Kosmos\-2\.5: a multimodal literate model\.arXiv preprint arXiv:2309\.11419\.Cited by:[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- NVIDIA \(2026\)NVIDIA jetson orin series\.Note:[https://www\.nvidia\.com/en\-us/autonomous\-machines/embedded\-systems/jetson\-orin/](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/)Accessed: 2026\-05\-06Cited by:[§3\.1](https://arxiv.org/html/2607.08029#S3.SS1.p2.1)\.
- OpenCompass \(2026\)Open vlm leaderboard\.Note:[https://huggingface\.co/spaces/opencompass/open\_vlm\_leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard)Accessed: 2026\-05\-06Cited by:[Appendix B](https://arxiv.org/html/2607.08029#A2.p1.1.1),[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- J\. Saad\-Falcon, A\. Narayan, H\. O\. Akengin, J\. Griffin, H\. Shandilya, A\. G\. Lafuente, M\. Goel, R\. Joseph, S\. Natarajan, E\. K\. Guha,et al\.\(2025\)Intelligence per watt: measuring intelligence efficiency of local ai\.arXiv preprint arXiv:2511\.07885\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p3.1),[3rd item](https://arxiv.org/html/2607.08029#S4.I6.i3.p1.2),[§4\.5](https://arxiv.org/html/2607.08029#S4.SS5.p2.7),[Table 12](https://arxiv.org/html/2607.08029#S4.T12.6.8.1.1)\.
- A\. Sharshar, L\. U\. Khan, W\. Ullah, and M\. Guizani \(2025\)Vision\-language models for edge networks: a comprehensive survey\.IEEE Internet of Things Journal\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1),[§2](https://arxiv.org/html/2607.08029#S2.p1.1),[§2](https://arxiv.org/html/2607.08029#S2.p3.1)\.
- A\. Steiner, A\. S\. Pinto, M\. Tschannen, D\. Keysers, X\. Wang, Y\. Bitton, A\. Gritsenko, M\. Minderer, A\. Sherbondy, S\. Long,et al\.\(2024\)Paligemma 2: a family of versatile vlms for transfer\.arXiv preprint arXiv:2412\.03555\.Cited by:[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- Q\. Sun, Y\. Cui, X\. Zhang, F\. Zhang, Q\. Yu, Y\. Wang, Y\. Rao, J\. Liu, T\. Huang, and X\. Wang \(2024\)Generative multimodal models are in\-context learners\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 14398–14409\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1)\.
- Z\. Wu, X\. Chen, Z\. Pan, X\. Liu, W\. Liu, D\. Dai, H\. Gao, Y\. Ma, C\. Wu, B\. Wang,et al\.\(2024\)Deepseek\-vl2: mixture\-of\-experts vision\-language models for advanced multimodal understanding\.arXiv preprint arXiv:2412\.10302\.Cited by:[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)Smoothquant: accurate and efficient post\-training quantization for large language models\.InInternational conference on machine learning,pp\. 38087–38099\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p2.1),[§2](https://arxiv.org/html/2607.08029#S2.p2.1)\.
- M\. Xu, W\. Yin, D\. Cai, R\. Yi, D\. Xu, Q\. Wang, B\. Wu, Y\. Zhao, C\. Yang, S\. Wang,et al\.\(2024\)A survey of resource\-efficient llm and multimodal foundation models\.arXiv preprint arXiv:2401\.08092\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.08029#S3.SS1.p1.1),[1st item](https://arxiv.org/html/2607.08029#S4.I3.i1.p1.1)\.
- Y\. Xue, Y\. Huang, J\. Shao, and J\. Zhang \(2025\)Vlmq: efficient post\-training quantization for large vision\-language models via hessian augmentation\.arXiv preprint arXiv:2508\.03351\.Cited by:[§2](https://arxiv.org/html/2607.08029#S2.p3.1)\.
- X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer \(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§3\.2](https://arxiv.org/html/2607.08029#S3.SS2.p1.1)\.
- B\. Zhou, Y\. Hu, X\. Weng, J\. Jia, J\. Luo, X\. Liu, J\. Wu, and L\. Huang \(2024\)Tinyllava: a framework of small\-scale large multimodal models\.arXiv preprint arXiv:2402\.14289\.Cited by:[§1](https://arxiv.org/html/2607.08029#S1.p1.1)\.

## Appendix AExperimental Setup

To support reproducibility, we report the generation hyperparameters used for each VLM wrapper\. Model weights are not fine\-tuned, and all evaluations share identical VLMEvalKit benchmark prompts, prompt templates, and the same exact\-matching answer\-extraction protocol, applied uniformly across every model and every quantization configuration\. All five models are evaluated with greedy decoding \(do\_sample=False\); the decoding configuration is held fixed across all quantization settings, so within\-model comparisons \(Δ\\DeltaMME\) are not confounded by decoding differences\.

Table 13:Generation configuration of each VLM wrapper\. All models use greedy decoding \(do\_sample=False\); sampling parameters \(temperature, top\-pp, top\-kk\) are inapplicable and omitted\. Settings are held fixed across all quantization configurations\.†Kosmos\-2\.5\-chat additionally appliesno\_repeat\_ngram\_size==3from its released configuration; given MME’s short Yes/No answers, this has a negligible effect on accuracy\.ModelDecodingMax New TokensQwen3\-VL\-2B\-InstructGreedy16384DeepSeek\-VL2\-TinyGreedy2048PaliGemma\-3B\-mix\-448Greedy512LLaVA\-OneVision\-0\.5BGreedy2048Kosmos\-2\.5\-chat†Greedy1024
## Appendix BComparison with OpenVLM Leaderboard

To verify our evaluation pipeline, we compare the MME Total Score produced by our measurement tool \(in baseline mode\) with the publicly reported scores on the Open VLM Leaderboard\(OpenCompass,[2026](https://arxiv.org/html/2607.08029#bib.bib22)\)\. Table[14](https://arxiv.org/html/2607.08029#A2.T14)demonstrates that, for the three models \(DeepSeek\-VL2\-Tiny, PaliGemma\-3B\-mix\-448, and LLaVA\-OneVision\-0\.5B\), our results closely match the leaderboard scores, supporting the robustness of our evaluation setup\. For the remaining two models, the leaderboard and our experiments use different model versions: the leaderboard reportsQwen2\-VL\-2BandKosmos\-2, whereas our experiments useQwen3\-VL\-2B\-InstructandKosmos\-2\.5\-chatrespectively\. The score differences for these two models therefore reflect the use of different model versions rather than discrepancies in our evaluation pipeline\.

Table 14:Comparison of MME Total Score between the Open VLM Leaderboard and our evaluation tool \(baseline mode\) on two hardware platforms \(AGX Orin and Jetson NX\)\.†The leaderboard reports Qwen2\-VL\-2B, while our experiments use Qwen3\-VL\-2B\-Instruct\.‡The leaderboard reports Kosmos\-2, while our experiments use Kosmos\-2\.5\-chat\.ModelLeaderboardMME ScoreOurs \(AGX\)BaselineOurs \(NX\)BaselineQwen2/3\-VL\-2B†1899\.11983\.322021\.78DeepSeek\-VL2\-Tiny1905\.51916\.711911\.12PaliGemma\-3B\-mix\-4481686\.11658\.101661\.01LLaVA\-OneVision\-0\.5B1392\.91359\.981361\.83Kosmos\-2\.5\-chat‡721\.1662\.05662\.30
## Appendix CMME Total Score across Quantization Configurations

Figure[1](https://arxiv.org/html/2607.08029#A4.F1)shows the MME total scores for all five quantization configurations \(C0–C5\) for each model, as measured on both hardware platforms\. Each configuration is evaluated overn≥3n\\geq 3independent runs, and the outcomes are visualized using bar charts, with each bar representing the mean score across runs\. The figure represents the full set of results that are summarized in Table[1](https://arxiv.org/html/2607.08029#S0.T1)\.

## Appendix DComprehensive Results across All Quantization Configurations

Table[15](https://arxiv.org/html/2607.08029#A4.T15)presents the full set of per\-model, per\-configuration measurements for all five sVLMs, evaluated under seven quantization configurations \(cfg0–cfg6\) on both the Jetson Orin NX and Jetson Orin AGX platforms\. For each configuration, we report the MME total score, peak VRAM usage \(GB\), end\-to\-end inference latency \(ms\), average power draw \(W\), and total energy consumption \(J\)\. Configuration definitions follow the notation introduced in Table[2](https://arxiv.org/html/2607.08029#S0.T2):cfg0is the FP16 baseline;cfg1applies LLM INT4 only;cfg2applies Projector INT8 only;cfg3combinescfg1andcfg2;cfg4applies Vision INT8 only;cfg5combinescfg1andcfg4; andcfg6applies all three components simultaneously \(Vision INT8 \+ Projector INT8 \+ LLM INT4\)\. This table constitutes the complete experimental record underlying the summarized results in Table[1](https://arxiv.org/html/2607.08029#S0.T1)\. Across models, LLM INT4 \(cfg1\) consistently reduces VRAM by approximately 40–50% but increases latency on both platforms, while Vision INT8 \(cfg4\) incurs a disproportionate latency penalty for SigLIP\-based encoders \(PaliGemma\-3B\-mix\-448 and DeepSeek\-VL2\-Tiny\)\. Energy consumption generally scales with latency, and accuracy degradation under quantization varies substantially across model architectures and configuration types\.

Table 15:Comprehensive Evaluation of Small VLMs across Quantization Configurations on Jetson Orin NX and AGX\.ModelConfigJetson Orin NXJetson Orin AGXMMEVRAMLat\.PowerEnergyMMEVRAMLat\.PowerEnergyScore\(GB\)\(ms\)\(W\)\(J\)Score\(GB\)\(ms\)\(W\)\(J\)Qwen3\-VL\-2B\-Instructcfg0 \(Base\)2021\.784\.06908\.1113\.3312\.441983\.324\.06533\.859\.074\.91cfg12077\.822\.131406\.2913\.5019\.042069\.052\.13744\.099\.767\.08cfg21989\.893\.95901\.8813\.2212\.322005\.343\.95537\.528\.604\.74cfg32046\.352\.041445\.8213\.4119\.422064\.462\.04801\.649\.197\.19cfg41973\.513\.771206\.7811\.4514\.272028\.413\.77738\.987\.535\.73cfg52029\.101\.851739\.6612\.2521\.432078\.731\.85984\.378\.168\.06cfg6 \(All\)2055\.091\.761761\.4612\.0521\.422055\.091\.75760\.8210\.658\.06DeepSeek\-VL2\-Tinycfg0 \(Base\)1911\.126\.824602\.5810\.8550\.051916\.716\.822945\.877\.6322\.57cfg11952\.543\.517883\.9611\.5591\.071962\.103\.516302\.316\.5341\.23cfg21916\.986\.814603\.1310\.8750\.141924\.606\.812958\.247\.6722\.78cfg31951\.553\.518015\.7711\.4591\.781950\.763\.516213\.226\.5740\.91cfg41896\.986\.447482\.609\.9574\.651909\.746\.444233\.307\.5632\.24cfg51953\.903\.1310865\.2910\.69116\.201953\.113\.137627\.906\.6350\.84cfg6 \(All\)1949\.643\.1210885\.1510\.67116\.261949\.643\.126098\.567\.5846\.67PaliGemma\-3B\-mix\-448cfg0 \(Base\)1661\.015\.852440\.2914\.5435\.851658\.105\.85835\.3015\.5313\.28cfg11598\.893\.183127\.3914\.5545\.831598\.743\.181033\.1315\.3316\.09cfg21663\.685\.862452\.4814\.5336\.011653\.475\.86825\.0315\.6713\.25cfg31600\.243\.183132\.6314\.5545\.881600\.373\.181053\.6715\.1516\.21cfg41668\.285\.493336\.2312\.9043\.401656\.975\.491203\.1213\.1616\.09cfg51589\.832\.814027\.4213\.1553\.261589\.192\.811420\.9113\.2018\.98cfg6 \(All\)1588\.562\.814003\.5713\.2153\.191588\.562\.801045\.4122\.3823\.78LLaVA\-OneVision\-0\.5Bcfg0 \(Base\)1361\.831\.891601\.3913\.3421\.561369\.732\.251522\.3614\.2521\.74cfg11141\.811\.451701\.1713\.4423\.041206\.151\.761558\.4814\.0522\.05cfg21360\.411\.881606\.9013\.3321\.631363\.282\.261531\.8014\.1921\.78cfg31137\.591\.451703\.5013\.4223\.051207\.481\.761561\.7914\.1422\.24cfg41369\.201\.572751\.9410\.9730\.511365\.301\.912084\.7512\.1625\.60cfg51149\.081\.132859\.8911\.0931\.991215\.451\.402130\.1012\.0626\.06cfg6 \(All\)1215\.941\.122883\.2411\.0432\.191215\.941\.401558\.0020\.4932\.39Kosmos\-2\.5\-chatcfg0 \(Base\)662\.306\.4516225\.2812\.11196\.82662\.056\.454181\.4414\.7962\.17cfg1651\.765\.5916495\.2812\.25202\.37649\.845\.594300\.8414\.7463\.68cfg2664\.276\.4516227\.3712\.12196\.98663\.306\.454196\.0714\.9062\.84cfg3653\.925\.5816488\.0912\.28202\.72656\.765\.584334\.1414\.9365\.03cfg4666\.485\.9817773\.3911\.95212\.64663\.275\.984748\.0614\.2567\.98cfg5656\.675\.1318046\.5812\.09218\.46650\.595\.134864\.0914\.1068\.89cfg6 \(All\)652\.515\.1218152\.6612\.07219\.40652\.515\.123721\.9724\.3891\.22
- Note:cfg0 \(FP16 base\), cfg1 \(LLM INT4\), cfg2 \(Proj INT8\), cfg3 \(cfg1\+cfg2\), cfg4 \(Vis INT8\), cfg5 \(cfg1\+cfg4\), and cfg6 \(Vis INT8 \+ Proj INT8 \+ LLM INT4\)\. Metrics default to: Latency \(ms\), Avg\. Power \(W\), and Energy \(J\)\.

![Refer to caption](https://arxiv.org/html/2607.08029v1/x1.png)Figure 1:MME total score across quantization configurations \(defined in the legend\) for five sVLMs on the AGX Orin and Orin NX platforms\. Bars show the mean forn≥3n\\geq 3samples; note that each subplot has its own independently scaled y\-axis, which does not necessarily start at zero\.
## Appendix EIsolated Vision\-Path Profiling for the SigLIP INT8 Latency Anomaly

To isolate the vision INT8 anomaly from the host LLM, projector, tokenizer, and decoding loop, we profile the vision path alone using NVTX\-annotated ranges with NVIDIA Nsight Systems \(50 iterations; Nsight Systems 2026\.3\.1\.157\)\. For PaliGemma2 and Kosmos\-2\.5 we measure the outerisolated\_vision\_encoderrange\. For DeepSeek\-VL2\-Tiny, whose standalone vision call does not reproduce its multimodal path, we instead profile the generation\-freeprepare\_inputs\_embedspath\. We note that this difference in measured scope means the DeepSeek\-VL2\-Tiny numbers are not strictly comparable to the encoder\-only ranges of the other two models; we report it separately and interpret it qualitatively rather than in direct head\-to\-head comparison\. As shown in Table[16](https://arxiv.org/html/2607.08029#A5.T16), the large slowdown is reproduced for SigLIP\-style encoders on both platforms \(PaliGemma2:4\.93×4\.93\\timeson AGX,4\.32×4\.32\\timeson NX; DeepSeek\-VL2\-Tiny:3\.10×3\.10\\times\), confirming that the anomaly is not an artifact of the decoding loop or a single AGX\-specific measurement\. The non\-SigLIP Kosmos\-2\.5 control, by contrast, slows by only1\.19×1\.19\\times\(AGX\) and1\.16×1\.16\\times\(NX\)\.

Nsight kernel summaries explain this contrast \(Table[17](https://arxiv.org/html/2607.08029#A5.T17)\)\. Under BitsAndBytes vision INT8, the PaliGemma2 vision path is restructured into repeated cycles of activation quantization, INT8 CUTLASS GEMM, dequantization, and synchronization\. Quantization and dequantization kernels alone account for22\.822\.8–25\.7%25\.7\\%of listed INT8 GPU kernel time \(NX and AGX, respectively\), accompanied by a7\.4×7\.4\\timesrise in kernel launches and a sharp increase in memory copies\. This points to a fragmented execution path rather than a faster GEMM substitution\. The non\-SigLIP Kosmos\-2\.5 control, already dominated by softmax, copy, and elementwise kernels, dilutes the same overhead to6\.06\.0–6\.5%6\.5\\%\. We restrict the kernel\-level breakdown in Table[17](https://arxiv.org/html/2607.08029#A5.T17)to these two models, as the latency results for DeepSeek\-VL2\-Tiny are reported only on AGX\.

These results lead us to revise H2 toward a narrower, deployment\-specific interpretation: the disproportionate slowdown arises from the interaction between SigLIP\-style encoders, the BitsAndBytes INT8 execution path, and the Jetson Orin Ampere architecture, rather than from any inherent inefficiency of SigLIP or of INT8 vision quantization in general\.

Table 16:Impact of BitsAndBytes vision INT8 quantization on isolated vision\-encoding latency \(ms\), with a non\-SigLIP Kosmos\-2\.5 control\. Ratio==Vision INT8 / FP16\.ModelPlat\.EncoderFP16 \(ms\)INT8 \(ms\)Ratio↑\\uparrowPaliGemma2AGXSigLIP\-So400m195\.99195\.99966\.09966\.09×\\times4\.93PaliGemma2NXSigLIP\-So400m313\.14313\.141353\.531353\.53×\\times4\.32DeepSeek\-VL2\-TinyAGXSigLIP\-style1145\.811145\.813555\.383555\.38×\\times3\.10Kosmos\-2\.5AGXPix2Struct\-style7058\.227058\.228373\.138373\.13×\\times1\.19Kosmos\-2\.5NXPix2Struct\-style10 701\.1110\\,701\.1112 428\.3712\\,428\.37×\\times1\.16Table 17:Kernel\-level breakdown of BitsAndBytes vision INT8 on the isolated vision path\. Quant/dequant share is the fraction of listed INT8 GPU kernel time spent inkInt8VectorQuant\(activation quantization\) andkdequant\_mm\_int32\_fp16\(dequantization\)\. Kernel\-launch ratio is INT8 relative to FP16\. DeepSeek\-VL2\-Tiny is omitted here because its latency is reported on AGX only \(Table 14\); for reference, its AGX quant/dequant share is 22\.5%\.ModelPlat\.EncoderQ/Dq shareLaunches↑\\uparrowPaliGemma2AGXSigLIP25\.7%7\.4×\\timesPaliGemma2NXSigLIP22\.8%7\.4×\\timesKosmos\-2\.5AGXPix2Struct\-style6\.5%2\.8×\\timesKosmos\-2\.5NXPix2Struct\-style6\.0%2\.8×\\times
## Appendix FNon\-BitsAndBytes AWQ Sanity Check

To examine whether the LLM INT4 latency trend reported in RQ3 \(Section[4\.3](https://arxiv.org/html/2607.08029#S4.SS3)\) is specific to the BitsAndBytes backend, we additionally evaluate a non\-BitsAndBytes AWQ baseline on Qwen3\-VL\-2B\-Instruct \(Table[18](https://arxiv.org/html/2607.08029#A6.T18)\)\. This experiment is intended as a sanity check rather than a full replacement for the component\-wise BitsAndBytes study\. We apply AWQ W4A16 exclusively to the text\-decoder Linear layers, excluding the vision encoder, the visual merger, and the lm\_head, with calibration on 512 held\-out Flickr30k image\-text samples\. For the AWQ sanity\-check run, we report median TPOT rather than relying only on mean end\-to\-end latency, because a small number of Qwen3\-VL samples produced unusually long responses that would otherwise inflate the mean end\-to\-end latency\. The AWQ baseline nearly preserves the MME accuracy of the FP16/BF16 baseline and, unlike BitsAndBytes INT4, does not reproduce the TPOT penalty, indicating that the TPOT increase under BitsAndBytes LLM INT4 is tied to the BitsAndBytes execution path in our runtime, rather than being an inherent property of 4\-bit LLM quantization\. However, AWQ does not reduce the peak inference VRAM in our current runtime\. This suggests that peak allocation during multimodal inference is likely dominated by runtime activations, visual features, the KV cache, and the compressed\-tensor execution workspace, rather than by the quantized weight storage alone\. We therefore report AWQ as a non\-BitsAndBytes sanity\-check backend, not as a substitute for the component\-wise BitsAndBytes analysis that forms the basis of this study\.

Table 18:Non\-BitsAndBytes AWQ sanity check on Qwen3\-VL\-2B\-Instruct, compared with the FP16/BF16 baseline and BitsAndBytes \(BnB\) LLM INT4\.ModelBackendQuant\. comp\.Calib\.MMEPeak VRAMMedian TPOTQwen3\-VL\-2BFP16/BF16NoneNone2021\.784\.06 GB111\.1 msQwen3\-VL\-2BBnB INT4LLMNone2077\.822\.13 GB173\.1 msQwen3\-VL\-2BAWQ W4A16LLMFlickr30k 5122020\.934\.08 GB107\.5 ms