Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed

# Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed
Source: [https://arxiv.org/html/2608.11981](https://arxiv.org/html/2608.11981)
2ndKaijie ZhuAffiliation:Institute of Automation, CAS Beijing, China ORCID 0009\-0002\-6220\-14763rdHaobo XuAffiliation:Tsinghua University Beijing, China ORCID 0009\-0007\-8311\-79584thYichen WuAffiliation:Harvard Medical School Boston, U\.S\. ORCID 0000\-0003\-2859\-32855thZhichao LuAffiliation:City University of Hong Kong Hong Kong, China ORCID 0000\-0002\-4618\-35736thQingfu ZhangAffiliation:City University of Hong Kong Hong Kong, China ORCID 0000\-0003\-0786\-06717thZhenan SunAffiliation:Institute of Automation, CAS Beijing, China ORCID 0000\-0003\-4029\-9935

###### Abstract

Small Language Models \(SLMs\) have emerged as a more efficient alternative to traditional Large Language Models \(LLMs\), offering promising potential in resource\-constrained scenarios\. Existing approaches to building SLMs typically follow two paths: training compact models from scratch, or compressing larger pre\-trained models using methods such as pruning, quantization, or distillation\. As language models become increasingly integrated into real\-world applications, ensuring their trustworthiness has become a critical concern\. However, how to build trustworthy SLMs remains an underexplored question\. In this work, we present a comprehensive evaluation of SLM trustworthiness across multiple dimensions, including fairness, robustness, privacy, and ethics\. We first examine the effects of pruning and quantization, and find that quantization is significantly more effective in preserving trustworthiness compared to pruning\. More importantly, we demonstrate that compressing a reliable large model via quantization can produce SLMs with superior trustworthiness and adaptability compared to using small models trained from scratch\. Furthermore, knowledge distillation from trustworthy teacher models can further enhance the reliability of SLMs\. We hope our findings provide practical guidance and a foundation for future research into the development and deployment of trustworthy small language models\.

###### Index Terms:

Small language models, Pruning, Quantization, Knowledge distillation, Trustworthiness

## IIntroduction

Large Language Models \(LLMs\) have demonstrated remarkable performance across a wide range of natural language processing tasks\[[26](https://arxiv.org/html/2608.11981#bib.bib23),[27](https://arxiv.org/html/2608.11981#bib.bib16),[10](https://arxiv.org/html/2608.11981#bib.bib21)\]\. This success is largely attributed to their massive parameter scales—models with over 7 billion parameters have become a popular baseline\. However, deploying such large models incurs significant computational and memory costs, making them impractical for resource\-constrained environments such as edge devices\. As a result, Small Language Models \(SLMs\), typically with fewer than 1–2 billion parameters, have attracted increasing attention for their efficiency during inference\. There are two primary approaches to obtaining SLMs: \(1\) designing and training compact models from scratch using curated datasets and optimized architectures\[[28](https://arxiv.org/html/2608.11981#bib.bib19),[1](https://arxiv.org/html/2608.11981#bib.bib17)\], and \(2\) compressing larger LLMs via techniques like pruning, quantization, and distillation\[[9](https://arxiv.org/html/2608.11981#bib.bib12),[8](https://arxiv.org/html/2608.11981#bib.bib8),[22](https://arxiv.org/html/2608.11981#bib.bib13),[20](https://arxiv.org/html/2608.11981#bib.bib11),[18](https://arxiv.org/html/2608.11981#bib.bib3)\]\.

![Refer to caption](https://arxiv.org/html/2608.11981v1/teaser.png)Fig\. 1:Our evaluation framework for assessing the trustworthiness of SLMs, including state\-of\-the\-art pruning and quantization methods, a comparison between pre\-trained SLMs and compressed larger models, and the impact of distillation\. Our results suggest quantization as the preferred compression technique, while surpassing pre\-trained SLMs\. Distillation could provide additional improvements\.The growing deployment of LLMs in real\-world applications brings trustworthiness to the forefront, involving aspects such as safety, fairness, robustness, privacy, and ethical alignment\[[3](https://arxiv.org/html/2608.11981#bib.bib38)\]\. Numerous studies\[[15](https://arxiv.org/html/2608.11981#bib.bib26),[41](https://arxiv.org/html/2608.11981#bib.bib35),[5](https://arxiv.org/html/2608.11981#bib.bib36)\]have investigated trustworthiness in LLMs, particularly for models exceeding 7B parameters or proprietary LLMs\. However, how to ensure the trustworthiness of SLMs remains an open question\. Prior works\[[6](https://arxiv.org/html/2608.11981#bib.bib27),[14](https://arxiv.org/html/2608.11981#bib.bib29),[4](https://arxiv.org/html/2608.11981#bib.bib37)\]have explored the effect of compression on trustworthiness, but few directly compare pre\-trained SLMs with compressed larger models under a unified framework\.

In this work, we conduct a comprehensive analysis to understand how to build more trustworthy SLMs, using fairness, robustness, privacy, and ethics as key evaluation dimensions\. First, we evaluate the impact of pruning and quantization on model trustworthiness\. Our findings across different model families and sizes suggest that pruning can impair reliability, whereas quantization largely preserves the trustworthiness of the original full\-precision models\. Therefore, we advocate using quantization as a more reliable path toward building SLMs\. Second, we directly compare several pre\-trained SLMs \(under 1B parameters\) with quantized larger models\. The results consistently show that quantized LLMs outperform pre\-trained SLMs across all trustworthiness dimensions, indicating that compressing a reliable large model is more effective than training a small model from scratch\. Lastly, we explore the effect of knowledge distillation and find that distilling from a more trustworthy teacher can further enhance the reliability of SLMs\. Our main contributions are summarized as follows:

- •We conduct a thorough evaluation of compressed SLMs and recommend quantization over pruning as a more effective and reliable technique for preserving trustworthiness\.
- •We highlight a key insight: quantizing a larger, more trustworthy model yields more robust and flexible SLMs compared to directly using pre\-trained small models\.
- •We discover that knowledge distillation effectively improves SLM trustworthiness by leveraging the guidance of stronger teacher models\.

## IIRelated Work

### II\-APre\-Trained Small Language Models

Recently, small language models \(SLMs\) have shown strong potential in resource\-constrained scenarios compared to their large\-scale counterparts\. Several works focus on designing SLMs from scratch\. For example, MobiLlama\[[28](https://arxiv.org/html/2608.11981#bib.bib19)\]improves efficiency by reducing redundancy in transformer blocks, while SmolLM2\[[1](https://arxiv.org/html/2608.11981#bib.bib17)\]maximizes model performance through a multi\-stage rebalancing of diverse training data sources\. In addition, the Qwen 2\.5\[[27](https://arxiv.org/html/2608.11981#bib.bib16)\]and Llama 3\.2\[[10](https://arxiv.org/html/2608.11981#bib.bib21)\]series have released pre\-trained SLMs with parameter scales ranging from 0\.5B to 1B\. In this research, we explore the trustworthiness differences between such pre\-trained SLMs and compressed LLMs\.

### II\-BModel Compression

Network Pruningis a widely used compression technique that reduces model size by eliminating redundant or unimportant weights\[[11](https://arxiv.org/html/2608.11981#bib.bib9),[31](https://arxiv.org/html/2608.11981#bib.bib41),[32](https://arxiv.org/html/2608.11981#bib.bib1)\]\. For large language models, unstructured pruning sets individual unimportant weights to zero, while N:M semi\-structured pruning enforces a constraint that at least N out of every contiguous M weights must be zero\. This semi\-structured format is particularly advantageous on NVIDIA GPUs, as it enables acceleration of matrix multiply\-accumulate operations\. Among representative methods, SparseGPT\[[8](https://arxiv.org/html/2608.11981#bib.bib8)\]improves the efficiency of the traditional Optimal Brain Surgeon \(OBS\) algorithm by adjusting the remaining weights to minimize the loss change caused by pruning\. Wanda\[[25](https://arxiv.org/html/2608.11981#bib.bib10)\]proposes to leverage input activations as the importance indicator, while achieving performance comparable to SparseGPT\. Structured pruning\[[23](https://arxiv.org/html/2608.11981#bib.bib39)\]often causes notable performance degradation for LLMs without additional retraining; therefore, we do not include it in our exploration\.

Network Quantizationreduces the memory footprint of models by converting weight matrices into low\-bit representations\[[36](https://arxiv.org/html/2608.11981#bib.bib7),[21](https://arxiv.org/html/2608.11981#bib.bib40),[39](https://arxiv.org/html/2608.11981#bib.bib2),[40](https://arxiv.org/html/2608.11981#bib.bib5),[19](https://arxiv.org/html/2608.11981#bib.bib6),[38](https://arxiv.org/html/2608.11981#bib.bib30),[37](https://arxiv.org/html/2608.11981#bib.bib31)\]\. Post\-training quantization has gained popularity for LLMs due to its low computational cost and the absence of retraining requirements\. GPTQ\[[9](https://arxiv.org/html/2608.11981#bib.bib12)\]leverages second\-order information to perform error compensation during quantization\. AWQ\[[22](https://arxiv.org/html/2608.11981#bib.bib13)\]identifies and protects salient weights with the assistance of the activation matrix, preserving model accuracy\. DuQuant\[[20](https://arxiv.org/html/2608.11981#bib.bib11)\]introduces rotation and permutation transformations to enhance low\-bit weight\-activation quantization\.

Knowledge Distillation\(KD\) is a key technique for both compressing LLMs and improving their downstream performance\. The central idea is to transfer the knowledge encoded in a high\-capacity teacher model into a smaller student model, typically by training the student to mimic the teacher’s output distributions\[[13](https://arxiv.org/html/2608.11981#bib.bib14),[33](https://arxiv.org/html/2608.11981#bib.bib4),[16](https://arxiv.org/html/2608.11981#bib.bib28),[30](https://arxiv.org/html/2608.11981#bib.bib24),[17](https://arxiv.org/html/2608.11981#bib.bib20),[35](https://arxiv.org/html/2608.11981#bib.bib18)\]\. This paradigm enables compact models to retain much of the performance of their larger counterparts while significantly reducing computational overhead\. Moreover, KD provides an effective means to extract task\-specific or domain\-specific knowledge from proprietary or closed\-source models, serving as a practical alternative to direct access\[[34](https://arxiv.org/html/2608.11981#bib.bib15)\]\. In this work, we provide empirical insights that inform the trustworthy use of compressed models with these three techniques\.

ModelEthicsPrivacyRobustnessFairnessOverallGemma\-1\.1\-7B82\.19%60\.13%68\.90%44\.73%63\.99%Unstructured\-Sparsegpt81\.12%58\.40%63\.84%38\.43%60\.45%Unstructured\-Wanda80\.90%56\.86%64\.41%29\.82%58\.00%4:8\-Sparsegpt77\.36%59\.24%65\.20%32\.66%58\.61%4:8\-Wanda74\.87%52\.70%63\.57%54\.50%61\.41%2:4\-Sparsegpt72\.92%56\.37%61\.82%24\.43%53\.89%2:4\-wanda73\.08%55\.35%59\.27%36\.23%55\.98%Llama\-3\.1\-8B81\.29%49\.16%71\.62%36\.03%59\.52%Unstructure\-Sparsegpt80\.05%46\.08%60\.57%44\.08%57\.69%Unstructured\-Wanda77\.18%51\.56%61\.04%31\.76%55\.38%4:8\-Sparsegpt76\.74%53\.40%54\.87%44\.10%57\.28%4:8\-Wanda74\.98%48\.67%57\.29%46\.19%56\.78%2:4\-Sparsegpt66\.02%47\.75%56\.76%44\.65%53\.79%2:4\-Wanda59\.61%42\.22%47\.27%72\.04%55\.29%Qwen2\.5\-7B83\.78%41\.53%67\.05%28\.33%55\.17%Unstructured\-Sparsegpt84\.29%44\.22%58\.64%29\.51%54\.17%Unstructured\-Wanda84\.42%38\.38%61\.01%28\.79%53\.15%4:8\-Sparsegpt83\.99%40\.87%60\.27%30\.42%53\.89%4:8\-Wanda83\.86%38\.41%56\.35%36\.14%53\.69%2:4\-Sparsegpt82\.12%40\.36%53\.37%30\.65%51\.62%2:4\-Wanda82\.20%39\.48%56\.48%29\.89%52\.01%

TABLE I:Trustworthiness assessment for pruned LLMs\.
### II\-CModel Trustworthiness

TrustLLM\[[15](https://arxiv.org/html/2608.11981#bib.bib26)\]evaluates the reliability of language models across six key dimensions\. Truthfulness measures whether a model conveys accurate information\. Safety focuses on preventing harmful, unsafe, or unlawful outputs\. Fairness ensures that models do not introduce bias or discrimination across different demographics\. Robustness captures a model’s ability to maintain stable performance under diverse inputs or conditions\. Privacy emphasizes protecting individual autonomy and sensitive data\. Together, these dimensions provide a comprehensive framework for assessing the trustworthiness of LLMs\. For SLMs, prior research\[[6](https://arxiv.org/html/2608.11981#bib.bib27)\]has mainly examined the safety of quantized LLMs, while\[[14](https://arxiv.org/html/2608.11981#bib.bib29)\]focuses on the trustworthiness of compressed large models\. In contrast, our work investigates the trustworthiness of SLMs by directly comparing pre\-trained SLMs with compressed larger models\.

## IIIPreliminary

Quantization\.The generalbb\-bit uniform quantization𝒬b​\(⋅\)\\mathcal\{Q\}\_\{b\}\(\\cdot\)can be represented as:

𝐱^=𝒬b​\(𝐱\)=s⋅ΠΩ⁡\(b\)​\(𝐱/s\),\\hat\{\\mathbf\{x\}\}=\\mathcal\{Q\}\_\{b\}\(\\mathbf\{x\}\)=s\\cdot\\Pi\_\{\\Omega\(b\)\}\(\\mathbf\{x\}/s\),\(1\)wheressis the quantization step size, andΠΩ⁡\(b\)\\Pi\_\{\\Omega\(b\)\}is the projection function onto the set ofbb\-bit integersΩ⁡\(b\)=\{0,1,…,2b−1\}\\Omega\(b\)=\\\{0,1,\.\.\.,2^\{b\}\-1\\\}\.

Knowledge Distillation\.Given an inputxx, the teacher model produces logitsz\(T\)z^\{\(T\)\}and the student model producesz\(S\)z^\{\(S\)\}\. With temperatureTT, the softened probabilities are

pi\(T\)=exp⁡\(zi\(T\)/T\)∑jexp⁡\(zj\(T\)/T\),qi=exp⁡\(zi\(S\)/T\)∑jexp⁡\(zj\(S\)/T\)\.p\_\{i\}^\{\(T\)\}=\\frac\{\\exp\(z\_\{i\}^\{\(T\)\}/T\)\}\{\\sum\_\{j\}\\exp\(z\_\{j\}^\{\(T\)\}/T\)\},\\quad q\_\{i\}=\\frac\{\\exp\(z\_\{i\}^\{\(S\)\}/T\)\}\{\\sum\_\{j\}\\exp\(z\_\{j\}^\{\(S\)\}/T\)\}\.\(2\)The supervised distillation\[[13](https://arxiv.org/html/2608.11981#bib.bib14)\]minimizes the KL divergence:

ℒKD=∑ipi\(T\)​log⁡pi\(T\)qi\.\\mathcal\{L\}\_\{\\text\{KD\}\}=\\sum\_\{i\}p\_\{i\}^\{\(T\)\}\\log\\frac\{p\_\{i\}^\{\(T\)\}\}\{q\_\{i\}\}\.\(3\)

## IVSafety Evaluation for Small LLMs

In this section, we present a series of experiments to address our core research question:How can we obtain more trustworthy small language models?To systematically explore this, we further investigate the following sub\-questions:

- •RQ1: Are pruned large language models trustworthy?
- •RQ2: Are quantized large language models trustworthy?
- •RQ3: Are compressed LLMs more trustworthy than pre\-trained SLMs?
- •RQ4: Can distillation improve trustworthiness?

ModelEthicsPrivacyRobustnessFairnessOverallLlama\-3\.2\-1B73\.98%60\.21%59\.43%70\.81%66\.11%1B\-INT4\-AWQ69\.71%59\.71%54\.17%79\.11%65\.67%1B\-INT4\-GPTQ72\.32%57\.74%56\.19%68\.45%63\.67%Llama\-3\.2\-3B82\.88%60\.15%66\.42%43\.30%63\.19%3B\-INT4\-AWQ81\.27%56\.58%58\.80%56\.51%63\.29%3B\-INT4\-GPTQ81\.93%57\.53%67\.21%59\.52%66\.55%Qwen2\.5\-0\.5B65\.71%27\.56%62\.42%48\.99%51\.17%0\.5B\-INT4\-AWQ57\.56%27\.77%53\.34%48\.67%46\.83%0\.5B\-INT4\-GPTQ69\.26%27\.91%62\.17%52\.37%52\.93%Qwen2\.5\-1\.5B77\.52%40\.89%69\.38%71\.82%64\.90%1\.5B\-INT4\-AWQ73\.83%38\.97%68\.64%71\.89%63\.33%1\.5B\-INT4\-GPTQ78\.63%40\.38%67\.34%69\.07%63\.86%Qwen2\.5\-3B80\.13%40\.48%68\.79%29\.99%54\.85%3B\-INT4\-AWQ79\.00%40\.93%68\.49%32\.10%55\.13%3B\-INT4\-GPTQ80\.43%44\.98%68\.75%29\.82%56\.00%Qwen2\.5\-7B83\.78%41\.53%67\.05%28\.33%55\.17%7B\-INT4\-AWQ83\.79%39\.02%67\.68%28\.71%54\.80%7B\-INT4\-GPTQ83\.87%40\.15%68\.09%29\.18%55\.32%

TABLE II:Trustworthiness assessment for quantized LLMs\.### IV\-ASetup

Language Models and Compression Methods\.We conduct comprehensive evaluations across a range of instruction\-tuned language models\. For pre\-trained small language models, we select the following models: h2o\-danube3\-500m\-Chat\[[24](https://arxiv.org/html/2608.11981#bib.bib22)\], MobiLlama\-500m\-Chat\[[28](https://arxiv.org/html/2608.11981#bib.bib19)\], SmolLM2\-360M\-Instruct\[[1](https://arxiv.org/html/2608.11981#bib.bib17)\], and Qwen2\.5\-0\.5B\-Instruct\[[27](https://arxiv.org/html/2608.11981#bib.bib16)\]\. For compression methods, we adopt widely used techniques, including SparseGPT\[[8](https://arxiv.org/html/2608.11981#bib.bib8)\]and Wanda\[[25](https://arxiv.org/html/2608.11981#bib.bib10)\]for pruning, as well as GPTQ\[[9](https://arxiv.org/html/2608.11981#bib.bib12)\]and AWQ\[[22](https://arxiv.org/html/2608.11981#bib.bib13)\]for quantization\. All compression experiments are calibrated using the WikiText\-v2 dataset\. To obtain compressed SLMs, we apply these compression methods to commonly used LLMs, such as Gemma\[[26](https://arxiv.org/html/2608.11981#bib.bib23)\], Llama\[[10](https://arxiv.org/html/2608.11981#bib.bib21)\], and Qwen\[[27](https://arxiv.org/html/2608.11981#bib.bib16)\]\. For distillation, we conduct supervised knowledge distillation\[[13](https://arxiv.org/html/2608.11981#bib.bib14)\]on Qwen2\.5 models using Alpaca dataset\[[2](https://arxiv.org/html/2608.11981#bib.bib25)\]\.

Trustworthiness Measurement\.We adopt the TrustLLM\[[15](https://arxiv.org/html/2608.11981#bib.bib26)\]benchmark as the primary framework for assessing trustworthiness\. Specifically, we evaluate models across four dimensions: machine ethics, privacy, robustness, and fairness\. For ethics, we use the ETHICS\[[12](https://arxiv.org/html/2608.11981#bib.bib33)\]and Social\-Chem\-101\[[7](https://arxiv.org/html/2608.11981#bib.bib34)\]datasets to assess implicit ethics, and the MoralChoice dataset for explicit ethics evaluation\. For privacy, we adopt agreement tests on private information usage and privacy scenario tasks\. For robustness, we use AdvGLUE\[[29](https://arxiv.org/html/2608.11981#bib.bib32)\]and AdvInstruction to measure resistance to natural noise, and further examine performance on out\-of\-distribution \(OOD\) detection and generalization tasks\. For fairness, we evaluate from three perspectives: disparagement, representation bias, and preference bias in subjective choices\. We report the average accuracy on sub\-tasks under each dimension, and the overall average score is used as the indicator of model trustworthiness\.

### IV\-BPruning Influence

Pruning generally harms trustworthiness\.We conduct comprehensive evaluations of both unstructured and semi\-structured pruning on various large language models, particularly those at the 7B scale\. The results, summarized in Table[I](https://arxiv.org/html/2608.11981#S2.T1), pruned LLMs show an obvious degradation in trustworthiness after pruning\. For example, Gemma\-1\.1\-7B and LLaMA\-3\.1\-8B exhibit noticeable declines, with LLaMA\-3\.1\-8B dropping by nearly 10% in robustness\. These findings suggest that pruning substantially reduces model trustworthiness across multiple dimensions\.

Semi\-structured pruning increases degradation\.As shown in Table[I](https://arxiv.org/html/2608.11981#S2.T1), applying semi\-structured pruning further reduces trustworthiness compared to unstructured pruning\. In particular, the performance under 2:4 sparsity is worse than under 4:8 sparsity, indicating that more restrictive sparsity patterns introduce greater degradation\. This suggests that rigid pruning constraints can distort important representation subspaces, thereby amplifying the loss of trustworthiness across multiple dimensions\.Considering that unstructured pruning offers little inference speedup and semi\-structured pruning leads to low trustworthiness, we do not recommend using pruned models as trustworthy SLMs\.

ModelEthicsPrivacyRobustnessFairnessOverallQwen2\.5\-1\.5B77\.52%40\.89%69\.38%71\.82%64\.90%1\.5B\-INT8\-GPTQ77\.68%41\.08%69\.13%71\.85%64\.93%1\.5B\-INT4\-GPTQ78\.63%40\.38%67\.34%69\.07%63\.86%1\.5B\-INT3\-GPTQ77\.31%40\.72%67\.01%68\.23%63\.32%Qwen2\.5\-3B80\.13%40\.48%68\.79%29\.99%54\.85%3B\-INT8\-GPTQ80\.26%40\.35%68\.70%29\.54%54\.71%3B\-INT4\-GPTQ80\.43%44\.98%68\.75%29\.82%56\.00%3B\-INT3\-GPTQ80\.02%41\.32%68\.65%29\.13%54\.78%

TABLE III:Trustworthiness assessment under different quantized bits for Qwen2\.5 models\.ModelEthicsPrivacyRobustnessFairnessOverallh2o\-danube3\-500m\-Chat61\.92%29\.02%56\.85%43\.27%47\.76%MobiLlama\-500m\-Chat55\.79%32\.33%48\.02%45\.69%45\.46%SmolLM2\-360M\-Instruct51\.47%48\.59%60\.16%47\.45%51\.92%Qwen2\.5\-0\.5B\-Instruct65\.71%27\.56%62\.42%48\.99%51\.17%Qwen2\.5\-1\.5B\-Instruct77\.52%40\.89%69\.38%71\.82%64\.90%Qwen2\.5\-1\.5B\-INT4\-AWQ73\.83%38\.97%68\.64%71\.89%63\.33%Qwen2\.5\-1\.5B\-INT4\-GPTQ78\.63%40\.38%67\.34%69\.07%63\.86%Qwen2\.5\-1\.5B\-INT8\-GPTQ77\.68%41\.08%69\.13%71\.85%64\.93%

TABLE IV:Trustworthiness comparison among pre\-trained SLMs and compressed LLMs\.ModelEthicsPrivacyRobustnessFairnessOverallQwen2\.5\-7B83\.78%41\.53%67\.05%28\.33%55\.17%Qwen2\.5\-3B80\.13%40\.48%68\.79%29\.99%54\.85%KD\-3B82\.12%41\.36%69\.88%32\.45%56\.45%

TABLE V:Exploration of distillation impact\.
### IV\-CQuantization Influence

Quantization has relatively minor influence, especially for larger models\.As shown in Table[II](https://arxiv.org/html/2608.11981#S4.T2), quantization maintains the trustworthiness of Qwen2\.5\-7B with minimal degradation\. We further extend our evaluation to 4\-bit quantization on smaller Qwen models and the LLaMA\-3\.2 series\. According to Table[II](https://arxiv.org/html/2608.11981#S4.T2), the quantized models generally exhibit comparable performance to their full\-precision counterparts, especially in the case of larger models\. For instance, the Qwen2\.5\-0\.5B model experiences a moderate 4% drop under AWQ quantization, whereas GPTQ yields more stable results, even improving the accuracy\. For models larger than 1\.5B, the decrease in trustworthiness remains minimal \(less than 2%\)\.These findings highlight quantization as a viable strategy for building trustworthy SLMs, with larger\-scale models demonstrating greater resilience to low\-bit compression\.

GPTQ offers more reliable trustworthiness than AWQ\.As presented in Table[II](https://arxiv.org/html/2608.11981#S4.T2), models quantized using GPTQ typically achieve higher accuracy in trustworthiness evaluations compared to those quantized with AWQ\. This trend holds across most of the settings in our experiments\. Interestingly, we observe that for certain models, such as LLaMA\-3\.2\-3B and Qwen2\.5\-3B, GPTQ quantized variants even outperform their FP16 counterparts\. These results further demonstrate that quantized models, particularly when using GPTQ, can not only reduce memory consumption but also maintain or even enhance trustworthiness, highlighting the potential of low\-bit quantization in building efficient and reliable SLMs\.

Quantization showcases robustness towards different compression ratios\.We further analyze the effect of different quantization levels on trustworthiness\. As shown in Table[III](https://arxiv.org/html/2608.11981#S4.T3), GPTQ exhibits stable performance across 3–8 bits on Qwen2\.5, with no systematic degradation as the bit\-width decreases\. This trend suggests that low\-bit quantization largely preserves the reliability\-related behaviors learned by the full\-precision model, and the trustworthiness scores are relatively insensitive to the compression ratio within this range\. Overall, these results support our conclusion that quantization \(especially GPTQ\) is a robust and practical strategy for obtaining trustworthy SLMs under varying efficiency constraints\.

### IV\-DDirectly Using SLMs or Compressing Larger LLMs?

Based on our analysis of pruning and quantization effects on LLMs, we focus on quantization as the primary technique to compare compressed LLMs against directly pre\-trained SLMs\. In this subsection, we evaluate several pre\-trained SLMs and quantized versions of larger models\. Specifically, we examine four pre\-trained SLMs with fewer than 1B parameters to provide a comprehensive perspective\.

Performance Analysis\.As shown in Table[IV](https://arxiv.org/html/2608.11981#S4.T4), these pre\-trained SLMs generally achieve lower trustworthiness scores, typically around 50In contrast, we apply 4\-bit quantization to Qwen2\.5\-1\.5B, which can reduce weight memory by approximately 4× and potentially deliver up to 3× speedup with optimized kernels\[[22](https://arxiv.org/html/2608.11981#bib.bib13)\]\. This makes the quantized 1\.5B model comparable in practical deployment efficiency to smaller SLMs \(e\.g\., 0\.5B models\), while retaining stronger representations\. Consequently, the quantized Qwen2\.5\-1\.5B model achieves substantially higher trustworthiness \(around 63%\) than the pre\-trained SLMs\. This gain is driven by the minimal degradation introduced by quantization and the stronger base capability of the 1\.5B model compared to smaller counterparts \(e\.g\., Qwen2\.5\-0\.5B\)\.

Memory and Latency Discussion\.On RTX\-4090 with batch=1 decoding, generation latency is often memory\-bound\. Although the 1\.5B model has ~3× more parameters, INT4 quantization reduces weight memory traffic by ~4×, which can largely offset the increased parameter count in practice\. As a result, Qwen2\.5\-1\.5B\-INT4\-GPTQ typically exhibits comparable ms/token to Qwen2\.5\-0\.5B FP16 \(often within ~1\.2–1\.5×\), rather than being strictly slower, while achieving markedly better trustworthiness\. Moreover, quantization offers a flexible knob to produce SLMs of different sizes and efficiency levels by adjusting bit\-width, enabling smooth trade\-offs between memory, speed, and trustworthiness\.

From this analysis, we derive an important insight: instead of relying solely on small models trained from scratch,leveraging quantization on larger pre\-trained LLMs can yield compact, efficient, and more trustworthy SLMs\.

### IV\-EDistillation Enhancement

We explore whether knowledge distillation can improve SLM trustworthiness\. Specifically, Qwen2\.5\-3B is distilled from Qwen2\.5\-7B using the Alpaca dataset, with results listed in Table[V](https://arxiv.org/html/2608.11981#S4.T5)\. The distilled model shows improvements across all four trustworthiness categories, demonstrating the benefit of transferring knowledge from a more reliable model\. We expect that distilling from larger and more trustworthy teacher models could yield even greater improvements in the reliability of small language models\.

## VConclusion

This work explores effective strategies for developing trustworthy Small Language Models \(SLMs\) through a systematic empirical study\. Our findings reveal three important observations: \(1\) Quantization offers a more reliable means than pruning for preserving model trustworthiness; \(2\) Compressing a well\-aligned large model via quantization results in more robust and adaptable SLMs than directly using pre\-trained small models; \(3\) Knowledge distillation serves as a complementary approach to further improve the reliability of compact models\. Overall, our study sheds light on practical and scalable pathways to build efficient SLMs without compromising their trustworthiness, and lays the foundation for future efforts in this direction\.

## References

- \[1\]L\. B\. Allal, A\. Lozhkov, E\. Bakouch,et al\.\(2025\)SmolLM2: when smol goes big–data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-A](https://arxiv.org/html/2608.11981#S2.SS1.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[2\]R\. Bommasani\(2021\)On the opportunities and risks of foundation models\.arXiv preprint arXiv:2108\.07258\.Cited by:[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[3\]Y\. Chang, X\. Wang, J\. Wang, Y\. Wu, L\. Yang, K\. Zhu, H\. Chen, X\. Yi, C\. Wang, Y\. Wang,et al\.\(2024\)A survey on evaluation of large language models\.ACM transactions on intelligent systems and technology15\(3\),pp\. 1–45\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1)\.
- \[4\]K\. Chen, J\. Zhang, J\. Hu, Y\. Wang, J\. Lou, Z\. Feng, and M\. SongAssessing safety risks and quantization\-aware safety patching for quantized large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1)\.
- \[5\]B\. C\. Das, M\. H\. Amini, and Y\. Wu\(2025\)Security and privacy challenges of large language models: a survey\.ACM Computing Surveys57\(6\),pp\. 1–39\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1)\.
- \[6\]K\. Egashira, M\. Vero, R\. Staab, J\. He, and M\. Vechev\(2024\)Exploiting llm quantization\.Advances in Neural Information Processing Systems37,pp\. 41709–41732\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1),[§II\-C](https://arxiv.org/html/2608.11981#S2.SS3.p1.1)\.
- \[7\]M\. Forbes, J\. D\. Hwang, V\. Shwartz, M\. Sap, and Y\. Choi\(2020\)Social chemistry 101: learning to reason about social and moral norms\.arXiv preprint arXiv:2011\.00620\.Cited by:[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p2.1)\.
- \[8\]E\. Frantar and D\. Alistarh\(2023\)Sparsegpt: massive language models can be accurately pruned in one\-shot\.InInternational conference on machine learning,pp\. 10323–10337\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[9\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[10\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-A](https://arxiv.org/html/2608.11981#S2.SS1.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[11\]S\. Han, H\. Mao, and W\. J\. Dally\(2015\)Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding\.arXiv preprint arXiv:1510\.00149\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p1.1)\.
- \[12\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Critch, J\. Li, D\. Song, and J\. Steinhardt\(2020\)Aligning ai with shared human values\.arXiv preprint arXiv:2008\.02275\.Cited by:[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p2.1)\.
- \[13\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1),[§III](https://arxiv.org/html/2608.11981#S3.p2.2),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[14\]J\. Hong, J\. Duan, C\. Zhang, Z\. Li, C\. Xie, K\. Lieberman, J\. Diffenderfer, B\. Bartoldson, A\. Jaiswal, K\. Xu,et al\.\(2024\)Decoding compressed trust: scrutinizing the trustworthiness of efficient llms under compression\.arXiv preprint arXiv:2403\.15447\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1),[§II\-C](https://arxiv.org/html/2608.11981#S2.SS3.p1.1)\.
- \[15\]Y\. Huang, L\. Sun, H\. Wang, S\. Wu, Q\. Zhang, Y\. Li, C\. Gao, Y\. Huang, W\. Lyu, Y\. Zhang,et al\.\(2024\)Trustllm: trustworthiness in large language models\.arXiv preprint arXiv:2401\.05561\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1),[§II\-C](https://arxiv.org/html/2608.11981#S2.SS3.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p2.1)\.
- \[16\]Y\. Jiang, H\. Lin, Y\. Bai, B\. Peng, Z\. Liu, Y\. Lyu, Y\. Yang, and J\. Dong\(2025\)Image\-level memorization detection via inversion\-based inference perturbation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 47960–47979\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1)\.
- \[17\]Z\. Li, H\. Lin, Y\. Xiao, Z\. Li, X\. Song, Z\. Zheng, Y\. He, H\. Yao, K\. Ding, C\. Yu,et al\.\(2026\)IV\-cot: implicit visual chain\-of\-thought for structure\-aware text\-to\-image generation\.arXiv preprint arXiv:2606\.24849\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1)\.
- \[18\]H\. Lin, X\. Jia, S\. Liu, S\. Xia, W\. Huang, H\. Xu, J\. Li, Y\. Xiao, X\. Xing, Z\. Guo,et al\.\(2026\)Efficient diffusion language models: a comprehensive survey\.Authorea Preprints\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1)\.
- \[19\]H\. Lin, X\. Jia, H\. Xu, B\. Yao, X\. Guo, Y\. Wu, Z\. Lu, Y\. Wei, Q\. Zhang, and Z\. Sun\(2026\)DuQuant\+\+: fine\-grained rotation enhances microscaling fp4 quantization\.arXiv preprint arXiv:2604\.17789\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[20\]H\. Lin, H\. Xu, Y\. Wu, J\. Cui, Y\. Zhang, L\. Mou, L\. Song, Z\. Sun, and Y\. Wei\(2024\)Duquant: distributing outliers via dual transformation makes stronger quantized llms\.Advances in Neural Information Processing Systems37,pp\. 87766–87800\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[21\]H\. Lin, H\. Xu, Y\. Wu, Z\. Guo, R\. Zhang, Z\. Lu, Y\. Wei, Q\. Zhang, and Z\. Sun\(2025\)Quantization meets dllms: a systematic study of post\-training quantization for diffusion llms\.arXiv preprint arXiv:2508\.14896\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[22\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, X\. Dang, and S\. Han\(2023\)AWQ: activation\-aware weight quantization for llm compression and acceleration\.arXiv preprint arXiv:2306\.00978\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1),[§IV\-D](https://arxiv.org/html/2608.11981#S4.SS4.p2.1)\.
- \[23\]X\. Ma, G\. Fang, and X\. Wang\(2023\)Llm\-pruner: on the structural pruning of large language models\.Advances in neural information processing systems36,pp\. 21702–21720\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p1.1)\.
- \[24\]P\. Pfeiffer, P\. Singer, Y\. Babakhin, G\. Fodor, N\. Dhankhar, and S\. S\. Ambati\(2024\)H2o\-danube3 technical report\.arXiv preprint arXiv:2407\.09276\.Cited by:[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[25\]M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter\(2023\)A simple and effective pruning approach for large language models\.arXiv preprint arXiv:2306\.11695\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[26\]G\. Team, T\. Mesnard, C\. Hardin, R\. Dadashi, S\. Bhupatiraju, J\. Love,et al\.\(2024\)Gemma: open models based on gemini research and technology\.arXiv preprint arXiv:2403\.08295\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[27\]Q\. Team\(2024\)Qwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-A](https://arxiv.org/html/2608.11981#S2.SS1.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[28\]O\. Thawakar, A\. Vayani, S\. Khan, H\. Cholakal, R\. M\. Anwer, M\. Felsberg, T\. Baldwin, E\. P\. Xing, and F\. S\. Khan\(2024\)Mobillama: towards accurate and lightweight fully transparent gpt\.arXiv preprint arXiv:2402\.16840\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p1.1),[§II\-A](https://arxiv.org/html/2608.11981#S2.SS1.p1.1),[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p1.1)\.
- \[29\]B\. Wang, C\. Xu, S\. Wang, Z\. Gan, Y\. Cheng, J\. Gao, A\. H\. Awadallah, and B\. Li\(2021\)Adversarial glue: a multi\-task benchmark for robustness evaluation of language models\.arXiv preprint arXiv:2111\.02840\.Cited by:[§IV\-A](https://arxiv.org/html/2608.11981#S4.SS1.p2.1)\.
- \[30\]S\. Xia, H\. Lin, Y\. Wu, Y\. Zhou, Z\. Li, Z\. Wan, X\. Xing, Y\. Zheng, X\. Li, C\. Shan,et al\.\(2025\)Medrek: retrieval\-based editing for medical llms with key\-aware prompts\.arXiv preprint arXiv:2510\.13500\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1)\.
- \[31\]X\. Xing, Z\. Liu, S\. Xiao, B\. Gao, Y\. Liang, W\. Zhang, H\. Lin, G\. Li, and J\. Zhang\(2025\)EfficientLLM: scalable pruning\-aware pretraining for architecture\-agnostic edge language models\.arXiv preprint arXiv:2502\.06663\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p1.1)\.
- \[32\]H\. Xu, S\. Chen, R\. Qiu, Y\. Yan, C\. Luo, M\. Cheng, J\. He, and H\. Tong\(2026\)Prune as you generate: online rollout pruning for faster and better rlvr\.arXiv preprint arXiv:2603\.24840\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p1.1)\.
- \[33\]H\. Xu, Y\. Yan, D\. Wang, Z\. Xu, Z\. Zeng, T\. F\. Abdelzaher, J\. Han, and H\. Tong\(2024\)Slog: an inductive spectral graph neural network beyond polynomial filter\.InForty\-first International Conference on Machine Learning,Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1)\.
- \[34\]X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, C\. Xu, D\. Tao, and T\. Zhou\(2024\)A survey on knowledge distillation of large language models\.arXiv preprint arXiv:2402\.13116\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1)\.
- \[35\]J\. Yang, Y\. Wu, W\. Li, H\. Lin, R\. Wang, X\. Feng, and X\. Jia\(2026\)MAC\-splat: multi\-attribute consistency for high\-fidelity sparse\-view reconstruction\.arXiv preprint arXiv:2607\.10792\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p3.1)\.
- \[36\]L\. Yang, H\. Gong, H\. Lin, Y\. Wu, Z\. Sun, and Q\. Gu\(2024\)Dopq\-vit: towards distribution\-friendly and outlier\-aware post\-training quantization for vision transformers\.arXiv preprint arXiv:2408\.03291\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[37\]L\. Yang, H\. Lin, Y\. Wu, C\. Shan, Z\. Sun, and Q\. Gu\(2026\)Reshape and rotate: adaptive weight reshaping and fine\-grained rotation for ultra\-low\-bit diffusion transformers quantization\.Neurocomputing,pp\. 133830\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[38\]L\. Yang, H\. Lin, Y\. Wu, Z\. Sun, and Q\. Gu\(2026\)DapQ\-dit: distribution\-aware post\-training quantization for efficient generative tasks in diffusion transformers\.InProceedings of the 2026 International Conference on Multimedia Retrieval,pp\. 2371–2380\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[39\]L\. Yang, H\. Lin, T\. Zhao, Y\. Wu, H\. Zhu, R\. Xie, Z\. Sun, Y\. Wang, and Q\. Gu\(2025\)LRQ\-dit: log\-rotation post\-training quantization of diffusion transformers for image and video generation\.arXiv preprint arXiv:2508\.03485\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[40\]J\. Zhang, Y\. Hsieh, Z\. Wang, H\. Lin, X\. Wang, Z\. Wang, Y\. Lei, and M\. Zhang\(2026\)QuantVLA: scale\-calibrated post\-training quantization for vision\-language\-action models\.arXiv preprint arXiv:2602\.20309\.Cited by:[§II\-B](https://arxiv.org/html/2608.11981#S2.SS2.p2.1)\.
- \[41\]K\. Zhu, J\. Wang, J\. Zhou, Z\. Wang, W\. Ye, Y\. Zhang, N\. Zhenqiang Gong,et al\.\(2023\)Promptbench: towards evaluating the robustness of large language models on adversarial prompts\.arXiv e\-prints,pp\. arXiv–2306\.Cited by:[§I](https://arxiv.org/html/2608.11981#S1.p2.1)\.