Little Brains, Big Feats: Exploring Compact Language Models

Hugging Face Daily Papers Papers
compact-language-models retrieval-augmented-generation russian-language quantization cpu-inference qwen edge-deployment

Summary

This paper benchmarks 17 compact language models (1B-8B parameters) as generators in Russian-language RAG systems under CPU-only inference, finding that Qwen-family models offer strong quality-latency tradeoffs for private, GPU-free deployment.

While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention. In this study, we investigate how smaller language models perform during the generation stage within a Retrieval-Augmented Generation (RAG) system. To benchmark these models effectively, we utilised both open-source and proprietary datasets covering diverse subject areas and question types. Our findings demonstrate that a RAG system with small language models can be executed directly on-device without requiring any GPU hardware within a reasonable time. The experimental code and links to the supplementary materials can be accessed through the GitHub repository: https://github.com/SibNN/SLM-RAG-EVAL.
Original Article
View Cached Full Text

Cached at: 07/01/26, 07:41 AM

Paper page - Little Brains, Big Feats: Exploring Compact Language Models

Source: https://huggingface.co/papers/2606.30062 overview Can small language models be strong enough for practical RAG generation without GPUs?

We benchmark 17 compact language models from 1B to 8B parameters as generators in Russian-language Retrieval-Augmented Generation systems. All candidate models were evaluated as local GGUF variants, including Q4_K_M and Q5_K_M quantized models, under CPU-only inference constraints.

The evaluation uses a 500-sample benchmark built from five Russian-language QA datasets, including open-source and proprietary domain-specific data. Responses are assessed with a multi-judge LLM-as-a-Judge setup across correctness, answer relevance, faithfulness, context relevance, and latency.

A clear pattern emerges: Qwen-family models dominate the top-performing SLM tier in this setting. Qwen3-8B-Q4_K_M achieved the strongest overall SLM quality, reaching 0.72 correctness and 0.83 faithfulness, approaching the GPT-5-mini baseline on correctness. At the same time, Qwen3-4B-Instruct-2507-Q5_K_M provided the best practical quality–latency trade-off, with 0.71 correctness, 0.89 answer relevance, 0.80 faithfulness, and substantially lower CPU latency than the 8B model. Qwen2.5-7B-Instruct-Q4_K_M was also a strong candidate, showing high answer relevance and faithfulness with moderate latency.

Our findings suggest that carefully selected quantized SLMs, especially from the Qwen family, can be competitive RAG generators while enabling local, private, and GPU-free deployment. The work is especially relevant for on-device AI, privacy-sensitive applications, edge deployment, and production RAG systems with limited compute budgets.

Accepted to ECML PKDD 2026 Applied Data Science Track. Author’s preprint version.

Similar Articles

Diffusion Language Models: An Experimental Analysis

arXiv cs.AI

A systematic experimental analysis evaluating eight state-of-the-art Diffusion Language Models across multiple benchmarks, analyzing trade-offs between generation quality and computational efficiency.

Exploring Lightweight Large Language Models for Court View Generation

arXiv cs.CL

This paper systematically explores the capabilities of lightweight (<2B) large language models for criminal court view generation, investigating trade-offs between model architecture, size, and impact on charge prediction. The authors also introduce CVGEvalKit, an evaluation framework with three public datasets.