Qwen3.8-27B Hybrid IQ4_XS quantization for 16GB gang
Summary
This is a quantized version of the Qwen3.8-27B AI model using IQ4_XS quantization, optimized for 16GB RAM systems, with instructions for local deployment using various tools like llama.cpp and Ollama.
View Cached Full Text
Cached at: 08/16/26, 04:07 PM
jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller · Hugging Face
Source: https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller NotebooksGoogle ColabKaggleLocal AppsSettingsllama.cppHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
# Run inference directly in the terminal:
./llama-cli -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Use Docker
docker model run hf.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
LM StudioJanOllamaHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with Ollama:
ollama run hf.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Unsloth StudioHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller to start chatting
PiHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with Pi:
Start the llama.cpp server
# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Configure the model in Pi
# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
"providers": {
"llama-cpp": {
"baseUrl": "http://localhost:8080/v1",
"api": "openai-completions",
"apiKey": "none",
"models": [
{
"id": "jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS"
}
]
}
}
}
Run Pi
# Start Pi in your project directory:
pi
OpenClawnewHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with OpenClaw:
Start the llama.cpp server
# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Configure OpenClaw
# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
--auth-choice custom-api-key \
--custom-base-url http://127.0.0.1:8080/v1 \
--custom-model-id "jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS" \
--custom-provider-id llama-cpp \
--custom-compatibility openai \
--custom-text-input \
--accept-risk \
--skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Docker Model RunnerHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with Docker Model Runner:
docker model run hf.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
LemonadeHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/
lemonade pull jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Run and chat with the model
lemonade run user.Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller-IQ4_XS
List all available models
lemonade list
Hermes AgentHow to use jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Configure Hermes
# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller:IQ4_XS
Run Hermes
hermes
Similar Articles
Maxing out 64GB of RAM - Qwen3.5 122B A10B at UD-Q2_K_XL w/ MTP fully replaced Qwen3 Next 80B at UD-Q4_K_XL for me
A user compares running quantized Qwen3 Next 80B and Qwen3.5 122B on a 64GB RAM system, noting the trade-offs in speed, quality, and memory usage for local LLM inference.
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
UPDATE: Qwen-27B-IQ4_KS and Qwen-27B-IQ_KS_KT for ik_llama.cpp, especially for NVIDIA with 16GB VRAM
New GGUF quantizations of Qwen3.6-27B optimized for 16GB VRAM NVIDIA GPUs, including an experimental Trellis variant, with perplexity benchmarks.
Qwen 3.6 27B is the sweet spot for local development
Qwen 3.6 27B is praised as a powerful local AI model that outperforms expectations for general intelligence, suitable for practical tasks like code generation, and runs easily with llama.cpp.
Qwen 3.6 27B on 24GB VRAM setup: backend comparisons, quant choice and settings (llama.cpp, ik_llama.cpp, BeeLlama, vllm)
The article compares llama.cpp backends for running Qwen 3.6 27B on an RTX 3090 24GB, finding ik_llama.cpp with IQ4_KS quantization yields the best performance (1261 tok/s prefill, 72.9 tok/s decode).