club-3090 / qengine / RedHatAI/Qwen3.8-27B-INT4
Summary
A user reports that running the quantized RedHatAI/Qwen3.8-27B-INT4 model with QEngine on dual RTX 3090s is significantly faster than their previous FP8 vLLM setup, sharing links to the config and model repo.
Similar Articles
club-3090 adds experimental FP8 support for Qwen3.6-27B!
club-3090 adds experimental FP8 support for Qwen3.6-27B, enabling dual RTX 3090 rigs to run the model with performance similar to unquantized BF16.
@leopardracer: https://x.com/leopardracer/status/2055341758523883631
A user shares their experience setting up a dual-GPU local AI lab with RTX 4080 Super and 5060 Ti, running Qwen 3.6 models via llama.cpp and llama-swap to reduce API costs and enable unrestricted experimentation.
Success running Qwen 3.8 27B EXL3 on RTX 3060 + 5060 Ti
A user successfully runs the Qwen 3.8 27B AI model on a mixed setup of RTX 3060 and 5060 Ti GPUs using tensor parallelism with exllamav3, achieving around 50 tokens per second with MTP enabled.
Wow! Qwen 3.6:35b-a3b on a 3090... pretty amazing.
A user shares impressive results running a quantized Qwen 3.6:35b-a3b model on a used RTX 3090, achieving 160 tokens per second output after fitting the model into VRAM, and demonstrates vision capabilities with a 75-second video processing time.
I pushed Qwen3.8-27B to 99 tps single request and 1150 tps with a batch request on a RTX 3090
The author optimized the Qwen3.8-27B model inference on an RTX 3090 GPU, achieving up to 99 tokens per second for single requests and 1150 tps with batch processing through various quantization and optimization techniques, and released the updated code on GitHub.