Success running Qwen 3.8 27B EXL3 on RTX 3060 + 5060 Ti

Sharing my successful experience here. My setup: RTX 5060 Ti 16GB (x16 PCIe slot) RTX 3060 12GB (x4 PCIe slot) RAM: 32GB CPU: Ryzen 5 5600 OS: Linux So I have a total of 28GB. Average tk/s: 50, and this is using MTP so it varies a lot between 40 and 60 but it mostly floats at 50. Without MTP it goes down to 22 tk/s. Inference engine: exllamav3 via https://github.com/theroyallab/tabbyAPI Model: https://huggingface.co/turboderp/Qwen3.8-27B-exl3 Here's my config.yml file: model: # https://huggingface.co/turboderp/Qwen3.8-27B-exl3 model_name: Qwen3.8-27B-exl3-5.0bpw model_dir: /home/ccidral/ai/models max_seq_len: 102400 cache_size: 102400 cache_mode: Q8 gpu_split_auto: true tensor_parallel: true tensor_parallel_backend: native reasoning: true reasoning_start_token: "<think>" reasoning_end_token: "</think>" tool_format: qwen3_coder draft_model: draft_mode: mtp draft_cache_mode: Q8 network: host: 127.0.0.1 port: 5000 As you can see it uses tensor parallelism. Regarding VRAM usage, I still have 1.3GB left on the 3060 and 0.6GB on the 5060, which is something I'm still trying to fix by manually tweaking the GPU split but for some reason the inference engine goes nuts and hangs. I think it balances well but my desktop environment takes about 1GB of VRAM which I think is causing the unbalance (I'm considering changing my CPU for one with iGPU). I already used it to write a small rust TUI app using ratatui. It's pretty good. I'm quite satisfied but if you see any room for improvements or better battle-tested alternatives, please feel free to share. Also happy to answer any questions.