I pushed Qwen3.8-27B to 99 tps single request and 1150 tps with a batch request on a RTX 3090

I'm back. Yesterday I released the first version of hyper-optimized Qwen3.8-27B inference engine for a RTX 3090, reaching 82 tps on single request and 672 peak. Over the last 24 hours I've been exploring how to improve it further (without a quality degredation). Yesterday we had these 3 optimizations: - fp8 kv cache - lm_head int8 - embed_tokens in8 Now now I added: - fp16 recurrent state - int8 activations, MLP - int8 activations on everything - draft_sample_method=probablistic (which samples from the MTP distribution rather than taking the argmax) - does not work for greedy sampling. For single user mode, the user can enable ctx=fast, which uses FlashAttention, bf16 kv, 64k context and gives the user 99 tps, but at the tradeoff of less context. Also the prefill is now up to 1812 tps at 1k and 1000 tps at 100k context - an increase of about 50% to 25%. I've updated the repo will everything you need to start flying: https://github.com/syv-ai/qwen38-27b-rtx3090 Right now I am working on getting KVarN working, so we can reduce the memory footprint of the context, so we can fit the full 262k instead of 195k.