Putting the question before the context took my local Qwen from 89% to 100% on a decision benchmark, and from ~400 ms to ~80 ms

Reddit r/LocalLLaMA News

Summary

Switching the order of question and context in prompts for local Qwen models improved accuracy from 89% to 100% and reduced latency from ~400 ms to ~80 ms on a decision benchmark.

Small, free finding. I use local Qwen models for typed decisions: a state plus a question with fixed allowed answers, and I read the probability of each answer from the logits of one forward pass instead of generating text. I used to build the prompt like you would for a human: situation first, then the question. Flipping it (question and allowed answers first, state last) changed both numbers at once on my benchmark of 38 short situations, Qwen3.6-35B-A3B at 4-bit on an M2 Max: accuracy 89% to 100% in English, 89% to 97% in German, median latency ~400 ms to ~80 ms. The speed part is boring: the question block is now an identical prefix across calls, so it gets computed once and cached. The accuracy part I can only guess at, probably the model reads the state differently when it already knows what it is looking for. Caveats: small set, short states. On long documents question-first still helped accuracy, but made those cases 6 to 8 times slower, because the document can no longer be cached across questions. Curious whether anyone sees the same effect with normal generation, e.g. JSON extraction or tool routing. Code and benchmark: https://github.com/Micha0827/snapjudge
Original Article

Similar Articles

Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context

Reddit r/LocalLLaMA

The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.