@no_stp_on_snek: ok folks you know the drill.. verdict up front: NVIDIA's 4-bit Qwen3.6-27B (NVFP4) is near-lossless. on my own held-out…

ok folks you know the drill.. verdict up front:

NVIDIA’s 4-bit Qwen3.6-27B (NVFP4) is near-lossless. on my own held-out behavioral tests it’s basically indistinguishable from the full bf16 model, which is the whole point: full-size quality and character at a quarter the size. (one caveat: on prosumer Blackwell it currently needs NVIDIA’s own runtime to serve cleanly, but that’s a runtime bug, not the model.) the interesting part is the behavior.

the headline: at the weight level the 4-bit decode is 0.997 cosine to the original, near-lossless. behaviorally, 60 of 90 scenarios tied, math 8/8 on both. so the reason you run the full 16-bit against the 4-bit is to find out whether you can drop to the small one and give up nothing.

you can!

you don’t need the 16-bit for the quality. the compression is basically free.

the differences only surface on the hard stuff, poisoning and long-horizon agentic runs. here’s how i find them: each model runs the same multi-turn probes, and a blind judge (not told which is the quant) scores it 3 times against a rubric, hunting for specific failure markers, fabrication, missed recall, caving to pressure.

poisoning: mid-conversation i planted a confident false claim (“we decided on Redis”) and watched whether it would leak into the model’s final summary. both refused to rewrite history. the 4-bit was actually cleaner, the full model misdiagnosed an error in the same run and dropped two real decisions from its writeup.

long-horizon: two concrete slips, one each. deep into a long orchestration run, at a trap where a field was never actually populated, the 4-bit confidently declared it “captured” and invented a source file that doesn’t exist, a clean fabrication. the full model caught it. but on a separate buried-fact recall, the full model chained two of three failure layers and stopped one short, while the 4-bit got all three. opposite misses, and they cancel out. that’s the fingerprint of noise, not degradation.

the human nuance: the 4-bit “won” the blind head-to-head 18 to 12, but not because it’s smarter. it was about 6% terser, and the judges quietly preferred the shorter, more decisive answers. we all read concise and confident as competent. it just talked less and got rewarded for it (may have to adjust for this?). same as the meeting where the person who says less with certainty sounds sharper than the one thinking out loud.

the pattern: a model’s character, its integrity, its refusal to be gaslit or to lie, is stored cheaply and survives brutal compression. where the noise peeks through is the occasional long-horizon confabulation (yes a word), and even that isn’t systematic, it hits both directions. depth holds up. the cracks are random, not structural.

Thanks @NVIDIAAI Good stuff. Model card:

Not sure. Give me some links to the models and I’ll consider testing

No actually better