为Qwen模型添加对‘wait’、‘maybe’和‘perhaps’的对数惩罚可提升其准确性

Reddit r/LocalLLaMA 新闻
logit-penalty qwen-model llama-cpp math-500 quantization accuracy-improvement

摘要

一项实验使用llama.cpp对Qwen模型中的特定过度思考令牌应用对数惩罚，从而在各种量化下提高了MATH-500基准的准确性，带来了显著的准确性提升和推理令牌的减少。

Meta发布了一篇引人注目的论文 https://arxiv.org/pdf/2606.00206，但它没有考察llama.cpp支持的多种量化方法。因此，我在50个随机MATH-500问题（https://huggingface.co/datasets/HuggingFaceH4/MATH-500）上进行了一次测试，并在https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF的各种量化上运行，尝试了 --logit-bias 466-2 --logit-bias 694-2 --logit-bias 1362-2 \ --logit-bias 1412-2 --logit-bias 1921-2 --logit-bias 1990-2 \ --logit-bias 2086-2 --logit-bias 2361-2 --logit-bias 2441-2 \ --logit-bias 2493-2 --logit-bias 2892-2 --logit-bias 3222-2 \ --logit-bias 3315-2 --logit-bias 3384-2 --logit-bias 3404-2 \ --logit-bias 3482-2 --logit-bias 3655-2 --logit-bias 4213-2 \ --logit-bias 4370-2 --logit-bias 4598-2 --logit-bias 4611-2 \ --logit-bias 4808-2 --logit-bias 5752-2 --logit-bias 6970-2 \ --logit-bias 7014-2 --logit-bias 7643-2 --logit-bias 8106-2 \ --logit-bias 10179-2 --logit-bias 10451-2 --logit-bias 11746-2 \ --logit-bias 13264-2 --logit-bias 13428-2 --logit-bias 14673-2 \ --logit-bias 15029-2 --logit-bias 16036-2 --logit-bias 21143-2 \ --logit-bias 21979-2 --logit-bias 33955-2 --logit-bias 35999-2 \ --logit-bias 36563-2 --logit-bias 37201-2 --logit-bias 37781-2 \ --logit-bias 41484-2 --logit-bias 62586-2 --logit-bias 66073-2 \ --logit-bias 73071-2 --logit-bias 84485-2 --logit-bias 85152-2 \ --logit-bias 95500-2 这些对应论文中的过度思考标记：[ " perhaps", " maybe", " wait", " Wait", " actually", " hold", " Hmm", " hmm", " Alternatively", " alternatively", " However", " however", " instead", " Instead", " But", " but", " though", " although", " yet", " rather", " unless", " otherwise", " nonetheless", " nevertheless", " regardless", " still", " anyway", " Or", " or", " either", " whether", " uncertain", " unsure", " possibly", " might", " could", " another", " different", " reconsider", " rethink", " backtrack", " retry", " revisit", " doubt", " confused", " wrong", " mistake", " error", " incorrect" ]。以下是结果，令人惊讶的是，即使是BF16也带来了更好的准确性。注意事项是这只在一个模型上进行了一次测试。试试看是否有帮助！格式 准确率：基线 → 惩罚 推理令牌 BF16 74% → 84% −19.4% Q8_0 76% → 80% −11.0% Q4_K_M 60% → 66% −14.8% Q3_K_M 52% → 66% −17.5% Q2_K 12% → 24% −11.5%
查看原文

相似文章

不流行观点：Qwen 3.8 27b 不是过度思考者

Reddit r/LocalLLaMA

文章认为，Qwen 3.8 27b 增加的推理令牌使用量与其他中国AI模型如 GLM 和 DeepSeek 相似，用户的挫败感源于硬件限制。它建议使用推理预算可以保持性能优于 Qwen 3.6。