E-Evaluation

Reddit r/AI_Agents 10/10/26, 08:17 AM Papers

llm-judge evaluation g-eval chain-of-thought nlg scoring-stability

Summary

这篇文章科普了微软研究院的论文《G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment》，介绍其通过链式思维生成评分清单并结合各分数的概率加权平均，解决 LLM 评分不稳定的问题。

Here is an annoying thing that happens when you use AI to grade text You ask it to rate a piece of writing from 1 to 10. The first time, it give it an 8. You run the exact same text again, and it gives it a 7. Nothings changed in the text.What happened? Under the hood, the AI does not actually know whether it is a 7 or 8. It might think there us a 40% chance it is an 8, and a 35% chance it is 7. That is why AI grading can feel so random. But a popular method called G-Eval fixes this in a really smart way: Make it think first: Before asking for a score have the AI write out step by step cheaklist point by chain of thought. Look under the hood: Do not just take the single number it splits out. Look at how confident it was in every number from 1 to 10. Take the weighted average: If it was split between 7 and 8, you dont take the highest number. You calculate the average based on its confidence and get somethings like 7.4. Suddenly, your evaluations stop bouncing between whole numbers on every run. You get a calm, smooth score that actually make sense. This entire framework was introduced by the researchers at Microsoft in their paper: "G-EVAL : NLG Evaluation using GPT-4 with Better Human Alignment". Real question have you moved to LLM as a judge in production, or are you still playing it safe with tradational metrics and human review?

Original Article

Similar Articles

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

arXiv cs.CL

该论文提出了一种基于回归的元评估框架，用于评估多语言风格个性化文本生成中的自动指标，发现经过校准的简单字符串和嵌入指标在40种评估设置下可媲美LLM-judge。

Beyond Reference-Based Evaluation: Reward Models for Meta-Evaluation of Grammatical Error Correction

arXiv cs.CL

This paper introduces RM-EVAL, a reward model trained on human preference data for reference-free meta-evaluation of grammatical error correction, and shows how it can improve GEC systems via reward-guided text generation.

What it took to get an LLM judge from garbage to 27 of 30 agreement with humans (live, at a conference)

Reddit r/AI_Agents

作者在 AI Engineer World's Fair 上的 evals 演讲总结，记录了如何将一个 LLM 评审器从完全不可用提升到与人类判断 30 次中有 27 次一致：包括改用点式评分避免偏好偏差、校准上下文、更换模型以及应对谄媚倾向等步骤。

Review Arcade: On the Human Alignment and Gameability of LLM Reviews

Hugging Face Daily Papers

This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.

Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

arXiv cs.LG

This paper introduces Elmes+, an automated framework for constructing fine-grained evaluation rubrics for LLMs in long-tail educational scenarios, and presents the Edu-330 benchmark covering 330 scenarios across 11 subjects. The framework uses a multi-agent engine and self-evolving module to co-optimize evaluation criteria and test data, revealing multidimensional educational capability differences among top LLMs.