Tag
Gemini 4 Argon dramatically outperforms Gemini 3.1 Pro Preview on a creative writing benchmark (0.80 vs -2.17), winning all 50 matched story comparisons, while Muse Wonder scores 1.96 versus 0.56 for Muse Spark 1.3. The leaderboard covers 59 writer models with over 110,000 judgments from cross-family judges.
作者分享了一套基于真实会话失败用例构建 AI agent 回归测试集的流程：先手动标注 traces，再按失败类型分组、写廉价检查项，并用 Kitaru 的 MCP server 与重放机制加速评测集的维护。
Edu-QuRating presents a pipeline for multi-dimensional educational data scoring using LLM judges to create reusable scorers, enhancing language model pre-training and post-training performance.
This paper proposes a risk-controlled framework for using LLMs as judges in factual evaluation, calibrating uncertainty thresholds to maintain a user-specified error rate and routing to retrieval-augmented mode when needed, achieving higher coverage with provable reliability guarantees.
This paper presents a pre-registered audit of whether LLM-judged helpfulness can reliably distinguish answer-giving from pedagogical guidance in AI tutors. The authors find that general-purpose helpfulness is not a dependable pedagogy signal, recommending pedagogy-targeted rubrics and deterministic process measures instead.
SenseJudge is a human-centric framework for customizable LLM judging that adapts to diverse user preferences, outperforming existing methods. It also introduces SenseBench, a benchmark derived from real-world multi-turn interactions.
OpenAI introduces PaperBench, a benchmark evaluating AI agents' ability to replicate state-of-the-art AI research by replicating 20 ICML 2024 papers with 8,316 gradable tasks. The best-performing model (Claude 3.5 Sonnet) achieves only 21% replication score, below human PhD-level performance, highlighting current limitations in autonomous research capabilities.