Tag
CrossAudit introduces a git-native protocol for auditing autonomous research pipelines using AI agents from different vendors to ensure oversight and integrity in agentic science.
This paper presents a unified benchmark comparing outline-driven frameworks for long-form generation with LLMs, evaluating outlines independently and finding that performance varies with framework-granularity match, supporting the decoupling of outline and writing stages.
The paper proposes information satisfaction as a reader-centered axis for summarization evaluation, showing that current metrics fail to capture user-specific informational needs and agree poorly with human judgment.
RubricForge induces evaluation rubrics from labeled trajectories to reduce over-crediting in language model agent evaluation, enhancing faithfulness without environment access.
LinkedIn presents a self-evolving agentic customer support system that integrates RAG with evolutionary auto-prompting and modular evaluation, achieving significant gains in production A/B tests including a 9.0-point increase in QA self-serve and 30.6-point improvement in routing accuracy.
The paper introduces SurveyReview, a reviewer-aligned multi-dimensional benchmark for evaluating survey papers, along with SurveyAlign, a fine-tuned model that substantially improves alignment with human reviewers over prompt-based GPT-5.2 judging.
This paper tests whether persisting evidence generated in one call and using it as the exclusive input for a later verdict (evidence locking) improves or harms LLM-as-judge evaluation. Across 24,000 judgments, they find that evidence locking reduces agreement with human preferences by 4-6 points and increases answer-order inconsistency compared to structured one-call judging.
Introduces SkillTV-Bench, a 681-case benchmark for evaluating skill-aware trajectory verification in LLM agents, along with SkillTV-Evolve, a method that externalizes verification knowledge as a reusable JudgeSkill, improving judge accuracy by 14.8 percentage points.
This paper introduces VIVID, the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese, comprising 1,636 idioms and proverbs. Evaluation of eight state-of-the-art models reveals significant gaps, with Vietnamese-specialized models drastically underperforming multilingual systems and even top models achieving less than 50% correctness on average.
Introduces TQLite, a distillation framework that uses a multi-LRM jury to train small language models for real-time MQM-based translation quality evaluation, achieving performance far exceeding off-the-shelf SLMs while remaining cost-effective.
This paper studies whether cheap open-weight LLMs can judge natural-language mathematical proofs as reliably as frontier models at far lower cost. On IMO-GradingBench, three cheap judges match frontier pass/fail agreement, and the authors recommend an all-three-pass consensus rule for cost-effective deployment.
This paper introduces the 'Agentic Formalism Trap' and an Evaluative Dissonance Index, showing how LLM-as-a-Judge systems can be misled by structural formalism and consensus mimicry rather than semantic truth, based on 22,500 trajectories across multiple domains.
TAPR is a lightweight model trained with reinforcement learning to rewrite user prompts into task-optimized prompts, improving downstream LLM performance on benchmarks like Natural Questions and GSM8K.
Google announces GA of Agent and Model Evaluations in Gemini Enterprise Agent Platform, enabling consistent measurement and monitoring of AI agents in dev and production with pre-built metrics, adaptive rubrics, simulators, and online monitors.
This paper presents the first systematic study of faithfulness in document-grounded podcast generation, introducing a turn-level LLM-as-a-judge evaluation framework and a model-agnostic catch-n-repair method that improves faithfulness across domains.
This paper audits the reliability of LLM-as-judge evaluation by showing that changing the evaluator model can shift scores even when candidate responses are fixed, and it examines scaling and upgrade paths for Qwen3 and MiniMax models, concluding that judge upgrades are not interchangeable and proposing best practices for reporting.
Introduces JAM, a theory-agnostic framework for personality recognition that uses LLMs as judges to improve metric alignment in prototypical networks, achieving better cross-framework generalization.
This paper evaluates the reliability of Gemini models as audio judges for scoring full-duplex voice agent conversations, finding that Gemini 2.5 Flash shows strong agreement with human raters on most dimensions, though model swaps require re-validation.
This paper presents an empirical study evaluating RAG evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik) by comparing them to human judgments and standard recall metrics, using a question-answering dataset created from business data.
This paper identifies a structural flaw in reference-free LLM judges used in self-play training, showing they score plausibility rather than correctness, leading to reward hacking where policies learn to produce plausible-but-wrong answers. The authors propose a hidden-anchor audit and a de-anchored reward to mitigate this issue.