Tag
This paper identifies perceptual judgment bias in multimodal LLM judges, where they over-reward fluent but visually wrong responses, and proposes a dataset PPJD and a trained model Perception-Judge using GRPO with batch-ranking reward to mitigate this bias and improve perception-grounded evaluation.
This paper proposes a Deep Research pipeline that improves literature search recall by an order of magnitude and argues that human citation lists are not reliable ground truth for evaluation.
This paper identifies weighting noise in LLM judges for multi-stakeholder tasks and proposes DecompR, a method that decouples utility estimation from aggregation using counterfactually calibrated weights.
Voxyz announces a new GBrain feature that enables agents to iteratively improve skills using LLM-as-judge evaluation and an overnight optimization cycle.
This paper proposes a two-stage sampling design where LLM evaluations are used to augment, rather than replace, human ratings, and provides guidance on determining sample sizes for human and LLM reviews using a doubly robust estimator from missing data literature.
This paper introduces Omni-DuplexEval, a benchmark and automatic evaluation framework for real-time duplex interaction in multimodal large language models, assessing continuous response generation and proactive event detection in streaming scenarios.
Practical findings from auditing a production customer support RAG system reveal that heuristic evaluators give false signal, retrieval bugs often masquerade as LLM failures, and the Pareto frontier for cost and quality is often not where expected. Sweeping models showed that replacing the incumbent (Gemini Flash Lite Preview) with Gemma 4 26B achieved a 19% quality improvement at 79% lower cost.
This paper investigates central tendency bias in multimodal LLMs used for clinical ordinal scoring of the Clock Drawing Test, finding that LLMs compress predictions toward the middle of the scale, disproportionately affecting critical extremes. The study extends the LLM-as-judge bias literature to clinical assessment, highlighting the need for calibration-aware evaluation before deployment.
This paper presents empirical measurements of information density in web pages from the perspective of LLM agents, using a curated benchmark of 100 URLs across five categories. It finds that structural extraction reduces token count by an average of 71.5% while preserving answer quality, and reveals an undocumented compression layer in Claude Code.
This study analyzes how modifications to evaluation rubrics, such as shifting from holistic to analytic criteria, impact the agreement between human raters and AI autoraters. The findings suggest that providing examples and reducing bias improves agreement, while higher complexity tends to decrease it.
Researchers from PNNL and Washington University introduce a systematic framework to test how five LLMs detect subtle semantic changes in documents, revealing positional bias, context coherence effects, and model-specific scoring fingerprints.