Tag
Introduces Janus, a playground system for implementing and evaluating user-involved agentic permission management designs, demonstrating the importance of user input and AI augmentation for privacy and security.
MetaHOPE is a metaphor-oriented evaluation framework for analyzing translation errors in machine translation and large language models. The paper proposes an error severity-aware annotation framework and evaluates models like GoogleMT, GPT5.4, and Hunyuan-7b on English-Chinese metaphor translation.
Introduces TRIE, an evaluation framework for stochastic PDE surrogates that tests reproduction of invariant measures, trustworthy predictive uncertainty, and efficiency. Benchmarks pointwise-trained neural surrogates, approximate uncertainty methods, and generative models on two SPDEs, finding generative models most consistent.
This paper introduces a cross-evaluation framework for benchmarking LLMs on Arabic cultural and sociolinguistic knowledge, using human SME ground truth and automated judges. The authors contribute a dataset of prompt-rubric pairs for Egyptian and Iraqi Arabic, evaluating frontier LLMs and finding that cultural reasoning remains a primary failure mode for automated grading.
SkillCoach introduces a self-evolving rubric framework that evaluates and enhances LLM agent skill-use by analyzing skill selection, following, composition, and reflection, providing process-level supervision beyond outcome-only metrics.
CLExEval introduces a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking, revealing failure patterns such as verbosity bias, hidden knowledge paradox, and reasoning-to-output mismatch in models like GPT-4o-mini and HuatuoGPT-o1.
This paper introduces PSALM, an LLM-as-a-judge framework that evaluates copyright infringement under EU law by assessing stylistic and narrative appropriation beyond verbatim memorization, finding that fine-tuning induces systematic stylistic similarity and that existing safeguards are insufficient.
This article argues for the need of a benchmark leaderboard that compares AI model harnesses (e.g., KimiCode vs OpenCode vs Codex) rather than just models themselves, proposing a repo to test model+harness combinations on cost, runtime, token usage, and score.
MedEvoEval is a longitudinal evaluation framework for doctor agents that simulates outpatient episodes, assessing how agents acquire evidence, use resources, and evolve across episodes through memory and reflection mechanisms.
TerraProbe introduces a five-layer oracle evaluation framework to detect deceptive fixes in LLM-assisted Terraform security repair, revealing that such fixes are systemic across models like Gemini, GPT-4o, and Claude. The paper provides a taxonomy of deceptive fixes and a replication package for evaluating IaC security repairs.
Introduces AgentOdyssey, a procedural text game generation framework designed to evaluate agents on test-time continual learning abilities including exploration, episodic memory, world knowledge acquisition, skill learning, and long-horizon planning. The framework highlights significant gaps between current agents and human performance.
This paper studies small language models (SLMs) as closed-loop policies for graph algorithm execution, evaluating both step accuracy and rollout reliability across multiple graph procedures. The results show a gap between local decision quality and global execution reliability, especially for weighted algorithms.
RubricsTree proposes a scalable, expert-aligned evaluation framework for personal health agents using over 100 atomic Boolean rubrics, achieving up to 66% relative gains on HealthBench across Gemini, GPT, and Qwen model families.
Evalatro is an open benchmark where LLMs play the real game Balatro via a text-based interface, with fixed seeds, a public leaderboard, and the goal of clearing Ante 12. Early results show models struggle, with none reaching the target.
This paper from eBay presents a modular two-agent simulation framework for evaluating conversational shopping assistant architectures, enabling controlled comparisons of responder designs. Key findings include that rolling-window memory outperforms intent-extraction memory by 35% in speed, and that systematic failure analysis reduced failure rates by 62%.
This paper presents SkillJuror, a framework for evaluating how different organizational paradigms for Agent Skills affect LLM agent runtime behavior. A study on 82 tasks shows that Progressive Disclosure increases resource diversity and uptake events compared to a flat baseline, with task-dependent outcome improvements.
Introduces BioDivergence, a benchmark and evaluation framework for detecting context-conditioned contradictions in biomedical abstracts, featuring a six-class conflict taxonomy and a silver dataset of 11,865 claim pairs.
Introduces PoQ-Judge, a multi-architecture evaluation framework with reference-free judge models (TextCNN, MiniLM, DeBERTa) for cost-aware Proof-of-Quality in decentralized LLM inference, achieving high correlation with ground-truth proxies while eliminating the need for reference answers.
This paper introduces EngVQA, a multimodal benchmark for evaluating engineering reasoning in vision-language models, along with an 8-stage automatic evaluation framework that enables fine-grained analysis of reasoning failures. It reveals substantial limitations in current VLMs' engineering reasoning capabilities.
This paper introduces CapCode, a capped evaluation framework that uses randomized test outputs to detect coding agents that game unit tests, and CapReward, a reward design that penalizes reward hacking in reinforcement learning for coding tasks.