evaluation-benchmark

Tag

Cards List
#evaluation-benchmark

ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?

Hugging Face Daily Papers ↗ · 2026-08-04 Cached

Introduces ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning in LLM agents, showing that while sequential execution improves performance, current methods struggle to consolidate experience into robust, transferable skills.

0 favorites 0 likes
#evaluation-benchmark

Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

arXiv cs.CL ↗ · 2026-07-27 Cached

This paper presents a longitudinal evaluation instrument for agent memory that avoids label-error and contamination by generating facts before text, and demonstrates that short-horizon benchmarks can mis-rank memory architectures compared to long-horizon performance. It also releases the Veracium open-source library for agent memory evaluation.

0 favorites 0 likes
#evaluation-benchmark

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Hugging Face Daily Papers ↗ · 2026-07-27 Cached

ClinFusion is a vision-centric multimodal large language model for holistic medical understanding that unifies 2D and 3D medical image analysis using a cascaded vision encoder. It achieves state-of-the-art results on 20 out of 24 benchmarks and outperforms proprietary models like GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks.

0 favorites 0 likes
#evaluation-benchmark

@rohanpaul_ai: LLMs can accept the same false claim differently depending on its tone, certainty, and grammatical form. Small wording …

X AI KOLs Timeline ↗ · 2026-07-22 Cached

This paper presents EoBench, a benchmark testing LLMs' acceptance of false claims phrased in 19 different styles, finding that tone, certainty, and grammatical form significantly affect model responses, with larger and instruction-tuned models showing more resistance.

0 favorites 0 likes
#evaluation-benchmark

MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

arXiv cs.CL ↗ · 2026-07-22 Cached

MedDDC-Eval introduces a diagnosis-decoupled evaluation testbed for multi-turn medical consultation agents, isolating the policy-elicited conversation history from diagnosis generation to enable cleaner measurement of evidence acquisition and diagnostic usefulness.

0 favorites 0 likes
#evaluation-benchmark

ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models

Hugging Face Daily Papers ↗ · 2026-07-22 Cached

Introduces ENTRAP-VL, a taxonomically structured dataset of 1,500 items to probe contextual entrainment in vision-language models, examining how textual and visual context independently influence model outputs.

0 favorites 0 likes
#evaluation-benchmark

Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

arXiv cs.LG ↗ · 2026-07-13 Cached

This paper frames machine unlearning in LLMs as an asymmetric generalization problem, introduces the SUITE evaluation protocol and training corpus to address under- and over-forgetting, and presents JensUn++, an algorithm achieving the best forget-retain utility trade-off across three LLMs.

0 favorites 0 likes
#evaluation-benchmark

MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

arXiv cs.AI ↗ · 2026-07-10 Cached

MentalHospital is a virtual environment designed to evaluate AI agents and human experts on psychiatric clinical encounters, covering interviewing, examination, diagnosis, and treatment planning. Experiments compare human experts, trainees, and various LLMs on objective and subjective metrics.

0 favorites 0 likes
#evaluation-benchmark

Overview of the TalentCLEF 2026: Skill and Job Title Intelligence for Human Capital Management

arXiv cs.CL ↗ · 2026-07-01 Cached

This paper presents an overview of the second edition of the TalentCLEF challenge at CLEF 2026, which includes tasks on job-person matching and job-skill matching in English and Spanish, attracting over 400 submissions.

0 favorites 0 likes
#evaluation-benchmark

Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History

arXiv cs.CL ↗ · 2026-06-10 Cached

This paper introduces Engram, an open-source bi-temporal memory engine for LLM agents that retrieves a compact context slice (∼9.6k tokens) to outperform the full-history baseline (79k tokens) by 10.4 accuracy points on LongMemEval, using a hybrid read path fusing dense, lexical, graph, and temporal signals.

0 favorites 0 likes
#evaluation-benchmark

Building Better Activation Oracles

arXiv cs.LG ↗ · 2026-06-03 Cached

This paper presents improvements to Activation Oracles (AOs) for interpreting residual stream activations, including a new conversational dataset, multi-layer injections, and on-policy training. The authors also release AObench, the first comprehensive evaluation suite for AO quality.

0 favorites 0 likes
#evaluation-benchmark

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Hugging Face Daily Papers ↗ · 2026-05-28 Cached

CausaLab is a scalable environment for evaluating LLM agents on interactive causal discovery, assessing both predictive accuracy and faithful recovery of underlying causal mechanisms. Experiments reveal a gap between prediction and mechanism recovery, highlighting limits in current LLM agents as experimental causal reasoners.

0 favorites 0 likes
#evaluation-benchmark

@_akhaliq: LongMINT Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

X AI KOLs Following ↗ · 2026-05-21 Cached

LongMINT is a benchmark for evaluating memory under multi-target interference in long-horizon agent systems.

0 favorites 0 likes
#evaluation-benchmark

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Hugging Face Daily Papers ↗ · 2026-05-13 Cached

EVA-Bench introduces a comprehensive end-to-end framework for evaluating voice agents, simulating realistic multi-turn conversations and measuring performance across voice-specific failure modes with novel accuracy (EVA-A) and experience (EVA-X) metrics. The benchmark includes 213 scenarios across enterprise domains and a perturbation suite for accent and noise robustness, revealing substantial gaps in current systems.

0 favorites 0 likes
#evaluation-benchmark

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

arXiv cs.AI ↗ · 2026-05-11 Cached

This article introduces TeamBench, a benchmark for evaluating agent coordination under enforced role separation, addressing issues where prompt-only roles may bypass intended constraints.

0 favorites 0 likes
#evaluation-benchmark

DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents

Hugging Face Daily Papers ↗ · 2026-05-06 Cached

This paper introduces the DecodingTrust-Agent Platform (DTap), a controllable and interactive red-teaming platform for evaluating AI agent security across multiple domains. It also presents DTap-Red, an autonomous agent for discovering attack strategies, and DTap-Bench, a large-scale dataset for risk assessment.

0 favorites 0 likes
#evaluation-benchmark

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies

Hugging Face Daily Papers ↗ · 2026-05-06 Cached

This paper introduces SWE-WebDevBench, a comprehensive 68-metric framework for evaluating AI-powered application development platforms as virtual software agencies. The study highlights critical gaps in current platforms regarding specification understanding, backend reliability, production readiness, and security.

0 favorites 0 likes
#evaluation-benchmark

OpenGame: Open Agentic Coding for Games

Papers with Code Trending ↗ · 2026-04-20 Cached

OpenGame is an open-source agentic framework for end-to-end web game creation, powered by the specialized GameCoder-27B model and evaluated via the new OpenGame-Bench benchmark.

0 favorites 0 likes
#evaluation-benchmark

Measuring the performance of our models on real-world tasks

OpenAI Blog ↗ · 2025-09-25 Cached

OpenAI introduces GDPval, a new evaluation framework measuring AI model performance on economically valuable, real-world tasks across 44 occupations in the top 9 US GDP-contributing industries. The benchmark includes 1,320 specialized tasks based on actual professional work products, representing a progression from academic benchmarks to more realistic occupational assessments.

0 favorites 0 likes
← Previous
← Back to home

Submit Feedback