Interactive Evaluation Requires a Design Science

Hugging Face Daily Papers Papers
interactive-evaluation ai-evaluation llm-evaluation agent-benchmarks evaluation-paradigm design-science taxonomy

Summary

This position paper argues that interactive AI evaluation should be treated as a design science paradigm, proposing a two-axis taxonomy and reporting standards for assessing dynamic system behavior through trajectories.

AI evaluation is undergoing a structural change. Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to build interactive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previous evaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomes interaction-generated trajectories, while the evaluation procedure must assess process, recoverability, coordination, robustness, and system-level performance. Building on this definition, we propose a two-axis taxonomy, derive design principles and reporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.
Original Article
View Cached Full Text

Cached at: 05/20/26, 10:40 PM

Paper page - Interactive Evaluation Requires a Design Science

Source: https://huggingface.co/papers/2605.17829 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Interactive evaluation represents a principled paradigm shift requiring new frameworks for assessing system behavior through dynamic trajectories rather than static responses.

AI evaluation is undergoing a structural change.Large language models(LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to buildinteractive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principledevaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previousevaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomesinteraction-generated trajectories, while the evaluation procedure must assess process,recoverability,coordination,robustness, andsystem-level performance. Building on this definition, we propose atwo-axis taxonomy, derivedesign principlesandreporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2605\.17829

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.17829 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.17829 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.17829 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Position: Behavioral Systems Require Behavioral Tests

arXiv cs.AI

This position paper argues that AI agents must be evaluated like behavioral systems through systematic observation and testing of their processes, not just performance outcomes, and proposes a research agenda to develop rigorous behavioral tests for better understanding and safety.

AI Evaluation Should Work With Humans

arXiv cs.AI

This position paper argues that AI evaluation should pivot to assessing human-AI teams rather than superhuman performance to foster better societal outcomes.

The Evaluation Trap: Benchmark Design as Theoretical Commitment

arXiv cs.AI

This paper identifies the 'evaluation trap' where AI benchmarks inadvertently stabilize dominant paradigms by narrowing what counts as progress, and introduces Epistematics, a meta-evaluative methodology to ensure evaluation criteria discriminate true capability from proxy behaviors.