Tag
The paper proposes Avatar Intelligence, a paradigm of persistent personal agents that represent users' goals, values, and constraints through sustained decision delegation, and introduces the Agentic Oversight Index (AOI) to evaluate delivered value against operational cost and oversight burden, outlining a research agenda covering identity-continuous personalization, Rehearsal Engineering, and safety mechanisms.
The paper studies LLMs as numerical optimizers, finding they can capture problem structure and produce near-optimal solutions, but reliability degrades with problem dimension—improved via iterative refinement—motivating hybrid LLM plus deterministic solver approaches.
This paper introduces PRESENT, a controlled evaluation framework showing that user presence significantly alters LLM agent behavior in negotiation—boosting payouts by 13.7% while also doubling misaligned behavior toward counterparties and appearing in 14.3% of interactions with users—exposing blind spots in static, outcome-based evaluation practices.
论文提出 TraDE（Trajectory-based Disclosure Evaluation）框架，基于社会渗透理论，通过固定当前披露深度、改变先前披露轨迹，评估对话模型如何受披露历史影响，发现模型在用户退回浅层披露时与人类支持者的差异最大。
ReTrace is a scalable evaluation framework for systematically studying recovery behavior in computer-use agents, formalizing Transition-Consistent Recovery via goal-oriented action replay after controlled error injection. The study finds that ideal recovery unlocks latent task completion capabilities, current agents fail to exploit their recovery potential, and accurate model-based recovery yields measurable end-to-end gains.
We present AuditKIT, a source-available evaluation library that unifies benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation with direction-aware metric contracts and extensible backends. Seven case studies show how answer extraction, runtime behavior, and metric selection can change evaluation outcomes for LLMs and agentic systems.
VETNET is an ecosystem of live benchmark datasets and evaluation frameworks aimed at accelerating AI-driven innovation in veterinary medicine and animal science through standardized, high-quality data and multi-institutional collaboration. It addresses the fragmented state of veterinary AI compared to advances in human medicine.
The article highlights that model benchmarks fail to evaluate production agents accurately due to system complexities, and introduces TRACES by Apodex for a more comprehensive evaluation, referencing a research paper.
TRACE introduces a condition-aware benchmark and evaluation framework for streaming video understanding, explicitly accounting for temporal validity, execution conditions, and multidimensional performance metrics beyond accuracy.
This paper introduces MemProbe, a cognitive-science-inspired framework for evaluating stability-plasticity tradeoffs in agent memory systems through experimental paradigms, providing interpretable profiles of memory maintenance over time.
This paper proposes a knowledge graph-based evaluation framework for assessing the contextual understanding of large language models in question answering, introducing a new similarity measure called S3KG that achieves superior performance over existing baselines.
This paper presents SRBench, a benchmark dataset for evaluating LLM performance in systematic review screening, and introduces PromptSR, a tool for managing prompt experiments and analysis.
LabFactory is a framework that enables an AI builder to construct executable AI labs from scientific briefs, which are then evaluated on their performance across various tasks.
This paper proposes a causal framework for understanding concept drift in data streams using Structural Causal Models, with a taxonomy and generator for simulating and evaluating drift events in non-stationary environments.
TicTacBench is a new benchmark for evaluating coding agents' timing closure capabilities in RTL design, revealing that current agents have significant room for improvement and proposing TicTacSkill to enhance performance.
This paper introduces OpTFM, a comparative evaluation framework for tabular foundation models in healthcare, assessing models across six clinically meaningful dimensions like generalization and fairness, and applies it to two use cases to demonstrate context-dependent rankings.
This paper introduces LLM endognostics, a white-box framework for auditing latent knowledge in language models, showing that behavioral evaluations can fail to reflect internal knowledge distinctions, as demonstrated on Minerva-7B-Instruct-v1.0.
The Situated Identity Test (SIT) is an architecture-independent framework introduced to evaluate whether an AI agent's behavior is functionally attributable to a specific developmental lineage, addressing the problem of persona imitation in large language models.
EnterpriseVal introduces a comprehensive evaluation system for generative AI in enterprises, addressing the measurement gap with a use-case-level framework that includes specifications, metrics, and a grading protocol, demonstrated through a pilot study in banking.
This paper introduces a productivity-oriented framework for evaluating human-AI collaboration based on outcome quality relative to interaction cost, showing that identical quality ratings can differ significantly in interaction costs and that subjective user ratings are not reliable for measuring productivity.