Tag
This paper proposes a generalized style-aware full-duplex framework with a lightweight turn controller LPS-TC, introduces a large-scale dataset WildTurn for real-world conversations, and presents a two-tier evaluation scheme to enhance proactive spoken interactions and response quality in dialogue systems.
E-Commerce Bench is an open-source benchmark that evaluates LLM agents on long-horizon autonomous business operation in e-commerce, featuring multi-store negotiation and dynamic events over a simulated year.
This paper presents an LLM-as-a-Judge evaluation framework for agentic AI in drug discovery, validated through human alignment studies with expert annotators. It optimizes the judge to improve alignment with human judgment and provides insights for reusable evaluation in scientific domains.
The paper presents a multi-criteria framework for evaluating socio-technical interventions, using misinformation as a case study, and reveals trade-offs between effectiveness, user acceptance, and implementation feasibility.
The paper proposes Schrödinger Repo, an evaluation framework for coding agents that dynamically instantiates repositories to address data leakage in benchmarks, showing that current LLMs may depend on memorized cues.
This paper proposes MAVEN, a hierarchical framework for evaluating multimodal content against macro-societal values, featuring a benchmark and optimized evaluators for scalable assessment.
This paper introduces MÖVE, a holistic evaluation framework for LLMs in the German public sector, examining governance dimensions like energy consumption, provider transparency, and knowledge of German-party positions, revealing trade-offs that necessitate context-specific model selection.
This paper introduces MuseCPEval, a framework with tailored metrics to evaluate the preservation of unchanged musical attributes in music editing systems during tasks like timbre transfer and genre transformation.
This paper introduces WitnessSim, a deposition simulator using controllable legal personas, and an evaluation framework for assessing behavioral realism in legal simulations.
This paper proposes a framework for evaluating agentic learning harnesses in cybersecurity without labeled benchmarks, using a teacher-student model based on the scaling hypothesis to proxy performance improvements.
This paper systematically evaluates seven frontier AI agents on long-horizon tasks, revealing they function more as engineering optimizers than autonomous researchers, with recommendations for improving training and experience management.
Introduces an operating-point-aware evaluation framework for adaptive data cleaning that corrects for removal-budget confounding. Experiments on CIFAR-10 and ImageNet-100 show that apparent performance gains vanish when operating points are matched, suggesting that many improvements reflect smaller removal budgets rather than genuine corruption discrimination.
smevals is an open-source framework for running evals against small and large models, with a structured vocabulary for tasks, configs, runs, and graders, plus a CLI for execution and grading.
This paper introduces HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution to assess whether vision-language models can act through a physical body. Testing nine state-of-the-art VLMs on 1,218 episodes across 41 scenes, the best model achieves only a 16.8% success rate, revealing that current VLMs lack embodied self-awareness.
The paper introduces Elenchos, a generative evaluation framework for abductive reasoning in LLMs, where models must infer hidden rule changes from behavioral differences under black-box access. It finds a detection-attribution dissociation: models detect alterations but struggle to identify the specific mutations, especially under interacting mutations.
RouteRec is a framework for evaluating request-level hard selection versus item-level learned aggregation of heterogeneous recommender agents, including an LLM reranker, under cost constraints. Experiments on MovieLens-1M show that item-level aggregation significantly outperforms request-level selection.
This paper introduces TradeLens, a trace-grounded diagnostic toolkit for evaluating whether LLM-based agentic trading systems convert their reasoning and tool-use costs into measurable incremental profit, analyzing failure patterns across models like DeepSeek-V3.2 and GLM-4.7.
This paper presents a black-box evaluation framework to assess LLMs' ability to generate Design Structure Matrices (DSMs) from structured technical documentation. It introduces reproducible metrics and a composite quality score, showing that while LLMs can produce plausible DSMs, they remain sensitive to ambiguity and prompt formulation.
Proposes cross-survey transfer as a rigorous evaluation framework for LLM-based human survey simulation, finding that zero-shot LLMs achieve 52% accuracy on unseen items.
Replit AI head shares engineering practices for a production-grade Agent continual learning system, breaking improvements into three layers: Model, Harness, and Context, and building a closed loop of offline validation and online A/B testing, allowing the Agent to continuously improve without updating the model.