Tag
EconSkills introduces a skill library and evaluation framework for web agents to transfer and retrieve procedural knowledge for live economic data retrieval, showing improved efficiency in controlled transfer and competitive performance at library scale.
Google DeepMind has launched a new institute to advance discussions on AGI, featuring essays on AI transparency, evaluation frameworks, and safety principles.
This paper evaluates whether LLM-generated cyberbullying dialogues faithfully reproduce the social dynamics of authentic interactions, finding that while high-level structures are preserved, finer-grained details are systematically distorted in a model-dependent manner.
This paper introduces a ground-truth-as-code framework for evaluating data-science agents on continuously updated data, achieving improved agreement with human evaluators and reduced token consumption.
This research explores sycophancy in large language models when providing romantic relationship advice, revealing that perspective-driven query framing influences behavior more than grammatical mood, and that models tend to increase sycophancy over dialogue turns.
The paper introduces Fuse, a multi-agent simulation framework for evaluating social reasoning in LLM assistants by providing verifiable ground truth through user-mediated interactions, validated with a human study and applied to 12 LLMs.
This paper introduces LogiMed-RoB, a benchmark for evaluating large language models' hierarchical logical consistency in medical risk-of-bias assessment, revealing that high atomic consistency can conceal critical reasoning flaws in clinical deployment.
This paper presents a systematic black-box framework for evaluating agentic AI systems, introducing a taxonomy of risks and automated red teaming methods. Empirical validation across agent architectures reveals critical vulnerabilities, with high rates of governance and privacy risks.
ContractEval is a diagnostic framework that evaluates procedural conformance in LLM agents by making active obligations explicit and matching them against evidence, detecting failures missed by traditional judges.
This paper critiques optimization-based balancing strategies for multimodal sentiment analysis, showing they fail due to conflating fitting speed with discriminative importance, and proposes a new research agenda focusing on held-out modality valuation.
The paper introduces a three-tier evaluation framework for unsupervised narrative label generation and compares clustering-based and graph-community-based methods for discovering disinformation narratives, releasing human-validated labels to support taxonomy development.
CriticGen is a fine-grained, generation-aware evaluation framework for large language models that generates sample-specific evaluation criteria to provide actionable feedback for improving answer quality.
This paper introduces Deep Persona, a psychologically grounded architecture for role-playing agents, and proposes an evaluation framework. It evaluates LLMs and finds systematic limitations in emotional expression despite high pragmatic fluency.
ObserverBench is a benchmark framework designed to evaluate whether internal estimators in AI models are adequate for guiding interventions, control, or safety tasks by reporting both estimation accuracy and downstream action performance.
Introduces READY, an evaluation framework for qualifying AI agents for enterprise deployment by measuring reliability, human oversight burden, and cost, enabling evidence-based deployment decisions.
The paper introduces Blueprint, a safety-evaluation framework that uses WorldviewSim and Monte Carlo Tree Search to optimize multi-turn jailbreak attacks against large language models, achieving high attack success rates with few queries and revealing model-specific vulnerabilities.
The paper proposes EGT-KG, a retrieval framework that uses evidence-grounded typed knowledge graphs to enhance scientific question-answering with small language models, showing improved performance over standard RAG methods in experiments.
SAGE is a novel evaluation framework for task-oriented dialogue agents that grounds assessments in dialogue state changes and uses abstention to provide cost-effective, accurate turn-level judgments without expensive LLM calls.
The paper introduces AVA, a framework to evaluate NLP embeddings' ability to capture ontological reasoning, finding significant limitations and challenging the assumption that strong NLP performance translates to Semantic Web competence.
This paper presents a preliminary study evaluating the accuracy of large language models in verifying causal medical hypotheses, finding that while they exhibit strong recall, they often fail to provide valid scientific evidence or reject unsupported claims.