Open-source lab for running controlled experiments on tool-using agents (vary tool names / personas / history, measure the effect)
Summary
An open-source lab framework for running controlled experiments on tool-using agents, allowing variation of tool names, personas, and history to measure effects.
Similar Articles
SciToolAgent-Evo: An Ontology-Aware Self-Evolving Agent for Open-World Scientific Tool Acquisition
Presents SciToolAgent-Evo, an ontology-aware self-evolving LLM agent for open-world scientific tool acquisition, along with the OpenSciToolBench benchmark of 900 realistic tasks. The agent uses an evolving memory and LinUCB-based bandit gate to dynamically explore and acquire novel tools.
How do you visualise agent tool use?
A discussion or tool for visualizing how AI agents use tools, likely to aid understanding and debugging of agent behavior.
Open-source devtool for AI agent projects
AgentLantern is an open-source devtool for AI agent projects that helps document, analyze, validate, and visualize agent workflows, with initial support for CrewAI and plans to extend to other frameworks.
Set-shifting Behavioral Test for Harnessed Agents
This paper introduces a benchmark for evaluating how LLM agents adapt their tool choice when the reliable tool silently changes mid-session, borrowing set-shifting from cognitive psychology. It tests open-weight LLMs and identifies distinct failure modes based on tool-set framing.
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
Introduces ToolBench-X, a benchmark for evaluating large language model agents under various tool-environment reliability hazards, revealing a substantial gap in performance compared to clean environments.