Open-source lab for running controlled experiments on tool-using agents (vary tool names / personas / history, measure the effect)
Summary
An open-source lab framework for running controlled experiments on tool-using agents, allowing variation of tool names, personas, and history to measure effects.
Similar Articles
How do you visualise agent tool use?
A discussion or tool for visualizing how AI agents use tools, likely to aid understanding and debugging of agent behavior.
Open-source devtool for AI agent projects
AgentLantern is an open-source devtool for AI agent projects that helps document, analyze, validate, and visualize agent workflows, with initial support for CrewAI and plans to extend to other frameworks.
Set-shifting Behavioral Test for Harnessed Agents
This paper introduces a benchmark for evaluating how LLM agents adapt their tool choice when the reliable tool silently changes mid-session, borrowing set-shifting from cognitive psychology. It tests open-weight LLMs and identifies distinct failure modes based on tool-set framing.
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
Introduces ToolBench-X, a benchmark for evaluating large language model agents under various tool-environment reliability hazards, revealing a substantial gap in performance compared to clean environments.
@omarsar0: Interesting interpretability paper on tool-using agents. The authors probe hidden states and find the model often recog…
This paper introduces a model-adaptive definition of tool necessity and finds a 26-54% mismatch between LLMs' internal recognition that a tool is needed and their actual tool-call actions, concentrated in the cognition-to-action transition. It reveals a 'knowing-doing gap' where the model often knows it should call a tool but fails to do so due to late-layer geometry rotating the signal nearly orthogonal to the action.