Tag
STOCKTAKE is a 26-week supply-chain benchmark using a POMDP with a fair oracle to separately measure failures of perception and action in LLM agents. Results show that agents often correctly diagnose hidden state changes but fail to act appropriately, indicating a gap between stated beliefs and costly actions.
This paper introduces a model-adaptive definition of tool necessity and finds a 26-54% mismatch between LLMs' internal recognition that a tool is needed and their actual tool-call actions, concentrated in the cognition-to-action transition. It reveals a 'knowing-doing gap' where the model often knows it should call a tool but fails to do so due to late-layer geometry rotating the signal nearly orthogonal to the action.
This paper introduces a model-adaptive definition of tool necessity for LLMs, revealing a substantial mismatch between when a model should use a tool and when it actually does. The authors decompose tool use into cognition and action stages, finding that the majority of errors occur in translating recognition into action, identifying a 'knowing-doing gap' in LLM tool use.