I put a conformal risk control gate between an LLM agent and its write actions on tau-bench. The bound held. The gate didn't beat asking every time [R]
My agent said it was 100% sure. The user had asked it to change two items in an order. It changed one. So I built gate-calibrated: a barrier between a tool-using agent and the environment. Before any action that writes (modify an order, change a payment method), the gate computes two confidence signals, the model's own probability estimate and the agreement of k resampled next actions, maps them to a calibrated probability of success, and compares it to a threshold. The thresholds come from conformal risk control: you pick an alpha, and the bound targets the share of conversations that execute a wrong write without asking. Actions in the irreversible tier (cancel, return, exchange) always ask, under every policy I compared. It is an open implementation of the idea in ToolChain-CRC (arXiv 2606.18467) and CORA (arXiv 2604.09155). Evaluated on tau-bench retail, 3,340 simulated conversations total, agent and simulated user both glm-5.3-flash. The risk control worked. At alpha 0.02, 1.7% of 460 test conversations executed a wrong write without asking. At alpha 0.05, 3.7%. At 0.10 and 0.20, 3.0%. All at or under target in point estimates, with wide CIs, because 460 trajectories is not a lot. tradeoff curve: wrong writes executed vs approvals requested https://preview.redd.it/6yjfiph4mlvh1.png?width=1050&format=png&auto=webp&s=dc280e5abbc7d2043f64476fd44b945ca01d6c2a The gate did not. At alpha 0.10 and above, the learned threshold is so low the gate never asks on a write, which makes it the same policy as a fixed 0.7 confidence cutoff. At alpha 0.02 it asks a lot, and per approval it avoids 0.058 wrong writes, against 0.062 for confirming every write (difference CI [-0.04, 0.03]). Confirming every write also gets you to zero executed wrong writes, at 1.57 approvals per conversation. The calibrated gate cannot reach zero at any alpha. The diagnosis is the part I keep coming back to. Verbalized confidence was 0.9 or higher on 548 of 549 write steps I collected, including all 43 wrong ones. Self-agreement of 5 resamples was 0.9 or higher on 758 of 780 steps. Both signals are saturated, so the calibrator mostly learned the base rate plus one real feature: the weight it puts on step position (-0.80 on log(1 + step index)) says later steps are riskier. Position in the conversation is doing the work confidence was supposed to do. Code, data and the negative results: https://github.com/Gjusev/gate-calibrated If verbalized confidence and self-agreement are this saturated even on an easy benchmark, what would you rank write steps by instead? A separate verifier model, checks on the tool arguments, something else?
The author shares a first working version of a multi-step agent guardrail that re-checks the facts an irreversible action depends on right before execution, plus a benchmark reporting catch rate and false-alarm rate separately — achieving 0% false alarms after conservative normalization while running on a local Qwen2.5-7B model.
This paper identifies a silent failure mode in tool-using LLM agents where policy violations occur without tool errors or agent self-reporting. The authors propose and evaluate lightweight deterministic pre-execution gates that significantly reduce such failures in the τ²-bench airline domain.
This paper proposes Action-Conditioned Risk Gating, a lightweight reinforcement learning method for risk-sensitive control under partial observability that uses a compact finite-history proxy state and an action-conditioned near-term risk predictor to balance safety and performance.