@johnschulman2: Bullish on this direction. Having a metric for explanation quality makes it possible to hillclimb, and counterfactual s…
Summary
John Schulman highlights research by Adam Karvonen and colleagues on using counterfactual simulatability as a metric to improve AI explanation quality. They developed a dataset and pipeline that trains models to generate better post-hoc explanations of their own behavior, showing generalization to held-out evaluations.
View Cached Full Text
Cached at: 09/06/26, 08:46 AM
Bullish on this direction. Having a metric for explanation quality makes it possible to hillclimb, and counterfactual simulatability seems right. Adam et al. created a dataset+pipeline that creates more diverse+realistic test cases than prior work & do interesting exps on it. Can also train models to write better post-hoc explanations of their behavior, as highlighted in this thread.
Adam Karvonen (@a_karvonen): Can a model learn to explain its behaviors, such as why it ignored a user request or made a coding error?
We trained models on thousands of explanations of their own in-the-wild behaviors.
Training on this single general dataset shows generalization to held-out evals.
🧵
Similar Articles
CEL: Comprehensive Counterfactual Explanations Library and Benchmark
Introduces CEL, a unified library and benchmark for counterfactual explanations, providing standardized implementations of 14 methods across 18 datasets to enable fair comparison in explainable AI.
Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments
This paper introduces Explanation Quality Markers (EQMs), a set of 60 reasoning patterns scored by LLMs to measure the quality of natural-language explanations in forecasting tournaments. Analyzing over 55,000 forecast-rationale pairs, EQMs predict accuracy at both forecast and forecaster levels, outperforming previous methods.
ExplainBench: Evaluating Code Explanations from Agents
ExplainBench is a new benchmark for evaluating the trustworthiness of explanations generated by coding agents, revealing that explanation quality is a distinct axis of agent evaluation and that an audit agent can automatically improve explanation reliability.
CAS: A Causal Attribution Score for Local and Global Explainable Artificial Intelligence
Introduces CAS, a causal attribution score for local and global explainable AI that separates predictive importance from causal effect heterogeneity, demonstrated on benchmarks and empirical datasets.
Effective Explanations Support Planning Under Uncertainty
This research paper introduces a computational model that evaluates the effectiveness of procedural explanations by simulating how they guide action planning under uncertainty. Through four experiments, the authors demonstrate that explanations scored higher by their model are judged more helpful and lead to better navigation performance.