Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Hugging Face Daily Papers Papers
interpretability ai-safety activation-explanations decodability reconstruction verification

Summary

This paper demonstrates that reconstruction-based tests for activation explanations can be gamed, producing high scores while specific claims remain false. It proposes RECAP, which trains linear heads alongside the target model to keep designated internal content reliably decodable and independently verifiable against probes, improving safety auditing.

Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims: if flipping a claim does not change the reconstruction, the claim is never penalized. We show the test is passed in two ways, neither faithful. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent, so the score tracks gist, not specific facts. Under exact synthetic ground truth, the standard recipe develops co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, and fixes that leave the target model unchanged do not help. We contribute two audit protocols, the grounded-vs-true cross and the evaluator swap, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors): linear heads trained alongside the target model to keep designated content decodable. On RECAP-trained sandbox models, fresh verbalizers state the designated content truly and the codes vanish, at a +0.001-nat cost. This replicates on a pretrained Pythia-160M: the content becomes reliably probe-decodable, though a fresh verbalizer conveys it only in part (truth 0.44-0.46 vs a near-zero control). For interpretability, high reconstruction does not certify individual claims. For AI safety, RECAP makes designated internal content independently checkable against probes rather than asserted by prose a model can game: an independent probe scores the verbalizer's true claims above its false ones (AUC 0.96, vs 0.82 without RECAP). Against an adversary that edits an explanation to maximize the reconstruction score while lying (suppressing ~87% of its lie penalty), the RECAP probe still flags the lies (AUC 0.95) while the control probe collapses to chance (0.51).
Original Article
View Cached Full Text

Cached at: 07/24/26, 05:07 AM

Paper page - Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Source: https://huggingface.co/papers/2607.20379

Abstract

Natural-languageautoencodersscoreexplanationsofhiddenactivationsbyreconstruction:anexplanationisdeemedfaithfuliftheactivationcanberegeneratedfromit.Thetestisstructurallyinsensitivetoindividualfalseclaims:ifflippingaclaimdoesnotchangethereconstruction,theclaimisneverpenalized.Weshowthetestispassedintwoways,neitherfaithful.OnareleasedQwen-2.5-7Bverbalizer,explanationsreconstructwellabovechancewhile~2%ofspecificclaimsarereconstruction-dependent,sothescoretracksgist,notspecificfacts.Underexactsyntheticgroundtruth,thestandardrecipedevelopsco-adaptedprivatecodes(falsewordingthereconstructiondependson)in5/5runs,andfixesthatleavethetargetmodelunchangeddonothelp.Wecontributetwoauditprotocols,thegrounded-vs-truecrossandtheevaluatorswap,andRECAP(ReadableEncodingsviaCo-trainedAuxiliaryPredictors):linearheadstrainedalongsidethetargetmodeltokeepdesignatedcontentdecodable.OnRECAP-trainedsandboxmodels,freshverbalizersstatethedesignatedcontenttrulyandthecodesvanish,ata+0.001-natcost.ThisreplicatesonapretrainedPythia-160M:thecontentbecomesreliablyprobe-decodable,thoughafreshverbalizerconveysitonlyinpart(truth0.44-0.46vsanear-zerocontrol).Forinterpretability,highreconstructiondoesnotcertifyindividualclaims.ForAIsafety,RECAPmakesdesignatedinternalcontentindependentlycheckableagainstprobesratherthanassertedbyproseamodelcangame:anindependentprobescorestheverbalizer’strueclaimsaboveitsfalseones(AUC0.96,vs0.82withoutRECAP).Againstanadversarythateditsanexplanationtomaximizethereconstructionscorewhilelying(suppressing~87%ofitsliepenalty),theRECAPprobestillflagsthelies(AUC0.95)whilethecontrolprobecollapsestochance(0.51).

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.20379

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.20379 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.20379 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.20379 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

REVES: REvision and VErification--Augmented Training for Test-Time Scaling

Hugging Face Daily Papers

Proposes REVES, a two-stage iterative framework that alternates between data augmentation and policy optimization to improve LLM reasoning by leveraging intermediate correction steps, achieving superior performance on coding benchmarks and constraint satisfaction problems.

Prover-Verifier Games improve legibility of language model outputs

OpenAI Blog

OpenAI researchers found that optimizing language models purely for correct answers reduces human interpretability, and propose 'prover-verifier games' where a prover generates solutions and a verifier checks them, improving legibility for both humans and AI systems.