Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations
Summary
This paper systematically tests linear probes for deception detection in large language models, finding they fail under distributional shifts but style-augmented probes recover performance, and revealing that deception is encoded through distributed sub-threshold features.
View Cached Full Text
Cached at: 06/03/26, 07:36 AM
Paper page - Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations
Source: https://huggingface.co/papers/2605.27958
Abstract
Linear probes for deception detection in large language models fail under distributional shifts despite high performance on clean data, revealing that deception is encoded through distributed sub-threshold features rather than simple linear directions.
Linear probestrained onLLM activationsare increasingly proposed asdeception-detection metrics, yet reportAUROCexceeding 0.96 on clean benchmarks while collapsing underdistributional shift. This paper systematically pressure-tests probe-based metrics across theGemma 3 model family(1B-27B parameters), diagnosing why they fail rather than merely documenting that they fail. We test four hypotheses about deception encoding: (1) single linear direction, (2) multi-dimensional subspace, (3)convex conic hull, (4)entropy proxy. Our design includescross-domain transfermatrices,multi-dimensional probe analysiswithpermutation null baselines,entropy-residualizationtests, and distractor evaluations across 8stylistic shifts. We find that: (a) probes achieve near-perfectAUROC(>=0.998) on clean data but collapse understylistic shifts; style-augmented probes recover near-perfect detection (meanAUROC0.979-0.983) on unseen styles; (b) the single-direction hypothesis is rejected (k=1 captures only 0.61-0.80AUROC), withcross-domain transferfailure confirmed as geometric rather thanlayer-mismatch-driven; (c) the entropy-proxy hypothesis is rejected (max |rho|=0.454, max Delta-AUROCafter residualization=0.004); and (d) deception does not form a significant linear subspace (per-domain k*=0), yet multi-dimensional probes (k>=5) recover the signal through distributed sub-threshold features. Probe fragility reflects distributional narrowness rather than an architectural limitation: style-augmented probes recover near-perfect detection at both 4B and 27B, establishing that the inverse scaling pattern is a training-distribution artifact rather than a genuine scale-dependent phenomenon.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2605\.27958
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.27958 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.27958 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.27958 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs
This paper systematically studies how lie typology, representation depth, probe expressivity, and sparse features impact deception detection in LLMs, finding that detection performance is highly dependent on training data and representation choice.
When LLMs Learn to Be Consistently Wrong: A Multi-Model Study of Linear Representations of Synthetic Deception
This paper studies synthetic dishonesty in LLMs by fine-tuning honest and deceptive variants of five transformer models and finding that robust, domain-invariant dishonesty representations can be rapidly entrenched via modest supervised fine-tuning, with implications for activation-based monitoring.
Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States
This paper demonstrates that linear probes on LLM hidden states detect task format confounds (e.g., source identity, response length) rather than distinct reasoning modes, using residualization and causal steering to show that high probe accuracy is due to superficial features, not computational structure.
Scaling Trends for Lie Detector Oversight in Preference Learning
This paper scales the SOLiD lie-detector oversight method to larger LLMs (up to 405B parameters) and evaluates it in realistic preference-learning settings, finding that undetected deception decreases with model scale but that the method is sensitive to distribution shift between training data.
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
This paper evaluates four lie detection methods for language models across prompted lying and trained model organisms, finding that activation- and logprob-based detectors drop sharply on trained model organisms while a chain-of-thought judge remains strong. It introduces new testbeds and the Did-You-Lie (DYL) follow-up probe method, releasing datasets and model organisms.