Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection

# Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection
Source: [https://arxiv.org/html/2607.21776](https://arxiv.org/html/2607.21776)
Tamer Aldwairi Temple University aldwairi@temple\.edu

###### Abstract

Talking\-face \(TF\) deepfake generation synthesizes photorealistic facial video from a static source image and an audio signal, producing forgeries that current image\-based detectors consistently fail to identify\. Unlike face\-swap manipulation, TF synthesis has no underlying real video from which to inherit physiological characteristics, making remote photoplethysmography \(rPPG\) a uniquely motivated detection modality for this forgery category\. We propose a detection framework that extracts per\-video rPPG waveforms via RhythmFormer and trains a suite of lightweight classifiers to distinguish real from synthesized physiological signals\. Evaluated on the TF subset of Celeb\-DF\+\+ under a strict subject\-independent protocol, where test identities are completely separated from training identities, our 1D ResNet achieves an AUC of 0\.806 and EER of 27\.8%, placing it within 2\.4 points of the best published general\-purpose detector \(Effort, ICML 2025\) while operating exclusively on the physiological channel\. We document a controlled reproduction study of DeepFakesON\-Phys, the representative prior rPPG detector, demonstrating degradation from AUC 0\.999 on legacy face\-swap data to 0\.622 on the TF subset of Celeb\-DF\+\+\. We further show that detection difficulty is strongly method\-dependent: AUC ranges from 0\.985 \(Real3DPortrait\) to 0\.690 \(IP\-LAP\) across the seven TF generators, with the ranking remaining perfectly stable across all evaluation protocols\. This spread reflects an interpretable physiological property of each generator rather than evaluation noise, and constitutes the primary theoretical contribution of the work\.

## 1Introduction

Deepfake video generation has advanced to the point where forgeries routinely deceive both human viewers and automated forensic systems\. The Celeb\-DF\+\+ benchmark\[[9](https://arxiv.org/html/2607.21776#bib.bib1)\]catalogues 22 contemporary synthesis methods organized into three categories: face\-swap \(FS\), face\-reenactment \(FR\), and talking\-face \(TF\)\. Among these, TF presents a distinctive forensic challenge\. TF methods, including SadTalker\[[24](https://arxiv.org/html/2607.21776#bib.bib2)\], IP\-LAP\[[25](https://arxiv.org/html/2607.21776#bib.bib3)\], AniTalker\[[11](https://arxiv.org/html/2607.21776#bib.bib4)\], EDTalk\[[16](https://arxiv.org/html/2607.21776#bib.bib5)\], Real3D\-Portrait\[[21](https://arxiv.org/html/2607.21776#bib.bib6)\], EchoMimic\[[2](https://arxiv.org/html/2607.21776#bib.bib7)\], and FLOAT\[[8](https://arxiv.org/html/2607.21776#bib.bib8)\], synthesize the entire facial appearance from a single static source frame conditioned on audio, leaving no real video substrate from which physiological characteristics can be inherited\.

This generative property has direct consequences for forensic strategy\. In FS forgeries, the source video’s spatiotemporal structure, including subtle skin\-surface reflectance variations induced by pulsatile blood flow, is partially preserved beneath the swapped region\. Remote photoplethysmography \(rPPG\) methods, which recover cardiac pulse waveforms from these microscopic color variations\[[17](https://arxiv.org/html/2607.21776#bib.bib9),[18](https://arxiv.org/html/2607.21776#bib.bib10)\], are therefore working against a residual real physiological signal in FS settings\. In TF synthesis, no such substrate exists\. Any apparent physiological signal in a TF\-generated video must be produced de novo by the generator itself, and current generators make no attempt to reproduce coherent rPPG patterns\.

Prior rPPG\-based deepfake detectors, FakeCatcher\[[3](https://arxiv.org/html/2607.21776#bib.bib11)\], DeepFakesON\-Phys\[[7](https://arxiv.org/html/2607.21776#bib.bib12)\], DeepRhythm\[[15](https://arxiv.org/html/2607.21776#bib.bib13)\], and subsequent work\[[1](https://arxiv.org/html/2607.21776#bib.bib14),[10](https://arxiv.org/html/2607.21776#bib.bib15),[19](https://arxiv.org/html/2607.21776#bib.bib16)\], were developed and evaluated exclusively on FS data, where rPPG is theoretically least advantaged\. The TF detection literature, meanwhile, is dominated by audio\-visual approaches that exploit lip\-audio temporal inconsistency\[[12](https://arxiv.org/html/2607.21776#bib.bib18),[6](https://arxiv.org/html/2607.21776#bib.bib17)\]\. To our knowledge, no prior work applies rPPG\-based detection specifically to TF forgeries, where the physiological argument is strongest\.

We address this gap with the following contributions:

- •First rPPG\-based detection framework specifically targeting TF deepfakes, leveraging RhythmFormer\[[26](https://arxiv.org/html/2607.21776#bib.bib23)\]for waveform extraction and lightweight 1D classifiers for discrimination, achieving AUC \(Area Under the ROC Curve\) 0\.806 within 2\.4 points of the best published general\-purpose detector while using only the physiological channel\.
- •Controlled reproduction of DeepFakesON\-Phys on Celeb\-DF\+\+ TF data, documenting an AUC degradation from 0\.999 \(original FS evaluation\) to 0\.622, quantifying the distribution shift between 2020\-era FS training data and contemporary TF forgeries\.
- •Method\-stratified analysis across all seven TF generators revealing a stable AUC range of 0\.690–0\.985, directly traceable to each method’s pixel synthesis strategy and constituting an interpretable forensic characterization of the TF generation landscape\.
- •Subject\-independent evaluation protocol with disjoint celebrity identity partitions, producing a methodologically rigorous baseline immune to cross\-identity physiological leakage\.

Our code and results are publicly available at[https://github\.com/AI\-Advanced\-Vision\-Forensics\-Lab/rPPG\-TalkingFace\-Detector](https://github.com/AI-Advanced-Vision-Forensics-Lab/rPPG-TalkingFace-Detector)\.

## 2Related Work

### 2\.1Talking\-Face Synthesis

TF synthesis methods animate a still source image with audio to produce a target\-speaker video\. SadTalker\[[24](https://arxiv.org/html/2607.21776#bib.bib2)\]estimates 3D morphable model coefficients from audio and renders head motion via a learned motion field\. IP\-LAP\[[25](https://arxiv.org/html/2607.21776#bib.bib3)\]uses landmark\-guided attention to preserve identity while synchronizing lip motion\. AniTalker\[[11](https://arxiv.org/html/2607.21776#bib.bib4)\]disentangles motion from appearance via a universal motion representation\. EDTalk\[[16](https://arxiv.org/html/2607.21776#bib.bib5)\]employs explicit facial component decomposition for fine\-grained lip and expression control\. Real3D\-Portrait\[[21](https://arxiv.org/html/2607.21776#bib.bib6)\]lifts the 2D synthesis problem into a 3D rendering framework\. EchoMimic\[[2](https://arxiv.org/html/2607.21776#bib.bib7)\]and FLOAT\[[8](https://arxiv.org/html/2607.21776#bib.bib8)\]represent recent diffusion\-based approaches to audio\-driven portrait animation\. Across all these methods, the fundamental operational property is shared: the output video has no temporally coherent real video underlying it, precluding any inheritance of source\-video physiology\.

### 2\.2Remote Photoplethysmography

rPPG recovers cardiac pulse signals from subtle periodic color variations in facial skin caused by pulsatile blood flow\[[17](https://arxiv.org/html/2607.21776#bib.bib9)\]\. Classical approaches use handcrafted signal decomposition\[[5](https://arxiv.org/html/2607.21776#bib.bib20),[18](https://arxiv.org/html/2607.21776#bib.bib10)\]\. Deep learning methods have substantially improved robustness: PhysNet\[[22](https://arxiv.org/html/2607.21776#bib.bib21)\]and variants apply 3D convolutions to spatiotemporal face volumes; PhysFormer\[[23](https://arxiv.org/html/2607.21776#bib.bib22)\]introduces transformer architectures to capture long\-range temporal dependencies\. RhythmFormer\[[26](https://arxiv.org/html/2607.21776#bib.bib23)\]proposes a periodic sparse attention mechanism that learns to focus on physiologically relevant temporal positions, achieving state\-of\-the\-art performance on the UBFC\-rPPG, PURE, and MAHNOB benchmarks\.

### 2\.3rPPG\-Based Deepfake Detection

FakeCatcher\[[3](https://arxiv.org/html/2607.21776#bib.bib11)\]pioneered the use of rPPG for deepfake forensics, classifying videos via statistical features derived from spatially pooled rPPG signals\. DeepFakesON\-Phys\[[7](https://arxiv.org/html/2607.21776#bib.bib12)\]extended this direction with a convolutional attention network operating on facial regions, reporting AUC 0\.999 on Celeb\-DF\-v2\. DeepRhythm\[[15](https://arxiv.org/html/2607.21776#bib.bib13)\]proposed attentional heartbeat rhythm analysis via dual\-spatial\-temporal attention maps\. Liang and Deng\[[10](https://arxiv.org/html/2607.21776#bib.bib15)\]demonstrated that rPPG rhythmic patterns are discriminative not only for detection but also for forgery categorization\. Wu et al\.\[[19](https://arxiv.org/html/2607.21776#bib.bib16)\]proposed multi\-scale spatial\-temporal rPPG maps with local attention and long\-distance interaction modules\.

All of the above approaches were developed and evaluated on FS data\. None targets TF synthesis, where the absence of a real physiological substrate creates a fundamentally stronger case for rPPG\-based discrimination\.

### 2\.4Talking\-Face Deepfake Detection

Dedicated TF detection methods have been proposed primarily from an audio\-visual perspective\. Haliassos et al\.\[[6](https://arxiv.org/html/2607.21776#bib.bib17)\]demonstrated that lip motion features learned for speech recognition are transferable to forgery detection\. Liu et al\.\[[12](https://arxiv.org/html/2607.21776#bib.bib18)\]explicitly model the temporal inconsistency between audio and visual lip motion in lip\-syncing deepfakes, achieving strong results on several benchmarks\. Datta et al\.\[[4](https://arxiv.org/html/2607.21776#bib.bib19)\]exploit spatial\-temporal patterns in the mouth region via a vision temporal transformer\. These approaches uniformly require synchronized audio\. In contrast, our method operates on the RGB visual signal alone, making it complementary to and applicable in audio\-absent or audio\-compromised settings\.

### 2\.5The Celeb\-DF\+\+ Benchmark

Celeb\-DF\+\+\[[9](https://arxiv.org/html/2607.21776#bib.bib1)\]covers three commonly encountered forgery scenarios: Face\-swap \(FS\), Face\-reenactment \(FR\), and Talking\-face \(TF\)\. Each scenario contains a substantial number of high\-quality forged videos, generated using a total of 22 various recent DeepFake methods\. The benchmark establishes three evaluation protocols: GF\-eval \(cross\-method generalization\), GFQ\-eval \(cross\-quality\), and GFD\-eval \(cross\-dataset\)\. Notably, robust results are limited to FS forgeries \(AUC\>\>85%\), while FR and TF forgeries yield degraded detection rates \(AUC 50–70%\), confirming TF as the hardest category for existing detectors\. The best published single\-model result on Celeb\-DF\+\+ is 83\.0% AUC \(Effort\[[20](https://arxiv.org/html/2607.21776#bib.bib25)\], ICML 2025\)\.

## 3Methodology

### 3\.1Problem Formulation

We cast TF deepfake detection as a binary classification problem over per\-video rPPG waveforms\. Formally, letV=\{v1,…,vN\}V=\\\{v\_\{1\},\\ldots,v\_\{N\}\\\}denote a corpus of facial videos, each assigned a binary labelyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}indicating real \(y=0y=0\) or TF\-synthesized \(y=1y=1\)\. Our goal is to learn a classifierf:ℝ160→\[0,1\]f:\\mathbb\{R\}^\{160\}\\to\[0,1\]that maps per\-video rPPG waveforms𝐱∈ℝ160\\mathbf\{x\}\\in\\mathbb\{R\}^\{160\}to fake probabilities\.

![Refer to caption](https://arxiv.org/html/2607.21776v1/x1.png)

Figure 1:Overview of the proposed rPPG\-based TalkingFace detection pipeline\. Input video frames are processed by MediaPipe BlazeFace for face detection and cropped with a 1\.5×\\timesbounding box expansion\. Temporal windowing produces 160\-frame clips at strides=60s=60\. RhythmFormer extracts a pulse waveform𝐱∈ℝ160\\mathbf\{x\}\\in\\mathbb\{R\}^\{160\}, which is per\-window z\-score normalized before classification by the 1D ResNet \(165K parameters\)\.
### 3\.2Waveform Extraction

Face Detection\.We replace the RhythmFormer reference pipeline’s RetinaFace detector with the MediaPipe BlazeFace short\-range model \(Tasks API\), eliminating a TensorFlow dependency while improving throughput and robustness across the head\-pose variation common in Celeb\-DF\+\+\. Face bounding boxes are expanded by a factor of 1\.5 with center preservation to provide contextual margin\. The full pipeline is illustrated in Fig\.[1](https://arxiv.org/html/2607.21776#S3.F1)\.

Temporal Windowing\.For each video, frames are resampled to yield a clip of exactly 160 frames\. For videos of native lengthT<160T<160frames \(minimumT≥60T\\geq 60\), frames are selected via linear interpolation of frame indices, preserving approximate temporal phase relationships\. For a videoVVof lengthT≥160T\\geq 160frames, we extract a set of windows\{Wk\}\\\{W\_\{k\}\\\}of fixed lengthL=160L=160using a sliding stridess\. The starting frame index of thekk\-th window is given by

start​\(k\)=k⋅s,k=0,1,…,⌊T−Ls⌋,\\text\{start\}\(k\)=k\\cdot s,\\quad k=0,1,\\ldots,\\left\\lfloor\\frac\{T\-L\}\{s\}\\right\\rfloor,\(1\)such that windowWkW\_\{k\}consists of framesV\[start\(k\):start\(k\)\+L\]V\[\\text\{start\}\(k\):\\text\{start\}\(k\)\+L\]\. Section[4\.3](https://arxiv.org/html/2607.21776#S4.SS3)describes the specific stride used for real\-video augmentation of the training set\.

RhythmFormer Inference\.Each 160\-frame clip is resized to128×128128\\times 128, globally z\-score normalized across the spatial\-temporal dimensions, and passed through RhythmFormer\[[26](https://arxiv.org/html/2607.21776#bib.bib23)\]loaded with the UBFC\_cross checkpoint\. RhythmFormer outputs a pulse waveform𝐱∈ℝ160\\mathbf\{x\}\\in\\mathbb\{R\}^\{160\}\.

Per\-Window Normalization\.To remove identity\-specific baseline amplitude \(a DC component that encodes subject identity rather than liveness information\), each waveform is independently z\-score normalized:

𝐱~=𝐱−μ𝐱σ𝐱\+ϵ,ϵ=10−6\.\\tilde\{\\mathbf\{x\}\}=\\frac\{\\mathbf\{x\}\-\\mu\_\{\\mathbf\{x\}\}\}\{\\sigma\_\{\\mathbf\{x\}\}\+\\epsilon\},\\quad\\epsilon=10^\{\-6\}\.\(2\)This operation preserves the oscillation shape and frequency content of the rPPG signal while erasing absolute amplitude and mean, rendering the representation more identity\-agnostic\.

### 3\.3Classifier Architectures

We evaluate three core architectures of constrained scale \(56K, 165K, 69K parameters\), designed to characterize the discriminative information available in the extracted waveforms\. Two additional Toeplitz\-based representations were tested and are discussed in Section[3\.6](https://arxiv.org/html/2607.21776#S3.SS6)as a negative result\.

1D Residual Network \(1D ResNet\)\.Three stages of basic residual blocks are used, with configuration\(2,2,2\)\(2,2,2\)and channel widths 32, 64, 128\. Each basic block consists of two Conv1d layers with batch normalization and a skip connection\. A stem convolution and global average pooling precede the final linear classifier\. This architecture totals 165K parameters and serves as the primary classifier throughout our experiments\.

1D Convolutional Network \(1D CNN\)\.Three sequential Conv1d blocks with channel dimensions1→32→64→1281\\to 32\\to 64\\to 128and kernel sizes 9, 7, 5 are applied in sequence\. Each block performs convolution, batch normalization, ReLU activation, and dropout\. A global average pooling operation is followed by a linear projection to a scalar logit\. This architecture totals 56K parameters\.

1D Transformer\.A patch size of 8 yields 20 tokens per sequence, with a model dimension of 64, 4 attention heads, and 2 encoder layers\. The architecture uses learned positional embeddings, a learnable \[CLS\] token, and pre\-layer normalization\. This architecture totals 69K parameters\.

### 3\.4Training Protocol

All classifiers are trained with a weighted binary cross\-entropy loss to address class imbalance\. For a batch ofNNsamples with ground\-truth labelsyiy\_\{i\}and predicted probabilitiesy^i\\hat\{y\}\_\{i\}, the loss is defined as

ℒWBCE=−1N​∑i=1N\[w\+⋅yi​log⁡\(y^i\)\+\(1−yi\)​log⁡\(1−y^i\)\],\\mathcal\{L\}\_\{\\text\{WBCE\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\Big\[w\_\{\+\}\\cdot y\_\{i\}\\log\(\\hat\{y\}\_\{i\}\)\+\(1\-y\_\{i\}\)\\log\(1\-\\hat\{y\}\_\{i\}\)\\Big\],\(3\)where the positive class weightw\+=n−/n\+w\_\{\+\}=n\_\{\-\}/n\_\{\+\}is the ratio of negative \(real\) to positive \(fake\) samples in the training set, compensating for the 1:7 real\-to\-fake imbalance in our corpus\.

We use AdamW with weight decay for 30 epochs at batch size 64\. The best\-performing configuration, confirmed via cross\-validation, uses lr=10−3=10^\{\-3\}, wd=5×10−4=5\\times 10^\{\-4\}, and dropout=0\.5=0\.5for the 1D ResNet, and lr=5×10−4=5\\times 10^\{\-4\}, wd=10−3=10^\{\-3\}, and dropout=0\.1=0\.1for the Transformer\.

### 3\.5Data Augmentation

To reduce overfitting to the rPPG characteristics of training identities, we apply two augmentations to training waveforms only, never to validation or test data\. First, additive Gaussian noise𝜼∼𝒩​\(0,0\.052\)\\boldsymbol\{\\eta\}\\sim\\mathcal\{N\}\(0,0\.05^\{2\}\)is injected at every epoch, preventing memorization of clean physiological signatures\. Second, a contiguous temporal segment of lengthm∼𝒰​\[10,25\]m\\sim\\mathcal\{U\}\[10,25\]is zeroed out at a random position, analogous to SpecAugment\[[14](https://arxiv.org/html/2607.21776#bib.bib24)\]for audio, encouraging the classifier to exploit distributed temporal features rather than relying on specific waveform segments\.

### 3\.6Ablation Study and Failed Representations

In addition to the three core architectures, we evaluated two alternative representations based on a Toeplitz reformulation of the waveform, to test whether explicit pairwise lag\-domain structure would aid discrimination\. In the first, the waveform𝐱\\mathbf\{x\}is reshaped into a symmetric Toeplitz matrix𝐓∈ℝ160×160\\mathbf\{T\}\\in\\mathbb\{R\}^\{160\\times 160\}, where each entry is defined as

Ti​j=x\|i−j\|,i,j∈\{1,…,160\},T\_\{ij\}=x\_\{\|i\-j\|\},\\quad i,j\\in\\\{1,\\ldots,160\\\},\(4\)exposing pairwise lag\-domain relationships as a two\-dimensional structure, processed by a three\-block 2D convolutional network totaling 32K parameters\. In the second, a patch size of 16 is applied to the same Toeplitz matrix, yielding 100 tokens for a 90K\-parameter Vision Transformer with a model dimension of 64, 4 attention heads, and 2 encoder layers\. Both representations were ultimately discarded: under the per\-window z\-score normalization of Equation \([2](https://arxiv.org/html/2607.21776#S3.E2)\), the 2D CNN collapses to a degenerate decision threshold, while the ViT shows no improvement over the 1D baselines\. We attribute this to both representations’ dependence on absolute amplitude structure, which normalization explicitly removes, and report this as a rigorous negative result rather than an omission\.

## 4Experimental Setup

### 4\.1Dataset

We evaluate on the TF subset of Celeb\-DF\+\+\[[9](https://arxiv.org/html/2607.21776#bib.bib1)\], which contains 590 real videos sourced from 59 unique celebrity identities and 17,500 TF\-forged videos spanning seven synthesis methods \(2,500 per method\)\. rPPG waveforms are extracted from all videos, yielding 585 real waveforms \(585 source videos with stride\-60 temporal windowing produce 2,435 waveforms in total\) and 17,500 fake waveforms across the seven methods\.

### 4\.2Subject\-Independent Data Partition

To prevent cross\-identity physiological leakage, we partition data at the identity level\. The 59 unique celebrity identities are assigned to disjoint splits: 41 identities to training, 9 to validation, and 9 to test\. All videos from a given identity are assigned exclusively to one split, with zero overlap verified before any experiments are run\. The resulting partition is summarized in Table[1](https://arxiv.org/html/2607.21776#S4.T1)\.

Table 1:Dataset Split StatisticsThe test identities \(id0, id4, id6, id11, id13, id16, id23, id27, id54\) are completely withheld from all training and hyperparameter selection procedures\. Per\-method fake counts in the training split range from 1,682 \(Real3DPortrait\) to 1,720 \(EDTalk\), reflecting the unequal distribution of identities across method\-specific generation pipelines\.

### 4\.3Temporal Cropping

Training real videos undergo temporal windowing at strides=60s=60\(Equation \([1](https://arxiv.org/html/2607.21776#S3.E1)\)\) at extraction time, yielding between 2 and 9 waveforms per source video \(median: 3\)\. This produces 2,435 real training waveforms from 585 source videos, achieving a training ratio of approximately 1:7 \(real:fake\), compensated byw\+≈7w\_\{\+\}\\approx 7in the weighted BCE loss\.

### 4\.4Cross\-Validation Protocol

Hyperparameter selection uses 5\-fold StratifiedGroupKFold with celebrity identity as the grouping key, ensuring that all temporal\-crop waveforms derived from the same source video remain in the same fold and that validation folds contain identities completely absent from training folds\. This produces 75 training runs per experimental condition\.

### 4\.5Evaluation Metrics

We report AUC \(area under the ROC curve\) as the primary discriminability metric\. EER \(equal error rate\) provides a threshold\-independent operating point standard in biometric evaluation\. For multi\-seed experiments, we report mean±\\pmstandard deviation across 5 random seeds to characterize result stability\.

## 5Experiments and Results

### 5\.1Reproduction Study: DeepFakesON\-Phys

To establish a reference baseline and quantify distribution shift between 2020\-era face\-swap data and contemporary TF forgeries, we execute the official DeepFakesON\-Phys pipeline\[[7](https://arxiv.org/html/2607.21776#bib.bib12)\]on our TF corpus using released model weights and preprocessing code\. Several compatibility patches are required for current software dependencies \(NumPy 2\.x, TensorFlow legacy Keras mode, headless OpenCV\)\. With all patches applied and the official preprocessing unchanged, we measure a video\-level AUC of 0\.622 on our TF corpus, compared to 0\.999 reported by the original authors on Celeb\-DF\-v2 face\-swap data\. We note that the score convention in DeepFakesON\-Phys isP​\(real\)P\(\\text\{real\}\); AUC is computed with label inversion accordingly\.

The 0\.377\-point degradation reflects two compounding factors\. First, there is a severe distribution shift: DeepFakesON\-Phys was trained exclusively on face\-swap forgeries, which partially preserve source\-video physiology, whereas TF synthesis produces entirely different rPPG disruption patterns\. Second, the Haar cascade face detector in the reference pipeline exhibits limited robustness on the non\-frontal head poses common in TF outputs, producing degraded crops that corrupt physiological signal estimation\.

### 5\.2Technique Isolation

Table[2](https://arxiv.org/html/2607.21776#S5.T2)reports the effect of per\-waveform z\-score normalization and waveform augmentation evaluated independently under 5\-fold cross\-validation on the full 17,500\-fake training corpus\. Neither technique produces a meaningful improvement over the baseline: z\-score normalization yields a CV AUC change of\+\+0\.006 for the 1D ResNet and augmentation produces−\-0\.007, with similarly marginal effects across all four architectures\. HP sensitivity is equally low, with a maximum AUC spread of 0\.008 across all tested learning rate, weight decay, and dropout configurations for the 1D ResNet\. These findings confirm that the rPPG signal quality of each generation method, rather than training configuration or preprocessing choice, is the primary determinant of detection performance\.

Table 2:Technique Isolation — 5\-Fold CV AUC \(1D ResNet\)
### 5\.3Main Results

Table[3](https://arxiv.org/html/2607.21776#S5.T3)presents the primary evaluation of our 1D ResNet on the 18\-identity test set \(val and test identities combined, never used in training\), averaged across 5 random seeds\. The combined evaluation over all seven TF methods yields AUC0\.806±0\.0030\.806\\pm 0\.003, representing a 0\.184\-point improvement over the DeepFakesON\-Phys reproduction baseline and placing our method within 2\.4 points of the best published general\-purpose detector on Celeb\-DF\+\+ \(Effort\[[20](https://arxiv.org/html/2607.21776#bib.bib25)\], 0\.830, ICML 2025\), despite operating exclusively on the physiological rPPG channel\.

Table 3:Main Results — 1D ResNet, 18\-Identity Eval \(5 Seeds\)The 1D ResNet consistently outperforms the Transformer across all evaluation conditions, and both architectures substantially outperform the DeepFakesON\-Phys reproduction baseline\.

### 5\.4Per\-Method Analysis

The most theoretically significant finding of this work is the strong and stable dependence of detection difficulty on the specific TF generation method\. Fig\.[2](https://arxiv.org/html/2607.21776#S5.F2)shows the ROC curves for each method under the 18\-identity subject\-independent protocol, and Table[4](https://arxiv.org/html/2607.21776#S5.T4)reports the corresponding AUC values\.

![Refer to caption](https://arxiv.org/html/2607.21776v1/x2.png)Figure 2:ROC curves for the 1D ResNet evaluated per TalkingFace generation method under the 18\-identity subject\-independent protocol \(5\-seed mean\)\. Line style and color together encode method identity for grayscale readability\. The AUC spread of 0\.295 across methods reflects the physiological detectability of each synthesis strategy rather than evaluation noise\.![Refer to caption](https://arxiv.org/html/2607.21776v1/x3.png)Figure 3:Representative rPPG waveforms \(z\-score normalized\) extracted by RhythmFormer from a real video, an IP\-LAP fake, and a Real3DPortrait fake\. The real waveform exhibits irregular cardiac variation\. IP\-LAP produces a near\-sinusoidal artifact that mimics physiological periodicity, explaining its low detection AUC \(0\.690\)\. Real3DPortrait produces incoherent noise with no periodic structure, making it trivially detectable \(AUC 0\.985\)\.Table 4:Per\-Method Evaluation — 1D ResNet, 18\-Identity Protocol \(5 Seeds\)The AUC spread of 0\.295 across methods is far larger than the variance attributable to random seed or hyperparameter choice, confirming that method identity is the dominant factor in detection difficulty\. Crucially, this ranking is perfectly consistent across every evaluation protocol we apply, including the CV AUC, 9\-identity test, 18\-identity test, and per\-method isolated training, with no shuffling of the ordering across conditions\. This stability across protocols indicates that the difficulty ordering reflects a genuine underlying property of each generator’s rPPG output rather than a statistical artifact, one that is directly traceable to each method’s pixel synthesis strategy\. Methods that synthesize pixel values from learned latent representations with no reference to source\-frame temporal structure \(Real3DPortrait, EDTalk, SadTalker\) produce outputs with no coherent rPPG signal, making them trivially detectable\. The detector simply identifies the absence of physiological periodicity\. Methods that warp or redistribute the original source frame’s pixels \(FLOAT, EchoMimic\) partially preserve the source image’s color structure, accidentally producing temporal color variation that superficially resembles rPPG activity, making them harder to detect\.

IP\-LAP occupies an exceptional position\. Even when trained exclusively on IP\-LAP fakes with balanced data and tuned hyperparameters, the 1D ResNet achieves only AUC 0\.690, which is barely above chance and substantially below every other method\. IP\-LAP’s landmark\-guided attention mechanism preserves the source frame’s identity structure more faithfully than any other method, which appears to also preserve sufficient temporal color coherence that the rPPG signal in IP\-LAP fakes is physiologically plausible rather than simply absent or noisy\. Notably, the Transformer outperforms the 1D ResNet specifically on IP\-LAP \(0\.728 vs 0\.707 in isolated training\), suggesting that attention\-based temporal modeling captures subtle waveform irregularities that the convolutional approach misses\. This architectural sensitivity to the hardest method is an observation for future detector design\.

### 5\.5Isolated Per\-Method Training

To further characterize the learnability of each method’s rPPG signature in isolation, we conduct an additional experiment in which the model is trained and evaluated exclusively on one method at a time, using 2,435 real and 2,500 fake videos in a near balanced 1:1 ratio under the same subject\-independent identity split\. Results are summarized in Table[5](https://arxiv.org/html/2607.21776#S5.T5)\.

Table 5:Isolated Per\-Method Training — 1D ResNet vs Transformer, 18\-Identity Protocol \(5 Seeds\)\. Comparison against the combined baseline \(AUC 0\.806\)\.All methods improve substantially under isolated training, with the sole exception of IP\-LAP, which drops 0\.099 AUC below the combined baseline\. This inversion reveals that IP\-LAP’s rPPG signal is so similar to real physiology that the model cannot learn a reliable decision boundary from IP\-LAP fakes alone\. The model requires exposure to the clearly absent physiological signals in other methods to calibrate its internal representation of what “fake” looks like\. Without that anchor, classification of IP\-LAP collapses\. This is perhaps the strongest evidence we have that IP\-LAP represents a qualitatively different detection regime from the other six methods\.

## 6Discussion

### 6\.1rPPG as a Forensic Modality for Talking\-Face Detection

Our results confirm that rPPG\-based detection is a well\-motivated and effective approach for TF forgeries specifically\. Using only the physiological channel, with no texture features, no frequency analysis, no audio, our 165K\-parameter 1D ResNet achieves AUC 0\.806, within 2\.4 points of the best published general\-purpose detector trained on visual features across all forgery categories\. The theoretical argument holds empirically: the absence of a real video substrate in TF synthesis makes rPPG detection considerably stronger here than in face\-swap settings, where prior rPPG detectors were consistently evaluated and where real physiological signal residue confounds the detection task\.

### 6\.2The Real Identity Ceiling

The primary structural limitation of our evaluation is the real identity ceiling of Celeb\-DF\+\+\. Although we use 17,500 fake videos spanning seven methods, all real video comes from only 59 unique celebrities\. This means our subject\-independent protocol, while methodologically sound, is evaluated on a narrow slice of physiological diversity\. The per\-identity AUC std of 0\.033 at full scale \(vs 0\.167 at the original 2,500\-fake scale\) shows that scaling fake volume stabilizes detection substantially, but the real identity bottleneck remains\. Expanding the real video pool through multi\-dataset fusion or collecting additional real data is the most direct path to closing this gap\.

### 6\.3Method\-Dependent Detectability as a Forensic Property

The stable 0\.295 AUC spread across methods is a practically useful finding beyond this paper\. It implies that the forensic difficulty of a TF generation method can be predicted from its pixel synthesis strategy without running detection experiments, where methods that warp source frames are harder than methods that synthesize pixels from scratch\. This provides a principled basis for prioritizing detector development effort and for anticipating which future TF methods will be most resistant to physiological detection\.

### 6\.4Limitations and Future Work

The per\-method evaluation uses a positional assignment of method labels to fake videos based on the known Celeb\-DF\+\+ generation ordering, since method names are not recoverable from filenames\. While this assignment is consistent with the dataset structure, it cannot be independently verified at the video level\. Additionally, the 1:7 real\-to\-fake training imbalance, compensated by weighted BCE, may not be fully resolved at threshold\-dependent metrics such as precision; future work should explore oversampling strategies for the real class\. Furthermore, rPPG extraction is inherently sensitive to melanin concentration, ambient lighting conditions, and skin tone\[[13](https://arxiv.org/html/2607.21776#bib.bib26)\]\. While our subject\-independent protocol prevents cross\-identity physiological leakage, it does not stratify evaluation across demographic groups\. Future work must evaluate detection performance across the Fitzpatrick skin type scale to ensure equitable detection rates and identify whether rPPG\-based detectors exhibit systematic performance disparities correlated with subject skin tone\.

We further acknowledge the dual\-use nature of this work: while our findings advance the detection of talking\-face deepfakes, the detailed characterization of each generator’s rPPG vulnerability profile, particularly the identification of IP\-LAP’s landmark\-guided synthesis as physiologically plausible, could inform adversarial synthesis strategies designed to evade physiological detectors\. We release our findings in the interest of defensive research, consistent with responsible disclosure norms in the security community\.

## 7Conclusion

We have presented the first rPPG\-based detection framework specifically targeting talking\-face deepfake synthesis, evaluated under a rigorous subject\-independent protocol on the Celeb\-DF\+\+ benchmark\. Our 1D ResNet achieves AUC 0\.806 across 18 held\-out celebrity identities, representing a 0\.184\-point improvement over the leading prior rPPG detector reproduced on TF data and placing our method within 2\.4 points of the best published general\-purpose detector while operating exclusively on the physiological signal channel\.

The central finding of this work is that detection difficulty is strongly and stably determined by the pixel synthesis strategy of each TF generation method\. Real3DPortrait, which generates pixels entirely from a learned latent representation, is detected at AUC 0\.985, while IP\-LAP, which preserves the source frame’s identity structure through landmark\-guided attention, resists detection at AUC 0\.690, which is a gap of nearly 30 AUC points across the same detection pipeline\. This spread is consistent across every evaluation protocol we apply, confirming that it reflects a genuine forensic property of each generator rather than an evaluation artifact\. Most strikingly, IP\-LAP is the only method for which isolated per\-method training performs worse than the combined\-method baseline, indicating that its rPPG output is physiologically indistinguishable from real physiology without calibration from other method types\.

These findings establish a methodological foundation for rPPG\-based TF detection and identify IP\-LAP\-style identity\-preserving synthesis as the primary open challenge for physiological forensics\. Future directions include multi\-dataset real video fusion to address the 59\-identity ceiling, adversarial rPPG injection to harden detectors against physiologically plausible synthesis, and fusion of rPPG features with complementary visual or frequency\-domain signals to close the remaining gap to general\-purpose detection performance\.

## References

- \[1\]G\. Boccignoneet al\.\(2022\)Deepfake video detection through optical flow based CNN\.InInt\. Conf\. Pattern Recog\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1)\.
- \[2\]Z\. Chenet al\.\(2024\)EchoMimic: lifelike audio\-driven portrait animations through editable landmark conditions\.arXiv:2407\.08136\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[3\]U\. A\. Ciftci, I\. Demir, and L\. Yin\(2020\)FakeCatcher: detection of synthetic portrait videos using biological signals\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.21776#S2.SS3.p1.1)\.
- \[4\]S\. K\. Datta, S\. Jia, and S\. Lyu\(2025\)Detecting lip\-syncing deepfakes: vision temporal transformer for analyzing mouth inconsistencies\.arXiv:2504\.01470\.Cited by:[§2\.4](https://arxiv.org/html/2607.21776#S2.SS4.p1.1)\.
- \[5\]G\. de Haan and V\. Jeanne\(2013\)Robust pulse rate from chrominance\-based rPPG\.IEEE Transactions on Biomedical Engineering60\(10\),pp\. 2878–2886\.Cited by:[§2\.2](https://arxiv.org/html/2607.21776#S2.SS2.p1.1)\.
- \[6\]A\. Haliassoset al\.\(2021\)Lips don’t lie: a generalisable and robust approach to face forge detection\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.21776#S2.SS4.p1.1)\.
- \[7\]J\. Hernandez\-Ortega, R\. Tolosana, J\. Fierrez, and A\. Morales\(2020\)DeepFakesON\-Phys: DeepFakes detection based on heart rate estimation\.arXiv:2010\.00400\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.21776#S2.SS3.p1.1),[§5\.1](https://arxiv.org/html/2607.21776#S5.SS1.p1.1),[Table 3](https://arxiv.org/html/2607.21776#S5.T3.2.4.2.1)\.
- \[8\]T\. Ki, D\. Min, and G\. Chae\(2025\)FLOAT: generative motion latent flow matching for audio\-driven talking portrait\.InInt\. Conf\. Comput\. Vis\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[9\]Y\. Li, D\. Zhu, X\. Cui, and S\. Lyu\(2025\)Celeb\-DF\+\+: a large\-scale challenging video deepfake benchmark for generalizable forensics\.arXiv:2507\.18015\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.5](https://arxiv.org/html/2607.21776#S2.SS5.p1.1),[§4\.1](https://arxiv.org/html/2607.21776#S4.SS1.p1.1)\.
- \[10\]T\. Liang and X\. Deng\(2021\)Identifying deepfake videos with optical flow and biological signals\.arXiv:2106\.11468\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.21776#S2.SS3.p1.1)\.
- \[11\]T\. Liuet al\.\(2024\)AniTalker: animate vivid and diverse talking faces via identity\-decoupled facial motion encoding\.arXiv:2405\.03121\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[12\]X\. Liuet al\.\(2024\)Lips are lying: spotting the temporal inconsistency between audio and visual in lip\-syncing deepfakes\.arXiv:2401\.15668\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.21776#S2.SS4.p1.1)\.
- \[13\]E\. M\. Nowara, D\. McDuff, and A\. Veeraraghavan\(2020\)A meta\-analysis of the impact of skin tone and gender on non\-contact photoplethysmography measurements\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\. Worksh\.,Cited by:[§6\.4](https://arxiv.org/html/2607.21776#S6.SS4.p1.1)\.
- \[14\]D\. S\. Parket al\.\(2019\)SpecAugment: a simple data augmentation method for automatic speech recognition\.InInterspeech,Cited by:[§3\.5](https://arxiv.org/html/2607.21776#S3.SS5.p1.2)\.
- \[15\]H\. Qiet al\.\(2020\)DeepRhythm: exposing deepfakes with attentional visual heartbeat rhythms\.InACM Int\. Conf\. Multimedia,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.21776#S2.SS3.p1.1)\.
- \[16\]S\. Tanet al\.\(2024\)EDTalk: efficient disentanglement for emotional talking head synthesis\.InEur\. Conf\. Comput\. Vis\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[17\]W\. Verkruysse, L\. O\. Svaasand, and J\. S\. Nelson\(2008\)Remote plethysmographic imaging using ambient light\.Optics Express16\(26\),pp\. 21434–21445\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.21776#S2.SS2.p1.1)\.
- \[18\]W\. Wang, A\. C\. den Brinker, S\. Stuijk, and G\. de Haan\(2017\)Algorithmic principles of remote PPG\.IEEE Transactions on Biomedical Engineering64\(7\),pp\. 1479–1491\.Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.21776#S2.SS2.p1.1)\.
- \[19\]Y\. Wuet al\.\(2023\)Detecting deepfake videos with temporal dropout 3DCNN\.InIJCAI,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.21776#S2.SS3.p1.1)\.
- \[20\]Z\. Yan, J\. Wang, P\. Jin, K\. Zhang, C\. Liu, S\. Chen, T\. Yao, S\. Ding, B\. Wu, and L\. Yuan\(2025\)Orthogonal subspace decomposition for generalizable AI\-generated image detection\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 70268–70288\.Cited by:[§2\.5](https://arxiv.org/html/2607.21776#S2.SS5.p1.1),[§5\.3](https://arxiv.org/html/2607.21776#S5.SS3.p1.1),[Table 3](https://arxiv.org/html/2607.21776#S5.T3.2.5.3.1)\.
- \[21\]Z\. Yeet al\.\(2024\)Real3D\-Portrait: one\-shot realistic 3D talking portrait synthesis\.InInt\. Conf\. Learn\. Represent\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[22\]Z\. Yuet al\.\(2019\)Remote photoplethysmograph signal measurement from facial videos using spatio\-temporal networks\.InBrit\. Mach\. Vis\. Conf\.,Cited by:[§2\.2](https://arxiv.org/html/2607.21776#S2.SS2.p1.1)\.
- \[23\]Z\. Yuet al\.\(2023\)PhysFormer\+\+: facial video\-based physiological measurement with slowfast temporal difference transformer\.Int\. J\. Comput\. Vis\.\.Cited by:[§2\.2](https://arxiv.org/html/2607.21776#S2.SS2.p1.1)\.
- \[24\]W\. Zhanget al\.\(2023\)SadTalker: learning realistic 3D motion coefficients for stylized audio\-driven single image talking face animation\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[25\]Z\. Zhonget al\.\(2023\)Identity\-preserving talking face generation with landmark and appearance priors\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\.,Cited by:[§1](https://arxiv.org/html/2607.21776#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.21776#S2.SS1.p1.1)\.
- \[26\]B\. Zou, Z\. Guo, J\. Chen, and H\. Ma\(2025\)RhythmFormer: extracting patterned rPPG signals based on periodic sparse attention\.Pattern Recognition164\.Cited by:[1st item](https://arxiv.org/html/2607.21776#S1.I1.i1.p1.1),[§2\.2](https://arxiv.org/html/2607.21776#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.21776#S3.SS2.p3.2)\.