Before the Warning Comes Too Late: Incremental Phone-Scam Detection from Speech

# Before the Warning Comes Too Late: Incremental Phone-Scam Detection from Speech
Source: [https://arxiv.org/html/2609.20223](https://arxiv.org/html/2609.20223)
Khang Nhat Hoang Vo1,2Anh Trac Duc Dinh3,4Tai Tien Ta4Tho Quan4

1Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE 2National University of Singapore, Singapore 3Center for AI Reseach \(CAIR\), VinUniversity, Hanoi 4Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology \(HCMUT\), VNU\-HCM, Ho Chi Minh City, Vietnam Correspondence:[Khang\.Vo@mbzuai\.ac\.ae](mailto:[email protected]),[qttho@hcmut\.edu\.vn](mailto:[email protected])

###### Abstract

We study weakly supervised incremental telecom fraud detection from raw telephone audio, where training provides only conversation\-level labels and predictions must be updated before a call ends\. We introduce StreamFraudNet, which processes incoming audio through overlapping bounded\-context windows using a frozen self\-supervised speech encoder, recurrent temporal modeling, and learned aggregation of latent window scores\. On a controlled English benchmark, StreamFraudNet achieves a ROC–AUC of0\.99530\.9953, significantly outperforming acoustic and mean\-pooling baselines while remaining competitive with strong global temporal models\. The model produces its first prediction after 10 seconds of audio, updates every 2 seconds, and operates faster than real time on the evaluated server hardware\. Ablations identify recurrent temporal context as the principal contributor to performance\. These results demonstrate that fraud risk can be scored incrementally from raw speech without transcripts or temporal annotations, while highlighting the need for latency\-aware training to improve early prediction\.

Before the Warning Comes Too Late: Incremental Phone\-Scam Detection from Speech

Khang Nhat Hoang Vo1,2††thanks:Work done while visiting the National University of Singapore\.Anh Trac Duc Dinh3,4Tai Tien Ta4Tho Quan41Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE2National University of Singapore, Singapore3Center for AI Reseach \(CAIR\), VinUniversity, Hanoi4Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology\(HCMUT\), VNU\-HCM, Ho Chi Minh City, VietnamCorrespondence:[Khang\.Vo@mbzuai\.ac\.ae](mailto:[email protected]),[qttho@hcmut\.edu\.vn](mailto:[email protected])

## 1Introduction

Phone\-scam detection is inherently time\-sensitive\. Recent LLM\-based systems analyze ongoing calls and issue warnings before harmful actions are completed\(Shenet al\.,[2025a](https://arxiv.org/html/2609.20223#bib.bib29),[b](https://arxiv.org/html/2609.20223#bib.bib30); Alhulifiet al\.,[2025](https://arxiv.org/html/2609.20223#bib.bib47); Nicholas and Ng,[2024](https://arxiv.org/html/2609.20223#bib.bib48)\)\. This motivates a setting beyond post\-call classification: estimating fraud risk from a partially observed conversation and updating that estimate as new evidence arrives\. The need is substantial\. Global telecom fraud losses reached USD38\.9538\.95billion in 2023\(Communications Fraud Control Association,[2023](https://arxiv.org/html/2609.20223#bib.bib26)\), while recent work shows how language models, speech recognition, and speech synthesis can be combined into scalable scam pipelines\(Gresselet al\.,[2024](https://arxiv.org/html/2609.20223#bib.bib28); United Nations Office on Drugs and Crime,[2025](https://arxiv.org/html/2609.20223#bib.bib27)\)\.

Existing conversational fraud detectors commonly rely on ASR transcripts, large audio–language models, or richer reasoning\-oriented supervision\(Wanget al\.,[2026](https://arxiv.org/html/2609.20223#bib.bib32); Erlacher,[2025](https://arxiv.org/html/2609.20223#bib.bib45); Chenet al\.,[2026](https://arxiv.org/html/2609.20223#bib.bib46); Shenet al\.,[2025a](https://arxiv.org/html/2609.20223#bib.bib29),[b](https://arxiv.org/html/2609.20223#bib.bib30); Maet al\.,[2025](https://arxiv.org/html/2609.20223#bib.bib31)\)\. These approaches can model semantic content effectively, but may require an intermediate transcription stage, high\-capacity models, or fine\-grained annotations\. We instead study whether fraud risk can be updated directly from raw telephone audio using only binary conversation\-level labels\.

This setting is weakly supervised because the model is told whether a call is fraudulent, but not when fraudulent evidence appears\. It is related to multiple\-instance learning\(Ilseet al\.,[2018](https://arxiv.org/html/2609.20223#bib.bib35)\), where a collection of instances receives one global label\. However, standard multiple\-instance aggregation is often permutation invariant, whereas conversational evidence is ordered and may accumulate across utterances\. A suitable model must therefore preserve local temporal structure while learning from coarse conversation\-level supervision\.

Figure[1](https://arxiv.org/html/2609.20223#S1.F1)illustrates this challenge\. A fraudulent call may contain both suspicious and apparently benign exchanges, with informative evidence appearing only at particular moments\. A single post\-call prediction cannot describe how the estimated risk changes as the conversation unfolds\. We therefore introduceStreamFraudNet, a bounded\-context model that produces an initial fraud score from the first observed window and updates the conversation\-level prediction as additional audio arrives\.

![Refer to caption](https://arxiv.org/html/2609.20223v1/x1.png)Figure 1:An SSN scam example in which high latent fraud scores occur only for selected utterances\. The scores illustrate how estimated risk can vary across a conversation, but should not be interpreted as supervised fraud\-span annotations\.StreamFraudNet divides incoming audio into overlapping bounded\-context windows\. A frozen Wav2Vec2 encoder\(Baevskiet al\.,[2020](https://arxiv.org/html/2609.20223#bib.bib33)\)extracts frame\-level speech representations, attention pooling forms chunk\-level vectors, and a recurrent module captures temporal dependencies within each observed window\. A learned aggregator then combines all window scores available so far into the current conversation\-level prediction\. Because no temporal labels are provided, the intermediate outputs are treated as latent risk scores rather than verified localizations of fraudulent content\.

Our contributions are as follows\.

- •We formulate telecom fraud detection as weakly supervised incremental scoring from raw audio using only conversation\-level labels\.
- •We propose StreamFraudNet, a bounded\-context architecture with a frozen speech encoder, recurrent temporal modeling, and incremental aggregation\. Its0\.9530\.953M\-parameter task head produces the first score after 10 seconds and updates it every 2 seconds\.
- •Through controlled multi\-seed experiments, we show that recurrent temporal context is the principal architectural contributor\. StreamFraudNet remains competitive with strong temporal baselines and operates faster than real time, while bounded windows primarily enable incremental operation rather than higher final\-call accuracy\.

## 2Related Work

### 2\.1Metadata\- and Graph\-Based Fraud Detection

Much telecom fraud research relies on Call Detail Records and interaction metadata, including call duration, frequency, timing, and communication structure\(Prasadet al\.,[2020](https://arxiv.org/html/2609.20223#bib.bib36); Huet al\.,[2024](https://arxiv.org/html/2609.20223#bib.bib37); Caoet al\.,[2024](https://arxiv.org/html/2609.20223#bib.bib38)\)\. Graph\-based methods represent users, calls, or transactions as relational structures and learn patterns associated with fraudulent behavior\.Huet al\.\([2024](https://arxiv.org/html/2609.20223#bib.bib37)\)introduced cost\-sensitive graph learning for telecom fraud detection, whileCaoet al\.\([2024](https://arxiv.org/html/2609.20223#bib.bib38)\)combined graph contrastive learning with adaptive augmentation\.

Related transaction\-fraud methods model class imbalance, neighborhood structure, and temporal dependencies\(Tianet al\.,[2024](https://arxiv.org/html/2609.20223#bib.bib39); Tian and Liu,[2024](https://arxiv.org/html/2609.20223#bib.bib40); Xieet al\.,[2024](https://arxiv.org/html/2609.20223#bib.bib41)\)\. Although these studies demonstrate the value of relational and temporal information, they operate on metadata or transaction graphs rather than spoken conversations\. They therefore motivate temporal modeling broadly but are not direct baselines for raw\-audio fraud detection\.

### 2\.2Acoustic and Conversational Fraud Detection

Earlier audio\-based work focused on acoustic and spectro\-temporal properties of recorded calls\.Prasadet al\.\([2020](https://arxiv.org/html/2609.20223#bib.bib36)\)combined audio and metadata to characterize robocalls, whileElizalde and Emmanouilidou \([2021](https://arxiv.org/html/2609.20223#bib.bib42)\)evaluated handcrafted acoustic descriptors and spectrogram\-based classifiers for robocall and voicemail\-spam detection\. These studies establish that call audio contains discriminative information, but primarily address recorded\-message classification rather than fraud evidence that develops throughout an interactive conversation\.

A complementary direction models conversational semantics\.Shenet al\.\([2025a](https://arxiv.org/html/2609.20223#bib.bib29)\)examined the capabilities and limitations of LLM\-based phone\-scam detection, whileShenet al\.\([2025b](https://arxiv.org/html/2609.20223#bib.bib30)\)studied real\-time warnings from partially observed call transcripts\. Such systems can reason over linguistic content as a call unfolds, but depend on automatic speech recognition and operate mainly in the textual domain\.

More recent work uses end\-to\-end audio–language models\.Maet al\.\([2025](https://arxiv.org/html/2609.20223#bib.bib31)\)introduced TeleAntiFraud\-28K and adapted Qwen2\-Audio for telecom fraud classification and reasoning\.Wanget al\.\([2026](https://arxiv.org/html/2609.20223#bib.bib32)\)subsequently proposed audio–text fraud detection with reinforcement learning and dynamic risk assessment\. These approaches use high\-capacity audio–language models and reasoning\-oriented supervision\. Our setting is complementary: we study bounded\-context fraud scoring directly from raw audio using a compact trainable head and only binary conversation\-level labels\.

### 2\.3Weakly Supervised Temporal Modeling

Our supervision setting is related to multiple\-instance learning, where a set of instances receives a single bag\-level label\.Ilseet al\.\([2018](https://arxiv.org/html/2609.20223#bib.bib35)\)introduced attention\-based aggregation that learns instance importance from global supervision\. Standard multiple\-instance models are typically permutation invariant\(Leeet al\.,[2019](https://arxiv.org/html/2609.20223#bib.bib49); Rymarczyket al\.,[2021](https://arxiv.org/html/2609.20223#bib.bib50)\), however, and do not explicitly represent the order in which conversational evidence appears\.

Weakly supervised sound event detection similarly learns frame\- or segment\-level scores from clip\-level labels\.Miyazakiet al\.\([2020](https://arxiv.org/html/2609.20223#bib.bib43)\)applied self\-attention in this setting, whileDeshmukhet al\.\([2021](https://arxiv.org/html/2609.20223#bib.bib44)\)introduced auxiliary self\-supervised objectives for learning from coarse annotations\. Sound events are often temporally localized acoustic phenomena\(Kotuset al\.,[2014](https://arxiv.org/html/2609.20223#bib.bib51); Politiset al\.,[2021](https://arxiv.org/html/2609.20223#bib.bib52)\), whereas telecom fraud may depend on evidence distributed across multiple utterances\. StreamFraudNet therefore combines conversation\-level supervision with ordered, bounded\-context temporal modeling\. Because timestamped fraud annotations are unavailable, its window outputs are treated as latent risk scores rather than verified event localizations\.

## 3Proposed Methodology

### 3\.1Problem Formulation

Let𝐱=\(x1,…,xC\)\\mathbf\{x\}=\(x\_\{1\},\\ldots,x\_\{C\}\)denote a telephone conversation divided intoCCfixed\-duration audio chunks, where eachxc∈ℝTx\_\{c\}\\in\\mathbb\{R\}^\{T\}containsTTwaveform samples\. Each conversation has a binary labely∈\{0,1\}y\\in\\\{0,1\\\}, indicating a legitimate or fraudulent call\. During training, only this conversation\-level label is available; no chunk\-, utterance\-, or timestamp\-level annotations are provided\.

We formulate the task as weakly supervised incremental fraud scoring\. At timett, letCtC\_\{t\}denote the number of chunks observed so far\. The observed audio is divided into overlapping windows ofkkchunks with stridess\. Theii\-th complete window is

𝐰i=\(xi​s\+1,xi​s\+2,…,xi​s\+k\)\.\\mathbf\{w\}\_\{i\}=\\left\(x\_\{is\+1\},x\_\{is\+2\},\\ldots,x\_\{is\+k\}\\right\)\.\(1\)The valid indices arei∈\{0,…,Nt−1\}i\\in\\\{0,\\ldots,N\_\{t\}\-1\\\}, where the number of complete windows available at timettis

Nt=⌊Ct−ks⌋\+1,N\_\{t\}=\\left\\lfloor\\frac\{C\_\{t\}\-k\}\{s\}\\right\\rfloor\+1,\(2\)whereCt≥kC\_\{t\}\\geq k\. At the end of the conversation,Ct=CC\_\{t\}=C, and the total number of windows isN=⌊\(C−k\)/s⌋\+1N=\\lfloor\(C\-k\)/s\\rfloor\+1\.

A window detectorfθf\_\{\\theta\}maps each observed window to a latent fraud score:

y^i=fθ​\(𝐰i\)∈\[0,1\]\.\\hat\{y\}\_\{i\}=f\_\{\\theta\}\(\\mathbf\{w\}\_\{i\}\)\\in\[0,1\]\.\(3\)
The scores available at timettform the ordered sequence𝐲^\(t\)=\(y^0,…,y^Nt−1\)\\hat\{\\mathbf\{y\}\}^\{\(t\)\}=\(\\hat\{y\}\_\{0\},\\ldots,\\hat\{y\}\_\{N\_\{t\}\-1\}\)are combined into the current predictiony^global\(t\)=gψ​\(𝐲^\(t\)\)\\hat\{y\}\_\{\\mathrm\{global\}\}^\{\(t\)\}=g\_\{\\psi\}\(\\hat\{\\mathbf\{y\}\}^\{\(t\)\}\)\. As new chunks arrive, additional windows become available andy^global\(t\)\\hat\{y\}\_\{\\mathrm\{global\}\}^\{\(t\)\}is updated\. Because temporal annotations are unavailable, the individualy^i\\hat\{y\}\_\{i\}values are treated as latent risk scores rather than supervised localizations of fraudulent content\.

### 3\.2Model Architecture

StreamFraudNet contains a bounded\-context window detector and an incremental aggregator\. The detector combines a frozen speech encoder, attention pooling, a BiLSTM, and a window classifier, while the aggregator combines all window scores observed so far\. Figure[2](https://arxiv.org/html/2609.20223#S3.F2)summarizes the model\.

![Refer to caption](https://arxiv.org/html/2609.20223v1/x2.png)Figure 2:Overview of StreamFraudNet\. StreamFraudNet processes overlappingkk\-chunk windows using a frozen Wav2Vec2 encoder, attention pooling, and a BiLSTM\. The resulting latent window scores are incrementally aggregated into the current call\-level score\.#### 3\.2\.1Bounded\-Context Window Detector

For theii\-th window, letxi,j=xi​s\+jx\_\{i,j\}=x\_\{is\+j\}denote itsjj\-th chunk, withj∈\{1,…,k\}j\\in\\\{1,\\ldots,k\\\}\. A frozen Wav2Vec2 encoderℰϕ\\mathcal\{E\}\_\{\\phi\}extracts frame\-level representations:

𝐅i,j=ℰϕ​\(xi,j\)∈ℝL×D,\\mathbf\{F\}\_\{i,j\}=\\mathcal\{E\}\_\{\\phi\}\(x\_\{i,j\}\)\\in\\mathbb\{R\}^\{L\\times D\},\(4\)whereLLis the number of encoder frames andDDis the representation dimension\. Wav2Vec2 provides self\-supervised speech representations without requiring transcripts at inference time\(Baevskiet al\.,[2020](https://arxiv.org/html/2609.20223#bib.bib33)\)\. We freeze the encoder to reduce trainable parameters and training\-time memory\.

Frame\-level attention pooling converts the variable\-length encoder output into a fixed\-dimensional chunk representation\. The normalized attention weight for frameℓ\\ellis

ai,j,ℓ=exp⁡\(𝐮⊤​𝐅i,j,ℓ\)∑m=1Lexp⁡\(𝐮⊤​𝐅i,j,m\)a\_\{i,j,\\ell\}=\\frac\{\\exp\\left\(\\mathbf\{u\}^\{\\top\}\\mathbf\{F\}\_\{i,j,\\ell\}\\right\)\}\{\\sum\_\{m=1\}^\{L\}\\exp\\left\(\\mathbf\{u\}^\{\\top\}\\mathbf\{F\}\_\{i,j,m\}\\right\)\}\(5\)where𝐮∈ℝD\\mathbf\{u\}\\in\\mathbb\{R\}^\{D\}is a trainable attention vector\. The pooled chunk representation is

𝐜i,j=∑ℓ=1Lai,j,ℓ​𝐅i,j,ℓ\.\\mathbf\{c\}\_\{i,j\}=\\sum\_\{\\ell=1\}^\{L\}a\_\{i,j,\\ell\}\\mathbf\{F\}\_\{i,j,\\ell\}\.\(6\)
Thekkpooled chunk representations form an ordered window sequence:

𝐂i=\(𝐜i,1,…,𝐜i,k\)∈ℝk×D\.\\mathbf\{C\}\_\{i\}=\\left\(\\mathbf\{c\}\_\{i,1\},\\ldots,\\mathbf\{c\}\_\{i,k\}\\right\)\\in\\mathbb\{R\}^\{k\\times D\}\.\(7\)
A bidirectional LSTM models temporal dependencies among chunks within the current window:

𝐇i=BiLSTM⁡\(𝐂i\)∈ℝk×2​H\\mathbf\{H\}\_\{i\}=\\operatorname\{BiLSTM\}\(\\mathbf\{C\}\_\{i\}\)\\in\\mathbb\{R\}^\{k\\times 2H\}\(8\)whereHHis the hidden dimension in each direction\. Bidirectionality is restricted to the current observed window\. The detector may use all chunks within𝐰i\\mathbf\{w\}\_\{i\}, but it cannot access audio from future windows\.

A second attention layer summarizes the contextualized chunk representations\. The normalized weight assigned to chunkjjis

bi,j=exp⁡\(𝐯⊤​𝐇i,j\)∑r=1kexp⁡\(𝐯⊤​𝐇i,r\)b\_\{i,j\}=\\frac\{\\exp\\left\(\\mathbf\{v\}^\{\\top\}\\mathbf\{H\}\_\{i,j\}\\right\)\}\{\\sum\_\{r=1\}^\{k\}\\exp\\left\(\\mathbf\{v\}^\{\\top\}\\mathbf\{H\}\_\{i,r\}\\right\)\}\(9\)where𝐯∈ℝ2​H\\mathbf\{v\}\\in\\mathbb\{R\}^\{2H\}is a trainable attention vector\. The resulting window representation is

𝐡i=∑j=1kbi,j​𝐇i,j\.\\mathbf\{h\}\_\{i\}=\\sum\_\{j=1\}^\{k\}b\_\{i,j\}\\mathbf\{H\}\_\{i,j\}\.\(10\)
A multilayer perceptron maps the window representation to a latent fraud score:

y^i=σ​\(MLP⁡\(𝐡i\)\)\.\\hat\{y\}\_\{i\}=\\sigma\\left\(\\operatorname\{MLP\}\(\\mathbf\{h\}\_\{i\}\)\\right\)\.\(11\)
Applying the detector to all complete windows available at timettyields the ordered score sequence𝐲^\(t\)=\(y^0,…,y^Nt−1\)\\hat\{\\mathbf\{y\}\}^\{\(t\)\}=\(\\hat\{y\}\_\{0\},\\ldots,\\hat\{y\}\_\{N\_\{t\}\-1\}\), which is passed to the incremental aggregator\.

#### 3\.2\.2Incremental Conversation\-Level Aggregator

At each inference step, the aggregator combines all window scores observed up to timett\. The normalized aggregation weight for windowiiis

γi\(t\)=exp⁡\(q​y^i\)∑r=0Nt−1exp⁡\(q​y^r\)\\gamma\_\{i\}^\{\(t\)\}=\\frac\{\\exp\\left\(q\\hat\{y\}\_\{i\}\\right\)\}\{\\sum\_\{r=0\}^\{N\_\{t\}\-1\}\\exp\\left\(q\\hat\{y\}\_\{r\}\\right\)\}\(12\)whereq∈ℝq\\in\\mathbb\{R\}is a trainable scalar\. The current conversation\-level prediction is

y^global\(t\)=∑i=0Nt−1γi\(t\)​y^i\.\\hat\{y\}\_\{\\mathrm\{global\}\}^\{\(t\)\}=\\sum\_\{i=0\}^\{N\_\{t\}\-1\}\\gamma\_\{i\}^\{\(t\)\}\\hat\{y\}\_\{i\}\.\(13\)
When a new complete window becomes available, its score is appended to𝐲^\(t\)\\hat\{\\mathbf\{y\}\}^\{\(t\)\}\. The aggregation weights and conversation\-level prediction are then recomputed using all windows observed so far\. The aggregator therefore produces updated fraud scores during the conversation and does not require the call to end\. The score\-dependent weighting allows different windows to contribute unequally to the current prediction\.

### 3\.3Weakly Supervised Training

During training, all complete windows from a conversation are processed\. Lettendt\_\{\\mathrm\{end\}\}denote the end of the conversation, such thatNtend=NN\_\{t\_\{\\mathrm\{end\}\}\}=N\. The final training prediction isy^global=y^global\(tend\)\\hat\{y\}\_\{\\mathrm\{global\}\}=\\hat\{y\}\_\{\\mathrm\{global\}\}^\{\(t\_\{\\mathrm\{end\}\}\)\}\.

All trainable components are optimized jointly using the conversation\-level binary cross\-entropy objective:

ℒBCE=−y​log⁡\(y^global\)−\(1−y\)​log⁡\(1−y^global\)\.\\mathcal\{L\}\_\{\\mathrm\{BCE\}\}=\-y\\log\\left\(\\hat\{y\}\_\{\\mathrm\{global\}\}\\right\)\-\(1\-y\)\\log\\left\(1\-\\hat\{y\}\_\{\\mathrm\{global\}\}\\right\)\.\(14\)
Gradients propagate through the conversation\-level aggregator, window classifier, BiLSTM, and attention modules, while the pretrained speech encoder remains fixed\. No auxiliary objective is applied to individual window scores, intermediate conversation prefixes, or temporal locations\.

The model is therefore trained to discriminate complete conversations using only conversation\-level labels\. Intermediate predictions are available because the same aggregator can be applied to any observed prefix, but these predictions are not explicitly optimized for early detection\. Likewise, the training objective does not guarantee that an individual window score corresponds to a temporally localized fraud event\.

### 3\.4Incremental Inference and Efficiency

After the firstkkchunks have been observed, StreamFraudNet produces one window score and its first conversation\-level prediction\. The prediction is then updated wheneverssadditional chunks arrive\. For chunk durationΔ\\Delta, the initial observation duration isk​Δk\\Delta, and the update interval iss​Δs\\Delta\.

Under the default configurationk=5k=5,s=1s=1, andΔ=2\\Delta=2seconds, the first prediction is produced after 10 seconds of audio and updated every 2 seconds thereafter\. At each update, the aggregator recomputes the conversation\-level score using all complete windows observed up to that point\.

StreamFraudNet is incremental at the window level rather than strictly causal at the frame level\. The BiLSTM uses both temporal directions within a complete observed window, but neither the window detector nor the aggregator can access future, unheard audio\.

We measure end\-to\-end computational throughput using the real\-time factor

RTF=tprocessingtaudio,\\mathrm\{RTF\}=\\frac\{t\_\{\\mathrm\{processing\}\}\}\{t\_\{\\mathrm\{audio\}\}\},\(15\)wheretprocessingt\_\{\\mathrm\{processing\}\}is the time required to decode, preprocess, encode, and classify the audio, andtaudiot\_\{\\mathrm\{audio\}\}is its duration\. An RTF below11indicates that the complete inference pipeline processes audio faster than it is received\.

RTF measures computational throughput rather than fraud\-onset detection delay\. Fraud\-onset delay also depends on when discriminative evidence first appears and cannot be measured directly without timestamped fraud annotations\.

## 4Experimental Evaluation

### 4\.1Datasets and Experimental Setup

##### Datasets\.

We evaluate on two conversation\-level fraud benchmarks\. The synthetic English benchmark is constructed from the Scam Dialogue Dataset111[https://huggingface\.co/datasets/BothBosu/scam\-dialogue](https://huggingface.co/datasets/BothBosu/scam-dialogue)\. Each dialogue is synthesized with Edge\-TTS using two voices sampled from a pool of 47 English speakers \(Figure[3](https://arxiv.org/html/2609.20223#S4.F3)\)\. The dataset contains 1,600 balanced conversations, divided into 1,280 training and 320 held\-out test examples\. Each dialogue is synthesized with Edge\-TTS using two voices sampled from 47 English\-speaking speakers; their accent\-locale distribution is reported in Figure[3](https://arxiv.org/html/2609.20223#S4.F3)\.

![Refer to caption](https://arxiv.org/html/2609.20223v1/images/voice_distribution_pie.png)Figure 3:Distribution of accent locales across the 47 English\-speaking voices used to synthesize the controlled English benchmark\.We additionally evaluate on TeleAntiFraud\-28K, a Mandarin audio\-text benchmark containing 28,511 telephone conversations with conversation\-level fraud labelsMaet al\.\([2025](https://arxiv.org/html/2609.20223#bib.bib31)\)\. We follow its official split of 21,490 training and 7,021 test conversations\. Table[1](https://arxiv.org/html/2609.20223#S4.T1)summarizes both datasets\.

Table 1:Dataset statistics\.
##### Model configuration\.

All recordings are converted to mono and resampled to 16 kHz\. For the synthetic benchmark, we use a frozen Wav2Vec2\-Base encoderBaevskiet al\.\([2020](https://arxiv.org/html/2609.20223#bib.bib33)\)\. Conversations are divided into 2\-second chunks and truncated or padded to 47 chunks\. StreamFraudNet uses five\-chunk windows, stride one, a single\-layer BiLSTM with 128 hidden units per direction, and dropout 0\.2\. The resulting local context spans 10 seconds, with score updates every 2 seconds\.

For TeleAntiFraud\-28K, we use a frozen Mandarin Wav2Vec2\-Large\-XLSRConneauet al\.\([2021](https://arxiv.org/html/2609.20223#bib.bib54)\)encoder, retain at most 30 chunks, and set the stride to two\. The task\-specific architecture is otherwise unchanged\.

##### Training and evaluation\.

Models are trained with AdamW and binary cross\-entropy for up to 25 epochs, using cosine learning\-rate decay and early stopping on validation F1\. Learning rates are selected separately under the same validation protocol\. Experiments use NVIDIA GPUs with 40 GB memory and a 12\-core CPU; end\-to\-end inference results are reported in Section[4\.9](https://arxiv.org/html/2609.20223#S4.SS9)\.

Controlled experiments use seeds 13, 37, and 73, with mean and sample standard deviation reported across runs\. Three\-seed ensembles average prediction scores and use validation\-selected thresholds\. We estimate confidence intervals using 10,000 label\-stratified bootstrap samples and apply paired bootstrap tests for ROC–AUC and F1, exact McNemar tests for accuracy, and Holm correction within each test family\.

##### Baselines\.

We compare StreamFraudNet with three groups of baselines\. Acoustic controls use call duration or log\-Mel features with linear and temporal CNN classifiers, testing whether simple signal\-level cues are sufficient\. Representation baselines apply mean pooling, Transformer attention, or BiLSTM attention to frozen Wav2Vec2 features, isolating the contribution of temporal modeling\. We additionally include Wav2Vec2\-Large as an encoder capacity control and WavLM\-Base\+ with the matched BiLSTM\-attention head\(Chenet al\.,[2022](https://arxiv.org/html/2609.20223#bib.bib34)\)to assess sensitivity to the pretrained speech encoder\.

### 4\.2Conversation\-Level Classification

ModelHead params\.ROC–AUCF1Ens\. AUC95% CIAcoustic controlsDuration mean–linear†20\.6236±0\.40680\.6236\\pm 0\.40680\.7235±0\.04920\.7235\\pm 0\.04920\.8585\[0\.8174,0\.8958\]\[0\.8174,0\.8958\]Log\-Mel mean–linear†1610\.6355±0\.01680\.6355\\pm 0\.01680\.7019±0\.00580\.7019\\pm 0\.00580\.6402\[0\.5778,0\.7002\]\[0\.5778,0\.7002\]Log\-Mel temporal CNN†127K0\.8718±0\.01580\.8718\\pm 0\.01580\.7943±0\.01490\.7943\\pm 0\.01490\.8795\[0\.8406,0\.9143\]\[0\.8406,0\.9143\]Wav2Vec2 controlsWav2Vec2 mean–linear†7690\.7934±0\.01460\.7934\\pm 0\.01460\.7295±0\.02350\.7295\\pm 0\.02350\.8020\[0\.7544,0\.8477\]\[0\.7544,0\.8477\]Wav2Vec2\-Large mean–linear†1\.03K0\.7244±0\.04010\.7244\\pm 0\.04010\.7189±0\.01060\.7189\\pm 0\.01060\.6830\[0\.6221,0\.7407\]\[0\.6221,0\.7407\]Wav2Vec2 Transformer\-attention7\.98M0\.9784±0\.01160\.9784\\pm 0\.01160\.9525±0\.01720\.9525\\pm 0\.01720\.9818\[0\.9654,0\.9944\]\[0\.9654,0\.9944\]Wav2Vec2 BiLSTM\-attention0\.953M0\.9990±0\.0008\\mathbf\{0\.9990\\pm 0\.0008\}0\.9833±0\.0066¯\\underline\{0\.9833\\pm 0\.0066\}0\.9992¯\\underline\{0\.9992\}\[0\.9975,1\.0000\]\[0\.9975,1\.0000\]Alternative encoder and proposed modelWavLM BiLSTM\-attention‡0\.953M0\.9990±0\.0009\\mathbf\{0\.9990\\pm 0\.0009\}0\.9834±0\.0049\\mathbf\{0\.9834\\pm 0\.0049\}0\.9998\\mathbf\{0\.9998\}\[0\.9994,1\.0000\]\\mathbf\{\[0\.9994,1\.0000\]\}StreamFraudNet0\.953M0\.9953±0\.0022¯\\underline\{0\.9953\\pm 0\.0022\}0\.9656±0\.01360\.9656\\pm 0\.01360\.9962\[0\.9916,0\.9994\]\[0\.9916,0\.9994\]

- •Best values are shown in bold and second\-best distinct values are underlined\. Head parameters exclude the frozen speech encoder\.
- †\\daggerSignificantly lower ROC–AUC and F1 than StreamFraudNet after Holm correction\.
- ‡\\ddaggerSignificantly higher ensemble ROC–AUC than StreamFraudNet after Holm correction\.

Table 2:Conversation\-level performance on the synthetic test set\. Results are mean±\\pmsample standard deviation over three seeds\.Table[2](https://arxiv.org/html/2609.20223#S4.T2)compares StreamFraudNet with acoustic, representation, and temporal baselines\. StreamFraudNet significantly outperforms the acoustic and mean\-pooling controls in both ROC–AUC and F1 \(pHolm=0\.0044p\_\{\\mathrm\{Holm\}\}=0\.0044\)\. It also uses approximately 0\.95 million task\-head parameters, compared with 7\.98 million for Wav2Vec2 Transformer\-attention, while obtaining higher mean ROC–AUC and F1\. The difference between these two models, however, is not significant after Holm correctionHolm \([1979](https://arxiv.org/html/2609.20223#bib.bib53)\)\.

StreamFraudNet has essentially the same task\-head size as the Wav2Vec2 and WavLM BiLSTM\-attention baselines\. Its difference from Wav2Vec2 BiLSTM\-attention is not significant, whereas WavLM BiLSTM\-attention obtains a significantly higher ensemble ROC–AUC, with a difference of0\.003630\.00363\(95%​CI=\[0\.00059,0\.00801\]95\\%\\ \\mathrm\{CI\}=\[0\.00059,0\.00801\],pHolm=0\.0128p\_\{\\mathrm\{Holm\}\}=0\.0128\)\. These results show that StreamFraudNet provides competitive performance with a compact task\-specific head and bounded\-context incremental operation, although it does not achieve the strongest offline classification result\. Complete pairwise tests and secondary metrics are reported in Appendix[A](https://arxiv.org/html/2609.20223#A1)\.

### 4\.3Component Analysis

StreamFraudNet combines bounded windows, recurrent modeling, attention pooling, and a nonlinear classifier\. To identify which choices are essential, we remove or simplify one component at a time under the same evaluation protocol, and present the results at Table[3](https://arxiv.org/html/2609.20223#S4.T3)\.

Table 3:Component analysis on the synthetic test set\. Performance is reported as mean±\\pmsample standard deviation across three seeds\. Paired effects are included only when significant after Holm correction; “n\.s\.” denotes no significant difference\.Removing recurrent context causes the only significant degradation, reducing ROC–AUC by0\.027970\.02797and F1 by0\.076370\.07637\(pHolm=0\.0044p\_\{\\mathrm\{Holm\}\}=0\.0044; Table[3](https://arxiv.org/html/2609.20223#S4.T3)\)\. Bidirectionality, attention pooling, and the nonlinear head provide no individually significant gain\. The global single\-window variant is numerically stronger but not significantly different, indicating that bounded windows primarily enable frequent updates rather than better offline classification\.

### 4\.4Speech Encoder Training Policy

Freezing the speech encoder reduces training cost but may limit adaptation to fraud\-specific cues\. We therefore compare frozen, partially fine\-tuned, and fully fine\-tuned encoders in Table[4](https://arxiv.org/html/2609.20223#S4.T4)\.

Table 4:Effect of the speech\-encoder training policy\. Performance is reported as mean±\\pmsample standard deviation across three seeds\. Peak memory denotes training\-time GPU memory\.Fine\-tuning the final two layers provides no performance gain while increasing peak memory from 0\.681 to 2\.586 GiB\. Full fine\-tuning requires substantially more memory and is sensitive to the learning rate:10−510^\{\-5\}is unstable, while10−610^\{\-6\}remains below the frozen configuration\.

The frozen ensemble exceeds stable full fine\-tuning by 0\.00949 ROC–AUC \(95%​CI=\[0\.00246,0\.01809\]95\\%\\ \\mathrm\{CI\}=\[0\.00246,0\.01809\],p=0\.0034p=0\.0034\)\. Freezing is therefore justified by its stronger stability–cost trade\-off, although the encoder remains part of the inference pipeline\.

### 4\.5Hyperparameter Robustness

Window size and stride control the amount of local context and the update frequency, while hidden dimension and dropout affect capacity and regularization\. We vary each factor independently to determine whether the results depend on a narrow configuration \(Table[5](https://arxiv.org/html/2609.20223#S4.T5)\)\.

Table 5:Hyperparameter sensitivity on the synthetic test set\. Values are mean±\\pmsample standard deviation across three seeds\. No tested variant differs significantly from the default after Holm correction\.No tested variant differs significantly from the default after Holm correction\. Performance is particularly stable across hidden dimensions and dropout values, while temporal settings produce larger numerical variation\. Stride three gives the highest mean scores but updates only every 6 seconds\. We retain stride one because it provides 2\-second updates while remaining statistically competitive, prioritizing responsiveness over the strongest offline mean\.

### 4\.6Incremental Prediction

Final\-call metrics do not show whether useful predictions are available during an ongoing conversation\. We therefore evaluate progressively longer audio prefixes, beginning at 10 seconds when the first complete window is available \(Table[6](https://arxiv.org/html/2609.20223#S4.T6)\)\.

Table 6:Performance as a function of observed audio duration\. Values are mean±\\pmsample standard deviation across three seeds\.StreamFraudNet reaches a ROC–AUC of0\.9532±0\.01780\.9532\\pm 0\.0178after 10 seconds and0\.9842±0\.00600\.9842\\pm 0\.0060after 20 seconds, showing that useful predictions can be produced before the call ends\.

The global BiLSTM remains numerically stronger at every prefix\. Thus, the results support partial\-conversation fraud scoring, but not a distinct early\-detection advantage from bounded\-window processing alone\.

### 4\.7Label Efficiency

To test whether the model remains effective with limited coarse supervision, we train StreamFraudNet and the matched global BiLSTM using progressively smaller subsets of the conversation\-level labels\. Table[7](https://arxiv.org/html/2609.20223#S4.T7)summarizes the results; full mean and standard deviation values are provided in Appendix[A\.6](https://arxiv.org/html/2609.20223#A1.SS6)\.

Table 7:Label efficiency\. Entries report mean ROC–AUC/F1 across three seeds\.Performance improves consistently with additional labels, and StreamFraudNet reaches a ROC–AUC of 0\.9739 using half of the training set\. The global BiLSTM remains stronger at every label budget, indicating that StreamFraudNet can learn from reduced conversation\-level supervision but is not more label\-efficient than the matched global model\.

### 4\.8External Evaluation

The controlled English benchmark uses balanced synthetic speech\. To assess performance beyond this setting, we evaluate on the larger Mandarin TeleAntiFraud\-28K benchmarkMaet al\.\([2025](https://arxiv.org/html/2609.20223#bib.bib31)\)\. Published Qwen2\-Audio results provide contextual reference, although the systems use different modalities and supervision \(Table[8](https://arxiv.org/html/2609.20223#S4.T8)\)\.

Table 8:Results on TeleAntiFraud\-28K\. Qwen2\-Audio results are taken from the original publication\. Best results are shown in bold\.StreamFraudNet achieves 0\.7131 accuracy and 0\.7058 F1\. Its performance is close to Qwen2\-Audio ASR\-text despite operating directly on audio with only binary conversation labels\. Qwen2\-Audio Think remains substantially stronger, indicating the benefit of larger audio\-language models and reasoning supervision\.

These results support cross\-language and cross\-dataset applicability, but should not be interpreted as a controlled model ranking\. Full precision and recall results are reported in Appendix[A\.4](https://arxiv.org/html/2609.20223#A1.SS4)\.

### 4\.9End\-to\-End Efficiency

Incremental detection requires the complete pipeline, rather than only the task head, to process audio faster than it arrives\. We therefore measure latency from raw WAV input through preprocessing, encoder inference, classification, and aggregation \(Table[9](https://arxiv.org/html/2609.20223#S4.T9)\)\.

Table 9:End\-to\-end inference efficiency on server\.Both evaluated server configurations achieve an RTF below11, demonstrating faster\-than\-real\-time end\-to\-end processing\. These measurements establish processing throughput rather than mobile deployability or fraud\-onset latency\. Additional CPU\-only results are reported in Appendix[A\.7](https://arxiv.org/html/2609.20223#A1.SS7)\. These measurements establish server\-side throughput, not mobile deployment or fraud\-onset latency\. Model footprint, memory use, and detailed timing settings are provided in Appendix[A\.5](https://arxiv.org/html/2609.20223#A1.SS5)\.

## 5Conclusion

We introduced StreamFraudNet for weakly supervised incremental fraud scoring from raw telephone audio\. The model combines a frozen speech encoder with bounded\-context recurrent modeling and requires only conversation\-level labels\. Its main advantage is fixed\-context, frequently updated scoring with a compact task\-specific head and modest training requirements\. Future work should explore latency\-aware training and evaluate detection delay on natural multilingual calls\.

## Limitations

The controlled English benchmark is constructed from scripted dialogues and synthetic speech\. Although this design supports balanced and reproducible evaluation, it does not capture the full variability of real calls, including spontaneous speech, interruptions, background noise, channel distortion, regional accents, and evolving scam strategies\. Evaluation on TeleAntiFraud\-28K extends the study to Mandarin conversations, but the comparison with published Qwen2\-Audio systems is contextual rather than controlled because the models use different inputs, supervision, and training procedures\. StreamFraudNet is trained only with conversation\-level labels\. Consequently, its window\-level outputs are latent risk scores rather than validated localizations of fraudulent evidence\. The prefix experiments show that useful predictions are available before a call ends, but they do not measure delay relative to the true onset of fraudulent content\. Moreover, the matched global BiLSTM remains numerically stronger at every evaluated prefix\. Timestamped annotations and latency\-aware objectives are therefore needed to evaluate and improve fraud\-onset detection directly\.

The efficiency results are limited to the evaluated server GPU and CPU\. They establish faster\-than\-real\-time processing on these platforms, but not deployment on mobile phones, embedded processors, or telecommunication edge devices\. Although the task\-specific head contains only 0\.953 million trainable parameters, the complete inference pipeline contains 95\.325 million parameters because the frozen speech encoder remains required\.

Finally, raw telephone audio may contain sensitive personal information\. Practical deployment would require appropriate consent, secure processing and storage, restricted retention, and validation across languages, demographic groups, and acoustic conditions\. Fraud predictions should support human or policy\-based review rather than automatically blocking calls, since false positives may disrupt legitimate communication and false negatives may leave users unprotected\.

## References

- M\. Alhulifi, R\. Alharbi, and A\. Alsoubai \(2025\)Anti\-scam: user\-centric evaluation of llm\-powered and content\-based phone scam detection\.InCompanion Publication of the 2025 Conference on Computer\-Supported Cooperative Work and Social Computing,CSCW Companion ’25,New York, NY, USA,pp\. 536–539\.External Links:ISBN 9798400714801Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1)\.
- A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli \(2020\)wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 12449–12460\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p5.1),[§3\.2\.1](https://arxiv.org/html/2609.20223#S3.SS2.SSS1.p2.2),[§4\.1](https://arxiv.org/html/2609.20223#S4.SS1.SSS0.Px2.p1.1)\.
- J\. Cao, X\. Cui, and C\. Zheng \(2024\)TFD\-GCL: telecommunications fraud detection based on graph contrastive learning with adaptive augmentation\.In2024 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–7\.Cited by:[§2\.1](https://arxiv.org/html/2609.20223#S2.SS1.p1.1)\.
- R\. Chen, W\. Liang, Z\. Gong, L\. Ai, and J\. Hirschberg \(2026\)Detecting mental manipulation in speech via synthetic multi\-speaker dialogue\.InProceedings of the 16th International Workshop on Spoken Dialogue System Technology,G\. Riccardi, S\. M\. Mousavi, M\. I\. Torres, K\. Yoshino, Z\. Callejas, S\. A\. Chowdhury, Y\. Chen, F\. Bechet, J\. Gustafson, G\. Damnati, A\. Papangelis, L\. F\. D’Haro, J\. Mendonça, R\. Bernardi, D\. Hakkani\-Tur, G\. ”\. Di Fabbrizio, T\. Kawahara, F\. Alam, G\. Tur, and M\. Johnston \(Eds\.\),Trento, Italy,pp\. 428–440\.External Links:[Link](https://aclanthology.org/2026.iwsds-1.41/)Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p2.1)\.
- S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao, J\. Wu, L\. Zhou, S\. Ren, Y\. Qian, Y\. Qian, J\. Wu, M\. Zeng, X\. Yu, and F\. Wei \(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.16\(6\),pp\. 1505–1518\.Cited by:[§4\.1](https://arxiv.org/html/2609.20223#S4.SS1.SSS0.Px4.p1.1)\.
- Communications Fraud Control Association \(2023\)Global telecom fraud loss survey 2023\.Industry ReportCommunications Fraud Control Association\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1)\.
- A\. Conneau, A\. Baevski, R\. Collobert, A\. Mohamed, and M\. Auli \(2021\)Unsupervised Cross\-Lingual Representation Learning for Speech Recognition\.InInterspeech 2021,pp\. 2426–2430\.External Links:ISSN 2958\-1796Cited by:[§4\.1](https://arxiv.org/html/2609.20223#S4.SS1.SSS0.Px2.p2.1)\.
- S\. Deshmukh, B\. Raj, and R\. Singh \(2021\)Improving weakly supervised sound event detection with self\-supervised auxiliary tasks\.InProceedings of Interspeech 2021,pp\. 596–600\.Cited by:[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p2.1)\.
- B\. Elizalde and D\. Emmanouilidou \(2021\)Detection of robocall and spam calls using acoustic features of incoming voicemails\.45\(1\),pp\. 060004\.Cited by:[§2\.2](https://arxiv.org/html/2609.20223#S2.SS2.p1.1)\.
- L\. Erlacher \(2025\)S2S\-sed: a speech\-to\-speech approach for detection of social engineering attacks in audio conversations\.InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security,CCS ’25,New York, NY, USA,pp\. 4854–4856\.External Links:ISBN 9798400715259Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p2.1)\.
- G\. Gressel, R\. Pankajakshan, and Y\. Mirsky \(2024\)Discussion paper: exploiting LLMs for scam automation: a looming threat\.InProceedings of the 3rd ACM Workshop on the Security Implications of Deepfakes and Cheapfakes,Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1)\.
- S\. Holm \(1979\)A simple sequentially rejective multiple test procedure\.6\(2\),pp\. 65–70\.External Links:ISSN 03036898, 14679469,[Link](http://www.jstor.org/stable/4615733)Cited by:[§4\.2](https://arxiv.org/html/2609.20223#S4.SS2.p1.1)\.
- X\. Hu, H\. Chen, J\. Zhang, H\. Chen, S\. Liu, X\. Li, Y\. Wang, and X\. Xue \(2024\)GAT\-COBO: cost\-sensitive graph neural network for telecom fraud detection\.10\(4\),pp\. 528–542\.Cited by:[§2\.1](https://arxiv.org/html/2609.20223#S2.SS1.p1.1)\.
- M\. Ilse, J\. M\. Tomczak, and M\. Welling \(2018\)Attention\-based deep multiple instance learning\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 2127–2136\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p1.1)\.
- J\. Kotus, K\. Lopatka, and A\. Czyzewski \(2014\)Detection and localization of selected acoustic events in acoustic field for smart surveillance applications\.68\(1\),pp\. 5–21\.External Links:ISSN 1573\-7721Cited by:[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p2.1)\.
- J\. Lee, Y\. Lee, J\. Kim, A\. Kosiorek, S\. Choi, and Y\. W\. Teh \(2019\)Set transformer: a framework for attention\-based permutation\-invariant neural networks\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 3744–3753\.External Links:[Link](https://proceedings.mlr.press/v97/lee19d.html)Cited by:[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p1.1)\.
- Z\. Ma, P\. Wang, M\. Huang, J\. Wang, K\. Wu, X\. Lv, Y\. Pang, Y\. Yang, W\. Tang, and Y\. Kang \(2025\)TeleAntiFraud\-28k: an audio\-text slow\-thinking dataset for telecom fraud detection\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 5853–5862\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.20223#S2.SS2.p3.1),[§4\.1](https://arxiv.org/html/2609.20223#S4.SS1.SSS0.Px1.p2.1),[§4\.8](https://arxiv.org/html/2609.20223#S4.SS8.p1.1)\.
- K\. Miyazaki, T\. Komatsu, T\. Hayashi, S\. Watanabe, T\. Toda, and K\. Takeda \(2020\)Weakly\-supervised sound event detection with self\-attention\.In2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 66–70\.Cited by:[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p2.1)\.
- P\. Y\. J\. Nicholas and P\. C\. Ng \(2024\)ScamDetector: Leveraging Fine\-Tuned Language Models for Improved Fraudulent Call Detection\.InTENCON 2024 \- 2024 IEEE Region 10 Conference \(TENCON\),pp\. 101\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1)\.
- A\. Politis, A\. Mesaros, S\. Adavanne, T\. Heittola, and T\. Virtanen \(2021\)Overview and evaluation of sound event localization and detection in DCASE 2019\.29,pp\. 684–698\.Cited by:[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p2.1)\.
- S\. Prasad, E\. Bouma\-Sims, A\. K\. Mylappan, and B\. Reaves \(2020\)Who’s calling? characterizing robocalls through audio and metadata analysis\.In29th USENIX Security Symposium \(USENIX Security 20\),pp\. 397–414\.External Links:ISBN 978\-1\-939133\-17\-5Cited by:[§2\.1](https://arxiv.org/html/2609.20223#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.20223#S2.SS2.p1.1)\.
- D\. Rymarczyk, A\. Borowa, J\. Tabor, and B\. Zielinski \(2021\)Kernel Self\-Attention for Weakly\-supervised Image Classification using Deep Multiple Instance Learning\.In2021 IEEE Winter Conference on Applications of Computer Vision \(WACV\),Vol\.,Los Alamitos, CA, USA,pp\. 1720–1729\.External Links:ISSNCited by:[§2\.3](https://arxiv.org/html/2609.20223#S2.SS3.p1.1)\.
- Z\. Shen, K\. Wang, Y\. Zhang, G\. Ngai, and E\. Y\. Fu \(2025a\)Combating phone scams with LLM\-based detection: where do we stand? \(student abstract\)\.39\(28\),pp\. 29487–29489\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1),[§1](https://arxiv.org/html/2609.20223#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.20223#S2.SS2.p2.1)\.
- Z\. Shen, S\. Yan, Y\. Zhang, X\. Luo, G\. Ngai, and E\. Y\. Fu \(2025b\)“It warned me just at the right moment”: exploring LLM\-based real\-time detection of phone scams\.InExtended Abstracts of the 2025 CHI Conference on Human Factors in Computing Systems,Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1),[§1](https://arxiv.org/html/2609.20223#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.20223#S2.SS2.p2.1)\.
- Y\. Tian, G\. Liu, J\. Wang, and M\. Zhou \(2024\)ASA\-GNN: adaptive sampling and aggregation\-based graph neural network for transaction fraud detection\.11\(3\),pp\. 3536–3549\.Cited by:[§2\.1](https://arxiv.org/html/2609.20223#S2.SS1.p2.1)\.
- Y\. Tian and G\. Liu \(2024\)Spatial\-temporal\-aware graph transformer for transaction fraud detection\.20\(11\),pp\. 12659–12668\.Cited by:[§2\.1](https://arxiv.org/html/2609.20223#S2.SS1.p2.1)\.
- United Nations Office on Drugs and Crime \(2025\)Inflection point: global implications of scam centres, underground banking and illicit online marketplaces in southeast asia\.Technical reportUnited Nations Office on Drugs and Crime,Bangkok, Thailand\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p1.1)\.
- P\. Wang, Z\. Ma, X\. Dai, Y\. Liu, S\. Feng, X\. Yang, W\. Hu, Z\. Wang, M\. Pan, L\. Yuan, and D\. Wang \(2026\)SAFE\-QAQ: end\-to\-end slow\-thinking audio\-text fraud detection via reinforcement learning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 26138–26157\.Cited by:[§1](https://arxiv.org/html/2609.20223#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.20223#S2.SS2.p3.1)\.
- Y\. Xie, G\. Liu, M\. Zhou, L\. Wei, H\. Zhu, R\. Zhou, and L\. Cao \(2024\)A spatial\-temporal gated network for credit card fraud detection by learning transactional representations\.21\(4\),pp\. 6978–6991\.Cited by:[§2\.1](https://arxiv.org/html/2609.20223#S2.SS1.p2.1)\.

## Appendix AAdditional Experimental Results

Unless otherwise stated, the following experiments use the synthetic English benchmark, training seeds 13, 37, and 73, and the evaluation protocol in Section[4](https://arxiv.org/html/2609.20223#S4)\.

### A\.1Secondary Metrics and Seed Variation

Table[10](https://arxiv.org/html/2609.20223#A1.T10)complements the primary ROC–AUC and F1 results with threshold\-independent, thresholded, and correlation\-based metrics\.

Table 10:Additional StreamFraudNet results on the synthetic test set\. Confidence intervals are estimated using 10,000 label\-stratified dialogue bootstraps\.The three individual runs are reported in Table[11](https://arxiv.org/html/2609.20223#A1.T11)\. ROC–AUC varies little across seeds, whereas F1 exhibits greater variation because it depends on the selected decision threshold\.

Table 11:Per\-seed StreamFraudNet performance\.
### A\.2Complete Statistical Comparisons

Table[12](https://arxiv.org/html/2609.20223#A1.T12)reports the corrected pairwise comparisons underlying the significance markers in Table[2](https://arxiv.org/html/2609.20223#S4.T2)\. “n\.s\.” denotes a difference that is not significant after Holm correction\.

Table 12:Pairwise comparisons against StreamFraudNet\. The direction refers to the ensemble ROC–AUC difference\.For the component analysis, removing recurrent context decreases ROC–AUC by 0\.02797, with a 95% confidence interval of\[0\.01398,0\.04422\]\[0\.01398,0\.04422\], and decreases F1 by 0\.07637, with an interval of\[0\.04264,0\.11200\]\[0\.04264,0\.11200\]\. Both comparisons havepHolm=0\.0044p\_\{\\mathrm\{Holm\}\}=0\.0044\. No other component ablation remains significant after correction\.

### A\.3Encoder Fine\-Tuning Diagnostics

The aggregate encoder\-policy comparison is reported in Table[4](https://arxiv.org/html/2609.20223#S4.T4)\. Table[13](https://arxiv.org/html/2609.20223#A1.T13)provides the per\-seed results needed to explain the high variance observed under full fine\-tuning with learning rate10−510^\{\-5\}\.

Table 13:Per\-seed ROC–AUC under full encoder fine\-tuning\.The instability at learning rate10−510^\{\-5\}is concentrated in seed 13\. Reducing the learning rate eliminates this failure, but the resulting ensemble remains below the frozen encoder, as discussed in Section[4\.4](https://arxiv.org/html/2609.20223#S4.SS4)\.

### A\.4Full TeleAntiFraud\-28K Results

Table[14](https://arxiv.org/html/2609.20223#A1.T14)supplements the accuracy and F1 values reported in Section[4\.8](https://arxiv.org/html/2609.20223#S4.SS8)with precision and recall\.

Table 14:Full TeleAntiFraud\-28K results\. Published Qwen2\-Audio values are contextual rather than controlled comparisons\.StreamFraudNet has higher precision but lower recall than Qwen2\-Audio ASR\-text, explaining their similar accuracy but slightly different F1 scores\.

### A\.5Model Footprint and Inference Protocol

The full model footprint and streaming configuration are reported in Table[15](https://arxiv.org/html/2609.20223#A1.T15)\. Although only 0\.953 million task\-head parameters are trained, the frozen speech encoder remains part of the deployed pipeline\.

Table 15:Model footprint and streaming configuration\.The end\-to\-end measurements in Table[9](https://arxiv.org/html/2609.20223#S4.T9)include raw audio decoding, resampling, chunking, speech\-encoder inference, task\-head inference, and final aggregation\. They use batch size one, 10 warm\-up runs, and 100 timed conversations averaging 109\.6 seconds, with audio capped at 120 seconds\. These measurements quantify processing throughput on the evaluated server hardware rather than mobile deployment or fraud\-onset detection latency\.

### A\.6Label\-Efficiency Details

The main paper summarizes performance under reduced supervision using mean ROC–AUC/F1 pairs\. Table[16](https://arxiv.org/html/2609.20223#A1.T16)reports the complete mean and sample standard deviation across the three training seeds\. The subsets contain 102, 256, 512, and 1,023 labeled conversations, corresponding to 10%, 25%, 50%, and 100% of the available training data\.

Table 16:Performance under reduced conversation\-level supervision\. Values are mean±\\pmsample standard deviation across three seeds\.Performance generally improves as additional conversation\-level labels become available\. StreamFraudNet retains substantial discrimination with half of the training data, reaching a mean ROC–AUC of 0\.9739 and an F1 score of 0\.9210\. Its three\-seed ensemble reaches 0\.9853 ROC–AUC and 0\.9536 F1 in this setting; relative to full\-data training, the F1 reduction is not significant after correction, whereas the ROC–AUC reduction is significant\.

The global BiLSTM remains numerically stronger at every evaluated label budget\. The experiment therefore demonstrates that StreamFraudNet can learn from reduced conversation\-level supervision, but does not establish greater label efficiency than the matched global model\.

### A\.7CPU\-Only Inference Results

To assess computational efficiency under resource\-constrained conditions, we sample 500 training instances per dataset \(seed=42; 1,000 total\) and evaluate all models CPU\-only on commodity hardware \(8 cores, 16GB RAM\)\. We simulate real\-time streaming by segmenting audio into 2\-second chunks, using a 5\-chunk \(10\-second\) sliding window with stride 1\. Real\-time factor \(RTF\) is reported as the average ratio of inference time to audio duration per window, per dataset\.

Table 17:Computational efficiency under CPU\-only inference\.As shown in Table[17](https://arxiv.org/html/2609.20223#A1.T17), the computational cost varies from 69\.2 to 178\.7 GFLOPs due to the use of different pretrained size of Wav2Vec encoders\. Under CPU\-only inference, the proposed model achieves real\-time performance across all configurations, with RTFs ranging from 0\.0437 to 0\.4168 in a streaming setting\. Notably, even the most computationally demanding configuration \(178\.7 GFLOPs\) maintains an RTF well below 1, corresponding to approximately 2\.5× faster\-than\-real\-time processing\. These results demonstrate that the model remains deployable in resource\-constrained environments without GPU acceleration\.

## Appendix BStreaming Inference on Synthetic Fraudulent Conversations

Tables 19 and 20 provide a closer, interpretability\-oriented view of how StreamFraudNet distributes fraud evidence across sliding windows, complementing the aggregate ROC–AUC and F1 results reported in Section 4\. Rather than treating the aggregator’s output as a single opaque decision, these window\-level traces let us inspect*where*within the 10\-second receptive field the local detector concentrates its score mass, and whether that concentration aligns with human\-interpretable fraud cues\. As illustrated in Figure\.[1](https://arxiv.org/html/2609.20223#S1.F1), fraudulent intent is often expressed only at specific moments within an otherwise benign call, making call\-level classification insufficient\.

For the non\-scam call \(Table 17\), the local score spikes to 0\.7051 only in the very first window, which happens to contain a phone\-number exchange—a superficial pattern that co\-occurs with scam openings in the training distribution\. As the conversation moves into unrelated resolution dialogue \(apologizing, suggesting a phone directory, saying goodbye\), the score decays by two to three orders of magnitude within four to six seconds \(windows 4\.0–16\.0 s\) and remains near zero thereafter\. This confirms that the detector reacts to local lexical/acoustic surface features within its bounded window rather than propagating a persistent belief state about the call, and that any such false activation is short\-lived and is down\-weighted by the attention\-based aggregator \(Eq\. 14–15\) at the conversation level\.

For the scam call \(Table 18\), scores exceed 0\.95 in windows that restate one of a small set of lexical anchors—suspended,verify,national security,social security number—even across non\-contiguous windows \(e\.g\., 0\.0–10\.0 s, 6\.0–16\.0 s, 14\.0–24\.0 s, 34\.0–50\.0 s\)\. Notably, the model’s sensitivity appears keyed to the literal recurrence of these terms rather than to the underlying scam intent: window 12\.0–22\.0 s, which paraphrases the same suspension claim as“This suspension is part of a national\-level investigation”without repeating the anchor phrase verbatim, drops to 0\.3761 even though it is flanked by windows scoring above 0\.95 on either side\. A similar pattern appears later in the call: once the dialogue shifts into negotiation and stalling \(callback\-number requests, checking with a supervisor, procedural questions about visiting an office\), scores collapse to below 0\.05 for several consecutive windows \(64\.0–106\.0 s\) despite the manipulative intent of the conversation persisting throughout, then rebound sharply as soon as the caller reiterates the identity\-verification request \(112\.0–122\.0 s, 0\.9922\)\. This intermittency suggests that StreamFraudNet’s window\-level scores behave more like a keyword\-triggered detector operating within a short temporal horizon than a model that tracks cumulative deceptive framing across the call\.

These qualitative patterns are consistent with the quantitative findings in Section 4\.3 and the paper’s limitations: recurrent context within a window is the dominant architectural contributor \(Table 3\), but the BiLSTM’s temporal reach is confined to the currentkk\-chunk window and has no mechanism for carrying evidence forward once a window slides out of context\. Because no timestamped fraud annotations are available during training \(Section 3\.1\), these local scores are optimized purely to support the conversation\-level BCE objective and should therefore be read as latent risk indicators rather than validated localizations of fraudulent intent—the same caveat raised in the paper’s Limitations section, and one that these appendix examples make concrete rather than abstract\.

Table 18:Example of a Non\-Scam ConversationTable 19:Example of a Scam ConversationTable 20:Model Scores over Sliding Windows for a Fraudulent Conversation in Table[18](https://arxiv.org/html/2609.20223#A2.T18)Table 21:Model Scores over Sliding Windows for a Fraudulent Conversation in Table[19](https://arxiv.org/html/2609.20223#A2.T19)