TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

Hugging Face Daily Papers Papers
telecom-fraud-detection audio-benchmark evaluation-protocol chinese-language mixed-tree-generation ASR-LLM near-domain-negatives

Summary

The paper presents TeleAntiFraud 2.0, an audio-based benchmark for evaluating telecom fraud detection models using a mixed-tree generation pipeline and frozen monthly sets to address evolving fraud scripts and near-domain negatives.

Telecom fraud scripts evolve rapidly and are often designed to resemble routine service conversations, creating two key requirements for audio-based telecom-fraud evaluation. First, benchmarks must incorporate newly observed scam patterns without overwriting previously established test sets. Second, they must distinguish fraud from lawful, near-domain calls rather than relying on topic-separated negative examples. We present TeleAntiFraud 2.0, constructed with our Mixed-Tree Anti-Fraud Generation Pipeline and evaluated under a monthly frozen evaluation protocol. The pipeline transforms online fraud-case abstracts into profile-grounded scenarios, expands them through mixed-tree generation, realizes fraud and non-fraud dialogue paths under shared contexts, renders validated dialogues as role-matched speech, and freezes the resulting audio, labels, prompts, manifests, and provenance records for each monthly evaluation set. Each frozen set contains 900 Chinese calls, comprising 600 fraud and 300 near-domain non-fraud cases. Controlled text experiments show that three classifiers achieve perfect macro-averaged F1 (Macro-F1) when evaluated against unrelated or ordinary negatives, but drop to 0.65-0.68 with near-domain sibling negatives. Full-set audio and automatic-speech-recognition plus large-language-model (ASR+LLM) evaluations further reveal class-prior shortcuts, prediction collapse, and snapshot sensitivity. Together, these findings establish near-domain construction and collapse-aware reporting as core requirements for evaluating audio-based telecom-fraud models under realistic confusable conditions. The accompanying research artifact includes the construction code, evaluation scripts, manifests, and documentation. Our dataset and code are available at https://anonymous.4open.science/r/TeleAntiFraud-2_0-EEB2/.
Original Article
View Cached Full Text

Cached at: 09/21/26, 07:19 AM

Paper page - TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

Source: https://huggingface.co/papers/2609.18748 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Telecomfraudscriptsevolverapidlyandareoftendesignedtoresembleroutineserviceconversations,creatingtwokeyrequirementsforaudio-basedtelecom-fraudevaluation.First,benchmarksmustincorporatenewlyobservedscampatternswithoutoverwritingpreviouslyestablishedtestsets.Second,theymustdistinguishfraudfromlawful,near-domaincallsratherthanrelyingontopic-separatednegativeexamples.WepresentTeleAntiFraud2.0,constructedwithourMixed-TreeAnti-FraudGenerationPipelineandevaluatedunderamonthlyfrozenevaluationprotocol.Thepipelinetransformsonlinefraud-caseabstractsintoprofile-groundedscenarios,expandsthemthroughmixed-treegeneration,realizesfraudandnon-frauddialoguepathsundersharedcontexts,rendersvalidateddialoguesasrole-matchedspeech,andfreezestheresultingaudio,labels,prompts,manifests,andprovenancerecordsforeachmonthlyevaluationset.Eachfrozensetcontains900Chinesecalls,comprising600fraudand300near-domainnon-fraudcases.Controlledtextexperimentsshowthatthreeclassifiersachieveperfectmacro-averagedF1(Macro-F1)whenevaluatedagainstunrelatedorordinarynegatives,butdropto0.65-0.68withnear-domainsiblingnegatives.Full-setaudioandautomatic-speech-recognitionpluslarge-language-model(ASR+LLM)evaluationsfurtherrevealclass-priorshortcuts,predictioncollapse,andsnapshotsensitivity.Together,thesefindingsestablishnear-domainconstructionandcollapse-awarereportingascorerequirementsforevaluatingaudio-basedtelecom-fraudmodelsunderrealisticconfusableconditions.Theaccompanyingresearchartifactincludestheconstructioncode,evaluationscripts,manifests,anddocumentation.Ourdatasetandcodeareavailableathttps://anonymous.4open.science/r/TeleAntiFraud-2_0-EEB2/.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.18748

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.18748 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.18748 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.18748 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Poster: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

arXiv cs.CL

This paper introduces the first public multimodal dataset of 100 Turkish scam and benign phone calls, evaluating seven LLMs under raw audio, ASR transcripts, and human-corrected transcripts. Results show transcript-based inputs outperform direct audio, highlighting the need for inclusive AI safety research in low-resource languages.