You can beat SOTA Time Series Anomaly Detection methods with a 100 year old algorithm [R]

Reddit r/MachineLearning News
time-series-anomaly-detection benchmark-critique statistical-process-control ai-research community-reflection

Summary

The author critiques current Time Series Anomaly Detection benchmarks, showing that a simple Statistical Process Control algorithm can outperform state-of-the-art methods, suggesting benchmarks are too trivial and calling for more challenging problems in the community.

You can beat SOTA Time Series Anomaly Detection methods with a 100 year old algorithm Time Series Anomaly Detection (TSAD) seems to be one of the hottest topics in NeurIPS, SIGKDD, VLDB etc. Many (perhaps most) papers evaluate on Paparrizos’ TSB-AD-M benchmark… However, I tested these benchmark datasets and found that in most cases I could beat the SOTA TSAD methods with a 100-year-old algorithm, simple Statistical Process Control (SPC). In the attached example, SPC gets perfect results. If we can beat the SOTA papers with 100-year-old algorithm, we probably should not be too impressed with them [b]. I really think this calls for some introspection by the community. To be clear, I make no claims (here) about the proposed algorithms in all these paper. But the TSB-AD benchmark is obviously too trivial to make meaningful claims on [a][b]. The example shown is one of the ECG traces but look at dozen of traces marked “TAO”, they are even more trivial to solve with SPC [a][c]. I do not claim to have solved the triviality problem, but I have done 90% of the work to introduce more challenging TSAD problems ([d] sled dogs, [e] Tuna, Fuel Cells, Smart Manufacturing etc.). TLDR: I think the TSAD community needs more introspection on benchmarks. Most progress over the last decade seems to be illusionary. [a] https://www.youtube.com/watch?v=VftCMSI3C_s [b] https://www.dropbox.com/scl/fi/31zuyhejb6sdjrom20frn/Problems-with-Time-Series-Anomaly-Detection.pptx?rlkey=mvcj1wz5s45kgazezopnih2h7&dl=0 [c] https://www.dropbox.com/scl/fi/42fkf9q9hft2224dnm83v/The-TSB-AD-Benchmarks-are-Nonsense.pptx?rlkey=5fwjopie5ncjhkgr0wqhdm2lp&dl=0 [d] https://www.linkedin.com/feed/update/urn:li:activity:7488825356494237696/ [e] https://www.dropbox.com/scl/fi/hettphvtpyrksggfect9d/Tutorial-on-Pan-Matrix-Profile.pptx?rlkey=p59gd2w56fxl9kl2fh5q819oo&dl=0
Original Article

Similar Articles

Bridging Classification and Reconstruction: Cooperative Time Series Anomaly Detection

arXiv cs.LG

This paper proposes CoAD, a novel framework that unifies Outlier Exposure (classification) and Masked Autoencoder (reconstruction) paradigms for time series anomaly detection, addressing their respective limitations. Extensive experiments show that CoAD significantly outperforms state-of-the-art methods while being lightweight and fast.

Anomalies in Multivariate Time Series Benchmarks Are Mostly Univariate

arXiv cs.LG

This paper introduces a diagnostic framework for multivariate time series anomaly detection benchmarks and finds that labeled anomalies are mostly detectable from individual channels, challenging the need for cross-channel modeling. The authors call for more structurally diverse evaluation sets.

TPA-AD: A Two-Stage Pseudo Anomaly-Guided Method for Bearing Time-Series Anomaly Detection

arXiv cs.LG

TPA-AD is a two-stage pseudo anomaly-guided method for bearing time-series anomaly detection that generates pseudo-anomalous windows near normal boundaries using reconstruction models and contrastive learning, then scores anomalies with KNN—without requiring real anomaly samples during training. It is evaluated on bearing fault and degradation datasets, including high-speed train axle-box bearing data.