Prediction, Event and Anomaly

Estimated reading time: 3 minutes

Anomaly Detection Software

Name Language Pitch  
Etsy’s Skyline Python Skyline is a real-time anomaly detection system, built to enable passive monitoring of hundreds of thousands of metrics  
Linkedin’s luminol Python Luminol is a light weight python library for time series data analysis. The two major functionalities it supports are anomaly detection and correlation. It can be used to investigate possible causes of anomaly.  
Ele.me’s banshee Mentat’s datastream.io Python An open-source framework for real-time anomaly detection using Python, Elasticsearch and Kibana.
MIDAS C++/Python/Golang/Ruby/Rust/R MIDAS detects anomalies in dynamic graphs in real-time  

This section includes some time-series software for anomaly detection-related tasks, such as forecasting.

Name Language Pitch
Facebook’s Prophet Python/R Prophet is a procedure for forecasting time series data. It is based on an additive model where non-linear trends are fit with yearly and weekly seasonality, plus holidays.
PyFlux Python The library has a good array of modern time series models, as well as a flexible array of inference options (frequentist and Bayesian) that can be applied to these models.
Pyramid Python Porting of R’s auto.arima with a scikit-learn-friendly interface.
SaxPy Python General implementation of SAX, as well as HOTSAX for anomaly detection.

Benchmark Datasets

  • Numenta’s NAB – NAB is a novel benchmark for evaluating algorithms for anomaly detection in streaming, real-time applications. It is comprised of over 50 labeled real-world and artificial timeseries data files plus a novel scoring mechanism designed for real-time applications.
  • Yahoo’s Webscope S5 – The dataset consists of real and synthetic time-series with tagged anomaly points. The dataset tests the detection accuracy of various anomaly-types including outliers and change-points.

Time Series Anomaly Detection

  • SAX
    • HOT SAX: Finding the Most Unusual Time Series Subsequence: Algorithms and Applications, Eamonn Keogh, Jessica Lin, Ada Fu, 2005 - Paper, Materials * LSTM
    • LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection, Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, Gautam Shroff, 2016 - Paper
    • Credit Card Transactions, Fraud Detection, and Machine Learning: Modelling Time with LSTM Recurrent Neural Networks, Bénard Wiese and Christian Omlin, 2009 - Springer
    • Long Short Term Memory Recurrent Neural Network Classifier for Intrusion Detection, Jihyun Kim, Jaehyun Kim, Huong Le Thi Thu, and Howon Kim - Paper
    • Deep Recurrent Neural Network-based Autoencoders for Acoustic Novelty Detection, Erik Marchi Fabio Vesperini, Stefano Squartini, and Bjo ̈rn Schuller - Paper
    • A Novel Approach for Automatic Acoustic Novelty Detection Using a Denoising Autoencoder with Bidirectional LSTM Neural Networks, Erik Marchi, Fabio Vesperini, Florian Eyben, Stefano Squartini, Bjo ̈rn Schuller - Paper * Transfer learning
    • Transfer Representation-Learning for Anomaly Detection, Jerone T. A. Andrews, Thomas Tanay, Edward J. Morton, Lewis D. Griffin, 2016 - Paper * Anomaly Detection Based on Sensor Data in Petroleum Industry Applications, Luis Martí,1, Nayat Sanchez-Pi, José Manuel Molina, and Ana Cristina Bicharra Garcia - Paper * Anomaly detection in aircraft data using recurrent nueral networks (RNN), Anvardh Nanduri, Lance Sherry - Paper * Bayesian Online Changepoint Detection, Ryan Prescott Adams, David J.C. MacKay - Paper * Anomaly Detection in Aviation Data using Extreme Learning Machines, Vijay Manikandan Janakiraman, David Nielsen - Paper
classification, detection, anomaly, event, time-series, deep, keras, LTSM