scalable anomaly detection with spark and sos
play

Scalable Anomaly Detection with Spark and SOS Strata NYC - PowerPoint PPT Presentation

May 12, 2023 •203 likes •871 views

Scalable Anomaly Detection with Spark and SOS Strata NYC September 26, 2019 Hi there, my name is Jeroen Janssens Today SOS, World! Anomalies and outliers Evaluating outlier-selection algorithms Various approaches to outlier

  1. Scalable Anomaly Detection with Spark and SOS Strata NYC September 26, 2019

  2. Hi there, my name is Jeroen Janssens

  3. Today ● SOS, World! ● Anomalies and outliers ● Evaluating outlier-selection algorithms ● Various approaches to outlier selection ● Stochastic Outlier Selection ● Conclusion

  4. SOS, World! 01-sos-world.ipynb

  5. Implementations of SOS ● Python: http://bit.ly/sos-python ● Spark: http://bit.ly/sos-spark ● R: http://bit.ly/sos-r ● Flink: http://bit.ly/sos-flink

  6. Anomalies and outliers

  7. An anomaly is an observation or event that deviates qualitatively from what is considered to be normal, according to a domain expert.

  8. Detecting anomalies is important ● Expensive ● Dangerous Mess up your model ●

  9. Human anomaly detection may suffer from ● Fatigue ● Information overload Emotional bias ●

  10. Feature-vector representation

  11. Dissimilarity-matrix representation

  12. From anomaly to outlier

  13. An outlier is a data point that deviates quantitatively from the majority of the data points, according to an outlier-selection algorithm.

  14. The symbiotic relationship between the domain expert and the algorithm

  15. Data flow diagram Data flow diagram illustrating the relationship between the domain expert (square) and the outlier-selection algorithm (top circle).

  16. Six Euler diagrams (1/2)

  17. Six Euler diagrams (2/2)

  18. Evaluating outlier-selection algorithms

  19. Confusion matrix Computer says no.

  20. Four possible outcomes

  21. Evaluation Illustration of relabelling a multi-class data set into multiple one-class data sets.

  22. Anomalies are rare In order to evaluate the algorithm we simulate anomalies to be rare. Banana for scale.

  23. Outlier scores The dashed line indicates the threshold chosen by the domain expert.

  24. ROC curve An ROC curve plots the false alarm rate against the hit rate for all possible thresholds.

  25. Various approaches to outlier selection

  26. Distribution-based outlier selection

  27. Distance-based outlier selection Size does matter

  28. Density-based outlier-selection

  29. Stochastic Outlier Selection

  30. Stochastic Outlier Selection ● Unsupervised outlier selection algorithm ● Employs concept of affinity (inspired by t-SNE) One parameter: perplexity ● Computes outlier probabilities ●

  31. t-Distributed Neighbor Embedding (t-SNE; Van der Maaten, Hinton) employs affinity to perform dimensionality reduction

  32. A data point is selected as an outlier when all the other data points have insufficient affinity with it.

  33. From input to output

  34. From feature matrix to dissimilarity matrix

  35. From input to output

  36. Smooth neighborhoods

  37. Affinity between data points

  38. From input to output

  39. From affinity to binding probability The binding matrix B is obtained by normalising each row in the affinity matrix A.

  40. Binding probabilities form a graph

  41. Binding probabilities form a graph

  42. Stochastic Neighbor Graph A data point belongs to the outlier class when no it is not selected by any other data points.

  43. Three SNGs The three SNGs Ga, Gb, and Gc are sampled from the discrete probability distribution P(G).

  44. Set of all SNGs

  45. Approximating outlier probabilities by sampling SNGs

  46. Demo: Sampling SNGs in CoffeeScript and D3 http://bit.ly/sos-d3

  47. Computing outlier probabilities through marginalisation

  48. Computing outlier probabilities in closed form

  49. Proof!

  50. Selecting outliers

  51. Adaptive variances via the perplexity parameter

  52. Continuous binary search

  53. Perplexity influences outlier probabilities

  54. Evaluation and comparison

  55. Putlier-score plots

  56. Real-world datasets

  57. Synthetic datasets

  58. Synthetic datasets

  59. Synthetic datasets

  60. Synthetic datasets

  61. SOS performs significantly better

  62. Spark implementation of SOS

  63. Spark implementation of SOS ● Developed by Fokko Driesprong ● Works with DataFrame API ● Available on GitHub ● Plan is to make it part of MLLib

  64. SOS on PySpark 92-pyspark-sos.ipynb

  65. Summary Outlier selection can support the detection of anomalies ● SOS is an intuitive and probabilistic algorithm to select outliers ● SOS has a very good performance ● No free lunch ●

  66. Thank you! Here are some links ● Blog: http://bit.ly/sos-blog ● D3 Demo: http://bit.ly/sos-d3 ● Python implementation: http://bit.ly/sos-python ● Spark implementation: http://bit.ly/sos-spark ● R implementation: http://bit.ly/sos-r ● Flink implementation: http://bit.ly/sos-flink

Recommend

Scalable Architecture for Anomaly Detection and Visualization in Power Generating Assets

Scalable Architecture for Anomaly Detection and Visualization in Power Generating Assets

Scalable Architecture for Anomaly Detection and Visualization in Power Generating Assets Paras Jain, Chirag Tailor , Sam Ford, Liexiao (Richard) Ding, Michael Phillips, Fang (Cherry) Liu, Nagi Gabraeel, Polo Chau 1/13 Background

325 views • 13 slides
Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier) in a dataset is an instance that is abnormal or unlikely based on the rest of the dataset If the instances in the dataset are labeled as

760 views • 19 slides
What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining

What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining

DataCamp Anomaly Detection in R ANOMALY DETECTION IN R What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining the term anomaly Anomaly: a data point or collection of data points that do not follow the

870 views • 21 slides
Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative

Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative

Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative Anomaly Detection Motivation Developing an anomaly detection system Anomaly detection vs. supervised learning Choosing what features

514 views • 34 slides
Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier Detection Automatically identify data points that are somehow different from the rest Working assumption: There are considerably

984 views • 72 slides
Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier Detection Automatically identify data points that are somehow different from the rest Working assumption: There are considerably

905 views • 74 slides
Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview Prior Work in Network Anomaly Detection The 1999 DARPA Intrusion Detection Evaluation Packet Header Anomaly Detection (PHAD) Application

575 views • 18 slides
<Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection

<Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection

Data Council Singapre 2019 Autoencoder Forest for Anomaly Detection from IoT Time Series <Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection Autoencoder for anomaly detection Autoencoder

540 views • 27 slides
In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong, Alan Fern, Thomas G. Dietterich and Md Amran Siddiqui School of EECS Anomaly Detection Goal: Identify rare or strange objects 2 Anomaly

756 views • 51 slides
Dataflow Anomaly Detection Presented By Archana Viswanath Computer Science and Engineering The

Dataflow Anomaly Detection Presented By Archana Viswanath Computer Science and Engineering The

Dataflow Anomaly Detection Presented By Archana Viswanath Computer Science and Engineering The Pennsylvania State University Anomaly Intrusion Detection Systems Anomaly Intrusion Detection Systems Model normal behavior. Attack - Any

417 views • 20 slides
Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and

Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and

Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and Mohammad Zulkernine School of Computing Queens University, Kingston Ontario, Canada K7L 3N6 {zhang, mzulker} @cs.queensu.ca Abstract- Anomaly

284 views • 6 slides
Efficient Anomaly Detection by Isolation using Nearest Neighbour Ensemble Tharindu

Efficient Anomaly Detection by Isolation using Nearest Neighbour Ensemble Tharindu

Information Technology Efficient Anomaly Detection by Isolation using Nearest Neighbour Ensemble Tharindu Rukshan Bandaragoda Kai Ming Ting David Albrecht Fei Tony Liu Jonathan R. Wells Outline Overview of anomaly detection

657 views • 41 slides
Isolation trees Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Isolation

Isolation trees Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Isolation

DataCamp Anomaly Detection in R ANOMALY DETECTION IN R Isolation trees Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Isolation tree DataCamp Anomaly Detection in R Isolation tree plots DataCamp Anomaly Detection in R

631 views • 19 slides
On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman.

On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman.

Introduction Feature Extraction Entropy Calculation Anomaly detection Classification Open Issues On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman. Cinvestav, Campus Guadalajara, Mexico November 2015

690 views • 24 slides
ACCT 420: Topic modeling and anomaly detection Session 8 Dr. Richard M. Crowley 1 Front matter

ACCT 420: Topic modeling and anomaly detection Session 8 Dr. Richard M. Crowley 1 Front matter

ACCT 420: Topic modeling and anomaly detection Session 8 Dr. Richard M. Crowley 1 Front matter 2 . 1 Learning objectives Theory: NLP Anomaly detection Application: Understand annual report readability Examine the

1.12k views • 85 slides
An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate

An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate

An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate April 25, 2010 The 3rd CAIDA-WIDE-CASFI Joint Measurement Workshop @Osaka 1 2010 4 25 Background Anomaly Detection:

331 views • 20 slides
Anomaly Detection with State Space Models Multi-dimensional State Space Models SVD Method EM

Anomaly Detection with State Space Models Multi-dimensional State Space Models SVD Method EM

Anomaly Detection CAMCOS 2009 Introduction ADAPT Anomaly Detection with State Space Models Multi-dimensional State Space Models SVD Method EM Algorithm Kalman Filter Maja Derek, Kate Isaacs, Duncan McElfresh, Jennifer Alarm Murguia,

1.4k views • 96 slides
ACCT 420: Topic modeling and anomaly detection Session 9 Dr. Richard M. Crowley 1 Front matter

ACCT 420: Topic modeling and anomaly detection Session 9 Dr. Richard M. Crowley 1 Front matter

ACCT 420: Topic modeling and anomaly detection Session 9 Dr. Richard M. Crowley 1 Front matter 2 . 1 Learning objectives Theory: NLP Anomaly detection Application: Understand the distribution of readability Examine

1.31k views • 82 slides
Anomaly Detection for the CERN Large Hadron Collider injection magnets Armin Halilovic KU

Anomaly Detection for the CERN Large Hadron Collider injection magnets Armin Halilovic KU

Anomaly Detection for the CERN Large Hadron Collider injection magnets Armin Halilovic KU Leuven - Department of Computer Science In cooperation with CERN 2018-07-27 0 Outline 1 Context 2 Data 3 Preprocessing 4 Anomaly Detection 5

792 views • 51 slides
An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar,

An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar,

An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar, David Andersen, Hyong Kim, Hui Zhang Carnegie Mellon University Entropy-based Anomaly Detection Goal: detect abnormal behavior scan activity,

447 views • 34 slides
Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time

Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time

Deep Learning Frameworks with Spark and GPUs Abstract Spark is a powerful, scalable, real-time data analytics engine that is fast becoming the de facto hub for data science and big data. However, in parallel, GPU clusters is fast becoming the

389 views • 26 slides
Evading Network Anomaly Detection Sytems - Fogla,Lee Divya Muthukumaran Intrusion detection

Evading Network Anomaly Detection Sytems - Fogla,Lee Divya Muthukumaran Intrusion detection

Evading Network Anomaly Detection Sytems - Fogla,Lee Divya Muthukumaran Intrusion detection Systems Signature Based IDS Monitor packets on the network Compare them against database of signatures/attributes from known threats

476 views • 26 slides
Anomaly-based Bot Server (and more!) Detection Jim Binkley jrb@cs.pdx.edu Portland State

Anomaly-based Bot Server (and more!) Detection Jim Binkley jrb@cs.pdx.edu Portland State

Anomaly-based Bot Server (and more!) Detection Jim Binkley jrb@cs.pdx.edu Portland State University Computer Science outline background experimental flow tuples botnet server mesh detection botnet client mesh detection

364 views • 23 slides
Detecting Attacks Anomaly-based Detection Signature-based Signature-based (Misuse)

Detecting Attacks Anomaly-based Detection Signature-based Signature-based (Misuse)

Detecting Attacks Anomaly-based Detection Signature-based Signature-based (Misuse) Detection Intrusion Detection Host-based Network-based Boriana Ditcheva and Lisa Fowler Active/Passive University of North Carolina at

401 views • 12 slides

More recommend