Quantifying and Detecting Incidents in IoT Big Data Analytics - PowerPoint PPT Presentation

Quantifying and Detecting Incidents in IoT Big Data Analytics Hong-Linh Truong Faculty of Informatics, TU Wien, Austria hong-linh.truong@tuwien.ac.at http://rdsea.github.io Acknowledgment: with a lot of discussion with Manfred Halper and our industrial partners. Note: Ongoing work Dagstuhl Seminar 17441, Big Stream Processing 1 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Case Study BTS Public cloud infrastructures BigQuery Private cloud infrs. Base Transceiver Station (BTS) Influxdb IoT MQTT Sensor Gateway Broker Hadoop FS G. Storage Actuator Analytics Optimizer Analytics Analytics  Large-scale systems (1K+ BTS)  Flexible back-end clouds  Generic enough for other applications (e.g., in smart agriculture)  With bad infrastructures for IoT and connectivity Dagstuhl Seminar 17441, Big Stream Processing 2 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Challenges The ultimate goal of the (domain) data scientist is to meet Quality of Analytics (QoA) QoA: cost, performance (response time), quality of data (up-to-date ness, accuracy) (Remember Christoph Quix’s talk about quality) But there are many interactions that might cause incidents Hong-Linh Truong , Aitor Murguzur, Erica Yang, Challenges in Enabling Quality of Analytics in the Cloud, ACM JDIQ Challenge paper, 2017. Dagstuhl Seminar 17441, Big Stream Processing 3 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Problem 1: the complexity of software stacks and subsystems Source: Simplified version of the Web Analytics Web design from I & A Computing Lab, VN services Service www.inacomputing.com Client Kibana Visualization analytics analytics Planner analytics BatchAnalytics results Enrichment Manager Apache Spark Service RabbitMQ notification analytics result BTS result Big data storage (Hadoop result Monitoring Apache Nifi FS/Google Storage) result ElasticSearch SFTP result data Ingestion Service Analytics BTS result BigQuery Service Monitoring result result MQTT result Streaming Data result Processing Dagstuhl Seminar 17441, Big Stream Processing 4 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Porblem 2: Complexity of the underlying virtual computing and network infrastructures IoT Big Data Analytics The SINC Concept: http://sincconcept.github.io  Heavily based on virtual resources  IoT, Network functions and Clouds  (Remember Manfred Hauswirth’s talk yesterday about fog/edge computing and NFV/5G networks) Dagstuhl Seminar 17441, Big Stream Processing 5 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Problem 3: Elasticity Management Tien-Dung Nguyen, Hong Linh Truong, Georgiana Copil, Duc-Hung Le, Daniel Moldovan, Schahram Dustdar: On Developing and Operating of Data Elasticity Management Process. ICSOC 2015: 105-119 Dagstuhl Seminar 17441, Big Stream Processing 6 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Our ideas for incident monitoring and analytics  Classification of incidents:  to quantify incidents and identify possible data sources, monitoring techniques and analytics.  Measurement/Instrumentation:  to provide mechanisms for measurement and data collection for incidents.  Incident analytics:  to find out the root cause and dependencies of incidents. Hong Linh Truong, Manfred Halper: Classifying Incidents in Cloud-based IoT Big Data Analytics, Working paper, 2017. Dagstuhl Seminar 17441, Big Stream Processing 7 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

W3H: what, when, where and how for incidents Too complex with many types of software. Can we have a simplified taxonomy for mapping incidents? Analysis/ Data IoT Transformation Storage Gateway Task Resulting analytics IoT Message Analysis/ Broker/Data …. Sensor Transformation Logistics Task Service Large number Large-scale Other Complex big data of data brokers & data systems in processing sources (e.g., transfer/logistics the pipeline frameworks IoT devices) services Hong Linh Truong, Manfred Halper: Classifying Incidents in Cloud-based IoT Big Data Analytics, Working paper, 2017. Dagstuhl Seminar 17441, Big Stream Processing 8 Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Monitoring and Analytics Not just fast, distributed and cross layer monitoring  Hard to collect some incident related data for quality of data Points of data data at collections for incident rest detection Data Storage VM/Container Data (Provider j) In-processing Source data Data Resulting Source Analysis Task data Original data in Data …. data motion ML Library Source Data Broker Other systems in the VM/Container VM/Container Large number of pipeline (Provider i) (Provider k) data sources (e.g., IoT) Analytics: will be based on big data principles with ML Large-scale brokers and storage but dependency analysis is not trivial Complex big data processing frameworks and ML Dagstuhl Seminar 17441, Big Stream Processing 9 applications (e.g., Spark) Systems, 31 Oct, 2017. (c) Hong-Linh Truong

Thanks for your attention! Hong-Linh Truong Faculty of Informatics TU Wien rdsea.github.io Dagstuhl Seminar 17441, Big Stream Processing 10 Systems, 31 Oct, 2017. (c) Hong-Linh Truong