An Open-Source Streaming Machine Learning and Real-Time Analytics Architecture Using an IoT example (incubating) (incubating) Fred Melo William Markito @fredmelo_br @william_markito 1
Traditional Data Analytics - Limitations Store Analyti cs Data Lake HDFS No real-time Hard to change information Labor intensive ETL based Inefficient Data-source specific 2
Stream-based, Real-Time Closed-Loop Analytics In-Memory Real- Data Stream Pipeline Time Data Expert System / Data Lake HDFS Machine Learning Continuous Learning Multiple Data Continuous Sources Improvement Real-Time Processing Continuous Adapting Store Everything 3
A Streaming Machine Learning for IoT Example Predictive Maintenance Scenario Evaluates LIVE DATA “ According to historical Real-Time trends, there’s an 80% chance this equipment would fail in the next 12 Sensor Data hours" Live data becomes historical over time Smart System Learns with HISTORICAL TRENDS Historical " How were the temperature and vibration sensors reading when the latest failures happened? " 4
Streaming Machine Learning Info Look at past trends Machine Learning (for similar input) Evaluate current input Analysis Score / Predict 5
Streaming Machine Learning Info Filter Machine Learning Analysis [ json ] 6
Streaming Machine Learning Info Filter Enrich Machine Learning Analysis 7
Streaming Machine Learning Info Filter Enrich Transform Machine Learning Analysis 8
Streaming Machine Learning Info ML Model Filter Enrich Transform Analysis 9
Streaming Machine Learning Info ML Model Filter Enrich Transform Analysis Transform 10
Streaming Machine Learning ML Model In-Memory Data Grid Update Push Front-end 11
Streaming Machine Learning Supervised Learning Example Neural Network Real-time In-Memory Data Grid scoring Train 12
A Streaming Machine Learning Reference Architecture Other Sources and Destinations Distributed Computing JMS Fast Data Sink Ingest Transform SpringXD Store / Analyze Predict / Machine Learning 13
Indoors Localization - Applied Example 14
Trilateration and its limitations Noisy Data Physical Barriers Large Overlap Areas Moving Targets Innacuracy Large Overlap Areas 15
Particle Filters - Calculating the optimum solution 16
Particle Filters - Calculating the optimum solution 17
The Solution 1. Capture signal strength 2. Calculate distance from antenna 3. Trilaterate different sensors to predict location in real-time 4. Show on a map with live updates 18
Architecture Overview Calculate Device Predict Distance Location Groovy Spring Boot + Distance JSON HTTP Transform Sink Ingest SpringXD GUI Application Platform 19
Geode Basic Concepts • Cache • Configurable through XML, ,Java • Region • Distributed j.u.Map on steroids • Highly available, redundant • Member • Locator, Server, Client • Callbacks • Listener, Writer, AsyncEventListener, Parallel/Serial 20
Introduction to SpringXD Runs as a distributed application or as a single node 21
Spring XD A stream is composed from modules . Each module is deployed to a container and its channels are bound to the transport . 22
Demo
Why have we selected those projects In-memory & Persistent • Iterative & Exploratory Productivity • • Highly Consistent model • Built-in connectors • Extreme transaction Web based REPL • • Cloud Agnostic • processing Multiple Interpreters • Highly Scalable • Thousands of concurrent • Apache Geode • Easy to setup • clients Apache Spark • Streams without coding • Markdown Reliable event model • • Flink • Python… • 24
Source code and detailed instructions available at: https://github.com/Pivotal-Open-Source-Hub/WifiAnalyticsIoT Follow us on GitHub! Fred Melo William Markito @fredmelo_br @william_markito 25 25
Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode (incubating), R and Spring XD Room: Tohotom - 14:30, Sep 30 Fred Melo, Pivotal, William Markito, Pivotal Fred Melo William Markito @fredmelo_br @william_markito 26 26
Recommend
More recommend
