CIRUS : A Cloud Infrastructure for Real-time Ubilytics (aka - PowerPoint PPT Presentation

CIRUS : A Cloud Infrastructure for Real-time Ubilytics (aka ubiquitous big data analytics) Didier Donsez Université de Grenoble LIG / ERODS P.N@imag.fr 1 21/06/14 D. Donsez, CIRUS, EclipseCon 2014 From Processing.org

Thanks to Manfred for the introduction of my talk 3 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Web X.0 (X > 3) Emerging ICT domains ● Cloud Computing ● Big Data Analytics ● Internet of (Every)Things ● Social Networks ● Mobile computing ● Crowd sourcing ● Open data ● ... 4 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Internet of Things (IoT) Industrial IoT (IIoT) phones sensor RFID / NFC nodes SCADA robot Instrumentation Communication Mediation Decision Action Mining 5 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Big Picture of Internet of (Every)Things, Data and Services Network cell size WAN MAN LAN WLAN WSN PAN BAN Home Automation Smart Public Space Smart Cities Geographic SOHO Smart Building Scale Urban Spaces Industry 4.0 21/06/2014 6 6 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Big Picture of Internet of (Every)Things, data and services 7 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Internet(s) of Data, Things and Services ● Internet of (Chatty) Things ● Internet of Everything ● Internet of People ● Internet of My Things ● Industrial Internet of Things (IIoT) : Industry 4.0 ● Fog Computing ● Cyber-Physical Systems ● ... 8 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

What is Cloud Computing ? ● On-demand computing – IaaS, PaaS, SaaS – Public, Private, Hybrid, Community, User-Centric, Souverain ● Advantages – Virtualization, TCO, Resilience, Elasticity, Energy efficency, Big Data Analytics … ● Drawbacks – Confidentiality (Privacy ,Industrial properties, …) – Souverainety 9 21/06/14 D. Donsez, Intergciels IoT

Cloud Services Models Software as a Service Saleforce, Steam, ... Platform as a Service Google App Engine, Amazon Hadoop ... Infrastructure as a Service Amazon EC2, ... Virtual/Physical Infrastructure (FaaS) Smart Green Grid IT cooling H 2 10

Cloud Computing : UbiCloud, Cloud of Things, ... ● U b i C l o u d – C l o u d s w i t h / f o r U b i - t e r m i n a l s ( s m a r t p h o n e s , t a b l e t cars, IDS, ...) ● Cloud of Things (CoT) – Cloud for Things (data collection and long-term storage ...) – T h i n g s a r e f a c i l i t i e s i n t h e F a a S 11 21/06/14 D. Donsez, Intergciels IoT

Cloud of Things Xively, Software as a Service (SaaS) Axeda, (on-demand access to any applications) Eurotech Platform as a Service (PaaS) (on-demand platform of delivering your own application) Deltadrone, Cloud Inrastructure as a Service (IaaS) Robotics (on-demand cpu/storage/nw infrastructures) Virtual/Physical Infrastructure (FaaS) Smart Green Grid IT cooling H 2 12 21/06/14 D. Donsez, Intergciels IoT

What is Big Data (Analytics) ? 13 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Big data is like teenage sex ● “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…” ● Dan Ariely, Professor at Duke University, TED speaker 14 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

The Data Deluge and the next IoT Data Deluge 1ZB=10^12 GB http://www.snia.org/sites/default/files2/ABDS2012/Tutorials/RobPeglar_Introduction_Analytics%20_Big%20Data_Hadoop.pdf 15 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

The 4+1V of Big Data V olume V elocity V ariety Ve racity + V alue 16 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

IoT Big Data 5V V o l u m e V elocity V ariety V eracity V alue 17 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Big Data and IoT ● Ubilytics : Ubi quitous big data ana lytics ● Realtime prediction on the sensor data flows ● For realtime decision (ie action) ● Mixin with other data sources ● Corporate data ● Open Data (gov, ...) ● Crowd-sourced data ● Social networks posts/tweets ● ... Voir Talk ICAR 2013 : Big Data par Jean-Laurent Philippe, http://erods.liglab.fr/icar2013/programme.html#intel 18 21/06/14 D. Donsez, Intergciels IoT

How analyze the Iot Data Deluge ? ● Fastly (hours) and Very Fastly (milliseconds) ● For speeding and improving decision supports → Business Intelligence tools (OLTP, OLAP, …) can't ! Now computing models are avalaible • Massively distributed, on-demand, fault tolerant • But • All old-fashioned statistical/prediction methods must be rethink 19 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Computing Models for Big Data ● P o s t - P r o c e s s i n g B a t c h ● Continuous Event Streaming – 100 MB/s of live data – TBs / PBs of stored data ● High-latency Decision Support ● Low-latency Decision Support → Map Reduce → Event Stream Processing H a d o o p , S c i D B , S p a r k , G i r a p h , . . . – – S t o r m , S 4 , S a m z a , M i l l w h e e l , . . ● Map-Update ● Discretized Stream ● MUD8P Processing ● Spark Streaming 20 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Event Stream Processing ● M a s s i v e l y d i s t r i b u t e d p r o c e s s i n g o f c o n t i n u o u s flows of events (sensors data, ...) – L o w - l a t e n c y ( f e w m i l l i s e c a f t e r ) mutable state mutable state input records node 1 node 3 output records node 5 input records node 2 Node 4 21

Example : Event Stream Processing Trending Topics VoIPSTREAM (VS) Twitter Sentiment Analysis (SA) From Maycon Bordin's ms thesis 22

Lambda Architecture Nathan Marz (Twitter, Backtype) ● Combine Low and High Latency BD stacks Batch layer can compute the analytics model of the Speed Layer l e d o M l a c i t y l a n A http://jameskinley.tumblr.com/post/37398560534/the-lambda-architecture-principles-for-architecting

Ubilytics Ubiquitous Big Data Analytics ● Motivation – PaaS for « Ubilytics » ● Autonomic : scalability, fault tolerance – End-to-End ● From sensors, gateways and lambda architecture (cloud) – « Simple as Possible » ● → for IoT SMEs & their IoT data scientists ● Problem – Huge variety of needs – Huge variety of technologies ● 24 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

New trendy Job : IoT Data Scientist Gartner says big data creates big jobs: 4.4 million IT jobs globally to support big d a t a b y 2 0 1 5 . http://www.gartner.com/newsroom/id/22 07915 The U.S. could face a shortage by 2018 of 140,000 to 190,000 people with "deep analytical talent" and of 1.5 million people capable of analyzing data in ways that enable business decisions. (McKinsey & Co) Big Data industry is worth more than $100 billion growing at almost 10% a year (roughly twice as fast as the software business) http://nirvacana.com/thoughts/becoming-a-data-scientist/ How can this guy deal with this deluge of technologies ? How to make this guy productive ?

Who is able to develop/ deploy Ublilytics infrastructures ? IoT Data Scientist Big Data Cloud Analytics Computing Internet of Things 26 21/06/14 D. Donsez, CAL & CIEL 2014

Towards "Ubilytics" PaaSs Send selected In Elastic sensors meausrements Hybrid Cloud M2M Mashup, NoSQL Store Gateway In elastic Cloud MongoDB, Reporting, @ Home, Office, City, Dashboard, . Cassandra, HDFS Message Broker Warehouse Message Broker (history charts, ...) (OpenHAB, IoTSys) or MaaS or PSaaS or MaaS or PSaaS Mosquitto, RabbitMQ, … Mosquitto, RabbitMQ, … Protocols : MQTT, AMQP, Protocols : MQTT, AMQP, Storing MapReduce STOMP, XMPP, CoAP, STOMP, XMPP, CoAP, agregates WebRTC, Motwin ... Hadoop WebRTC, Motwin ... SmartPhone Computed @ Car, City, … prediction model Realtime ESP Sensors data messages Storm, Samza, S4, ie energy Consumption, Predictions MQTT, AMQP, Embedded boards temperature, images , ... Spark Streaming, … STOMP ... Trends, ... smartphones topologies by millions Monitoring Placement (static,dynamic) Deployment & (Re)Configuration (Roboconf) 27 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Ubilytics Example Energy Consumption Forecast Domain : Smart Grid • 2125 individual smart plugs in 40 houses • measuring and sending instant load (W) and cumulative load (kW) Sensor Dataset*: • Events contain instant load (W) and cumulative load (kW) • 130 millions events/day (on one month) • 3 GB/day Challenge* : Forecasts loads at 1min, 5min, 15min, 60min and 120min Goal : anticipate electricity demand for ajusting the production (ie. save energy and avoid blackout) * DEBS GC 2014 http://www.cse.iitb.ac.in/debs2014/?page_id=42# 28

Ubilytics Example Energy Consumption Forecast MQTT Broker + Storm Topology + Cassandra DB on a Azure VM cluster instant load (W) and cumulative load (kW) 29

Smart Campus (ie Small Smart City) OpenHAB, Galileo, MQTT, Storm, Azure, ... 30 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Smart Campus 31 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Smart Campus 32 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

SmartCampus 33 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Smart Campus 34 21/06/14 D. Donsez, CIRUS, EclipseCon 2014

Q & A Q & A 35 21/06/14 D. Donsez, CAL & CIEL 2014