Bridging the Gap between Smart Home Platforms and Machine Learning - PowerPoint PPT Presentation

René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann Bridging the Gap between Smart Home Platforms and Machine Learning using Relational Reference Attribute Grammars Models@run.time 2019 September 17, 2019

Context and Goal https://www.youtube.com/watch?v=HSFO1aD_w8o Smart Home Platforms + Machine Learning with RAGs 2 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Context and Goal Control lights Recognize arrival https://www.youtube.com/watch?v=HSFO1aD_w8o Smart Home Platforms + Machine Learning with RAGs 2 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Two Important Trends More “Smart” Devices Commercial Machine Learning — “20 billion internet-connected things by — “Machine learning has progressed 2020” [Gartner, Inc., 2017] dramatically over the past two decades, — Middleware platforms manage large from laboratory curiosity to a practical number of devices, abstract hardware technology in widespread commercial details, and provide static rules use” [Jordan and Mitchell, 2015] Smart Home Platforms + Machine Learning with RAGs 3 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Two Important Trends More “Smart” Devices Commercial Machine Learning — “20 billion internet-connected things by — “Machine learning has progressed 2020” [Gartner, Inc., 2017] dramatically over the past two decades, — Middleware platforms manage large from laboratory curiosity to a practical number of devices, abstract hardware technology in widespread commercial details, and provide static rules use” [Jordan and Mitchell, 2015] Combination of Smart Home Platforms and Machine Learning — Middleware only use static rules, machine learning alone can not be used easily at runtime — Very few approaches integrating both, and none uses Open Source Middleware Platforms Smart Home Platforms + Machine Learning with RAGs 3 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

State-of-the-art: Middleware Platforms openHAB Written in Java, based on Eclipse Smarthome. Concepts: Thing, item, type Home Assistant Written in Python. Concepts: state object, event, zone, scene ioBroker Written in JavaScript. Concepts: Adapter, instance, (UI) object FHEM Written in Perl. Concepts: Devices, events Smart Home Platforms + Machine Learning with RAGs 4 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Research Questions RQ1 How to integrate multiple smart home middleware platforms with multiple machine learning components? RQ2 Which model elements and relations between them are necessary for such an integration? RQ3 How to ease the selection of relevant inputs for different instances of machine learning approaches? newData ◮ Machine Smart Home Learning Middleware 1 Component 1 Middleware 2 Component 2 Middleware 3 Component 3 ◭ classify Devices Smart Home Platforms + Machine Learning with RAGs 5 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Research Questions RQ1 How to integrate multiple smart home middleware platforms with multiple machine learning components? RQ2 Which model elements and relations between them are necessary for such an integration? RQ3 How to ease the selection of relevant inputs for different instances of machine learning approaches? newData ◮ Machine Smart Home Learning Middleware 1 Component 1 Runtime Middleware 2 Component 2 Model Middleware 3 Component 3 ◭ sync ◮ ◭ classify Devices Smart Home Platforms + Machine Learning with RAGs 5 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Research Questions RQ1 How to integrate multiple smart home middleware platforms with multiple machine learning components? RQ2 Which model elements and relations between them are necessary for such an integration? RQ3 How to ease the selection of relevant inputs for different instances of machine learning approaches? newData ◮ Machine Smart Home Learning Middleware 1 Component 1 Runtime Middleware 2 Component 2 Model Middleware 3 Component 3 ◭ sync ◮ ◭ classify Devices Smart Home Platforms + Machine Learning with RAGs 5 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Solution: A Structured Model Smart- phone OpenHAB Smart- watch Smart Light MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- phone OpenHAB Create a Structured Knowledge Base Smart- integrating all investigated watch MQTT middleware platforms Broker Knowledge Base Smart Light MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- phone OpenHAB Smart- watch MQTT Broker Knowledge Base Smart Light OpenHAB Home I/O Simulator Brightness Sensor MAPE Control Flow Data Flow Component

Solution: A Structured Model Extend the Resulting Knowledge Base Smart- M A phone with new information needed for self-adaptive behaviour OpenHAB Smart- E P watch MQTT Broker Knowledge Base Smart Light OpenHAB Home I/O Simulator Brightness Sensor MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- M A phone Activity OpenHAB Learning Component Smart- E P watch MQTT Broker Knowledge Base Smart Light Preference OpenHAB Learning Home I/O Simulator Component Enable Machine Learning Algorithms Brightness to learn user’s preferences Sensor MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- M A phone Activity OpenHAB Learning Component Smart- E P watch MQTT Broker Knowledge Base Smart Light Preference Represent the Relations OpenHAB Learning Home I/O Simulator between machine learning Component models and elements in the Brightness structured knowledge base Sensor MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- M A phone Activity OpenHAB Learning Component Smart- E P watch MQTT Broker Knowledge Base Smart Light Preference OpenHAB Learning Home I/O Simulator Component Brightness Sensor MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- M A phone Activity Home Learning Assistant Component Smart- E P watch MQTT Broker Knowledge Base Smart Light Preference OpenHAB Learning Home I/O Simulator Component Brightness Sensor MAPE Control Flow Data Flow Component

Solution: A Structured Model Smart- M A phone Activity Home Learning Assistant Component Smart- E P watch MQTT Broker Knowledge Base Smart Preference Light Learning Component OpenHAB Home I/O Simulator with Decision Brightness Trees Sensor MAPE Control Flow Data Flow Component

The Five Aspects (to be) Modelled with RAGs Smart Home User Entity Model affects has located in Preferences valid for valid for performs nearby happens in Location Activity located in Smart Home Platforms + Machine Learning with RAGs 7 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

The Five Aspects (to be) Modelled with RAGs: Smart Home Entities Thing Channel * * 1 Smart ThingType * Home * 1 Entity Channel Model ChannelType Category * * Item ItemMetaData * * * linkedItems Group * * Smart Home Platforms + Machine Learning with RAGs 8 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

The Five Aspects (to be) Modelled with RAGs: Smart Home Entities 1 SmartHomeEntityModel ::= Thing* Group* ThingType* ChannelType* ChannelCategory* ItemCategory* /ActivityItem:Item/ ; 2 ThingType : DescribableModelElement ::= Parameter* ; 3 rel ThingType.ChannelType* -> ChannelType ; 4 abstract Item : LabelledModelElement ::= <_fetched_data:boolean> MetaData:ItemMetaData* [ItemObserver] ; 5 rel Item.Category? -> ItemCategory ; 6 rel Item.Controlling* <-> Item.ControlledBy* ; 7 8 abstract ItemWithBooleanState : Item ::= <_state:boolean> ; 9 abstract ItemWithStringState : Item ::= <_state:String> ; 10 abstract ItemWithDoubleState : Item ::= <_state:double> ; 11 ColorItem : Item ::= <_state:TupleHSB> ; 12 DateTimeItem : Item ::= <_state:Instant> ; 13 ContactItem : ItemWithBooleanState ; Smart Home Platforms + Machine Learning with RAGs 8 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019

Architecture Details (1/2) Modelled using Relational RAGs — Declarative specification of structure, and its relations — Analyses on this structure — Definition of relations rel MachineLearningModel.RelevantItem* <-> Item.RelevantInMachineLearningModel* ; rel MachineLearningModel.TargetItem* <-> Item.TargetInMachineLearningModel* ; Smart Home Platforms + Machine Learning with RAGs 9 / 16 René Schöne, Johannes Mey, Boqi Ren and Uwe Aßmann September 17, 2019