Unsupervised Recognition of Interleaved Activities of Daily Living - PowerPoint PPT Presentation

Unsupervised Recognition of Interleaved Activities of Daily Living through Ontological and Probabilistic Reasoning Gabriele Civitarese Daniele Riboni Timo Sztyler Heiner Stuckenschmidt Univ. of Cagliari Univ. of Milano Univ. of Mannheim Univ. of Mannheim Italy Italy Germany Germany IEEE International Conference on Pervasive Computing and 14.09.2016 1 Communications 2016

MOTIVATION ACM International Joint Conference on 14.09.2016 2 Pervasive and Ubiquitous Computing 2016

14.09.2016 Scenario Recognizing activities of daily living in a smart-home to support healthcare, home automation, a �o�e i�depe�de�t life, … We rely on unobtrusive se�so�s … IEEE International Conference on Pervasive Computing and Gabriele Civitarese 3 Communications 2016

14.09.2016 State of the Art and Open Issues Most a�ti�it� �e�og�itio� s�ste�s �el� o� … … supe��ised -based approaches: acquire expensive labeled data sets often user/environment-specific … k�o�ledge -based approaches: unfeasible to enumerate all activity patterns We propose an unsupervised method to recognize complex/interleaved ADLs Based on hybrid ontological – probabilistic reasoning ACM International Joint Conference on Gabriele Civitarese 4 Pervasive and Ubiquitous Computing 2016

14.09.2016 Our approach … … o�e��o�es d�a��a�ks of supe��ised -based approach not user/environment- spe�ifi�, �o e�pe�si�e data set, … … �elies o� se�a�ti� �elatio�s �a�ti�ities↔ e�e�ts� derived from ontological reasoning … �e�og�izes i�te�lea�ed a�ti�ities inferred by a probabilistic model ACM International Joint Conference on Gabriele Civitarese 5 Pervasive and Ubiquitous Computing 2016

MODEL AND SYSTEM ACM International Joint Conference on 14.09.2016 6 Pervasive and Ubiquitous Computing 2016

14.09.2016 Recognized System overview activity instances 3. Markov Logic Network (MLN) / MAP Inference MLN knowledge base 2. Statistical analysis of events semantic Event(se 1 ,et 1 ,t 1 ) correlations 1. Semantic Semantic integration correlation layer reasoner ACM International Joint Conference on Gabriele Civitarese 7 Pervasive and Ubiquitous Computing 2016

14.09.2016 1. Semantic Correlation Reasoner Why do we use Ontology (OWL2)? to de�i�e se�a�ti� �o��elatio�s �e�e�t t�pe ↔ a�ti�it� �lass� Ontology / Axioms OWL2 Reasoner infers {turn on stove} is a predictive sensor event type for {Prepare hot meal} and {Prepare tea} interact PPM Matrix stove silverware_drawer freezer prepare Hot meal 0.5 0.33 0.5 Cold meal 0.0 0.33 0.5 Tea 0.5 0.33 0.0 ACM International Joint Conference on Gabriele Civitarese 8 Pervasive and Ubiquitous Computing 2016

14.09.2016 2. Statistical Analysis of Events Input : PPM matrix and temporally ordered events infers most probable activity class for each event allows to define activity boundaries (activity instance candidate) activity instance candidate Temporal extension of MLN (MLN NC ) Our ontology Knowledge Base is translated Events into the MLN NC model ACM International Joint Conference on Gabriele Civitarese 9 Pervasive and Ubiquitous Computing 2016

14.09.2016 3. MLN / MAP Inference Observed predicates  0.5: hot meal  0.5: cold meal ADL  0.0: tea hot meal?  0.5: hot meal Event 1: opens freezer (1:00pm)  0.0: cold meal Event 2: turns on stove (1:02pm) cold meal?  0.5: tea tea? Hidden predicates belong to ADL & Sensor Event Sensor Event Hot meal Freezer Stove Gabriele Civitarese 10

EXPERIMENTS ACM International Joint Conference on 14.09.2016 11 Pervasive and Ubiquitous Computing 2016

14.09.2016 Data Sets We consider two well- k�o�� data sets … 1. CASAS (controlled environment) • Interleaved ADLs of twenty-one subjects • Sensors: movement, water, interaction, door, phone • Activities: fill medications dispenser, watch DVD, water plants, a�s�e� the pho�e, �lea�, �hoose outfit, … 2. SmartFaber (uncontrolled environment) • An elderly woman diagnosed with Mild Cognitive Impairment • Sensors: magnetic, motion, presence, temperature • A�ti�ities: taki�g �edi�i�es, �ooki�g, … IEEE International Conference on Pervasive Computing and ACM International Joint Conference on Gabriele Civitarese 12 Pervasive and Ubiquitous Computing 2016 Communications 2016

14.09.2016 CASAS (1/2) MLN NC (Dataset) MLN NC (Ontology) • Our approach outperforms HMM HMM (related work) ontological reasoning is effective Candidate • Refinement improves boundary precision Refined 0.9 3 2.5 0.85 F-Measure Minutes 2 0.8 1.5 0.75 1 0.7 0.5 0.65 0 0.6 Delta-Start Delta-Dur ac1 ac2 ac3 ac4 ac5 ac6 ac7 ac8 IEEE International Conference on Pervasive Computing and ACM International Joint Conference on Gabriele Civitarese 13 Communications 2016 Pervasive and Ubiquitous Computing 2016

14.09.2016 SmartFaber (2/2) MLN NC (Dataset) • unsupervised and supervised-based MLN NC (Ontology) results are comparable Supervised / SmartFarber • results were penalized by a poor Candidate choice of sensors Refined 25 0.9 0.85 20 F-Measure 0.8 Minutes 15 0.75 10 0.7 5 0.65 0.6 0 ac9 ac10 ac11 Delta-Start Delta-Dur ACM International Joint Conference on Gabriele Civitarese 14 Pervasive and Ubiquitous Computing 2016

DISCUSSION / FUTURE WORK ACM International Joint Conference on 14.09.2016 15 Pervasive and Ubiquitous Computing 2016

14.09.2016 Discussion Results with two large datasets of interleaved ADLs were positive, but... • … k�o�ledge e�gi�ee�i�g is �e�ui�ed ��uild o�tolog�� existing smart-home ontologies can be reused • … it is �uestio�a�le if o�e o�tolog� �a� �o�e� e�e�� ho�e adaptation/extension should be performed (semi-) automatically IEEE International Conference on Pervasive Computing and ACM International Joint Conference on Gabriele Civitarese 16 Pervasive and Ubiquitous Computing 2016 Communications 2016

14.09.2016 Future Work Extensive real- �o�ld e�pe�i�e�ts should sho� … … if a�d ho� the o�tolog� has to �e adapted … �hat happe�s i� a �ulti -user environment Ca� a�ti�e lea��i�g allo� to … … fi�e - tu�e e�isti�g �odels? �use�’s e��i�o��e�t/ha�its� … e�ol�e the o�tolog� a��o�di�g to the �u��e�t �o�te�t? ACM International Joint Conference on Gabriele Civitarese 17 Pervasive and Ubiquitous Computing 2016

THANK YOU FOR YOUR ATTENTION ACM International Joint Conference on 14.09.2016 18 Pervasive and Ubiquitous Computing 2016

BACKUP SLIDES ACM International Joint Conference on 14.09.2016 19 Pervasive and Ubiquitous Computing 2016

14.09.2016 Semantic Integration Layer • collects events data from a sensor network • applies preprocessing rules to detect operations Example f�idge doo� se�so� sig�aled �1�  the ope�atio� is �ope�i�g the f�idge� <Event(se 1 , et 1 , t 1 �, …, E�e�t� se k ,et k ,t k )> ACM International Joint Conference on Gabriele Civitarese 20 Pervasive and Ubiquitous Computing 2016

14.09.2016 MLN Model (detailed) Ontological constraints time-aware inference PPM Matrix temporal *PriorProbability knowledge-based Statistical analysis of events *InstanceCandidate / *Event Observed predicates *Event (SenEvent, EventType, Time) *PriorProbability (SenEvent, ADL, ActivClass, p) *InstanceCandidate (ADL, Start, Stop) Hidden predicates OccursIn (SenEvent, ADL) InstanceClass (ActivClass, ADL) Gabriele Civitarese 21