Independent LifeStyle Assistant (I.L.S.A.) A NIST ATP Program - PowerPoint PPT Presentation

Independent LifeStyle Assistant ™ (I.L.S.A.) A NIST ATP Program Karen Zita Haigh Karen.haigh@honeywell.com Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Team Members • Joe Keller Honeywell: Behavioral Informatics, Inc. • Liana Kiff • Anthony Glascok • Stephen Metz • David Kutzik • John Allen • Charles Obranovich • Peter Bergstrom • Olu Olofinboba EverCare, Inc. • Peter Bullemer • John Phelps • Todd Carpenter • Nancy Williams • Tom Plocher • Zhao Chen • Michelle Raymond • Gary Determan SIFT, LLC • Dal Vernon Reising • Wende Dewing • Harry Funk • Rose Mae Richardson • Michael Dorneich • Chris Miller • Victor Riley • Kevin Driscoll • Jeff Rye • Anthony Faltesek University of Minnesota: • Jon Schewe • Denis Foo Kune • Kathleen Krichbaum • Tricia Syke • Christopher Geib • David Toms • Michael Good Weiser Scott & Assoc., Inc. • Ryan Vanriper • Valerie Guralnik • Janet Myers • Don Vu • Karen Haigh • Tom Wagner • Steven Harp • Rand Whillock • Steve Hickman • Stephen Whitlow • Geoffrey Ho • Woodrow Winchester • Raj Gopal Prasad Kantamneni • Peggy Wu Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

In a Nutshell Program Objective Develop an intelligent home automation system with situation awareness and decision-making capability based on integration of diverse sensors, devices, and appliances to support caregivers and enable elderly users to live independently at home. Programmatics: Benefits: ∎ A NIST Advanced Technology Program ∎ Support elder independent living » 2.5 years (Nov ’00 – Mar ’03) ∎ Provide peace of mind to caregivers » $5.3 Million ∎ Support efficient quality care for ∎ Lead by Honeywell caregiving organizations » Behavioral Informatics, Inc. ∎ Cost savings for government and industry » SIFT, LLC ∎ Market growth for in-home product » United Health Group EverCare producers » University of Minnesota School of Nursing Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

The Vision ∎ Gather information about elder, activity, and home status by listening to the home and communicating with devices ∎ Assess the need for assistance based on the system’s understanding the elder’s condition and what activities are going on inside the home ∎ Respond to a given situation by providing assistance to the elder and getting help when necessary ∎ Share health and status information with authorized caregivers to help improve the quality and timely delivery of care Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

The Vision Lois is doing fine. I’ll check on her again this afternoon. Lois is fine. Lois ate breakfast at 8: 20. I t’s tim e to take your m edicine! 10:00 A.M. Time for medicine Lois is in the living room. Mom’s having a good day! Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Finding Relevant Features Factors contributing to institutionalization ∎ caregiver burnout ∎ medication mgmt, medical monitoring ∎ mobility, wandering, toileting, dementia, safety ∎ usability Technological feasibility & match ∎ demonstrable in 30 months ∎ fits I.L.S.A. vision of passive monitoring & support Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Initial Feature Set Monitoring Functions Service Features ∎ Mobility (general activity level) ∎ Reminders ∎ Verify medication taken ∎ Internet & phone access to elder ∎ Panic button activation activity ∎ Caregiver to-do lists ∎ Toileting ∎ Coordinate multiple caregivers ∎ Eating ∎ Environment (comfort/intrusion) Usability Features ∎ Password-free elder interactions Response Functions ∎ Operational modes ∎ Alarms ∎ Queries to elders ∎ Alerts ∎ Feature Controls ∎ Notifications ∎ Activity Reports User Interfaces ∎ Elder: Phone, webpad, eFrame ∎ Caregiver: Web, phone, email Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Softw are Architecture Requirements Each ILSA client and home will be very different and have specialized needs, so the system must be: ∎ rapidly deployable, ∎ easily configurable, ∎ highly modular, and ∎ adaptive to the environment. Modularity is critical both to functionality as well as expandability for a number of reasons: ∎ Integrate 3rd party functional units ∎ Flexibility of sensor and actuator suites ∎ Expansion of ILSA capabilities over time Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Agent Architecture Highly distributed -- can compute anywhere Highly modular -- can change or incorporate agents Agent Architecture Actuators & Sensors Displays Environment Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Layered Agents Response Execution Response Execution Talks to devices (displays & actuators) Unlayered Agents Unlayered Response Planning Response Planning Based on situation, creates general response plan -- what to do or who to talk to, how to present it, on what device Intent Inference Intent Inference Infer goals of actors; put multiple events together. Agents Situation Assessment & Response Monitoring Situation Assessment & Response Monitoring Based on evidence, predict ramifications. Validating Validating Increase confidence of patterns, eliminate false positives, weigh competing hypothesized patterns. Clustering Clustering Combine multiple sensor reports into a single event. Log Adapter Adapter Hardware Hardware Sensors Actuators Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Agent Architecture Response Execution Response Execution Response Plan/Exec Talks to devices (displays & actuators) Web Pager Phone Customization Response Planning Response Planning IDS Schedule Based on situation, creates general response plan -- what to do or who to talk to, how to Machine Learning present it, on what device Medication Mobility Intent Inference Intent Inference Eating Infer goals of actors; put multiple events Client Agent together. Home Agent Situation Assessment & Situation Assessment & Agent Response Monitoring Response Monitoring CG Log Mgr Based on evidence, predict ramifications. Clustering Clustering Combine multiple sensor Event Recog reports into a single event. Agent Layer Agent Layer Sensor Adapter Log Adapter Adapter Device Layer Device Layer Sensors Actuators Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

ILSA Agents Agents group functionality, e.g. ∎ Mobility monitor ∎ Medication monitor ∎ Client interaction module ∎ Device controllers Agents group technical capability, e.g. ∎ Machine Learning ∎ Task tracking ∎ Response Planning Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Device Agents Intelligent, coordinated integration of multiple sensors, effectors and and displays ∎ Use standard communication protocols and the Ontology to seamlessly incorporate new devices » sensing into the situation-aware infrastructure » actuation / displays from response planner ∎ Cluster information from low cost, fault- vulnerable devices of disparate types to provide information about the client’s behaviour Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Task Tracking Recognize what the client is doing: ∎ Considers all hypotheses and actively reweights them as new evidence is added ∎ Can recognize that one sensor sequence may mean two different things (competing possibilities), ∎ Be aware of how confident it is in the recognized sequence (e.g. competing possibilities, or noisy sensors), ∎ Handle missed actions (e.g. when a sensor failed) ∎ Recognize what the person was TRYING to do, even if they didn't actually succeed or have not yet completed the task Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Response Planning Given a (set of) recognized situations, decide what to do: ∎ who : client, caregiver, house, external environment ∎ what : gather more evidence, interact (alarm, alert, remind, notify) ∎ where : location of devices ∎ when : degree of intrusiveness (severity) ∎ how : multiple devices, presentation format ...in a coordinated way, without overloading the resources (device or human) Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Adaptive User Interfaces Adaptive Interaction Design ∎ Use models of domain, task, and user(s) to dynamically design and create interactions ∎ Incorporate more divergent multi-modal devices ∎ Support less capable audiences, with changing capabilities ∎ Support a more varied, less predictable home situation Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Machine Learning Learn models of the actors and environment to automatically improve the performance of the system: ∎ what is normal / unusual, for elder, caregiver and other environmental factors ∎ what is the most effective technique to use ∎ understand sensor reliability ∎ etc Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.

Domain agent example: Medication Situation assessment from sensor events Asks Task Tracker for client intent Requests alerts and notifications for anomalous events Reminds according to schedule and recent activity Uses machine learning to adjust schedule, and likely task performance Karen Haigh, Autonomous Agents and Multi-Agent Systems, July 2002.