Aging in Place Marjorie Skubic Associate Professor, Electrical and - PDF document

3/31/2009 Recognition Technology for Aging in Place Marjorie Skubic Associate Professor, Electrical and Computer Engineering Dept. Director, Center for Eldercare and Rehabilitation Technology University of Missouri Research Team  MU Engineering  Faculty: Marge Skubic, Jim Keller, Henry He, Harry Tyrer, Tina Smilkstein, Tony Han  Students/Staff: Derek Anderson, Tim Havens, Bob Luke, Isaac Sledge, Shuang Wang, Fang Wang, Erik Stone, Kevin Reed, Nick Harvey, Wenqing Dai, Tanvi Banerjee, Dane Guevara, Chad Godsy, Matt Nevels, Ian Kable  MU Nursing  Faculty: Marilyn Rantz, Myra Aud, Bonnie Wakefield, Greg Alexander  Students: Jie Yu, Jean Krampe  MU Health Informatics / Physical Therapy / Social Work  Faculty: Mihail Popescu, Carmen Abbott, Deb Oliver, Colleen Galambos  Students: Saurav Garg , Rohan Ohol, Jarod Giger  MU Family Medicine  Richelle Koopman  University of Washington  Faculty: George Demiris 1

3/31/2009 Outline  Motivation  TigerPlace  Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors  Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project  Conclusions  What do the residents think? TigerPlace A new retirement community developed by the University of Missouri in affiliation with Americare Systems, Inc. 2

3/31/2009 The Challenge Give us technology for TigerPlace that will help elder residents stay as active and functionally independent as possible. Initial Focus Group Results  Older adults were already using computers in sophisticated ways  Participants were willing to accept technology if it  Met a need  Had an appropriate interface (watch sensory limitations)  Wearable sensors were considered obtrusive  Privacy concerns with cameras were raised  But participants said they might accept silhouette-based cameras Demiris, Rantz, Aud, Marek, Tyrer, Skubic & Hussam, Informatics and The Internet in Medicine , 2004. 3

3/31/2009 TigerPlace Designed for Aging in Place TigerPlace Residents  34 residents, aged from about 70 to 90  4 married couples + 26 singles + 2 dogs, 1 cat  90% have a chronic illness  e.g., arthritis, heart disease, diabetes, stroke risk  60% have multiple chronic illnesses  Some early stage Alzheimers  15% use a walker, 1 wheelchair, 1 in leg braces  Residents tend to be socially active 11 11 4

3/31/2009 Trajectory of Functional Decline  Detect falls Functional Decline  Assess gait & activity level  Assess normal patterns  Recognize pattern changes  Detect acute illness onset or Time Time Current trend changes Aimed trend (with technology) Rantz, Marek, Aud, Tyrer, Skubic, Demiris & Hussam, Nursing Outlook , 2005 Our Goal: Keep them functionally active! 5

3/31/2009 Outline  Motivation  TigerPlace  Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors  Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project  Conclusions  What do the residents think? Design Decisions  Do not put any sensors or devices on the person.  The system should be transparent to the elder.  Minimize changes to the environment. 6

3/31/2009 Integrated Sensor Network Physiological (1) sensor network stove sensor Data monitor bed sensor Web Database Activity motion sensors server analysis Residents, Family and Web Customization interface Healthcare Fusion + (6) manager (5) reasoning Providers engine Alert (4) (3) manager (2) Activity Video sensor analysis network video Silhouette sensors extraction Skubic, Alexander, Popescu, Rantz, Keller, Technology & Health Care , in press. 16 Basic Sensors 7

3/31/2009 Video Processing Low cost Young Elliott cameras (on Anderson Tripods for the picture) Priceless $100 each “Little Elliott”: Low cost CPU with GPU for processing power $500 A typical sensor network 14 motion sensors 1 Bed sensor unit 1 Stove temp unit 1 PC appliance 8

3/31/2009 Outline  Motivation  TigerPlace  Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors  Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project  Conclusions  What do the residents think? Sensor Network Installations Res.# 2005 2006 2007 2008 # Mo. 1 17 2 37 3 30 4 28 5 5 6 8 7 22 8 22 9 22 10 12 11 12 12 12 13 11 14 10 15 8 16 3 17 3 21 9

3/31/2009 Case Study #1 Increase in Bed Restlessness Following Surgery Heart attack and bypass surgery Rantz, Skubic, Miller, and Krampe, ICOST 2008 Case Study #1 Return to Normal Bed Restlessness Following Cardiac Rehabilitation 10

3/31/2009 Case Study #2 Bradycardia (slow pulse rate: 1 -30 beats per minute) ER visit Hospitalization Rantz, Skubic, Miller, and Krampe, ICOST 2008 Case Study #2 Decrease in Bed Restlessness ER visit Hospitalization 11

3/31/2009 Motion Sensor Density Map Each line is one day Black indicates time out of apartment Color indicates density of motion activity One resident in March later in October Wang and Skubic, IE 2008 29 Two different residents A more active pattern A pattern showing irregular activity Wang and Skubic, IE 2008 30 12

3/31/2009 Combining Motion Sensor and Bed Sensor Data  32 dimensional feature vectors  Bed Features: time awake/asleep, # OOB morning/night, time in bed morning/night, nap time, time OOB morning/night, amount slept, stats. rest/pulse/breath firings  Motion Features: # daily bathroom visits, puttering PCA reduction index, time in each room, stats. of motion firings Sledge, Keller & Alexander, EMBC 2008 31 A Temporal View 32 13

3/31/2009 Growing Neural Gas for Temporal Clustering 10/10/2005 to 01/18/2006 10/10/2005 to 06/17/2006 Sledge & Keller, ICPR 2008 33 10/10/2005 to 11/14/2006 Interview Results  No privacy concerns with motion and bed monitoring  Some discomfort with bed sensor  Three stages in adjustment to sensor network  Familiarization (2-3 weeks)  Adjustment and curiosity (2-3 weeks)  Integration: participants forget about the sensors after about one month and have not changed their activities  Residents have asked to see the data themselves  Residents participate because they are helping us Demiris, Parker Oliver, Dickey, Skubic, Rantz, Technology and Health Care , 2008. 34 14

3/31/2009 Outline  Motivation  TigerPlace  Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors  Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project  Conclusions  What do the residents think? Use Silhouettes for Privacy 15

3/31/2009 Silhouette Improvement Fuse color and texture Undetected Original Image Color Feature Improvement Undetected Fusion (no post processing) Texture Feature Results of Silhouette Extraction RGB only Color and Texture RGB Method Color/Texture Method Percentage of Percentage of Pixels Percentage of Percentage of Pixels Foreground Pixels Incorrectly Classified Foreground Pixels Incorrectly Classified Found as Foreground Found as Foreground Sequence 1 99% 9% 70% 7% Sequence 2 97% 5% 66% 6% Sequence 3 96% 8% 79% 6% 47 Total 98% 7% 69% 6% Luke, Anderson, Keller & Skubic, 2008. 16

3/31/2009 Outline  Motivation  TigerPlace  Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors  Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project  Conclusions  What do the residents think? 17

3/31/2009 Creation of Voxel Person Silhouette Raw Video 1 Raw Video 2 Silhouette Extraction Extraction Voxel Camera Space Silhouette 1 Silhouette 2 Voxel Lists Intersection Voxel Person Voxel Space 18

3/31/2009 Voxel Person in Action Fuzzy Activity Analysis Initial Implementation - 24 Fuzzy Rules INPUTS OUTPUTS 1. Mean Red = Upright 2. Eigen based height Green = In between 3. Max eigenvector and Blue = On the ground ground plane normal similarity Anderson, Luke, Keller, Skubic, Rantz & Aud, IEEE TFS , 2009. 19

3/31/2009 membership … Sum 1 Sum 2 Sum G Level 2: Fuzzy logic for reasoning about activity { Sum 1 , Sum 2 , … , Sum G } Linguistic Summarizations membership S 1 S 2 S 3 time Level 1: Fuzzy logic for state based reasoning … … Voxel Person … … time t-1 time t Silhouette Extraction Linguistic Summarization  <Person> is <State> in <Location> for <Time>  where <State>, <Location>, and <Time> are linguistic variables LINGUISTIC SUMMARY EXAMPLES: Derek is upright in the living room for a moderate amount of time. Derek is on the ground in the living room for a moderate amount of time. 60 20