THUIR at the NTCIR-14 Lifelog-3 (LIT Task): How does lifelog help the user’s status recognition Isadora Nguyen Van Khan, Pranita Shrestha, Min Zhang, Yiqun Liu and Shaoping Ma Tsinghua University z-m@tsinghua.edu.cn June 12, Tokyo, 2019
Introduction Ø User’s statuses can be used for self-monitoring or as feature for other works Ø Previous researches focused on physical or health statuses Ø Continue to enhance the user’s life according to his/her personal data Ø Statuses are a good indicators of both mental and physical health of a user 2
Related Work Ø Activities analysis Ø Characterizing [Wang 2013] , matching [Yamauchi 2016] , annotation [Xu 2017] , recognition [Doherty 2011] Ø Link lifestyle traits to groups of people [Doherty 2011] Ø Statistics to give meaningful results to users Ø Health analysis Ø Sleep, psychology, mood … [Soleimanidjan 2017-2018, Iijima 2016] Ø Detection and prediction Ø Personnalized advices Ø User Status has never been studied before in lifelogging: Ø Indoor v.s. Outdoor, Alone v.s. Not alone, Working v.s. Not working Ø Can be used as the basis of activities and health analysis 3
Lifelog Insight task: User Status Detection Is the user alone or is there at least one person in his/her surroundings? Is the user inside or outside? Is the user working? Ø Features Ø Non-visual features (e.g. biometrics, activity, environment) Ø Visual based features Ø Merged features 4
Features Non Visual Features Segmentation : subset of following images that describe the same scene One Sample per Image One Sample per Category Features Segmentation Features Users UserID UserID Average heart rate, Average Biometrics Heart Rate, Calories Calories, Sum of calories Sum steps, Average steps, Activities Steps, Activity Activity Location, City, Longitude, Location, City, Longitude, Environment Latitude, Time Latitude, Begin time, End time 5
Features Visual Features {"categories": [{"name": "others_", "score": 0.046875}, {"name": "object_screen", "score": 0.85546875}], "color": {"dominantColorForeground": "Black", "dominantColorBackground": "Grey", "dominantColors": ["Grey", "Black"], "accentColor": "858246", "isBwImg": false, "isBWImg": false}, "tags": [{"name": "indoor", "confidence": 0.9856416583061218}, {"name": "computer", "confidence": 0.9771684408187866}, {"name": "desk", "confidence": 0.8973867297172546}, {"name": "keyboard", "confidence": 0.8705658316612244}, {"name": "electronics", "confidence": 0.8533153533935547}, {"name": "display", "confidence": 0.3655721843242645}, {"name": "desktop", "confidence": 0.2967212498188019}, {"name": "mac", "confidence": 0.2967212498188019}, {"name": "laptop", "confidence": 0.0638200324050663}, {"name": "office", "confidence": 0.0339242832307443}, {"name": "monitor", "confidence": 0.023137086807164493}], "description": {"tags": ["indoor", "computer", "desk", "keyboard", "electronics", "table", "monitor", "sitting", "laptop", "white", "small", "desktop", "mouse", "holding", "standing", "video"], "captions": [{"text": "a desktop computer sitting on top of a desk", "confidence": 0.888279407495522}]}, "requestId": "a32a79a3- e008-430f-a265-a5de764ee846", "metadata": {"width": 3264, "height": 2448, "format": "Jpeg"}} (By MS Vision API) Clusters Cooking, teeth, brushing, covered, dirty, cluttered, hot, preparing, blender, pot, pan, toothbrush Markov Cluster Algorithm Riding, sidewalk, walking, carrying Boy, child, little, baby Nodes : words Edges : words describe the same segmentation pizza,fruit,blurry,restaurant,image,commercial, bar Pre-processing : cleaning/edge elimination/triangle elimination (S. Van Dongen 2000) 6
Features Merged Features 7
Models Ø Non-visual features and merged features: Ø Combination of Adaptive Boosting and C4.5 or Random Tree Ø Combination of Bagging and C4.5 or LMT Ø Random Forest Ø Visual features: Ø For each tag describing an image: we give the value of the cluster Ø The main present value is considered the right one 8
Results and Analysis ) Ø Accuracy: 1 𝐷 #,--'#. ( Adaboost + RT 𝑂 #$%&&'& ) ) 𝐷 #,--'#. + 𝐷 )0#,--'#. )∈#$%&&'& Ø Use non-visual features: *Segmentation: a series of continuous similar images Highest Highest Experiment Accuracy Accuracy (images) (segmentation*) Inside or 88.6% 79.3% Outside Alone or 72.1% 54.5% not Alone Working or not 79.1% 70.5% Working 11
Results and Analysis ) Ø Accuracy: 1 𝐷 #,--'#. ( Adaboost + RT 𝑂 #$%&&'& ) ) 𝐷 #,--'#. + 𝐷 )0#,--'#. )∈#$%&&'& Ø Use non-visual features: *Segmentation: a series of continuous similar images Highest Highest Effective Feature Non Effective Accuracy Accuracy Experiment Categories Feature Categories (images) (segmentation*) Inside or 1-Time, 2-Calories, 88.6% 79.3% Steps and Activity Outside 3-Latitude/Longitude 1-Time, Alone or 72.1% 54.5% 2-Latitude/Longitude UserID and City not Alone 3-Heart Rate Working or 1-Heart Rate, 2-Time, not 79.1% 70.5% 3-Latitude and UserID and Steps Working Longitude 12
Results and Analysis Ø Use visual features (image based): Ø Inside/Outside recognition: 95.9% Ø Alone/Not Alone recognition: 55.1% Ø Working/Not Working recognition: 76.4% Ø Use merged features (image-based): Experiment Highest Accuracy Correspondent Model Inside or Outside 99.7% Bagging + LMT Recognition Alone or not Alone 66.2% Random Forest Recognition Working or not Working 76.5% Bagging + LMT Recognition 13
Summary Ø First work on user status recognition with lifelog data Ø Success in the status recognition Ø Inside or outside: 99.6% with merged features Ø Alone or not alone: 72% with non-visual features Ø Working or not working: 79% with non-visual features Ø Limitations : for visual and merged features, the number of annotated samples is not sufficient enough 15
Summary Ø First work on user status recognition with lifelog data Ø Success in the status recognition Ø Inside or outside: 99.6% with merged features Ø Alone or not alone: 72% with non-visual features Ø Working or not working: 79% with non-visual features Ø Limitations : for visual and merged features, the number of annotated samples is not sufficient enough 16
More on Life Moment Search 17
z-m@tsinghua.edu.cn
Recommend
More recommend
