Selecting Home Appliances with Smart Glass based on Contextual Information Ubicomp 2016 *Quan Kong(Osaka University) Takuya Maekawa(Osaka University) Taiki Miyanishi(ATR) Takayuki Suyama(ATR) Advanced Telecommunications Research Advanced Telecommunications Research Osaka University Osaka University Institute International Institute International
Introduction- Control home appliances And In more Free your hands direct way Advanced Telecommunications Research Osaka University Institute International
Approaches for Home Appliances Control “ Alex, open the Voice Gesture curtain on the kitchen’s north wall.” Amazon echo Wearable Camera IR Emitter Wearable IR Camera Ben Z., S UI ’ 14 Shi F., 06 Advanced Telecommunications Research Osaka University Institute International
Our Approach Home Network Living Room + Television ・ Activity ・ Indoor position “ ON” Advanced Telecommunications Research Osaka University Institute International
Feature of Our Approach – Context-aware appliance selection Google Glass Context Information Activities Indoor positions non-paramet ric ・ Orientation Camera unsupervised learning ・ Accelerator ・ Light sensor … Why cont ext informat ion ? Related to the home appliances Distinguish between different appliances Advanced Telecommunications Research Osaka University Institute International
System Overview- Sensors Used in Our System Google Glass Smartphone Wi-Fi Light sensor Screen Accelerator Orientation Camera Microphone Orientation Microphone Camera Wi-Fi Accelerator Light sensor Appliance Image Activity Indoor position Head direction (First person view) Information Information Advanced Telecommunications Research Osaka University Institute International
System Overview- Initialization and Model Update Initialization 1 Stand in front of the appliance and take a 10 seconds video Select the prepared name of appliance in glass application 2 3 Images extracted from the video used as the training data for initializing Use the trained appliance selection model in the daily life 4 wrong estimation is corrected by the user and used as training data Update the model by using the collected history daily data 5 Advanced Telecommunications Research Osaka University Institute International
Proposed Method Feature Train(ed) Train(ed) Extraction Activity Positional Clust ering Model Model Train Dat a Feat ure Dat a (t est dat a) Estimated Appliance Attention Detection 0.981 Act 1 0.765 Act 2 Activity Multiple 0.543 Act 3 + … Kernel … Learning 0.923 Pos1 Position 0.865 Pos2 Classification CNN feature Appliance Image Sensor Data 0.443 Pos3 fc6:4096 … … Context feature dimensional Extract Deep Convolut ional CNN fc6 Neural Net work 1 Detect the user’s attention using orientation data Extract the attention time’s image feature and estimate the activity & position ( IGMM ) 2 3 Extracted above information as the input of appliance selection model (MKL) Advanced Telecommunications Research Osaka University Institute International
Proposed Method – Image feature extraction with DCNN CaffeNet Reference Model: ILSVR-2012 image feature Advanced Telecommunications Research Osaka University Institute International
Proposed Method- Unsupervised activity recognition and indoor positioning Learning Act ivit y and Posit ion Model ・ Use non-parametric learning approach IGMM for activity and position clustering Bedroom Kitchen 𝐸 � Test data Wi-Fi Position IGMM Feature Extraction Sleep Cook - Wi-Fi Activity - microphone IGMM - light - light 𝐸 � - Acc - Acc - microphone 𝐸 � : the invers of the distance between the test data and each cluster Feat ure ext ract ion of IGMM input Accelerator 3-axis combination signal Light sensor Average of illumination Signal strength values Microphone Average MFCC components Wi-Fi Advanced Telecommunications Research Osaka University Institute International
Proposed Method –Appliance selection using MKL A linear combinat ion of mult iple base kernels for image and cont ext feat ure Camera Deep image convolutional neural network Image features Wi-Fi Position Multiple Selected IGMM kernel Appliance learning Activity describe a different IGMM - light property of the data - Acc with multiple kernels context features - microphone Mult iple Kernel Learning 𝒍 𝒅𝒑𝒐𝒖𝒇𝒚𝒖,∗ : 𝑠𝑏𝑒𝑗𝑏𝑚 𝑐𝑏𝑡𝑗𝑡 𝑔𝑣𝑜𝑑𝑢𝑗𝑝𝑜 （ 𝑔𝑝𝑠 𝑑𝑝𝑜𝑢𝑓𝑦𝑢 ） 𝒍 𝒋𝒏𝒉,∗ : 𝑞𝑝𝑚𝑧𝑜𝑝𝑛𝑗𝑏𝑚 𝑙𝑓𝑠𝑜𝑓𝑚 �𝑔𝑝𝑠 𝑗𝑛𝑏𝑓� Decision Function : f �𝑦 ∗ � � 𝑏 � 𝑓 ��� 𝑙 ���,∗ � 𝑓 ������� 𝑙 �������,∗ � 𝑐 Advanced Telecommunications Research Osaka University Institute International
Evaluation – Data set Device for dat a collect ion floor plan and appliances Google Glass, Nexus 5 (in pocket) Sampling rate : 30Hz front door Semi-naturalistic collection toilet protocol faucet Activities follow the instruction 3 users X 10 sessions activities Activities fan prepare meals lounge bedroom air drawer lighting conditioner eat meals Random kitchen wash dishes TV kitchen faucet bedroom … lighting lighting lounge air conditioner watch TV kitchen curtain lounge curtain sleep Advanced Telecommunications Research Osaka University Institute International
Evaluation Result - Leave-one-session out cross validat ion 0.945 Proposed : Effect of context Proposed 0.936 activity + position + camera 0.955 0.886 Proposed w/o pos : Proposed w/ o pos 0.878 10% activity+ camera 0.894 0.912 Proposed w/o act : Proposed w/ o act 0.897 position + camera 0.928 0.859 Proposed w/ cam : Proposed w/ cam 0.862 only camera 0.857 0.812 SVM w/ cam : SVM w/ cam 0.812 only camera 0.813 F-measure SVM all : 0.844 SVM all 0.844 Recall activity + position + camera 0.845 Precision Advanced Telecommunications Research Osaka University Institute International
Evaluation Result – Confusion Matrix Visual confusion matrix of Proposed w/ cam Visual confusion matrix of Proposed • • air conditioner and lighting were relatively poor air conditioner and lighting were increased • can’ t distinguish between kitchen lighting and about 14%on average of F-measure • F-measure improved by about 10% on total bedroom lighting • average drawer performed not well Advanced Telecommunications Research Osaka University Institute International
Evaluation Result –Transition of Average F-measures 1 Proposed Proposed (reuse) 0.9 Proposed (imgaenet ) Reusing ot her users’ dat a 0.8 User1 F-measure User2 Activity User3 + Activity 0.7 + Activity Position Ut ilizing online image dat abase + Position Appliance Image Sensor Data Appliance Image Sensor Data Position -collect online 32% 28% 0.6 Appliance Image Sensor Data images for each category -find top-k similar 0.5 images for each appliance 0.4 0 2 4 6 8 10 12 sessions Advanced Telecommunications Research Osaka University Institute International
Conclusion We proposed a new method of appliance selection with a smart glass based on position and activity contextual information The effectiveness of contextual information in an appliance selection task has been confirmed in a real experiment environment. Context based method can also be used to enhance the performance of such other appliance selection approaches as speech, gaze direction, and beacon- based approaches Advanced Telecommunications Research Osaka University Institute International
Recommend
More recommend
