entry Application Using EMG Signals Qiang Yang, Yongpan Zou * , Meng - PowerPoint PPT Presentation

ArmIn: Explore the Feasibility of Designing a Text- entry Application Using EMG Signals Qiang Yang, Yongpan Zou * , Meng Zhao, Jiawei Lin, Kaishun Wu Shenzhen University 2020/2/8

Part 1 Motivation Part 2 System Overview Outline Part 3 Challenges & solutions Part 4 Evaluation Part 5 Conclusion

Motivation 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n Traditional keyboard or touch screen is too small on wearable devices

Extended keyboard 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n Infrared ray Bluetooth Flexible material 5 C o n c l u s i o n Large / Expensive !

ArmIn: EMG-based virtual keyboard 1 Motivation Low-cost 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d Small scale S o l u t i o n s 4 E va l u a t i o n Commercial hardware 5 C o n c l u s i o n Bind on your arm and input on the virtual keyboard!

EMG Signal collection 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n Stick electrodes on your forearm

System workflow 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n

System workflow 4. How to enhance the text-entry performance? 2. How to remove the effect of channel asynchrony? 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n 1. How to eliminate noise inference? 3. How to choose an effective model to recognize keystrokes?

Challenges 1 Motivation 1. How to eliminate noise pollution? 2 S y s t e m O v e r v i e w 2. How to remove the effect of channel asynchrony? 3 C h a l l e n g e s a n d S o l u t i o n s 3. How to choose an effective recognition model? 4 E va l u a t i o n 5 C o n c l u s i o n 4. How to enhance the text-entry performance?

1. How to eliminate noise pollution? 1 Motivation 2 S y s t e m O v e r v i e w Baseline wandering (BW) 3 C h a l l e n g e s a n d S o l u t i o n s Power line interference (PLI) 4 E va l u a t i o n Gaussian white noise (WGN) 5 C o n c l u s i o n

1. How to eliminate noise pollution? Power line interference (PLI) Produced by alternating current(AC) at 50Hz, 150Hz,… 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n Elliptic filter-based 3-order notch filter <15Hz 5 C o n c l u s i o n Baseline wandering (BW) Gaussian white noise (WGN) Bandpass Butterworth filter Soft threshold wavelet-based denoising

2. How to remove the effect of channel asynchrony? 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n Electrodes are attached at different positions of muscles, EMG 5 C o n c l u s i o n signals cannot be captured simultaneously in multi channels.

2. How to remove the effect of channel asynchrony? Observation SE can be used as a weight to balance 1 Motivation EMG signal and noise. 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d Because of the randomness of noises, S o l u t i o n s SE can be regarded as an indicator. 4 E va l u a t i o n Real EMG signal owns more power 5 C o n c l u s i o n so that can be described by RMS.

2. How to remove the effect of channels asynchrony? Definition: C(w) 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n Where w i denotes the i th window, i means the SE of i th window in j th channel, SE j RMS j is defined as the RMS of i th window in j th channel.

2. How to remove the effect of channels asynchrony? C(w) 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n Revision: 5 C o n c l u s i o n 1. Short pause or shift (1-0-1) → (1-1-1) Use threshold T to encode C(w), 2. Short time < 5 windows (0.3s) → 0 then endpoints can be detected.

2. How to remove the effect of channel asynchrony? 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n Endpoints can be detected even though that EMG signals of each channel are asynchronous.

4. How to choose an effective recognition model? 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n 5 C o n c l u s i o n Feature selection (Wrapper method) 10-fold cross validation

4. How to choose an effective recognition model? SVM / KNN / random forests (RF) / Discriminant Analysis (DA)? 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s Penalty coefficient C 4 E va l u a t i o n Kernel function coefficient γ 5 C o n c l u s i o n SVM

4. How to choose an effective recognition model? SVM / KNN / random forests (RF) / Discriminant Analysis (DA)? 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d Achieve a balance between S o l u t i o n s training time and performance 4 E va l u a t i o n 5 C o n c l u s i o n 5 KNN

4. How to choose an effective recognition model? SVM / KNN / random forests (RF) / Discriminant Analysis (DA)? 1 Motivation RF 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d Trees: Number of trees S o l u t i o n s Dim: number of branches in each node 4 E va l u a t i o n Discriminant Analysis(DA) Performance 5 C o n c l u s i o n ☆ Liner Discriminant Analysis(LDA) 84.43% DA Diaglinear Discriminant Analysis(DDA) 82.57% Quadratic Discriminant Analysis(QDA) 83.25%

4. How to enhance the text-entry performance? Intended word ： 𝑋 = 𝑥 1 𝑥 2 … 𝑥 𝑜 … Recognized letters ： T = 𝑢 1 𝑢 2 … 𝑢 𝑜 … 1 Motivation ሻ max 𝑄 𝑋 𝐽 ≈ max 𝑄(𝐽|𝑋ሻ × 𝑄(𝑋 2 S y s t e m O v e r v i e w 𝑊 𝑊 𝑜 𝑄 𝑚 𝑗 𝑥 𝑗 = ς 𝑗 𝑜 𝐷𝑁(𝑥 𝑗 , 𝑚 𝑗 ሻ 𝑄 𝐽 𝑋 = ς 𝑗 3 C h a l l e n g e s a n d S o l u t i o n s ሻ 𝑛𝑏𝑦 𝑄 𝑋 𝐽 ≈ 𝑛𝑏𝑦 𝑄(𝐽|𝑋ሻ × 𝑄(𝑋 𝑊 𝑊 𝑜 𝑄 𝑚 𝑗 𝑥 𝑗 × 𝑄(𝑋ሻ ς 𝑗 ≈ 𝑛𝑏𝑦 4 E va l u a t i o n 𝑊 𝑜 𝐷𝑁(𝑥 𝑗 , 𝑚 𝑗 ሻ × 𝑄(𝑋ሻ ς 𝑗 ≈ 𝑛𝑏𝑦 𝑄 𝑋 𝑑𝑏𝑜 𝑐𝑓 𝑝𝑐𝑢𝑏𝑗𝑜𝑓𝑒 𝑔𝑠𝑝𝑛 𝑑𝑝𝑠𝑞𝑣𝑡 𝑊 5 C o n c l u s i o n 𝐷𝑁(𝑥 𝑗 , 𝑚 𝑗 ሻ is the confusion matrix of letters recognition. Bayesian-based correction method

Experiment setup 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s Experiments on printed and ArmIn prototype physical keyboard 4 E va l u a t i o n 5 C o n c l u s i o n For left hand key area, 8 participants X 16 letters X 130 repetitions X 2 keyboards 8 participants X 15 words X 30 times

Evaluation 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s SVM achieves the best average accuracy ( 89.5% ) over all participants with the lowest variance 4 E va l u a t i o n (0.17%). 5 C o n c l u s i o n Although SVM has a higher training overhead threshold, it still achieves the highest accuracy when the training sample number reaches 40. We use it as optimal model. Among 8 participants, the best performance of them is 95.1% and the worst is 82.9%

Evaluation 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n For printed and physical keyboards, the average recognition accuracy can achieve about 89.5% and 87.5% , respectively 5 C o n c l u s i o n The lowest accuracy among all letters is 85.6% , which means that ArmIn holds a stable recognition accuracy among different letters.

Evaluation 1 Motivation 2 S y s t e m O v e r v i e w 3 C h a l l e n g e s a n d S o l u t i o n s 4 E va l u a t i o n With one candidate word, the accuracy rises to 43.6%. When two candidate words are displayed, the system can achieve 92.5% accuracy. 5 C o n c l u s i o n The performance can be enhanced further by considering more candidate words, e.g., 93% accuracy for three candidate words.

Conclusion We design and implement ArmIn with commercial EMG electrodes which can recognize fine-grained keystrokes. 1 Motivation 2 S y s t e m O v e r v i e w We conduct experiment to evaluate its performance, and results show ArmIn can recognize keystrokes and word with 3 C h a l l e n g e s a n d S o l u t i o n s accuracy of 89.5% and 92.5% (providing two candidates), respectively. 4 E va l u a t i o n 5 C o n c l u s i o n We prove the feasibility of designing a text-entry application using EMG signals, which opens up a new vision of HCI applications using EMG techniques.

THANK YOU Questions? Qiang Yang Shenzhen University yangqiang2016@email.szu.edu.cn