and Electrooculography Features Ruofei Du 1 , Renjie Liu 1 , - PowerPoint PPT Presentation

Online Vigilance Analysis Combining Video and Electrooculography Features Ruofei Du 1 , Renjie Liu 1 , Tianxiang Wu 1 , Baoliang Lu 1234 1 Center for Brain-like Computing and Machine Intelligence Department of Computer Science and Engineering 2 MOE-Microsoft Key Lab. for Intelligent Computing and Intelligent Systems 3 Shanghai Key Laboratory of Scalable Computing and Systems 4 MOE Key Laboratory of Systems Biomedicine Shanghai Jiao T ong University 800 Dongchuan Road, Shanghai 200240, China Shanghai Jiao T ong University ICONIP 2012

Outline • Motivation • Introduction • System Overview • Video Features • Electrooculography • Conclusion and Future Work Shanghai Jiao T ong University ICONIP 2012

Motivation • 60 600, 0, 00 000 0 people die from traffic accidents every year, and • 10 10,0 ,000 00,0 ,000 00 people get injured throughout the world. • 60 60% % of adult drivers – about 16 168 million people – say they have driven a vehicle while feeling drowsy in 2004 in the U.S. Drowsy driving results in 550 550 deaths, 71 71,0 ,000 00 injuries, and $1 $12. 2.5 billion in monetary losses. • In China, 45 45.7 .7% % accidents on the highway are caused by fatigued driving. Shanghai Jiao T ong University ICONIP 2012

Introduction Vid Video eo EO EOG EEG EEG Intrusive Least Moderate Most Moderate, Most Moderate, need to Accuracy influenced by accurate denoise. luminance Eye movement, Eye blinks, Delta waves (Slow Features yawn state and movement Wave Sleep) and theta facial orientation. and energy. waves (drowsiness) Shanghai Jiao T ong University ICONIP 2012

System Overview Shanghai Jiao T ong University ICONIP 2012

System Overview Train T est Shanghai Jiao T ong University ICONIP 2012

System Overview stimulus Black screen 5~7s 500ms One trial Shanghai Jiao T ong University ICONIP 2012

System Overview Shanghai Jiao T ong University ICONIP 2012

Visual Features • Video signals: By infrared cameras, 640 × 480, 30 frames/s • Face Detection: Haar-like cascade Adaboost classifier. • Active Shape Model: Locate the landmarks on the face. Shanghai Jiao T ong University ICONIP 2012

Visual Features • PERCLOS (percentage of closure): • Blink frequency, etc.: • Yawn frequency: • Body Posture: (By ASM) Shanghai Jiao T ong University ICONIP 2012

Linear Dynamic System 𝑄 𝑦 𝑢 𝑨 𝑢 = 𝑂 𝑦 𝑢 𝑨 𝑢 + ഥ 𝑥, 𝑅 𝑄 𝑨 𝑢 𝑨 𝑢−1 = 𝑂 𝑨 𝑢 𝐵𝑨 𝑢−1 + ҧ 𝑤, 𝑆 Shanghai Jiao T ong University ICONIP 2012

Electrooculography Shanghai Jiao T ong University ICONIP 2012

Forehead Signals Separated by ICA HEO VEO Shanghai Jiao T ong University ICONIP 2012

Electrooculography • Filter the vertical EOG signal by a low-pass filter with a frequency of 10Hz. • Adjust the amplitude of the signals. • Computer the difference of signals for the extraction of eye blinks. • 𝐸 𝑗 = 𝑊 𝑗 + 𝑗 − 𝑊 𝑗 × 𝑆 • where V denotes the signal, R as the sampling rate • Slow Eye Movement (SEM) and Rapid Eye Movement (REM) are extracted according to different kinds of time threshold. • Fourier transformation: 0.5Hz and 2Hz to process the horizontal EOG. • The sampling rate: 125Hz, time window: 8 seconds. Shanghai Jiao T ong University ICONIP 2012

Electrooculography Shanghai Jiao T ong University ICONIP 2012

Conclusion Shanghai Jiao T ong University ICONIP 2012

Conclusion Shanghai Jiao T ong University ICONIP 2012

Future Work • Smaller EOG chip: to • Comprehensive feature: depth information and grip power. • Robustness and stability: Various luminance, moving car, actual environment... Shanghai Jiao T ong University ICONIP 2012

Thank you BCMI: We are family! http://bcmi.sjtu.edu.cn Shanghai Jiao T ong University ICONIP 2012

Reference 1. May, , J., , Bald ldwin, in, C.: Driver r fa fatig igue ue: : The imp mportance tance of i f ide dentif tifyin ying g ca causa sal l fa fact ctor ors s of f fa fatig igue ue when co consi side dering ing de detect ctio ion n and co d counte terme measu asure tech chnolo ologi gies. s. Trans nspo portation tation Re Rese search ch Part t F : Traffi fic Psychology and Behaviour 12(3) (November 2009) 218– 224 224 2. Stutt tts, J., , Wilk lkins ins, J., Vaugh ghn, n, B.: Why do do pe peopl ple have dr drowsy sy dr driving ing cr crash shes. s. A Fo Foundat dation ion fo for Traffi fic Safety (202/638) 3. W Wang ang, Q., Yang, g, J., Re Ren, M., Zheng, g, Y.: Driver r fa fatigu tigue e de detect ctio ion: n: a su survey. Intellig ligent nt Control ol and d Automat tomation ion, 2006. WCICA A 2006. The Sixth th World rld Congr gress ss on 2 (2006) 8587 – 8591 8591 4. J Ji, , Q., Yang, g, X.: Re Real-time time eye, ga gaze, and fa d face ce po pose se track acking ing fo for mo monito itoring ring dr driver vigi gilanc lance. . Re Real-Time Time Ima magi ging ng 8(5) ( (2002) 357 – 377 5. Dinge ges, D., Malli llis, M., Mais islin lin, G., Powell, ll, I., et al.: : Fi Final al repo port: t: Evaluation aluation of f tech chniq nique ues s fo for ocu cular ar me measu surem ement nt as s an inde dex of fa f fatig igue ue and as d as the ba basi sis s fo for alertnes tness s ma manage ageme ment nt. . National tional Highway Traffi fic Safety Administration (HS 808762) (1998) 6. F Fletche tcher r LAp Apostol stoloffn ffnPete terso son L, L, e.a e.a.: : Visi sion n in and o d out of v f vehicl cles. s. IEEE Trans ns on Intelli llige gent nt Trans anspo porta tation tion Syst stems ms 18(3) 7. Ma, J., Shi, L., , Lu, , B.: Vigi gilanc ance est stimation mation by by u usi sing g elect ctroo oocu culog lograph raphic ic fe featur ures. s. Proce ceedi ding ngs s of 3 f 32nd I d Internat rnation ional al Confe ferenc nce of th f the IEEE E Engi gine neerin ering g in Medi dici cine ne and d Biolo logy gy Soci ciety ty (2010) 6591 – 6594 6594 8. W Wei, i, Z., Lu, , B.: On Online ine vigi gilanc lance anay aysi sis ba base sed d on elect ctroo oocu culog lograph raphy. . Internat rnatio iona nal l Joint int Confe ferenc nce on Neural ral Network works ( s (2012) 9. Cootes es, T., , T aylor lor, , C., Coope per, , D., Graham, aham, J., et al.: : Activ ctive e sh shape pe mo mode dels s - their r traini ining ng and ap d appl plicat cation ion. . Comp mputer r Visi sion and Im d Image ge Unde derst stand anding ng 61(1) (1995) 38 – 59 59 10 10. Viola, la, P ., Jone nes, s, M.: : Ra Rapi pid d obj bject ct de detect ction on usi sing g a bo boost sted ca d casc scade de of si f simp mple fe featur tures es. Proce ceedi ding ngs s of C f Comp mputer r Visi sion and P d Patt ttern rn Re Reco cogn gnition ition 1 ( (2001) I – 511 511 – I – 51810 51810 Ru Ruo-Fe Fei i Du, Re Ren-Ji Jie Liu, u, Tian an-Xiang Xiang Wu, , Bao-Lian Liang g Lu 11. Delorme rme, A., , Make keig ig, , S.: Eegl glab ab: an ope pen so source ce toolb lbox x fo for analy alysi sis o s of si f singl gle-tria trial l eeg eeg dy dynami mics cs incl clud uding ing inde depe pende dent nt co comp mponent t analys alysis. s. Journ urnal l of n f neuros osci cienc nce me methods ds 134(1) (2004) 9 – 21 21 12. Dinge ges, D., Grace ce, , R. R.: Percl clos os: A valid lid ps psych choph physi siolo ologi gical cal me measu sure of a f alertnes tness as s as ass ssess ssed d by by ps psych chomo motor tor vigi gilanc lance. . Fe Fede deral l High ghway way Administration. Offi fice of motor carriers, T ech ch. Re Rep. MCRT RT-98 98-00 006 (1998) 13. A. Bullin ling, g, J. Ward, d, H.G., , Trost ster er, , G.: Eye mo moveme ment nt analys alysis s fo for act ctivity vity reco cogn gnition ition usi sing g elect ctroo oocu culog lograph raphy. . Pattern ttern Analys alysis s and d Mach chine ine Intellige ligenc nce, , IEEE E Trans ansactio ctions ns (99) ( (2011) 1 – 1 14. Shi, L., Lu, B.: Of Off-line ine and o d on-lin line vigi gilanc lance est stimation mation ba base sed o d on linear ear dy dynamical mical sy syst stem an m and d ma manifold fold learnin rning. g. Engi gine neering ring in Medi dici cine ne and d Biolog logy Soci ciety ty (EMBC BC), , 2010 Annu nnual al Internat rnation ional al Confe ferenc nce of th f the IEEE E (Sept ptemb mber 2010) 6 6587 – 6590 6590 15. Chang ng, C., Lin, n, C.: Libs bsvm: : a libr brar ary y fo for su supp pport vect ctor r ma mach chines nes. ACM Trans nsactio actions ns on Intellig ligent nt Syst stems ms and d T ech chnolog logy y (TIST) 2(3) ( (2011) 27 Shanghai Jiao T ong University ICONIP 2012