Attention detection in driver simulator project #4 INPG : L. - PowerPoint PPT Presentation

Attention detection in driver simulator project #4 •INPG : L. Bonnaud, A. Caplier •UCL : D. Trevisan, B. Macq •Participants : A. Benoit, G. Chanel, P. Ngo, V. Levacic, C. Thillou •Guest star : L. Lawson, Burak

Introduction Heart rate EEG 8/19/2005 2

Dialog controller Alarm Loss of watchfulness (presentation task) Redundancy / Redundancy / Equivalence Equivalence Graphical EEG ECG Head Yawn ? Sonic frequency epresentation interpreter interpreter inclination ? representation Head Mouth EEG Eye ECG Screen Speakers motion motion Blinking detector detector detector Physiological Provided sensors Camera EEG: electroencephalogram To be developped ECG: electrocardiogram 8/19/2005 3

Architecture 8/19/2005 4

5 Challenges • Driver Simulator • Attention detection – Biological signals – Stress detection – Video-based information – Fatigue detection • Fusion • Fission • Integration – Distributed architecture – OpenInterface 8/19/2005 5

Driver Simulator 8/19/2005 6

Driver Simulator • TORCS - GPL program well built with source code well structured (C++ and OpneGl) • Force Feedback with controlled level of wheel vibration • Message (color) • Button click (user’s interaction) • Multi-thread server • The network protocol used is TCP/IP. We used a “GPL” library called Openthreads to allow threads access global variables with a Mutual Exception implementation 8/19/2005 7

Attention Detection • Video-based system • Fatigue detection – Eyes – Yawn – Head movement 8/19/2005 8

Attention Detection • Biological-based system (stress detection) • ECG and GSR • 3 situations : – rest / relaxation – Stress stimuli while reading • Hand clapping • Light in eyes • Answering simple question • Telephone call, “your dead” (killer game) – Playing with the driving simulator (difficult tracks) • GSR acquisition and analysis can be integrated in real time • New experiments for detecting relax situation 8/19/2005 9

Fusion Video image Video image Face-based detection Face-based detection Head, eyelid and Head, eyelid and Contextual Contextual Bayesian Information Information yawn movement yawn movement Network for Physical Physical fitness fitness detecting Data Fusion Data Fusion Sleep history Sleep history Fatigue State Bayesian Network Bayesian Network Time of day Time of day Temperature Temperature No No Fatigue ? Fatigue ? Fatigue ? Yes Yes Activate Data Fission Activate Data Fission component component 8/19/2005 10

Fusion Contextual Bayesian information Network for detecting Prior probability Fatigue State video-based information Qiang Ji, Zhiwei Zhu and Peilin Lan, Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue, IEEE Transactions on Vehicular Technology, Vol. 53, No. 4, July, 2004, p1052-1068]. 8/19/2005 11

Fission Data fission responsibility is to collect the data from data fusion and to generate a XML message that is sent to the driver simulator Fatigue range [0,33] [33,66] [66,100] Message « » « Tired » « A sleep » Message color « » « Green « Red » » Shaking power « 0 » « 0 » « 100 » 8/19/2005 12

Architecture 8/19/2005 13

OpenInterface Integration Each component is Mouse Mouse Speech Speech registered into O P E N I N T E R F A C E O P E N I N T E R F A C E O P E N I N T E R F A C E O P E N I N T E R F A C E Component Component Component Component F F OpenInterface CIDL CIDL C++ C++ Java Java Platform using the XML XML Fusion for Fusion for i i Face Face Component Interface fatigue fatigue Description Language s s detection detection C++ C++ Start Start (CIDL) and described s s CIDL CIDL CIDL CIDL in XML OpenInterface OpenInterface C++ C++ Bayes C++ C++ Bayes XML XML XML XML Kernel Kernel MatLab MatLab i i Image Viewer Image Viewer Components properties Components properties Network Network Component Component C++ C++ o o Java Java Graphic Editor (Java) Graphic Editor (Java) The registered MatLab MatLab Execution pipeline Execution pipeline components n n Camera Camera properties are Driver Driver MatLab MatLab retrieved by Components properties Components properties the Graphic C++ C++ Editor (Java). 8/19/2005 14

Future Works • Integrate biological signals for fatigue detection • Usability tests to assess interface interactions. • Improve the Bayesian Network to take account more specialized information about head orientation once this information is available in the head detection code. • Transform the face detection component into 3 OpenInterface components. 8/19/2005 15

Conclusion � 1 st goal: real time distributed system based on video data, integrated under OpenInterface, for driver attention level analysis with feed-back to the user � 2 nd goal: multimodal system taking into account the biological signals - Stress 8/19/2005 16

TEAM 8/19/2005 17

TEAM 8/19/2005 18

DEMO • Have Fun and come to play during our Demo session! 8/19/2005 19

Questions? 8/19/2005 20