SHAHAB iCV Research Group iCV Image Processing & Mathematical - PowerPoint PPT Presentation

SHAHAB iCV Research Group

iCV • Image Processing & Mathematical Modelling • Computer Vision & Robotics • Computer Graphics & 3D Modelling

iCV • Image Processing & Mathematical Modelling • Computer Vision & Robotics • Computer Graphics & 3D Modelling

Facial Multi-emotional Expression Recognition

Human-Computer Interaction

Human-Computer Interaction • Aging society

Human-Computer Interaction • Robots as colleagues

Human-Computer Interaction • Machine should understand us • Understanding through: – Audio – Vision • Emotion

Human-Computer Interaction • Automatic Emotion recognition – Face – Gesture – Audio • Facial Emotion

Human-Computer Interaction • Facial Emotion – Angry – Contempt – Fear – Disgust – Happy – Sadness – Surprise

Human-Computer Interaction • Are these 7 emotions realistic? – Yes – No • Are these 7 emotions enough? – Yes – No

Dominant and complementary expressions • Dominant emotions are recognised using all extracted feature points • Complementary emotions are recognised from other features depending on the dominant emotion

Complementary emotions • Belong to the same group of seven classical emotions • Are not as meaningful in the context of the whole face as the dominant emotion • The features used for complementary emotion classification depend on the dominant emotion

Complementary emotions cont. • Calder et al. (2000): – Anger, fear and sadness are more easily recognisable from the upper face of a person – Disgust and happiness are easier to recognise from the lower face – Surprise can be equally recognised from both areas

New Sets of Emotions

Proposed method • Face localisation and segmentation by Viola- Jones face detection algorithm • Feature extraction by Local Gabor Binary Patterns (LGBP) • Euclidean distances between corresponding points on neutral and full emotion • C-support vector classification

Flowchart

Dominant and complementary emotion combinations • As anger, fear and sadness influence mostly the upper half of the face, we extract their complementary emotion for those dominant emotions from the lower half of the face • In cases of contempt, disgust, happiness and surprise complementary emotions are extracted from the upper face

Extracted features A total of 68 intransient feature points extracted using Local Gabor Binary Patterns (FERA 2015 baseline system) 37 points on upper face, 31 on lower face.

Division of the face

Results for dominant emotion recognition CK+ recognition rate % JAFFE recognition rate % Angry 100 67 Contempt 89 - Disgust 98 55 Fear 96 84 Happy 100 87 Sad 89 45 Surprise 100 73 Cohan-Kanade: 327 JAFFE: 213

Results for complementary emotion recognition for CK+ Angry Contempt Disgust Fear Happy Sad Surprise Angry 34 3 1 2 1 2 2 Contempt 3 8 1 1 0 3 2 Disgust 7 2 49 1 0 0 0 Fear 3 1 1 14 2 1 3 Happy 3 2 4 5 51 2 2 Sad 4 1 2 3 0 15 3 Surprise 5 2 3 3 2 3 65

The first row from left to right: angry, fearfully angry and surprisingly angry. The second row from left to right: angrily fearful, fear, happily fearful and sadly fearful. The third row from left to right angrily sad, fearfully sad and sad.

The first row from left to right: angry, disgustingly angry, fearfully angry, sadly angry and surprisingly angry. The second row from left to right: fearfully angry, disgustingly feared, fear, sadly feared and surprisingly feared. The third row from left to right: angrily sad, happily sad, sad and surprisingly sad.

Conclusion • Further research • Inclusion of psychologists • Database development • Real world implementation