真实世界人脸表情识别 Weihong Deng （邓伟洪） Beijing University of Posts and Telecommunications http://whdeng.cn/Emotion/projects.html
Outlines Intr In trod oduc uction tion & Bac ackgrou kground nd 01 Facial Expression Databases 02 Our Works 03 Latest Survey 04
Evolution Creates Facial Expressions Share similar facial muscles • Charles Darwin theorized that emotional expression was evoluted by natural selection • Important for survival: Fear expression can directly let our eyes absorb more light and our lungs take more air. • Improve group fitness: Surprise indicates something new happen; Sadness is a signal to the group that something is wrong.
Basic Emotions are Universal Common muscle group Basic Expressions Paul Ekman • Paul Ekman designed the acknowledged Facial Action Coding System (FACS) • Paul Ekman claimed that Basic emotional expressions are in fact universal across cultures acted by similar muscle group. • In 1960s, Paul Ekman identified six core expressions: happiness, fear, surprise, disgust, sadness, anger
Outliness Introduction & Background 01 Fac acia ial Expre pression ssion Dat atab abas ases es 02 Our Works 03 Latest Survey 04
Prototype Databases 2007 CK+ MMI • Previous widely-used facial expression Oulu-CASIA 2009 datasets are lab-controlled and small- TFD scale . JAFFE 2011 Multi-PIE … … 2013 MMI: CK+: 2900 videos, 75 subjects 596 videos, 123 subjects 2015 2017 Oulu-CASIA: JAFFE: 2880 videos, 80 subjects 213 images, 10 females
Posed Spontaneous Mirco Facial Expression Recognition 2013 Acted Facial Expression In The Wild (FER-2013) (AFEW) • 35887 images from the Internet Micro-Expression Datasets: • 1809 videos from movies and TV shows • 48x48 pixels in grayscale • SMIC • 7 basic facial expressions • 184 emotion-related keywords • CASME, CASME II, CAS(ME) 2 • Three annotators • 7 basic facial expressions • MEVIEW (Micro-Expressions VIdEos in the • More than 330 subjects, age 1~77 years Wild) Suppressed emotion, difficult to observe
Lab-controlled Moive In-the-Wild AffectNet EmotioNet • 1,000,000 images from Internet • 1,000,000 images from Internet • 1,250 emotion-related keywords • 457 concepts of emotion-related keywords • 8 basic emotion categories • 23 basic and compound emotion categories • Valence and Arousal • Action Units
Advanced Databases 2007 CK+ MMI Oulu-CASIA 2009 • Datasets collected from real world are TFD more diverse and naturalistic, most of JAFFE which contain large-scale samples. 2011 Multi-PIE … … • Facial expression datasets in-the-wild FER2013: https://github.com/npinto/fer2013 2013 FER2013 SFEW, AFEW: https://cs.anu.edu.au/few/ EmotioNet: http://cbcsl.ece.ohio-state.edu/dbform_emotionet.html EmotiW AffectNet: http://mohammadmahoor.com/affectnet/ RAF-DB, RAF-ML: http://whdeng.cn/Emotion/projects.html 2015 EmotioNet Aff-Wild: https://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge/ ExpW: http://mmlab.ie.cuhk.edu.hk/projects/socialrelation/index.html RAF-DB AffectNet 2017 Li, S., & Deng, W. Deep facial expression recognition: A survey. CoRR … … abs/1804.08348 (2018).
Contents In-the-wild Expression Labeling Movies Dataset Bias Lab-controlled Latest Survey Posed Spontaneous Micro-expression Discussions Datasets: from Basic to Complex
Outlines Introduction & Background 01 Facial Expression Databases 02 Our ur Wor orks ks 03 Latest Survey 04
Two annotation Challenges 1 Crowd-sourcing 315 volunteers online 0.8 Probability Each image labelled 40 times 0.6 0.29 0.4 0.24 1,200,000 0.2 Learning from 0.12 0.12 0.2 Annotation labels 0.02 0 labels 0 Basic & Compound & Blended
Real-world Affective Face Database (RAF-DB) Keywords ‘smile’ ‘crying’ download parse ~ 30,000 1. XML Collection Collection ’s API ‘OMG’… URLs response images 60,000 images downloaded Image Collection Flickr (Image social network) https://api.flickr.com/services/rest/?method=flickr.photos.search&api_key={}&text={}&tags={} &per_page={}&page={}&sort=relevance XML response → Interpreted into URLs of the images → Download S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality preserving learning for expression recognition in the wild,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2017, pp. 2584 – 2593.
Real-world Affective Face Database (RAF-DB) Learning from Crowd-sourcing 1,200,000 2. labels 315 volunteers online Annotation labels Each image labelled 40 times Annotation Image Annotation Crowd-sourcing 315 well-trained annotators were asked to label facial images with one of the seven basic categories Each image is annotated enough times independently, i.e., around 40 times in our experiment.
Real-world Affective Face Database (RAF-DB) Filter out Optimal EM unreliable labels Reliability 3. Reliability Estimation framework Reliability Estimation Filter noisy annotators and labels an Expectation Maximization (EM) framework was used to iteratively optimize and assess the target parameters of each labeler’s reliability.
Real-world Affective Face Database (RAF-DB) • Database Statistics • 29672 number of real-world images , • a 7-dimensional expression distribution vector for each image, • two different subsets: single-label subset , including 7 classes of basic emotions; two-tab subset, including 12 classes of compound emotions, • 5 accurate landmark locations , 37 automatic landmark locations , race , age range and gender attributes annotations per image.
Compound Emotions Background ? While past research had identifed facial expressions associated with a single internally felt category (e.g, the facial expression of happiness when we feel joyful), we have recently studied facial expressions observed when people experience compound emotions (e.g, the ? facial expression of happy surprise when we feel joyful in a surprised way, as, for example, at a surprise birthday party) ? S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions 1. “ Nonverbal communication ”, M. Anderson, 1987. of emotion ,” Proceedings of the National Academy of Sciences, 2. “ Facial expression and emotion”, P. Ekman, 1993. vol. 111, no. 15, pp. E1454 – E1462, 2014. 3. “Compound facial expressions of emotion”, Martinez et al. PNAS 2014.
Real-world Affective Face Database (RAF-DB)
Action Units: RAF-DB is more diverse RAF-DB CK+ AU1,2 AU1,2 AU1,2 AU1,2 AU1,2 AU5 Surprise AU5 AU5 AU25, AU25 AU25, AU26 AU27 AU6 AU6 AU6 AU6 AU6 Joy AU12 AU12 AU12 AU12 AU12 AU25 AU25 AU26 AU26 AU1 AU4 AU1,4 AU1,4 AU1,4 AU5 Fear AU5 AU7 AU7 AU26 AU26 AU20, AU20 AU26 AU27 AU27 AU25 , AU25 AU5 AU4 AU7 AU4 AU4 AU7 Anger AU9 AU7 AU7 AU10 AU10 AU24 AU26 AU24 AU25 AU25 AU27 AU26 AU17 AU17 AU5 AU4 AU4 AU4 AU7 AU10 AU5 Disgust AU7 AU7 AU9 AU9 AU10 AU20, AU17 AU25 AU17 AU1,4 AU1,4 AU1,4 Sadness AU7 AU15 AU15 AU15 AU10 AU25 AU17 AU25 AU17 AU17 S. Li and W. Deng, “Reliable crowdsourcing and deep locality preserving learning for unconstrained facial expression recognition ,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 356 – 370, Jan 2019.
DLP-CNN: Deep Locality-preserving CNN C: The convolution layer P: The max-pooling layer R: The ReLU layer F: The fully connected layer Softmax Loss R F C R C R F C R P P C R Input R P C R C λ Locality- preserving Face Images Our goal: Loss 2 min 𝑇 𝑗𝑘 ||𝑦 𝑗 − 𝑦 𝑘 || 2 𝑗,𝑘 Discriminative Features Separable Features Locality Preserving Loss: 1, 𝑦 𝑘 is among 𝑙 nearest neighbors of 𝑦 𝑗 𝐩𝐬 𝑜 𝑇 𝑗𝑘 = 𝑦 𝑗 is among k nearest neighbors of 𝑦 𝑘 𝑀 𝑚𝑞 = 1 ||𝑦 𝑗 − 1 2 2𝑜 𝑦 || 2 0, 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 𝑙 i=1 𝑦∈𝑂 𝑙 𝑦 𝑗
DLP-CNN: Deep Locality-preserving CNN S. Li and W. Deng, “Reliable crowdsourcing and deep locality preserving learning for unconstrained facial expression recognition ,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 356 – 370, Jan 2019.
DLP-CNN: Experiment Results Table 1. Expression recognition performance of different DCNNs on RAF. The metric is the mean diagonal value of the confusion matrix. [6] [7] [8] [6] [7] [8] 6. Simonyan & Zisserman, arXiv:1409.1556 (2014). 7. Krizhevsky et al. NIPS, 1097 – 1105 (2012). 8. Wen et al. ECCV, 499 – 515 (2016).
Recommend
More recommend
