Multi-modal Face Recognition Hu Han hanhu@ict.ac.cn http: / / vipl.ict.ac.cn/ members/ hhan 2016/ 04/ 06
2 Trend on multi-modal (face) recognition Multi-modal & cross-modal FR Conclusion and discussion hanhu@ict.ac.cn Related work Background Outline 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Background Unconstrained sensing & uncooperative subject scenario poses great challenges to unimodal FR system Institute of Computing Technology, Chinese Academy of Sciences “We are particularly interested in reviewing video “It’s our intention to go through footage captured by bystanders with cell phones every frame of every video.” – Boston Police Commissioner, or personal cameras near either of the blasts… In an investigation of this nature, no detail is too Ed Davis small.” – Attorney General, Eric H. Holder Jr. 2016/4/6 hanhu@ict.ac.cn 3
Background Unimodal FR A manually selected probe face image of the suspect (Tamerlan Tsarnaev) with the best quality is matched with its true mate by a COTS with rank-5000 among a 1M gallery set Institute of Computing Technology, Chinese Academy of Sciences Probe 1 M 2016/4/6 hanhu@ict.ac.cn 4
Background Challenges Low quality surveillance videos and images Intentional thwarting of identification (e.g. sunglasses and hats) Daunting amount of data Institute of Computing Technology, Chinese Academy of Sciences Videos or images are n/ a … Multi-modal FR is a possible solution Advances in computing and imaging tech. RGB, depth, NIR, 3D, sketch, etc. Multi-modality, multi-view, multi-biometrics 2016/4/6 hanhu@ict.ac.cn 5
Background Institute of Computing Technology, Chinese Academy of Sciences Human operators Top K Matche manually review K*n s images ( n = # of images in the face media collection) Top K Matche … … s Traditional Forensic Investigation Workflow Top K 2016/4/6 hanhu@ict.ac.cn 6 Matche s
7 Trend on multi-modal (face) recognition Multi-modal & cross-modal FR Conclusion and discussion hanhu@ict.ac.cn Related work Background Outline 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Related work Multi-modal FR 2D + 3D Beumier and Acheroy, PRL’01 Chang et al., ACM-W’03 … 2D + depth Institute of Computing Technology, Chinese Academy of Sciences Lu and Jain, TPAMI’06 2D + 3D + NIR Bowyer et al., 2003-2011 Most are: Per-modal matching + score-level fusion 2016/4/6 hanhu@ict.ac.cn 8
Related work Cross-modal FR Modality transformation Wang & Tang, TPAMI’09 (sketch vs. photo) Gao et al., TCSVT’12 (sketch vs. photo) 3D face modeling, Blanz & Vetter’03 (2D vs. 3D) … Institute of Computing Technology, Chinese Academy of Sciences Invariant features Lei & Li, CVPR’09 VIS-NIR Klare & Jain, TPAMI’13; Han & Jain TIFS’13; Klum et al., TIFS’14 VIS-NIR, forensic sketch, VIS-TIR 2016/4/6 hanhu@ict.ac.cn 9
Outline Background Related work Multi-modal & cross-modal FR Multi-modal FR Trend on multi-modal (face) recognition Institute of Computing Technology, Chinese Academy of Sciences Conclusion and discussion 2016/4/6 hanhu@ict.ac.cn 10
11 . . . Multi-modal face recognition manually review 1* K ? Human operators images Sketch 3D hanhu@ict.ac.cn Still image Video 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Multi-modal face recognition A hierarchical quality-based fusion 30-40 岁 男性 白人 … Image/video Sketch 3D Institute of Computing Technology, Chinese Academy of Sciences Quality measures： Q1 Q2 Q3 Q4 1M Mugshot，True mate is matched at rank-112 (vs. rank-5000 in unimodal) 2016/4/6 hanhu@ict.ac.cn 12
Multi-modal face recognition Face Track All Frame Pairs Extraction U u 1 , u 2 ,..., u a ... … … ... V v 1 , v 2 ,..., v b Institute of Computing Technology, Chinese Academy of Sciences Matching a Face COTS Face Matcher Track from a Video ... Multi ‐ t √ frame Same s u 1 , v 1 ... Score ‐ level … Fusion: Similarity s U , V • mean Matrix • median Not • max Same t ... s u a , v b • min 2016/4/6 hanhu@ict.ac.cn 13 a b
14 Pose Correction via 3D Face Modeling Multi-modal face recognition hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
15 Multi- modal 4,249 gallery images, 596 probe subjects Multi-modal face recognition Videos Close set identification hanhu@ict.ac.cn Images 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
16 Multi-modal face recognition 4,249 gallery images + 1M background mugshots, 596 probe subjects Close set identification hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Multi-modal face recognition Open set identification The person of interest may not be present in legacy face databases The gallery consists of 596 subjects with at least two images in the LFW database and at least one video in the YTF database Institute of Computing Technology, Chinese Academy of Sciences 2016/4/6 hanhu@ict.ac.cn 17
18 Multi-modal face recognition hanhu@ict.ac.cn Quality based fusion 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
19 A Case Study on the Boston Bomber Multi-modal face recognition (Gallery of one million mugshots) hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Multi-modal face recognition Forensic Sketches from Low Quality Video Institute of Computing Technology, Chinese Academy of Sciences Retrieval ranks without and with demographic filtering 2016/4/6 hanhu@ict.ac.cn 20 are given as #(#)
21 CNNs D CNNs RGB Deep multi-modal FR CNNs D hanhu@ict.ac.cn CNNs CNNs RGB CNNs 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
22 900,000 RGBD images of 700 subjects Deep RGBD face recognition Deep multi-modal FR Accuracy 0.93 0.86 0.98 hanhu@ict.ac.cn Deep RGB-D Modality Depth RGB 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Outline Background Related work Multi-modal & cross-modal FR Cross-modal FR Trend on multi-modal (face) recognition Institute of Computing Technology, Chinese Academy of Sciences Conclusion and discussion 2016/4/6 hanhu@ict.ac.cn 23
Cross-modal face recognition Compatible with huge existing 2D face images RGBD vs. RGB Modality is not available NIR vs. VIS Institute of Computing Technology, Chinese Academy of Sciences Sketch vs. photo 2016/4/6 hanhu@ict.ac.cn 24
Cross-modal face recognition Sketch to mugshot matching Viewed sketch: drawing/ synthesizing a sketch while looking at a subject/ photo Forensic sketch: drawing/ synthesizing a sketch based on verbal description from the Institute of Computing Technology, Chinese Academy of Sciences victim or eyewitness COTS matcher for photo-to-photo matching can achieve over than 85% rank-1 identification rate for viewed sketch in 2013, while its performance for forensic sketch identification is less than 10% 2016/4/6 hanhu@ict.ac.cn 25
26 Cross-modal face recognition Sketch to mugshot matching hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Sketch to mugshot matching Sketch leads to arrest of suspects Institute of Computing Technology, Chinese Academy of Sciences Timothy McVeigh (the Ted Kaczynski David Berkowitz Oklahoma City bomber) (the Unabomber) (Son of Sam) 2016/4/6 hanhu@ict.ac.cn 27
Sketch to mugshot matching Local and holistic matching Local matching: component based rep. Institute of Computing Technology, Chinese Academy of Sciences Hu Han, Brendan Klare, Kathryn Bonnen, and Anil K. Jain. Matching Composite Sketches to Face Photos: A Component Based Approach. IEEE Transactions on Information Forensics and Security (T-IFS), vol. 8, no. 1, pp. 191-204, Jan. 2013. 2016/4/6 hanhu@ict.ac.cn 28
29 Sketch to mugshot matching Component based rep. is an inverse process of sketch composition hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
30 Holistic matching: dense keypoint features Sketch to mugshot matching Local and holistic matching hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
31 Sketch to mugshot matching hanhu@ict.ac.cn Complementarity 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
32 software-generated Sketch to mugshot matching hanhu@ict.ac.cn and forensic sketch Hand-drawn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
33 Sketch to mugshot matching Software-generated viewed sketch hanhu@ict.ac.cn 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
34 cross-spectral Generalized cross-modal FR Enrolled VIS matching in nighttime FR and hanhu@ict.ac.cn Cross-distance 150m NIR at night 2016/4/6 Institute of Computing Technology, Chinese Academy of Sciences
Recommend
More recommend
