Improving RGB-D face recognition via transferring pretrained 2D - PowerPoint PPT Presentation

Improving RGB-D face recognition via transferring pretrained 2D networks Xingwang Xiong, Xu Wen, and Cheng Huang INSTITUTE O https://github.com/XingwXiong/Face3D-Pytorch OF C COMPUTING T TECHNOLOGY 3D Face Recognition Algorithm Challenge (3DFRAC) ICT, Chinese Academy of Sciences

Face Representations 3D Face Images RGB image Point cloud 3D Mesh Depth image 3DFRAC Bench 19

3D Face Recognition Algorithm Challenge n RGB-D Face Recognition n The value in the depth image reflects the distance of scene object surface from the viewpoint . RGB-D image Depth camera (Kinect V2) RGB image Depth image 3DFRAC Bench 19

Why do we need RGB-D FR ? n 2D FR is sensitive to external variations n Poses n Facial expressions n Illuminations n Extra low-level patterns on depth images n Smooth variations n Contracts n Borders & global layouts n Face Anti-spoofing (ICPR 2018) 3DFRAC Bench 19

Open-set vs. Closed-set FR n Open-set FR n Classification n Closed-set FR n Face embedding n Similarity comparison • SVM • KNN n 3DFRAC n a closed-set problem 3DFRAC Bench 19 CVPR 2017

RGB Images vs. Depth Images RGB images Depth images n High frequency patterns n Low frequency patterns n Textures & Details n Smooth variations n Contracts n Borders & Global layouts n Easy to obtain n Massive scale n Not enough to learn a deep n ∼ 3.3 million faces 1 CNN n ∼ 9K identities 1 n ∼ 403K million faces 2 n ∼ 1.2K identities 2 1. VGGFace2 2. Intellifusion RGB-D face dataset 3DFRAC Bench 19

Goal n To leverage both conventional RGB-based works and depth features 3DFRAC Bench 19

Inter-modal Transfer Learning 2D pretrained network RGB-D network Copy weights 3DFRAC Bench 19

Inter-modal Transfer Learning n Use ResNet-50 as the backbone network n Copy pretrained weights of middle layers n Fine-tune the whole model with 224x224 RGB-D images 3DFRAC Bench 19

Preprocessing n Face Detection & Alignment n MTCNN (SPL 2016) n Randomly horizontally flipping n Normalizing n 0-1 range Face Detection MTCNN Locate the face 3DFRAC Bench 19

Results n 94.64% accuracy on the Intellifusion RGB-D dataset n Won 1 st on 3DFRAC Method Input data CNN Models Accuracy (%) Pretrained on ImageNet RGB images ResNet-50 94.47 Train from scratch RGB-D images RGB-D ResNet50 88.36 Pretrained on ImageNet RGB-D images RGB-D ResNet50 94.64 3DFRAC Bench 19

Conclusions n Inter-modal transfer learning from pretrained 2D networks to RGB-D networks improves recognition accuracy n Code is open-sourced n https://github.com/XingwXiong/Face3D-Pytorch n Email n xiongxingwang@ict.ac.cn 3DFRAC Bench 19

3DFRAC Bench 19