Real-time 3D Eyelids Tracking From Semantic Edges Quan Wen, Feng - PowerPoint PPT Presentation

Real-time 3D Eyelids Tracking From Semantic Edges Quan Wen, Feng Xu, Ming Lu, Jun-Hai Yong Tsinghua University

Background Facial capture and animation is crucial in many applications Face capture in computer games Face animation in movies -1-

Background Facial tracking focus less on the eyes [Bouaziz et al. 2013] [Cao et al. 2014] [Li et al. 2013] [Cao et al. 2015] [Hsieh et al. 2015] [Liu et al. 2015] -2-

Background Facial organs tracking [Bérard et al. 2016] [Edwards et al. 2016] [Wang et al. 2016] [Wood et al. 2016] [Bermano et al. 2015] [Wu et al. 2016] [Wen et al. 2016] -3-

Our Work A real-time 3D eyelids tracking system -4-

Overview ? Input Eyelid image Reconstruction -5-

Overview Input Eyelid Eyelid image features Reconstruction -6-

Overview Eyelid models Input Eyelid Eyelid image features Reconstruction -7-

Overview Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Face & Eyeball Fitting Final Result [Wen et al. 2016] Input Depth Face & Eyeball Result -8-

Overview Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Face & Eyeball Fitting Final Result [Wen et al. 2016] Input Depth Face & Eyeball Result -9-

Overview Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Eyelid Result Face & Eyeball Fitting [Wen et al. 2016] Input Depth Face & Eyeball Result -10-

Overview Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Face & Eyeball Fitting Final Result [Wen et al. 2016] Input Depth Face & Eyeball Result -11-

Overview Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Face & Eyeball Fitting Final Result [Wen et al. 2016] Input Depth Face & Eyeball Result -12-

Eyelid Edge Detection and Identification Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Face & Eyeball Fitting Final Result [Wen et al. 2016] Input Depth Face & Eyeball Result -13-

Semantic Eyelid Edges Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge -14-

Semantic Eyelid Edges Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge -15-

Semantic Eyelid Edges Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge -16-

Semantic Eyelid Edges Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge -17-

Semantic Eyelid Edges Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge -18-

Semantic Eyelid Edges Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge -19-

Network 1-channel HED Sigmoid Cross-entropy [Xie and Tu 2015] DNN in Loss HED -20-

Network 1-channel HED Sigmoid Cross-entropy [Xie and Tu 2015] DNN in Loss HED ··· ··· Training Set -21-

Network 1-channel HED Sigmoid Cross-entropy [Xie and Tu 2015] DNN in Loss HED ··· ··· Training Set Network output -22-

Network 1-channel HED Sigmoid Cross-entropy [Xie and Tu 2015] DNN in Loss HED 4-channel Proposed eyelid edge Sigmoid Cross-entropy detection and identification Proposed Loss DNN -23-

Network 1-channel HED Sigmoid Cross-entropy [Xie and Tu 2015] DNN in Loss HED No double-fold ··· ··· 4-channel Sigmoid No bulge Cross-entropy Proposed Loss Training Set DNN -24-

Network 1-channel Sigmoid Cross-entropy DNN in Loss HED 4-channel Sigmoid Network output Cross-entropy Proposed Loss DNN -25-

Eyelid Edge Detection and Identification Results -26-

Eyelid Linear Models Edge Maps for Training Two Eyelid Linear Models Eyelid Edge Detection & Identification Curve-based Input Color Edge Result Eyelid Reconstruction Face & Eyeball Fitting Final Result [Wen et al. 2016] Input Depth Face & Eyeball Result -27-

Shape Linear Rig Eyelid shape categories Position Contour shape Bulge Double-fold -28-

Shape Linear Rig Linear rig 𝐶 𝑗𝑒 𝑗𝑒 models in 𝐶 𝑗𝑒 𝑐 𝑙 𝐶 𝑗𝑒 = 𝑐 𝑙 𝑗𝑒 |𝑙 = 0, … , 𝑂 𝑗𝑒 − 1 , 𝑂 𝑗𝑒 = 29 𝑂 𝑗𝑒 number of 𝑐 𝑙 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑐 0 𝑐 11 𝑐 21 𝑐 23 (basic) (contour: downturned) (double-fold: single) (bulge: parallel) -29-

Shape Linear Rig Synthesized shape model of a specific user 𝑂 𝑗𝑒 −1 𝑗𝑒 + 𝑗𝑒 − 𝑐 0 𝑗𝑒 (𝑐 𝑙 𝑗𝑒 ) 𝐹 𝑂 = 𝑐 0 𝑥 𝑙 𝑙=1 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑐 0 𝑐 11 𝑐 21 𝑐 23 (basic) (contour: downturned) (double-fold: single) (bulge: parallel) -30-

Shape Linear Rig Synthesized shape model of a specific user 𝑗𝑒 basic model in 𝐶 𝑗𝑒 𝑂 𝑗𝑒 −1 𝑐 0 𝑗𝑒 + 𝑗𝑒 − 𝑐 0 𝑗𝑒 (𝑐 𝑙 𝑗𝑒 ) 𝐹 𝑂 = 𝑐 0 𝑥 𝑙 𝑙=1 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑐 0 𝑐 11 𝑐 21 𝑐 23 (basic) (contour: downturned) (double-fold: single) (bulge: parallel) -31-

Shape Linear Rig Synthesized shape model of a specific user 𝑗𝑒 basic model in 𝐶 𝑗𝑒 𝑂 𝑗𝑒 −1 𝑐 0 𝑗𝑒 + 𝑗𝑒 − 𝑐 0 𝑗𝑒 shape models in 𝐶 𝑗𝑒 𝑗𝑒 (𝑐 𝑙 𝑗𝑒 ) 𝐹 𝑂 = 𝑐 0 𝑥 𝑙 𝑐 𝑙 𝑙=1 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑐 0 𝑐 11 𝑐 21 𝑐 23 (basic) (contour: downturned) (double-fold: single) (bulge: parallel) -32-

Shape Linear Rig Synthesized shape model of a specific user 𝑗𝑒 basic model in 𝐶 𝑗𝑒 𝑂 𝑗𝑒 −1 𝑐 0 𝑗𝑒 + 𝑗𝑒 − 𝑐 0 𝑗𝑒 shape models in 𝐶 𝑗𝑒 𝑗𝑒 (𝑐 𝑙 𝑗𝑒 ) 𝐹 𝑂 = 𝑐 0 𝑥 𝑙 𝑐 𝑙 𝑗𝑒 weight of 𝑐 𝑙 𝑗𝑒 𝑥 𝑙 𝑙=1 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑐 0 𝑐 11 𝑐 21 𝑐 23 (basic) (contour: downturned) (double-fold: single) (bulge: parallel) -33-

Shape Linear Rig Synthesized shape model of a specific user 𝑂 𝑗𝑒 −1 𝑗𝑒 ( = + 𝑥 𝑙 − ) 𝑙=1 𝑗𝑒 𝑗𝑒 𝑗𝑒 𝑐 0 𝑐 k 𝑐 0 𝐹 𝑂 (basic) (shape) (basic) User-specific shape model -34-

Pose Linear Rig Generic linear rig 𝐶 𝑓𝑦𝑞 𝑓𝑦𝑞 models in 𝐶 𝑓𝑦𝑞 𝑐 𝑙 𝐶 𝑓𝑦𝑞 = 𝑐 𝑙 𝑓𝑦𝑞 |𝑙 = 0, … , 𝑂 𝑓𝑦𝑞 − 1 , 𝑂 𝑓𝑦𝑞 = 23 𝑂 𝑓𝑦𝑞 number of 𝑐 𝑙 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑐 0 𝑐 1 𝑐 3 𝑐 5 (basic) (close) (inner close) (outer close) -35-

Pose Linear Rig Generic pose rig 𝐶 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 Basic model 𝑐 0 Pose models 𝑐 𝑙 -36-

Pose Linear Rig Generic pose rig 𝐶 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 Basic model 𝑐 0 Pose models 𝑐 𝑙 User-specific basic model 𝐹 𝑂 -37-

Pose Linear Rig Generic pose rig 𝐶 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 Basic model 𝑐 0 Pose models 𝑐 𝑙 ? User-specific pose rig 𝐶 𝑓𝑦𝑞′ User-specific 𝑓𝑦𝑞′ User-specific pose models 𝑐 𝑙 basic model 𝐹 𝑂 -38-

Pose Linear Rig Generic pose rig 𝐶 𝑓𝑦𝑞 𝑓𝑦𝑞 𝑓𝑦𝑞 Basic model 𝑐 0 Pose models 𝑐 𝑙 Deformation transfer User-specific pose rig 𝐶 𝑓𝑦𝑞′ User-specific 𝑓𝑦𝑞′ User-specific pose models 𝑐 𝑙 basic model 𝐹 𝑂 -39-

Pose Linear Rig User-specific eyelid model in tracking 𝑂 𝑓𝑦𝑞 −1 𝑓𝑦𝑞′ + 𝑓𝑦𝑞′ − 𝑐 0 𝑓𝑦𝑞 (𝑐 𝑙 𝑓𝑦𝑞′ ) 𝐹 𝑄 = 𝑐 0 𝑥 𝑙 𝑙=1 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑐 0 𝑐 1 𝑐 3 𝑐 5 (basic) (close) (inner close) (outer close) -40-

Pose Linear Rig User-specific eyelid model in tracking 𝑂 𝑓𝑦𝑞 −1 𝑓𝑦𝑞′ basic model in 𝐶 𝑓𝑦𝑞′ 𝑐 0 𝑓𝑦𝑞′ + 𝑓𝑦𝑞′ − 𝑐 0 𝑓𝑦𝑞 (𝑐 𝑙 𝑓𝑦𝑞′ ) 𝐹 𝑄 = 𝑐 0 𝑥 𝑙 𝑙=1 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑐 0 𝑐 1 𝑐 3 𝑐 5 (basic) (close) (inner close) (outer close) -41-

Pose Linear Rig User-specific eyelid model in tracking 𝑂 𝑓𝑦𝑞 −1 𝑓𝑦𝑞′ basic model in 𝐶 𝑓𝑦𝑞′ 𝑐 0 𝑓𝑦𝑞′ + 𝑓𝑦𝑞′ − 𝑐 0 𝑓𝑦𝑞 (𝑐 𝑙 𝑓𝑦𝑞′ ) 𝑓𝑦𝑞′ pose models in 𝐶 𝑓𝑦𝑞′ 𝐹 𝑄 = 𝑐 0 𝑥 𝑙 𝑐 𝑙 𝑙=1 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑓𝑦𝑞′ 𝑐 0 𝑐 1 𝑐 3 𝑐 5 (basic) (close) (inner close) (outer close) -42-