face2f ace2face ace
play

Face2F ace2Face: ace: Real-time Face Capture and Reenactment of - PowerPoint PPT Presentation

Jan 07, 2023 •422 likes •807 views

IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2F ace2Face: ace: Real-time Face Capture and Reenactment of RGB-Videos Justus Thies 1 , Michael Zollhfer 2 , Marc Stamminger 1 , Christian Theobalt 2 , Matthias Niener 3

  1. IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2F ace2Face: ace: Real-time Face Capture and Reenactment of RGB-Videos Justus Thies 1 , Michael Zollhöfer 2 , Marc Stamminger 1 , Christian Theobalt 2 , Matthias Nießner 3 1 University of Erlangen-Nuremberg 2 Max-Planck-Institute for Informatics 3 Stanford University

  2. IEEE 2016 Conference on Computer Vision and Pattern Recognition Related Work • Offline • Online Creating a Photoreal Digital Actor: Real-time Expression Transfer for Facial Reenactment The Digital Emily Project RGB-D Special Hardware Vdub: Modifying Face Video of Actors for Face2Face: Real-time Face Capture and Reenactment Plausible Visual Alignment to a Dubbed Audio Track Of RGB-Videos RGB

  3. IEEE 2016 Conference on Computer Vision and Pattern Recognition Related Work • Offline • Online Creating a Photoreal Digital Actor: Real-time Expression Transfer for Facial Reenactment The Digital Emily Project RGB-D Special Hardware Vdub: Modifying Face Video of Actors for Face2Face: Real-time Face Capture and Reenactment Plausible Visual Alignment to a Dubbed Audio Track Of RGB-Videos RGB

  4. IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model Face Model Face Capture Reenactment Results

  5. IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model • Face Capture • Energy Formulation • Non-rigid Model-based Bundling Face Model Face Capture Reenactment Results

  6. IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model • Face Capture • Energy Formulation • Non-rigid Model-based Bundling • Reenactment • Mouth Retrieval • Comparisons Face Model Face Capture Reenactment Results

  7. IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model • Face Capture • Energy Formulation • Non-rigid Model-based Bundling • Reenactment • Mouth Retrieval • Comparisons • Results / Live Demo Face Model Face Capture Reenactment Results

  8. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model

  9. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model 𝑸 Face Model Face Capture Reenactment Results

  10. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 Face Model Face Capture Reenactment Results

  11. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 𝑸 = 6 +80 Face Model Face Capture Reenactment Results

  12. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 +80 𝑸 = 6 +80+80 Face Model Face Capture Reenactment Results

  13. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 +80+80 𝑸 = 6 +80+80+76 Face Model Face Capture Reenactment Results

  14. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 +80+80+76 𝑸 = 6 +80+80+76+27= 269 Face Model Face Capture Reenactment Results

  15. IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model 𝑄 Face Model Face Capture Reenactment Results

  16. IEEE 2016 Conference on Computer Vision and Pattern Recognition Face Capture

  17. IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = Face Model Face Capture Reenactment Results

  18. IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = 𝐹 𝑑𝑝𝑚 𝑄 Distance in RGB Color Space Color Consistency 𝒎 𝟑,𝟐 − 𝒐𝒑𝒔𝒏 Face Model Face Capture Reenactment Results

  19. IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = 𝐹 𝑑𝑝𝑚 𝑄 +𝐹 𝑛𝑠𝑙 𝑄 Distance in Image Space Color Feature Consistency Similarity Face Model Face Capture Reenactment Results

  20. IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = 𝐹 𝑑𝑝𝑚 𝑄 +𝐹 𝑛𝑠𝑙 𝑄 +𝐹 𝑠𝑓𝑕 (𝑄) Color Feature Regularization Consistency Similarity −𝟒 𝝉 +𝟒 𝝉 𝟘𝟘, 𝟖% Face Model Face Capture Reenactment Results

  21. IEEE 2016 Conference on Non-rigid Model-based Computer Vision and Pattern Recognition Bundling 𝑜 𝐹 𝑢𝑝𝑢𝑏𝑚 𝑸 = 𝐹 𝑗 𝑸 → 𝑛𝑗𝑜 𝑗=0 Face Model Face Capture Reenactment Results

  22. IEEE 2016 Conference on Non-rigid Model-based Computer Vision and Pattern Recognition Bundling • Iterative Reweighted Least Squares (IRLS) 𝑲 𝑼 𝑲𝚬𝑸 = −𝑲 𝑼 𝑮  Gauss-Newton: 𝑲(𝑸) = Face Model Face Capture Reenactment Results

  23. IEEE 2016 Conference on Non-rigid Model-based Computer Vision and Pattern Recognition Bundling Hierarchy Levels Input Model Face Model Face Capture Reenactment Results

  24. IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Face Model Face Capture Reenactment Results

  25. IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Comparison

  26. IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Comparison Face Model Face Capture Reenactment Results

  27. IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Comparison Face Model Face Capture Reenactment Results

  28. IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment

  29. IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment Online RGB-Tracking Reenactment Pose Source Actor Per Frame Expression Transfer Illumination Mouth Retrieval Expression Identity Compositing Preprocessed Video Tracking Target Actor Pose Per Frame Illumination Expression Identity Face Model Face Capture Reenactment Results

  30. IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment Online RGB-Tracking Reenactment Pose Source Actor Per Frame Expression Transfer Illumination Mouth Retrieval Expression Identity Compositing Preprocessed Video Tracking Target Actor Pose Per Frame Illumination Expression Identity Face Model Face Capture Reenactment Results

  31. IEEE 2016 Conference on Computer Vision and Pattern Recognition Mouth-Retrieval Face Model Face Capture Reenactment Results

  32. IEEE 2016 Conference on Computer Vision and Pattern Recognition Mouth-Retrieval Face Model Face Capture Reenactment Results

  33. IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment Comparison Face Model Face Capture Reenactment Results

  34. IEEE 2016 Conference on Computer Vision and Pattern Recognition Live-Demo

  35. IEEE 2016 Conference on Computer Vision and Pattern Recognition Limitations / Future Work • Assumption of Lambertian surface and smooth illumination • No occlusion handling • No person specific details (fine scale details / wrinkles) • Reenactment relies on a training sequence (Mouth retrieval) Face Model Face Capture Reenactment Results

  36. IEEE 2016 Conference on Computer Vision and Pattern Recognition Conclusion • First Real-time Facial Reenactment only based on RGB-videos • Non-Rigid Model-Based Bundling • Sub-Space Deformation Transfer • Image-Based Mouth Synthesis Face Model Face Capture Reenactment Results

  37. IEEE 2016 Conference on Computer Vision and Pattern Recognition Thank You!

  38. IEEE 2016 Conference on Computer Vision and Pattern Recognition References • O. Alexander, M. Rogers, W. Lambeth, M. Chiang, and P. Debevec. The Digital Emily Project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses , pages 12:1 – 12:15. ACM, 2009. • P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt. Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. In Computer Graphics Forum . Wiley-Blackwell, 2015. • F. Shi, H.-T. Wu, X. Tong, and J. Chai. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM TOG , 33(6):222, 2014. • C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG , 20(3):413 – 425, 2014. • J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt. Real-time expression transfer for facial reenactment. ACM Transactions on Graphics (TOG) ,34(6), 2015. • V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In Proc. SIGGRAPH , pages 187 – 194. ACM Press/Addison-Wesley Publishing Co., 1999. Face Model Face Capture Reenactment Results

More recommend