Model-Based Teeth Reconstruction Chenglei Wu 2 , Derek Bradley 1 , - PowerPoint PPT Presentation

Model-Based Teeth Reconstruction Chenglei Wu 2 , Derek Bradley 1 , Pablo Garrido 3 , Michael Zollhöfer 3 , Christian Theobalt 3 , Markus Gross 1,2 , Thabo Beeler 1 1 Disney Research 2 ETH Zurich 3 Max Planck Institute for Informatics

MOTIVATION Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 2 2

MOTIVATION Input Image Beeler11 Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 3 3

MOTIVATION Berard 16 Input Image Beeler11 Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 4 4

MOTIVATION Berard 16 Bermano 16 Input Image Beeler11 Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 5 5

MOTIVATION Berard 16 Bermano 16 Beeler12 Input Image Beeler11 Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 6 6

MOTIVATION Berard 16 Bermano 16 Beeler12 Nagano15 Input Image Beeler11 Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 7 7

MOTIVATION Berard 16 Bermano 16 Beeler12 Nagano15 Input Image Beeler11 Hu14 Michael Zollhöfer Michael Zollhöfer Model-Based Teeth Reconstruction Model-Based Teeth Reconstruction 8 8

MOTIVATION Teeth are missing! Input Image Beeler11 Michael Zollhöfer Model-Based Teeth Reconstruction 9

MOTIVATION Teeth are missing! Input Image Beeler11 Michael Zollhöfer Model-Based Teeth Reconstruction 10

MOTIVATION Teeth are missing! Complex Appearance Input Image Beeler11 Michael Zollhöfer Model-Based Teeth Reconstruction 11

MOTIVATION Teeth are missing! Complex Appearance Featureless Input Image Beeler11 Michael Zollhöfer Model-Based Teeth Reconstruction 12

MOTIVATION Teeth are missing! Complex Appearance Featureless Occlusions Input Image Beeler11 Michael Zollhöfer Model-Based Teeth Reconstruction 13

MOTIVATION Teeth are missing! Complex Appearance Featureless Occlusions Input Image Our Reconstruction Michael Zollhöfer Model-Based Teeth Reconstruction 14

RELATED WORK Michael Zollhöfer Model-Based Teeth Reconstruction 15

RELATED WORK Medical Dentistry From CT images [ Omachi07 , Yanagisawa14] Michael Zollhöfer Model-Based Teeth Reconstruction 16

RELATED WORK Medical Dentistry From CT images [ Omachi07 , Yanagisawa14] Intra-Oral Scanners [3M True Definition, iTero , 3Shape TRIOS] Michael Zollhöfer Model-Based Teeth Reconstruction 17

RELATED WORK Medical Dentistry Photogrammetric Reconstruction From CT images Shape from Shading [ Omachi07 , Yanagisawa14] [Carter10, Farag13 , Mostafa14] Intra-Oral Scanners [3M True Definition, iTero , 3Shape TRIOS] Michael Zollhöfer Model-Based Teeth Reconstruction 18

RELATED WORK Medical Dentistry Photogrammetric Reconstruction From CT images Shape from Shading [ Omachi07 , Yanagisawa14] [Carter10, Farag13 , Mostafa14] Contours and Feature Points [ Zheng11 ] Intra-Oral Scanners [3M True Definition, iTero , 3Shape TRIOS] Michael Zollhöfer Model-Based Teeth Reconstruction 19

RELATED WORK Medical Dentistry Photogrammetric Reconstruction From CT images Shape from Shading [ Omachi07 , Yanagisawa14] [Carter10, Farag13 , Mostafa14] Contours and Feature Points [ Zheng11 ] Intra-Oral Scanners [3M True Definition, iTero , 3Shape TRIOS] Statistical Models [ Mehl05 , Buchaillard07] Michael Zollhöfer Model-Based Teeth Reconstruction 20

HUMAN TEETH Scans Michael Zollhöfer Model-Based Teeth Reconstruction 21

HUMAN TEETH Scans Michael Zollhöfer Model-Based Teeth Reconstruction 22

HUMAN TEETH Scans Michael Zollhöfer Model-Based Teeth Reconstruction 23

OVERVIEW Scans Michael Zollhöfer Model-Based Teeth Reconstruction 24

OVERVIEW Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 25

OVERVIEW Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 26

OVERVIEW Teeth Prior Model Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 27

OVERVIEW Teeth Prior Model Teeth Reconstruction Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 28

OVERVIEW Teeth Prior Model Teeth Reconstruction 3D Reconstruction Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 29

OVERVIEW Teeth Prior Model Teeth Reconstruction 3D Reconstruction Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 30

OVERVIEW Teeth Prior Model Teeth Reconstruction 3D Reconstruction Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 31

TEETH MODEL • 86 high resolution plaster cast scans Michael Zollhöfer Model-Based Teeth Reconstruction 32

TEETH MODEL • 86 high resolution plaster cast scans Michael Zollhöfer Model-Based Teeth Reconstruction 33

PREPROCESSING Teeth Segmentation Fitted Template Michael Zollhöfer Model-Based Teeth Reconstruction 34

PREPROCESSING Teeth Segmentation Fitted Template Input Scan Michael Zollhöfer Model-Based Teeth Reconstruction 35

PREPROCESSING Teeth Segmentation Fitted Template Input Scan Michael Zollhöfer Model-Based Teeth Reconstruction 36

PREPROCESSING Teeth Segmentation Fitted Template Input Scan Michael Zollhöfer Model-Based Teeth Reconstruction 37

PARAMETRIC TEETH MODEL 𝐣 𝐂 𝛖 𝐣 𝐚 𝛖 = 𝐓 𝑼 τ 𝐁 𝛖 + 𝛃 𝛖 𝐣 Michael Zollhöfer Model-Based Teeth Reconstruction 38

PARAMETRIC TEETH MODEL 𝐣 𝐂 𝛖 𝐣 𝐚 𝛖 = 𝐓 𝑼 τ 𝐁 𝛖 + 𝛃 𝛖 𝐣 𝐁 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 39

PARAMETRIC TEETH MODEL 𝐣 𝐂 𝛖 𝐣 𝐚 𝛖 = 𝐓 𝑼 τ 𝐁 𝛖 + 𝛃 𝛖 𝐣 𝐁 𝛖 𝐣 𝐂 𝛖 𝐣 𝛃 𝛖 shape of tooth 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 40

PARAMETRIC TEETH MODEL 𝐣 𝐂 𝛖 𝐣 𝐚 𝛖 = 𝐓 𝑼 τ 𝐁 𝛖 + 𝛃 𝛖 𝐣 𝐁 𝛖 𝑼 τ 𝐣 𝐂 𝛖 𝐣 𝛃 𝛖 position of tooth 𝛖 shape of tooth 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 41

PARAMETRIC TEETH MODEL 𝐣 𝐂 𝛖 𝐣 𝐚 𝛖 = 𝐓 𝑼 τ 𝐁 𝛖 + 𝛃 𝛖 𝐣 𝐁 𝛖 𝐓 𝑼 τ 𝐣 𝐂 𝛖 𝐣 𝛃 𝛖 Scale and position position of tooth 𝛖 shape of tooth 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 42

GAUSSIAN DISTRIBUTION 𝐣 }) 𝒀 = (𝐓 , 𝑼 τ , {𝛃 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 43

GAUSSIAN DISTRIBUTION • All dimensions modeled via multivariate Gaussians 𝐣 }) 𝒀 = (𝐓 , 𝑼 τ , {𝛃 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 44

GAUSSIAN DISTRIBUTION • All dimensions modeled via multivariate Gaussians 𝐣 }) 𝒀 = (𝐓 , 𝑼 τ , {𝛃 𝛖 Michael Zollhöfer Model-Based Teeth Reconstruction 45

GAUSSIAN DISTRIBUTION • All dimensions modeled via multivariate Gaussians 𝐣 }) 𝒀 = (𝐓 , 𝑼 τ , {𝛃 𝛖 𝒒 𝒀 Teeth Prior Michael Zollhöfer Model-Based Teeth Reconstruction 46

OVERVIEW Teeth Prior Model Teeth Reconstruction 3D Reconstruction Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 47

OVERVIEW Teeth Prior Model Teeth Reconstruction 3D Reconstruction Dental Scans Contour Extraction Static Images/Video Clip Scans Michael Zollhöfer Model-Based Teeth Reconstruction 48

CONTOUR EXTRACTION Scans Michael Zollhöfer Model-Based Teeth Reconstruction 49

CONTOUR EXTRACTION • Teeth are featureless Scans Michael Zollhöfer Model-Based Teeth Reconstruction 50

CONTOUR EXTRACTION • Teeth are featureless  Except the silhouette Scans Michael Zollhöfer Model-Based Teeth Reconstruction 51

CONTOUR EXTRACTION • Teeth are featureless  Except the silhouette . Scans Michael Zollhöfer Model-Based Teeth Reconstruction 52

CONTOUR EXTRACTION • Teeth are featureless  Except the silhouette .  Boosted Edge Learning (BEL) Scans Michael Zollhöfer Model-Based Teeth Reconstruction 53

FEATURE FUSION Scans Michael Zollhöfer Model-Based Teeth Reconstruction 54

FEATURE FUSION Input Scans Michael Zollhöfer Model-Based Teeth Reconstruction 55

FEATURE FUSION BEL Input Scans Michael Zollhöfer Model-Based Teeth Reconstruction 56

FEATURE FUSION BEL Input Gabor Scans Michael Zollhöfer Model-Based Teeth Reconstruction 57

FEATURE FUSION BEL Fusion Input Gabor Scans Michael Zollhöfer Model-Based Teeth Reconstruction 58

FEATURE FUSION Formulas in BEL the paper! Fusion Input Gabor Scans Michael Zollhöfer Model-Based Teeth Reconstruction 59