Data sets Niloy Mit Ni Mitra Ias asonas Kok okkin inos os - PowerPoint PPT Presentation

Deep Learning for Graphics Data sets Niloy Mit Ni Mitra Ias asonas Kok okkin inos os Pau aul l Gu Guer errero Vl Vladim imir ir Ki Kim Kos ostas Rematas Tob obias Ri Ritschel UCL UCL/Facebook UCL Adobe Research U Washington UCL

Timetable Niloy Iasonas Paul Vova Kostas Tobias Introduction X X X X Theory X NN Basics X X Supervised Applications Data X Unsupervised Applications X Beyond 2D X X X Outlook X X X X X X EG Course “Deep Learning for Graphics” 2

Modalities • 2D images • 3D datasets • Data augmentation • Synthetic image training data EG Course “Deep Learning for Graphics” 3

2D image data sets EG Course “Deep Learning for Graphics” 4

Image Datasets • MNIST • Handwritten digits • 28x28 images • 10 classes • 60k train/10k test EG Course “Deep Learning for Graphics” 5

Image Datasets • CIFAR10 • Object images • 10 classes • 32 x 32 pixels • 50k train/10k test EG Course “Deep Learning for Graphics” 6

Image Datasets • PASCAL VOC • Multiple objects per image • 20 classes • Labels for classification, segmentation, detection EG Course “Deep Learning for Graphics” 7

Image Datasets • ImageNet • The main ”fuel” for deep learning • 1000 classes • Classification/Detection (200 classes) • Structure from WordNet EG Course “Deep Learning for Graphics” 8

Image Datasets • MS COCO • Boost to DL class/instance segmentation and keypoint detection • 80 classes • 200k images • Instance segmentation masks (>1 mil) • Human keypoints (250k) EG Course “Deep Learning for Graphics” 9

LDR 2 HDR • http://hdrv.org/hdrcnn/material/testset/index.html EG Course “Deep Learning for Graphics” 10

3D data sets EG Course “Deep Learning for Graphics” 11

3D Datasets • ShapeNet • Similar to ImageNet but for CAD models • 55 common categories • 10k+ models • ShapeNetCore • 12k models • Additional annotations (real world dimensions, materials, … ) EG Course “Deep Learning for Graphics” 12

3D Datasets • SMPL • Parametric human shape model • 72 parameters control pose and human shape • Fully differentiable • Useful for human shape estimation, motion capture etc EG Course “Deep Learning for Graphics” 13

Data Augmentation EG Course “Deep Learning for Graphics” 14

Data Augmentation • Augment existing data with image operations to reduce overfitting • Much larger dataset • Learn expected transformations EG Course “Deep Learning for Graphics” 15

Data Augmentation • Mirror EG Course “Deep Learning for Graphics” 16

Data Augmentation • Mirror • Rotation EG Course “Deep Learning for Graphics” 17

Data Augmentation • Mirror • Rotation • Translation EG Course “Deep Learning for Graphics” 18

Data Augmentation • Mirror • Rotation • Translation • Zoom EG Course “Deep Learning for Graphics” 19

Data Augmentation • Mirror • Rotation • Translation • Zoom • Blur EG Course “Deep Learning for Graphics” 20

Data Augmentation • Mirror • Rotation • Translation • Zoom • Blur • Noise EG Course “Deep Learning for Graphics” 21

Data Augmentation • Mirror • Rotation • Translation • Zoom • Blur • Noise • Color transforms Library: https://github.com/codebox/image_augmentor EG Course “Deep Learning for Graphics” 22

Creating Your Own EG Course “Deep Learning for Graphics” 23

Material/Illumination EG Course “Deep Learning for Graphics” 24

Material/Illumination EG Course “Deep Learning for Graphics” 25

Decomposition EG Course “Deep Learning for Graphics” 26

Synthetic data EG Course “Deep Learning for Graphics” 27

Synthetic Data EG Course “Deep Learning for Graphics” 28

Synthetic Data for DL • 3D models + renderer = unlimited data • Suitable for data hungry approaches such as deep networks • Higher fidelity -> smaller discrepancy between synthetic and real EG Course “Deep Learning for Graphics” 29

How To Generate Synthetic Data • What you need • 3D models with task-specific annotation • Renderer • Example: Indoor depth estimation (McCormac et al) + = 3D Room with furniture NVIDIA OptiX renderer 30 Data with annotation EG Course “Deep Learning for Graphics”

How To Generate Synthetic Data • How much fidelity? OpenGL Natural Illumination + AO Path tracer, global Illumination (VRay) EG Course “Deep Learning for Graphics” 31

Selecting viewpoint • Object in the center • Sample from the hemisphere • Multiple FOV, Target, Up, etc • Scene • Simulate human camera path • Optimize camera position for a particular objective (eg segmentation) EG Course “Deep Learning for Graphics” 32

Selecting Appearance • Materials • Capture the variability of real world objects • BRDF, textures (MERL DB) • Illumination • Capture the effect of environment • Increase realism • Laval indoor DB http://indoor.hdrdb.com/ EG Course “Deep Learning for Graphics” 33

Thank you! http://geometry.cs.ucl.ac.uk/dl4g/ EG Course “Deep Learning for Graphics” 34