au autoenc ncoders
play

Au Autoenc ncoders Prof. Leal-Taix and Prof. Niessner 1 Mac - PowerPoint PPT Presentation

Feb 03, 2023 •300 likes •978 views

Au Autoenc ncoders Prof. Leal-Taix and Prof. Niessner 1 Mac Machine e lear earning Unsupervised learning Supervised learning Labels or target classes Goal: learn a mapping from input to label Classification,

  1. Au Autoenc ncoders Prof. Leal-Taixé and Prof. Niessner 1

  2. Mac Machine e lear earning Unsupervised learning Supervised learning Labels or target • classes • Goal: learn a mapping from input to label Classification, • regression Prof. Leal-Taixé and Prof. Niessner 2

  3. Mac Machine e lear earning Unsupervised learning Supervised learning CAT DOG DOG CAT CAT DOG Prof. Leal-Taixé and Prof. Niessner 3

  4. Mac Machine e lear earning Unsupervised learning Supervised learning CAT No label or target class • • Find out properties of DOG DOG the structure of the data Clustering (k-means, • CAT PCA) CAT DOG Prof. Leal-Taixé and Prof. Niessner 4

  5. Mac Machine e lear earning Unsupervised learning Supervised learning CAT DOG DOG CAT CAT DOG Prof. Leal-Taixé and Prof. Niessner 5

  6. Mac Machine e lear earning Unsupervised learning Supervised learning CAT DOG DOG CAT CAT DOG Prof. Leal-Taixé and Prof. Niessner 6

  7. Un Unsupervi vised le learning with au autoenc ncoders Prof. Leal-Taixé and Prof. Niessner 7

  8. Au Auto toenc ncoders • Unsupervised approach for learning a lower- dimensional feature representation from unlabeled training data Prof. Leal-Taixé and Prof. Niessner 8

  9. Au Auto toenc ncoders • From an input image to a feature representation (bottleneck layer) x • Encoder: a CNN in our case z Input Image Conv Prof. Leal-Taixé and Prof. Niessner 9

  10. Au Auto toenc ncoders • Why do we need this dimensionality reduction? • To capture the patterns, the most meaningful factors of variation in our data • Other dimensionality reduction methods? Prof. Leal-Taixé and Prof. Niessner 10

  11. Au Auto toenc ncoder: tr traini ning ng Reconstruction Loss (like L1, L2) Input Image Output Image Conv Transpose Conv Prof. Leal-Taixé and Prof. Niessner 11

  12. Au Auto toenc ncoder: tr traini ning ng Input images Reconstruction x’ Input x Reconstructed images Latent space z dim (z) < dim (x) Prof. Leal-Taixé and Prof. Niessner 12

  13. Au Auto toenc ncoder: tr traini ning ng • No labels required Reconstruction x’ • We can use unlabeled data to Input x first get its structure Latent space z dim (z) < dim (x) Prof. Leal-Taixé and Prof. Niessner 13

  14. Au Auto toenc ncoder: Use : Use C Case ses Embedding of MNIST numbers Prof. Leal-Taixé and Prof. Niessner 14

  15. Au Auto toenc ncoder for pre-tr traini ning ng • Test case: medical applications based on CT images – Large set of unlabeled data. – Small set of labeled data. • We cannot do: take a network pre-trained on ImageNet. Why? • The image features are different CT vs natural images Prof. Leal-Taixé and Prof. Niessner 15

  16. Au Auto toenc ncoder for pre-tr traini ning ng • Test case: medical applications based on CT images – Large set of unlabeled data. – Small set of labeled data. • We can do: pre-train our network using an autoencoder to “learn” the type of features present in CT images Prof. Leal-Taixé and Prof. Niessner 16

  17. Au Auto toenc ncoder for pre-tr traini ning ng • Step 1: Unsupersived training with autoencoders Reconstruction Input Prof. Leal-Taixé and Prof. Niessner 17

  18. Au Auto toenc ncoder for pre-tr traini ning ng • Step 2: Supervised training with the labeled data Reconstruction Input Throw away the decoder Prof. Leal-Taixé and Prof. Niessner 18

  19. Auto Au toenc ncoder for pre-tr traini ning ng • Step 2: Supervised training with the labeled data Backprop x y as always Loss z Input y ∗ Ground truth labels for supervised learning Prof. Leal-Taixé and Prof. Niessner 19

  20. Wh Why usi using au autoen oencoder oders? • Use 1: pre-training, as mentioned before – Image à same image reconstructed – Use the encoder as “feature extractor” • Use 2: Use them to get pixel-wise predictions – Image à semantic segmentation – Low-resolution image à High-resolution image – Image à Depth map Prof. Leal-Taixé and Prof. Niessner 20

  21. Au Autoenc ncoders fo for pi pixel xel-wis wise predic ictio ions ns Prof. Leal-Taixé and Prof. Niessner 21

  22. Se Semanti ntic c Se Segmenta ntati tion n (FCN) • Recall the Fully Convolutional Networks Can we do better? Prof. Leal-Taixé and Prof. Niessner [Long et al. 15] Fully Convolutional Networks for Semantic Segmetnation (FCN) 22

  23. Se SegNet Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 23

  24. Se SegNet • Enc Encoder : normal convolutional filters + pooling • De Decoder : Upsampling + convolutional filters Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 24

  25. Se SegNet • Enc Encoder : normal convolutional filters + pooling • De Decoder : Upsampling + convolutional filters Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 25

  26. Se SegNet • Enc Encoder : normal convolutional filters + pooling • De Decoder : Upsampling + convolutional filters • The convolutional filters in the decoder are learned using backprop and their goal is to refine the upsampling Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 26

  27. Rec Recal all tran anspos osed ed con onvol olution on • Transposed convolution Output 5x5 - Unpooling - Convolution filter (learned) - Also called up-convolution (never deconvolution) Input 3x3 Prof. Leal-Taixé and Prof. Niessner 27

  28. Se SegNet • Enc Encoder : normal convolutional filters + pooling • De Decoder : Upsampling + convolutional filters ax layer: The output of the soft-max classifier is • Softmax a K channel image of probabilities where K is the number of classes. Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 28

  29. Upsampli ling Prof. Leal-Taixé and Prof. Niessner 29

  30. Ty Types of upsa upsampl plings gs • 1. Interpolation ? Prof. Leal-Taixé and Prof. Niessner 30

  31. Ty Types of upsa upsampl plings gs • 1. Interpolation Original image Nearest neighbor interpolation Bilinear interpolation Bicubic interpolation Prof. Leal-Taixé and Prof. Niessner 31 Image: Michael Guerzhoy

  32. Ty Types of upsa upsampl plings gs • 1. Interpolation Few artifacts Prof. Leal-Taixé and Prof. Niessner 32 Image: Michael Guerzhoy

  33. Ty Types of upsa upsampl plings gs • 2. Fixed unpooling efficient + CONVS A. Dosovitskiy, “Learning to Generate Chairs, Tables and Cars with Convolutional Networks“. TPAMI 2017 Prof. Leal-Taixé and Prof. Niessner 33

  34. Ty Types of upsa upsampl plings gs • 3. Unpooling: “à la DeconvNet” Keep the locations where the max came from Prof. Leal-Taixé and Prof. Niessner 34

  35. Ty Types of upsa upsampl plings gs • 3. Unpooling: “à la DeconvNet” Now: convolutional filters are LEARNED In DeConvNet: we convolve with the transpose of the learned filter Prof. Leal-Taixé and Prof. Niessner 35

  36. Ty Types of upsa upsampl plings gs • 3. Unpooling: “à la DeconvNet” Keep the details of the structures Prof. Leal-Taixé and Prof. Niessner 36

  37. U-Net o Net or s skip con connecti ection ons i s in au autoenc ncoders Prof. Leal-Taixé and Prof. Niessner 38

  38. Ski Skip Conne nnecti ctions ns • U-Net Pass the low- level information High-level information Recall ResNet O. Ronneberger et al. “U-Net: Convolutional Networks for Biomedical Image Segmentation”. MICCAI 2015 Prof. Leal-Taixé and Prof. Niessner 39

  39. Ski Skip Conne nnecti ctions ns • U-Net: zoom in append O. Ronneberger et al. “U-Net: Convolutional Networks for Biomedical Image Segmentation”. MICCAI 2015 Prof. Leal-Taixé and Prof. Niessner 40

  40. Ski Skip Conne nnecti ctions ns • Concatenation connections C. Hazirbas et al. “Deep depth from focus”. ACCV 2018 Prof. Leal-Taixé and Prof. Niessner 41

  41. Ski Skip Conne nnecti ctions ns • Widely used in Autoencoders • At what levels the skip connections are needed depends on your problem Prof. Leal-Taixé and Prof. Niessner 42

  42. Au Autoenc ncoders in in Vi Visi sion on Prof. Leal-Taixé and Prof. Niessner 43

  43. Se SegNet Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 44

  44. Se SegNet Input Ground truth SegNet Badrinarayanan et al. „SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation“. TPAMI 2016 Prof. Leal-Taixé and Prof. Niessner 45

  45. Mon Monoc ocular ar dep depth • Unsupervised monocular depth estimation R. Garg et al. „Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue“ ECCV 2016 Prof. Leal-Taixé and Prof. Niessner 46

Recommend

More recommend