DEEP LEARNING applications Julia Rabetti Giannella Research assistant at VISGRAF Lab PhD in Design and Technology (PPDESDI-UERJ) juliagiannella@gmail.com
APPLICATIONS • Colorization of Black and White Images • Adding Sounds To Silent Movies • Object Classification in Photographs • Automatic Handwriting Generation • Character Text Generation. • Image Caption Generation. • Automatic Game Playing • Artistic style transfer Source: http://machinelearningmastery.com/inspirational-applications-deep-learning/
1) Colorization of Black and White Images • problem of adding color to black and white photographs • traditionally, this was done by hand with human effort • CV task attacked by different approaches • topic of relative importance in SIGGRAPH and EUROGRAPH • DL approach involves the use of very large CNN and supervised layers that recreate the image with the addition of color
Paper Colorful Image Colorization (ECCV, 2016) Source: http://richzhang.github.io/colorization/
Network architecture Source: https://arxiv.org/pdf/1603.08511.pdf
Semantic interpretability of results Source: http://richzhang.github.io/colorization/
[Algorithmia] Demo Source: http://demos.algorithmia.com/colorize-photos/
Dana Keller - designer and photo colorizer Source: https://www.youtube.com/watch?v=bYHnWhZkAIc Source: http://www.danarkeller.com/about/
Comparing Keller Algorithmia
Comparing Keller Algorithmia
Comparing Keller Algorithmia
Comparing Keller Algorithmia
Comparing Keller Algorithmia
Comparing Keller Algorithmia
2) Object Classification in Photographs • task requires the classification of objects within a photograph as one of a set of previously known objects • State-of-the-art results have been achieved on benchmark examples of this problem using very large CNN • derives from image classification task • breakthrough: ImageNet Classification with Deep Convolutional Neural Networks (Krizhevsky et al., 2012) • AlexNet won ILSVRC-2012 challenge Source: http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
Classification with localization • more complex variation of this task involves specifically identifying one or more objects within the scene of the photograph and drawing a box around them • GoogLeNet won ILSVRC-2014 challenge in this task Source: https://research.googleblog.com/2014/09/building-deeper-understanding-of-images.html
2.1) DL and RIO2016 • VISGRAF project (out 2016) • task: automatically classify and cluster images by subject features related to the Olympic Games, Olympic Torch • CNN model and supervised learning • TensorFlow (open source software library) • Inception-v3 ( Going Deeper with Convolutions , 2015) • transfer learning (manually labeled 100 examples) Source: http://lvelho.impa.br/dl_rio2016/index.html Source: https://arxiv.org/abs/1409.4842
Confidence score A subset of 12 from 2091 images with confidence score over 83% for the Olympic torch category Source: http://lvelho.impa.br/dl_rio2016/metodologia.html
Torch Mosaic Source: http://lvelho.impa.br/dl_rio2016/mosaico.html
Torch Mosaic Source: http://lvelho.impa.br/dl_rio2016/mosaico.html
2.2) Twitter Facial Analysis Reveals Demographics of Presidential Campaign Followers • (Mit Technology Review, march 2016) • IN: Conference on Web and Social Media • understand follower demographics of Trump and Clinton by crossing Twitter metadata and facial features • a CNN model on followers’ profile images extracts information on gender, race and age Source: https://www.technologyreview.com/s/601074/twitter-facial-analysis-reveals-demographics-of-presidential- campaign-followers/?utm_campaign=add_this&utm_source=email&utm_medium=post Source: https://arxiv.org/abs/1603.03097
A Comparison of the Trumpists and Clintonists C"lintonists" in the Twitter Sphere Source: https://arxiv.org/abs/1603.03097
2.3) NVIDIA DRIVENet Demo - Visualizing a Self-Driving Car Source: https://www.youtube.com/watch?v=HJ58dbd5g8g
3) Artistic style transfer • task: separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic images • A Neural Algorithm of Artistic Style (Gatys et al., 2015) Source: https://arxiv.org/abs/1508.06576
Convolutional Neural Network (CNN) Source: https://arxiv.org/abs/1508.06576
An example The style transfer algorithm crosses a photo with a painting style; for example Neil deGrasse Tyson in the style of Kadinsky’s Jane Rouge Bleu. Photo by Guillaume Piolle, used with permission. Source: https://research.googleblog.com/2016/02/exploring-intersection-of-art-and.html
3.1) DeepDream • computer vision program created by Google • given an input image returns a version with h"allucinogenic" appearance • originates in a CNN codenamed Inception after the film of the same name developed for the ILSVRC-2014 • CNN can also be run in reverse, to do synthesis • enhance faces and certain animals -> pareidolia results Source: http://deepdreamgenerator.com/ Source: https://en.wikipedia.org/wiki/DeepDream
3.1) DeepDream Source: http://deepdreamgenerator.com/ Source: https://en.wikipedia.org/wiki/DeepDream
3.2) Prisma App • photo-editing application that utilizes a neural network and to transform the image into an artistic effect • became popular on July 2016 • created by Alexey Moiseenkov • reference A Neural Algorithm of Artistic Style (2016) Source: http://prisma-ai.com/ Source: https://en.wikipedia.org/wiki/Prisma_(app)
3.2) Prisma App
3.2) Prisma App
3.3) Artistic style transfer (video) • Artistic style transfer for videos (Ruder et al.,2016) Source: https://arxiv.org/abs/1604.08610 Source: https://www.youtube.com/watch?v=Khuj4ASldmU
3.4) Supercharging Style Transfer for video • A Learned Representation For Artistic Style (Dumoulin et al., 2016) • CNN that learns multiple styles at the same time • method enables style interpolation Source: https://arxiv.org/abs/1610.07629 Source: https://research.googleblog.com/2016/10/supercharging-style-transfer.html
3.4) Supercharging Style Transfer for video Source: https://www.youtube.com/watch?v=6ZHiARZmiUI
Recommend
More recommend
