language based colorization of scene sketches
play

Language-based Colorization of Scene Sketches Changqing Zou* 1,2 , - PowerPoint PPT Presentation

Mar 17, 2024 •878 likes •1.52k views

Language-based Colorization of Scene Sketches Changqing Zou* 1,2 , Haoran Mo* 1 , Chengying Gao 1 , Ruofei Du 3 , Hongbo Fu 4 Sun Yat-sen University 1 Huawei Noahs Ark Lab 2 Google 3 City University of Hong Kong 4 Nov. 20 th , 2019

  1. Language-based Colorization of Scene Sketches Changqing Zou* 1,2 , Haoran Mo* 1 , Chengying Gao 1 , Ruofei Du 3 , Hongbo Fu 4 Sun Yat-sen University 1 Huawei Noah’s Ark Lab 2 Google 3 City University of Hong Kong 4 Nov. 20 th , 2019 SA2019.SIGGRAPH.ORG CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  2. Motivation: Abstract Data and Human Cognition • Sparse • Highly abstract SA2019.SIGGRAPH.ORG 1 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  3. Motivation: Abstract Data Understanding • Lots of early exploration with computational models [Eitz et. al 2012, Li et. al 2013, Schneider et. al 2014, Li et. al 2015] • Limited ability of understanding object-level sketches [1] M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? In SIGGRAPH, 2012. [2] Y. Li, Y. Song, and S. Gong. Sketch recognition by ensemble matching of structured features. In BMVC, 2013. [3] R. G. Schneider and T. Tuytelaars. Sketch classification and classification-driven analysis using fisher vectors. In SIGGRAPH Asia, 2014. [4] Y. Li, T. M. Hospedales, Y. Song, and S. Gong. Free-hand sketch recognition by multikernel feature learning. CVIU, 2015. SA2019.SIGGRAPH.ORG 2 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  4. Motivation: Sketch Understanding TU-Berlin / Sketchy / QuickDraw SA2019.SIGGRAPH.ORG 3 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  5. Motivation: Sketch Understanding vs SketchyScene [Zou et. al 2018] Scene-level sketch: • Interaction among multiply objects • More empty region, lack of contextual information [1] Changqing Zou, et. al. SketchyScene: Richly-Annotated Scene Sketches. In ECCV, 2018. SA2019.SIGGRAPH.ORG 4 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  6. Motivation: Scene Sketch Understanding Scene sketch colorization Natural language SA2019.SIGGRAPH.ORG 5 6 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  7. Motivation: Why Language-based? • Natural: easily adopted by novice users • Touchless: friendly for people with upper limb impairments • Effective: support batch-processing colorization “the bus is yellow with blue windows” SA2019.SIGGRAPH.ORG 7 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  8. Motivation: Language-based Sketch Colorization • Toy problem, but not simple …… SA2019.SIGGRAPH.ORG 8 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  9. Challenges A. Understanding scene-level sketch is very hard • Too abstract • Lack of contextual information CMPlaces Photo-Sketching SketchyScene [Castrejon et. al 2016] [Li et. al 2019] [Zou et. al 2018] SA2019.SIGGRAPH.ORG 9 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  10. Challenges B. Multimodal learning between language and scene sketch • Mapping between language and target objects “the dog on the rightmost has orange body” SA2019.SIGGRAPH.ORG 10 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  11. Challenges B. Multimodal learning between language and scene sketch • One or multiple objects with single instruction “the two trees on the left of the house are light green” SA2019.SIGGRAPH.ORG 10 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  12. Challenges B. Multimodal learning between language and scene sketch • Various free expressions of location “the dog in the middle is gray ” / “the dog near the house is gray” SA2019.SIGGRAPH.ORG 10 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  13. Challenges C. Multimodal learning between language and object sketch • Object-part-level colorization • Various free expressions of colors “the bus is dark (navy/…) blue with white windows ” SA2019.SIGGRAPH.ORG 11 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  14. Inspiration: Drawing and Intelligence Development • Sensitive to line drawing and color • Mode of thinking and creation SA2019.SIGGRAPH.ORG 12 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  15. Inspiration: Language and Literacy Development • Embedding voice in traditional drawings supports children’s literacy development [Raffle et. al 2007] SA2019.SIGGRAPH.ORG 13 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  16. Related Work SA2019.SIGGRAPH.ORG CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  17. Related Work A. Language-based Image Segmentation • Fusion of textual and visual information • Only natural images • Only one binary mask for single or multiple target objects “ second vase from right ” “ the bottom two luggage cases being rolled ” [Ye et. al 2019] Our work SA2019.SIGGRAPH.ORG 14 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  18. Related Work B. Language-based Image Colorization • Language-based image editing (LBIE) [Chen et. al 2018] • Require pair-wise scene-level sketch and color image “ The flower has red petals with yellow stigmas in the middle ” Scene sketch [Chen et. al 2018] Our work [1] Jianbo Chen, et. al. Language-Based Image Editing With Recurrent Attentive Models. In CVPR, 2018. SA2019.SIGGRAPH.ORG 15 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  19. Our Work SA2019.SIGGRAPH.ORG CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  20. Main contributions • Language-based colorization system for scene sketches • Language-based instance segmentation network for scene sketches • Three large-scale datasets for language-based scene sketch colorization SA2019.SIGGRAPH.ORG 16 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  21. Proposed approach A. System pipeline • Divide-and-conquer and progressive strategy • Two modes (foreground and background) • Three models (instance matching, foreground colorization, background colorization) System pipeline SA2019.SIGGRAPH.ORG 17 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  22. Proposed approach B.1 Instance Matching Model • Training: two phases for binary mask (b) generation • Inferring: fuse binary mask with instance segmentation results Foreground mode SA2019.SIGGRAPH.ORG 18 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  23. Proposed approach B.2 Foreground Colorization Model • GAN + fusion module • Colorize objects from different categories Foreground mode SA2019.SIGGRAPH.ORG 20 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  24. Proposed approach B.3 Background Colorization Model • cGAN + two-branch decoder • Colorization branch • Explicit segmentation branch (segmentation loss) Background mode SA2019.SIGGRAPH.ORG 21 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  25. Datasets • MATCHING dataset : 38k groups of text-based instance segmentation data. SA2019.SIGGRAPH.ORG 22 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  26. Datasets • FOREGROUND dataset : 4k groups of text-based sketch object colorization data. SA2019.SIGGRAPH.ORG 22 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  27. Datasets • BACKGROUND dataset : 20k groups of text-based background colorization data. SA2019.SIGGRAPH.ORG 22 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  28. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Single object “the bus is yellow with blue windows” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  29. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Single object “the bus is yellow with blue windows” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  30. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Multiple objects “ all the trees are dark green” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  31. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Multiple objects “ all the trees are dark green” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

Recommend

Colorization using Optimization Anat Levin Dani Lischinski Yair Weiss Colorization Colorization

Colorization using Optimization Anat Levin Dani Lischinski Yair Weiss Colorization Colorization

Colorization using Optimization Anat Levin Dani Lischinski Yair Weiss Colorization Colorization : a computer-assisted process of adding color to a monochrome image or ) movie. (Invented by Wilson Markle, 1970 Motivation Colorizing black

366 views • 20 slides
Automatic Colorization Gustav Larsson TTI Chicago / University of Chicago Joint work with

Automatic Colorization Gustav Larsson TTI Chicago / University of Chicago Joint work with

Automatic Colorization Gustav Larsson TTI Chicago / University of Chicago Joint work with Michael Maire and Greg Shakhnarovich NVIDIA @ SIGGRAPH 2016 Colorization Let us first define colorization Colorization Definition 1: The inverse

693 views • 50 slides
Learning Representations for Automatic Colorization Gustav Larsson, Michael Maire, Greg

Learning Representations for Automatic Colorization Gustav Larsson, Michael Maire, Greg

Learning Representations for Automatic Colorization Gustav Larsson, Michael Maire, Greg Shakhnarovich TTI Chicago / University of Chicago ECCV 2016 Colorization Let us first define colorization Colorization Definition 1: The inverse of

778 views • 65 slides
DRAFTING THE UNIVERSAL LANGUAGE SKETCHING Rough Sketches are the most common recording method.

DRAFTING THE UNIVERSAL LANGUAGE SKETCHING Rough Sketches are the most common recording method.

DRAFTING THE UNIVERSAL LANGUAGE SKETCHING Rough Sketches are the most common recording method. The term rough describes the state of the design ideas. It suggests that the designs are incomplete and unrefined. Refined Sketches are

329 views • 12 slides
Scene Graphs Scene Representation How does one describe the objects in a 3D scene? Scene

Scene Graphs Scene Representation How does one describe the objects in a 3D scene? Scene

Scene Graphs Scene Representation How does one describe the objects in a 3D scene? Scene Graphs 1 Scene Modeling Languages / API From the low level to the high level State Machine Model -- OpenGL Graph Based Scene Languages

483 views • 17 slides
COLORIZATION USING KNET Jeffrey Lu and Kevin Liu 6.338/18.337 Fall 2017 MOTIVATION Image

COLORIZATION USING KNET Jeffrey Lu and Kevin Liu 6.338/18.337 Fall 2017 MOTIVATION Image

JULIA IMAGE COLORIZATION USING KNET Jeffrey Lu and Kevin Liu 6.338/18.337 Fall 2017 MOTIVATION Image colorization for 6.869 Computer Vision (Jeffrey) Cool application of deep learning Restoring old black and white photos Abstract

390 views • 10 slides
2004: Poisson Matting 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: The

2004: Poisson Matting 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: The

2004: Poisson Matting 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: The non-photorealistic camera 2004: Colorization 2004: Colorization 2004: Colorization 2004: Colorization 2004: Colorization 2005: Color2Gray

626 views • 30 slides
Physical Sketches CPSC 581 - Fall 2015 Motivation Experience your sketches in a more physical

Physical Sketches CPSC 581 - Fall 2015 Motivation Experience your sketches in a more physical

Physical Sketches CPSC 581 - Fall 2015 Motivation Experience your sketches in a more physical form Test your interaction design with potential end-users Better insights by exploring in 3D! Architecture scale model Industrial Design

417 views • 27 slides
Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a

Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a

Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a collection of images l 3D model-based rendering (MBR): A scene is represented by a 3D model plus texture maps l Differences u Many scene details

979 views • 55 slides
Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai

Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai

Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai Sheng Tai, Vatsal Sharan, Peter Bailis Stanford University 1 Motivation: Monitoring production data streams Billions of events / day of mobile app

245 views • 23 slides
Episode 42: I Made Slides 10 February 2019 The Three-Act, Seven Scene Structure Act I:

Episode 42: I Made Slides 10 February 2019 The Three-Act, Seven Scene Structure Act I:

Episode 42: I Made Slides 10 February 2019 The Three-Act, Seven Scene Structure Act I: Act III: Scene 1: The Inciting Incident Scene 5: Dark Moment Scene 2: Choice to Engage Scene 6: Climax Act II: Scene 7: Resolution Scene 3: The

217 views • 10 slides
COMPSCI 326 Web Programming Week 09: ER Diagram Sketches Agenda 4:00 4:35 ER Diagram

COMPSCI 326 Web Programming Week 09: ER Diagram Sketches Agenda 4:00 4:35 ER Diagram

COMPSCI 326 Web Programming Week 09: ER Diagram Sketches Agenda 4:00 4:35 ER Diagram Sketches 4:35 5:10 Team Lightning Data Talks 2 minute presentation on: 1. 1-2 sentences overviewing your startups application 2. 1-2

632 views • 22 slides
Lo Local Im Image Pri riors for Automatic Im Image Colorization with Simultaneous Cla lassifi

Lo Local Im Image Pri riors for Automatic Im Image Colorization with Simultaneous Cla lassifi

Le Let th there be Color!: Jo Joint End-to to-end Le Learning of f Global and Lo Local Im Image Pri riors for Automatic Im Image Colorization with Simultaneous Cla lassifi fication Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa

525 views • 50 slides
Bloom Filters, Count Sketches and Adaptive Sketches Rice University Anshumali Shrivastava

Bloom Filters, Count Sketches and Adaptive Sketches Rice University Anshumali Shrivastava

Bloom Filters, Count Sketches and Adaptive Sketches Rice University Anshumali Shrivastava anshumali@rice.edu 29th August 2016 Anshumali Shrivastava (COMP 640) Sketching 29th August 2016 1 / 22 Basics: Universal Hashing Basic tool for

542 views • 33 slides
Learning to Infer Program Sketches Maxwell Nye, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama

Learning to Infer Program Sketches Maxwell Nye, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama

Learning to Infer Program Sketches Maxwell Nye, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama Goal: We want to automatically write code from the kinds of specifications humans can easily provide, such as examples or natural language

1.13k views • 23 slides
Volumetric Scene Reconstruction Volumetric Scene Reconstruction Goal Goal from Multiple

Volumetric Scene Reconstruction Volumetric Scene Reconstruction Goal Goal from Multiple

Image Image- -Based Scene Reconstruction Based Scene Reconstruction Volumetric Scene Reconstruction Volumetric Scene Reconstruction Goal Goal from Multiple Views from Multiple Views Automatic construction of photo Automatic

378 views • 21 slides
GPU-Based Scene Generation for Flight Simulation Tim Woodard Chief Technology Officer Diamond

GPU-Based Scene Generation for Flight Simulation Tim Woodard Chief Technology Officer Diamond

GPU-Based Scene Generation for Flight Simulation Tim Woodard Chief Technology Officer Diamond Visionics www.dvcsim.com GPU Technology Conference 2015 Deferred Commitment Traditional Scene Generation 3 Database Generation Image

470 views • 23 slides
Twitter = Colloquial Language? Paraphrasing Macbeth Act 2, Scene 3 Using Twitter Kara Campbell,

Twitter = Colloquial Language? Paraphrasing Macbeth Act 2, Scene 3 Using Twitter Kara Campbell,

Twitter = Colloquial Language? Paraphrasing Macbeth Act 2, Scene 3 Using Twitter Kara Campbell, Roberta Coleman, Adrienne Law, Angela Li, Kevin Solis

275 views • 16 slides
Image-based Rendering Can we model and rendering this? What do we want to do with the model?

Image-based Rendering Can we model and rendering this? What do we want to do with the model?

Image-based Rendering Can we model and rendering this? What do we want to do with the model? Image-Based Modeling Images (photographs, renderings) are used to determine Scene Appearance Scene Geometry Lighting

581 views • 14 slides
Querying GIS with Animated Spatial Sketches Volker Haarslev and Michael Wessel University of

Querying GIS with Animated Spatial Sketches Volker Haarslev and Michael Wessel University of

Querying GIS with Animated Spatial Sketches Volker Haarslev and Michael Wessel University of Hamburg, Germany haarslev@informatik.uni-hamburg.de http://kogs-www.informatik.uni-hamburg.de/~haarslev/ Motivation VISCO's Language Elements

424 views • 15 slides
1 SETTING THE SCENE Main references: Ziegler A and Knig I. A Statistical approach to genetic

1 SETTING THE SCENE Main references: Ziegler A and Knig I. A Statistical approach to genetic

A tour in genetic epidemiology Chapter 7: Perspectives on family-based GWAs PERSPECTIVES ON FAMILY-BASED GWAs 1 Setting the scene

1.09k views • 88 slides
Attention-based Model and It Its Application in in Scene Text xt Recognition Baoguang Shi (

Attention-based Model and It Its Application in in Scene Text xt Recognition Baoguang Shi (

Attention-based Model and It Its Application in in Scene Text xt Recognition Baoguang Shi ( ) Huazhong University of Science and Technology March 23, 2016 Part 1: Introduction to Attention- Based Models 3/23/2016 VALSE Panel 2

656 views • 33 slides
Scene Representation How does one describe the objects in a Scene Graphs 3D scene? Scene

Scene Representation How does one describe the objects in a Scene Graphs 3D scene? Scene

Scene Representation How does one describe the objects in a Scene Graphs 3D scene? Scene Graphs State Machine Model API Scene Modeling Languages / API From the low level to the high level State Machine Model State Machine

436 views • 6 slides
Colorful Image Colorization Richard Zhang, Phillip Isola, Alexei A. Efros Presenters: Aditya

Colorful Image Colorization Richard Zhang, Phillip Isola, Alexei A. Efros Presenters: Aditya

Colorful Image Colorization Richard Zhang, Phillip Isola, Alexei A. Efros Presenters: Aditya Sankar and Bindita Chaudhuri Introduction Fully automatic approach (self-supervised deep learning algorithm) Aim: estimate the 2 unknown

706 views • 16 slides

More recommend