Eurographics 2011 John Collomosse and Jan Eric Kyprianidis Centre - PowerPoint PPT Presentation

Artistic Stylization of Images and Video Eurographics 2011 John Collomosse and Jan Eric Kyprianidis Centre for Vision Speech and Signal Processing (CVSSP) University of Surrey, United Kingdom Hasso-Plattner-Institut, University of Potsdam, Germany http://kahlan.eps.surrey.ac.uk/EG2011

Artistic Stylization Resources  Texts  Web Bibliographies Strothotte & Schlechtweg http://video3d.ims.tuwien.ac.at/~stathis/nprl ISBN: 1558607870 ib/index.php Gooch & Gooch http://isgwww.cs.uni- ISBN: 1568811330 magdeburg.de/~stefans/npr/nprpapers.html http://www.red3d.com/cwr/npr/ (dated) Romero & Machado ISBN: 3540728767  Tutorials  Main Publication Forums NPAR (Symposium on Non-photorealistic Animation) SIGGRAPH 99 (Green et al.) – 2D/3D NPR Held in Annecy even years, at SIGGRAPH odd years. SIGGRAPH 02 (Hertzmann) – 2D NPR SIGGRAPH 03 (Sousa et al.) – 2D/3D NPR IEEE Trans Visualization and Comp. Graphics ( TVCG ) IEEE Computer Graphics and Applications ( CG&A ) Eurographics 05,06 and... Eurographics and Computer Graphics Forum SIGGRAPH 06 (Sousa et al) – 3D NPR SIGGRAPH, SIGGRAPH Asia and ACM ToG SIGGRAPH 10 (McGuire) – 3D NPR for Games EG Symposium on Rendering ( EGSR ) ACM/EG Symposium on Computer Animation ( EGSA ) Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 2

Artistic Stylization Non-Photorealistic Rendering (NPR) Coined by Salesin et al., 1994 Stylized Rendering Aesthetic Rendering Artistic Stylization Artistic Rendering Anatomy of the Human Body H. Gray, 1918 Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 3

Motivation Artistic Stylization Communication  Why? Comprehension Aesthetics Animation Visualization  Artistic Stylization can  Simplify and structure the presentation of content  Selectively guide attention to salient areas of content and influence perception  Learn and emulate artistic styles  Provide assistive tools to artists and animators (not replace the artist!)  Help us to design effective visual interfaces Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 4 Tatzgurn et al. NPAR 2010

Motivation Artistic Stylization  Rendering real images/video footage in to pseudo-artistic styles  Convergence of Computer Vision, Graphics (and HCI) Analysis Render Representation Image Processing / Vision Computer Graphics User Interaction  Visual analysis enables new graphics. Graphical needs motivate new vision. Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 5

Chronology Late 1980s Video painting Perceptual UI & Space-time Anisotropy / filters Advances in media segmentation video emulation H. Winnemoeller [SIGGRAPH 06] J. Wang [SIGGRAPH 04] J. Kyprianidis [TPCG 08] D. Strassman (SIGGRAPH 86) P. Litwinowicz (SIGGRAPH 97) D. Decarlo [SIGGRAPH 02] J. Collomosse [TVCG 05] 1990 1997 1998 2000 2002 2005 2006 2010 Semi-automatic Fully automatic Automatic NPAR 2010 painting systems painting perceptual Grand challenges User evaluation A. Hertzmann (SIGGRAPH 98) J. Collomosse [EvoMUSART 05] P. Haeberli (SIGGRAPH 90) Treveatt/Chen [EGUK 97] P. Litwinowicz [SIGGRAPH 97] T. Isenberg [NPAR 06] Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 6

Interactions with Vision Late 1980s Video painting Perceptual UI & Space-time Anisotropy / filters Rendering process is guided by... Advances in media segmentation video User emulation subconscious interaction Low-level image Higher level computer H. Winnemoeller [SIGGRAPH 06] User concious interaction Direct Anisotropic filtering J. Wang [SIGGRAPH 04] J. Kyprianidis [TPCG 08] processing vision D. Strassman (SIGGRAPH 86) P. Litwinowicz (SIGGRAPH 97) D. Decarlo [SIGGRAPH 02] J. Collomosse [TVCG 05] 1990 1997 1998 2000 2002 2005 2006 2010 Semi-automatic Fully automatic Automatic NPAR 2010 painting systems painting perceptual Grand challenges User evaluation A. Hertzmann (SIGGRAPH 98) J. Collomosse [EvoMUSART 05] P. Haeberli (SIGGRAPH 90) Treveatt/Chen [EGUK 97] P. Litwinowicz [SIGGRAPH 97] T. Isenberg [NPAR 06] Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 7

Tutorial Structure Part III: Anisotropy and Filtering Late 1980s Video painting Perceptual UI & Space-time Anisotropy / filters Rendering process is guided by... Advances in media segmentation video (70 min) User emulation subconscious interaction Low-level image Higher level computer H. Winnemoeller [SIGGRAPH 06] User concious interaction Direct Anisotropic filtering J. Wang [SIGGRAPH 04] J. Kyprianidis [TPCG 08] processing vision D. Strassman (SIGGRAPH 86) P. Litwinowicz (SIGGRAPH 97) D. Decarlo [SIGGRAPH 02] J. Collomosse [TVCG 05] BREAK! 1990 1997 1998 2000 2002 2005 2006 2010 Semi-automatic Fully automatic Automatic Part IV: Future NPAR 2010 painting systems painting perceptual Part I: Classical algorithms (30 min) Challenges (20 min) Grand challenges User evaluation Part II: Vision for Stylisation (60 min) A. Hertzmann (SIGGRAPH 98) J. Collomosse [EvoMUSART 05] P. Haeberli (SIGGRAPH 90) Treveatt/Chen [EGUK 97] P. Litwinowicz [SIGGRAPH 97] T. Isenberg [NPAR 06] Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 8

Artistic Stylization of Images and Video Part I – Classical Algorithms / Stroke Based Rendering Eurographics 2011 John Collomosse Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey, United Kingdom

References  Paint by numbers: Abstract image representations P. Haeberli, SIGGRAPH 1990  Almost Automatic Computer Painting P. Haggerty, IEEE CG & A 1991  Orientable Textures for Image based Pen-and-Ink Illustration D. Salisbury et al., SIGGRAPH 1997  Processing images and video for an impressionist effect P. Litwinowicz, SIGGRAPH 1997  Statistical techniques for the automated synthesis of non-photorealistic images S. Treavett and M. Chen, Eurographics UK 1997.  Automatic Painting based on Local Source Image Approximation Shiraishi and Yamaguchi, NPAR 2000.  Painterly Rendering with Curved Strokes of Multiple Sizes A. Hertzmann, SIGGRAPH 1998.  Paint by Relaxation A. Hertzmann, CGI 2001  Fast Paint Texture A. Hertzmann, NPAR 2002 Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 10

Time to Palette  Early painting systems lacked appropriate UI for rich digital painting Xerox superpaint (1980s) Windows Vista Paint 2007 Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 11

Paint by numbers: Abstract Image Representations Haeberli. (1990)  Stroke based rendering (SBR)  Painting is a manually ordered list of strokes, placed interactively.  Stroke attributes sampled from the photo. same geometry Photo Canvas stroke click Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 12

Paint by numbers: Abstract Image Representations Haeberli. (1990)  Stroke colour and orientation are sampled from the source image  Stroke order and scale are user-selected  Addition of RGB noise generates an impressionist effect Photo credit: Haeberli ’90. Sobel Edge detection -1 -2 -1 0 0 0 1 2 1 Edge Mag. Edge orient. Paintings with / without orientable strokes Orientation Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 13

Paint by numbers: Abstract Image Representations Haeberli. (1990)  More stylised orientation effects with a manually defined orientation field Orientation field Painterly Rendering Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 14 Photo credit: Haeberl ’90.

Paint by numbers: Abstract Image Representations Haeberli. (1990) All tutorial code at http://kahlan.eps.surrey.ac.uk/EG2011 Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 15

Orientable Textures for Image-based Pen-and-Ink Illustration Salisbury et al. (1997)  Very similar system for pen-and-ink rendering of photos  User defined orientation field. • Regions manually drawn and marked up with orientation  Stroke (line) placement automatic. Strokes clipped to keep within regions. Manually defining regions of the orientation field Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 16 Photo credit: Salisbury’97.

Almost automatic computer painting Haggerty (1991)  Stroke colour and orientation are sampled from the source image  Stroke order and scale are user-selected  Scale sampled from Sobel edge magnitude Fully automated  Regularly place strokes. Order of strokes randomly generated Photo credit: Haeberli ’90. Loss of detail in important regions Interactive (Haeberli) Pseudo-random (as Haggerty) Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 17

Processing Images & Video for Impressionist Effect Litwinowicz (1997) Sobel edge direction No clipping Clipping Image edge seed Stroke grows from seed point bidirectionaly until edge pixels encountered Eurographics 2011 • Artistic Stylization of Images and Video • Part I • 18 Photo credit: Litwinowicz ‘97