Sisley the Abstract Painter Mingtian Zhao Song-Chun Zhu University - PowerPoint PPT Presentation

NPAR 2010 Sisley the Abstract Painter Mingtian Zhao Song-Chun Zhu University of California, Los Angeles & Lotus Hill Institute

Motivation Wheatstack (Thaw, Sunset) Claude Monet 1890–91 “I considered that the painter had no right to paint indistinctly . . . and I noticed with surprise and confusion that the picture not only gripped me, but impressed itself ineradicably on my memory.” — Wassily Kandinsky

Perceptual Ambiguity • Indistinction, Confusion: Perceptual Ambiguity • The Mechanism of Abstract Arts [Berlyne 1971] Perceptual Ambiguity ⇒ Mental Efforts ⇒ Arousal Changes ⇒ Aesthetic Pleasures • Where does perceptual ambiguity come from? what we see = arg max P ( interpretation | image )

More Abstract Arts No. 5, 1948 Violin and Guitar Jackson Pollock Pablo Picasso

More Abstract Arts Le Mont Sainte-Victoire The Red Vineyard Paul C´ ezanne Vincent van Gogh

More Abstract Arts • They all preserve certain features and free others. • Features are marginal statistics. • Projection Pursuit [Friedman & Tukey 1974] • Semantic Fidelity vs. Uncertainty & Ambiguity • Paths of Perception

Our Approach • Interactive Image Parsing [Tu et al. 2005] ⋄ Interactive Segmentation ⋄ Hierarchical Organization ⋄ Category Labeling (optional but recommended) • Customization and Rendering ⋄ Customized Perceptual Ambiguity Levels ⋄ Stochastic Operations on Color/Shape/Texture • Computation and Control ⋄ Kernel Density Estimation ⋄ Belief Propagation ⋄ Servomechanism

Parse Tree seascape sailboat sea buildings trees sky sail hull

Numerical Measure of Perceptual Ambiguity • Assume parse tree structure is obvious, and • Perceptual ambiguity is only with categories L = ( ℓ 1 , ℓ 2 , · · · , ℓ K ) • Uncertainty in p ( L ) : Information/Shannon Entropy � H ( L ) | I = − p ( L| I ) log p ( L| I ) L

Stochastic Operations • Color ⋄ hue shift σ ∝ � ∆ h ∼ G T (0 , σ 2 , − 3 σ, 3 σ ) H , σ max = 15 ◦ • Shape ⋄ boundary pixel shift ( ∆ x , ∆ y ∼ G T ) ⋄ image warping • Texture ⋄ Painterly Rendering (adapted from [Zeng et al. 2009])

Computation & Control • Parse Tree as Markov random field (MRF) ⋄ Hammersley-Clifford � � p ( L| I ) = 1 φ i ( ℓ i ) ψ ij ( ℓ i , ℓ j ) Z i ∈V � i , j �∈E ⋄ Unary Term: Local Evidence φ i ( ℓ i ) ← p ( ℓ i | I i ) ⋄ Binary Term: Compatibility ψ ij ( ℓ i , ℓ j ) ← � f ( ℓ i , ℓ j )

Computation & Control • Local Evidence by Kernel Density Estimation Close: high-weight votes Distant: low-weight votes • The LHI Image Database

Computation & Control • Local Evidence by Kernel Density Estimation � p ( ℓ i | I i ) ∝ ϕ ( I i , J n ) 1 ( ℓ i = ℓ n ) n • The Exponential Kernel ϕ ( I i , J n ) = exp {− λ � s ( I i ) − s ( J n ) �} s : Opponent-SIFT [van de Sande et al. 2010] • The use of manually labeled categories • Can we plug in p ( ℓ i | I i ) to get p ( L| I ) and H ( L ) | I ?

Computation & Control • Problem of computing H ( L ) | I : The space of L is usually huge. | Ω L | = (# categories ) # nodes • Approximation with p ( ℓ i | I ) instead of p ( ℓ i | I i ) � H ( L ) | I ≈ w i H ( ℓ i ) | I w i ∝ size( i ) i � H ( ℓ i ) | I = − p ( ℓ i | I ) log p ( ℓ i | I ) L

Computation & Control • Belief Propagation on MRF [Yedidia et al. 2001] j m ij I i m ji p ( ℓ i | I i ) b i ( ℓ i ) i m ki m ik k

Computation & Control • Belief Propagation on MRF [Yedidia et al. 2001] ⋄ Update Messages � � p ( ℓ i | I i ) � m ij ( ℓ j ) = f ( ℓ i , ℓ j ) m ki ( ℓ i ) ℓ i k ∈ ∂ i \ j ⋄ Update Beliefs � b i ( ℓ i ) ∝ p ( ℓ i | I i ) m ji ( ℓ i ) j ∈ ∂ i ⋄ p ( ℓ i | I ) ← b i ( ℓ i ) after convergence

Computation & Control • Relative Abstract Level � i w i H ( ℓ i ) | I � � � H = ∈ [0 , 1] � � � � Ω ℓ i � i w i log • Servomechanism ⋄ If � out < � H ( t ) H ( t ) in � 2 � � � � � H ( t + 1) H ( t ) H ( t ) = in out I ⋄ If � out > � H ( t ) H ( t ) in � 2 � � � � � 1 − � 1 − � H ( t + 1) H ( t ) H ( t ) = 1 − in in out

Experimental Results � H ≈ 0 � � � H ≈ 0 . 25 H ≈ 0 . 5 H ≈ 0 . 75

Experimental Results

Experimental Results

Experimental Results Different Paths of Perception

Evaluations • Comparative Human Experiments Photographs Paintings Original Synthesized Paintings Alley Flying Bird Buildings Butterfly

Evaluations • Recognition Accuracy Photographs Original Paintings Synthesized Paintings Horizontal axis: reported categories Vertical axis: true categories Darkness of grids: frequencies

Evaluations • Response Speed ⋄ ANOVA F -test: p -value= 2 . 955 × 10 − 8 ⋄ Tukey’s HSD ∆ ¯ Group Pair t (ms) p -value Photographs vs. Original Paintings 2165 < 0 . 01 Photographs vs. Synthesized Paintings 1183 0 . 03 Original vs. Synthesized Paintings − 982 0 . 11

Summary & Future Studies • Abstract Arts Rendering by Perceptual Ambiguity Computation and Control • Convergence control • Variable parse tree structures • Mixture modeling of abstract arts More Results and Demo System: http://www.stat.ucla.edu/ ∼ mtzhao/research/sisley/