Scattering Bricks to Build Invariants Joan Bruna, Joakim Anden, - PowerPoint PPT Presentation

Stable Translation Invariants • Invariance to translations x c ( t ) = x ( t − c ) Φ ( x c ) = Φ ( x ) . ∀ c ∈ R , Φ ( x ) Not stable 0 t | τ 0 ( t ) | k x k k Φ ( x ) � Φ ( x τ ) k � sup Φ ( x τ ) 0 • Lipschitz stable to deformations x τ ( t ) = x ( t − τ ( t )) small deformations of x = small modifications of Φ ( x ) ⇒ ⇤ τ , ⇧ Φ ( x τ ) � Φ ( x ) ⇧ ⇥ C sup t | ⌃ τ ( t ) | ⇧ x ⇧ . deformation size

Stable Translation Invariants • Invariance to translations x c ( t ) = x ( t − c ) Φ ( x c ) = Φ ( x ) . ∀ c ∈ R , Φ ( x ) Not stable 0 t | τ 0 ( t ) | k x k k Φ ( x ) � Φ ( x τ ) k � sup Φ ( x τ ) Fourier invariants 0 are not stable either. • Lipschitz stable to deformations x τ ( t ) = x ( t − τ ( t )) small deformations of x = small modifications of Φ ( x ) ⇒ ⇤ τ , ⇧ Φ ( x τ ) � Φ ( x ) ⇧ ⇥ C sup t | ⌃ τ ( t ) | ⇧ x ⇧ . deformation size

Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) ψ λ ( t ) = 2 − j ψ (2 − j t ) with λ = 2 − j . • Dilated: ψ λ � ( t ) | ˆ | ˆ ψ λ ( ω ) | 2 | ˆ ψ λ � ( ω ) | 2 φ ( ω ) | 2 ψ λ ( t ) λ � λ 0 ω

Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) ψ λ ( t ) = 2 − j ψ (2 − j t ) with λ = 2 − j . • Dilated: ψ λ � ( t ) | ˆ | ˆ ψ λ ( ω ) | 2 | ˆ ψ λ � ( ω ) | 2 φ ( ω ) | 2 ψ λ ( t ) x ( ω ) ˆ λ � λ 0 ω Z x ? λ ( t ) = x ( u ) λ ( t − u ) du • Wavelet transform: ✓ x ? � ( t ) ◆ Wx = x ? λ ( t ) t, λ

Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) ψ λ ( t ) = 2 − j ψ (2 − j t ) with λ = 2 − j . • Dilated: ψ λ � ( t ) | ˆ | ˆ ψ λ ( ω ) | 2 | ˆ ψ λ � ( ω ) | 2 φ ( ω ) | 2 ψ λ ( t ) x ( ω ) ˆ λ � λ 0 ω Z x ? λ ( t ) = x ( u ) λ ( t − u ) du • Wavelet transform: ✓ x ? � ( t ) ◆ Wx = x ? λ ( t ) t, λ � Wx � 2 = � x � 2 . Unitary:

Image Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) , t = ( t 1 , t 2 ) ψ λ ( t ) = 2 − j ψ (2 − j rt ) with λ = (2 j , r ) rotated and dilated: real parts imaginary parts

Image Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) , t = ( t 1 , t 2 ) ψ λ ( t ) = 2 − j ψ (2 − j rt ) with λ = (2 j , r ) rotated and dilated: ω 2 real parts imaginary parts | ˆ ψ λ ( ω ) | 2 ω 1

Image Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) , t = ( t 1 , t 2 ) ψ λ ( t ) = 2 − j ψ (2 − j rt ) with λ = (2 j , r ) rotated and dilated: ω 2 real parts imaginary parts | ˆ ψ λ ( ω ) | 2 ω 1

Image Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) , t = ( t 1 , t 2 ) ψ λ ( t ) = 2 − j ψ (2 − j rt ) with λ = (2 j , r ) rotated and dilated: ω 2 real parts imaginary parts | ˆ ψ λ ( ω ) | 2 ω 1

Image Wavelet Transform • Complex wavelet: ψ ( t ) = ψ a ( t ) + i ψ b ( t ) , t = ( t 1 , t 2 ) ψ λ ( t ) = 2 − j ψ (2 − j rt ) with λ = (2 j , r ) rotated and dilated: ω 2 real parts imaginary parts | ˆ ψ λ ( ω ) | 2 ω 1 ✓ x ? � ( t ) ◆ • Wavelet transform: Wx = x ? λ ( t ) t, λ Unitary: � Wx � 2 = � x � 2 .

Wavelet Translation Invariance x ? λ 1 ( t ) = x ? a λ 1 ( t ) + i x ? b λ 1 ( t )

Wavelet Translation Invariance q λ 1 ( t ) | 2 + | x ? b | x ? a λ 1 ( t ) | 2 pooling | x ? λ 1 ( t ) | = • The modulus | x ? λ 1 | is a regular envelop

Wavelet Translation Invariance | x ? λ 1 | ? � ( t ) • The modulus | x ? λ 1 | is a regular envelop • The average | x ? λ 1 | ? � ( t ) is invariant to small translations relatively to the support of φ .

Wavelet Translation Invariance | x ? λ 1 | ? � ( t ) • The modulus | x ? λ 1 | is a regular envelop • The average | x ? λ 1 | ? � ( t ) is invariant to small translations relatively to the support of φ . Z φ → 1 | x ? λ 1 | ? � ( t ) = lim | x ? λ 1 ( u ) | du = k x ? λ 1 k 1

Recovering Lost Information | x ? λ 1 | | x ⇤⇥ λ 1 | ⇤ �

Recovering Lost Information | x ? λ 1 | | x ⇤⇥ λ 1 | ⇤ � • The high frequencies of | x ? λ 1 | are in wavelet coe ffi cients: ✓ ◆ | x ? λ 1 | ? � ( t ) W | x ? λ 1 | = | x ? λ 1 | ? λ 2 ( t ) t, λ 2

Recovering Lost Information | x ? λ 1 | | x ⇤⇥ λ 1 | ⇤ � • The high frequencies of | x ? λ 1 | are in wavelet coe ffi cients: ✓ ◆ | x ? λ 1 | ? � ( t ) W | x ? λ 1 | = | x ? λ 1 | ? λ 2 ( t ) t, λ 2 • Translation invariance by time averaging the amplitude: | | x ? λ 1 | ? λ 2 | ? � ( t ) ∀ � 1 , � 2 ,

Deep Convolution Network x

Deep Convolution Network x x ? � | W 1 | | x ? λ 1 |

Deep Convolution Network x x ? � | W 1 | | x ? λ 1 | | x ? λ 1 | ? � | W 2 | || x ? λ 1 | ? λ 2 |

Deep Convolution Network x x ? � | W 1 | | x ? λ 1 | | x ? λ 1 | ? � | W 2 | || x ? λ 1 | ? λ 2 | || x ? λ 1 | ? λ 2 | ? � | W 3 | ||| x ? λ 1 | ? λ 2 | ? λ 3 |

Scattering Vector Network ouptut:   x ⇤� ( u ) | x ⇤ ⇥ λ 1 | ⇤ � ( u )     || x ⇤⇥ λ 1 | ⇤ ⇥ λ 2 | ⇤ � ( u ) Sx =     ||| x ⇤⇥ λ 2 | ⇤ ⇥ λ 2 | ⇤ ⇥ λ 3 | ⇤ � ( u )   ... u, λ 1 , λ 2 , λ 3 ,...

Amplitude Modulation log( λ 1 ) | x ? λ 1 ( t ) | First − order windowed scattering (small scale) log( ! 1 ) t t log( λ 1 ) | x ? λ 1 | ? � ( t ) First − order windowed scattering (large scale) log( ! 1 ) t t log( λ 2 ) || x ? λ 1 | ? λ 2 | ? � ( t ) for � 1 = log(1977) Second − order windowed scattering (large scale) Band #75 log( ! 2 ) 18 Hz t t

Amplitude Modulation log( λ 1 ) | x ? λ 1 ( t ) | First − order windowed scattering (small scale) log( ! 1 ) t t log( λ 1 ) | x ? λ 1 | ? � ( t ) First − order windowed scattering (large scale) log( ! 1 ) t t log( λ 2 ) || x ? λ 1 | ? λ 2 | ? � ( t ) for � 1 = log(1977) Second − order windowed scattering (large scale) Band #75 log( ! 2 ) 18 Hz t t

Amplitude Modulation log( λ 1 ) | x ? λ 1 ( t ) | First − order windowed scattering (small scale) log( ! 1 ) t t log( λ 1 ) | x ? λ 1 | ? � ( t ) First − order windowed scattering (large scale) log( ! 1 ) t t log( λ 2 ) || x ? λ 1 | ? λ 2 | ? � ( t ) for � 1 = log(1977) Second − order windowed scattering (large scale) Band #75 log( ! 2 ) 18 Hz t t

Amplitude Modulation log( λ 1 ) | x ? λ 1 ( t ) | First − order windowed scattering (small scale) 1977 Hz log( ! 1 ) t t log( λ 1 ) | x ? λ 1 | ? � ( t ) First − order windowed scattering (large scale) log( ! 1 ) t t log( λ 2 ) || x ? λ 1 | ? λ 2 | ? � ( t ) for � 1 = log(1977) Second − order windowed scattering (large scale) Band #75 log( ! 2 ) 18 Hz t t

Cascade of Contractions x | W 1 | | W 2 | | W 3 |

Cascade of Contractions x x ? � | W 1 | | x ? λ 1 | ? � | W 2 | || x ? λ 1 | ? λ 2 | ? � | W 3 |

Cascade of Contractions x x ? � | W 1 | | x ? λ 1 | ? � | W 2 | || x ? λ 1 | ? λ 2 | ? � | W 3 | • Cascade of contractive operators ⇤ | W k | x � | W k | x 0 ⇤ ⇥ ⇤ x � x 0 ⇤

Cascade of Contractions x x ? � | W 1 | | x ? λ 1 | ? � | W 2 | || x ? λ 1 | ? λ 2 | ? � | W 3 | • Cascade of contractive operators ⇤ | W k | x � | W k | x 0 ⇤ ⇥ ⇤ x � x 0 ⇤ with � | W k | x � = � x � .

Scattering Properties   x ⇤� ( u ) | x ⇤ ⇥ λ 1 | ⇤ � ( u )     Sx = || x ⇤⇥ λ 1 | ⇤ ⇥ λ 2 | ⇤ � ( u )     ||| x ⇤⇥ λ 2 | ⇤ ⇥ λ 2 | ⇤ ⇥ λ 3 | ⇤ � ( u )   ... u, λ 1 , λ 2 , λ 3 ,... Theorem : For appropriate wavelets, a scattering is contractive k Sx � Sy k  k x � y k preserves norms k Sx k = k x k

Scattering Properties   x ⇤� ( u ) | x ⇤ ⇥ λ 1 | ⇤ � ( u )     Sx = || x ⇤⇥ λ 1 | ⇤ ⇥ λ 2 | ⇤ � ( u )     ||| x ⇤⇥ λ 2 | ⇤ ⇥ λ 2 | ⇤ ⇥ λ 3 | ⇤ � ( u )   ... u, λ 1 , λ 2 , λ 3 ,... Theorem : For appropriate wavelets, a scattering is contractive k Sx � Sy k  k x � y k preserves norms k Sx k = k x k stable to deformations x τ ( t ) = x ( t − τ ( t )) k Sx � Sx τ k  C sup t | r τ ( t ) | k x k

Scattering Properties   x ⇤� ( u ) | x ⇤ ⇥ λ 1 | ⇤ � ( u )     Sx = || x ⇤⇥ λ 1 | ⇤ ⇥ λ 2 | ⇤ � ( u )     ||| x ⇤⇥ λ 2 | ⇤ ⇥ λ 2 | ⇤ ⇥ λ 3 | ⇤ � ( u )   ... u, λ 1 , λ 2 , λ 3 ,... Theorem : For appropriate wavelets, a scattering is contractive k Sx � Sy k  k x � y k preserves norms k Sx k = k x k stable to deformations x τ ( t ) = x ( t − τ ( t )) k Sx � Sx τ k  C sup t | r τ ( t ) | k x k ⇒ linear discriminative classification from Φ x = Sx

Linearized Classification Joan Bruna computed with PCA. X 2 X 1 MNIST data basis:

Linearized Classification Joan Bruna • Each class X k is represented by a scattering centroid E ( SX k ) A ffi ne space model A k = E ( SX k ) + V k . computed with PCA. E ( SX 2 ) X 2 X 1 E ( SX 1 ) A 2 S A 1 MNIST data basis:

Linearized Classification Joan Bruna • Each class X k is represented by a scattering centroid E ( SX k ) A ffi ne space model A k = E ( SX k ) + V k . computed with PCA. E ( SX 2 ) X 2 X 1 E ( SX 1 ) A 2 S A 1 x MNIST data basis:

Linearized Classification Joan Bruna • Each class X k is represented by a scattering centroid E ( SX k ) A ffi ne space model A k = E ( SX k ) + V k . computed with PCA. E ( SX 2 ) X 2 X 1 E ( SX 1 ) A 2 Sx S A 1 x MNIST data basis:

Linearized Classification Joan Bruna • Each class X k is represented by a scattering centroid E ( SX k ) A ffi ne space model A k = E ( SX k ) + V k . computed with PCA. E ( SX 2 ) X 2 X 1 E ( SX 1 ) A 2 Sx S A 1 x MNIST data basis:

Scattering Moments The scattering transform of a stationary process X ( t )   X ? � ( t ) | X ? λ 1 | ? � ( t )     SX ( t ) = || X ? λ 1 | ? λ 2 | ? � ( t )     ||| X ? λ 2 | ? λ 2 | ? λ 3 | ? � ( t )   ... λ 1 , λ 2 , λ 3 ,... is an estimator of the expected scattering of X ( t )   E ( X ) E ( | X ? λ 1 | )     SX = E ( || X ? λ 1 | ? λ 2 | )     E ( ||| X ? λ 2 | ? λ 2 | ? λ 3 | )   ... λ 1 , λ 2 , λ 3 ,...

Textures with Same Spectrum x ( t ): stationary process Fourier Textures Power Spectrum x ( t ) ω 1 ω 2 ω 1 ω 2

Textures with Same Spectrum x ( t ): stationary process Fourier Wavelet Scattering Textures Power Spectrum x ( t ) | x ? λ 1 | ? � ω 1 ω 2 ω 1 ω 2 window size = image size

Textures with Same Spectrum x ( t ): stationary process Fourier Wavelet Scattering Textures Power Spectrum x ( t ) | x ? λ 1 | ? � || x ? λ 1 | ? λ 2 | ? � ω 1 ω 2 ω 1 ω 2 window size = image size

Sounds with Same Spectrum Fourier X : stationary process Spectrum log( λ 1 ) ω | x ⇥� λ 1 | ( t ) J. McDermott First − order windowed scattering (small scale) log( � 1 ) t log( λ 1 ) | x ? λ 1 | ? � ( t ) 2 s window First − order windowed scattering (large scale) log( � 1 ) log( λ 2 ) t || x ? λ 1 | ? λ 2 | ? � ( t ) for � 1 = 2000 Second − order windowed scattering (large scale) Band #51 log( � 2 ) t t

Sounds with Same Spectrum Fourier X : stationary process Spectrum log( λ 1 ) ω | x ⇥� λ 1 | ( t ) J. McDermott First − order windowed scattering (small scale) log( � 1 ) t log( λ 1 ) | x ? λ 1 | ? � ( t ) 2 s window First − order windowed scattering (large scale) log( � 1 ) log( λ 2 ) t || x ? λ 1 | ? λ 2 | ? � ( t ) for � 1 = 2000 Second − order windowed scattering (large scale) Band #51 log( � 2 ) t t

Representation of Random Processes • An expected scattering is a non-complete representation   E ( X ) = E ( U 0 X ) E ( | X ⇥ � λ 1 | ) = E ( U 1 X )     SX = E ( || X ⇥ � λ 1 | ⇥ � λ 2 | ) = E ( U 2 X )     E ( ||| X ⇥ � λ 2 | ⇥ � λ 2 | ⇥ � λ 3 | ) = E ( U 3 X )   ... λ 1 , λ 2 , λ 3 ,... Theorem (Boltzmann) The distribution p ( x ) which satisfies Z R N U m x p ( x ) dx = E ( U m X ) R and maximizes the entropy − p ( x ) log p ( x ) dx ∞ p ( x ) = 1 ⇣ ⌘ X Z exp λ m . U m x can be written: m =1

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party

Synthesis from Second Order Joakim Anden Joan Bruna • Maximum entropy estimation of X ( t ) : - Gaussian model from N power spectrum coe ffi cients. - Scattering model from (log 2 N ) 2 / 2 1st & 2nd orders. J. McDermott textures Original Gaussian Model Scattering Moments JackHammer Water Applause Paper Cocktail Party Not good for everything: learn from mistakes.

Classification of Textures J. Bruna CUREt database 61 classes � Texte Supervised Linear Sx x y Classifier: PCA/SVM Training Fourier Histogr. Scattering per class Spectr. Features 46 1% 1% 0 . 2 %

Wavelet Transform on a Group Laurent Sifre • Roto-translation group G = { g = ( r, t ) ∈ SO (2) × R 2 } ( r, t ) . x ( u ) = x ( r − 1 ( u − t ))

Wavelet Transform on a Group Laurent Sifre • Roto-translation group G = { g = ( r, t ) ∈ SO (2) × R 2 } ( r, t ) . x ( u ) = x ( r − 1 ( u − t )) Z 0 � 1 g ) dg 0 X ( g 0 ) φ ( g • Averaging on G : X ~ φ ( g ) = G