An ideal observer model for grouping and contour integration in - PowerPoint PPT Presentation

Departement of Systems and Computational Biology Albert Einstein College of Medicine An ideal observer model for grouping and contour integration in natural images Jonathan Vacher With: Ruben Coen-Cagli (Albert Einstein College of Medicine, New-York) and Pascal Mamassian (LSP, ´ Ecole Normale Sup´ erieure, Paris). ECVP 29/08/2019 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 1 /13

Image Segmentation vs Visual Segmentation jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 2 /13

Image Segmentation vs Visual Segmentation Great progress with deep learning: ◮ mostly supervised learning ( ∼ top-down approach) ◮ smart architecture but different from biological vi- sion ◮ performance oriented ◮ work as a black box jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 2 /13

Image Segmentation vs Visual Segmentation Great progress with deep learning: ◮ mostly supervised learning ( ∼ top-down approach) ◮ smart architecture but different from biological vi- sion ◮ performance oriented ◮ work as a black box Visual segmentation is more involved ! ◮ variable but consistent across humans ◮ top-down + bottom-up processing Our goal is to craft an open box model with all the ingredient of vision ! jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 2 /13

Grouping and contours integration: a basis for visual segmentation ? jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 3 /13

Grouping and contours integration: a basis for visual segmentation ? Contour perception (Field et al. 1993) jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 3 /13

Grouping and contours integration: a basis for visual segmentation ? Contour perception (Field et al. 1993) Texture perception (Landy et al. 2001) jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 3 /13

Grouping and contours integration: a basis for visual segmentation ? Contour perception (Field et al. 1993) Texture perception (Landy et al. 2001) Artificial stimuli ! jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 3 /13

Grouping and contours integration: a basis for visual segmentation ? Contour perception (Field et al. 1993) Texture perception (Landy et al. 2001) Artificial stimuli ! Tractable models and well-controlled experiments . . . jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 3 /13

Grouping and contours integration: a basis for visual segmentation ? Contour perception (Field et al. 1993) Texture perception (Landy et al. 2001) Artificial stimuli ! Tractable models and well-controlled experiments . . . but how to generalize to natural images ? jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 3 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! ◮ To guide future model driven psychophysical experiments (ongoing work, not presented here) jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! ◮ To guide future model driven psychophysical experiments (ongoing work, not presented here) Several constraints: jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! ◮ To guide future model driven psychophysical experiments (ongoing work, not presented here) Several constraints: ◮ Image statistics jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! ◮ To guide future model driven psychophysical experiments (ongoing work, not presented here) Several constraints: ◮ Image statistics ◮ Cortical features jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Toward an ideal observer model for visual segmentation An ideal observer for visual segmentation of natural images ! ◮ To guide future model driven psychophysical experiments (ongoing work, not presented here) Several constraints: ◮ Vision psychophysics ◮ Image statistics ◮ Cortical features jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 4 /13

Representation and non-Gaussian statistics of natural images How are images represented ? jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Representation and non-Gaussian statistics of natural images How are images represented ? ⇒ decomposition in a wavelet basis (receptive fields) (Bell et al. 1997; Olshausen et al. 1996) ◮ X = ( X 1 , . . . , X n ) T = ( � w 1 , I � , . . . , � w n , I � ) T ( w k ) k = I = jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Representation and non-Gaussian statistics of natural images How are images represented ? ⇒ decomposition in a wavelet basis (receptive fields) (Bell et al. 1997; Olshausen et al. 1996) ◮ X = ( X 1 , . . . , X n ) T = ( � w 1 , I � , . . . , � w n , I � ) T What are the coefficient statistics ? jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Representation and non-Gaussian statistics of natural images How are images represented ? ⇒ decomposition in a wavelet basis (receptive fields) (Bell et al. 1997; Olshausen et al. 1996) ◮ X = ( X 1 , . . . , X n ) T = ( � w 1 , I � , . . . , � w n , I � ) T What are the coefficient statistics ? Non-Gaussian ! (Wainwright et al. 2000) Histogram of x 1 0 . 3 0 . 2 0 . 1 0 . 0 − 5 0 5 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Representation and non-Gaussian statistics of natural images How are images represented ? ⇒ decomposition in a wavelet basis (receptive fields) (Bell et al. 1997; Olshausen et al. 1996) ◮ X = ( X 1 , . . . , X n ) T = ( � w 1 , I � , . . . , � w n , I � ) T What are the coefficient statistics ? Non-Gaussian ! (Wainwright et al. 2000) Definition (Gaussian Scale Mixture) Gaussian vector of visual features (G ∼ N (0 , Σ) ) × Contrast between features (Z ∼ L ( ν ) ) X = ZG . Density of X 1 knowing Z = z Histogram of x 1 0 . 4 0 . 3 Density Z = z ր 0 . 2 0 . 2 0 . 1 0 . 0 0 . 0 − 5 − 5 0 5 0 5 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Representation and non-Gaussian statistics of natural images How are images represented ? ⇒ decomposition in a wavelet basis (receptive fields) (Bell et al. 1997; Olshausen et al. 1996) ◮ X = ( X 1 , . . . , X n ) T = ( � w 1 , I � , . . . , � w n , I � ) T What are the coefficient statistics ? Non-Gaussian ! (Wainwright et al. 2000) Definition (Gaussian Scale Mixture) Gaussian vector of visual features (G ∼ N (0 , Σ) ) × Contrast between features (Z ∼ L ( ν ) ) X = ZG . Density of X 1 knowing Z = z Z : random variable 0 . 4 0 . 3 0 . 3 Density Z = z ր 0 . 2 0 . 2 0 . 2 0 . 1 0 . 1 0 . 0 0 . 0 0 . 0 − 5 − 5 − 5 0 5 0 0 5 5 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Representation and non-Gaussian statistics of natural images How are images represented ? ⇒ decomposition in a wavelet basis (receptive fields) (Bell et al. 1997; Olshausen et al. 1996) ◮ X = ( X 1 , . . . , X n ) T = ( � w 1 , I � , . . . , � w n , I � ) T What are the coefficient statistics ? Non-Gaussian ! (Wainwright et al. 2000) Definition (Gaussian Scale Mixture) Gaussian vector of visual features (G ∼ N (0 , Σ) ) × Contrast between features (Z ∼ L ( ν ) ) X = ZG . Density of X 1 knowing Z = z Z : random variable 0 . 4 ◮ Heavy-tailed distribu- 0 . 3 0 . 3 Density tions . . . Z = z ր 0 . 2 0 . 2 0 . 2 ◮ and non-linear depen- 0 . 1 0 . 1 dencies ! 0 . 0 0 . 0 0 . 0 − 5 − 5 − 5 0 5 0 0 5 5 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 5 /13

Gaussian Scale Mixture explains surround modulation in V1 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 6 /13

Gaussian Scale Mixture explains surround modulation in V1 Taken from Coen-Cagli, Dayan, et al. 2012 jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 6 /13

Gaussian Scale Mixture explains surround modulation in V1 Taken from Coen-Cagli, Dayan, et al. 2012 GSM ⇒ normalization: X = ( X ( c ) , X ( s ) ) , G = ( G ( c ) , G ( s ) ) X ( c ) G ( c ) ∝ � k w k X 2 ν + � k jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 6 /13

Gaussian Scale Mixture explains surround modulation in V1 Taken from Coen-Cagli, Dayan, et al. 2012 Interpretation: GSM ⇒ normalization: ◮ G : vector of neurons responses (Coen-Cagli X = ( X ( c ) , X ( s ) ) , G = ( G ( c ) , G ( s ) ) and Schwartz 2013; Orb´ an et al. 2016) X ( c ) ◮ Z : normalization (canonical across the brain G ( c ) ∝ � Carandini et al. 2012) k w k X 2 ν + � k jonathan.vacher@einstein.yu.edu https://jonathanvacher.github.io 6 /13