Marta Favali Thesis director: Alessandro Sarti Thesis co-director: - PowerPoint PPT Presentation

École Doctorale Cerveau-Cognition Comportement Doctorate in Theoretical Neuroscience CAMS (CNRS-EHESS) Doctorate in Mathematics Department of Mathematics (University of Bologna) Marta Favali Thesis director: Alessandro Sarti Thesis co-director: Giovanna Citti Title of the project: Formal models of visual perception based on cortical architectures. 1

Objectives • Mathematical models of the primary visual cortex • Mathematical models of visual perception Methods and development of work : • The neurogeometry of the visual cortex • Models of cortical connectivity, with different stochastic kernels • Spectral analysis of connectivity matrix • Simulations (Kanizsa figures and retinal images). 2

Individuation of perceptual units: the association fields Field et al, 1993 3

Mathematical models of the functional architecture of V1 — J.J. Koenderink, A.J van Doorn, Representation of local geometry in the visual system. , Biol. Cybernet. 55,367-375, 1987. — J. Petitot, The neurogeometry of pinwheels as a sub-Riemannian contact structure , in Journal Physiol, Pages 97(2-3):265-309, 2003. — G. Citti, A.Sarti, A cortical based model of perceptual completion in the roto-translation space , Journal of Mathematical Imaging and Vision, 24(3):307-326, 2006 — S.W. Zucker, Differential geometry from the Frenet point of view: boundary detection, stereo, texture and color ., In: Paragios, N., Chen, Y., Faugeras, O. (eds.) Handbook of Mathematical Models in Computer Vision, pp. 357-373. Springer, US, 2006. — A.Sarti, G. Citti, J. Petitot , The symplectic structure of the primary visual cortex , Biol. Cybern. 98, 33-48, 2008. — R. Duits, E.M. Franken, Left invariant parabolic evolution equations on SE(2) and contour enhance- ment via invertible orientation scores, part I: Linear left-invariant diffusion equations on SE(2) , Q. Appl. Math. 68, 255-292, 2010. 4

The neurogeometry of V1 Hypercolumnar structure Hubel-Wiesel, 1965 Receptive profile of a simple cell and its representation as a even-symmetric and odd- symmetric Gabor filters. y 2 ) [ − ( ! x 2 + ! + i ! y 1 σ ] σ 2 ϕ ( x , y , θ ) = 2 πσ 2 e ! x = x cos( θ ) + y sin( θ ) ! y = − x sin( θ ) + y cos( θ ) Daugman, 1985 5

• Simple cells are modeled with Gabor filters and represent a group: ! ! ! X 3 = ( − sin θ ,cos θ ,0) X 1 = (cos θ ,sin θ ,0) X 2 = (0,0,1) 6

Output of simple cells: dx ' dy ' ϕ x , y , θ ( x ' , y ' ) I ( x ' , y ' ) ∫ h ( x , y , θ ) = Lifting: nonmaximal suppression max θ ( h ( x , y , θ )) = h ( x , y , θ ) Sarti Citti, 2006 7

! ! X 1 = (cos θ ,sin θ ,0) X 2 = (0,0,1) X 1 = cos( θ ) ∂ x + sin( θ ) ∂ y X 2 = ∂ θ X 3 = [ X 2 , X 1 ] = − sin( θ ) ∂ x + cos( θ ) ∂ y ! ! ! generator of the tangent space. X 1 , X 2 , X 3 Citti-Sarti, 2006 8

Differential model of Citti-Sarti X 1 = cos( θ ) ∂ x + sin( θ ) ∂ y X 2 = ∂ θ ! ! γ ' ( t ) = X 1 ( γ ) + k X 2 ( γ ) γ (0) = ( x 0 , y 0 , θ 0 ) Citti-Sarti, 2006 9

The Fokker Planck operator has a nonnegative fundamental solution Γ 1 that satisfies: X 1 Γ 1 (( x , y , θ ),( x ' , y ' , θ ' )) + σ 2 X 22 Γ 1 (( x , y , θ ),( x ' , y ' , θ ' )) = δ ( x , y , θ ) 3 : ω 1 2 : 3 : 3 0 25 15 10 5 30 50 y x The Sub-Riemannian Laplacian operator has a nonnegative fundamental solution that satisfies: Γ 2 σ 2 1 X 11 Γ 2 (( x , y , θ ),( x ' , y ' , θ ' )) + σ 2 2 X 22 Γ 2 (( x , y , θ ),( x ' , y ' , θ ' )) = δ ( x , y , θ ) ω 2 Sanguinetti Citti Sarti, 2008 10

The connectivity map measured by Bosking in tree shrew: Bosking et al, 1997 Maximum values along dimension of the connectivity kernels associated to the θ fundamental solution of a FP (left) and SRL equations (right). 11

Affinity Matrix Propagation of to close cells: h ( x i , y i , θ i ) 20 N ∑ ω (( x i , y i , θ i ),( x j , y j , θ j )) h ( x j , y j , θ j ) 40 j = 1 A i , j = ω (( x i , y i , θ i ),( x j , y j , θ j )) 60 80 20 40 60 80 Individuation of perceptual units: Kanizsa figure 12

Numerical algorithm 1. Define the affinity matrix from the approximated connectivity A i , j kernel. 2. Solve the eigenvalue problem , where the order of i is such A i , j u i = λ i u i that is decreasing. λ i 3. Find and represent on the segments the eigenvector associated to its u 1 largest eigenvalue. 13

The affinity matrix is updated First eigenvector of the affinity matrix, removing the detected perceptual using the fundamental solutions of FP unit; the first eigenvector of the and SRL equations. new matrix is visualized. 14

(a) In red the first eigenvectors of the affinity matrix using both connectivity kernel. (b) (c) F., Citti, Sarti: “Local and global gestalt laws: A neurally based spectral approach”, submitted to (d) Neural Computation, 2015. 15

Individuation of perceptual units: retinal images Analyzed problems: bifurcation crossing disconnected vessels In collaboration with TU/e 16

— In presence of an input stimuli, the visual cortex codifies the features of position and orientation. Image patch: crossing Oriented segments Lifted image 3 : 2 : 3 : 3 1 0 11 21 21 11 1 y x — The proposed method models the connectivity as the fundamental solution of the Fokker-Planck equation. 17

— In order to measure the distances between intensities we introduce the kernel : ω 3 2( f i − f j ( − 1 )) 2 σ 2 ω 3 ( f i , f j ) = e — The final connectivity kernel can be written as the product of the two components: ω (( x i , y i , θ i , f i ),( x j , y j , θ j , f j )) = ω 1 (( x i , y i , θ i ),( x j , y j , θ j )) ω 3 ( f i , f j ) — Starting from that connectivity kernel it is possible to extract perceptual units from images by means of spectral analysis of suitable affinity matrix: A ij = ω (( x i , y i , θ i , f i ),( x j , y j , θ j , f j )) 20 40 60 80 18 20 40 60 80

Normalized Spectral Clustering 1. After defining the affinity matrix from the connectivity kernel A − 1 2. We evaluate the normalized affinity matrix where is the diagonal P = D A D degree matrix having elements: n ∑ d i = a i , j j = 1 3. Solve the eigenvalue problem: Pu m = λ m u m 4. Define the thresholds and evaluate the largest integer K such that λ τ ε , τ m > 1 − ε for m = 1,..., K Eigenvalues 1 0.8 0.6 0.4 0.2 Shi Malik, 2000 0 5 10 15 Meila Shi, 2001 19

Normalized Spectral Clustering 5. Define the clusters from the eigenvector u K 6. Find and remove the clusters that contain less than a minimum cluster size elements. Perceptual units Image patch 20

F., Abbasi, Romeny, Sarti: “Analysis of Vessel Connectivities in Retinal Images by Cortically Inspired Spectral 21 Clustering”, submitted to JMIV, 2015.

Conclusion — We have presented a neurally based model for figure-ground segmentation using spectral methods. • Different connectivity kernels are compatible with the functional architecture of V1, we have compared their properties and modelled them as fundamental solution of Fokker Planck, Sub-Riemannian Laplacian equations. • With this model we have identified perceptual units of different Kanizsa figures and retinal images. • We have shown how this can be considered a good quantitative model for the constitution of perceptual units. 22

Future work • Our method represents some limitations at blood vessels with high curvature. These structures will be analyzed in an higher dimensional group (Engel group) adding other features. • Other images containing tree structures will be analyzed. • We will compare the results obtained with this model with functional fMRI data, that represent measurements of cortical neural activity. 23

Thanks for your attention 24