Microscopic Super-Resolution in the lateral plane by sparse - PowerPoint PPT Presentation

Microscopic Super-Resolution in the lateral plane by sparse regularization in the covariance domain Vasiliki Stergiopoulou Laure Blanc-Féraud (CNRS), Luca Calatroni (CNRS), Sebastien Schaub (IMEV) and Henrique de Morais Goulart (ENSEEIHT) Thematic Day: "Non-convex sparse optimization" Friday, October 9

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work 2 / 23

Diffraction Limited Resolution of Conventional Fluorescence Microscopy ◮ Spatial resolution is limited by light diffraction phenomena ◮ Smallest resolvable distance (Rayleigh Criterion): d = 0 . 61 λ ( ≈ 200 nm ) NA λ : emission wavelength, NA : Numerical Aperture. 3 / 23

Diffraction Limited Resolution of Conventional Fluorescence Microscopy ◮ Spatial resolution is limited by light diffraction phenomena ◮ Smallest resolvable distance (Rayleigh Criterion): d = 0 . 61 λ ( ≈ 200 nm ) NA λ : emission wavelength, NA : Numerical Aperture. 3 / 23

State-of-the art methods for SR STED (Hell and Wichmann, 1994) STimulation-Emission-Depletion ◮ Depletes some of the excited fluorophores and limits the area of illumination SMLM (Betzig, Zhuang, Hess, 2006) Single Molecule Localization Microscopy ◮ Activation of few molecules, imaging and localization http://zeiss-campus. magnet.fsu.edu/ 4 / 23

State-of-the art methods for SR STED (Hell and Wichmann, 1994) STimulation-Emission-Depletion ◮ Depletes some of the excited fluorophores and limits the area of illumination ◮ Requires special equipment, potentially harmful excitation levels SMLM (Betzig, Zhuang, Hess, 2006) Single Molecule Localization Microscopy ◮ Activation of few molecules, imaging and localization ◮ Time consuming acquisition, poor temporal resolution, potentially harmful excitation levels http://zeiss-campus. magnet.fsu.edu/ 4 / 23

State-of-the art methods for SR STED (Hell and Wichmann, 1994) STimulation-Emission-Depletion ◮ Depletes some of the excited fluorophores and limits the area of illumination ◮ Requires special equipment, potentially harmful excitation levels SMLM (Betzig, Zhuang, Hess, 2006) Single Molecule Localization Microscopy ◮ Activation of few molecules, imaging and localization ◮ Time consuming acquisition, poor temporal resolution, potentially harmful excitation levels http://zeiss-campus. magnet.fsu.edu/ We aim to design a Super-Resolution model with the following features: ◮ improved temporal resolution ◮ harmless excitation levels ◮ use of standard equipment / conventional fluorophores 4 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work 5 / 23

Temporal fluctuations Temporal profile of a fluorescent molecule: Temporal profile of a pixel (real data): Idea : Acquire short videos with high-density of molecules per frame and use a reconstruction algorithm that exploits spatial and temporal independence of the emitters 6 / 23

Related approaches SOFI (Dertinger et al. , 2009) Super resolution Optical Fluctuation Imaging ◮ Shrinkage of PSF via computation of higher-order statistics SRRF (Gustafsson et al. , 2016) Super-Resolution Radial Fluctuations ◮ Non-linear transformation of each frame based on radial symmetry (the degree of local gradient convergence) SPARCOM (Solomon et al. , 2019) SPARsity based super-resolution COrrelation Mi- croscopy ◮ Exploits sparsity in the correlation domain 7 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work 8 / 23

Method CO ℓ 0 RME CO variance-based ℓ 0 super- R esolution M icroscopy with intensity E stimation 1st Step: Support Estimation (Covariance domain) ⇒ Support Ω = 9 / 23

Method CO ℓ 0 RME CO variance-based ℓ 0 super- R esolution M icroscopy with intensity E stimation 1st Step: Support Estimation (Covariance domain) ⇒ Support Ω = 2nd Step: Intensity Estimation 9 / 23

Image model Y t = M q ( H ( X t )) + N t + B t = 1 , ..., T : all the frames Y t X t Y t ∈ R N × N raw data X t ∈ R L × L high-resolution image L = qN M q : down-sizing operator H : convolution operator q=4 N t : additive white Gaussian noise B : stationary background 10 / 23

1st Step: Support Estimation vectorized form ◮ Y t = M q ( H ( X t )) + N t + B − − − − − − − − − − → y t = Ψx t + n t + b 11 / 23

1st Step: Support Estimation vectorized form ◮ Y t = M q ( H ( X t )) + N t + B − − − − − − − − − − → y t = Ψx t + n t + b x 1 x 1 x 2 x 3 x T y 1 y 1 y 2 y T Ψ Ψ . . . . . . . . . . . . . . . . . = = . . . . . . . . . . Ψ T R x R y Ψ . . . . . . . = . . . . . . . . . . . . R x = diag ( r x ) T 1 ( y t − µ y )( y t − µ y ) T Ry = � r x ∈ R L 2 T − 1 t =1 11 / 23

1st Step: Support Estimation vectorized form ◮ Y t = M q ( H ( X t )) + N t + B − − − − − − − − − − → y t = Ψx t + n t + b ◮ Covariance matrices: R y = ΨR x Ψ T ◮ R x is diagonal, r x = diag ( R x ) 11 / 23

1st Step: Support Estimation vectorized form ◮ Y t = M q ( H ( X t )) + N t + B − − − − − − − − − − → y t = Ψx t + n t + b ◮ Covariance matrices: R y = ΨR x Ψ T + R n ◮ R x is diagonal, r x = diag ( R x ) ◮ R n = s I , s ∈ R + 11 / 23

1st Step: Support Estimation vectorized form ◮ Y t = M q ( H ( X t )) + N t + B − − − − − − − − − − → y t = Ψx t + n t + b ◮ Covariance matrices: R y = ΨR x Ψ T + R n ◮ R x is diagonal, r x = diag ( R x ) ◮ R n = s I , s ∈ R + 11 / 23

1st Step: Support Estimation vectorized form ◮ Y t = M q ( H ( X t )) + N t + B − − − − − − − − − − → y t = Ψx t + n t + b ◮ Covariance matrices: R y = ΨR x Ψ T + R n ◮ R x is diagonal, r x = diag ( R x ) ◮ R n = s I , s ∈ R + ◮ R y = ΨR x Ψ T + s I vectorized form − − − − − − − − − − → r y = ( Ψ ⊙ Ψ ) r x + s I v Problem l 2 - l 0 : 1 2 � r y − ( Ψ ⊙ Ψ ) r x − s I v � 2 arg min F + λ � r x � 0 r x ≥ 0 ,s ≥ 0 r x � = 0 − → Support Ω 11 / 23

1st Step: Support Estimation l 2 - l 0 by continuous relaxation 1 with noise variance estimation G ( r x , s ) = 1 2 � r y − ( Ψ ⊙ Ψ ) r x − s I v � 2 2 + λ � r x � 0 → ˜ G ( r x , s ) = 1 2 � r y − ( Ψ ⊙ Ψ ) r x − s I v � 2 2 + Φ CEL0 ( r x ) L 2 L 2 √ λ − � a i � 2 � � 2 λ Φ CEL0 ( r x ) = � φ CEL0 ( � a i � , λ ; ( r x ) i ) = � | ( r x ) i | − √ 1 {| ( r x ) i |≤ 2 � a i � 2 λ � ai � } i =1 i =1 and � a i � = � ( Ψ ⊙ Ψ ) i � 1 Soubies et al., A Continuous Exact l0 penalty (CEL0) for least squares regularized problem, SIAM J. Imaging Sciences, 2015 2 Ochs et al., On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision, SIAM J. Imaging Sciences, 2014 12 / 23

1st Step: Support Estimation l 2 - l 0 by continuous relaxation 1 with noise variance estimation G ( r x , s ) = 1 2 � r y − ( Ψ ⊙ Ψ ) r x − s I v � 2 2 + λ � r x � 0 → ˜ G ( r x , s ) = 1 2 � r y − ( Ψ ⊙ Ψ ) r x − s I v � 2 2 + Φ CEL0 ( r x ) L 2 L 2 √ λ − � a i � 2 � � 2 λ Φ CEL0 ( r x ) = � φ CEL0 ( � a i � , λ ; ( r x ) i ) = � | ( r x ) i | − √ 1 {| ( r x ) i |≤ 2 � a i � 2 λ � ai � } i =1 i =1 and � a i � = � ( Ψ ⊙ Ψ ) i � Alternating minimization: Iterative Reweighted l 1 algorithm 2 & explicit expression: Require: r x 0 ∈ R L 2 , s 0 ∈ R + repeat rn x calculate the weights: ω i L 2 rn r x n +1 = arg 1 2 � r y − ( Ψ ⊙ Ψ ) r x − s n I v � 2 x | ( r x ) i | + X ≥ 0 ( r x ) min 2 + λ � ω i rx ∈ R L 2 i =1 s n +1 = arg min 1 2 � r y − ( Ψ ⊙ Ψ ) r n + 1 − s I v � 2 2 + X ≥ 0 ( s ) x s ∈ R until convergence print r x , s 1 Soubies et al., A Continuous Exact l0 penalty (CEL0) for least squares regularized problem, SIAM J. Imaging Sciences, 2015 2 Ochs et al., On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision, SIAM J. Imaging Sciences, 2014 12 / 23

2nd Step: Intensity estimation x ∈ R L 2 x ∈ R | Ω | ✘ ✘✘✘ The model: y t = Ψx t + n t + b ( x i − x j ) 2 + α �∇ b � 2 2 � Y1 T − Ψ Ω x − T b � 2 1 2 + β � � arg min 2 x ≥ 0 , b ≥ 0 i ∈ Ω j ∈ N ( i ) ∩ Ω 13 / 23

Table of Contents Introduction Exploiting temporal fluctuations COL0RME Results Conclusions and Future work 14 / 23

Simulated acquisition Ground truth Observation (Low Background) (SNR = 21.26dB) Size of the image: 160 x 160 Size of the image: 40 x 40 Fluorescent Molecules (FM): 5471 Pixel size: 100 µm Average FM/pixel/frame: 10.7 Video rate: 100 fps Acquisition time: 10 s Spatial pattern: Microtubules dataset from the SMLM challenge 2016 3 Temporal profiles: SOFI simulation tool (Girsault et al. . 2016) 3 http://bigwww.epfl.ch/smlm/datasets/index.html 15 / 23

Simulated results Ground Truth COL0RME COL0RME T = 100 T = 700 SRRF SRRF T = 100 T = 700 16 / 23

Simulated results Ground Truth COL0RME COL0RME T = 100 T = 700 SPARCOM SPARCOM T = 100 T = 700 16 / 23