Deep learning for MR imaging and analysis Shanshan Wang Paul C. - PowerPoint PPT Presentation

Deep learning for MR imaging and analysis Shanshan Wang Paul C. Lauterbur Research Center for Biomedical Imaging Shenzhen Institutes of Advanced Technology (SIAT) 2020-04-15

Learning Reconstruction and Analysis Image reconstruction a. Background b. Linear reconstruction c. Non-linear iterative reconstruction d. Deep learning-based reconstruction Image analysis a. Stroke lesion segmentation b. Breast tumor classification and segmentation c. Cervical cancer classification Summary a. Summary

Background: MRI A powerful tool for clinical diagnosis A powerful tool for scientific research

Challenges SNR ( Signal To Noise Ratio ) Correlated &interacting ➢ Highly dependent on the doctor's experience Imaging Time ➢ Tedious and cumbersome Resolution manual review Short imaging time may cause issues like ➢ Imaging data-heavy nature low resolution and SNR, while long time may cause issues like claustrophobia, requires new solutions like AI motion artifact and signal distortion Challenges In Imaging Challenges In Diagnosis

Factors determining Acquisition Time Ky Recontruction N Kx MR Image Sampling Data (K-Space) T = N × TR Scan time acquisition lines repetition Acquire an image using Spin Echo Sequence: T1 Weighted Image (T1w) ： TR=800ms ， N=256 ， T=3.4min T2 Weighted Image (T2w) ： TR=2000ms, N=256, T=8.5min

Fast MRI Techniques MR Physics (1970’s) ➢ Pulse sequence design • ➢ Hardware (2000’s) Parallel imaging with phased array coils • Image reconstruction from incomplete K-space data (past decade) ➢ Modeling using priori knowledge • Radio Frequency Pulse Reconstruction Encoding Phased Array Coil Image for diagnosis K-Space Data

Image Reconstruction Incomplete K-space data Reconstructed image Pr(x) Different prior knowledge 1 2 + 𝑔 𝑦 = 2 𝐹𝑦 − 𝑧 2 𝜇𝑄 𝑠 (𝑦) y : Given measurements Sanity Prior or regularization x : Unknown to be recovered (relation to measurements) E : Encoding matrix

MR Recon from Incomplete K-space Data Image domain Sparsity domain Big datasets Ky Big dataset Wavelet transform / TV collection Kx Deep prior Inverse transform learning Partial K-space 3 rd phase Deep learning 1 st phase linear 2 nd phase Nonlinear MR reconstruction reconstruction iterative reconstruction (CNN, ADMM-NET,VN-net, (IFFT, SENSE, SMASH, (CS, low-rank, dictionary Automap, MoDL, U- NET, …) GRAPPA, ….) learning, …) [1] Shanshan Wang, et al. . IEEE International [1] Pruessmann, Klaas P., et al. Magnetic [1] Justin H, et al. Magnetic Resonance in Symposium on Biomedical Imaging, 2016. Resonance in Medicine , 1999. Medicine, 2016 [2] Bo Zhu, Nature, 2018. [2] Sodickson, Daniel K., and Warren J. [2] Zhi-Pei Liang, et al. IEEE Transactions [3] Florian Knoll, et al. Magnetic Resonance Manning. Magnetic Resonance in Medicine , 1997. in Medical Image, 2003. Imaging, 2018. [3] Griswold, Mark A., et al. Magnetic Resonance in [3] Zhou, Yihang, et al. IEEE International [4] Yang, Guang, et al. IEEE Transactions in Medicine, 2002. Symposium on Biomedical Imaging, 2015. Medical Image, 2018. [4] Lustig et al . Magnetic Resonance in Medicine, [4] Shanshan Wang ,et al, IEEE [5] J. Schlemper, et al. IEEE Transactions in 2007. Transactions Medical Imaging, 37(1):251- Medical Image, 2018. [5] Jianhua Luo, Shanshan Wang , et al. Journal of 261, 2018 Magnetic Resonance 224 (2012): 82-93

Our Work - 1 st phase ➢ Layer deconvolution spectral (LDS) analysis method ➢ Main steps • Estimate the truncated k-space from the image containing truncation artifact • Compute sparse representation parameters from truncated k- space data Use computed parameters to • recover Missing k-space data • Obtain the artifact-removed image through Inverse transform of updated k-space data Convolution/deconvolution is always a very powerful tool ! Jianhua Luo, Shanshan Wang , et al. Journal of Magnetic Resonance 224 (2012): 82-93.

Our Work - 1 st phase • (a) Initial ZF image containing truncation • Results of removing truncation artefact in real artefacts (with truncation frequency c = 64). (b – d) MR images. (a and c) Represent respectively a are respectively the images after removing the stomach MR image and a brain MR image artefacts in (a) using the Hamming window, the having truncation artefacts. (b and d) Show the TV and the proposed LDS methods. images after removing artefacts in (a) and (c), respectively, using the proposed LDS method. Jianhua Luo, Shanshan Wang , et al. Journal of Magnetic Resonance 224 (2012): 82-93.

1 st phase Sub-summary ➢ 1 st phase Linear analytical reconstruction ➢ Pros: • Simple and straightforward model Easy implementation • ➢ Cons: Object prior is not considered • • Long scan (acquisition) time

Compressed Sensing (CS) • Incoherent projection • Signal sparsity Sparse unknown vector 𝐵 𝑌 𝑐 𝑜 × 1 vector 𝑛 × 1 = measurements 𝑙 # of non-zeros 𝑛 ≅ 𝑙 log 𝑜 ≪ 𝑜 Wakin, Michael, et al. "Compressive imaging for video representation and coding." Picture Coding Symposium . Vol. 1. No. 13. 2006. Takhar, Dharmpal, et al. "A new compressive imaging camera architecture using optical-domain compression." Computational Imaging IV . Vol. 6065. International Society for Optics and Photonics, 2006.

2 nd phase nonlinear reconstruction Dynamic imaging Example 2 Example 1 time course Low rank Prior knowledge can be roughly categorized as non-adaptive and adaptive ones. Non-adaptive: Fixed transform, statistical modelling, model fitting, low-rank. Adaptive: Dictionary learning, data-driven tight frame

Our Work- 2 nd phase ➢ Sparse representation: based on fixed and adaptive dictionary • Dictionary learning in Fenchel-dual space Impulse-noise removal with L1-L1 minimization • Improving image reconstruction accuracy with good convergence property • Dictionary Learning Shanshan Wang , et al. IEEE Transactions on Image Processing 22.12 (2013): 5214-5225. Shanshan Wang , et al. Signal Processing 93.9 (2013): 2696-2708. Dong Pei, Shanshan Wang* , et al. IEEE Transactions on Image Processing 25.11 (2016): 5035-5049 Qiegen Liu, Shanshan Wang , et al. IEEE Transactions on Image Processing 22.12 (2013): 4652-4663

Our Work – Dictionary learning for MRI ➢ Multi-channel correlation and multi-layer sparse development Label DL-PI SparseSENSE • Parallel imaging with L2,1 norm adaptive joint sparse coding • One - layer and two - layer tight Sparse BLIP CaLM MRI Proposed frame learning for MR imaging Figure. Achieves 6X acceleration in 2D with the • Improve the accuracy of image smallest reconstruction error. reconstruction and accelerate the convergence. Shanshan Wang , Dong Liang,et al, IEEE Transactions on Medical Imaging,37(1):251-261, 2018 Shanshan Wang , Dong Liang, et al, BioMed Research International （ SCI ） , 2860643, 2016 Qiegen Liu, Shanshan Wang , et al, IEEE Transactions on Medical Imaging, 2013

Our Work - Iterative Feature Extraction ➢ Iterative Feature Refinement-Compressed Sensing (IFR-CS) consists of three main steps: 2 + 𝜇 𝑀1 𝑣 𝑀1 𝑣 = arg min 𝐽 − 𝑣 2 • Sparsity-promoting denoising 𝑣 𝐽 𝑢 = 𝑣 + 𝑈 ⊗ 𝑤 Feature refinement • 2 + 𝜈 𝐽 − 𝐽 𝑢 𝐽 = arg min 𝐺 𝑞 𝐽 − 𝑔 2 • Tikhonov regularization 𝐽 IFR-CS Shanshan Wang , Dong Liang, et al. Physics in Medicine and Biology, 2016 ， 61, 3291 － 3316

Our Work - Iterative Feature Extraction ➢ Extracting fine structure and details from the residual image This paper were selected as Featured • Article and 2016 Research Highlight by Physics in Medicine and Biology. = Detected structure Feature descriptor Residual image The technology has applied for a patent : 201410452350.1 Physics in Medicine and Biology （ SCI ）

Our Work - Iterative Feature Extraction ➢ Results: Reference IRM-TV DLMRI Proposed Reconstruction Error Enlarged region

2 nd phase Sub-summary ➢ 2 nd phase Nonlinear Iterative reconstruction Pros: ➢ • Improved image quality • Image prior is included in the model Nice theoretical explanation • ➢ Cons: Long reconstruction time • • Limited prior model capacity • Hand tuned parameters

1 ST Work in This Area going beyond CS ➢ Deep Learning (DL) MRI beyond Compressed Sensing (CS) ( Citation 256 ) Scan Recon Linear Scan Recon CS-MRI Scan Recon DL-MRI Training data Offline Training Θ Online Reconstruction Input Reconstruction S Wang , L. Ying, D Liang etal,. IEEE International Symposium on Biomedical Imaging 2016: 514-517.

Pioneering work Combination with CS-MRI reconstruction methods Sequential model: Initialize CS Reconstructed MR Network Prediction Reconstruction model image Integration model: S Wang , L. Ying, D Liang, “Accelerating Magnetic Resonance Imaging via Deep Learning,” IEEE -ISBI 2016: 514-517.

Initial Results ➢ 3T scanner ➢ 32-channel coil ➢ T1-weighted (spoiled GRE) ➢ TE min full Sampling mask Reference Initialization ➢ TR = 7.5ms ➢ 256×256, ➢ thickness = 17mm ➢ R = 3 Network Output Final reconstruction Error S Wang , L. Ying, D Liang, “Accelerating Magnetic Resonance Imaging via Deep Learning,” IEEE -ISBI 2016: 514-517.