Enhao Gong, PhD Candidate, Electrical Engineering, Stanford University Dr. John Pauly, Professor in Electrical Engineering, Stanford University Dr. Greg Zaharchuk, Associate Professor in Radiology, Stanford University
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Prof. John Pauly Dr. Morteza Mardani Prof. Greg Zaharchuk Dr. Jia Guo
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Medical imaging interior clinical analysis and medical intervention visual representation PET fMRI Structural Imaging Functional Imaging Image source: Wikipedia, NIVIDA
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Image Reconstruction Pathology Detection Diagnosis Assistance • From anatomy to image • From image to labels • From pathology to diagnosis • Reconstruction and Restoration • Tumor Segmentation • Prescription and Treatment • Denoising and Super-resolution • Pathology Detection • Prognostic Prediction Prognosis Scanner Analytics Diagnosis Treatment Images Enhancement & Augmentation Pathology Image source: blogs.nvidia.com
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Magnetic Resonance Imaging ü Great tissue contrast for distinguish normal tissue vs pathology ü No exposure to ionizing radiation q Samples in Fourier domain ( k -space) • Need some “magic” transform to convert to image domain Scanner Signal Image
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Reconstruction of sparse sampled complex signal in Fourier domain ( k -space) of images • Much harder image restoration tasks Reconstruction Under-sampled Recovered k -space k-space 20 20 40 40 60 60 80 80 100 100 120 120 140 140 160 160 180 180 20 40 60 80 20 40 60 80
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Reconstruction of sparse sampled complex signal in Fourier domain ( k -space) of images • Harder image restoration tasks Reconstruction Under-sampled Recovered k -space k-space 20 20 40 40 60 60 80 80 100 100 120 120 140 140 160 160 180 180 20 40 60 80 20 40 60 80 Image Super-resolution Image Restoration
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Reconstruction of sparse sampled complex signal in Fourier domain ( k -space) of images • Solved with constrained optimization Reconstruction Under-sampled Recovered • Reconstruction Model k -space k-space • Image: X • Acquisition Model: E 20 20 • Measured Signal: Y=EX 40 40 60 60 80 80 • Solve inverse problem with optimization 100 100 120 120 140 140 160 160 180 180 20 40 60 80 20 40 60 80 Regularizations Consistent with (Sparsity, Low-rank, Signal Model Dictionary) • Solving using Iterative Optimization
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Case #1 Multi-contrast (structure) MRI reconstruction Original Proposed Post-Diamox
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs By Courtesy, Center for Advanced Functional Neuroimaging, Stanford
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Initial Recon Ground-truth Recon Sequential Jointly+Local Patch à Image • Intermediate • Train Local deep network • Image synthesis reconstruction for patch-patch regression • Generate Improved • Sequential and • Jointly improve multi- reconstruction independent contrast local patch
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Initial Recon Ground-truth Recon Sequential Jointly+Local Patch à Image • Intermediate • Train Local deep network • Image synthesis reconstruction for patch-patch regression • Generate Improved • Sequential and • Jointly improve multi- reconstruction independent contrast local patch
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs CNN Sequential Jointly+Local Patch à Image • Intermediate • Train Local deep network • Image synthesis reconstruction for patch-patch regression • Generate Improved • Sequential and • Jointly improve multi- reconstruction independent contrast local patch
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs CNN Sequential Jointly+Local Patch à Image • Intermediate • Train Local deep network • Image synthesis reconstruction for patch-patch regression • Generate Improved • Sequential and • Jointly improve multi- reconstruction independent contrast local patch
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs CNN Convention Using Data Methods Deep Learning Augmentation • Intermediate • Train Local deep network • Image synthesis reconstruction for patch-patch regression • Generate Improved • Sequential and • Jointly improve multi- reconstruction independent contrast local patch
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs method details Convolutional Encoder-Decoder with bypasses O. Ronneberger, et al. 2015 Convolutional Encoder-Decoder with downsample poolings H. Noh, et al. ICCV2015 1. M. Uecker et al. MRM 2014 2.M. Uecker, et al. BART
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs residual learning results Training Testing Initial Residual Predicted Residual Reduced Residual Initial Residual Predicted Residual Reduced Residual
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs residual learning results Residual Residual R2 Residual R2 Artifact Fitting RMSE Norm (Before) Norm (After) Reduction (%) Train 0.460 0.0102 0.0051 50% Test 0.729 0.0230 0.0192 17% Training Testing Initial Residual Predicted Residual Reduced Residual Initial Residual Predicted Residual Reduced Residual
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs clinical results in routine multi-contrast neuro-imaging protocol Improved GRE reconstruction on a hemorrhage subject ü Faster scan with less artifacts ü Accelerate FLAIR/GRE with T1w/T2w scan ü Reconstruction with sharable information ü Acceleration with preserved pathology
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs • Case #2 Perfusion MRI reconstruction Original Proposed Post-Diamox
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Background • Arterial Spin Labeling(ASL) ( ) x 100= Blood flow map
CS7415 Enhanced multi-contrast MRI reconstruction with Deep Learning & NVIDIA GPUs Detail Methods and innovations NLM Denoising • Extend conventional spatial-frequency Denoising • Non-Local-Mean (NLM) Filtering • Extend with multi-contrast information • Train deep network models for image-to-image regression tasks • Ground-truth with higher resolution and SNR • End-to-end Denoising and super-resolution • Use multi-contrast images/patches as inputs • More robustness, accuracy and avoid overfitting A. Buades, et.al. CVPR 2005
Step 1: Nonlinear ASL signal denoising using Non-Local (NLM) and MulC-contrast Guided Filter Nex=1 Low SNR ASL raw Nex=6 High SNR Ref ASL Training 𝒙 𝑩𝑻𝑴 = ∞ 𝒙 𝑩𝑻𝑴 = 𝟐𝟏𝟏 𝒙 𝑩𝑻𝑴 = 𝟔𝟏 𝒙 𝑩𝑻𝑴 = 𝟑𝟏 𝒙 𝑩𝑻𝑴 = 𝟐𝟏 𝒙 𝑩𝑻𝑴 = 𝟔 original recon more regulariza.on using nonlinear denoising Step 2: Generate patches from High-SNR Ref. ASL, Low-SNR raw ASL, multi-level denoised ASL and anatomical MR images Nex=6 Nex=1 High SNR Ref ASL Low SNR ASL Denoised ASL with different 𝒙 𝑩𝑻𝑴 𝑼𝟑𝒙 FSE 𝑸𝑬𝒙 Step 3: Training deep network to learn the nonlinear image restoration from multi-contrast patches Deep Convolutonal-Deconvolu4onal Neural Network Cost function Input Input by-passes connec.ons Output Compare vs. Ref Patches Patches … More Layers Output: restored high-SNR ref Multi-contrast patches Step 4: Generate the restored image from stored patches Output Patches High SNR Ref ASL Restored from Low−SNR Diff Original Low-SNR Original Diff
Recommend
More recommend
