Neuroimage Analysis for Automated Brain Disease Diagnosis Mingxia - PowerPoint PPT Presentation

Neuroimage Analysis for Automated Brain Disease Diagnosis Mingxia Liu University of North Carolina at Chapel Hill mxliu@med.unc.edu http://mingxia.web.unc.edu/ 07-17-2019

Background Healthy Brain vs. Alzheimer’s Sulcus Sulcus Gyrus Gyrus Ventricle Alzheimer’s Alzheimer’s Healthy Brain Healthy Brain

Background • Alzheimer’s Disease (AD) – A progressive disease Brain Health • Calling Need – Developing computer-aided methods for MCI/AD diagnosis Time Alzheimer’s Mild Cognitive Normal Control Impairment (MCI) Disease Stable MCI (sMCI) Progressive MCI (pMCI)

Background Biomarkers for early diagnosis of AD and MCI • Structural Magnetic Resonance Imaging (MRI) • FDG-Positron Emission Tomography (PET) Cerebrospinal Fluid (CSF) ‐‐‐ A β42, t ‐ tau and p ‐ tau • Multi-modal data Brain CSF Dura MRI PET CSF

Brain Disease Diagnosis – Typical Pipeline Neuroimaging Image Feature Classifier Data Preprocessing Extraction/Selection Learning sMRI PET fMRI Machine Learning & Deep Learning M. Liu , etc., Landmark-based Deep Multi-Instance Learning for Brain Disease Diagnosis, Medical Image Analysis , 2018. M. Liu , etc. Joint Classification and Regression via Deep Multi-task Multi-channel Learning for Alzheimer’s Disease Diagnosis. IEEE Trans. on Biomedical Engineering , 2018.

Challenges in Computer-aided Disease Diagnosis • Effective feature representation of neuroimages • Missing multi-modal data • Heterogeneous data at different imaging sites

Outline sMRI PET CSF sMRI • Structural MRI (sMRI) • Missing Data • Multi-modal Data Fusion • Domain Adaptation Single-modal Neuroimage Multi-modal Neuroimage

Part I. Single-modal Neuroimage Analysis • Structural MRI based Brain Disease Diagnosis

Anatomical Landmarks for Structural MRI • Landmark-based Deep Representation of sMRI C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics , 2019. M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering , 2019. M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis , 2018. M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics , 2018. J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing , 2017.

Anatomical Landmark-based Deep Network Pre-processed Deep Multi-channel Convolutional Landmark Discovery Patch Extraction MR Images Neural Network (CNN) ... ... ... CNN … … Training Data Training Data Training Data Training MRIs Landmark Detection Test MRI Disease Clinical Score Classification Prediction ata ata ata * Featured Article of IEEE Journal of Biomedical and Health Informatics, 2018

Anatomical Landmark-based Deep Network 1,740 landmarks via group Top 50 landmarks comparison between AD and NC 0.01 p -values 0 M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis , 2018. M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics , 2018.

Landmark-based Deep Network sMRI … Patch 1 Patch 2 Patch L Conv1- Conv1- Conv1- 32@3 × 3 × 3 32@3 × 3 × 3 32@3 × 3 × 3 Class Label Conv2- Conv2- Conv2- 32@3 × 3 × 3 32@3 × 3 × 3 32@3 × 3 × 3 Soft-max Max-pooling Max-pooling Max-pooling FC11-2 Global Conv3- Conv3- Conv3- 64@2 × 2 × 2 64@2 × 2 × 2 64@2 × 2 × 2 Representation? Global Conv4- Conv4- Conv4- … 64@2 × 2 × 2 64@2 × 2 × 2 64@2 × 2 × 2 Image-level FC10-8L Max-pooling Max-pooling Max-pooling Representation Conv5- Conv5- Conv5- 128@2 × 2 × 2 128@2 × 2 × 2 128@2 × 2 × 2 FC9-32L Conv6- Conv6- Conv6- 128@2 × 2 × 2 128@2 × 2 × 2 128@2 × 2 × 2 Max-pooling Max-pooling Max-pooling Concatenated Local … Patch-level FC7-128 FC7-128 FC7-128 FC8-32 FC8-32 FC8-32 Representation M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis , 2018. M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics , 2018.

Results of Classification and Regression 0.518 0.567 Correlation Coefficient 0.538 0.487 0.486 0.492 Overall Accuracy 0.46 7 0.468 0.431 0.404 0.325 0.289 Classification Results for AD vs. sMCI vs. pMCI vs. NC Regression Results for MMSE M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis , 2018. M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics , 2018.

End-to-end Disease Diagnosis with sMRI • Hierarchical Fully Convolutional network C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics , 2019. M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering , 2019. M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis , 2018. M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics , 2018. J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing , 2017.

Hierarchical Network for ROI Identification • Hierarchical Fully Convolutional Network (H-FCN) – Automatically and identify disease-related ROIs in the whole sMR image – Jointly learn multi-scale features and construct a classification model C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. 15 C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification Input: sMRI C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification Input: sMRI 1) Location proposals C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification PSN PSN Class_P 32 64 64 128 128 64 Input: sMRI 1) Location 2) Patch-level sub- proposals networks (PSN) ( shared weights ) C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification PSN PSN Class_P PSN 32 64 64 128 128 64 PSN PSN … … PSN PSN Input: sMRI PSN PSN 1) Location 2) Patch-level sub- proposals networks (PSN) ( shared weights ) C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification PSN PSN Class_P Conv_R Class_R PSN 32 64 64 128 128 64 PSN 64 PSN … … PSN PSN Input: sMRI PSN PSN 1) Location 2) Patch-level sub- 3) Region-level proposals networks (PSN) ( shared sub-networks weights ) C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification PSN PSN Class_P Conv_R Class_R PSN 32 64 64 128 128 64 PSN 64 PSN … … … PSN Conv_R Class_R PSN Input: sMRI PSN 64 PSN 1) Location 2) Patch-level sub- 3) Region-level proposals networks (PSN) ( shared sub-networks weights ) C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Hierarchical Network for ROI Identification Network Pruning: To remove less informative regions PSN PSN Class_P Conv_R Class_R PSN 32 64 64 128 128 64 PSN 64 Conv_S PSN 4 × 4 × 4 Conv 3 × 3 × 3 Conv … … … 64 1 × 1 × 1 Conv 2 × 2 × 2 max pooling Class_S Classification ( 1 × 1 × 1 Conv) PSN Conv_R Class_R PSN Input: Region-level Conv Subject-level Conv Output: sMRI Class label PSN Channel concatenation Spatial concatenation 64 PSN Potentially pruned sub-networks 1) Location 2) Patch-level sub- 3) Region-level 4) Subject-level Skipped connection Conv: Convolution proposals networks (PSN) ( shared sub-networks sub-network weights ) C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Identified Voxel-level Discriminative Locations 1 2 3 4 1 2 1 2 3 2 1 3 3 4 2 1 2 1 1 2 1 2 3 4 1 2 3 (a) AD Subject #1 (b) AD Subject #2 (c) AD Subject #3 1 2 3 1 2 3 1 2 3 2 3 1 1 3 3 2 2 1 1 2 3 1 2 3 1 2 3 (d) AD Subject #4 (e) AD Subject #5 (f) AD Subject #6 C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Identified Patch-level Discriminative Locations Sagittal View Axial View Coronal View 3D View C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .

Identified Region-level Discriminative Locations Sagittal View Axial View Coronal View 3D View C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence , 2019. C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019 .