ARTIFICIAL INTELLIGENCE FOR DIGITAL PATHOLOGY Kyunghyun Paeng, - PowerPoint PPT Presentation

ARTIFICIAL INTELLIGENCE FOR DIGITAL PATHOLOGY Kyunghyun Paeng, Co-founder and Research Scientist, Lunit Inc.

1. BACKGROUND: DIGITAL PATHOLOGY 2. APPLICATIONS BREAST CANCER • AGENDA • PROSTATE CANCER 3. DEMONSTRATIONS 4. CONCLUSION 2

BACKGROUND: DIGITAL PATHOLOGY DIAGNOSTIC PROCEDURE Patient Detection Diagnosis Treatment (X-ray, CT, MRI, ...) (biopsy, resection, ...) Radiology Pathology Oncology 3

BACKGROUND: DIGITAL PATHOLOGY LIMITATIONS OF CONVENTIONAL PATHOLOGY Slide Report Diagnosis (-) Archiving (biopsy, resection, ...) (-) Workflow Pathology (-) Analysis 4

BACKGROUND: DIGITAL PATHOLOGY RISE OF DIGITAL PATHOLOGY Diagnosis (+) Archiving (biopsy, resection, ...) (+) Workflow Pathology (+) Analysis Digital pathology 5

BACKGROUND: DIGITAL PATHOLOGY WHY DO WE NEED AI IN DIGITAL PATHOLOGY ? (+) Reproducibility (+) Accuracy (+) Workload reduction 25% disagreement among pathologists in breast biopsy report. “Diagnostic Concordance Among Pathologists Interpreting Breast Biopsy Specimens.”, JAMA, 2015. 6

BACKGROUND: DIGITAL PATHOLOGY CHALLENGES IN AI FOR DIGITAL PATHOLOGY ~ 100,000 pixels Grade 1 1. Gigapixel images Grade 2 Grade 3 2. Quality variation 3. Ambiguity in ground-truth definition 3! 4! 3? 4? 7

KEY APPLICATIONS: #1. Tumor proliferation score prediction in breast resection specimen. #2. Gleason score prediction in prostate biopsy specimen. 8

APPLICATION #1: BREAST CANCER WHAT IS TUMOR PROLIFERATION SCORE ? Breast resection specimen Proliferation score (in 10 consecutive HPFs) good Mitosis Score 1: ~6 mitosis Score 2: 6~10 mitosis prognosis Score 3: 10~ mitosis bad 9

APPLICATION #1: BREAST CANCER TUMOR PROLIFERATION SCORE PREDICTION Data statistics Tumor Proliferation Assessment Challenge 2016 TUPAC16 | MICCAI Grand Challenge Training dataset Test dataset Proliferation Proliferation 500 slides 321 slides score score , , Auxiliary dataset Mitosis #1 (x,y) 656 ROIs ... , from 73 slides Mitosis #N (x,y) 10

APPLICATION #1: BREAST CANCER TUMOR PROLIFERATION SCORE PREDICTION System overview Mitosis 1. The number of mitosis Detection 2. The number of cells ... Network Whole slide image Feature vector based on statistical information Stain normalization Tissue region extraction Support Vector Machine Auxiliary set for mitosis Patch extraction at x40 detection Proliferation score ROI detection using cell density Phase 2: Mitosis Phase 1: Handling whole slide images Phase 3: Score prediction detection 11

APPLICATION #1: BREAST CANCER TUMOR PROLIFERATION SCORE PREDICTION Phase 1: Handling whole slide images Resizing a whole slide image. • Finding a threshold. • Morphological operations. • ... Whole slide image Patch extraction with 10 HPFs size. • Cell detection in each patch. • Stain normalization Tissue region extraction 30 ROIs selection. • Stain normalization. • Patch extraction at x40 ROI detection using cell density 12

APPLICATION #1: BREAST CANCER TUMOR PROLIFERATION SCORE PREDICTION Phase 2: Mitosis detection 16 conv 1, resblock resblock resblock mitosis 3x3, 16 1.1, 3x3, 32 2.1, 3x3, 64 3.1, 3x3, 128 8 Mitosis Detection normal resblock resblock resblock Network 128 x 128 1.3, 3x3, 32 2.3, 3x3, 64 3.3, 3x3, 128 Global pooling layer • Based on Residual Network (ResNet). • 9 residual blocks = 21 layers architecture. • 2 step training procedure. Auxiliary set • Cropped global pooling layer. for mitosis detection Training step: , Inference step: 13

APPLICATION #1: BREAST CANCER TUMOR PROLIFERATION SCORE PREDICTION Phase 3: Score prediction Converting each WSI to a 21-dim feature vector. • 10-fold cross validation from 500 training samples. • 1. The number of mitosis Feature selection based on cross validation results. • 2. The number of cells Feature vector based on statistical information Support Vector Machine Proliferation score 14

APPLICATION #1: BREAST CANCER TUMOR PROLIFERATION SCORE PREDICTION Results Tumor Proliferation Assessment Challenge 2016 TUPAC16 | MICCAI Grand Challenge 15

APPLICATION #2: PROSTATE CANCER WHAT IS GLEASON SCORE ? Prostate biopsy specimen Grade 1 Core #1: 5+5 Core #2: 0 Core #3: 3+4 Grade 2 Core #4: 0 Grade 3 Grade 4 Grade 5 Grade 1, 2 Grade 3 Grade 4 Grade 5 16

APPLICATION #2: PROSTATE CANCER GLEASON SCORE PREDICTION Data statistics Training dataset Test dataset { Grade, { Grade, 900 slides 50 slides Contours } Contours } , , Dataset from medical centers The number of patients: 385 • The number of slides: 1152 • The number of cores: 4907 • The number of normal cores: 2872 • • The number of cancer cores: 2035 17

APPLICATION #2: PROSTATE CANCER GLEASON SCORE PREDICTION System overview Patch-based classification 1000 Normal Normal Gleason score 1100 Ranking loss Grade 3 classification network 1110 with thermometer code Grade 4 1111 Grade 5 Grade 3 Memory network-based refinement (25 neighbors) Memory network Grade 4 Embedded memory vector ... Refined Grade 5 output Embedding Query vector 18

APPLICATION #2: PROSTATE CANCER GLEASON SCORE PREDICTION Patch-based classification Baseline settings Normal • ResNet 101 architecture. ~75% 512x512 patch with 75% overlap. • Softmax loss with 4 class classification. • Grade 3 Key features for improving performance Normal patches from only fully normal slides. è +~5% gain • Ranking loss with thermometer code. è +2~3% gain • Grade 4 Not a classification problem! Ordering problem! 1000 1100 Network decodes from the left-most bit Grade 5 1110 to the right-most bit. 1111 19

APPLICATION #2: PROSTATE CANCER GLEASON SCORE PREDICTION Patch-level outputs Memory network-based refinement (25 neighbors) Refined output + ~5% gain 1D-CNN ... Softmax 25x1024 ... Weighting ... ... ... Embedding Innerproduct Memory vector ... (25x4dim) Attention vector ... 25x1 1x1024 Query vector (1x4dim) 20

APPLICATION #2: PROSTATE CANCER GLEASON SCORE PREDICTION Results Patch-level performance Baseline: 75% • + Data cleansing: 80% + Ranking loss: 82.8% + Memnet refinement: 87.5% Core-level performance Normal or cancer core? • AUC: 97.8% • • Gleason score prediction? Only 1 st major: 83% • Both: 76.7% • 21

DEMO #1: BREAST CANCER 22

DEMO #2: PROSTATE CANCER 23

CONCLUSION Artificial intelligence for digital pathology Lessons learned Challenge #1. How to handle gigapixel images ? (i.e., whole slide images) ü Consider how to sample patches. (patch size, sampling step, ...) è with pathologists. ü Consider how to construct whole pipeline from gigapixel images to diagnosis. Challenge #2. How to handle quality variation between slides ? ü Design image processing modules carefully. ü Do cross-validation to avoid overfitting. Challenge #3. How to handle ambiguous ground-truth ? ü Design task-specific loss. ü Sanitize training dataset as much as possible you can. ü Don’t be satisfied with patch-based results. 24

TEAM MEMBERS THANK YOU