Dermatologist-level classification of skin cancer with deep neural - PowerPoint PPT Presentation

Dermatologist-level classification of skin cancer with deep neural networks Enhancing the Expert Andre Esteva PI: Sebastian Thrun Stanford University 1

How can technology assist a human? 2

3

4

5

How can AI assist a dermatologist? 6

Skin Cancer 7

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US 8

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US ● 20% of Americans will get skin cancer 9

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US ● 20% of Americans will get skin cancer ● Actinic Keratosis (pre-cancer) affects 58 million Americans 10

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US ● 20% of Americans will get skin cancer ● Actinic Keratosis (pre-cancer) affects 58 million Americans ● 76,000 melanomas each year - 10,000 deaths 11

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US ● 20% of Americans will get skin cancer ● Actinic Keratosis (pre-cancer) affects 58 million Americans ● 76,000 melanomas each year - 10,000 deaths ● $8.1B in US annual costs for skin cancer 12

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US ● 20% of Americans will get skin cancer ● Actinic Keratosis (pre-cancer) affects 58 million Americans ● 76,000 melanomas each year - 10,000 deaths ● $8.1B in US annual costs for skin cancer 100 Stage I Survival Probability 75 Stage II 50 Stage III 25 Stage IV 0 1 5 10 15 13 Years

Skin Cancer ● 5.4M cases of non-melanoma skin cancer each year in US ● 20% of Americans will get skin cancer ● Actinic Keratosis (pre-cancer) affects 58 million Americans ● 76,000 melanomas each year - 10,000 deaths ● $8.1B in US annual costs for skin cancer 100 Stage I 0 1 2 3 4 Survival Probability 75 Stage II 50 Stage III 25 Stage IV 0 1 5 10 15 14 Years

15

16

Early detection is critical 17

6.3 billion smartphones 18

Skin Cancer Classification 19

Skin Cancer Classification ~130,000 images of skin 2000 diseases 20

Skin Cancer Classification ~130,000 images of skin 2000 diseases 21

Skin Cancer Classification Epidermal Lesions Melanocytic Lesions Melanocytic Lesions (Dermoscopy) Benign Malignant 22

Skin Cancer Classification Deep Convolutional Neural Network (Inception-v3) 23

Skin Cancer Classification Skin Lesion Image Deep Convolutional Neural Training Classes Network (Inception-v3) (757) Acral-lent. melanoma Amelanotic melanoma Lentigo melanoma ... Blue nevus Halo nevus Mongolian spot … Partitioning Algorithm 24

Skin Cancer Classification Skin Lesion Image Deep Convolutional Neural Training Classes Inference Classes Network (Inception-v3) (757) (varies by task) Acral-lent. melanoma Amelanotic melanoma 92% Malignant Lentigo melanoma ... Blue nevus Halo nevus 8% Benign Mongolian spot … Partitioning Algorithm 25

Skin Cancer Classification P = 0.1 P = 0.4 P = 0.05 P = 0.1 Training Classes P = 0.05 P = 0.05 Inference Classes P = 0.02 P = 0.03 26

Dermatologist-level performance 27

Skin Cancer Classification Validation set 28

Skin Cancer Classification Classifier Three-way accuracy Validation set Dermatologist 1 65.6% Dermatologist 2 66.0% CNN 69.5% CNN - PA 72.0% Disease classes: three-way classification 0. Benign single lesions 1. Malignant single lesions 2. Non-neoplastic lesions 29

Skin Cancer Classification Classifier Three-way accuracy Classifier Nine-way accuracy Validation set Dermatologist 1 65.6% Dermatologist 1 53.3% Dermatologist 2 66.0% Dermatologist 2 55.0% CNN 69.5% CNN 48.9% CNN - PA 72.0% CNN - PA 55.3% Disease classes: Disease classes: three-way classification nine-way classification 0. Benign single lesions 0. Cutaneous lymphoma and lymphoid infiltrates 1. Malignant single lesions 1. Benign dermal tumors, cysts, sinuses 2. Non-neoplastic lesions 2. Malignant dermal tumor 3. Benign epidermal tumors, hamartomas, milia, and growths 4. Malignant and premalignant epidermal tumors 5. Genodermatoses and supernumerary growths 6. Inflammatory conditions 7. Benign melanocytic lesions 8. Malignant Melanoma 30

Skin Cancer Classification Test set 31

Skin Cancer Classification Test set: Dermatologist Comparison (376 images) 32

Skin Cancer Classification Test set: Dermatologist Comparison (376 images) Carcinoma: 135 images Specificity Algorithm: AUC = 0.96 Dermatologists (25) Average Dermatologist Sensitivity Sensitivity 33

Skin Cancer Classification Test set: Dermatologist Comparison (376 images) Carcinoma: 135 images Melanoma: 130 images Melanoma: 111 dermoscopy images Specificity Specificity Specificity Algorithm: AUC = 0.96 Algorithm: AUC = 0.94 Algorithm: AUC = 0.91 Dermatologists (25) Dermatologists (22) Dermatologists (21) Average Dermatologist Average Dermatologist Average Dermatologist Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity 34

Skin Cancer Classification Test set: Total (1942 images) Carcinoma: 707 images Melanoma: 225 images Melanoma: 1010 dermoscopy images Specificity Specificity Specificity Algorithm: AUC = 0.96 Algorithm: AUC = 0.96 Algorithm: AUC = 0.94 Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity 35

How does the algorithm work? 36

T-SNE Visualization 37 Van der Maaten & Hinton, 2008

T-SNE Visualization Epidermal Benign Epidermal Malignant Melanocytic Benign Melanocytic Malignant Basal Cell Carcinomas Squamous Cell Carcinomas Nevi Melanomas Seborrheic Keratoses 38

T-SNE Visualization Epidermal Benign Basal Cell Carcinomas Epidermal Malignant Melanocytic Benign Melanocytic Malignant Squamous Cell Carcinomas Nevi Melanomas Seborrheic Keratoses 39

What is the network fixating on? 40

What is the network fixating on? Malignant Melanocytic Lesion Simonyan, Zisserman, 2014 41

What is the network fixating on? Malignant Melanocytic Lesion Benign Melanocytic Lesion Inflammatory Condition Malignant Epidermal Lesion Benign Epidermal Lesion Genodermatosis Malignant Dermal Lesion Benign Dermal Lesion Cutaneous Lymphoma 42

What does the network misclassify? 43

What does the network misclassify? CNN Dermatologist 1 Dermatologist 2 1 0 1 2 True Label 3 4 5 6 7 8 0 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 Predicted Label Predicted Label Predicted Label 44

Dermatologist-level Classification of Skin Cancer with Deep Neural Networks Andre Esteva* , Brett Kuprel*, Rob Novoa, Justin Ko, Susan Swetter, Helen Blau, Sebastian Thrun Nature, 2017 (Equal contribution authors*) 45

How can AI assist a dermatologist? 46

Community 47

Questions? esteva@cs.stanford.edu @andreesteva cs.stanford.edu/people/esteva 48