Introducing Space and Time in Local Feature-Based Endomicroscopic - PowerPoint PPT Presentation

Introducing Space and Time in Local Feature-Based Endomicroscopic Image Retrieval January 25, 2010 Barbara André Supervision Tom Vercauteren Nicholas Ayache 1 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

1. Introduction • Outline 1. Introduction 2. The Bag-of-Visual Words Method 3. Introducing Spatial Information 4. Introducing Temporal Information 5. Conclusion 2 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Medical Context 1. Introduction pCLE Probe-based Confocal Laser Endomicroscopy Colonic Polyp 3 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

pCLE Principle Fiber bundle scanned by the laser Laser source Avalanche photodiode Fiber bundle Oscillating mirror Surface scanned up to at 4 kHz 600 × 600 μm Frame rate: 12 Hz Galvanometric mirror 4 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Explore the Entire GI Tract Esophagus Duodenum and Stomach small intestine Bile duct Colon Rectum 5 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Clinical Need 1. Introduction Crypt Goblet Cell Benign Differentiate Neoplastic ( pathological ) Courtesy of Pr. Michael Wallace , Mayo Clinic, Jacksonville, USA Nuclei or membranes not visible… nucleo-cytoplasmic ratio ? Combination of local texture & shape features in pCLE images ? 6 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

CBIR Concept 1. Introduction Courtesy of Pr. Charles Lightdale Database Query Columbia-Presbyterian MC, New York, USA Courtesy of Courtesy of Courtesy of Pr. Charles Lightdale Dr. Caroline Loeser Pr. Michael Wallace Columbia-Presbyterian MC, New York, USA Yale University, New Haven, USA Mayo Clinic, Jacksonville, USA Colon Colon Colon Benign Benign Neoplastic 7 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

State of the Art in Computer Vision 1. Introduction Texture classes of the UIUCTex dataset [1] Classification accuracy = 98.7 % Database 25 classes 500 images CBIR method: Bag-of-Visual Words 8 [1] Zhang et al., IJCV 2007 [1] Zhang et al., IJCV 2007 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Methodology Benign Neoplastic 9 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

2. The Bag-of-Visual Words Method • Outline 1. Introduction 2. The Bag-of-Visual Words Method 3. Introducing Spatial Information 4. Introducing Temporal Information 5. Conclusion 10 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

BVW Pipeline 2. The Bag-of-Visual Words Method 1. Regions Salient Region Image I Detector x X Salient Region x x x x x Courtesy of Pr. Michael Wallace , Mayo Clinic, Jacksonville, USA 11 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

BVW Pipeline 2. The Bag-of-Visual Words Method Bag of features: 2. Regions Salient Region region descriptions Description Invariant Description x SIFT [1] Vector x x v 1 v 2 x x . Feature . x v 128 Courtesy of Pr. Michael Wallace , Mayo Clinic, Jacksonville, USA 12 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

BVW Pipeline 2. The Bag-of-Visual Words Method N training images 3. ( W 1 , ... , W K ) N bags of features: Clustering K visual words “ Visual Words “ One Bag are clusters ” for all images ” x x x x x x x x x x x x x x x x x x V x x x x x x x x x x x x x x x x x x x x x x x Feature x x x x xx xx x x V x x x x x x x x xx x xx x x x x x xx x xx x xx xx x x x x x x x x x x x x x x x x x x x x x x x x x V x x x x x x x x x x x x x x x x x x x x x x x x x x x x V x x Space x x x x x x x x x x x x V x x x x x x V x x x x V x x x x x x x x xx xx x x x x x x V V x x x x V VV x x x x x x x x x x x x x x x x e.g. SIFT Space V x x V V x x x x x x V V VV x x x x x x x x x x V V x x x x xx xx x x x x x x x x x x x x V x x x x x x x x V x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x xx xx x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 13 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

BVW Pipeline 2. The Bag-of-Visual Words Method Regions 4. Image Mapping to ( W 1 , ... , W K ) Signature Visual Words K visual words x Number of x x occurrences x x x Courtesy of Pr. Michael Wallace , 1 2 3 ... K Mayo Clinic, Jacksonville, USA Visual word 14 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

BVW Pipeline 2. The Bag-of-Visual Words Method Courtesy of Pr. Michael Wallace , Mayo Clinic, Jacksonville, USA Image I 1 5. Similarity Measure d ( I 1 , I 2 ) = � 2 ( Signature ( I 1 ) , Signature ( I 2 ) ) Image I 2 15 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

BVW Pipeline 2. The Bag-of-Visual Words Method 3. 1. Region Training Detector Clustering Visual Word Images Dictionary 4. Mapping to 2. Region Off- & Descriptor Visual Words line Image Signatures x x x x x x x x Courtesy of x x Pr. Michael Wallace , x x Mayo Clinic, USA 1. Region 5. 4. Mapping to Detector On- Similarity Visual Words line Measures 2. Region Descriptor OUPTUT: INPUT Most similar images Query Image 16 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

3. Introducing Space • Outline 1. Introduction 2. The Bag-of-Visual Words Method 3. Introducing Spatial Information 4. Introducing Temporal Information 5. Conclusion 17 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

From Sparse to Dense Detector 3. Introducing Space time Courtesy of Pr. Michael Wallace , Mayo Clinic, Jacksonville, USA Sparse detector… inconsistency ! Clinically relevant information is densely distributed. Dense Region Detection 18 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Bi-Scale Disc Description 3. Introducing Space Dense regular grid Disc Overlap Large discs: groups of cells Small discs: individual cells Bi-Scale Disc Description Courtesy of Pr. Michael Wallace , Mayo Clinic, Jacksonville, USA 19 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Introducing Space 3. Introducing Space Observation: Cellular architecture is substantial to establish a diagnosis Courtesy of Pr. Michael Wallace Mayo Clinic, Jacksonville, USA Assumption: Spatial relationship between local features statistically the same in the images with similar appearance 20 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Introducing Space 3. Introducing Space Idea: Spatial relationship Feature = Co-occurrence matrix of visual words Image I, K visual words M of size K x K w 1 … w i … w K w 1 w j w K Proba ( w i adjacent to w j in I ) 21 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Introducing Space 3. Introducing Space Supervised description S ( I ) = W . M of Spatial Features Co-occurrence Discriminant matrix linear combinaison ( LDA ) M 1 W M 2 Image I 2 Image I 1 S ( I 1 ) S ( I 2 ) E = | S 1 – S 2 | Benign space feature Neoplastic space feature Space Feature Difference 22 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Introducing Space 3. Introducing Space 1. Region 3. Clustering Training Detector Visual Word Images Dictionary & 2. Region 4. Mapping to Off- Image Descriptor Visual Words line Signatures 1. Region 5. Detector 4. Mapping to On- Similarity Threshold on E = Visual Words line Measures 2. Region Descriptor | S ( output ) – S( input ) | OUPTUT: INPUT Most similar Query images Outlier Image Rejection Retrieval Pipeline Nearest Neighbors Nearest Neighbors 23 B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Results: Benign Query 3. Introducing Space Benign Vote = 100 % Query 4 nearest neighbors B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Results: Benign Query 3. Introducing Space Outlier ! E > 2 100 % Benign Vote = 75 % E = 0.8 E = 1.8 E = 1.0 E = 4.0 Query 4 nearest neighbors B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Results: Neoplastic Query 3. Introducing Space Neoplastic Vote = 100 % Query 4 nearest neighbors B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010

Results: Neoplastic Query 3. Introducing Space Outlier ! E > 2 100 % Neoplastic Vote = 75 % E = 0.6 E = 2.1 E = 1.4 E = 0.9 Query 4 nearest neighbors B. Andr B. André é, MSR , MSR- -INRIA Workshop 2010 INRIA Workshop 2010