Processing Megapixel Images with Deep Attention-Sampling Models - PowerPoint PPT Presentation

Processing Megapixel Images with Deep Attention-Sampling Models Angelos Katharopoulos & Fran¸ cois Fleuret ICML, June 11, 2019 Funded by

How do DNNs process large images? Cropping and downsampling to a manageable resolution (e.g. 224 × 224) Dividing the image into patches and processing them separately ∗ image taken from the Imagenet dataset A. Katharopoulos Deep Attention-Sampling Models 2/9

Our contributions ◮ Disentangle the computational and memory requirements from the input resolution. ◮ Sample from a soft attention to only process a fraction of the image in high resolution. ◮ We derive gradients through the sampling for all parameters and train our models end-to-end. A. Katharopoulos Deep Attention-Sampling Models 3/9

Soft Attention Given an input x we define a neural network Ψ( x ) that uses attention � K � � � � Ψ( x ) = g a ( x ) i f ( x ) i = g E I ∼ a ( x ) [ f ( x ) I ] , i =1 where f ( x ) ∈ R K × D are the features and a ( x ) ∈ R K + is the attention distribution. A. Katharopoulos Deep Attention-Sampling Models 4/9

Attention Sampling We approximate Ψ( x ) by Monte Carlo    1 �  where Q = { q i ∼ a ( x ) | i ∈ { 1 , 2 , . . . , N }} . Ψ( x ) ≈ g f ( x ) q N q ∈ Q We show that ◮ Sampling from the attention is optimal to approximate Ψ( x ) if � f ( x ) i � = � f ( x ) j � ∀ i , j ◮ We can compute the gradients both for the parameters a ( · ) and f ( · ) A. Katharopoulos Deep Attention-Sampling Models 5/9

Processing Megapixel Images with Deep Attention-Sampling Models A. Katharopoulos Deep Attention-Sampling Models 6/9

Processing Megapixel Images with Deep Attention-Sampling Models A. Katharopoulos Deep Attention-Sampling Models 6/9

Processing Megapixel Images with Deep Attention-Sampling Models A. Katharopoulos Deep Attention-Sampling Models 6/9

Processing Megapixel Images with Deep Attention-Sampling Models A. Katharopoulos Deep Attention-Sampling Models 6/9

Processing Megapixel Images with Deep Attention-Sampling Models A. Katharopoulos Deep Attention-Sampling Models 6/9

Processing Megapixel Images with Deep Attention-Sampling Models A. Katharopoulos Deep Attention-Sampling Models 6/9

Qualitative evaluation of the attention distribution (1) Full Image Epithelial Cells Ilse et al. (2018) Attention Sampling A. Katharopoulos Deep Attention-Sampling Models 7/9

Qualitative evaluation of the attention distribution (2) Ground Truth Ilse et al. (2018) Attention Sampling Extracted patch A. Katharopoulos Deep Attention-Sampling Models 8/9

Thank you for your time! Speed limit sign detection 0 . 30 0 . 30 0 . 25 0 . 25 Test Error Test Error 0 . 20 0 . 20 0 . 15 0 . 15 0 . 10 0 . 10 0 500 1000 1500 20 40 60 80 100 Memory/sample (MB) Time/sample (s) Come talk to us at poster #3 at Pacific Ballroom . A. Katharopoulos Deep Attention-Sampling Models 9/9