HARDNET: CONVOLUTIONAL NETWORK FOR LOCAL IMAGE DESCRIPTION A n a - PowerPoint PPT Presentation

HARDNET: CONVOLUTIONAL NETWORK FOR LOCAL IMAGE DESCRIPTION A n a s t a s i i a Mi s h c h u k , D m y t r o Mi s h k i n , F i l i p R a d e n o v i c J i r i Ma t a s

OUTLINE  Short review of methods for learning of local descriptors  The L2-Net  HardNet loss and architecture  Benchmarks 2

TRAINING DATA 3 s e t s , 4 0 0 k p a t c h e s e a c h : • L i b e r t y ( s h o w n ) • N o t r e d a me • Y o s e mi t e S i z e : 6 4 x 6 4 , g r a y s c a l e . O b t a i n e d f r o m S f M mo d e l , 3 D p o i n t → D o G k e y p o i n t s U s e d i n a l l l e a r n e d d e s c r i p t o r s me a n t i o n e d i n t h i s p r e s e n t a t i o n 3 D i s c r i mi n a n t L e a r n i n g o f L o c a l I ma g e D e s c r i p t o r s B r o w n e t a l , P A MI 2 0 1 0

CONVEXOPT (SIMONYAN ET AL, 2012) C o n v e x o p t i mi z a t i o n p r o b l e m Global margin loss 4 S i mo n y a n e t a l , E C C V 2 0 1 2

MATCHNET 16 16 32 32 64 3x3 Conv 64 5x5 Conv 7x7 Conv 2x2 MP/2 pad 1 2x2 MP/2 pad 2 pad 1 ReLU ReLU ReLU 96 64 64 24 24 1 16 16 8 1 1 1 1 3x3 Conv 3x3 Conv 1x1 Conv 1x1 Conv 1x1 Conv 8x8 Conv 3x3 MP/2 pad 1 pad 1 Sofumax ReLU ReLU ReLU 2 128 256 256 ReLU ReLU 96 64 64 Wo r k s w e l l , b u t r e l y o n me t r i c n e t w o r k . A p p r o x i ma t e k N N me t h o d s , e . g . F L A N N c a n n o t b e a p p l i e d d i r e c t l y 5 H a n e t a l , C V P R 2 0 1 5 .

DEEPCOMPARE 16 16 32 32 64 3x3 Conv 64 5x5 Conv 7x7 Conv 2x2 MP/2 pad 1 2x2 MP/2 pad 2 pad 3 ReLU ReLU ReLU 256 192 192 96 96 1 8 1 1 1x1 Conv 1x1 Conv 8x8 Conv 2x2 MP/2 Sigmoid 1 ReLU 256 ReLU 256 256 Wo r k s w e l l , b u t r e l y o n me t r i c n e t w o r k . A p p r o x i ma t e k N N me t h o d s , e . g . F L A N N c a n n o t b e a p p l i e d d i r e c t l y 6 Z a g o r u y k o a n d K o mo d a k i s , C V P R 2 0 1 5

D e e p D e s c ( S i mo - S e r r a e t a l , 2 0 1 5 ) R e l a t i v e l y s h a l l o w a n d f a s t C N N . H a r d n e g a t i v e mi n i n g : C o n t r a s t i v e l o s s o n L 2 d i s t a n c e 64 58 3x3 29 23 L2Pool/4 1 4x4 6x6 2x2 5x5 L2Pool/3 8 4 7x7 Conv L2pool/2 Conv Conv 128 TanH TanH TanH 64 64 128 32 32 1 • E v e n s h a l l o w e r a n d f a s t e r C N N , T F e a t ( B a l n t a s e t a l , 2 0 1 6 ) • h a r d - n e g a t i v e mi n i n g : b y a n c h o r s w a p i n t r i p l e t . • t r i p l e t ma r g i n l o s s o n L 2 d i s t a n c e 32 26 13 1 8 8x8 Conv 2x2 MP/2 7x7 Conv 6x6 Conv 9 TanH 64 128 TanH 32 TanH 32 1 S i mo - S e r r a e t a l , I C C V 2 0 1 5 . B a l n t a s e t a l , B MV C 2 0 1 6

DESCRIPTOR COMPARISON D e s c r . # l a y e r s L o s s Ha r d mi n i n g K d - t r e e w/ p a r a ms r e a d y C o n v e x O p t 1 G l o b a l ma r g i n - + D e e p D e s c 3 C o n t r a s t i v e + + T F e a t 3 T r i p l e t ma r g i n + / - + Ma t c h N e t 8 C r o s s e n t r o p y - - D e e p C o mp 5 H i n g e - - 1 0 B a l n t a s e t a l , B MV C 2 0 1 6

L2NET. TIAN ET AL (CVPR 2017) 16 16 32 32 32 3x3 Conv 3x3 Conv 3x3 Conv 3x3 Conv 3x3 Conv pad 1 /2 pad 1 pad 1 /2 pad 1 pad 1 BN + ReLU BN + ReLU BN + ReLU BN + ReLU BN + ReLU 64 64 32 32 1 8 8 1 3x3 Conv 8x8 Conv pad 1 BN+ L2Norm BN + ReLU 128 128 128 1 1

L2NET: LOSS TERMS S o f t ma x o v e r r o w / c o l u mn o f d i s t a n c e ma t r i x 1 3

L2NET: LOSS TERMS S o f t ma x o v e r r o w / c o l u mn o f d i s t a n c e ma t r i x P e n a l t y o n d e s c r i p t o r c o mp o n e n t s c o r r e l a t i o n 1 4

L2NET: LOSS TERMS S o f t ma x o v e r r o w / c o l u mn o f d i s t a n c e ma t r i x P e n a l t y o n d e s c r i p t o r c o mp o n e n t s c o r r e l a t i o n S o f t ma x o v e r r o w / c o l u mn o f d i s t a n c e ma t r i x o f i n t e r me d i a t e 1 5 f e a t u r e s

HARDNET Triplet margin loss for hard negative P e n a l t y o n d e s c r i p t o r c h a n n e l s c o r r e l a t i o n S o f t ma x o v e r r o w / c o l u mn o f d i s t a n c e ma t r i x o f i n t e r me d i a t e 1 6 f e a t u r e s

HARDNET (OURS) 32 3x3 Conv 3x3 Conv 3x3 Conv 3x3 Conv 3x3 Conv pad 1 /2 pad 1 pad 1 /2 pad 1 pad 1 BN + ReLU BN + ReLU BN + ReLU BN + ReLU BN + ReLU 64 64 32 32 1 8 8 1 3x3 Conv 8x8 Conv pad 1 BN+ L2Norm BN + ReLU 128 128 128 1 7

BATCH SIZE INFLUENCE 1 8

DESCRIPTOR COMPARISON D e s c r . # l a y e r s L o s s Ha r d mi n i n g K d - t r e e w/ p a r a ms r e a d y C o n v e x O p t 1 G l o b a l ma r g i n - + D e e p D e s c 3 C o n t r a s t i v e + + T F e a t 3 T r i p l e t ma r g i n + / - + Ma t c h N e t 8 C r o s s e n t r o p y - - D e e p C o mp 5 H i n g e - - L 2 N e t 7 S o f t Ma x + + Ha r d N e t 7 T r i p l e t ma r g i n + + 1 9

Loss comparison on patch triplets 2 0

LOSSES COMPARISON, DERIVATIVES 2 1

LOSSES COMPARISON, DERIVATIVES s t n e e i l d a mp r g l l ma a t S x n e e i e d v i a t r a g g o e n N m o r f 2 2

LOSSES COMPARISON Contrastive Softmax (L2Net) Triplet margin FPR, Brown 0.009 0.009 0.006 Yos mAUC, W1BS 0.072 0.083 0.083 2 3 mAUC, HP-T 0.153 0.157 0.164

Results 2 4

RESULTS: BROWN DATASET 2 5

RESULTS: W1BS DATASET 2 6 N u i s a n c e f a c t o r : A p p e a r a n c e G e o me t r y L i g h t i n g S e n s o r Mi s h k i n e t a l , B MV C 2 0 1 5

HPATCHES DATASET D o G , H e s s i a n , H a r r i s – i n r e f . i ma g e ~ 1 3 0 0 p a t c h e s p e r i ma g e k e p t . R e p r o j e c t e d t o o t h e r i ma g e s w i t h 3 l e v e l s o f “ a ffjn e f r a me n o i s e ” a d d e d V : 5 7 i ma g e s i x p l e t s – p h o t o me t r i c c h a n g e s I : 5 9 i ma g e s i x p l e t s – g e o me t r i c c h a n g e s 2 7 B a l n t a s e t a l , C V P R 2 0 1 7

RESULTS: HPATCHES 2 8

RESULTS: MATCHING WITH VIEW SYNTH O n p a r wi t h D a t a s e t s a r e a l r e a d y s a t u r a t e d R o o t S I F T S t i l l c h a l l e n g i n g d u e t o mu l t i p l e n u i s a n c e f a c t o r s 2 9 Z i t n i c k a n d R a mn a t h , 2 0 1 1 , Mi s h k i n e t a l 2 0 1 5 , Mi k o l a j c z y k e t a l . 2 0 1 3 , H a u a g g e a n d S n a v e l y , 2 0 1 2 , K e l ma n e t a l , 2 0 0 7 , F e r n a n d o e t a l . 2 0 1 4

RESULTS: BOW OXFORD5K & PARIS 6K 3 0 P h i l b i n e t a l 2 0 0 7 , P h i l b i n e t a l 2 0 0 8

RESULTS: HQE OXFORD5K & PARIS 6K 3 1

Thank you for attention PDF: https://arxiv.org/abs/1705.10872 Source and models: https://github.com/DagnyT/hardnet 3 2