Feature Selection Matters for Anchor-Free Object Detection Chenchen - PowerPoint PPT Presentation

Carnegie Mellon Feature Selection Matters for Anchor-Free Object Detection Chenchen Zhu Carnegie Mellon University 04/29/2020

Carnegie Mellon Overview • Background • Motivation • Feature Selection in Anchor-Free Detection • General concept • Network architecture • Ground-truth and loss • Feature selection • Experiments 2

Carnegie Mellon Overview • Background • Motivation • Feature Selection in Anchor-Free Detection • General concept • Network architecture • Ground-truth and loss • Feature selection • Experiments 3

Carnegie Mellon Background A long-lasting challenge: scale variation 4

Carnegie Mellon Background Prior methods addressing scale variation Image pyramid 5

Carnegie Mellon Background Prior methods addressing scale variation Anchor boxes [Ren et al, Faster R-CNN] 6

Carnegie Mellon Background Prior methods addressing scale variation Pyramidal feature hierarchy, e.g. [Liu et al, SSD] 7

Carnegie Mellon Background Prior methods addressing scale variation Feature pyramid network [Lin et al, FPN, RetinaNet] 8

Carnegie Mellon Background Prior methods addressing scale variation Augmentation Balanced FPN [Pang et al, Libra R-CNN] HRNet [Wang et al] NAS-FPN [Ghiasi et al] EfficentDet [Tan et al] 9

Carnegie Mellon Background Combining feature pyramid with anchor boxes • Smaller anchor associated with lower pyramid levels (local fine-grained information) • Larger anchor associated with higher pyramid levels (global semantic information) large anchor-based anchors head medium anchors anchor-based head small anchors anchor-based head feature pyramid 10

Carnegie Mellon Overview • Background • Motivation • Feature Selection in Anchor-Free Detection • General concept • Network architecture • Ground-truth and loss • Feature selection • Experiments 11

Carnegie Mellon Motivation Implicit feature selection by anchor boxes • IoU-based 60x60 ad-hoc heuristics! • Heuristic guided large anchor-based anchors 50x50 head medium anchors anchor-based head 40x40 small anchors anchor-based head feature pyramid 12

Carnegie Mellon Motivation Problem: feature selection by heuristics may not be optimal. Question: how can we select feature level based on semantic information rather than just box size? Answer: allowing arbitrary feature assignment by removing the anchor matching mechanism (using anchor-free methods), selecting the most suitable feature level/levels. 13

Carnegie Mellon Overview • Background • Motivation • Feature Selection in Anchor-Free Detection • General concept • Network architecture • Ground-truth and loss • Feature selection • Experiments 14

Carnegie Mellon Feature Selection in Anchor-Free Detection The general concept • Each instance can be arbitrarily assigned to a single or multiple feature levels. anchor-free instance head feature selection anchor-free head anchor-free head feature pyramid 15

Carnegie Mellon Feature Selection in Anchor-Free Detection Instantiation • Network architecture • Ground-truth and loss • Feature selection: heuristic guided vs. semantic guided 16

Carnegie Mellon Feature Selection in Anchor-Free Detection Network architecture (on RetinaNet) class+box subnets class+box subnets class+box subnets feature pyramid 17

Carnegie Mellon Feature Selection in Anchor-Free Detection Network architecture (on RetinaNet) class subnet WxH WxH WxH x256 x256 xK x4 anchor-free class+box head subnets WxH WxH WxH x256 x256 x4 x4 box subnet 18

Carnegie Mellon Feature Selection in Anchor-Free Detection 19

Carnegie Mellon Feature Selection in Anchor-Free Detection Ground-truth and loss (similar to DenseBox [Huang et al]) anchor-free head class output “car” class for one feature level focal loss WxH xK IoU loss WxH x4 box output 20

Carnegie Mellon Feature Selection in Anchor-Free Detection 21

Carnegie Mellon Feature Selection in Anchor-Free Detection Question: what is a good representation of semantic information to guide feature selection? Our assumption: semantic information is encoded in the network loss . 22

Carnegie Mellon Feature Selection in Anchor-Free Detection focal loss IoU loss focal loss IoU loss focal loss IoU loss feature pyramid 23

Carnegie Mellon Feature Selection in Anchor-Free Detection Question: is it enough to select just one feature level for each instance? 24

Carnegie Mellon Feature Selection in Anchor-Free Detection Can we use similar features from multiple levels to further improve the performance? 25

Carnegie Mellon Feature Selection in Anchor-Free Detection Semantic guided feature selection: soft version instance b RoIAlign RoIAlign feature C selection net RoIAlign feature pyramid 26

Carnegie Mellon Feature Selection in Anchor-Free Detection anchor-free feature selection net head anchor-free head anchor-free head feature pyramid 27

Carnegie Mellon Overview • Background • Motivation • Feature Selection in Anchor-Free Detection • General concept • Network architecture • Ground-truth and loss • Feature selection • Experiments 28

Carnegie Mellon Experiments l Data u COCO Dataset, train set: train2017 , validation set: val2017 , test set: test-dev l Ablation study u Train on train2017 , evaluate on val2017 u ResNet-50 as backbone network l Runtime analysis u Train on train2017 , evaluate on val2017 u Run on a single 1080Ti with CUDA 10 and CUDNN 7 l Compare with state of the arts u Train on train2017 with 2x iterations, evaluate on test-dev 29

Carnegie Mellon Experiments Ablation study: the effect of feature selection Semantic guided Heuristic AP AP 50 AP 75 AP S AP M AP L guided Hard Soft selection selection RetinaNet  (anchor- 35.7 54.7 38.5 19.5 39.9 47.5 based)  35.9 54.8 38.1 20.2 39.7 46.5 Ours  (anchor- 37.0 55.8 39.5 20.5 40.1 48.5 free)  38.0 56.9 40.5 21.0 41.1 50.2 30

Carnegie Mellon Experiments Ablation study: the effect of feature selection Semantic guided Heuristic AP AP 50 AP 75 AP S AP M AP L guided Hard Soft selection selection RetinaNet  (anchor- 35.7 54.7 38.5 19.5 39.9 47.5 based)  35.9 54.8 38.1 20.2 39.7 46.5 Ours  (anchor- 37.0 55.8 39.5 20.5 40.1 48.5 free)  38.0 56.9 40.5 21.0 41.1 50.2 Anchor-free branches with heuristic feature selection can achieve comparable performance with anchor-based counterparts. 31

Carnegie Mellon Experiments Ablation study: the effect of feature selection Semantic guided Heuristic AP AP 50 AP 75 AP S AP M AP L guided Hard Soft selection selection RetinaNet  (anchor- 35.7 54.7 38.5 19.5 39.9 47.5 based)  35.9 54.8 38.1 20.2 39.7 46.5 Ours  (anchor- 37.0 55.8 39.5 20.5 40.1 48.5 free)  38.0 56.9 40.5 21.0 41.1 50.2 Hard version of semantic guided feature selection chooses more suitable feature levels than heuristic guided selection. 32

Carnegie Mellon Visualization of hard feature selection 33

Carnegie Mellon Experiments Ablation study: the effect of feature selection Semantic guided Heuristic AP AP 50 AP 75 AP S AP M AP L guided Hard Soft selection selection RetinaNet  (anchor- 35.7 54.7 38.5 19.5 39.9 47.5 based)  35.9 54.8 38.1 20.2 39.7 46.5 Ours  (anchor- 37.0 55.8 39.5 20.5 40.1 48.5 free)  38.0 56.9 40.5 21.0 41.1 50.2 Hard selection doesn’t fully explore the network potential. Using similarity from multiple features is helpful. 34

Carnegie Mellon Visualization of soft feature selection 35

Carnegie Mellon Visualization of soft feature selection 36

Carnegie Mellon Experiments Ablation study: the effect on different feature pyramids Heuristic Semantic Feature guided guided AP AP 50 AP 75 AP S AP M AP L pyramid selection selection  35.9 54.8 38.1 20.2 39.7 46.5 FPN  38.0 56.9 40.5 21.0 41.1 50.2  36.8 57.2 39.0 22.0 41.0 45.9 BFP  38.8 58.7 41.3 22.5 42.6 50.8 37

Carnegie Mellon Experiments Runtime analysis Runtime Backbone Method AP AP 50 (FPS) RetinaNet 35.7 54.7 11.6 (anchor-based) ResNet-50 Ours 38.8 58.7 14.9 (anchor-free) RetinaNet 37.7 57.2 8.0 (anchor-based) ResNet-101 Ours 41.0 60.7 11.2 (anchor-free) RetinaNet 39.8 59.5 4.5 (anchor-based) ResNeXt-101 Ours 43.1 63.7 6.1 (anchor-free) 38

Carnegie Mellon Experiments Comparison with state of the arts 39