Angular Visual Hardness Beidi Chen Department of Computer Science, - PowerPoint PPT Presentation

Angular Visual Hardness Beidi Chen Department of Computer Science, Rice University Collaborators: Weiyang Liu, Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan kautz, and Anima Anandkumar

PART Motivation 0

Motivation Background Discoveries Applications Conclusion Human Visual Hardness Image Degradation Semantic Ambiguity

Motivation Background Discoveries Applications Conclusion Gap between human visual system and CNNs Hard for Human and Easy for CNNs Easy for Human and Hard for CNNs e m a N Softmax Score s s a l C Nail Oil Filter Golf Ball Radio 0.93 0.998 0.001 0.001 0.2 0.2 1.0 1.0 H u m a n S e l e c t i o n F r e q u e n c y

Agenda Part 1 Background Part 2 Discoveries Part 3 Applications Part 4 Conclusion

PART 01 Background

Motivation Background Discoveries Applications Conclusion Do ImageNet Classifiers Generalize to ImageNet? Human Labeling Interface Recht et al. “Do ImageNet Classifiers Generalize to ImageNet?” ICML 2019

Motivation Background Discoveries Applications Conclusion Loss function of CNNs in visual recognition Softmax cross-entropy loss The magnitude information The angle between feature and classifier Model Confidence

PART Discoveries 02

Motivation Background Discoveries Applications Conclusion Proposal: Angular Visual Hardness (AVH) Given a sample x with label y : 0 1 2 3 4 Angle θ(x,w y) 5 6 Norm || x || 7 8 where, 9 Classifier w y wi is the classifier for the i-th class. Theoretical Foundation : Soudry et al. “The Implicit Bias of Gradient Descent on Separable Data” ICLR 2018

Motivation Background Discoveries Applications Conclusion Simple Example: AVH vs. ||x|| ||x|| Raw data Color map of AVH Color map of ||x|| ||x||

Motivation Background Discoveries Applications Conclusion CNN characteristics vs. human selection frequency

Motivation Background Discoveries Applications Conclusion AVH is well aligned with human frequency Spearman rank correlations

Motivation Background Discoveries Applications Conclusion Discovery 1 AVH hits a plateau very early even when the accuracy or loss is still improving Alexnet ResNet50 VGG19 Alexnet VGG19 Resnet50

Motivation Background Discoveries Applications Conclusion Discovery 2 AVH is an indicator of model’s generalization ability Alexnet ResNet50 VGG19 Alexnet VGG19 Resnet50

Motivation Background Discoveries Applications Conclusion Discovery 3 The norm of feature embeddings keeps increasing during training Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

Motivation Background Discoveries Applications Conclusion Discovery 4 AVH’s correlation with human selection frequency holds across models throughout training Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

Motivation Background Discoveries Applications Conclusion Discovery 5 The norm’s correlation with human selection frequency is not consistent Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

Motivation Background Discoveries Applications Conclusion Conjecture on training dynamic of CNNs ● Softmax cross-entropy loss will first optimize the angles among different classes while the norm will fluctuate and increase very slowly. ● The angles become more stable and change very slowly while the norm increases rapidly. ● Easy examples: the angles get decreased enough for correct classification, the softmax cross-entropy loss can be well minimized by increasing the norm. ● Hard examples: the plateau is cause by unable to decrease the angle to correctly classify examples or increase the norms otherwise hurting loss.

PART Applications 03

Motivation Background Discoveries Applications Conclusion Self-training and Domain Adaptation road sidewalk building wall fence pole tra ffi c lgt tra ffi c sgn vegetation terrain sky person rider car truck bus train motorcycle bike Zou et al. “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training” ECCV

Motivation Background Discoveries Applications Conclusion AVH for Self-training and Domain Adaptation Replace Softmax-based confidence with AVH-based one during sample selection: Similarly, AVH-based pseudo label

Motivation Background Discoveries Applications Conclusion Main Results

Motivation Background Discoveries Applications Conclusion Inner Metric Examples chosen by AVH but not Softmax

Motivation Background Discoveries Applications Conclusion AVH-based loss for Domain Generalization AVH-based Loss:

PART Conclusion 04

Motivation Background Discoveries Applications Conclusion Summary ● Propose AVH as a measure for visual hardness ● Validate that AVH has a statistically significant stronger correlation with human selection frequency ● Make observations on the dynamic evolution of AVH scores during ImageNet training ● Show the superiority of AVH with its application to self-training for unsupervised domain adaptation and domain generalization

Motivation Background Discoveries Applications Conclusion Discussions Trajectory of an adversarial example ● Connection to deep metric learning switching from one class to another ● Connection to fairness in machine learning Trajectory ● Connection to knowledge transfer and curriculum learning ● Uncertainty estimation (Aleatoric and Epistemic) ● Adversarial Example: A Counter Example?

THANKS Paper URL Contact: beidi.chen@rice.edu