Generalized Cross Entropy Loss for Noisy Labels Zhilu Zhang and Mert - PowerPoint PPT Presentation

Generalized Cross Entropy Loss for Noisy Labels Zhilu Zhang and Mert R. Sabuncu Cornell University Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 1

Cornell University Motivation Deep neural networks: • Often need lots of clean labeled data - can be expensive to obtain • Can overfit to noisy labels [Zhang et al. 2016] Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 2

Cornell University Symmetric Loss • A loss function is symmetric if • Symmetric loss can be tolerant to noisy labels [Ghosh et al. 2017] • MAE for classification with probabilistic outputs is symmetric Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 3

Cornell University Limitations of MAE • MAE is noise-robust but can converge to lower accuracy Much slower convergence Slight gap in test accuracy ResNet on CIFAR-10 Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 4

Cornell University Limitations of MAE • MAE is noise-robust but can converge to lower accuracy Using MAE, the highest ~ 20% accuracy achieved is 38.29% in 2000 epochs, and CCE achieved better performance after 7 epochs! ResNet on CIFAR-100 Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 4

Cornell University Generalized Cross Entropy (Lq Loss) CCE • Good convergence, but prone to label noise MAE • More noise robust, but bad convergence Use the Box-Cox Transformation to combine them Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 5

Cornell University Generalized Cross Entropy (Lq Loss) CCE MAE ! = 0 ! " [0,1] ! = 1 • Lq loss has bounded sum of losses for non zero q • The tighter the bound, the more noise robust the Lq loss Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 5

Cornell University Generalized Cross Entropy (Lq Loss) CCE MAE ! = 0 ! " [0,1] ! = 1 ResNet on CIFAR-10 Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 5

Cornell University Truncated Lq Loss • Propose the truncated Lq loss • Often has tighter bound • Use alternative convex search algorithm for optimization Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 6

Cornell University Experiments CIFAR-10 87.62% 87.13% 40% NOISE • ResNet on CIFAR-10, 67% 81.88% CIFAR-100 and 89.70% 89.83% 20% NOISE FASHION-MNIST with 83.72% 86.98% synthetic noise 60.00% 65.00% 70.00% 75.00% 80.00% 85.00% 90.00% 95.00% CIFAR-100 62.64% 61.77% 40% NOISE 9.03% • Consistent improvements 48.20% over CCE and MAE 67.61% 66.81% 20% NOISE 15.80% 58.72% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% Trunc Lq Lq (q = 0.7) MAE CCE Generalized Cross Entropy Loss for Noisy Labels – Poster # 101 7

Cornell University • Thank you very much for your attention! • Hope to see you at Poster #101