Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels Lu - PowerPoint PPT Presentation

Proprietary + Confidential Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels Lu Jiang, Di Huang, Mason Liu, Weilong Yang. Lu Jiang Di Huang Mason Liu Weilong Yang

Deep Learning on Noisy Labels Deep networks are very good at memorizing the noisy labels ( Zhang et al. 2017) . Memorization leads to a critical issue since noisy labels are inevitable in big data. Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." ICLR (2017). Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Controlled Noisy Labels Pergorming controlled experiments on noisy labels is essential in existing works. Correct label Wrong label noise level=20% 80% 40% Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Issues with Controlled Synthetic Labels Issue: existing studies only pergorm controlled experiments on synthetic labels (or random labels). 1. Contradictory fjndings. For example, DNNs are robust to massive label noise? Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Issues with Controlled Synthetic Labels Issue: existing studies only pergorm controlled experiments on synthetic labels (or random labels). 1. Contradictory fjndings. For example, DNNs are robust to massive label noise? (Zhang et al. 2017) (Rolnick et al. 2017) Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." ICLR (2017). Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Rolnick, D., et al. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694, 2017.

Issues with Controlled Synthetic Labels Issue: existing studies only pergorm controlled experiments on synthetic labels (or random labels). 2. Inconsistent empirical results We found that methods that pergorm well on synthetic noise may not work as well on real-world noisy labels. ● Motivation of our research project. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Our Contributions: We establish the fjrst benchmark of controlled real-world label noise (from the web). 1. 2. A simple but highly efgective method to overcome both synthetic and real-world noisy labels (best results on the WebVision benchmark) 3. We conduct the largest study by far into understanding deep neural networks trained on noisy labels across difgerent noise levels, noise types, network architectures, methods, and training setuings. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Contribution I: New Dataset First benchmark of controlled real-world label noise Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Datasets of noisy training labels real-world label synthetic noise image corruption (Hendrycks & Dietterich, 2019) content (Zhang et al., 2019a) adversarial attack Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Datasets of noisy training labels WebVision, uncontrolled Clothing1M etc. real-world our work ? Controlled → Missing label (Zhang et al. 2017) synthetic Controlled noise image corruption (Hendrycks & Dietterich, 2019) content (Zhang et al., 2019a) adversarial attack Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Construction of controlled synthetic label noise 1. Starts with a well-labeled dataset. Correct label 2. Randomly selects p% examples. 3. Independently flips each label to a random incorrect class (symmetric or asymmetric). 4. Repeats Step 1-3 with a different p (noise level) Mini-ImageNet Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Construction of controlled synthetic label noise 1. Starts with a well-labeled dataset. Correct label 2. Randomly selects p% examples. 3. Independently flips each label to a random incorrect class (symmetric or asymmetric). 4. Repeats Step 1-3 with a different p (noise level) noise level p = 20% Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Construction of controlled synthetic label noise 1. Starts with a well-labeled dataset. Correct label Wrong label 2. Randomly selects p% examples. 3. Independently flips each label to a random incorrect class (symmetric or asymmetric). 4. Repeats Step 1-3 with a different p (noise level) noise level p = 20% Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Construction of controlled synthetic label noise 1. Starts with a well-labeled dataset. Correct label Wrong label 2. Randomly selects p% examples. 3. Independently flips each label to a random incorrect class (symmetric or asymmetric). 4. Repeats Step 1-3 with a different p (noise level) noise level p = 40% This process generates controlled synthetic label noise. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Construction of uncontrolled web label noise label correctness unknown noise level p = ??% This process can automatically collect noisy labeled images from the web. But the noise level is fixed and unknown (unsuitable for controlled studies). Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

From uncontrolled to controlled noise Correct label Wrong label correct incorrect correct noise level p is known We have each retrieved image annotated by 3-5 works using Google Cloud Labeling Service https://cloud.google.com/ai-platform/data-labeling/docs Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Construction of our dataset 1. Starts with a well-labeled dataset. Correct label wrong label 2. Randomly selects p% examples. 3. Replaces the clean images with the incorrectly labeled web images while leaving the label unchanged*. 4. Repeats Step 1-3 with a different p (noise level) noise level p = 20% *We show that an alternative way to construct the dataset by removing all image-to-image results leads to consistent results in the Appendix Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Our Dataset: Controlled Noisy Labels from the Web Manually annotate 212K images through 800K annotations. We establish the fjrst benchmark of controlled web label noise for two classifjcation tasks: coarse (Mini-ImageNet) and fjne-grained (Stanford Cars) Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Our Dataset: Controlled Noisy Labels from the Web Manually annotate 212K images through 800K annotations. We establish the fjrst benchmark of controlled web label noise for two classifjcation tasks: coarse (Mini-ImageNet) and fjne-grained (Stanford Cars) Red noise: label noise from the web Blue noise: synthetic label noise Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Mini-ImageNet Stanford Cars Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Contribution II: New Method to overcome synthetic and real-world label noise Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Overview Problem : Given a noisy dataset of some unknown noise level, fjnd a robust learning method that generalizes well on the clean test data. Prior works : Many techniques tackle it from multiple directions, among others, Regularization (Azadi et al., 2016; Noh et al., 2017; etc.) ● Label cleaning (Reed et al., 2014; Goldberger, 2017; Li et al., 2017b; Veit et al., 2017; Song et al., 2019; etc.) ● Example weighting (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016; etc.) ● Data augmentation (Zhang et al., 2018; Cheng et al., 2019) ● … ... ● Our Method: a simple and efgective method called MentorMix. Why need yet another method? We show our method overcomes both synthetic and real-world noisy labels. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels Lu - PowerPoint PPT Presentation

Proprietary + Confidential Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels Lu Jiang, Di Huang, Mason Liu, Weilong Yang. Lu Jiang Di Huang Mason Liu Weilong Yang Deep Learning on Noisy Labels Deep networks are very good at

S8941 SYNTHETIC LABEL DATA FOR TRAINING DEEP LEARNING ISR ALGORITHMS WILL RORRER, PROGRAM

17,000 Young or old, once you lose your hearing you can never get it back New

Learning to denoise without clean data Joshua Batson hep-ai seminar 10/18/18 Noisy data is

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann

Distributed Synthetic Data Platform for Deep Learning Applications BITCOIN OR ETHER AMAZON DEEP

Interactive Clustering Barna Saha Clustering Learning over Noisy Data Learn a classifier or

Combating Label Noise in Deep Learning using Abstention Speaker: Sunil Thulasidasan

Identifiability of dynamic networks with noisy and noise-free nodes Paul Van den Hof Coworkers:

NOISE AT WORK AWARENESS SESSION FOR SUPERVISORS OBJECTIVES Understand what is noise

4 Slides! be S.T.I.L.L. in the midst of noise? Read Mark 6:30-34 aloud. S ___________START BY

From microscopic to macroscopic noise: the dynamics of transitions around noisy networks George

Image Classification Canxiang Yan, Cheng Niu and Jie Zhou WeChat AI CONTENT Noise in

Detecting Changes and Anomalies in Noisy Text Streams Jerry Wright Networking and Services

LEARNING IN GAMES WITH NOISY PAYOFF OBSERVATIONS Background and motivation Preliminaries The

Learning Sound Event Classifiers from Web Audio with Noisy Labels Eduardo Fonseca 1 , Manoj Plakal

E9 205 Machine Learning for Signal Procesing Deep Learning for Audio and Vision 20-11-2019

EchoPanel EchoPanel Acoustics 101 NRC = Noise Reduction Coefficient Noisy office/restaurant

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Unsupervised, Fast and Precise Recognition of Digital Arcs in Noisy Images T. P . NGUYEN, B.

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

Robust Power Estimation and Simultaneous Switching Noise Prediction Methods Using Machine Learning

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Linear Factor Models Lecture slides for Chapter 13 of Deep Learning www.deeplearningbook.org Ian

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning