a kernel theory of modern data augmentation
play

A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert - PowerPoint PPT Presentation

Nov 15, 2022 •223 likes •485 views

th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:309:00 P 9:00 PM A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert

  1. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert Gu, Alex Ratner, Virginia Smith, Chris De Sa, Chris Ré

  2. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy…

  3. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy… 3.7 pt. average gain across top ten CIFAR-10 models

  4. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy… 3.7 pt. average gain across top ten CIFAR-10 models 13.9 pt. average gain for CIFAR-100

  5. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy… 3.7 pt. average gain across top ten CIFAR-10 models 13.9 pt. average gain for CIFAR-100 A form of weak supervision: expresses domain knowledge (invariance)

  6. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM … but is not well understood

  7. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM … but is not well understood How does data augmentation affect the model? • Learning process • Parameters and decision surface

  8. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Augmentation as sequence modeling • TANDA [Ratner et al., 2017] • AutoAugment [Cubuk et al., 2018]

  9. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Augmentation as sequence modeling • TANDA [Ratner et al., 2017] • AutoAugment [Cubuk et al., 2018] Model augmentation as a Markov chain

  10. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Augmentation as kernels Base classifier: k-nearest neighbors + Data augmentation = Asymptotic kernel classifier

  11. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers

  12. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o x x o x x o x x Invariance

  13. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance

  14. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance Practical utility

  15. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance Practical utility speeding up training

  16. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance Practical utility as a speeding up diagnostic training

  17. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Model of data augmentation: kernel classifier n 1 X ` ( w > � ( x i )) min Non-augmented: w n Loss function i =1 Feature map

  18. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Model of data augmentation: kernel classifier n 1 X ` ( w > � ( x i )) min Non-augmented: w n Loss function i =1 Feature map n 1 X E z i ⇠ T ( x i ) ` ( w > � ( z i )) min Augmented: w n i =1 Transformed versions of data point

  19. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation effects n n 1 E z i ⇠ T ( x i ) ` ( w > � ( z i )) ≈ 1 X X ` ( w > E z i ⇠ T ( x i ) � ( z i )) n n i =1 i =1 Average of augmented features (i.e. kernel mean embedding)

  20. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation effects n n 1 E z i ⇠ T ( x i ) ` ( w > � ( z i )) ≈ 1 X X ` ( w > E z i ⇠ T ( x i ) � ( z i )) n n i =1 i =1 Average of augmented features (i.e. kernel mean embedding) 1 st order effect: induces invariance by feature averaging

  21. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation effects n n 1 E z i ⇠ T ( x i ) ` ( w > � ( z i )) ≈ 1 X X ` ( w > E z i ⇠ T ( x i ) � ( z i )) n n i =1 i =1 Average of augmented features (i.e. kernel mean embedding) 1 st order effect: 2 nd order effect: reduces induces invariance model complexity by feature via a data-dependent averaging regularization

  22. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A diagnostic: kernel alignment metric ψ ( x ) = E z ∼ T ( x ) φ ( z ) Averaged features: Kernel target alignment [Cristianini et al., 2002]: how well separated are features from different classes

  23. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A diagnostic: kernel alignment metric Kernel alignment Kernel alignment MNIST

  24. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A diagnostic: kernel alignment metric Kernel alignment Kernel alignment MNIST Kernel alignment correlates with accuracy.

  25. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Summary • Data augmentation + k-NN = asymptotic kernel classifier. • Data augmentation induces invariance and regularizes. • Application in speeding up training and diagnostics. Tri Dao trid@stanford.edu Poster #227 on Tuesday Jun 11 th at 6:30pm

Recommend

Data Augmentation in NLP 2020-03-21 Xiachong Feng Outline Why we need Data Augmentation?

Data Augmentation in NLP 2020-03-21 Xiachong Feng Outline Why we need Data Augmentation?

Data Augmentation in NLP 2020-03-21 Xiachong Feng Outline Why we need Data Augmentation? Data Augmentation in CV Widely Used Methods EDA Back-Translation Contextual Augmentation Methods based on Pre-trained Language

483 views • 46 slides
Population Based Augmentation Efficient Learning of Augmentation Policy Schedules Daniel Ho , Eric

Population Based Augmentation Efficient Learning of Augmentation Policy Schedules Daniel Ho , Eric

Population Based Augmentation Efficient Learning of Augmentation Policy Schedules Daniel Ho , Eric Liang, Ion Stoica, Pieter Abbeel, Xi Chen Efficiently learn data augmentation policies to improve neural network performance. Data Augmentation

335 views • 8 slides
SwitchOut: An Efficient Data Augmentation for Neural Machine Translation Xinyi Wang , Hieu

SwitchOut: An Efficient Data Augmentation for Neural Machine Translation Xinyi Wang , Hieu

SwitchOut: An Efficient Data Augmentation for Neural Machine Translation Xinyi Wang , Hieu Pham , Zihang Dai, Graham Neubig November 2, 2018 :equal contribution 1 / 41 Data Augmentation Neural models are data hungry, while

782 views • 41 slides
Does Data Augmentation Lead to Positive Margin? Dimitris Po-Ling Loh Shashank Rajput* Zhili

Does Data Augmentation Lead to Positive Margin? Dimitris Po-Ling Loh Shashank Rajput* Zhili

Does Data Augmentation Lead to Positive Margin? Dimitris Po-Ling Loh Shashank Rajput* Zhili Feng* Zachary Charles Papailiopoulos * Equal Contribution Data Augmentation (DA) DA means increasing the training set artificially. Used to

379 views • 15 slides
Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20,

Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20,

Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20, 2020 Table of contents 1. Motivation 2. Kernel methods 3. Basic kernel operations in feature space 4. Kernel-based algorithms 5. Summary 1/24

729 views • 31 slides
In Deep Learning Anima Anandkumar & Zachary Lipton DATA AUGMENTATION To improve

In Deep Learning Anima Anandkumar & Zachary Lipton DATA AUGMENTATION To improve

Addressing Data Scarcity In Deep Learning Anima Anandkumar & Zachary Lipton DATA AUGMENTATION To improve generalization, augment training data. In computer vision. Simple techniques: rotation, cropping, noise In speech

319 views • 27 slides
Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M.

Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M.

Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M. Magdon-Ismail CSCI 4100/6100 recap: Large Margin is Better Controling Overfitting Non-Separable Data 0.08 random hyperplane 2 w t w + C N E

576 views • 18 slides
Modern MDL meets Data Mining Insight, Theory, and Practice Jilles Kenji Vreeken Yamanishi

Modern MDL meets Data Mining Insight, Theory, and Practice Jilles Kenji Vreeken Yamanishi

2019 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Modern MDL meets Data Mining Insight, Theory, and Practice Jilles Kenji Vreeken Yamanishi CISPA Helmholtz Center The University of Tokyo for Information

636 views • 39 slides
1. Mid-Ocean Ridges 1. Mid-Ocean Ridges How Do We Know? We know about Mid-Ocean Ridges

1. Mid-Ocean Ridges 1. Mid-Ocean Ridges How Do We Know? We know about Mid-Ocean Ridges

Modern Theory The Modern Theory of Plate Tectonics is built on Alfred Wegeners theory of Continental Drift . Modern Theory of Two new discoveries (discoveries over the last 60 years) helped scientists Plate Tectonics develop the

461 views • 9 slides
image-augmentation April 9, 2019 1 Image Augmentation In [1]: % matplotlib inline import d2l

image-augmentation April 9, 2019 1 Image Augmentation In [1]: % matplotlib inline import d2l

image-augmentation April 9, 2019 1 Image Augmentation In [1]: % matplotlib inline import d2l import mxnet as mx from mxnet import autograd, gluon, image, init, nd from mxnet.gluon import data as gdata, loss as gloss, utils as gutils import sys

109 views • 8 slides
1 CPU Events: Interrupts and Exceptions CPU Events: Interrupts and Exceptions Protecting Entry

1 CPU Events: Interrupts and Exceptions CPU Events: Interrupts and Exceptions Protecting Entry

Processes and the Kernel Processes and the Kernel The kernel sets processes up process in private data data execution Processes, Protection and the Kernel: Processes, Protection and the Kernel: virtual contexts to address

540 views • 9 slides
Diverse Paraphrasing and its Effectiveness in Data Augmentation Ashutosh Kumar*, Satwik

Diverse Paraphrasing and its Effectiveness in Data Augmentation Ashutosh Kumar*, Satwik

Diverse Paraphrasing and its Effectiveness in Data Augmentation Ashutosh Kumar*, Satwik Bhattamishra*, Manik Bhandari, Partha Talukdar Machine and Language Learning Lab (MALL) Indian Institute of Science, Bangalore *Equal Contributions

905 views • 87 slides
Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Eleonora

Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Eleonora

Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Eleonora Cagli 1 , 3 ecile Dumas 1 C Emmanuel Prouff 2 , 3 1

1.28k views • 100 slides
Improving Molecular Design by Stochastic Iterative Target Augmentation Kevin Yang, Wengong Jin,

Improving Molecular Design by Stochastic Iterative Target Augmentation Kevin Yang, Wengong Jin,

Improving Molecular Design by Stochastic Iterative Target Augmentation Kevin Yang, Wengong Jin, Kyle Swanson, Regina Barzilay, Tommi Jaakkola 15-Second Overview Data augmentation approach: improve molecular optimization SOTA by > 10%

621 views • 50 slides
Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation

Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation

Motivation Related work Proposed approach Empirical results Conclusion Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation Devansh Bisla , Anna Choromanska, Russell S. Berman, Jennifer A. Stein, David

359 views • 15 slides
Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity

Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity

Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity Kernels in Different Applications M. Magdon-Ismail CSCI 4100/6100 recap: The Kernel Allows Us to Bypass Z -space Solve the QP 1

356 views • 12 slides
Paraphrase generation: adversarial examples / data augmentation CS 685, Fall 2020 Advanced

Paraphrase generation: adversarial examples / data augmentation CS 685, Fall 2020 Advanced

Paraphrase generation: adversarial examples / data augmentation CS 685, Fall 2020 Advanced Natural Language Processing Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst stuff from last time HW1

692 views • 53 slides
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich

Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich

Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology ACL 2019 Ran Zmigrod, Sebastian J. Mielke , Hanna Wallach, Ryan Cotterell University of Cambridge // Johns Hopkins University // Microsoft

778 views • 58 slides
ECE 417 Fall 2018 Lecture 19: Mini-Batch Training and Data Augmentation Mark Hasegawa-Johnson

ECE 417 Fall 2018 Lecture 19: Mini-Batch Training and Data Augmentation Mark Hasegawa-Johnson

Annealing Mini-Batch Training Data Augmentation Conclusions ECE 417 Fall 2018 Lecture 19: Mini-Batch Training and Data Augmentation Mark Hasegawa-Johnson University of Illinois October 25, 2018 Annealing Mini-Batch Training Data

501 views • 28 slides
Data Augmentation for Context-Sensitive Neural Lemmatization Using Inflection Tables and Raw

Data Augmentation for Context-Sensitive Neural Lemmatization Using Inflection Tables and Raw

Data Augmentation for Context-Sensitive Neural Lemmatization Using Inflection Tables and Raw Text Toms Bergmanis, Sharon Goldwater Lemmatization Sing Plural NOM ce cei GEN cea ceu DAT ceam ceiem ce ACC ceu

370 views • 26 slides
Modern MDL Meets Data Mining Insight, Theory, and Practice Part IV Dynamic Setting Kenji

Modern MDL Meets Data Mining Insight, Theory, and Practice Part IV Dynamic Setting Kenji

Modern MDL Meets Data Mining Insight, Theory, and Practice Part IV Dynamic Setting Kenji Yamanishi Graduate School of Information Science and Technology, the University of Tokyo August 4 th 2019 KDD Tutorial Part IV. Dynamic Setting

944 views • 60 slides
Embedding and Data Augmentation yanweifu@fudan.edu.cn

Embedding and Data Augmentation yanweifu@fudan.edu.cn

One-shot Learning in Semantic Embedding and Data Augmentation yanweifu@fudan.edu.cn http://yanweifu.github.io One-shot Learning: learning object categories from just a few images, by incorporating

724 views • 61 slides
MODERN 1 MODERN 2 MODERN 3 MODERN 4 MODERN A peep at some distant orb has power to raise

MODERN 1 MODERN 2 MODERN 3 MODERN 4 MODERN A peep at some distant orb has power to raise

MODERN 1 MODERN 2 MODERN 3 MODERN 4 MODERN A peep at some distant orb has power to raise and purify our thoughts like a strain picture, or a passage from the grander poets. It always does one good. 5 MODERN 6 MODERN 7 MODERN 8

524 views • 24 slides
Jitk: A trustworthy in-kernel interpreter infrastructure Xi Wang, David Lazar, Nickolai Zeldovich,

Jitk: A trustworthy in-kernel interpreter infrastructure Xi Wang, David Lazar, Nickolai Zeldovich,

Jitk: A trustworthy in-kernel interpreter infrastructure Xi Wang, David Lazar, Nickolai Zeldovich, Adam Chlipala, Zachary Tatlock MIT and University of Washington Modern OSes run untrusted user code in kernel In-kernel interpreters

1k views • 30 slides

More recommend