robust model training and generalisation with
play

Robust model training and generalisation with Studentising flows - PowerPoint PPT Presentation

Sep 21, 2022 •673 likes •983 views

Robust model training and generalisation with Studentising flows Simon Alexanderson Gustav Eje Henter {simonal,ghe}@kth.se Division of Speech, Music and Hearing (TMH), School of Electrical Engineering and Computer Science (EECS), KTH Royal

  1. Robust model training and generalisation with Studentising flows Simon Alexanderson Gustav Eje Henter {simonal,ghe}@kth.se Division of Speech, Music and Hearing (TMH), School of Electrical Engineering and Computer Science (EECS), KTH Royal Institute of Technology, Stockholm, Sweden 2020-07-11 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 1 / 11

  2. One-slide summary • We propose replacing Gaussian base distributions Z in normalising flows with multivariate Student’s t -distributions • Studentising flows • Our proposal is motivated through statistical robustness • Experiments show that the proposal stabilises training and leads to better generalisation Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 2 / 11

  3. Outline • What is robustness? • Robustness sits in the tails • Tails of flow-based models • Experimental findings Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 3 / 11

  4. Why do we need robustness? Generate some 1D standard normal data and fit a Gaussian: 0 . 3 0 . 25 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  5. Why do we need robustness? Generate some 1D standard normal data and fit a Gaussian: 0 . 3 Gauss. 0 . 25 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  6. Why do we need robustness? The fit changes if we add an outlying datapoint (red blob). 0 . 3 Gauss. 0 . 25 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  7. Why do we need robustness? The fit changes if we add an outlying datapoint (red blob). 0 . 3 Gauss. 0 . 25 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  8. Why do we need robustness? A fitted Student’s t -distribution (red plot) is more concentrated. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  9. Why do we need robustness? As the outlier is moved away, the Gaussian fit changes a lot. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  10. Why do we need robustness? As the outlier is moved away, the Gaussian fit changes a lot. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  11. Why do we need robustness? As the outlier is moved away, the Gaussian fit changes a lot. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  12. Why do we need robustness? As the outlier is moved away, the Gaussian fit changes a lot. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  13. Why do we need robustness? As the outlier is moved away, the Gaussian fit changes a lot. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  14. Why do we need robustness? As the outlier is moved away, the Gaussian fit changes a lot. 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  15. Why do we need robustness? In contrast, the Student’s t -distribution is statistically robust . 0 . 3 Gauss. 0 . 25 t ( ν = 1 . 5) 0 . 2 p ( x ) 0 . 15 0 . 1 0 . 05 0 -2 0 2 4 6 8 10 12 x Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 4 / 11

  16. Robust statistics Robust ( resistant ) estimator: Adversarially corrupting a fraction η of the data ( η < 1 / 2 ) only has a bounded effect on the estimated model parameters � θ Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 5 / 11

  17. Why is Student’s t robust? The probability density functions of Gaussians and Student’s t -distributions look similar. 0 . 5 Gauss. t ( ν = 4) 0 . 4 t ( ν = 15) 0 . 3 0 . 2 0 . 1 0 -10 -5 0 5 10 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 6 / 11

  18. Why is Student’s t robust? The associated loss functions (the negative log-likelihood, or NLL) exhibit differences in the tails. 25 Gauss. t ( ν = 4) 20 t ( ν = 15) 15 10 5 0 -10 -5 0 5 10 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 6 / 11

  19. Why is Student’s t robust? The influence function is the gradient of the NLL. It quantifies the effect of outliers. For the t -distribution the influence function is bounded. 5 Gauss. t ( ν = 4) t ( ν = 15) 0 − 5 -10 -5 0 5 10 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 6 / 11

  20. Why is Student’s t robust? Gradient clipping can also limit the influence of outliers, but need not converge on the maximum-likelihood model. 5 Gauss. t ( ν = 4) t ( ν = 15) Clipped Gauss. 0 − 5 -10 -5 0 5 10 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 6 / 11

  21. Related work Our findings complement those in concurrent work by Jaini et al. (2020) 1 • They show: • Lipschitz-continuous triangular flows f θ ( Z ) with Gaussian base distributions Z cannot represent fat-tailed data • For example: Glow with sigmoid-transformed scale factors • Using multivariate t ν -distributions allows modelling data with fat tails • We add to this: • The advantages of t ν -distributions can be understood through statistical robustness • Experimentally, these benefits extend to bounded data (no fat tails) 1 Jaini, P., Kobyzev, I., Yu, Y., and Brubaker, M. Tails of Lipschitz triangular flows. In Proc. ICML , 2020. Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 7 / 11

  22. Stable training Training loss of Glow models of 64 × 64 CelebA data trained using Adam. The red configuration is unstable. 8 t ( = 50), lr=1e-3 Gauss. no grad-clip, lr=1e-4 7 Gauss. w. grad-clip, lr=1e-3 Gauss. no grad-clip, lr=5e-4 6 Training loss 5 4 3 2 0 200 400 600 800 1000 Steps Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 8 / 11

  23. Stable training Reducing the learning rate (yellow), clipping gradients (green), or changing the base to a multivariate t ν -distribution (blue) stabilises training. 8 t ( = 50), lr=1e-3 Gauss. no grad-clip, lr=1e-4 7 Gauss. w. grad-clip, lr=1e-3 Gauss. no grad-clip, lr=5e-4 6 Training loss 5 4 3 2 0 200 400 600 800 1000 Steps Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 8 / 11

  24. Better generalisation on image data Test set negative log-likelihood on MNIST with and without outliers from greyscale CIFAR-10. ν = ∞ is the Gaussian baseline. Test Clean 1% outliers Train ν = 20 50 1000 20 50 1000 ∞ ∞ Clean NLL 1.16 1.13 1.13 1.17 1.63 1.27 1.26 1.31 ∆ 0 − 0.03 − 0.03 0.01 0 − 0.36 − 0.37 − 0.32 1% NLL 1.17 1.13 1.14 1.18 1.21 1.18 1.19 1.22 outliers ∆ 0 − 0.04 − 0.03 0.01 0 − 0.03 − 0.02 0.01 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 9 / 11

  25. Better generalisation on image data Test set negative log-likelihood on MNIST with and without outliers from greyscale CIFAR-10. ν = ∞ is the Gaussian baseline. Test Clean 1% outliers Train ν = 20 50 1000 20 50 1000 ∞ ∞ Clean NLL 1.16 1.13 1.13 1.17 1.63 1.27 1.26 1.31 ∆ 0 − 0.03 − 0.03 0.01 0 − 0.36 − 0.37 − 0.32 1% NLL 1.17 1.13 1.14 1.18 1.21 1.18 1.19 1.22 outliers ∆ 0 − 0.04 − 0.03 0.01 0 − 0.03 − 0.02 0.01 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 9 / 11

  26. Better generalisation on image data Test set negative log-likelihood on MNIST with and without outliers from greyscale CIFAR-10. ν = ∞ is the Gaussian baseline. Test Clean 1% outliers Train ν = 20 50 1000 20 50 1000 ∞ ∞ Clean NLL 1.16 1.13 1.13 1.17 1.63 1.27 1.26 1.31 ∆ 0 − 0.03 − 0.03 0.01 0 − 0.36 − 0.37 − 0.32 1% NLL 1.17 1.13 1.14 1.18 1.21 1.18 1.19 1.22 outliers ∆ 0 − 0.04 − 0.03 0.01 0 − 0.03 − 0.02 0.01 Alexanderson & Henter (KTH) Robustness through Studentising flows 2020-07-11 9 / 11

Recommend

Generalisation error in learning with random features and the hidden manifold model B. Loureiro F.

Generalisation error in learning with random features and the hidden manifold model B. Loureiro F.

Generalisation error in learning with random features and the hidden manifold model B. Loureiro F. Gerace M. Mzard L. Zdeborov F. Krzakala (IPhT) (IPhT) (ENS) (IPhT) (ENS) ICML 2020 TRAINING A NEURAL NET EXPECTATIONS error variance

779 views • 34 slides
Inductive Visual Localisation: Factorised Training for Superior Generalisation Ankush Gupta

Inductive Visual Localisation: Factorised Training for Superior Generalisation Ankush Gupta

Inductive Visual Localisation: Factorised Training for Superior Generalisation Ankush Gupta Andrea Vedaldi Andrew Zisserman Visual Geometry Group (VGG) University of Oxford 1 BMVC 2018, Newcastle upon Tyne | Ankush Gupta RNNs have a

420 views • 12 slides
More Regression Thomas J. Leeper Department of Political Science and Government Aarhus

More Regression Thomas J. Leeper Department of Political Science and Government Aarhus

OLS Model Fit Robust SEs GLM More Regression Thomas J. Leeper Department of Political Science and Government Aarhus University November 14, 2013 OLS Model Fit Robust SEs GLM OLS Model Fit Robust SEs GLM OLS Model Fit 1 2 Robust SEs

493 views • 14 slides
First attempts to Automatize Generalisation of Electronic Navigational Charts Weronika Socha

First attempts to Automatize Generalisation of Electronic Navigational Charts Weronika Socha

Presentation for the 16 th ICA Workshop on Generalisation and Map Production - Dresden, Germany, 23-24 August, 2013 by Weronika Socha First attempts to Automatize Generalisation of Electronic Navigational Charts Weronika Socha GIMA, Delft

582 views • 24 slides
How Robust are Thresholds for Community Detection? Ankur Moitra (MIT) Robust Statistics Summer

How Robust are Thresholds for Community Detection? Ankur Moitra (MIT) Robust Statistics Summer

How Robust are Thresholds for Community Detection? Ankur Moitra (MIT) Robust Statistics Summer School Let me tell you a story about the success of belief propagation and statistical physics THE STOCHASTIC BLOCK MODEL Introduced by Holland,

1.65k views • 119 slides
Reading the Tea Leaves: Model Uncertainty, Robust Forecasts, and the Autocorrelation of

Reading the Tea Leaves: Model Uncertainty, Robust Forecasts, and the Autocorrelation of

Introduction Robust Forecasting Empirical Methodology Empirical Analysis of Analysts Forecasts Decomposing the autocorrelation in forecast errors Reading the Tea Leaves: Model Uncertainty, Robust Forecasts, and the Autocorrelation of

637 views • 34 slides
On a Generalisation of Dillons APN Permutation Lo Perrin Anne Canteaut Sbastien Duval

On a Generalisation of Dillons APN Permutation Lo Perrin Anne Canteaut Sbastien Duval

On a Generalisation of Dillons APN Permutation Lo Perrin Anne Canteaut Sbastien Duval leo.perrin@uni.lu Anne.Canteaut@inria.fr Sebastien.Duval@inria.fr May 11, 2017 Introduction Butterflies Generalisation Properties Spectrum

566 views • 33 slides
Robust strategies and model selection Stefan Van Aelst Department of Applied Mathematics and

Robust strategies and model selection Stefan Van Aelst Department of Applied Mathematics and

Robust strategies and model selection Stefan Van Aelst Department of Applied Mathematics and Computer Science Ghent University, Belgium Stefan.VanAelst@UGent.be ERCIM09 - COMISEF/COST Tutorial Outline Regression model 1 Least squares 2

1.13k views • 93 slides
Unsupervised Monocular Depth Estimation CNN Robust to Training Data Diversity Valery

Unsupervised Monocular Depth Estimation CNN Robust to Training Data Diversity Valery

Unsupervised Monocular Depth Estimation CNN Robust to Training Data Diversity Valery Anisimovskiy, Andrey Shcherbinin, 15 May, 2020 Sergey Turko and Ilya Kurilin Problem Statement: Depth Sensors limitations IR depth sensor Stereo camera

670 views • 21 slides
Sta$s$cal model training DTW, EM, and HMM training DTW:

Sta$s$cal model training DTW, EM, and HMM training DTW:

Sta$s$cal model training DTW, EM, and HMM training DTW: no training per se each example = its own model does deal with sequences

487 views • 31 slides
Robust model aggregation for production forecasting of oil and gas Gilles Stoltz CNRS HEC

Robust model aggregation for production forecasting of oil and gas Gilles Stoltz CNRS HEC

The problem at hand How to combine the outputs Interval forecasts IRSDI-PGMO Robust model aggregation for production forecasting of oil and gas Gilles Stoltz CNRS HEC Paris Joint work with Rapha el Deswarte (Ecole Polytechnique), S

458 views • 20 slides
Update on Cuyama Basin Groundwater Modeling December 3, 2018 Approach for Cuyama Basin Model

Update on Cuyama Basin Groundwater Modeling December 3, 2018 Approach for Cuyama Basin Model

Cuyama Basin Groundw ater Sustainability Agency Update on Cuyama Basin Groundwater Modeling December 3, 2018 Approach for Cuyama Basin Model Development Develop a Robust and Defensible Integrated Water Resources Model Robust Model

999 views • 66 slides
Object detection &amp; classification for ADAS Robust for Bad situations Small object sizes

Object detection &amp; classification for ADAS Robust for Bad situations Small object sizes

Object detection &amp; classification for ADAS Robust for Bad situations Small object sizes Robust for occlusion Small model size SVNet @ NVIDIA TX2 Please click Icon for Video 5/19/2017 2 Robust detection for various

123 views • 11 slides
Robust Multilingual Part-of-Speech Tagging via Adversarial Training (NAACL 2018) Michihiro

Robust Multilingual Part-of-Speech Tagging via Adversarial Training (NAACL 2018) Michihiro

Robust Multilingual Part-of-Speech Tagging via Adversarial Training (NAACL 2018) Michihiro Yasunaga , Jungo Kasai, Dragomir Radev Department of Computer Science, Yale University .github.io Adversarial Examples Very close to the

408 views • 24 slides
A Step Beyond the State of the Art Robust Model Predictive Control Synthesis Methods by c

A Step Beyond the State of the Art Robust Model Predictive Control Synthesis Methods by c

A Step Beyond the State of the Art Robust Model Predictive Control Synthesis Methods by c Sa sa V. Rakovi (based on recent collaborative research with B. Kouvaritakis, M. Cannon &amp; C. Panos) ISR, University of Maryland

1.11k views • 86 slides
FR-Train: A Mutual Information-based Fair and Robust Training Yuji Roh, Kangwook Lee, Steven E.

FR-Train: A Mutual Information-based Fair and Robust Training Yuji Roh, Kangwook Lee, Steven E.

FR-Train: A Mutual Information-based Fair and Robust Training Yuji Roh, Kangwook Lee, Steven E. Whang, Changho Suh Yuji Roh , Data Intelligence Lab, KAIST Trustworthy AI AI has significant potential to help solve challenging problems,

646 views • 42 slides
Model-Robust Inference for Clinical Trials that Improve Precision by Stratified Randomization and

Model-Robust Inference for Clinical Trials that Improve Precision by Stratified Randomization and

Model-Robust Inference for Clinical Trials that Improve Precision by Stratified Randomization and Adjustment for Additional Baseline Variables Bingkai Wang 1 , Ryoko Susukida 2 , Ramin Mojtabai 2 , Masoumeh Amin-Esmaeili 2 , 3 , Michael Rosenblum

196 views • 19 slides
and a model for robust community participation Wyeth W. Wasserman 30 November 2010

and a model for robust community participation Wyeth W. Wasserman 30 November 2010

The wiki-based Transcription Factor Encyclopedia and a model for robust community participation Wyeth W. Wasserman 30 November 2010 http://www.cisreg.ca/tfe Transcription Factors 101 Diverse set of proteins sharing common functional role

458 views • 29 slides
Generalisation/Specialisation Example: code name Subclass = sub-entity = special case.

Generalisation/Specialisation Example: code name Subclass = sub-entity = special case.

Generalisation/Specialisation Example: code name Subclass = sub-entity = special case. name Course ClassesIn Room #seats More attributes and/or relationships. teacher A subclass shares the key of its parent. ISA #computers

272 views • 8 slides
Learning Step Size Controllers for Robust Neural Network Training Christian Daniel et al. Recent

Learning Step Size Controllers for Robust Neural Network Training Christian Daniel et al. Recent

Learning Step Size Controllers for Robust Neural Network Training Christian Daniel et al. Recent Trends in Automated Machine Learning Abeeha Shafiq 18.07.2019 Motivation Optimizers are sensitive to initial learning rate Good

438 views • 16 slides
Bootstrapping incremental dialogue systems from minimal data: the generalisation power of

Bootstrapping incremental dialogue systems from minimal data: the generalisation power of

Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars Arash Eshghi, Igor Shalyminov, Oliver Lemon Heriot-Watt University Presenter: Prashant Jayannavar Problem - Inducing task-based

472 views • 30 slides
Mismatched Models &amp; Can GP Regression Be Made Robust Against Model Mismatch? Peter Sollich

Mismatched Models &amp; Can GP Regression Be Made Robust Against Model Mismatch? Peter Sollich

Gaussian Process Regression with Mismatched Models &amp; Can GP Regression Be Made Robust Against Model Mismatch? Peter Sollich NEURIPS 2002 &amp; International Workshop on Deterministic and Statistical Methods in Machine Learning (2004)

253 views • 9 slides
Glitch-Resistant Masking Revisited or Why Proofs in the Robust Probing Model are Needed Thorben

Glitch-Resistant Masking Revisited or Why Proofs in the Robust Probing Model are Needed Thorben

Glitch-Resistant Masking Revisited or Why Proofs in the Robust Probing Model are Needed Thorben Moos 1 , Amir Moradi 1 , Tobias Schneider 2 and Franois-Xavier Standaert 2 Horst Grtz Institute for IT Security, Ruhr-Universitt Bochum,

694 views • 46 slides
Coherence, Similarity, and Concept Generalisation Roberto Confalonieri 1 , Oliver Kutz 1 , Nicolas

Coherence, Similarity, and Concept Generalisation Roberto Confalonieri 1 , Oliver Kutz 1 , Nicolas

Coherence, Similarity, and Concept Generalisation Roberto Confalonieri 1 , Oliver Kutz 1 , Nicolas Troquard 1 , Pietro Galliani 1 , Daniele Porello 1 , Rafael Pe naloza 1 , Marco Schorlemmer 2 1 Free University of Bozen-Bolzano, Italy 2 Artificial

443 views • 16 slides

More recommend