direction matters on the implicit regularization effect
play

Direction Matters: On the Implicit Regularization Effect of - PowerPoint PPT Presentation

Oct 05, 2022 •33 likes •193 views

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate Jingfeng Wu , Difan Zou, Vladimir Braverman, Quanquan Gu Johns Hopkins University & UCLA November 2020 Overview

  1. Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate Jingfeng Wu , Difan Zou, Vladimir Braverman, Quanquan Gu Johns Hopkins University & UCLA November 2020

  2. Overview • Background • SGD vs. GD: Different Convergence Directions • Small Learning Rate • Moderate Learning Rate • Direction Matters: SGD + Moderate LR is Good • Proof Sketches

  3. <latexit sha1_base64="KXCQlbNA4tNvFg4knZExHyj1hdI=">ACL3icbVDLSgMxFM34rPVdekmWIS6KTOC6KZQdKELFxXtAzp1yKSZNjTJDElGKeP8h9/gzh9wq38gbsSV4F+YabvQ6oHA4dxzOTfHjxhV2rbfrJnZufmFxdxSfnldW29sLHZUGEsManjkIWy5SNFGBWkrqlmpBVJgrjPSNMfnGTz5g2RiobiSg8j0uGoJ2hAMdJG8gr757iRGDpds9WIFuIBFOnDQRKXRVzL2EVpz0WkCXMObRzJX3CkW7bI8A/xJnQorVU/jgene9mlf4dLshjkRGjOkVNuxI91JkNQUM5Lm3ViRCOEB6pG2oQJxojrJ6G8p3DVKFwahNE9oOFJ/biSIKzXkvnFypPtqepaJ/83asQ6OgkVUayJwOgIGZQhzArCnapJFizoSEIS2puhbiPTD3a1PkrhQ9HIVkxznQNf0ljv+wclO0L09AxGCMHtsEOKAEHIqOAM1UAcY3IMn8AxerEfr1Xq3PsbWGWuyswV+wfr6Bk6UqsE=</latexit> Implicit Regularization: SGD >> GD n L S ( w ) = 1 X Loss ` i ( w ) n i =1 GD <latexit sha1_base64="EarS2g+hOyhHCn1+yqIviVW4JU=">ACMnicbVA9bxNBEN1LCARDiIGSZoUVyS5s3UWyQhkBRQqKIOIPyWdZc3tz9sp7H9qdw7JO/iX8Avr8gbSJ+AEgGoRERZue9TkFtjPSrp7em5k3ekGmpCHX/e7s7D7Ye/ho/3HlydODZ4fV5y+6Js21wI5IVar7ARhUMsEOSVLYzRCHCjsBdN3S73GbWRaXJB8wyHMYwTGUkBZKlRtT3jvsKIQOt0xme8yQu/3FpoDBfcRwL7jzWE/MPI/yRA8fqsMarW3JZbFt8G3h2onTZuvzXf/x6Pqr+8cNU5DEmJBQYM/DcjIYFaJC4aLi5wYzEFMY48DCBGI0w6I8ZMGPLBPyKNX2JcRL9v+JAmJj5nFgO2OgidnUluR92iCn6M2wkEmWEyZiZRTlilPKl1nxUGoUpOYWgNDS3srFBDQIsomucTz0qRig/E2Y9gG3eOW1265H21Cb9mq9tkr9prVmcdO2Ck7Y+eswT7wq7YNbtxLp0fzi/n96p1x7mbecnWyvn7DzPzrjI=</latexit> w w � η r L S ( w ) SGD <latexit sha1_base64="NEVHOYBfO6Mb1VrYie/t9hYiKGM=">ACMXicbVBNT9tAEF0HaCGlJaXHXlaJKgWhRjaiao9RufRIJRIixVE0Xo+TVdYf2h03sqz8C078jf4BrvAPcqt6KV/go3DgQAj7erpvZl5oxdkShpy3aVT29refV6d6/+Zv/tu4PG+8O+SXMtsCdSlepBAaVTLBHkhQOMo0QBwovg9nZSr/8hdrINLmgIsNRDJNERlIAWrcOJ1zX2FEoHU653P+mZd+tbXUGC64jwT2n2gILVZqPGvPj/i40XI7blX8OfAeQKvb9I+vlt3ifNz454epyGNMSCgwZui5GY1K0CSFwkXdzw1mIGYwaGFCcRoRmV1x4J/skzIo1TblxCv2McTJcTGFHFgO2OgqXmqrciXtGFO0bdRKZMsJ0zE2ijKFaeUr6LiodQoSBUWgNDS3srFDQIsoFuMRFZVK3wXhPY3gO+icd70vH/WkT+s7Wtcs+siZrM49ZV32g52zHhPsmt2wW3bn/HaWzh/n7q15jzMfGAb5fy/BzjzrO8=</latexit> w w � ⌘ r ` k ( w ) CIFAR-10, ResNet-18, w/o weight decay, w/o data augmentation Wu, Jingfeng, et al. "On the Noisy Gradient Descent that Generalizes as SGD." ICML 2020.

  4. Two More Figures about SGD (Less Relevant) 70 test DccurDcy (%) 60 50 40 GD (66.96) GLD const (66.66) 30 GLD GynDmLc (69.25) 20 GLD GLDg (67.96) 6GD (75.21) 10 0 2500 5000 7500 10000 12500 15000 17500 LterDtLon Wilson, Ashia C., et al. "The marginal value of adaptive gradient methods in machine learning." NIPS 2017. Zhu, Zhanxing, et al. "The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects." ICML 2019.

  5. SGD vs. GD: Learning Rate Matters! Small LR Moderate LR GD L L SGD L J Q1: Small LR, SGD ≈ GD? Q2: Moderate LR, SGD >> GD? Q3: GD is bad anyhow?

  6. In Theory, SGD ≈ GD or SGD ≠ GD ?? GD <latexit sha1_base64="EarS2g+hOyhHCn1+yqIviVW4JU=">ACMnicbVA9bxNBEN1LCARDiIGSZoUVyS5s3UWyQhkBRQqKIOIPyWdZc3tz9sp7H9qdw7JO/iX8Avr8gbSJ+AEgGoRERZue9TkFtjPSrp7em5k3ekGmpCHX/e7s7D7Ye/ho/3HlydODZ4fV5y+6Js21wI5IVar7ARhUMsEOSVLYzRCHCjsBdN3S73GbWRaXJB8wyHMYwTGUkBZKlRtT3jvsKIQOt0xme8yQu/3FpoDBfcRwL7jzWE/MPI/yRA8fqsMarW3JZbFt8G3h2onTZuvzXf/x6Pqr+8cNU5DEmJBQYM/DcjIYFaJC4aLi5wYzEFMY48DCBGI0w6I8ZMGPLBPyKNX2JcRL9v+JAmJj5nFgO2OgidnUluR92iCn6M2wkEmWEyZiZRTlilPKl1nxUGoUpOYWgNDS3srFBDQIsomucTz0qRig/E2Y9gG3eOW1265H21Cb9mq9tkr9prVmcdO2Ck7Y+eswT7wq7YNbtxLp0fzi/n96p1x7mbecnWyvn7DzPzrjI=</latexit> w w � η r L S ( w ) SGD <latexit sha1_base64="H8W0j+nYlbjAbB3JxTavAukChE=">ACeHicbVFNb9NAEF2bj5bwUReOXEZUiEYokV0JwbECDhw4FEHaSnFkjdfjdJX1h3bHRJGVH8SP4Qcgceml/4ALJzZODrTNSLt6eu/NzuhtWmtlOQx/ef6du/fu7+w+6D189PjJXrD/9NRWjZE0kpWuzHmKlrQqacSKNZ3XhrBINZ2lsw8r/ew7Gauq8hsvapoUOC1VriSyo5KgnUOsKWc0prDHAbQxt2raFsCTExuntqMIPSfxVobDeR9ew1ZfalDOiNstHYMNF5PWyWzFLJPgIByGXcFtEG3AwXH/z8/Bx98/TpLgKs4q2RUstRo7TgKa560aFhJTcte3Fiq3QY4pbGDJRZkJ235hJeOiaDvDLulAwd+39Hi4W1iyJ1zgL5wt7UVuQ2bdxw/m7SqrJumEq5HpQ3GriCVeKQKUOS9cIBlEa5XUFeoEuK3b9cm1IsuiE9F0x0M4b4PRoGL0Zhl9cQu/FunbFc/FCHIpIvBXH4pM4ESMhxaW34wXevfXB/+V319bfW/T80xcK/oH9F/wao=</latexit> w w � ⌘ r L S ( w ) + ⌘ ( r L S ( w ) � r ` k ( w )) Unbiased noise (scales with 𝜃 ) Theory disagrees with practice L “Easy” to prove SGD ≈ GD by concentration <= e.g., small LR • “Hard” to prove an inverse result <= no concentration! •

  7. <latexit sha1_base64="gE32TAgCiehRkp6UP49CDdg+CSs=">ACcnicbVFNb9NAEF2bj7Yp0BRu5TIQIbWCRnalqlyQqsKBA4eikrZSHKzxep2sumu7u2OqyPIP4sfwA1B75hfAnY3TA0kZaW3b97MW71NSiUtBcFPz793/8HDldW1zvqjx082uptPT21RGS4GvFCFOU/QCiVzMSBJSpyXRqBOlDhLt7P+mfhLGyL/QtBQjeNcZpIjOSruVlEqM7iCd7AL0dhgCp/i6ISjgu2rHaghai1qI9IGWik18BrqBdpeGqojQdi4y4kca3SzX6MJqmxJOVtwFMTd3tBP2gL7oLwFvQOd/782P1w8/047v6K0oJXWuTEFVo7DIOSRjUaklyJphNVpTIL3Ashg7mqIUd1a1A68ck0JWGHdygpb9d6JGbe1UJ06pkSZ2uTcj/9cbVpS9HdUyLysSOZ8bZUCKmCWNaTSCE5q6gByI91bgU/QICf3IwsuetqadFw4XIMd8HpXj/c7wefXUJHbF6r7Dl7ybZyA7YIfvIjtmAcXbteV7HW/d+1v+C783l/re7cwztlD+m79aJMCi</latexit> Small Learning Rate: SGD ≈ GD 𝜃 = 𝑒𝑢 → 0 GD <latexit sha1_base64="EarS2g+hOyhHCn1+yqIviVW4JU=">ACMnicbVA9bxNBEN1LCARDiIGSZoUVyS5s3UWyQhkBRQqKIOIPyWdZc3tz9sp7H9qdw7JO/iX8Avr8gbSJ+AEgGoRERZue9TkFtjPSrp7em5k3ekGmpCHX/e7s7D7Ye/ho/3HlydODZ4fV5y+6Js21wI5IVar7ARhUMsEOSVLYzRCHCjsBdN3S73GbWRaXJB8wyHMYwTGUkBZKlRtT3jvsKIQOt0xme8yQu/3FpoDBfcRwL7jzWE/MPI/yRA8fqsMarW3JZbFt8G3h2onTZuvzXf/x6Pqr+8cNU5DEmJBQYM/DcjIYFaJC4aLi5wYzEFMY48DCBGI0w6I8ZMGPLBPyKNX2JcRL9v+JAmJj5nFgO2OgidnUluR92iCn6M2wkEmWEyZiZRTlilPKl1nxUGoUpOYWgNDS3srFBDQIsomucTz0qRig/E2Y9gG3eOW1265H21Cb9mq9tkr9prVmcdO2Ck7Y+eswT7wq7YNbtxLp0fzi/n96p1x7mbecnWyvn7DzPzrjI=</latexit> w w � η r L S ( w ) <latexit sha1_base64="pkyzSQktJPiPlv6NuoczmDRe2JA=">ACL3icbVC9SgNBGNzN8a/qKXNhyLEwnAXEG2EoBYWFhFNDORC2Nvbi0v2ftjdM4Qj7+ErWPoCtpYiyBiJfgA9m4uKUziwMIwMx+zjBNxJpVpvhlT0zOzc/OZhezi0vLKam5tvSrDWBaISEPRc3BknIW0IpitNaJCj2HU6vnfZJ37+pUKyMLhS3Yg2fNwKmMcIVlpq5oq2yzowBHsgd0S2IXzpn1JMId8ZxcSsNOKRFC3B2lU9Zq5bNgpoBJYg3Jdmn352Xv9P2+3Mx92W5IYp8GinAsZd0yI9VIsFCMcNrL2rGkESZt3KJ1TQPsU9lI0uIe7GjFBS8U+gUKUvXvRYJ9Kbu+o5M+Vjdy3OuL/3n1WHmHjYQFUaxoQAZFXsxBhdAfClwmKFG8qwkmgum/ArnBAhOl5xp8btpSVYPY43PMEmqxYK1XzAv9ELHaIAM2kRbKI8sdIBK6AyVUQURdIce0RN6Nh6MV+PD+BxEp4zhzQYagfH9C87Dq+Q=</latexit> d w = � r L S ( w )d t Gradient Flow (GF) d w = � r L S ( w )d t + p η Σ ( w ) 1 2 d B t Stochastic Modified Equation (SME) 𝜃 = 𝑒𝑢 → 0 Higher order term SGD <latexit sha1_base64="H8W0j+nYlbjAbB3JxTavAukChE=">ACeHicbVFNb9NAEF2bj5bwUReOXEZUiEYokV0JwbECDhw4FEHaSnFkjdfjdJX1h3bHRJGVH8SP4Qcgceml/4ALJzZODrTNSLt6eu/NzuhtWmtlOQx/ef6du/fu7+w+6D189PjJXrD/9NRWjZE0kpWuzHmKlrQqacSKNZ3XhrBINZ2lsw8r/ew7Gauq8hsvapoUOC1VriSyo5KgnUOsKWc0prDHAbQxt2raFsCTExuntqMIPSfxVobDeR9ew1ZfalDOiNstHYMNF5PWyWzFLJPgIByGXcFtEG3AwXH/z8/Bx98/TpLgKs4q2RUstRo7TgKa560aFhJTcte3Fiq3QY4pbGDJRZkJ235hJeOiaDvDLulAwd+39Hi4W1iyJ1zgL5wt7UVuQ2bdxw/m7SqrJumEq5HpQ3GriCVeKQKUOS9cIBlEa5XUFeoEuK3b9cm1IsuiE9F0x0M4b4PRoGL0Zhl9cQu/FunbFc/FCHIpIvBXH4pM4ESMhxaW34wXevfXB/+V319bfW/T80xcK/oH9F/wao=</latexit> w w � ⌘ r L S ( w ) + ⌘ ( r L S ( w ) � r ` k ( w )) <latexit sha1_base64="2YkqCO3ZcVtUvasN3QP5Gwy0bRM=">AC43icbVLihNBFK1uX2N8RWenm8JhQBCd0DGjTDMbFwIjmIyA6meUF190ymHj1V1Uo+wvcyWz9MH/Anf9gdSc+knih4HDuvecbndeCW5dknyP4mvXb9y8tXO7d+fuvfsP+g8fja2uDYMR0Kbs5xaEFzByHEn4KwyQGUu4DS/OG7px/BWK7VB7eoIJO0VHzGXWBmvZ/kBxKrjwLGrbpkTE1eEKgslxoleFXmFheSno+xIRsdKcpfo4JK7SzLfjNetKl8gaKhoCjDW5ltm/wpZLTHJevVrY7idOx82OMQCVfyJ+J6Xc0eN0Z9WGm3Kbn19214a5zuvpn295JB0hXeBukK7B3uP/65e5X6k/YwhWa1BOWYoNZO0qRymafGcSYgpKgtVJRd0BImASoqwWa+s2/wfmAKPNMmPOVwx/674am0diHzMCmpm9vNXkv+rzep3exl5rmqageKLY1mtcBO4/bj4oIbYE4sAqDM8JAVszk1lLnwC6y5yEVn0guHSTfPsA3Gw0H6YpC8Cxc6QsvaQU/QU/QMpegAHaLX6ASNEIveRCby0ecY4i/x1/hqORpHq51dtFbxt1/se1a</latexit> ( Var[ ✏ ] = � 2 ⇒ � = O ( √ ⌘ ) Var[ ✏ 1 + · · · + ✏ η ] = ⌘� 2 ∼ O � ⌘ 2 �

Recommend

10. Regularization More on tradeoffs Regularization Effect of using different norms

10. Regularization More on tradeoffs Regularization Effect of using different norms

CS/ECE/ISyE 524 Introduction to Optimization Spring 201718 10. Regularization More on tradeoffs Regularization Effect of using different norms Example: hovercraft revisited Laurent Lessard (www.laurentlessard.com) Review of

407 views • 21 slides
The Implicit Regularization of Stochastic Gradient Flow for Least Squares Alnur Ali 1 , Edgar

The Implicit Regularization of Stochastic Gradient Flow for Least Squares Alnur Ali 1 , Edgar

The Implicit Regularization of Stochastic Gradient Flow for Least Squares Alnur Ali 1 , Edgar Dobriban 2 , and Ryan J. Tibshirani 3 1 Stanford University, 2 University of Pennsylvania, 3 Carnegie Mellon University Outline Overview

666 views • 27 slides
Implicit Regularization in Nonconvex Statistical Estimation Yuxin Chen Electrical Engineering,

Implicit Regularization in Nonconvex Statistical Estimation Yuxin Chen Electrical Engineering,

Implicit Regularization in Nonconvex Statistical Estimation Yuxin Chen Electrical Engineering, Princeton University Cong Ma Kaizheng Wang Yuejie Chi Princeton ORFE Princeton ORFE CMU ECE Nonconvex estimation problems are everywhere

1.03k views • 91 slides
Implicit Bias and Race Mikah K. Thompson, Esq. Director of Affirmative Action &amp; Adjunct

Implicit Bias and Race Mikah K. Thompson, Esq. Director of Affirmative Action &amp; Adjunct

Implicit Bias and Race Mikah K. Thompson, Esq. Director of Affirmative Action &amp; Adjunct Professor of Law University of Missouri-Kansas City Learning Objectives Define implicit bias Explore the effect that implicit bias has on

447 views • 17 slides
Regularization Overview Regularization Overview Problems &amp; Multicollinearity We will

Regularization Overview Regularization Overview Problems &amp; Multicollinearity We will

Regularization Overview Regularization Overview Problems &amp; Multicollinearity We will discuss three popular methods for obtaining better estimates of the linear model coefficients Regularization Techniques Principal

305 views • 12 slides
Whose and What Chatter Matters? The Effect of Tweets on Movie Sales Huaxia Rui (joint work with

Whose and What Chatter Matters? The Effect of Tweets on Movie Sales Huaxia Rui (joint work with

Whose and What Chatter Matters? The Effect of Tweets on Movie Sales Whose and What Chatter Matters? The Effect of Tweets on Movie Sales Huaxia Rui (joint work with Yizao Liu, and Andrew Whinston) Simon School of Business University of

294 views • 25 slides
Regularization Regularization is a general approach to add a complexity parameter to a

Regularization Regularization is a general approach to add a complexity parameter to a

Regularization Regularization is a general approach to add a complexity parameter to a learning algorithm. Requires that the model parameters be continuous. (i.e., Regression OK, IAML: Regularization and Ridge Regression Decision trees

204 views • 3 slides
Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall

Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall

Implicit surface modelling Spline regularization as a Gaussian process GPIS for 2D curves GPIS for 3D surfaces Summary Gaussian Process Implicit Surfaces Oliver Williams 1 Microsoft Research and Trinity Hall Cambridge, UK Gaussian Processes

647 views • 40 slides
Implicit Bias Implicit bias Implicit bias refers to attitudes or stereotypes that affect our

Implicit Bias Implicit bias Implicit bias refers to attitudes or stereotypes that affect our

Implicit Bias Implicit bias Implicit bias refers to attitudes or stereotypes that affect our understanding, actions and decisions in an unconscious manner. It's different from suppressed thoughts we might conceal to keep the peace; it's

634 views • 16 slides
Implicit Surfaces Implicit Surfaces An implicit surface is simply an iso-contour CIS 781 of a

Implicit Surfaces Implicit Surfaces An implicit surface is simply an iso-contour CIS 781 of a

Implicit Surfaces Implicit Surfaces An implicit surface is simply an iso-contour CIS 781 of a scalar function f(x,y,z)=0 . Roger Crawfis The term is usually used when modeling a surface, whereas iso-contour is used when visualizing a

388 views • 12 slides
Manifold Regularization Lorenzo Rosasco MIT, 9.520 L. Rosasco Manifold Regularization About

Manifold Regularization Lorenzo Rosasco MIT, 9.520 L. Rosasco Manifold Regularization About

Manifold Regularization Lorenzo Rosasco MIT, 9.520 L. Rosasco Manifold Regularization About this class Goal To analyze the limits of learning from examples in high dimensional spaces. To introduce the semi-supervised setting and the use of

437 views • 31 slides
The Right Culture Why culture matters Wh lt tt What a great culture looks like

The Right Culture Why culture matters Wh lt tt What a great culture looks like

4/30/2014 Building the Right Culture for Your Organization Barbara Schmidt Kemp North Star Consultants LLC Guiding organizations and leaders in the right direction 1 The Right Culture Why culture matters Wh lt tt What a

356 views • 17 slides
Overparametrization and the bias-variance dilemma Johannes Schmidt-Hieber joint work with Alexis

Overparametrization and the bias-variance dilemma Johannes Schmidt-Hieber joint work with Alexis

Overparametrization and the bias-variance dilemma Johannes Schmidt-Hieber joint work with Alexis Derumigny https://arxiv.org/abs/2006.00278.pdf 1 / 13 double descent and implicit regularization overparametrization generalizes well implicit

357 views • 13 slides
Regularization of Inverse Problems Matthias J. Ehrhardt January 28, 2019 What is an Inverse

Regularization of Inverse Problems Matthias J. Ehrhardt January 28, 2019 What is an Inverse

Regularization of Inverse Problems Matthias J. Ehrhardt January 28, 2019 What is an Inverse Problem? A : U V mapping between Hilbert spaces U , V , A L ( U , V ) physical model A , cause u and effect A ( u ) = Au . Direct /

900 views • 31 slides
Implicit Extremes and Implicit MaxStable Laws Stilian Stoev ( sstoev@umich.edu ) University of

Implicit Extremes and Implicit MaxStable Laws Stilian Stoev ( sstoev@umich.edu ) University of

Problem Formulation Limit Theory Implicit Extreme Value Laws Domains of Attraction An Example Implicit Extremes and Implicit MaxStable Laws Stilian Stoev ( sstoev@umich.edu ) University of Michigan, Ann Arbor September 19, 2014 Joint work

808 views • 34 slides
CS7015 (Deep Learning) : Lecture 8 Regularization: Bias Variance Tradeoff, l2 regularization,

CS7015 (Deep Learning) : Lecture 8 Regularization: Bias Variance Tradeoff, l2 regularization,

CS7015 (Deep Learning) : Lecture 8 Regularization: Bias Variance Tradeoff, l2 regularization, Early stopping, Dataset augmentation, Parameter sharing and tying, Injecting noise at input, Ensemble methods, Dropout Mitesh M. Khapra Department of

1.06k views • 86 slides
Regularization for Multi-Output Learning Lorenzo Rosasco 9.520 L. Rosasco Regularization for

Regularization for Multi-Output Learning Lorenzo Rosasco 9.520 L. Rosasco Regularization for

Regularization for Multi-Output Learning Lorenzo Rosasco 9.520 L. Rosasco Regularization for Multi-Output Learning About this class Goal In many practical problems, it is convenient to model the object of interest as a function with multiple

507 views • 47 slides
WINE &amp; DINE Direction Munich Rotkreuz Direction Frankfurt Direction Basle Vorarlberg

WINE &amp; DINE Direction Munich Rotkreuz Direction Frankfurt Direction Basle Vorarlberg

WINE &amp; DINE Direction Munich Rotkreuz Direction Frankfurt Direction Basle Vorarlberg KSSNACHT Zurich LUCERNE VITZNAU Lucerne Berne Rigi PARK HOTEL VITZNAU Buochs Pilatus Geneva Chiasso Direction Italy In the heart of

288 views • 24 slides
Iterative regularization for general inverse problems Guillaume Garrigos with L. Rosasco and S.

Iterative regularization for general inverse problems Guillaume Garrigos with L. Rosasco and S.

Iterative regularization for general inverse problems Guillaume Garrigos with L. Rosasco and S. Villa CNRS, cole Normale Suprieure Sminaire CVN - Centrale Suplec - 23 Jan 2018 Regularization of inverse problems 1 Regularization by

719 views • 56 slides
Regularization Effect of Large Initial Learning Rate Yuanzhi Li* Colin Wei* Tengyu Ma Carnegie

Regularization Effect of Large Initial Learning Rate Yuanzhi Li* Colin Wei* Tengyu Ma Carnegie

Regularization Effect of Large Initial Learning Rate Yuanzhi Li* Colin Wei* Tengyu Ma Carnegie Mellon University Stanford University Stanford University Large Initial Learning Rate is Crucial for Generalization Large Initial Learning Rate is

446 views • 29 slides
Implicit Surfaces CPSC 599.86 / 601.86 Sonny Chan University of Calgary (some board work happened

Implicit Surfaces CPSC 599.86 / 601.86 Sonny Chan University of Calgary (some board work happened

Implicit Surfaces CPSC 599.86 / 601.86 Sonny Chan University of Calgary (some board work happened here) Implicit Surface Rendering Kenneth Salisbury &amp; Christopher Tarr, 1997 Implicit Surfaces A surface defined by an implicit equation: S

398 views • 20 slides
When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really

When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really

When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters Yogesh K. Ramadass, AnanthaGroup Microsystems Technology Laboratory Outline Outline Outline Outline

507 views • 36 slides
LIC-Based Regularization of Multi-Valued Images David Tschumperl CNRS UMR 6072 (GREYC/ENSICAEN)

LIC-Based Regularization of Multi-Valued Images David Tschumperl CNRS UMR 6072 (GREYC/ENSICAEN)

LIC-Based Regularization of Multi-Valued Images David Tschumperl CNRS UMR 6072 (GREYC/ENSICAEN) - Image Team ICIP2005, Genova, 11-14 September 2005 Data Regularization Aim of regularization consists in transforming a noisy signal into a

583 views • 22 slides
Theory and Practice Implicit Leadership Theory (ILT) Romance of Leadership (RoL) Implicit

Theory and Practice Implicit Leadership Theory (ILT) Romance of Leadership (RoL) Implicit

Critical leadership: Theory and Practice Implicit Leadership Theory (ILT) Romance of Leadership (RoL) Implicit Leadership Theories ILTs It is all in our heads `General ideas about what leaders are like and how they behave are called

887 views • 24 slides

More recommend