nonconvex variance reduced optimization with arbitrary
play

Nonconvex Variance Reduced Optimization with Arbitrary Sampling - PowerPoint PPT Presentation

Nov 09, 2022 •780 likes •1.07k views

Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik Empirical Risk Minimization n x R d f ( x ) := 1 X min f i ( x ) n i =1 Empirical Risk Minimization n x R d f ( x ) := 1 X min f i (

  1. Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horváth Peter Richtárik

  2. Empirical Risk Minimization n x ∈ R d f ( x ) := 1 X min f i ( x ) n i =1

  3. Empirical Risk Minimization n x ∈ R d f ( x ) := 1 X min f i ( x ) n i =1 n is big

  4. Empirical Risk Minimization n x ∈ R d f ( x ) := 1 X min f i ( x ) n i =1 n is big non-convex, L i -smooth kr f i ( x ) � r f i ( y ) k  L i k x � y k

  5. Baseline Variance Reduced SGD Methods SVRG Johnson & Zhang NIPS 2013 SAGA Defazio, Bach & Lacoste-Julien NIPS 2014 SARAH Nguyen, Liu, Scheinberg & Takáč ICML 2017

  6. Baseline Variance Reduced SGD Methods SVRG Johnson & Zhang NIPS 2013 Uniform sampling SAGA Defazio, Bach & Lacoste-Julien NIPS 2014 Uniform sampling SARAH Nguyen, Liu, Scheinberg & Takáč ICML 2017 Uniform sampling

  7. Baseline Variance Reduced SGD Methods–Mini-batch SVRG Konečný & Richtárik FAMS 2017 SAGA Reddi, Hefny, Sra, Poczos, Smola CDC 2016 SARAH Nguyen, Liu, Scheinberg & Takáč 2017

  8. Baseline Variance Reduced SGD Methods–Mini-batch Mini-batch size SVRG Konečný & Richtárik FAMS 2017 Uniform sampling SAGA Reddi, Hefny, Sra, Poczos, Smola CDC 2016 Uniform sampling SARAH Nguyen, Liu, Scheinberg & Takáč 2017 Uniform sampling

  9. Contributions Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm • Construction of optimal minibatch sampling •

  10. Richtárik & Takáč (OL 2016; arXiv 2013) Contributions Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019) Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm • Construction of optimal minibatch sampling •

  11. Richtárik & Takáč (OL 2016; arXiv 2013) Contributions Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019) Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm • Construction of optimal minibatch sampling • First optimal/importance sampling for minibatches!

Recommend

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji 1 , Zhe Wang 1 , Yi Zhou 2 , Yingbin Liang 1 1 Ohio State University, 2 Duke University ICML 2019 K. Ji, Z. Wang, Y. Zhou, Y. Liang Zeroth-Order

327 views • 9 slides
Stochastic Cubic Regularization for Fast Nonconvex Optimization Nilesh Tripuraneni, Mitchell

Stochastic Cubic Regularization for Fast Nonconvex Optimization Nilesh Tripuraneni, Mitchell

Stochastic Cubic Regularization for Fast Nonconvex Optimization Nilesh Tripuraneni, Mitchell Stern, Chi Jin, Jeffrey Regier and Michael I. Jordan Achin Jain University of Pennsylvania STAT991, Spring 2019 Fast Nonconvex Optimization 1

422 views • 26 slides
Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov Julien Mairal andrei.kulunchakov@inria.fr julien.mairal.@inria.fr International Conference on Machine Learning, 2019 Poster event-4062, (Jun 12th,

317 views • 6 slides
Pathwise Coordinate Optimization for Nonconvex Sparse Learning Tuo Zhao

Pathwise Coordinate Optimization for Nonconvex Sparse Learning Tuo Zhao

Pathwise Coordinate Optimization for Nonconvex Sparse Learning Tuo Zhao http://www.princeton.edu/tuoz Department of Computer Science Johns Hopkins University Mar. 25. 2015 A General Theory of Pathwise Coordinate Optimization Collaborators

798 views • 45 slides
Accelerated Douglas-Rachford splitting and ADMM for structured nonconvex optimization Panos

Accelerated Douglas-Rachford splitting and ADMM for structured nonconvex optimization Panos

Accelerated Douglas-Rachford splitting and ADMM for structured nonconvex optimization Panos Patrinos KU Leuven (ESAT-STADIUS) joint work with Andreas Themelis and Lorenzo Stella LCCC Workshop Large-Scale and Distributed Optimization Lund,

591 views • 47 slides
Capstone Technology Industrial Plant Optimization in Reduced Dimensional Spaces Fields

Capstone Technology Industrial Plant Optimization in Reduced Dimensional Spaces Fields

Capstone Technology Industrial Plant Optimization in Reduced Dimensional Spaces Fields Optimization Lecture Toronto, ON Giles Laurier June 4, 2013 Agenda Review of optimization in oil refining Real Time Optimization Reduced Space

845 views • 56 slides
How to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization Frank E.

How to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization Frank E.

Motivation Contemporary Analyses Partitioning Regularization Methods Summary How to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization Frank E. Curtis , Lehigh University joint work with Daniel P. Robinson ,

734 views • 37 slides
Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie

Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie

Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie Johnson and Tong Zhang RJ Research Consulting Hong Kong University of Science and Technology 1 / 12 Training Deep Neural Networks

458 views • 21 slides
Nonconvex Distributed Optimization: Novel Algorithmic Design and Arbitrarily Precise Solutions

Nonconvex Distributed Optimization: Novel Algorithmic Design and Arbitrarily Precise Solutions

Nonconvex Distributed Optimization: Novel Algorithmic Design and Arbitrarily Precise Solutions Presenter: Zhiyu He Coauthors: Jianping He*, Cailian Chen and Xinping Guan Department of Automation Shanghai Jiao Tong University July 2020

716 views • 39 slides
A Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O ( 3 / 2 )

A Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O ( 3 / 2 )

Motivation Proposed Algorithm Theoretical Results Numerical Results Summary A Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O ( 3 / 2 ) Complexity Mohammadreza Samadi , Lehigh University joint work with

556 views • 39 slides
A Lagrangean Based Branch-and-Cut Algorithm for Global Optimization of Nonconvex Mixed-Integer

A Lagrangean Based Branch-and-Cut Algorithm for Global Optimization of Nonconvex Mixed-Integer

A Lagrangean Based Branch-and-Cut Algorithm for Global Optimization of Nonconvex Mixed-Integer Nonlinear Programs with Decomposable Structures Ramkumar Karuppiah and Ignacio E. Grossmann Department of Chemical Engineering Carnegie Mellon

588 views • 48 slides
Flexible ADMM for Block-Structured Convex and Nonconvex Optimization Zhi-Quan (Tom) Luo Joint

Flexible ADMM for Block-Structured Convex and Nonconvex Optimization Zhi-Quan (Tom) Luo Joint

Introduction The ADMM Algorithm The Main Result Flexible ADMM for Block-Structured Convex and Nonconvex Optimization Zhi-Quan (Tom) Luo Joint work with Mingyi Hong, Tsung-Hui Chang, Xiangfeng Wang, Meisam Razaviyanyn, Shiqian Ma University

770 views • 57 slides
Interior point method for nonlinear nonconvex optimization Ladislav Lukan, Ctirad Matonoha, Jan

Interior point method for nonlinear nonconvex optimization Ladislav Lukan, Ctirad Matonoha, Jan

Interior point method for nonlinear nonconvex optimization Ladislav Lukan, Ctirad Matonoha, Jan Vl cek Institute of Computer Science AS CR, Prague GAMM Workshop Applied and Numerical Linear Algebra September 11-12, 2008 Technische

842 views • 51 slides
Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization Feihu

Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization Feihu

Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization Feihu Huang 1 , Songcan Chen 2,3 and Heng Huang 1,4* 1. Department of Electrical and Computer Engineering, University of Pittsburgh, USA 2. College of

179 views • 14 slides
Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1

Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1

Variance of an Indicator Mathematics for Computer Science MIT 6.042J/18.062J IanindicatorwithE[I]=p: Var[I]:: = E[(I p) 2 ] Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1 variance.2

104 views • 6 slides
Inexact variable metric proximal gradient methods with line-search for convex and nonconvex

Inexact variable metric proximal gradient methods with line-search for convex and nonconvex

Inexact variable metric proximal gradient methods with line-search for convex and nonconvex optimization Silvia Bonettini Dipartimento di Scienze Fisiche, Optimization Algorithms and Software Informatiche e Matematiche for Inverse problemS

567 views • 17 slides
Nonconvex Demixing from Bilinear Measurements Yuanming Shi 1 Outline Motivations Blind

Nonconvex Demixing from Bilinear Measurements Yuanming Shi 1 Outline Motivations Blind

Nonconvex Demixing from Bilinear Measurements Yuanming Shi 1 Outline Motivations Blind deconvolution meets blind demixing T woVignettes: Implicitly regularized Wirtinger flow Why nonconvex optimization? Implicitly

764 views • 59 slides
Accelerating PDE-Constrained Optimization using Adaptive Reduced-Order Models: Application to

Accelerating PDE-Constrained Optimization using Adaptive Reduced-Order Models: Application to

Motivation ROM-Constrained Optimization Numerical Experiments Conclusion Accelerating PDE-Constrained Optimization using Adaptive Reduced-Order Models: Application to Topology Optimization Matthew J. Zahr Farhat Research Group Stanford

942 views • 72 slides
Accelerating PDE-Constrained Optimization using Progressively-Constructed Reduced-Order Models

Accelerating PDE-Constrained Optimization using Progressively-Constructed Reduced-Order Models

Motivation PDE-Constrained Optimization Reduced-Order Models ROM-Constrained Optimization Numerical Experiments Conclusion References Accelerating PDE-Constrained Optimization using Progressively-Constructed Reduced-Order Models Matthew J.

1.5k views • 135 slides
Mean-Variance Optimization Corporate Finance and Incentives Lars Jul Overby Department of

Mean-Variance Optimization Corporate Finance and Incentives Lars Jul Overby Department of

Mean-Variance Optimization Corporate Finance and Incentives Lars Jul Overby Department of Economics University of Copenhagen September 2010 Lars Jul Overby (D of Economics - UoC) Mean-Variance Optimization 09/10 1 / 21 Practical stuff

337 views • 21 slides
The Power of Nonconvex Optimization in Solving Random Quadratic Systems of Equations Yuxin Chen

The Power of Nonconvex Optimization in Solving Random Quadratic Systems of Equations Yuxin Chen

The Power of Nonconvex Optimization in Solving Random Quadratic Systems of Equations Yuxin Chen (Princeton) Emmanuel Cand` es (Stanford) Y. Chen, E. J. Cand` es, Communications on Pure and Applied Mathematics vol. 70, no. 5, pp. 822-883, May

1.3k views • 83 slides
OOPS 2020 Mean field methods in high-dimensional statistics and nonconvex optimization Lecturer:

OOPS 2020 Mean field methods in high-dimensional statistics and nonconvex optimization Lecturer:

OOPS 2020 Mean field methods in high-dimensional statistics and nonconvex optimization Lecturer: Andrea Montanari Problem session leader: Michael Celentano July 7, 2020 Problem Session 1 Problem 1: from Gordons objective to the fixed point

518 views • 9 slides
PDE-Constrained Optimization using Progressively-Constructed Reduced-Order Models Matthew J.

PDE-Constrained Optimization using Progressively-Constructed Reduced-Order Models Matthew J.

PDE-Constrained Optimization ROM-Constrained Optimization Numerical Experiments Conclusion PDE-Constrained Optimization using Progressively-Constructed Reduced-Order Models Matthew J. Zahr and Charbel Farhat Institute for Computational and

412 views • 23 slides
Reduced-Order Approaches for PDE-Constrained Multiobjective Optimization joint work with U

Reduced-Order Approaches for PDE-Constrained Multiobjective Optimization joint work with U

Reduced-Order Approaches for PDE-Constrained Multiobjective Optimization joint work with U Dubrovnik / TU Eindhoven / U Erlangen / TU Mnchen / U Paderborn Stefan Volkwein University of Konstanz, Chair Numerical Optimization New Trends in

284 views • 12 slides

More recommend