Some Recent Advances in Non-convex Optimization Purushottam Kar - PowerPoint PPT Presentation

Some Recent Advances in Non-convex Optimization Purushottam Kar IIT KANPUR

Outline of the Talk • Recap of Convex Optimization • Why Non-convex Optimization? • Non-convex Optimization: A Brief Introduction • Robust Regression : A Non-convex Approach • Robust Regression: Application to Face Recognition • Robust PCA : A Sketch and Application to Foreground Extraction in Images

Recap of Convex Optimization

Convex Optimization Convex set Convex function

Examples Linear Programming Quadratic Programming Semidefinite Programming

Applications Regression Classification Resource Allocation Clustering/Partitioning Signal Processing Dimensionality Reduction

Techniques • Projected (Sub)gradient Methods • Stochastic, mini-batch variants • Primal, dual, primal-dual approaches • Coordinate update techniques • Interior Point Methods • Barrier methods • Annealing methods • Other Methods • Cutting plane methods • Accelerated routines • Proximal methods • Distributed optimization • Derivative-free optimization

Why Non-convex Optimization?

Gene Expression Analysis DNA micro-array gene expression data … www.tes.com

Recommender Systems 𝑙 𝑜 = 𝑛

Image Reconstruction and Robust Face Recognition = + + 0.90 0.05 0.05 = ≈ + + 0.92 0.01 0.07 = ≈ + + 0.65 0.15 0.20

Image Denoising and Robust Face Recognition = = + + + + + ⋯ 𝑜

Large Scale Surveillance • Foreground-background separation = = + 𝑜 = + 𝑛 www.extremetech.com

Non Convex Optimization Sparse Recovery Matrix Completion Robust Regression Robust PCA

Non-convex Optimization: A Brief Introduction

Relaxation-based Techniques • “ Convexify ” the feasible set

Alternating Minimization Matrix Completion Robust PCA … also Robust Regression, coming up

Projected Gradient Descent Top 𝑡 elements by magnitude Perform 𝑙 -truncated SVD Sparse Recovery

Pursuit and Greedy Methods Set of “atoms” Sparse Recovery

Robust Regression: A Non-convex Approach

Linear Regression

Linear Regression image.frompo.com

Linear Regression with Noise ≈

Linear Regression with Noise Residual

Linear Regression with Noise

Linear Regression with Corruptions www.toonvectors.com

Robust Regression Corruptions are adversarial, adaptive, but only on a “few” locations

Robust Regression Corruptions are adversarial, adaptive, but only on a “few” locations Attempt 1 3

Robust Regression Corruptions are adversarial, adaptive, but only on a “few” locations Attempt 1 10

Robust Regression Corruptions are adversarial, adaptive, but only on a “few” locations Attempt 2 [Wright and Ma 2010*, Nguyen et al , 2013*]

Lessons from History If among these errors are some which appear too large to be admissible, then those equations which produced these errors will be rejected, as coming from too faulty experiments, and the unknowns will be determined by means of the other equations, which will then give much smaller errors Adrien-Marie Legendre, On the Method of Least Squares , 1805

Linear Regression with Corruptions

Linear Regression with Corruptions TORRENT-FC Thresholding Operator-based Robust RegrEssioN meThod [Bhatia et al , 2015]

TORRENT in Action!

Alt-Min in Theory Recovery Guarantees Robust against adaptive adversaries has access to data , gold model , and noise Requirement: Data needs to satisfy some “nice” properties Enough data needs to be present Guarantees: TORRENT will recover the gold model if i.e.

Alt-Min in Theory Convergence Rates Linear rate of convergence Suppose each alternation ≡ one step 1 After 𝑈 = log 𝜗 time steps Invariant: at time 𝑢 , “active set” s.t

Alt-Min in Practice Quality of Recovery [Bhatia et al 2015]

Alt-Min in Practice Speed of Recovery [Bhatia et al 2015]

Robust Regression: Application to Face Recognition Extended Yale B dataset, 38 people, 800 images

Face Recognition 10% noise 30% noise 50% noise 70% noise [Bhatia et al 2015]

Robust PCA: A Sketch and Application to Foreground Extraction in Images

The Alternating Projection Procedure [Netrapalli et al 2014]

Concluding Comments Non-convex optimization is an exciting area Widespread applications • Much better modelling of problems • Much more scalable algorithms • Provable guarantees So … • Full of opportunities • Full of challenges

Acknowledgements http://research.microsoft.com/en-us/projects/altmin/default.aspx Portions of this talk were based on joint work with Ambuj Tewari Kush Bhatia Prateek Jain U. Michigan, Ann Arbor Microsoft Research Microsoft Research

The Data Sciences Gang@IITK Medha Atre Sumit Ganguly Purushottam Harish Karnick Arnab Kar Bhattacharya Vinay Gaurav Indranil Saha Sandeep Shukla Namboodiri Piyush Rai Sharma

Machine Learning Vision, Image Databases, Processing Data Mining Our Strengths Online, Streaming Cyber-physical Algorithms Systems

Questions?

TORRENT as an Alt-Min Procedure • TORRENT indeed performs Alt-Min • Two variables in TORRENT – active set and model • encodes the complement of the corruption vector • TORRENT alternates between • Fixing model and choosing active set • Fixing active set and choosing model • Both steps reduce the residual as much as possible

Linear Regression with Corruptions TORRENT-GD Thresholding Operator-based Robust RegrEssioN meThod [Bhatia et al , 2015]

Linear Regression with Corruptions TORRENT-HYB Thresholding Operator-based Robust RegrEssioN meThod [Bhatia et al , 2015]

Some Recent Advances in Non-convex Optimization Purushottam Kar - PowerPoint PPT Presentation

Some Recent Advances in Non-convex Optimization Purushottam Kar IIT KANPUR Outline of the Talk Recap of Convex Optimization Why Non-convex Optimization? Non-convex Optimization: A Brief Introduction Robust Regression : A

Recent Advances in Photonic Recent Advances in Photonic effect employing IP- based distributed

Recent Advances in Biomolecular NMR Lucia Banci CERM University of Florence Recent Advances

Recent Advances in Biomolecular NMR Lucia Banci CERM University of Florence Recent Advances

Confl flict and Development: Recent Advances and Future Agendas Professor Patricia Justino

Recent Advances Aly Khawaja Outline STAR- CCM+: a complete simulation workflow Emphasis on

Recent advances in Mandelbrot martingales theory Julien Barral, Universit e Paris Nord

Recent Advances in Adap.ve Sampling and Reconstruc.on for Monte

Recent Advances in Adaptive Sampling and Reconstruction for Monte

Recent Advances in Two-loop Superstrings Eric DHoker Institut des Hautes Etudes Scientifiques,

Recent Advances in Adversarial Machine Learning Nicholas Carlini Google Research Recent

Recent advances on the acceleration of first-order methods in convex optimization . Juan

Some Recent Advances in Nonnegative Matrix Factorization and their Applications to Hyperspectral

Thank you for joining us. The program will commence momentarily. Recent Advances in Medical

Recent Theoretical Advances in Sparse Approximation Joel A. Tropp

Thank you for joining us. The program will commence momentarily. Recent Advances in Medical

Accuracy and Stability: recent advances in C.A.G.D. J.M. Pe na* *University of Zaragoza, Spain

From Declarative Signatures to Misuse IDS 4th International Symposium on Recent Advances in

Thank you for joining us. The program will commence momentarily. Recent Advances in Medical

Becoming Human Purpose: to re-evaluate recent advances in in th the sciences, and develop a

Recent Advances in Simulation for Marine Applications Milovan Peri CD-adapco, Nrnberg Office

Recent advances in fermentation technology for value-addition to cotton stalks and cottonseed Dr.

International conference on Recent Advances in Materials Science and Biophysics Department of

Recent advances in the development of ice cloud bulk scattering and absorption models for use with

Sediment Benchmarks Use in Screening and Recent Advances Sharon Thoms U.S. EPA Region 4