lecture 4 optimization
play

Lecture 4: Optimization Justin Johnson Lecture 4 - 1 September - PowerPoint PPT Presentation

Jun 07, 2023 •260 likes •1.12k views

Lecture 4: Optimization Justin Johnson Lecture 4 - 1 September 16, 2019 Waitlist Update We will open the course for enrollment later today / tomorrow Justin Johnson Lecture 4 - 2 September 16, 2019 Reminder: Assignment 1 Was due

  1. Lecture 4: Optimization Justin Johnson Lecture 4 - 1 September 16, 2019

  2. Waitlist Update We will open the course for enrollment later today / tomorrow Justin Johnson Lecture 4 - 2 September 16, 2019

  3. Reminder: Assignment 1 Was due yesterday! (But you do have late days…) Justin Johnson Lecture 4 - 3 September 16, 2019

  4. Assignment 2 • Will be released today • Use SGD to train linear classifiers and fully-connected networks • After today, can do linear classifiers section • After Wednesday, can do fully-connected networks • If you have a hard time computing derivatives, wait for next Monday’s lecture on backprop • Due Monday September 30, 11:59pm (two weeks from today) Justin Johnson Lecture 4 - 4 September 16, 2019

  5. Course Update - A1: 10% - A2: 10% - A3: 10% - A4: 10% - A5: 10% - A6: 10% - Midterm: 20% - Final: 20% Justin Johnson Lecture 4 - 5 September 16, 2019

  6. Course Update: No Final Exam - A1: 10% - A1: 10% - A2: 10% - A2: 13% - A3: 10% - A3: 13% - A4: 10% - A4: 13% Expect A5 and A6 - A5: 10% - A5: 13% to be longer than other homework - A6: 10% - A6: 13% - Midterm: 20% - Midterm: 25% - Final: 20% - Final Justin Johnson Lecture 4 - 6 September 16, 2019

  7. Last Time: Linear Classifiers Visual Viewpoint Algebraic Viewpoint Geometric Viewpoint One template Hyperplanes f(x,W) = Wx per class cutting up space Justin Johnson Lecture 4 - 7 September 16, 2019

  8. Last Time: Loss Functions quantify preferences We have some dataset of (x, y) - We have a score function: - We have a loss function : - Linear classifier Softmax SVM Full loss Justin Johnson Lecture 4 - 8 September 16, 2019

  9. Last Time: Loss Functions quantify preferences Q : How do we find the best W? We have some dataset of (x, y) - We have a score function: - We have a loss function : - Linear classifier Softmax SVM Full loss Justin Johnson Lecture 4 - 9 September 16, 2019

  10. Optimization Justin Johnson Lecture 4 - 10 September 16, 2019

  11. This image is CC0 1.0 public domain Walking man image is CC0 1.0 public domain Justin Johnson Lecture 4 - 11 September 16, 2019

  12. This image is CC0 1.0 public domain Walking man image is CC0 1.0 public domain Justin Johnson Lecture 4 - 12 September 16, 2019

  13. Idea #1: Random Search ch (bad idea!) Justin Johnson Lecture 4 - 13 September 16, 2019

  14. Idea #1: Random Search ch (bad idea!) 15.5% accuracy! not bad! Justin Johnson Lecture 4 - 14 September 16, 2019

  15. Idea #1: Random Search ch (bad idea!) 15.5% accuracy! not bad! (SOTA is ~95%) Justin Johnson Lecture 4 - 15 September 16, 2019

  16. Idea #2: Fo Follow the slope Justin Johnson Lecture 4 - 16 September 16, 2019

  17. Idea #2 : : Follow th the e slope In 1-dimension, the derivative of a function gives the slope: Justin Johnson Lecture 4 - 17 September 16, 2019

  18. Idea #2 : : Follow th the e slope In 1-dimension, the derivative of a function gives the slope: In multiple dimensions, the gradient is the vector of (partial derivatives) along each dimension The slope in any direction is the dot product of the direction with the gradient The direction of steepest descent is the negative gradient Justin Johnson Lecture 4 - 18 September 16, 2019

  19. current W: gradient dL/dW: [0.34, [?, -1.11, ?, 0.78, ?, 0.12, ?, 0.55, ?, 2.81, ?, -3.1, ?, -1.5, ?, 0.33,…] ?,…] loss 1.25347 Justin Johnson Lecture 4 - 19 September 16, 2019

  20. current W: W + h (first dim) : gradient dL/dW: [0.34, [0.34 + 0.0001 , [?, -1.11, -1.11, ?, 0.78, 0.78, ?, 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25322 Justin Johnson Lecture 4 - 20 September 16, 2019

  21. current W: W + h (first dim) : gradient dL/dW: [0.34, [0.34 + 0.0001 , [ -2.5 , -1.11, -1.11, ?, 0.78, 0.78, ?, 0.12, 0.12, ?, (1.25322 - 1.25347)/0.0001 0.55, 0.55, ?, = -2.5 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25322 Justin Johnson Lecture 4 - 21 September 16, 2019

  22. current W: W + h (second dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11 + 0.0001 , ?, 0.78, 0.78, ?, 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25353 Justin Johnson Lecture 4 - 22 September 16, 2019

  23. current W: W + h (second dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11 + 0.0001 , 0.6 , 0.78, 0.78, ?, 0.12, 0.12, ?, 0.55, 0.55, ?, (1.25353 - 1.25347)/0.0001 2.81, 2.81, ?, = 0.6 -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25353 Justin Johnson Lecture 4 - 23 September 16, 2019

  24. current W: W + h (third dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11, 0.6, 0.78, 0.78 + 0.0001 , ?, 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25347 Justin Johnson Lecture 4 - 24 September 16, 2019

  25. current W: W + h (third dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11, 0.6, 0.78, 0.78 + 0.0001 , 0.0 , 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, (1.25347 - 1.25347)/0.0001 -3.1, -3.1, ?, = 0.0 -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25347 Justin Johnson Lecture 4 - 25 September 16, 2019

  26. current W: W + h (third dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11, 0.6, 0.78, 0.78 + 0.0001 , 0.0 , 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, Numeric Gradient : ?, - Slow: O(#dimensions) -1.5, -1.5, ?, - Approximate 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25347 Justin Johnson Lecture 4 - 26 September 16, 2019

  27. Loss is a function of W want Justin Johnson Lecture 4 - 27 September 16, 2019

  28. Loss is a function of W: Analytic Gradient want Use calculus to compute an analytic gradient This image is in the public domain This image is in the public domain Justin Johnson Lecture 4 - 28 September 16, 2019

  29. current W: gradient dL/dW: [0.34, [-2.5, dL/dW = ... -1.11, 0.6, (some function 0.78, 0, data and W) 0.12, 0.2, 0.55, 0.7, 2.81, -0.5, -3.1, 1.1, -1.5, 1.3, 0.33,…] -2.1,…] loss 1.25347 Justin Johnson Lecture 4 - 29 September 16, 2019

  30. current W: gradient dL/dW: [0.34, [-2.5, dL/dW = ... -1.11, 0.6, (some function 0.78, 0, data and W) 0.12, 0.2, 0.55, 0.7, 2.81, -0.5, -3.1, (In practice we will compute 1.1, dL/dW using backpropagation; -1.5, 1.3, see Lecture 6) 0.33,…] -2.1,…] loss 1.25347 Justin Johnson Lecture 4 - 30 September 16, 2019

  31. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 31 September 16, 2019

  32. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 32 September 16, 2019

  33. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 33 September 16, 2019

  34. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 34 September 16, 2019

  35. Gradient Descent Iteratively step in the direction of the negative gradient (direction of local steepest descent) Hyperparameters : - Weight initialization method - Number of steps - Learning rate Justin Johnson Lecture 4 - 35 September 16, 2019

  36. negative Gradient Descent gradient original W direction Iteratively step in the direction of W_2 the negative gradient (direction of local steepest descent) Hyperparameters : - Weight initialization method - Number of steps - Learning rate W_1 Justin Johnson Lecture 4 - 36 September 16, 2019

  37. Gradient Descent Iteratively step in the direction of the negative gradient (direction of local steepest descent) Hyperparameters : - Weight initialization method - Number of steps - Learning rate Justin Johnson Lecture 4 - 37 September 16, 2019

  38. Batch ch Gradient Desce cent Full sum expensive when N is large! Justin Johnson Lecture 4 - 38 September 16, 2019

  39. cent (SGD) Stoch chastic c Gradient Desce Full sum expensive when N is large! Approximate sum using a minibatch of examples 32 / 64 / 128 common Hyperparameters : - Weight initialization - Number of steps - Learning rate - Batch size - Data sampling Justin Johnson Lecture 4 - 39 September 16, 2019

Recommend

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic:

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic:

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic: eigenvalue problems PDE Constrained Optimization We will now consider optimization based on a function that depends on the solution of a PDE Let us

604 views • 26 slides
CMU-Q 15-381 Lecture 8: Optimization I: Optimization for CSP Local Search Teacher: Gianni A.

CMU-Q 15-381 Lecture 8: Optimization I: Optimization for CSP Local Search Teacher: Gianni A.

CMU-Q 15-381 Lecture 8: Optimization I: Optimization for CSP Local Search Teacher: Gianni A. Di Caro L OCAL S EARCH FOR CSP Real-life CSPs can be very large and hard to solve Methods so far: construct a solution by assigning one

767 views • 19 slides
CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26

CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26

CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26 Overview Weve talked a lot about how to compute gradients. What do we actually do with them? Todays lecture: various things that can go

675 views • 29 slides
CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25

CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25

CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25 Overview Weve talked a lot about how to compute gradients. What do we actually do with them? Todays lecture: various things that can go

676 views • 28 slides
Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is

Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is

Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is More Than Half Done! If you have comments they are very welcome y y Lectures Lecture notes Weekly Homework Midterm

795 views • 42 slides
Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric

Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric

Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric Optimization Minimize/maximize something subject to some constraints Have seen: Linear Programming Minimum Enclosing Ball

337 views • 17 slides
Optimization X Heuristic Optimization R Lecture 1 f : X R Algorithm Engineering Group

Optimization X Heuristic Optimization R Lecture 1 f : X R Algorithm Engineering Group

Optimization X Heuristic Optimization R Lecture 1 f : X R Algorithm Engineering Group Hasso Plattner Institute, University of Potsdam Goal: 14 April 2015 Find z X such that f ( z ) f ( x ) for all x X (minimization) Find z

612 views • 5 slides
AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector

AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector

AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector optimization Note: last years midterm is now available on the website for practice Motivation: Optimization If the objective function or any of the

1.1k views • 48 slides
15-780 Graduate Artificial Intelligence: Optimization J. Zico Kolter (this lecture) and Ariel

15-780 Graduate Artificial Intelligence: Optimization J. Zico Kolter (this lecture) and Ariel

15-780 Graduate Artificial Intelligence: Optimization J. Zico Kolter (this lecture) and Ariel Procaccia Carnegie Mellon University Spring 2017 1 Outline Introduction to optimization Types of optimization problems, convexity Solving

831 views • 30 slides
Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013

Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013

Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013 Suvrit Sra Organizational Info Quiz coming up on 19th Feb. Project teams by 19th Feb Good if you can mix your research with class projects

1.19k views • 98 slides
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 13: Optimality Conditions for

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 13: Optimality Conditions for

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 13: Optimality Conditions for Convex Optimization Instructor: Shaddin Dughmi Announcements Today: short lecture wrapping up convex optimization Thursday: We begin combinatorial

396 views • 20 slides
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems Instructor: Shaddin Dughmi Announcements Homework: Due beginning of next class Must submit a hard copy, unless you have a good excuse If using late

934 views • 62 slides
Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos

Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos

Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Outline Optimization Challenges in Optimization Momentum Adaptive Learning Rate Parameter Initialization Batch

457 views • 45 slides
10 Analytics & Optimization From Code to Product gidgreen.com/course Lecture 10

10 Analytics & Optimization From Code to Product gidgreen.com/course Lecture 10

10 Analytics & Optimization From Code to Product gidgreen.com/course Lecture 10 Introduction Data collection Website metrics Optimization Competitive intelligence Surveys Tools and books From Code to Product

948 views • 68 slides
Lecture 6: Optimization CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline

Lecture 6: Optimization CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline

Lecture 6: Optimization CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline Optimization Challenges in Optimization Momentum Adaptive Learning Rate Parameter Initialization Batch Normalization CS109B, P

495 views • 47 slides
Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic

Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic

An Example Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic Methods for Continuous 1. An Application Example in Econometrics Optimization 2. Heuristics for Continuos Optimization Marco Chiarandini

410 views • 6 slides
ADVANCED ALGORITHMS Lecture 19: optimization, linear programming 1 ANNOUNCEMENTS HW 4 is

ADVANCED ALGORITHMS Lecture 19: optimization, linear programming 1 ANNOUNCEMENTS HW 4 is

ADVANCED ALGORITHMS Lecture 19: optimization, linear programming 1 ANNOUNCEMENTS HW 4 is due on Monday, November 5 lb mod 2 Project meetings air n L 2 LAST CLASS Optimization {x 1 , x 2 , , x n } are variables

641 views • 24 slides
EC400 Part II, Math for Micro: Lecture 1 Leonardo Felli NAB.SZT 9 September 2010 Course Outline

EC400 Part II, Math for Micro: Lecture 1 Leonardo Felli NAB.SZT 9 September 2010 Course Outline

EC400 Part II, Math for Micro: Lecture 1 Leonardo Felli NAB.SZT 9 September 2010 Course Outline Lecture 1: Tools for optimization (Quadratic forms). Lecture 2: Tools for optimization (Taylors expansion) and Unconstrained optimization.

916 views • 64 slides
Dimension Free Optimization and Non-Convex Optimization Instructor: Sham Kakade 1 Non-convex

Dimension Free Optimization and Non-Convex Optimization Instructor: Sham Kakade 1 Non-convex

CSE 547/Stat 548: Machine Learning for Big Data Lecture Dimension Free Optimization and Non-Convex Optimization Instructor: Sham Kakade 1 Non-convex optimization and Black-Box Oracle Complexity Suppose we are trying to minimize the function F (

273 views • 4 slides
Evolutionary Algorithm 2. Swarm Intelligence and Ant Colony Optimization Ant Colony Optimization

Evolutionary Algorithm 2. Swarm Intelligence and Ant Colony Optimization Ant Colony Optimization

Outline DM63 HEURISTICS FOR COMBINATORIAL OPTIMIZATION 1. Evolutionary Algorithms Lecture 10 Evolutionary Algorithm 2. Swarm Intelligence and Ant Colony Optimization Ant Colony Optimization Background Ant Colony Optimization: the

298 views • 12 slides
Convex Optimization for Data Science Gasnikov Alexander gasnikov.av@mipt.ru Lecture 6.

Convex Optimization for Data Science Gasnikov Alexander gasnikov.av@mipt.ru Lecture 6.

Convex Optimization for Data Science Gasnikov Alexander gasnikov.av@mipt.ru Lecture 6. Gradient-free methods. Coordinate descent February, 2017 1 Main books: Spall J.C. Introduction to stochastic search and optimization: estimation, simula- tion

454 views • 28 slides
Ant Colony Optimization Capacited Vehicle Routing Linear Ordering Exercises Marco Chiarandini

Ant Colony Optimization Capacited Vehicle Routing Linear Ordering Exercises Marco Chiarandini

Outline DM63 HEURISTICS FOR COMBINATORIAL OPTIMIZATION Ant Colony Optimization: the Metaheuristic Application Examples Generalized Assignment Problem (GAP) Lecture 11 Connection between ACO and other Metaheuristics Encodings Ant Colony

261 views • 5 slides
Optimization Problems and Wrap-Up CS 221 Lecture 14 Tue 6 December 2011 Agenda 1.

Optimization Problems and Wrap-Up CS 221 Lecture 14 Tue 6 December 2011 Agenda 1.

Optimization Problems and Wrap-Up CS 221 Lecture 14 Tue 6 December 2011 Agenda 1. Announcements 2. Solving Optimization Problems in Excel and MATLAB (Text Chapter 10) 3. Other nifty functions in (standard) MATLAB Image

485 views • 21 slides
Game Theory: Lecture #8 Outline: Individual Optimization Security Strategies Optimization

Game Theory: Lecture #8 Outline: Individual Optimization Security Strategies Optimization

Game Theory: Lecture #8 Outline: Individual Optimization Security Strategies Optimization and Strategy Previous focus: Given individual preferences, compute group preference? Given individual preferences, compute satisfactory

343 views • 9 slides

More recommend