lecture 20 regression
play

Lecture 20: Regression Dr. Chengjiang Long Computer Vision - PowerPoint PPT Presentation

Feb 22, 2024 •270 likes •879 views

Lecture 20: Regression Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture 20 April 16, 2018 Outline Regression Overview Linear

  1. Lecture 20: Regression Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu

  2. Recap Previous Lecture 2 C. Long Lecture 20 April 16, 2018

  3. Outline Regression Overview • Linear Regression • Support Vector Regression • Logistic Regression • Deep Neural Network for Regression • 3 C. Long Lecture 20 April 16, 2018

  4. Outline Regression Overview • Linear Regression • Support Vector Regression • Logistic Regression • Deep Neural Network for Regression • 4 C. Long Lecture 20 April 16, 2018

  5. Regression Overview Hierarchy clustering • Logistic Regression Gaussian Mixture Model • Neural Networks 5 C. Long Lecture 20 April 16, 2018

  6. One Example ...... 6 C. Long Lecture 20 April 16, 2018

  7. Evaluation Metrics Root mean-square error (RMSE) • RMSE is a popular formula to measure the error rate of a • regression model. However, it can only be compared between models whose errors are measured in the same units. Mean absolute error (MAE) • MAE has the same unit as the original data, and it can only be • compared between models whose errors are measured in the same units. It is usually similar in magnitude to RMSE, but slightly smaller. 7 C. Long Lecture 20 April 16, 2018

  8. Evaluation Metrics Relative Squared Error (RSE) • Unlike RMSE, the relative squared error (RSE) can be compared • between models whose errors are measured in the different units. mean value Relative Absolute Error (RAE) • Like RSE , the relative absolute error (RAE) can be compared • between models whose errors are measured in the different units.. 8 C. Long Lecture 20 April 16, 2018

  9. Outline Regression Overview • Linear Regression • Support Vector Regression • Logistic Regression • Deep Neural Network for Regression • 9 C. Long Lecture 20 April 16, 2018

  10. Linear Regression Given data with n dimensional variables and 1 target- • variable (real number) where The objective: Find a function f that returns the best fit. • Assume that the relationship between X and y is • approximately linear. The model can be represented as (w represents coefficients and b is an intercept) 10 C. Long Lecture 20 April 16, 2018

  11. Linear Regression To find the best fit , we minimize the sum of squared • errors -> Least square estimation The solution can be found by solving • (By taking the derivative of the above objective function w.r.t. ) In MATLAB, the back-slash operator computes a least • square solution. 11 C. Long Lecture 20 April 16, 2018

  12. Linear Regression To avoid over-fitting, a regularization term can be introduced (minimize a magnitude of w) 12 C. Long Lecture 20 April 16, 2018

  13. Outline Regression Overview • Linear Regression • Support Vector Regression • Logistic Regression • Deep Neural Network for Regression • 13 C. Long Lecture 20 April 16, 2018

  14. Support Vector Regression Find a function , f ( x ), with at most ε- deviation from • the target y 14 C. Long Lecture 20 April 16, 2018

  15. Support Vector Regression 15 C. Long Lecture 20 April 16, 2018

  16. Soft margin 16 C. Long Lecture 20 April 16, 2018

  17. How about a non-linear case? 17 C. Long Lecture 20 April 16, 2018

  18. Linear versus Non-linear SVR 18 C. Long Lecture 20 April 16, 2018

  19. Dual problem 19 C. Long Lecture 20 April 16, 2018

  20. Kernel trick 20 C. Long Lecture 20 April 16, 2018

  21. Dual problem for non-linear case 21 C. Long Lecture 20 April 16, 2018

  22. Architecture of a regression machine [Alex J. Smola et al. A tutorial on support vector regression, 2004.] URL: https://alex.smola.org/papers/2004/SmoSch04.pdf 22 C. Long Lecture 20 April 16, 2018

  23. Outline Regression Overview • Linear Regression • Support Vector Regression • Logistic Regression • Deep Neural Network for Regression • 23 C. Long Lecture 20 April 16, 2018

  24. Logistic Regression Takes a probabilistic approach to learning • discriminative functions (i.e., a classifier) Logistic regression model: • 24 C. Long Lecture 20 April 16, 2018

  25. Interpretation of Hypothesis Output 25 C. Long Lecture 20 April 16, 2018

  26. Another Interpretation Equivalently , logistic regression assumes that • In other words , logistic regression assumes that the log • odds is a linear function of x 26 C. Long Lecture 20 April 16, 2018

  27. Logistic Regression Assume a threshold and ... • 27 C. Long Lecture 20 April 16, 2018

  28. Non-Linear DecisionBoundary Can apply basis function expansion to features , same • as with linear regression ? 28 C. Long Lecture 20 April 16, 2018

  29. Logistic Regression 29 C. Long Lecture 20 April 16, 2018

  30. Logistic Regression Objective Function Can’t just use squared loss as in linear regression • – Using the logistic regression model results in a non-convex optimization 30 C. Long Lecture 20 April 16, 2018

  31. Deriving the Cost Function via Maximum Likelihood Estimation 31 C. Long Lecture 20 April 16, 2018

  32. Deriving the Cost Function via Maximum Likelihood Estimation 32 C. Long Lecture 20 April 16, 2018

  33. Intuition Behind the Objective 33 C. Long Lecture 20 April 16, 2018

  34. Intuition Behind the Objective 34 C. Long Lecture 20 April 16, 2018

  35. Intuition Behind the Objective 35 C. Long Lecture 20 April 16, 2018

  36. Regularized Logistic Regression We can regularize logistic regression exactly as before • 36 C. Long Lecture 20 April 16, 2018

  37. Gradient Descent for Logistic Regression 37 C. Long Lecture 20 April 16, 2018

  38. Gradient Descent for Logistic Regression 38 C. Long Lecture 20 April 16, 2018

  39. Gradient Descent for Logistic Regression 39 C. Long Lecture 20 April 16, 2018

  40. Multi-Class Classification 40 C. Long Lecture 20 April 16, 2018

  41. Multi-Class Logistic Regression 41 C. Long Lecture 20 April 16, 2018

  42. Multi-Class Logistic Regression Split into One vs Rest : • Train a logistic regression classifier for each class • i to predict the probability that y = i with 42 C. Long Lecture 20 April 16, 2018

  43. Implementing Multi-Class Logistic Regression 43 C. Long Lecture 20 April 16, 2018

  44. Outline Regression Overview • Linear Regression • Support Vector Regression • Logistic Regression • Deep Neural Network for Regression • 44 C. Long Lecture 20 April 16, 2018

  45. DNN Regression For a two-layer MLP: • The network weights are adjusted to minimize an • output cost function 45 C. Long Lecture 20 April 16, 2018

  46. Computing the Partial Derivatives for Regression We use SSE and for a two layer network the linear final outputs • can be written : We can then use the chain rules for derivatives , as for the Single • Layer Perceptron , to give the derivatives with respect to the two sets of weights 46 C. Long Lecture 20 April 16, 2018

  47. Deriving the Back Propagation Algorithm for Regression All we now have to do is substitute our derivatives into the weight • update equations Then if the transfer function f(x) is a Sigmoid we can use f′(x) = • f(x).(1 – f(x)) to give These equations constitute the Back-Propagation Learning • Algorithm for Regression. 47 C. Long Lecture 20 April 16, 2018

  48. Classification + Localization: Task 48 C. Long Lecture 20 April 16, 2018

  49. Idea #1: Localization as Regression 49 C. Long Lecture 20 April 16, 2018

  50. Simple Recipe for Classification + Localization Step 1: Train ( or download ) a classification model • ( AlexNet , VGG , GoogLeNet ) 50 C. Long Lecture 20 April 16, 2018

  51. Simple Recipe for Classification + Localization Step 2: Attach new fully-connected “regression head” • to the network 51 C. Long Lecture 20 April 16, 2018

  52. Simple Recipe for Classification + Localization Step 3: Train the regression head only with SGD and • L2 loss 52 C. Long Lecture 20 April 16, 2018

  53. Simple Recipe for Classification + Localization Step 4: At test time use both heads • 53 C. Long Lecture 20 April 16, 2018

  54. Per-class vs class agnostic regression 54 C. Long Lecture 20 April 16, 2018

  55. Where to attach the regression head? 55 C. Long Lecture 20 April 16, 2018

  56. Aside: Localizing multiple objects 56 C. Long Lecture 20 April 16, 2018

  57. Aside: Human Pose Estimation 57 C. Long Lecture 20 April 16, 2018

  58. DNN Regression Applications Great results in: • Computer Vision • ○ Object Localization / Detection as DNN Regression ○ Self-driving Steering Command Prediction ○ Human Pose Regression Finance • ○ Currency Exchange Rate ○ Stock Price Prediction ○ Forecasting Financial Time Series ○ Crude Oil Price Prediction 58 C. Long Lecture 20 April 16, 2018

Recommend

Todays lecture Logistic regression How can we use logistic regression for reranking? Shay

Todays lecture Logistic regression How can we use logistic regression for reranking? Shay

Todays lecture Logistic regression How can we use logistic regression for reranking? Shay Cohen How do we set the parameters of a logistic regression model? (based on slides by Sharon Goldwater) How is logistic regression related

652 views • 9 slides
CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.):

CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.):

CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.): Linear and Beyond 1. Review: Linear Regression (LR), LLSE 2. LR: Examples 3. Beyond LR: Quadratic Regression 4. Conditional Expectation (CE) and

2.12k views • 132 slides
Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335 Cost function Gradient descent Linear classifiers and decision boundaries Dan Sheldon Classification Example: Hand-Written Digits Input: 20

541 views • 5 slides
Mixed-effects regression and eye-tracking data Lecture 2 of advanced regression methods for

Mixed-effects regression and eye-tracking data Lecture 2 of advanced regression methods for

Mixed-effects regression and eye-tracking data Lecture 2 of advanced regression methods for linguists Martijn Wieling and Jacolien van Rij Seminar fr Sprachwissenschaft University of Tbingen LOT Summer School 2013, Groningen, June 25 1 |

717 views • 38 slides
Lecture 10: Classification and Logistic Regression CS109A Introduction to Data Science Pavlos

Lecture 10: Classification and Logistic Regression CS109A Introduction to Data Science Pavlos

Lecture 10: Classification and Logistic Regression CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Lecture Outline Why not Linear Regression? Binary Response & Logistic Regression Estimating the Simple

878 views • 58 slides
CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear Regression 1 / 30 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets, e.g. stock prices (hence

506 views • 34 slides
CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear Regression 1 / 26 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets, e.g. stock prices (hence

328 views • 29 slides
Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression

Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression

Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression Gradient Descent M. Magdon-Ismail CSCI 4100/6100 recap: Linear Classification and Regression The linear signal: s = w t x Good Features are Important

272 views • 23 slides
Introduction to mixed-effects regression Lecture 1 of advanced regression for linguists Martijn

Introduction to mixed-effects regression Lecture 1 of advanced regression for linguists Martijn

Introduction to mixed-effects regression Lecture 1 of advanced regression for linguists Martijn Wieling and Jacolien van Rij Seminar fr Sprachwissenschaft University of Tbingen LOT Summer School 2013, Groningen, June 24 1 | Martijn Wieling

1.02k views • 45 slides
Lecture 12: Effect modification, and confounding in logistic regression Ani Manichaikul

Lecture 12: Effect modification, and confounding in logistic regression Ani Manichaikul

Lecture 12: Effect modification, and confounding in logistic regression Ani Manichaikul amanicha@jhsph.edu 4 May 2007 Today n Categorical predictor n create dummy variables n just like for linear regression n Comparing nested models that

889 views • 70 slides
15.1 Last Lecture Want to solve a regression problem. confidence band f = argmin f 2

15.1 Last Lecture Want to solve a regression problem. confidence band f = argmin f 2

CS/CNS/EE 253: Advanced Topics in Machine Learning Topic: Nonparametric learning #4: Gaussian Process Regression Lecturer: Andreas Krause Scribe: Tim Black Date: March 1, 2010 15.1 Last Lecture Want to solve a regression problem. confidence band

256 views • 4 slides
Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Announcements Section : Friday 1:30-2:45pm : @ MD 123 (only this Friday) A-section: Today:

921 views • 65 slides
Binary Logistic Regression + Multinomial Logistic Regression Matt Gormley Lecture 10 Feb. 17,

Binary Logistic Regression + Multinomial Logistic Regression Matt Gormley Lecture 10 Feb. 17,

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Binary Logistic Regression + Multinomial Logistic Regression Matt Gormley Lecture 10 Feb. 17, 2020 1 Reminders

452 views • 43 slides
Lecture 11: Interpreting logistic regression models Ani Manichaikul amanicha@jhsph.edu 3 May

Lecture 11: Interpreting logistic regression models Ani Manichaikul amanicha@jhsph.edu 3 May

Lecture 11: Interpreting logistic regression models Ani Manichaikul amanicha@jhsph.edu 3 May 2007 Logistic regression n Framework and ideas of linear modelling similar to linear regression n Still have a systematic and probabilistic part to

1.29k views • 62 slides
Lecture 5: Multiple Linear Regression CS109A Introduction to Data Science Pavlos Protopapas and

Lecture 5: Multiple Linear Regression CS109A Introduction to Data Science Pavlos Protopapas and

Lecture 5: Multiple Linear Regression CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Lecture Outline Simple Regression: Predictor variables Standard Errors Evaluating Significance of Predictors Hypothesis

540 views • 50 slides
Lecture 6 Jan-Willem van de Meent Regression Curve Fitting (according to XKCD)

Lecture 6 Jan-Willem van de Meent Regression Curve Fitting (according to XKCD)

Unsupervised Machine Learning and Data Mining DS 5230 / DS 4420 - Fall 2018 Lecture 6 Jan-Willem van de Meent Regression Curve Fitting (according to XKCD) https://xkcd.com/2048/ Linear Regression Goal: Approximate points with a line or

768 views • 55 slides
Lecture 11: Logistic Regression Extensions and Evaluating Classification CS109A Introduction to

Lecture 11: Logistic Regression Extensions and Evaluating Classification CS109A Introduction to

Lecture 11: Logistic Regression Extensions and Evaluating Classification CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Lecture Outline Logistic Regression: a Brief Review Classification Boundaries

422 views • 41 slides
Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC

Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC

Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Module 2: Classification Why not Linear Regression? Binary

1.1k views • 77 slides
BS2247 Introduction to Econometrics Lecture 3: The simple regression model OLS, Algebraic

BS2247 Introduction to Econometrics Lecture 3: The simple regression model OLS, Algebraic

BS2247 Introduction to Econometrics Lecture 3: The simple regression model OLS, Algebraic properties, Goodness-of-fit Dr. Kai Sun Aston Business School 1 / 28 The Simple Regression Model: y = 0 + 1 x + u In the simple linear regression of

708 views • 54 slides
Lecture #4: Introduction to Regression Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos

Lecture #4: Introduction to Regression Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos

Lecture #4: Introduction to Regression Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Announcements Data Statistical Modeling Regression vs. Classification Error,

588 views • 56 slides
Lecture #11: Logistic Regression - Part II Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A

Lecture #11: Logistic Regression - Part II Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A

Lecture #11: Logistic Regression - Part II Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Logistic Regression: a Brief Review Classification Boundaries

793 views • 62 slides
Lecture 5: Logistic Regression Feb 10 2020 Lecturer: Steven Wu Scribe: Steven Wu Last lecture,

Lecture 5: Logistic Regression Feb 10 2020 Lecturer: Steven Wu Scribe: Steven Wu Last lecture,

CSCI 5525 Machine Learning Fall 2019 Lecture 5: Logistic Regression Feb 10 2020 Lecturer: Steven Wu Scribe: Steven Wu Last lecture, we give several convex surrogate loss functions to replace the zero-one loss func- tion, which is NP-hard to

449 views • 5 slides
Logistic Regression James H. Steiger Department of Psychology and Human Development Vanderbilt

Logistic Regression James H. Steiger Department of Psychology and Human Development Vanderbilt

Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Logistic Regression James H. Steiger

777 views • 54 slides
Lecture 10: Introduction to Logistic Regression Ani Manichaikul amanicha@jhsph.edu 2 May 2007

Lecture 10: Introduction to Logistic Regression Ani Manichaikul amanicha@jhsph.edu 2 May 2007

Lecture 10: Introduction to Logistic Regression Ani Manichaikul amanicha@jhsph.edu 2 May 2007 Logistic Regression n Regression for a response variable that follows a binomial distribution n Recall the binomial model n And the Binomial

1.04k views • 55 slides

More recommend