linear regression
play

Linear Regression David M. Blei COS424 Princeton University April - PowerPoint PPT Presentation

Sep 24, 2023 •205 likes •857 views

Linear Regression David M. Blei COS424 Princeton University April 10, 2008 D. Blei Linear Regression 1 / 65 Regression We have studied classification, the problem of automatically categorizing data into a set of discrete classes.

  1. Linear Regression David M. Blei COS424 Princeton University April 10, 2008 D. Blei Linear Regression 1 / 65

  2. Regression • We have studied classification, the problem of automatically categorizing data into a set of discrete classes. • E.g., based on its words, is an email spam or ham? • Regression is the problem of predicting a real-valued variable from data input. D. Blei Linear Regression 2 / 65

  3. Linear regression ● ● ● ● ● ● 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● ● ● ● ● ● response ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 ● ● ● ● −2 −1 0 1 2 input Data are a set of inputs and outputs D = { ( x n , y n ) } N n =1 D. Blei Linear Regression 3 / 65

  4. Linear regression ● ● ● ● ● ● 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● ● ● ● ● ● response ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 ● ● ● ● −2 −1 0 1 2 input The goal is to predict y from x using a linear function. D. Blei Linear Regression 4 / 65

  5. Examples ● ● ● ● ● ● 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● ● ● ● ● ● response ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 ● ● ● ● −2 −1 0 1 2 input • Given today’s weather, how much will it rain tomorrow? • Given today’s market, what will be the price of a stock tomorrow? • Given her emails, how long will a user stay on a page? • Others? D. Blei Linear Regression 5 / 65

  6. Linear regression ● ● ● ● ● ● 1.0 ( x n , y n ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● f ( x ) = β 0 + β x ● ● ● ● ● response ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 0.5 ● ● ● ● − 2 − 1 0 1 2 input D. Blei Linear Regression 6 / 65

  7. Multiple inputs • Usually, we have a vector of inputs, each representing a different feature of the data that might be predictive of the response. x = � x 1 , x 2 , . . . , x p � • The response is assumed to be a linear function of the input p � f ( x ) = β 0 + x i β i i =1 • Here, β ⊤ x = 0 is a hyperplane. D. Blei Linear Regression 7 / 65

  8. Multiple inputs Y • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • X 2 • • • X 1 D. Blei Linear Regression 8 / 65

  9. Flexibility of linear regression • This set-up is less limiting than you might imagine. • Inputs can be: • Any features of the data • Transformations of the original features, e.g., x 2 = log x 1 or x 2 = √ x 1 . • A basis expansion, e.g., x 2 = x 2 1 and x 3 = x 3 1 • Indicators of qualitative inputs, e.g., category • Interactions between inputs, e.g., x 1 = x 2 x 3 • Its simplicity and flexibility make linear regression one of the most important and widely used statistical prediction techniques. D. Blei Linear Regression 9 / 65

  10. Polynomial regression example 10 ● 8 6 ● response 4 ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 −1 0 1 2 input D. Blei Linear Regression 10 / 65

  11. Linear regression 10 ● 8 6 ● response 4 ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 −1 0 1 2 input f ( x ) = β 0 + β x D. Blei Linear Regression 11 / 65

  12. Polynomial regression 10 ● 8 6 ● response 4 ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 −1 0 1 2 input f ( x ) = β 0 + β 1 x + β 2 x 2 + β 3 x 3 D. Blei Linear Regression 12 / 65

  13. Fitting a regression ● ● ● • Given data D = { ( x n , y n ) } N ● 1.0 n =1 , ● ● ● ● ● ● ● ● ● ● find the coefficient β that can ● ● ● ● ● ● ● 0.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● predict y new from x new . ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● y ● ● ● • Simplifications: ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 ● • 0-intercept, i.e., β 0 = 0 ● ● ● ● ● ● ● ● ● • One input, i.e., p = 1 ● ● −1.0 ● ● ● ● ● ● ● ● • How should we proceed? ● −2 −1 0 1 2 x D. Blei Linear Regression 13 / 65

  14. Residual sum of squares ● ● ● 1.0 ● | ( y n − β x n ) | ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● ● y ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 0.5 ● ● ● ● ● ● ● ● ● ● ● ● − 1.0 ● ● ● ● ● ● ● ● ● − 2 − 1 0 1 2 x A reasonable approach is to minimize sum of the squared Euclidean distance between each prediction β x n and the truth y n N RSS ( β ) = 1 � ( y n − β x n ) 2 2 n =1 D. Blei Linear Regression 14 / 65

  15. RSS for two inputs Y • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • X 2 • • • X 1 D. Blei Linear Regression 15 / 65

  16. Optimizing β The objective function is N RSS ( β ) = 1 � ( y n − β x n ) 2 2 n =1 The derivative is N d � d β RSS ( β ) = − ( y n − β x n ) x n n =1 The optimal value is � N n =1 y n x n ˆ β = � n x 2 n D. Blei Linear Regression 16 / 65

  17. The optimal β ● ● ● ● • The optimal value is 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● ● ● ● � N ● ● ● ● ● ● ● n =1 y n x n ● ● ● ● ● ● ˆ ● ● β = ● ● ● ● ● ● ● ● ● ● ● n x 2 ● ● � 0.0 ● ● y ● ● ● ● ● ● ● n ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 ● ● ● ● ● ● ● ● ● ● • + values pull the slope up. ● ● −1.0 ● ● ● ● ● ● ● ● • − values pull the slope down ● −2 −1 0 1 2 x D. Blei Linear Regression 17 / 65

Recommend

Regression Methods 1. Linear Regression and Logistic Regression: definitions, and a common

Regression Methods 1. Linear Regression and Logistic Regression: definitions, and a common

0. Regression Methods 1. Linear Regression and Logistic Regression: definitions, and a common property CMU, 2004 fall, Andrew Moore, HW2, pr. 4 2. Linear Regression and Logistic Regression: Definitions Given an input vector X , linear

1.07k views • 22 slides
Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear

Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear

Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear models, assume that the response, Y , has a normal distribution with constant variance. Unlike linear models, however, the mean function may depend

159 views • 4 slides
LINEAR REGRESSION LINEAR REGRESSION - FROM A MACHINE LEARNING POINT OF VIEW 25 SIMPLE LINEAR

LINEAR REGRESSION LINEAR REGRESSION - FROM A MACHINE LEARNING POINT OF VIEW 25 SIMPLE LINEAR

LINEAR REGRESSION LINEAR REGRESSION - FROM A MACHINE LEARNING POINT OF VIEW 25 SIMPLE LINEAR REGRESSION 20 Sales 15 Starting point 10 Simplest parametric function 5 Easy to interpret the parameters: 0 50 100 150 200 250

1.12k views • 13 slides
Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied

Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied

Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied linear regression ; the output/label space Y was R . 90 80 delay 70 60 50 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 duration 1 / 68 Linear

607 views • 38 slides
Which models can be fit with linear regression? Simple linear regression in Matlab X = rand(3,3)

Which models can be fit with linear regression? Simple linear regression in Matlab X = rand(3,3)

Which models can be fit with linear regression? Simple linear regression in Matlab X = rand(3,3) b = pinv(X) * y X = b = 0.0467 0.5188 0.5317 14.3966 0.6587 0.3323 0.5070 50.9428 0.7573 0.0428 0.2532 -50.8102 y = rand(3,1) b = X \

643 views • 13 slides
Trendlines Simple Linear Regression Multiple Linear Regression Systematic Model

Trendlines Simple Linear Regression Multiple Linear Regression Systematic Model

Trendlines Simple Linear Regression Multiple Linear Regression Systematic Model Building Practical Issues Overfitting Categorical Variables Interaction Terms Non-linear Terms Linear y = a + bx Logarithmic

945 views • 65 slides
Outline The Simple Linear Regression Model (12.1) Fitting the Regression Line (12.2)

Outline The Simple Linear Regression Model (12.1) Fitting the Regression Line (12.2)

2/22/2007 219323 Probability and Statistics for Software Statistics for Software and Knowledge Engineers Lecture 13: Simple Linear Regression Simple Linear Regression and Correlation Monchai Sopitkamon, Ph.D. Outline The Simple Linear

968 views • 19 slides
Unit 7: Multiple linear regression 1. Introduction to multiple linear regression Sta 101 - Fall

Unit 7: Multiple linear regression 1. Introduction to multiple linear regression Sta 101 - Fall

Announcements Unit 7: Multiple linear regression 1. Introduction to multiple linear regression Sta 101 - Fall 2018 Project questions? Duke University, Department of Statistical Science Dr. Abrahamsen Slides posted at

537 views • 10 slides
Linear Models for Regression Greg Mori - CMPT 419/726 Bishop PRML Ch. 3 Regression Linear Basis

Linear Models for Regression Greg Mori - CMPT 419/726 Bishop PRML Ch. 3 Regression Linear Basis

Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Linear Models for Regression Greg Mori - CMPT 419/726 Bishop PRML Ch. 3 Regression Linear Basis Function Models

1.19k views • 84 slides
Regression 1: Linear Regression Marco Baroni Practical Statistics in R Outline Classic linear

Regression 1: Linear Regression Marco Baroni Practical Statistics in R Outline Classic linear

Regression 1: Linear Regression Marco Baroni Practical Statistics in R Outline Classic linear regression Linear regression in R Outline Classic linear regression Introduction Constructing the model Estimation Looking at the fitted model

1.13k views • 77 slides
Overview IAML: Linear Regression The linear model Fitting the linear model to data

Overview IAML: Linear Regression The linear model Fitting the linear model to data

Overview IAML: Linear Regression The linear model Fitting the linear model to data Probabilistic interpretation of the error function Nigel Goddard School of Informatics Examples of regression problems Dealing with multiple

325 views • 10 slides
Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear Regression M. Magdon-Ismail CSCI 4100/6100 recap: Approximation Versus Generalization VC Analysis Bias-Variance Analysis E out E in + ( d

390 views • 23 slides
Linear regression Linear regression is a simple approach to supervised learning. It assumes

Linear regression Linear regression is a simple approach to supervised learning. It assumes

Linear regression Linear regression is a simple approach to supervised learning. It assumes that the dependence of Y on X 1 , X 2 , . . . X p is linear. True regression functions are never linear! 7 6 f(X) 5 4 3 2 4 6 8 X

704 views • 48 slides
10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression another discriminative learning method As optimization Gradient descent As matrix inversion (Ordinary Least Squares) Overfitting

1.52k views • 97 slides
Introduction to Linear Regression Rebecca C. Steorts September 15, 2015 Today

Introduction to Linear Regression Rebecca C. Steorts September 15, 2015 Today

Introduction to Linear Regression Rebecca C. Steorts September 15, 2015 Today (Re-)Introduction to linear models and the model space What is linear regression Basic properties of linear regression Using data frames for statistical

849 views • 37 slides
Secure Linear Regression on Secure Linear Regression on Vertically Partitioned Datasets

Secure Linear Regression on Secure Linear Regression on Vertically Partitioned Datasets

Secure Linear Regression on Secure Linear Regression on Vertically Partitioned Datasets Vertically Partitioned Datasets Adria Gascon Phillipp Schoppmann Borja Balle Mariana Raykova Samee Zahur Jack Doerner David Evans Cryptography in the

233 views • 22 slides
Linear Regression Yijun Zhao Northeastern University Fall 2016 Yijun Zhao Linear Regression

Linear Regression Yijun Zhao Northeastern University Fall 2016 Yijun Zhao Linear Regression

Linear Regression Yijun Zhao Northeastern University Fall 2016 Yijun Zhao Linear Regression Regression Examples Any Attributes Continuous Value = x y { age , major , gender , race } GPA { income , credit score , profession }

613 views • 37 slides
CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.):

CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.):

CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.): Linear and Beyond 1. Review: Linear Regression (LR), LLSE 2. LR: Examples 3. Beyond LR: Quadratic Regression 4. Conditional Expectation (CE) and

2.12k views • 132 slides
Regression Methods 1. Linear Regression with only one parameter, and without offset; MLE and

Regression Methods 1. Linear Regression with only one parameter, and without offset; MLE and

0. Regression Methods 1. Linear Regression with only one parameter, and without offset; MLE and MAP estimation CMU, 2012 fall, Tom Mitchell, Ziv Bar-Joseph, midterm, pr. 3 2. Consider real-valued variables X and Y . The Y variable is

1.25k views • 86 slides
Multiple Linear Regression The population model In a simple linear regression model, a single

Multiple Linear Regression The population model In a simple linear regression model, a single

Multiple Linear Regression The population model In a simple linear regression model, a single response measure- ment Y is related to a single predictor (covariate, regressor) X for each observation. The critical assumption of the model is

859 views • 49 slides
Statistics for Applications Chapter 7: Regression 1/43 Heuristics of the linear regression (1)

Statistics for Applications Chapter 7: Regression 1/43 Heuristics of the linear regression (1)

Statistics for Applications Chapter 7: Regression 1/43 Heuristics of the linear regression (1) Consider a cloud of i.i.d. random points ( X i , Y i ) , i = 1 , . . . , n : 2/43 Heuristics of the linear regression (2) I Idea: Fit the best line

780 views • 44 slides
CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear Regression 1 / 26 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets, e.g. stock prices (hence

328 views • 29 slides
CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear Regression 1 / 30 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets, e.g. stock prices (hence

506 views • 34 slides
Concise Implementation of Linear Regression Concise Implementation of Linear Regression

Concise Implementation of Linear Regression Concise Implementation of Linear Regression

Concise Implementation of Linear Regression Concise Implementation of Linear Regression Generating Data Sets Generating Data Sets In [1]: from mxnet import autograd, nd num_inputs = 2 num_examples = 1000 true_w = nd.array([2, -3.4]) true_b =

782 views • 9 slides

More recommend