Linear Regression Part 2: Residuals and Errors INFO-1301, - PowerPoint PPT Presentation

Sep 13, 2022 •127 likes •288 views

Linear Regression Part 2: Residuals and Errors INFO-1301, Quantitative Reasoning 1 University of Colorado Boulder April 21, 2017 Prof. Michael Paul Fitting Linear Functions Where does a linear function such as y = 9.607x + 111.958 come

Linear Regression Part 2: Residuals and Errors INFO-1301, Quantitative Reasoning 1 University of Colorado Boulder April 21, 2017 Prof. Michael Paul
Fitting Linear Functions Where does a linear function such as “y = 9.607x + 111.958” come from? Want to pick slope and y-intercept (y= mx + b ) such that the line is as close as possible to the true data points • Want to minimize distance from each point to the line • We’ll be more concrete today
Fitting Linear Functions The process of picking the parameters of a function (e.g., m and b ) to make it is close as possible to a set of data points is regression If the function is linear (i.e., a line) then this is linear regression Statistical software such as MiniTab Express can perform linear regression automatically
Residuals The residual of a point ( x i , y i ) is the difference between the true y i value and the value you estimated based on your best-fit line: e i = y i – ( mx i + b ) Also referred to as the error of your line at that point The size of a residual is its absolute value: | e i |
Residuals The average residual size can tell you the average error you will make if you estimate new data points (e.g., interpolation or extrapolation) This is only true if the new data points follow the same pattern as the data you originally observed • More likely to be true for interpolation than extrapolation
Residual Plots Original data: Residuals:
Practice
Practice (7.21) Suppose we fit a regression line to predict the shelf life of an apple based on its weight. For a particular apple, we predict the shelf life to be 4.6 days. The apple’s residual is -0.6 days. Did we over or under estimate the shelf-life of the apple? Explain your reasoning.
Fitting Linear Functions How to choose m and b ? Pick them so that the residuals are as small as possible. • Could minimize the absolute value of the residuals • More common: minimize the square of the residuals • “least squares regression” • This will favor solutions where no residual is especially large (outliers penalized more)
Root Mean Squared Error 2 )/n MSE = ( Σ i e i n is the number of points RMSE = √ (MSE) Example: residuals are 1, 2, -1, 2, -2 • RMSE = [[(1) 2 +(2) 2 +(-1) 2 +(2) 2 +(-2) 2 ]/5] .5 = [[1+4+1+4+4]/5] .5 = [14/5] .5 = 1.67
Root Mean Squared Error • Generally: • 68% of residuals are within 1 RSME of line. • 95% of residuals are within 2 RSME of line. • Look familiar? • In “nice” cases, residuals form a normal distribution around the least square line • The mean residual is 0. • The standard deviation is the RMSE. • Can use Z-table to estimate probability of having an errors of a certain size.
Conditions for Least Squares Regression • Eyeball that the data is linear (fits a line) • Random residuals not too far from line (outliers can be a problem) • The size of the residuals roughly constant (not those that get larger at one end) • No repeating patterns in the data (e.g. time series)
Conditions for Least Squares Regression Figure 7.13, page 342
Practice
Practice
More Practice

Recommend

Regression Methods 1. Linear Regression and Logistic Regression: definitions, and a common

0. Regression Methods 1. Linear Regression and Logistic Regression: definitions, and a common property CMU, 2004 fall, Andrew Moore, HW2, pr. 4 2. Linear Regression and Logistic Regression: Definitions Given an input vector X , linear

1.07k views • 22 slides

Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear

Notes on the Non-linear Regression The model Non-linear regression models, like ordinary linear models, assume that the response, Y , has a normal distribution with constant variance. Unlike linear models, however, the mean function may depend

159 views • 4 slides

LINEAR REGRESSION LINEAR REGRESSION - FROM A MACHINE LEARNING POINT OF VIEW 25 SIMPLE LINEAR

LINEAR REGRESSION LINEAR REGRESSION - FROM A MACHINE LEARNING POINT OF VIEW 25 SIMPLE LINEAR REGRESSION 20 Sales 15 Starting point 10 Simplest parametric function 5 Easy to interpret the parameters: 0 50 100 150 200 250

1.12k views • 13 slides

Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied

Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied linear regression ; the output/label space Y was R . 90 80 delay 70 60 50 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 duration 1 / 68 Linear

607 views • 38 slides

Which models can be fit with linear regression? Simple linear regression in Matlab X = rand(3,3)

Which models can be fit with linear regression? Simple linear regression in Matlab X = rand(3,3) b = pinv(X) * y X = b = 0.0467 0.5188 0.5317 14.3966 0.6587 0.3323 0.5070 50.9428 0.7573 0.0428 0.2532 -50.8102 y = rand(3,1) b = X \

643 views • 13 slides

Trendlines Simple Linear Regression Multiple Linear Regression Systematic Model

Trendlines Simple Linear Regression Multiple Linear Regression Systematic Model Building Practical Issues Overfitting Categorical Variables Interaction Terms Non-linear Terms Linear y = a + bx Logarithmic

945 views • 65 slides

Outline The Simple Linear Regression Model (12.1) Fitting the Regression Line (12.2)

2/22/2007 219323 Probability and Statistics for Software Statistics for Software and Knowledge Engineers Lecture 13: Simple Linear Regression Simple Linear Regression and Correlation Monchai Sopitkamon, Ph.D. Outline The Simple Linear

968 views • 19 slides

Unit 7: Multiple linear regression 1. Introduction to multiple linear regression Sta 101 - Fall

Announcements Unit 7: Multiple linear regression 1. Introduction to multiple linear regression Sta 101 - Fall 2018 Project questions? Duke University, Department of Statistical Science Dr. Abrahamsen Slides posted at

537 views • 10 slides

Linear Models for Regression Greg Mori - CMPT 419/726 Bishop PRML Ch. 3 Regression Linear Basis

Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Linear Models for Regression Greg Mori - CMPT 419/726 Bishop PRML Ch. 3 Regression Linear Basis Function Models

1.19k views • 84 slides

Regression 1: Linear Regression Marco Baroni Practical Statistics in R Outline Classic linear

Regression 1: Linear Regression Marco Baroni Practical Statistics in R Outline Classic linear regression Linear regression in R Outline Classic linear regression Introduction Constructing the model Estimation Looking at the fitted model

1.13k views • 77 slides

Overview IAML: Linear Regression The linear model Fitting the linear model to data

Overview IAML: Linear Regression The linear model Fitting the linear model to data Probabilistic interpretation of the error function Nigel Goddard School of Informatics Examples of regression problems Dealing with multiple

325 views • 10 slides

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear Regression M. Magdon-Ismail CSCI 4100/6100 recap: Approximation Versus Generalization VC Analysis Bias-Variance Analysis E out E in + ( d

390 views • 23 slides

Linear regression Linear regression is a simple approach to supervised learning. It assumes

Linear regression Linear regression is a simple approach to supervised learning. It assumes that the dependence of Y on X 1 , X 2 , . . . X p is linear. True regression functions are never linear! 7 6 f(X) 5 4 3 2 4 6 8 X

704 views • 48 slides

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression another discriminative learning method As optimization Gradient descent As matrix inversion (Ordinary Least Squares) Overfitting

1.52k views • 97 slides

Introduction to Linear Regression Rebecca C. Steorts September 15, 2015 Today

Introduction to Linear Regression Rebecca C. Steorts September 15, 2015 Today (Re-)Introduction to linear models and the model space What is linear regression Basic properties of linear regression Using data frames for statistical

849 views • 37 slides

Secure Linear Regression on Secure Linear Regression on Vertically Partitioned Datasets

Secure Linear Regression on Secure Linear Regression on Vertically Partitioned Datasets Vertically Partitioned Datasets Adria Gascon Phillipp Schoppmann Borja Balle Mariana Raykova Samee Zahur Jack Doerner David Evans Cryptography in the

233 views • 22 slides

Linear Regression Yijun Zhao Northeastern University Fall 2016 Yijun Zhao Linear Regression

Linear Regression Yijun Zhao Northeastern University Fall 2016 Yijun Zhao Linear Regression Regression Examples Any Attributes Continuous Value = x y { age , major , gender , race } GPA { income , credit score , profession }

613 views • 37 slides

CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.):

CS70: Lecture 35. Regression (contd.): Linear and Beyond CS70: Lecture 35. Regression (contd.): Linear and Beyond 1. Review: Linear Regression (LR), LLSE 2. LR: Examples 3. Beyond LR: Quadratic Regression 4. Conditional Expectation (CE) and

2.12k views • 132 slides

Regression Methods 1. Linear Regression with only one parameter, and without offset; MLE and

0. Regression Methods 1. Linear Regression with only one parameter, and without offset; MLE and MAP estimation CMU, 2012 fall, Tom Mitchell, Ziv Bar-Joseph, midterm, pr. 3 2. Consider real-valued variables X and Y . The Y variable is

1.25k views • 86 slides

Multiple Linear Regression The population model In a simple linear regression model, a single

Multiple Linear Regression The population model In a simple linear regression model, a single response measure- ment Y is related to a single predictor (covariate, regressor) X for each observation. The critical assumption of the model is

859 views • 49 slides

Statistics for Applications Chapter 7: Regression 1/43 Heuristics of the linear regression (1)

Statistics for Applications Chapter 7: Regression 1/43 Heuristics of the linear regression (1) Consider a cloud of i.i.d. random points ( X i , Y i ) , i = 1 , . . . , n : 2/43 Heuristics of the linear regression (2) I Idea: Fit the best line

780 views • 44 slides

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear Regression 1 / 26 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets, e.g. stock prices (hence

328 views • 29 slides

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear

CSC321 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC321 Lecture 2: Linear Regression 1 / 30 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets, e.g. stock prices (hence

506 views • 34 slides

Concise Implementation of Linear Regression Concise Implementation of Linear Regression

Concise Implementation of Linear Regression Concise Implementation of Linear Regression Generating Data Sets Generating Data Sets In [1]: from mxnet import autograd, nd num_inputs = 2 num_examples = 1000 true_w = nd.array([2, -3.4]) true_b =

782 views • 9 slides