a motivation for polynomial regression
play

A motivation for polynomial regression We have obtained input-output - PowerPoint PPT Presentation

Nov 02, 2023 •282 likes •450 views

A motivation for polynomial regression We have obtained input-output pairs { ( x t , y t ) } t over the last 200 time steps and aim to model their relationship 3.0 3.0 3.0 2.5 2.5 2.5

  2. A motivation for polynomial regression We have obtained input-output pairs { ( x t , y t ) } t over the last 200 time steps and aim to model their relationship 3.0 3.0 3.0 ● ● ● ● ● ● 2.5 ● ● 2.5 2.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2.0 ● ● ● 2.0 2.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.5 ● 1.5 1.5 ● y ● ● y ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.0 ● 1.0 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● 0.5 0.5 ● ● ● ● ● ● ● ● ● 0.0 0.0 0.0 0 1 2 3 4 5 0 0 1 1 2 2 3 3 4 4 5 5 x x Using linear regression does not look like such a good idea... 2/14

  3. Linear regression A simple linear relation is assumed between x and y , i.e., y t = β 0 + β 1 x t + ε t , t = t n − n , . . . , t n where β 0 and β 1 are the model parameters (called intercept and slope ) ε t is a noise term, which you may see as our forecast error we want to minimize The linear regression model can be reformulated in a more compact form as y t = β ⊤ x t + ε t , t = t n − n , . . . , t n with � � � 1 � β 0 β = , x t = β 1 x t 3/14

  4. Least Squares (LS) estimation Now we need to find the best value of β that describes this cloud of point Under a number of assumptions, which we overlook here, the (best) model parameters ˆ β can be readily obtained with Least-Squares (LS) estimation The Least-Squares (LS) estimate ˆ β of the linear regression model parameters is given by � 2 t ε i 2 = arg min β � ˆ y t − β ⊤ x t = ( X ⊤ X ) − 1 X ⊤ y β = arg min β � � t with     1 x t n − n y t n − n � ˆ 1 x t n − n +1 y t n − n +1 � β 0     ˆ β = , X =  , y = . . .     ˆ . . . β 1     . . .    1 x t n y t n 4/14

  5. Extending to polynomial regression We could also assume more generally a polynomial relation between x and y , i.e., y t = β 0 + � P p =1 β p x p t + ε t , t = t n − n , . . . , t n where β p , p = 0 , . . . , P are the model parameters ε t is a noise term, which you may see as our forecast error we want to minimize This polynomial regression can be reformulated in a more compact form as y t = β ⊤ x t + ε t , i = t n − n , . . . , t n with     β 0 1 β 1 x t     β =  , x t =     · · · · · ·    x P β P t 5/14

  6. Least Squares (LS) estimation As the model is linear we can still use LS estimation! The Least-Squares (LS) estimate ˆ β of the linear regression model parameters is given by � 2 t ε t 2 = arg min β � ˆ y t − β ⊤ x t = ( X ⊤ X ) − 1 X ⊤ y � � β = arg min β t with x 2  x P    ˆ 1 x t n − n . . . y t n − n   β 0 t n − n t n − n x 2 x P 1 x t n − n +1 . . . y t n − n +1 ˆ β 1  t n − n +1 t n − n +1    ˆ   β =  , X =  , y = . . . . .       . . . . . · · ·     . . . . .     ˆ β P x 2 x P 1 x t n . . . y t n t n t n 6/14

  7. Going back to our example We apply polynominal regression with P = 2 (quadratic) and P = 3 (cubic) 3.0 3.0 3.0 3.0 ● ● ● ● ● ● ● ● ● ● 2.5 2.5 ● 2.5 2.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2.0 2.0 ● ● 2.0 2.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.5 1.5 ● 1.5 1.5 ● ● ● ● ● y ● y ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.0 1.0 ● 1.0 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 0.5 ● 0.5 0.5 ● ● ● ● ● ● ● ● ● 0.0 0.0 0.0 0.0 0 0 1 1 2 2 3 3 4 4 5 5 0 0 1 1 2 2 3 3 4 4 5 5 x x They both look quite nicer than the simple linear fit We are lucky here that the relationship truly is quadratic... if fitting higher-order polynominals, ˆ β i = 0 , p > 2 In general, higher-order may yield spurious results(!) 7/14

Recommend

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Announcements Section : Friday 1:30-2:45pm : @ MD 123 (only this Friday) A-section: Today:

921 views • 65 slides
with Polynomial Filters Josiah Manson and Scott Schaefer Texas A&M University Motivation

with Polynomial Filters Josiah Manson and Scott Schaefer Texas A&M University Motivation

Analytic Rasterization of Curves with Polynomial Filters Josiah Manson and Scott Schaefer Texas A&M University Motivation Motivation Motivation Motivation Motivation Motivation Motivation Constant colors Color gradients Motivation

976 views • 70 slides
Contents 1 Introduction 1 2 Linear Transformations 2 3 Polynomial Regression 3 4

Contents 1 Introduction 1 2 Linear Transformations 2 3 Polynomial Regression 3 4

Diagnostics and Transformations Part 2 Contents 1 Introduction 1 2 Linear Transformations 2 3 Polynomial Regression 3 4 Orthogonal Polynomials 6 5 Empirically Motivated Nonlinear Transformations 7 5.1 Mosteller-Tukey Bulging

382 views • 15 slides
Review:$$ Model$Selec5on$ Training$vs.$Test$errors$ Polynomial$regression$

Review:$$ Model$Selec5on$ Training$vs.$Test$errors$ Polynomial$regression$

Review:$$ Model$Selec5on$ Training$vs.$Test$errors$ Polynomial$regression$ Model$complexity:$Degree$of$polynomial$ Is$larger$always$be^er?$ Test$ Error$ Train$ Model$complexity$ Model$Selec5on$Criterion$

404 views • 14 slides
Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/ Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many slides attributable to: Prof. Mike Hughes Erik Sudderth

298 views • 28 slides
Polynomial Selection Thorsten Kleinjung Ecole Polytechnique F ed erale de Lausanne

Polynomial Selection Thorsten Kleinjung Ecole Polytechnique F ed erale de Lausanne

Polynomial Selection Thorsten Kleinjung Ecole Polytechnique F ed erale de Lausanne Contents Brief summary of polynomial selection (no root sieve) Motivation (lattice sieving, monic algebraic polynomial) General case (reduction to

411 views • 30 slides
h ( x ) } Although we can generally find the best set of weights efficiently, the exact form of

h ( x ) } Although we can generally find the best set of weights efficiently, the exact form of

Practical Use of Linear Regression 25 25 25 20 20 20 Sales Sales Sales 15 15 15 10 10 10 5 5 5 Class #03: Linear and Polynomial Regression Models 0 50 100 200 300 0 10 20 30 40 50 0 20 40 60 80 100 TV Radio

207 views • 3 slides
Beta Regression: Summary Shaken, Stirred, Mixed, and Partitioned Achim Zeileis, Francisco

Beta Regression: Summary Shaken, Stirred, Mixed, and Partitioned Achim Zeileis, Francisco

Overview Motivation Shaken or stirred: Single or double index beta regression for mean and/or precision in betareg Mixed: Latent class beta regression via flexmix Partitioned: Beta regression trees via party Beta Regression: Summary Shaken,

455 views • 6 slides
On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine

On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine

Index On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine Regression Characterization of vectors in SVM regression Procedure for Adding one vector Procedure for Removing one vector

520 views • 15 slides
Binary Decision Diagrams INF5140 1 Motivation Fig 1. Polynomial representation Boolean

Binary Decision Diagrams INF5140 1 Motivation Fig 1. Polynomial representation Boolean

Binary Decision Diagrams INF5140 1 Motivation Fig 1. Polynomial representation Boolean functions Extremely useful Some areas of application: Cryptography Propositional logic Verification/model checking Fig

322 views • 16 slides
Introduction to Machine Learning Polynomial Regression Models Learning goals Understand how to

Introduction to Machine Learning Polynomial Regression Models Learning goals Understand how to

Introduction to Machine Learning Polynomial Regression Models Learning goals Understand how to add flexibility to the linear model by using 2.0 f(x) for d = 1 (linear) f(x) for d = 5 1.5 polynomials f(x) for d = 25 1.0 0.5 y Understand

496 views • 8 slides
STAT 215 Polynomials, Multicollinearity Colin Reimer Dawson Oberlin College 4 November 2016

STAT 215 Polynomials, Multicollinearity Colin Reimer Dawson Oberlin College 4 November 2016

Outline Polynomial Regression Interactions Multicollinearity STAT 215 Polynomials, Multicollinearity Colin Reimer Dawson Oberlin College 4 November 2016 Outline Polynomial Regression Interactions Multicollinearity Outline Polynomial

517 views • 28 slides
CS 445 Introduction to Machine Learning Logistic Regression Instructor: Dr. Kevin Molloy Review

CS 445 Introduction to Machine Learning Logistic Regression Instructor: Dr. Kevin Molloy Review

CS 445 Introduction to Machine Learning Logistic Regression Instructor: Dr. Kevin Molloy Review Linear regression Finding the weights to assign to a polynomial so that the resulting line minimizes the "loss". ( ! ,

481 views • 11 slides
Diagnostics and Transformations Part 2 Bivariate Linear Regression James H. Steiger

Diagnostics and Transformations Part 2 Bivariate Linear Regression James H. Steiger

Introduction Linear Transformations Polynomial Regression Orthogonal Polynomials Empirically Motivated Nonlinear Transformations Diagnostics and Transformations Part 2 Bivariate Linear Regression James H. Steiger Department of Psychology

961 views • 46 slides
Polynomial Time corresponds to Solutions of Polynomial Ordinary Differential Equations of

Polynomial Time corresponds to Solutions of Polynomial Ordinary Differential Equations of

Polynomial Time corresponds to Solutions of Polynomial Ordinary Differential Equations of Polynomial Length Olivier Bournez, Daniel Graa and Amaury Pouly July 13, 2015 Main result and consequences Theorem (Informal) PTIME = PIVP of

633 views • 47 slides
Multiple Regression Analysis Independent Variables Mechanics and Interpretation of OLS

Multiple Regression Analysis Independent Variables Mechanics and Interpretation of OLS

Motivation for Multiple Regression The Model with k Multiple Regression Analysis Independent Variables Mechanics and Interpretation of OLS Interpreting the OLS Caio Vigo Regression Line The Expected Value of the OLS The University of

2.75k views • 89 slides
Logistic Regression James H. Steiger Department of Psychology and Human Development Vanderbilt

Logistic Regression James H. Steiger Department of Psychology and Human Development Vanderbilt

Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Logistic Regression James H. Steiger

777 views • 54 slides
Simple Linear Regression Chapter 10 1 Motivation Have data (sample, x s) Want to

Simple Linear Regression Chapter 10 1 Motivation Have data (sample, x s) Want to

4/29/2019 IMGD 2905 Simple Linear Regression Chapter 10 1 Motivation Have data (sample, x s) Want to know likely value B of next observation E.g., playtime versus skins owned A reasonable to compute mean (with Y

553 views • 32 slides
On Continuity of the Roots of a Parametric Zero Dimensional Multivariate Polynomial Ideal Yosuke

On Continuity of the Roots of a Parametric Zero Dimensional Multivariate Polynomial Ideal Yosuke

On Continuity of the Roots of a Parametric Zero Dimensional Multivariate Polynomial Ideal Yosuke Sato Ryoya Fukasaku Hiroshi Sekigawa Tokyo University of Science (Tokyo, Japan) 1 / 25 Motivation The roots of an unary polynomial are

1.11k views • 68 slides
Polynomial Hierarchy A polynomial-bounded version of Kleenes Arithmetic Hierarchy becomes

Polynomial Hierarchy A polynomial-bounded version of Kleenes Arithmetic Hierarchy becomes

Polynomial Hierarchy A polynomial-bounded version of Kleenes Arithmetic Hierarchy becomes trivial if P = NP . Karp, 1972 Computational Complexity, by Fu Yuxi Polynomial Hierarchy 1 / 44 Larry Stockmeyer and Albert Meyer introduced

584 views • 45 slides
Property of the interior polynomial from the HOMFLY polynomial

Property of the interior polynomial from the HOMFLY polynomial

Computing the HOMFLY polynomial using the combinatorics Properties of the interior polynomial Property of the interior polynomial from the HOMFLY polynomial 2017 12

554 views • 24 slides
Introduction to Machine Learning Classification: Logistic Regression

Introduction to Machine Learning Classification: Logistic Regression

Introduction to Machine Learning Classification: Logistic Regression compstat-lmu.github.io/lecture_i2ml MOTIVATION A discriminant approach for directly modeling the posterior probabilities ( x | ) of the labels is logistic regression . For

484 views • 9 slides
The NL-coflow polynomial (joint work with W. Hochstttler) MCW 2019 The NL-coflow polynomial

The NL-coflow polynomial (joint work with W. Hochstttler) MCW 2019 The NL-coflow polynomial

Discrete Mathematics and Optimization Johanna Wiehe: The NL-coflow polynomial (joint work with W. Hochstttler) MCW 2019 The NL-coflow polynomial Johanna Wiehe Discrete Mathematics and Optimization motivation undirected graph G = ( V , E )

782 views • 26 slides
Polygonal Splines and Their Applications Motivation Polynomial for Numerical Solution of PDE 1

Polygonal Splines and Their Applications Motivation Polynomial for Numerical Solution of PDE 1

Ming-Jun Lai Polygonal Splines and Their Applications Motivation Polynomial for Numerical Solution of PDE 1 Splines Spline Method for PDE GBC BB functions Ming-Jun Lai Polygonal Spline Space Department of Mathematics Hexahedral Spline

440 views • 40 slides

More recommend