Lecture 9: Nonparametric Regression (1) Applied Statistics 2015 1 / - PowerPoint PPT Presentation

Introduction Estimations: local modelling Cross Validation Assignments Lecture 9: Nonparametric Regression (1) Applied Statistics 2015 1 / 22

Introduction Estimations: local modelling Cross Validation Assignments An example: Pick-It Lottery The New Jersey Pick-It Lottery is a daily numbers game run by the state of New Jersey. Buying a ticket entitles a player to pick a number between 0 and 999. Half of the money bet each day goes into the prize pool. (The state takes the other half.) The state picks a winning number at random, and the prize pool is shared equally among all winning tickets. We analyze the first 254 drawings after the lottery started in 1975. Figure 1 shows a scatterplot of the winning numbers and their payoffs. 2 / 22

Introduction Estimations: local modelling Cross Validation Assignments An example: Pick-It Lottery ● 800 ● ● ● ● 600 ● ● ● ● ● ● ● ● Payoff ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 400 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 200 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 200 400 600 800 1000 Number Although all numbers are equally likely to win, numbers chosen by fewer people have bigger payoffs if they win because the prize is shared among fewer tickets. Question: can we find some pattern from the data? Are there numbers with larger payoffs? 3 / 22

Introduction Estimations: local modelling Cross Validation Assignments An example: Pick-It Lottery The question can be answered by regression analysis. Linear regression: assumes the linear relation between payoff and win- ning number. The blue dashed line is the least squares regression line, which shows a general trend of higher payoffs for larger winning num- bers. ● 800 ● ● ● ● 600 ● ● ● ● ● ● ● ● ● ● ● ● ● ● Payoff ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 400 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 200 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 200 400 600 800 1000 Number 4 / 22

Introduction Estimations: local modelling Cross Validation Assignments Nonparametric regression Nonparametric regression do not assume any parametric structure. It is also known as “learning a function” in the field of machine learning. There are n pairs of observations ( x 1 , Y 1 ) , . . . , ( x n , Y n ) . The response variable Y is related to the covariate x by the equations Y i = r ( x i ) + ǫ i , i = 1 , . . . , n where r is the regression function , E( ǫ i ) = 0 and Var( ǫ i ) = σ 2 . Here, we want to estimate r under weak assumptions without assuming a parametric model of r . We are treating the covariate x i as fixed – fixed design. For random design, the data are ( X i , Y i ) , i = 1 , . . . , n and r ( x ) is the conditional expectation of Y given that X = x : r ( x ) = E( Y | X = x ) . 5 / 22

Introduction Estimations: local modelling Cross Validation Assignments A general idea behind different estimations Note that Y i is the sum of r ( x i ) and some error, the expected value of which is zero. This motivates to estimate r ( x ) by the average of those Y i where X i is “close” to x . Different ways of averaging and different measures of closeness lead to different estimators. 6 / 22

Introduction Estimations: local modelling Cross Validation Assignments An Example The data are n = 60 pairs of observations from a certain regression model. How to construc ˆ r n , an etimator of r ? ● 2 ● ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Y ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● −1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 ● 0.0 0.2 0.4 0.6 0.8 1.0 x 7 / 22

Introduction Estimations: local modelling Cross Validation Assignments Estimator: Regressogram A regressogram is construced in a similar manner as that for histogram. Here we consider that x i ∈ [0 , 1] . Devide the unit interval into m equally spaced bins denoted by B 1 , B 2 , . . . , B m . Define the regressogram, g n ( x ) = 1 � ˆ for x ∈ B j , Y i , k j i : x i ∈ B j where k j is the number of points in B j . Here we use the convention 0 0 = 0 . 8 / 22

Introduction Estimations: local modelling Cross Validation Assignments Estimator: Regressogram A regressogram is construced in a similar manner as that for histogram. Here we consider that x i ∈ [0 , 1] . Devide the unit interval into m equally spaced bins denoted by B 1 , B 2 , . . . , B m . Define the regressogram, g n ( x ) = 1 � ˆ for x ∈ B j , Y i , k j i : x i ∈ B j where k j is the number of points in B j . Here we use the convention 0 0 = 0 . Regressogram (m=10) ● 2 ● ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Y ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● −1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 ● 0.0 0.2 0.4 0.6 0.8 1.0 8 / 22 x

Introduction Estimations: local modelling Cross Validation Assignments Estimator: Local average Fix h > 0 , � n i =1 I ( x − h < x i ≤ x + h ) Y i r n ( x ) = ˆ i =1 I ( x − h < x i ≤ x + h ) . � n x − xi � n 1 2 1 [ − 1 , 1) ( h ) Y i This is also called naive kernel estimator: ˆ r n ( x ) = i =1 2 1 [ − 1 , 1) ( x − xi h ) � n 1 i =1 9 / 22

Introduction Estimations: local modelling Cross Validation Assignments Estimator: Local average Fix h > 0 , � n i =1 I ( x − h < x i ≤ x + h ) Y i ˆ r n ( x ) = i =1 I ( x − h < x i ≤ x + h ) . � n x − xi � n 1 2 1 [ − 1 , 1) ( h ) Y i This is also called naive kernel estimator: ˆ r n ( x ) = i =1 2 1 [ − 1 , 1) ( x − xi h ) � n 1 i =1 Local Average (h=0.2) ● 2 ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Y ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● −1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 ● 0.0 0.2 0.4 0.6 0.8 1.0 x 9 / 22

Introduction Estimations: local modelling Cross Validation Assignments Nadaraya-Watson Estimator Replacing the box kernel by a general kernel in the local average estimator, we obtain the Nadaraya-Watson estimator of r : � x − x i � n � i =1 K Y i h � . r n ( x ) = ˆ � x − x i � n i =1 K h 10 / 22

Introduction Estimations: local modelling Cross Validation Assignments Nadaraya-Watson Estimator Replacing the box kernel by a general kernel in the local average estimator, we obtain the Nadaraya-Watson estimator of r : � x − x i � n � i =1 K Y i h � . r n ( x ) = ˆ � x − x i � n i =1 K h Nadaraya−Watson (h=0.2, kernel=guassian) ● 2 ● ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Y ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● −1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 ● 0.0 0.2 0.4 0.6 0.8 1.0 x 10 / 22