Regression Instructor: Prof. Shuai Huang Industrial and Systems - PowerPoint PPT Presentation

Lecture 4: Logistic Regression Instructor: Prof. Shuai Huang Industrial and Systems Engineering University of Washington

Extend linear model for classification 𝑞 • Need a mathematic transfer function to connect 𝛾 0 + σ 𝑗=1 𝛾 𝑗 𝑦 𝑗 with a binary outcome 𝑧 • How? • Logistic regression chooses to use 𝑞 𝒚 𝑞 1−𝑞 𝒚 = 𝛾 0 + σ 𝑗=1 log 𝛾 𝑗 𝑦 𝑗 . • Why?

Justification for the logistic regression model • It works in many applications • It leads to analytical tractability (in some senses) and encourages in- depth theoretical investigation • It has a strong tie with linear regression model. Therefore, methodologically there is much we can translate from linear regression to logistic regression. Conceptually, it inherits the aura of linear regression model and users can assume a similar degree of confidence of linear regression model onto the logistic regression model

Parameter estimation • The likelihood function is: 1−𝑧 𝑜 . 𝑞 𝒚 𝑜 𝑧 𝑜 1 − 𝑞 𝒚 𝑜 𝑂 𝑀 𝜸 = ς 𝑜=1 • We use the log-likelihood to turn products into sums: 𝑂 𝑚 𝜸 = σ 𝑜=1 𝑧 𝑜 log 𝑞 𝒚 𝑜 + 1 − 𝑧 𝑜 log 1 − 𝑞 𝒚 𝑜 . This could be further transformed into 𝑞 𝑞 𝛾 𝑗 𝑦 𝑜𝑗 − σ 𝑜=1 − log 1 + 𝑓 𝛾 0 +σ 𝑗=1 𝑂 𝑂 𝑚 𝜸 = σ 𝑜=1 𝑧 𝑜 𝛾 0 + σ 𝑗=1 𝛾 𝑗 𝑦 𝑜𝑗 , Then we can have 𝑂 σ 𝑜=1 𝑧 𝑜 log 𝑞 𝒚 𝑜 + 1 − 𝑧 𝑜 log 1 − 𝑞 𝒚 𝑜 , 𝑞 𝒚 𝑜 𝑂 𝑂 = σ 𝑜=1 − σ 𝑜=1 log 1 − 𝑞 𝒚 𝑜 𝑧 𝑜 log 1−𝑞 𝒚 𝑜 , 𝑞 𝑞 𝛾 𝑗 𝑦 𝑜𝑗 − σ 𝑜=1 𝑂 − log 1 + 𝑓 𝛾 0 +σ 𝑗=1 𝑂 = σ 𝑜=1 𝑧 𝑜 𝛾 0 + σ 𝑗=1 𝛾 𝑗 𝑦 𝑜𝑗 .

Application of the Newton-Raphson algorithm • The Newton-Raphson algorithm is an iterative algorithm that seeks updates of the current solution using the following formula: −1 𝜖𝑚 𝜸 𝜖 2 𝑚 𝜸 𝜸 𝑜𝑓𝑥 = 𝜸 𝑝𝑚𝑒 − 𝜖𝜸 . 𝜖𝜸𝜖𝜸 𝑈 • We can show that 𝜖𝑚 𝜸 𝑂 𝜖𝜸 = σ 𝑜=1 𝒚 𝑜 𝑧 𝑜 − 𝑞 𝒚 𝑜 , 𝜖 2 𝑚 𝜸 𝑂 𝑈 𝑞 𝒚 𝑜 𝜖𝜸𝜖𝜸 𝑈 = − σ 𝑜=1 𝒚 𝑜 𝒚 𝑜 1 − 𝑞 𝒚 𝑜 . • A certain structure can then be revealed if we rewrite it in matrix form: 𝜖𝑚 𝜸 𝜖𝜸 = 𝐘 𝑈 𝒛 − 𝒒 , 𝜖 2 𝑚 𝜸 𝜖𝜸𝜖𝜸 𝑈 = −𝐘 𝑈 𝐗𝐘 . where 𝐘 is the 𝑂 × 𝑞 + 1 input matrix, 𝒛 is the 𝑂 × 1 column vector of 𝑧 𝑗 , 𝒒 is the 𝑂 × 1 column vector of 𝑞 𝒚 𝑜 , and 𝐗 is a 𝑂 × 𝑂 diagonal matrix of weights with the n th diagonal element as 𝑞 𝒚 𝑜 1 − 𝑞 𝒚 𝑜 .

The updating rule Plugging this into the updating formula of the Newton-Raphson algorithm, −1 𝜖𝑚 𝜸 𝜖 2 𝑚 𝜸 𝜸 𝑜𝑓𝑥 = 𝜸 𝑝𝑚𝑒 − 𝜖𝜸 , 𝜖𝜸𝜖𝜸 𝑈 we can derive that −1 𝐘 𝑈 𝐗 𝒛 − 𝒒 , 𝜸 𝑜𝑓𝑥 = 𝜸 𝑝𝑚𝑒 + 𝐘 𝑈 𝐗𝐘 −1 𝐘 𝑈 𝐗 𝐘𝜸 𝑝𝑚𝑒 + 𝐗 −1 𝒛 − 𝒒 = 𝐘 𝑈 𝐗𝐘 , −1 𝐘 𝑈 𝐗𝒜 , = 𝐘 𝑈 𝐗𝐘 where 𝐴 = 𝐘𝜸 𝑝𝑚𝑒 + 𝐗 −1 𝒛 − 𝒒 .

Another look at the updating rule • This resembles the generalized least squares (GLS) estimator of a regression model, where each data point 𝒚 𝑜 , 𝑧 𝑜 is associated with a weight 𝑥 𝑜 to reduce the influence of potential outliers in fitting the regression model. 𝜸 𝑜𝑓𝑥 ⟵ arg min 𝒜 − 𝐘𝜸 𝑈 𝐗 𝒜 − 𝐘𝜸 . 𝜸 • For this reason, this algorithm is also called the Iteratively Reweighted Least Square or IRLS algorithm. 𝒜 is referred as the adjusted response. • Why the weighting makes sense? Or, what are the implications of this?

A summary of the IRLS algorithm Putting all these together, a complete flow of the IRLS is shown in below: • Initialize 𝜸 . 1 • Compute 𝒒 by its definition: 𝑞 𝒚 𝑜 = 𝛾𝑗𝑦𝑜𝑗 for 𝑜 = 1,2, … , 𝑂 . 𝑞 1+𝑓 − 𝛾0+σ 𝑗=1 • Compute the diagonal matrix 𝐗 , while the n th diagonal element as 𝑞 𝒚 𝑜 1 − 𝑞 𝒚 𝑜 for 𝑜 = 1,2, … , 𝑂 . • Set 𝒜 as = 𝐘𝜸 + 𝐗 −1 𝒛 − 𝒒 . −1 𝐘 𝑈 𝐗𝒜 . • Set 𝜸 as 𝐘 𝑈 𝐗𝐘 • If the stopping criteria is met, stop; otherwise go back to step 2.

R lab • Download the markdown code from course website • Conduct the experiments • Interpret the results • Repeat the analysis on other datasets