Lecture 5: Classification and dimension reduction Felix Held, - - PowerPoint PPT Presentation

Lecture 5: Classification and dimension reduction

Felix Held, Mathematical Sciences

MSA220/MVE440 Statistical Learning for Big Data 4th April 2019

Random Forests

• 1. Given a training sample with 𝑞 features, do for 𝑐 = 1, … , 𝐶

1.1 Draw a bootstrap sample of size 𝑜 from training data (with replacement) 1.2 Grow a tree 𝑈𝑐 until each node reaches minimal node size 𝑜min

1.2.1 Randomly select 𝑛 variables from the 𝑞 available 1.2.2 Find best splitting variable among these 𝑛 1.2.3 Split the node

• 2. For a new 𝐲 predict

Regression: ˆ 𝑔

𝑠𝑐(𝐲) = 1 𝐶 ∑ 𝐶 𝑐=1 𝑈𝑐(𝐲)

Classification: Majority vote at 𝐲 across trees Note: Step 1.2.1 leads to less correlation between trees built

• n bootstrapped data.

1/21

Comparison of RF, Bagging and CART

Toy example 𝑧 = 𝑦2

1 + 𝜁

where 𝜁 ∼ 𝑂(0, 1) 𝐲 ∼ 𝑂(𝟏, 𝚻), 𝐲 ∈ ℝ5, 𝚻𝑚𝑚 = 1, 𝚻𝑚𝑙 = 0.98, 𝑚 ≠ 𝑙

Training and test data were sampled from the true model. Results for RF, bagged CART and a single CART, using 𝑦1, … , 𝑦5 as predictor

• variables. (𝑜𝑢𝑠 = 50, 𝑜𝑢𝑓 = 100)
• 1.5

1.8 2.1 100 200 300

Number of trees Test error

2/21

Variable importance

• 1. Impurity index: Splitting on a feature leads to a reduction
• f node impurity. Summing all improvements over all

trees per feature gives a measure for variable importance

• 2. Out-of-bag error

▶ During bootstrapping for large enough 𝑜, each sample has

a chance of about 63% to be selected

▶ For bagging the remaining samples are out-of-bag. ▶ These out-of-bag samples for tree 𝑈𝑐 can be used as a test

set for that particular tree, since they were not used during training. Resulting in test error 𝐹0

▶ Permute variable 𝑘 in the out-of-bag samples and

calculate test error again 𝐹(𝑘)

1

▶ The increase in error

𝐹(𝑘)

1

− 𝐹0 ≥ 0 serves as an importance measure for variable 𝑘

3/21

RF applied to cardiovascular dataset

Monica dataset (http://thl.fi/monica, 𝑜 = 6367, 𝑞 = 11)

Predicting whether or not patients survive a 10 year period given a number of cardiovascular risk factors (class ratio 1.25 alive : 1 dead)

• 0.05

0.10 0.15 0.20 0.25 50 100 150 200

Number of Trees Out−of−bag error

Type

• Alive

Dead OOB

Error estimate

age angina diabetes hichol highbp hosp premi sex smstat stroke yronset 1 2 3

Decrease

Type

Alive Acc. Dead Acc. Mean Acc. Mean Gini

Variable importance 4/21

RF applied to heart disease dataset

South African coronary heart disease (SAheart) dataset

𝑜 = 462, 𝑞 = 9, predicting cholesterol levels in variable ldl

• 4

5 6 100 200 300

Number of Trees Out−of−bag error (MSE)

adiposity age alcohol chd famhist

• besity

sbp tobacco typea 0.0 0.5 1.0 1.5 2.0 2.5

Decrease

Type

Mean accuracy Mean MSE

Variable importance

4 8 12 16 100 125 150 175 200

sbp y

4 8 12 16 10 20 30

tobacco y

4 8 12 16 10 20 30 40

adiposity y

4 8 12 16 20 40 60 80

typea y

4 8 12 16 20 30 40

• besity

y

4 8 12 16 50 100 150

alcohol y

4 8 12 16 20 30 40 50 60

age y

4 8 12 16 Absent Present

famhist y

4 8 12 16 CHD No CHD

chd y

5/21

Principal Component Analysis

Projection onto a subspace

Assume 𝐲 ∈ ℝ𝑞. Given orthonormal vectors 𝐜1, … , 𝐜𝑛, i.e. ‖𝐜

𝑘‖ = 1

and 𝐜𝑈

𝑘 𝐜𝑙 = 0 for 𝑘 ≠ 𝑙

where 𝑛 < 𝑞, the projection of 𝐲 onto the 𝑛-dimensional linear subspace 𝑊

𝑛 = span(𝐜1, … , 𝐜𝑛) is

̂ 𝐲 =

𝑛

∑

𝑘=1

(𝐲𝑈𝐜

𝑘)𝐜 𝑘 = ( 𝑛

∑

𝑘=1

𝐜

𝑘𝐜𝑈 𝑘 )

⏟ ⎵ ⎵ ⏟ ⎵ ⎵ ⏟

Projection matrix

𝐲 The projection is orthogonal, i.e. (𝐲 − ̂ 𝐲)𝑈𝐜

𝑘 = 0

for all 𝐜

𝑘.

• ●
• 6/21

Rayleigh Quotient

Let 𝐁 ∈ ℝ𝑙×𝑙 be a symmetric matrix. For 𝟏 ≠ 𝐲 ∈ ℝ𝑙 define 𝐾(𝐲) = 𝐲𝑈𝐁𝐲 𝐲𝑈𝐲 𝐾(𝐲) is called the Rayleigh Quotient for 𝐁. Maximizing the Rayleigh Quotient The maximization problem max

𝐲

𝐾(𝐲) subject to 𝐲𝑈𝐲 = 1 is solved by a unit eigenvector 𝐲 of 𝐁 corresponding to the largest eigenvalue 𝜇 of 𝐁. Note: −𝐲 is also a solution.

7/21

Principal Component Analysis (PCA) (I)

Goal: Given continuous data, find an orthogonal coordinate system such that the variance of the data is maximal along each direction. Given data points 𝐲1, … , 𝐲𝑜 and a unit vector 𝐬, the variance of the data along 𝐬 is 𝑇(𝐬) =

𝑜

∑

𝑚=1

(𝐬𝑈(𝐲𝑚−𝐲))2 = (𝑜−1)𝐬𝑈ˆ 𝚻𝐬 where ˆ 𝚻 is the empirical covariance matrix.

• Axes

Cartesian Principal Component

8/21

Principal Component Analysis (PCA) (II)

Direction with maximal variance: Find 𝐬 such that max

𝐬

𝑇(𝐬) subject to ‖𝐬‖2 = 𝐬𝑈𝐬 = 1

▶ This is the same problem as maximizing the Rayleigh

Quotient for the matrix ˆ 𝚻.

▶ The solution is the eigenvector 𝐬1 of ˆ

𝚻 corresponding to the largest eigenvalue 𝜇1. How do we find the other directions? Project data on

• rthogonal complement of 𝐬1, i.e.

̂ 𝐲𝑚 = (𝐉𝑞 − 𝐬1𝐬𝑈

1 ) 𝐲𝑚

and repeat the procedure above.

9/21

Principal Component Analysis (PCA) (III)

Computational Procedure:

• 1. Centre and standardize the columns of the data matrix

𝐘 ∈ ℝ𝑜×𝑞

• 2. Calculate the empirical covariance matrix ˆ

𝚻 = 1 𝑜 − 1𝐘𝑈𝐘

• 3. Determine the eigenvalues 𝜇

𝑘 and corresponding

• rthonormal eigenvectors 𝐬

𝑘 of ˆ

𝚻 for 𝑘 = 1, … , 𝑞 and order them such that 𝜇1 ≥ 𝜇2 ≥ ⋯ ≥ 𝜇𝑞 ≥ 0

• 4. The vectors 𝐬

𝑘 give the direction of the principal

components (PC) 𝐬𝑈

𝑘 𝐲 and the eigenvalues 𝜇 𝑘 are the

variances along the PC directions Note: Set 𝐒 = (𝐬1, … , 𝐬𝑞) and 𝐄 = diag(𝜇1, … , 𝜇𝑞) then ˆ 𝚻 = 𝐒𝐄𝐒𝑈 and 𝐒𝑈𝐒 = 𝐒𝐒𝑈 = 𝐉𝑞

10/21

PCA and Dimension Reduction

Recall: For a matrix 𝐁 ∈ ℝ𝑙×𝑙 with eigenvalues 𝜇1, … , 𝜇𝑙 it holds that tr(𝐁) =

𝑙

∑

𝑘=1

𝜇

𝑘

For the empirical covariance matrix ˆ 𝚻 and the variance of the 𝑘-th feature Var[𝑦

𝑘]

tr(ˆ 𝚻) =

𝑞

∑

𝑘=1

Var[𝑦

𝑘] = 𝑞

∑

𝑘=1

𝜇

𝑘

is called the total variation. Using only the first 𝑛 < 𝑞 principal components leads to 𝜇1 + ⋯ + 𝜇𝑛 𝜇1 + ⋯ + 𝜇𝑞 ⋅ 100%

• f explained variance

11/21

PCA and Dimension Reduction: Example (I)

Variant of the MNIST handwritten digits dataset (𝑜 = 7291, 16 × 16 greyscale images, i.e. 𝑞 = 256)

Digit Frequency 0.16 1 0.14 2 0.10 3 0.09 4 0.09 5 0.08 6 0.09 7 0.09 8 0.07 9 0.09

7 3 6 6 5 4

12/21

PCA and Dimension Reduction: Example (II)

For standardized variables tr(ˆ 𝚻) = 𝑞 Typical selection rule: Components with 𝜇

𝑘 ≥ 1

𝑞 tr(ˆ 𝚻) (= 1)

• 0.1

1.0 10.0 100 200

Principal Component Eigenvalue

Scree plot

3 4 1 2

13/21

PCA and Dimension Reduction: Example (III)

Using the selection rule leads to 44

• components. Using the projection

̂ 𝐲 = (

44

∑

𝑘=1

𝐬

𝑘𝐬𝑈 𝑘 ) 𝐲

creates a reconstruction of 𝐲.

4 7 6 5

14/21

PCA and Dimension Reduction: Example (IV)

Projecting the digits onto the first two principal component directions gives a very clear distinction of digits 0 and 1.

• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• ●
• −10

−5 5 −10 10 20

r1

Tx = PC1

r2

Tx = PC2

Digit

• 1

Running QDA naively on all 256 variables to predict the digits does not work. Use the two most variable features across both classes. Table 1: Missclassifaction rate (20-fold CV)

1 Overall QDA + PCA 0.000 0.010 0.005 LDA + PCA 0.044 0.000 0.024 LDA + max var 0.007 0.024 0.015 QDA + max var 0.015 0.028 0.021 15/21

Singular Value Decomposition

Singular Value Decomposition (SVD)

The singular value decomposition (SVD) of a matrix 𝐘 ∈ ℝ𝑜×𝑞, 𝑜 ≥ 𝑞, is 𝐘 = 𝐕𝐄𝐖𝑈 where 𝐕 ∈ ℝ𝑜×𝑞 and 𝐖 ∈ ℝ𝑞×𝑞 with 𝐕𝑈𝐕 = 𝐉𝑞 and 𝐖𝑈𝐖 = 𝐖𝐖𝑈 = 𝐉𝑞 and 𝐄 ∈ ℝ𝑞×𝑞 is diagonal. Usually 𝑒11 ≥ 𝑒22 ≥ ⋯ ≥ 𝑒𝑞𝑞 Note: Due to the orthogonality conditions for 𝐕 and 𝐖 𝐘𝐘𝑈𝐕 = 𝐕𝐄2 𝐘𝑈𝐘𝐖 = 𝐖𝐄2

16/21

SVD and PCA

In PCA the empirical covariance matrix ˆ 𝚻 is in focus, whereas SVD focuses on the data matrix 𝐘 directly. Connection: For centred variables ˆ 𝚻 = 𝐘𝑈𝐘 𝑜 − 1 = 𝐖𝐄𝐕𝑈𝐕𝐄𝐖𝑈 𝑜 − 1 = 𝐖 ( 𝐄2 𝑜 − 1) 𝐖𝑈 The PC directions are in 𝐖 and the eigenvalues of ˆ 𝚻 are 𝑒2

𝑘𝑘/(𝑜 − 1).

Note: This is how PCA is typically calculated. SVD is a more general tool and is used in many other contexts as well.

17/21

SVD and best rank-𝑟-approximation / dimension reduction

Write 𝐯

𝑘 and 𝐰 𝑘 for the columns of 𝐕 and 𝐖, respectively. Then

𝐘 = 𝐕𝐄𝐖𝑈 =

𝑞

∑

𝑘=1

𝑒𝑘𝑘 𝐯

𝑘𝐰𝑈 𝑘

⏟

rank-1-matrix

Best rank-𝑟-approximation: For 𝑟 < 𝑞 𝐘𝑟 =

𝑟

∑

𝑘=1

𝑒𝑘𝑘𝐯

𝑘𝐰𝑈 𝑘

with approximation error ‖ ‖𝐘 − 𝐘𝑟‖ ‖

2 2 =

‖ ‖ ‖ ‖

𝑞

∑

𝑘=𝑟+1

𝑒𝑘𝑘𝐯

𝑘𝐰𝑈 𝑘

‖ ‖ ‖ ‖

2 2

=

𝑞

∑

𝑘=𝑟+1

𝑒2

𝑘 18/21

Connections to Discriminant Analysis

Discriminant Analysis and the Inverse Covariance Matrix

From PCA or SVD we get ˆ 𝚻 = 𝐖𝐄𝐖𝑈 where 𝐖𝑈𝐖 = 𝐖𝐖𝑈 = 𝐉𝑞 and 𝑒11 ≥ ⋯ ≥ 𝑒𝑞𝑞 ≥ 0. Then ˆ 𝚻−1 = 𝐖𝐄−1𝐖𝑈 = 𝐖𝐄−1/2𝐄−1/2𝐖𝑈 = (ˆ 𝚻−1/2)

𝑈 ˆ

𝚻−1/2 where (𝐄−1/2)𝑘𝑘 ∶= 1/√𝑒𝑘𝑘 and ˆ 𝚻−1/2 ∶= 𝐄−1/2𝐖𝑈. In DA the term involving the inverse covariance matrix is then (𝐲 − ˆ 𝝂)𝑈ˆ 𝚻−1(𝐲 − ˆ 𝝂) = (𝐲 − ˆ 𝝂)𝑈 (ˆ 𝚻−1/2)

𝑈 ˆ

𝚻−1/2(𝐲 − ˆ 𝝂) = (𝐖𝑈(𝐲 − ˆ 𝝂))

𝑈 𝐄−1 (𝐖𝑈(𝐲 − ˆ

𝝂)) = ∑

𝑘=1

1 𝑒𝑘𝑘 ( ̃ 𝑦

𝑘 − ̃

𝜈

𝑘)2

Inverse of the eigenvalues can lead to numerical instability!

19/21

Regularized Discriminant Analysis (RDA)

The empirical covariance matrix can be stabilized: ˆ 𝚻𝜇 ∶= ˆ 𝚻 + 𝜇𝐉𝑞 = 𝐖(𝐄 + 𝜇𝐉𝑞)𝐖𝑈 where 𝜇 > 0 is a tuning parameter.

▶ Using ˆ

𝚻𝜇 in LDA is called regularized discriminant analysis (RDA).

▶ Instead of 1/𝑒𝑘𝑘 the values 1/(𝑒𝑘𝑘 + 𝜇) are now involved. ▶ For small 𝑒𝑘𝑘 this can lead to numerical stability, whereas

large 𝑒𝑘𝑘 are not much affected.

▶ For large 𝜇 the 𝑒𝑘𝑘 will have diminishing impact and RDA

starts to become nearest centroids.

▶ RDA can be used with QDA as well by considering:

ˆ 𝚻𝑗,𝜇 ∶= ˆ 𝚻𝑗 ⏟

QDA

+𝜇 ˆ 𝚻 ⏟

LDA 20/21

Take-home message

▶ Random forests is very flexible and can determine

variable importance

▶ Principal component analysis gives a convenient

decomposition of the data with respect to variance

▶ Singular value decomposition is a universal workhorse for

dimension reduction

21/21