recap: Data Condensation and Nearest Neighbor Search Training Set - PowerPoint PPT Presentation

recap: Data Condensation and Nearest Neighbor Search Training Set Consistent S 2 Learning From Data Branch and bound for finding Lecture 18 nearest neighbors. x Radial Basis Functions S 1 Non-Parametric RBF Parametric RBF − − k -RBF-Network − → M. Magdon-Ismail CSCI 4100/6100 Lloyd’s algorithm for finding a good clustering. � A M Radial Basis Functions : 2 /31 c L Creator: Malik Magdon-Ismail RBF vs. k -NN − → Weighting the Data Points: α n Radial Basis Functions (RBF) k -Nearest Neighbor: Only considers k -nearest neighbors. Test point x . Most popular kernel: Gaussian each neighbor has equal weight φ ( z ) = e − 1 2 z 2 . α n : the weight of x n in g ( x ). What about using all data to compute g ( x )? � | | x − x n | | � α n ( x ) = φ r Window kernel, mimics k -NN, � 1 z ≤ 1 , φ ( z ) = RBF: Use all data. decreasing function of | | x − x n | | 0 z > 1 , data further away from x have less weight. � A c M Radial Basis Functions : 3 /31 � A c M Radial Basis Functions : 4 /31 L Creator: Malik Magdon-Ismail Weighting data points − → L Creator: Malik Magdon-Ismail Weighting depends on distance − →

Weighting the Data Points: α n Weighting the Data Points: α n Test point x . Test point x . Most popular kernel: Gaussian Most popular kernel: Gaussian φ ( z ) = e − 1 2 z 2 . φ ( z ) = e − 1 2 z 2 . α n : the weight of x n in g ( x ). α n : the weight of x n in g ( x ). � | | x − x n | | � � | | x − x n | | � α n ( x ) = φ α n ( x ) = φ r r Window kernel, mimics k -NN, Window kernel, mimics k -NN, � � 1 z ≤ 1 , 1 z ≤ 1 , φ ( z ) = φ ( z ) = weighting depends on the distance | | x − x n | | . . . relative to a scale parameter r 0 z > 1 , 0 z > 1 , � A M Radial Basis Functions : 5 /31 � A M Radial Basis Functions : 6 /31 c L Creator: Malik Magdon-Ismail Relative to scale r − → c L Creator: Malik Magdon-Ismail Determined by φ − → Weighting the Data Points: α n Weighting the Data Points: α n Test point x . Test point x . Most popular kernel: Gaussian Most popular kernel: Gaussian φ ( z ) = e − 1 2 z 2 . φ ( z ) = e − 1 2 z 2 . α n : the weight of x n in g ( x ). α n : the weight of x n in g ( x ). � | | x − x n | | � � | | x − x n | | � α n ( x ) = φ α n ( x ) = φ r r Window kernel, mimics k -NN, Window kernel, mimics k -NN, � 1 z ≤ 1 , � 1 z ≤ 1 , φ ( z ) = φ ( z ) = kernel φ determines how the weighting decreases with distance kernel φ determines how the weighting decreases with distance 0 z > 1 , 0 z > 1 , � A c M Radial Basis Functions : 7 /31 � A c M Radial Basis Functions : 8 /31 L Creator: Malik Magdon-Ismail Example Kernels φ − → L Creator: Malik Magdon-Ismail Nonparametric RBF final hypothesis − →

Nonparametric RBF – Regression Nonparametric RBF – Classification � | | x − x n | | � | | x − x n | | � � y y α n ( x ) = φ α n ( x ) = φ r r α n α n x ( x n , y n ) x ( x n , y n ) � N N � � � � � α n ( x ) α n ( x ) � � g ( x ) = · y n g ( x ) = sign · y n � N � N m =1 α m ( x ) m =1 α m ( x ) n =1 n =1 ր ր Weighted average of target values Weighted average of target values � A M Radial Basis Functions : 9 /31 � A M Radial Basis Functions : 10 /31 c L Creator: Malik Magdon-Ismail Nonparametric RBF – classsification − → c L Creator: Malik Magdon-Ismail Nonparametric RBF – logistic regression − → Nonparametric RBF – Logistic Regression Choice of Scale r Nearest Neighbor Choosing k : k = 1 k = 3 k = 11 k = 3 √ k = N � | | x − x n | | � y α n ( x ) = φ r CV α n x ( x n , y n ) Nonparametric RBF N � � r = 0 . 01 r = 0 . 05 r = 0 . 5 Choosing r : α n ( x ) � g ( x ) = · � y n = +1 � 1 � N r ∼ √ m =1 α m ( x ) 2 d N n =1 ր CV Weighted average of target values overfitting underfitting � A c M Radial Basis Functions : 11 /31 � A c M Radial Basis Functions : 12 /31 L Creator: Malik Magdon-Ismail Choosing the scale r − → L Creator: Malik Magdon-Ismail Highlights of Nonparametric RBF − →

Highlights of Nonparametric RBF Scaled Bumps on Each Data Point 6. Computationally demanding . } 1. Simple (‘smooth’ version of k -NN rule). N � � α n ( x ) 2. No training. � y g ( x ) = · y n � N m =1 α m ( x ) A good! method n =1 3. Near optimal E out . α n Weighted average of y n x ( x n , y n ) 4. Easy to justify classification to customer. 5. Can do classification, multi-class, regression, logistic regression. N � � y n � | | x − x n | | � � g ( x ) = · φ � N r m =1 α m ( x ) n =1 N � | | x − x n | | � � = w n ( x ) φ r n =1 Sum of bumps at x n scaled by w n ( x ) � A M Radial Basis Functions : 13 /31 � A M Radial Basis Functions : 14 /31 c L Creator: Malik Magdon-Ismail Bumps on Data Points − → c L Creator: Malik Magdon-Ismail Rewrite as weighted bumps − → Scaled Bumps on Each Data Point Scaled Bumps on Each Data Point N � � N � � α n ( x ) α n ( x ) � � g ( x ) = · y n y g ( x ) = · y n y � N � N m =1 α m ( x ) m =1 α m ( x ) n =1 n =1 α n α n Weighted average of y n Weighted average of y n x ( x n , y n ) x ( x n , y n ) N N � � � � y n � | | x − x n | | � y n � | | x − x n | | � � � g ( x ) = · φ g ( x ) = · φ � N � N r r m =1 α m ( x ) m =1 α m ( x ) n =1 n =1 y w n N N � | | x − x n | | � � | | x − x n | | � � � = w n ( x ) φ = w n ( x ) · φ r r x ( x n , y n ) n =1 n =1 Sum of bumps at x n scaled by w n ( x ) Sum of bumps at x n scaled by w n ( x ) � A c M Radial Basis Functions : 15 /31 � A c M Radial Basis Functions : 16 /31 L Creator: Malik Magdon-Ismail Weighted bumps, w n ( x ) − → L Creator: Malik Magdon-Ismail Nonparametric RBF: 3 point example − →

Nonparametric RBF: w n ( x ) Parametric RBF, w n – A Linear Model Nonparametric RBF Nonparametric RBF N N � | | x − x n | | � � | | x − x n | | � � � g ( x ) = w n ( x ) · φ g ( x ) = w n ( x ) · φ r r n =1 n =1 y y y y r = 0 . 1 r = 0 . 3 r = 0 . 1 r = 0 . 3 Only need to specify r . Only need to specify r . x x x x Parametric RBF Parametric RBF N N � | | x − x n | | � � | | x − x n | | � � h ( x ) = w n · φ � h ( x ) = w n · φ r r n =1 n =1 Fix r ; need to determine the parameters w n . Fix r ; need to determine the parameters w n . — fit the data. — fit the data. — overfit the data? — overfit the data? � A M Radial Basis Functions : 17 /31 � A M Radial Basis Functions : 18 /31 c L Creator: Malik Magdon-Ismail Parametric RBF − → c L Creator: Malik Magdon-Ismail Parametric RBF 3 point example − → Parametric RBF – A Linear Model RBF-Nonlinear Transform Depends on Data Nonparametric RBF N � | | x − x n | | � � h ( x ) = w n · φ = w t z N � | � | x − x n | | r � g ( x ) = w n ( x ) · φ n =1 r n =1 y y r = 0 . 1 r = 0 . 3 — Φ ( x 1 ) t — Only need to specify r .       φ 1 ( x ) — z t 1 — — Φ ( x 2 ) t — x x φ 2 ( x ) — z t 2 —       z = Φ ( x ) =  , φ n ( x )= φ ( ) . Z =  = | | x − x n | | . . . . . .       . . . r     — Φ ( x N ) t — φ N ( x ) — z t N — Parametric RBF r = 0 . 1 r = 0 . 3 N � | | x − x n | | � � h ( x ) = w n · φ r n =1 y y Fit the data ( h ( x n ) = y n ): w = Z † y = (Z t Z) − 1 Z t y Fix r ; need to determine the parameters w n . — fit the data. x x — overfit the data? � A c M Radial Basis Functions : 19 /31 � A c M Radial Basis Functions : 20 /31 L Creator: Malik Magdon-Ismail RBF-Nonlinear Transform − → L Creator: Malik Magdon-Ismail Solving for w − →