Diagnostics & Kernel Methods Visualized Ondej Bojar April 3, - PowerPoint PPT Presentation

Sep 08, 2023 •310 likes •1.45k views

Diagnostics & Kernel Methods Visualized Ondej Bojar April 3, 2019 NPFL104 Machine Learning Methods Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise stated Diagnostics

RBF Kernels … Totally “fmips” the space: 69/110 𝑙( x , y ) = 𝑓𝑦𝑞(−𝛿‖ x − y ‖ 2 ); 𝛿 > 0 1 exp(-0.5*x*x) 0.9 exp(-1*x*x) exp(-2*x*x) 0.8 exp(-3*x*x) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 -10 -5 0 5 10 • Each training point creates its bell. • Overall shape is the sum of the bells. • Kind of “all nearest neighbours”. • The axes are now closeness to other objects.
RBF Kernel Parameters Peaked must be close to training examples High can be far from training examples Low Afgected Points gamma High 𝐷 Complex Low High Low High Simple Smooth Low Variance Bias Model Decision Surface 70/110 • Does higher gamma lead to higher variance? • Choice critical for SVM performance. • Advised to use GridSearchCV for 𝐷 and gamma: • exponentially spaced probes • wide range
SVM RBF (C=0.05, gamma=2) 71/110
SVM RBF (C=0.1, gamma=2) 72/110
SVM RBF (C=0.2, gamma=2) 73/110
SVM RBF (C=0.5, gamma=2) 74/110
SVM RBF (C=0.6, gamma=2) 75/110
SVM RBF (C=0.7, gamma=2) 76/110
SVM RBF (C=1, gamma=2) 77/110
SVM RBF (C=2, gamma=2) 78/110
SVM RBF (C=1, gamma=2) 79/110
SVM RBF (C=0.5, gamma=2) 80/110
SVM RBF (C=0.5, gamma=5) 81/110
SVM RBF (C=0.5, gamma=10) 82/110
SVM RBF (C=0.5, gamma=5) 83/110
SVM RBF (C=0.5, gamma=2) 84/110
SVM RBF (C=0.5, gamma=1) 85/110
SVM RBF (C=0.5, gamma=0.7) 86/110
SVM RBF (C=0.5, gamma=0.5) 87/110
SVM RBF (C=0.5, gamma=0.2) 88/110
SVM RBF (C=0.5, gamma=0.1) 89/110
SVM RBF (C=0.5, gamma=0.05) 90/110
Cross-validation Heatmap http: //scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html 91/110
Multi-class SVM Two implementations in scikit-learn: 92/110 • SVC: one-against-one • 𝑜(𝑜 − 1)/2 classifjers constructed • supports various kernels, incl. custom ones • LinearSVC: one-vs-the-rest • 𝑜 classifjers trained
PAMAP-easy Training Data 93/110 34 1 12 13 16 33.5 17 2 24 3 33 4 5 6 7 32.5 32 31.5 31 30.5 30 60 80 100 120 140 160 180 200
Default View (every 200) 94/110
Default View (every 300) 95/110
Default View (every 400) 96/110
Regularization C=0.5 97/110

Recommend

1 Kernel methods & optimization One example of a kernel that is frequently used in practice

Machine Learning Class Notes 9-25-12 Prof. David Sontag 1 Kernel methods & optimization One example of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions is the Gaussian kernel,

241 views • 5 slides

Kernel Methods Lei Tang Arizona State University Jul. 26th, 2007 Lei Tang Kernel Methods

Kernel Methods Lei Tang Arizona State University Jul. 26th, 2007 Lei Tang Kernel Methods Introduction Linear parametric models for regression and classification. Memory-based methods: Parzen probability density estimation, k-nearest

1.01k views • 63 slides

Optimization for Kernel Methods S. Sathiya Keerthi Yahoo! Research, Burbank, CA, USA Kernel

Optimization for Kernel Methods S. Sathiya Keerthi Yahoo! Research, Burbank, CA, USA Kernel methods: Support Vector Machines (SVMs) Kernel Logistic Regression (KLR) Aim: To introduce a variety of optimization problems that arise in the

738 views • 59 slides

Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20,

Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20, 2020 Table of contents 1. Motivation 2. Kernel methods 3. Basic kernel operations in feature space 4. Kernel-based algorithms 5. Summary 1/24

729 views • 31 slides

Skoltech Skolkovo Institute of Science and Technology Kernel Methods Refresher Kernel trick:

Quadrature-based Features for Kernel Approximation Marina Munkhoeva , Yermek Kapushev, Evgeny Burnaev, Ivan Oseledets Skoltech Skolkovo Institute of Science and Technology Kernel Methods Refresher Kernel trick: compute

1.01k views • 17 slides

Machine Learning Kernel Methods Hamid R. Rabiee Mohammad H. Rohban Spring 2015

Machine Learning Kernel Methods Hamid R. Rabiee Mohammad H. Rohban Spring 2015 http://ce.sharif.edu/courses/93-94/2/ce717-1 Agenda Agenda Motivations Kernel Definition Mercers Theorem Kernel Matrix Kernel Construction

353 views • 20 slides

Bayesian Kernel Methods for Non-Gaussian Distributions Cameron MacKenzie and Theodore Trafalis

Bayesian Kernel Methods for Non-Gaussian Distributions Cameron MacKenzie and Theodore Trafalis School of Industrial Engineering University of Oklahoma INFORMS Annual Meeting November 9, 2010 Current Bayesian Kernel methods Combine

610 views • 14 slides

Predictive low-rank decomposition for kernel methods Francis Bach Michael Jordan Ecole des

Predictive low-rank decomposition for kernel methods Francis Bach Michael Jordan Ecole des Mines de Paris UC Berkeley ICML 2005 Predictive low-rank decomposition for kernel methods Kernel algorithms and low-rank decompositions

258 views • 24 slides

Kernel Methods - I Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia

Introduction Dual Representations Kernel Design Radial Basis Functions Summary Kernel Methods - I Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu

1.05k views • 22 slides

Can Deep Learning Be Interpreted with Kernel Methods ? Ben Edelman & Preetum Nakkiran

Can Deep Learning Be Interpreted with Kernel Methods ? Ben Edelman & Preetum Nakkiran Opening the black box of neural networks Weve seen various post-hoc explanation methods (LIME, SHAP , etc.), but none that are faithful and robust.

1.03k views • 47 slides

Kernel Methods for Cooperative Contextual Bandits Introduction Motivation UCB Algorithms Basic

Kernel Methods for Cooperative Contextual Bandits Dubey and Pentland ICML 2020 Kernel Methods for Cooperative Contextual Bandits Introduction Motivation UCB Algorithms Basic Cooperation Summary of Contributions Abhimanyu Dubey and Alex

283 views • 17 slides

Kernel Smoothing Methods (Part 1) Henry Tan Georgetown University April 13, 2015 Georgetown

Kernel Smoothing Methods (Part 1) Henry Tan Georgetown University April 13, 2015 Georgetown University Kernel Smoothing 1 Introduction - Kernel Smoothing Previously Basis expansions and splines. Use all the data to minimise least squares

1.33k views • 43 slides

Explicit Feature Methods for Accelerated Kernel Learning Purushottam Kar Quick Motivation

Explicit Feature Methods for Accelerated Kernel Learning Purushottam Kar Quick Motivation Kernel Algorithms (SVM, SVR, KPCA) have output = , Number of support vectors is typically

448 views • 20 slides

Two-Tailed PCR and other methods for Precision Diagnostics Challenges analyzing miRNAs (and

Two-Tailed PCR and other methods for Precision Diagnostics Challenges analyzing miRNAs (and other short NA) microRNAs are short (most 21-22 nt) and cannot fit two conventional PCR primers There is no common sequence feature to use for

479 views • 35 slides

Kernel-based Methods and Support Vector Machines Larry Holder CptS 570 Machine Learning

Kernel-based Methods and Support Vector Machines Larry Holder CptS 570 Machine Learning School of Electrical Engineering and Computer Science Washington State University References Muller et al., An Introduction to Kernel-Based

371 views • 26 slides

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020 Joint work with Behrooz Ghorbani, Theodor Misiakiewicz, and Andrea Montanari Song Mei (Stanford University) Neural Networks and Kernel Methods June

645 views • 26 slides

Nonparametric Methods Recap Aarti Singh Machine Learning 10-701/15-781 Oct 4, 2010

Nonparametric Methods Recap Aarti Singh Machine Learning 10-701/15-781 Oct 4, 2010 Nonparametric Methods Kernel Density estimate (also Histogram) Weighted frequency Classification - K-NN Classifier Majority vote Kernel

749 views • 41 slides

Kernel-based Methods and Support Vector Machines Larry Holder CSE 6363 Machine Learning

Kernel-based Methods and Support Vector Machines Larry Holder CSE 6363 Machine Learning Computer Science and Engineering University of Texas at Arlington References Muller et al., An Introduction to Kernel-Based Learning

807 views • 26 slides

An Introduction to Kernel Methods for Classification, Regression and Structured Data atsch

An Introduction to Kernel Methods for Classification, Regression and Structured Data atsch Gunnar R Computational Biology Center Sloan-Kettering Institute, New York City previous versions together with S oren Sonnenburg & Cheng

1.57k views • 115 slides

Distribution-Free Uncertainty Quantification for Kernel Methods by Gradient Perturbations Bal

Distribution-Free Uncertainty Quantification for Kernel Methods by Gradient Perturbations Bal azs Csan ad Cs aji & Kriszti an Bal azs Kis SZTAKI: Institute for Computer Science and Control MTA: Hungarian Academy of Sciences

510 views • 24 slides

Meta-parameters of kernel methods and their optimization Petra Vidnerov Roman Neruda

Meta-parameters of kernel methods and their optimization Petra Vidnerov Roman Neruda Institute of Computer Science Academy of Sciences of the Czech Republic ITAT 2014 Motivation Learning given set of data samples find underlying trend,

171 views • 15 slides

Kernel Methods for Topological Data Analysis Kenji Fukumizu The Institute of Statistical

Kernel Methods for Topological Data Analysis Kenji Fukumizu The Institute of Statistical Mathematics (Tokyo, Japan) Joint work with Genki Kusano and Yasuaki Hiraoka (Tohoku Univ.), supported by JST CREST STM2016 at ISM. July 22, 2016 1

481 views • 37 slides

Diagnostics: A focus on use in development of drugs for MDR pathogens John H. Rex, MD

Diagnostics: A focus on use in development of drugs for MDR pathogens John H. Rex, MD AstraZeneca Pharmaceuticals Rex - Diagnostics, EMA workshop 25-26 Oct 2012 1 Key ideas in advance New diagnostics are hard work Speed &

260 views • 14 slides

Kernel Methods for Predictive Sequence Analysis Cheng Soon Ong 1 , 2 and Gunnar Rtsch 1 1

Kernel Methods for Predictive Sequence Analysis Cheng Soon Ong 1 , 2 and Gunnar Rtsch 1 1 Friedrich Miescher Laboratory, Tbingen 2 Max Planck Institute for Biological Cybernetics, Tbingen Tutorial at the German Conference on Bioinformatics

583 views • 30 slides