Feature Selection Gavin Brown www.cs.man.ac.uk/~gbrown The Usual - PowerPoint PPT Presentation

Aug 28, 2022 •538 likes •724 views

COMP61011 : Machine Learning Feature Selection Gavin Brown www.cs.man.ac.uk/~gbrown The Usual Supervised Learning Approach data + labels Learning Algorithm Model Predicted label Testing data Predicting Recurrence of Lung Cancer ofi Only a

COMP61011 : Machine Learning Feature Selection Gavin Brown www.cs.man.ac.uk/~gbrown
The Usual Supervised Learning Approach data + labels Learning Algorithm Model Predicted label Testing data
Predicting Recurrence of Lung Cancer ofi Only a few genes actually matter! Need small, interpretable subset to help doctors! fl
Text classi fi cation.... is this news story “ i nterestin g” ? “ B ag-of-Words ” representation: x = { 0 , 3 , 0 , 0 , 1 , ..., 2 , 3 , 0 , 0 , 0 , 1 } one entry per word! Easily 50,000 words! Very sparse - easy to over fi t! Need accuracy, otherwise we lose visitors to our news website!
The Usual Supervised Learning Approach ????? data + labels Learning Algorithm – OVERWHELMED! Model Predicted label Testing data
With big data….  Time complexity  Computational cost  Cost in data collection  Over-fitting  Lack of interpretability Feature selection
Some things matter, Some do not. Relevant features - those that we need to perform well Irrelevant features - those that are simply unnecessary Redundant features - those that become irrelevant in the presence of others
3 main categories of Feature Selection techniques: Wrappers, Filters, Embedded methods
Wrappers: Evaluation method Feature set Pros :  Model-oriented  Usually gets good performance for the model you choose. Trains a model Cons : Trains a model  Hugely computationally expensive. Outputs accuracy
Wrappers: Search strategy  With an exhaustive search 101110000001000100001000000000100101010 20 features … 1 million feature sets to check 25 features … 33.5 million sets 30 features … 1.1 billion sets  Need for a search strategy  Sequential forward selection  Recursive backward elimination  Genetic algorithms  Simulated annealing  …
Wrappers: Sequential Forward Selection
Search Complexity for Sequential Forward Selection
Feature Selection (2): Filters
Search Complexity for Filter Methods Pros :  A lot less expensive! Cons :  Not model-oriented
Feature Selection (3): Embedded methods Principle : the classifier performs feature selection as part of the learning procedure Example : the logistic LASSO (Tibshirani, 1996) With Error Function: Cross-entropy error Regularizing term Pros :  Performs feature selection as part of learning the procedure Cons :  Computationally demanding
Conclusions on Feature Selection Potential benefits Wrappers generally infeasible on the modern “big data” problem. Filters mostly heuristics, but can be formalized in some cases. - Manchester MLO group works on this challenge.
This is the End of the Course Unit…. That’s it. We’re done. Exam in January – past papers on website. MSc students: Projects due Friday, 4pm CDT/MRes students: 1 week later. You need to submit a hardcopy to SSO: - your 6 page (maximum) report You need to send by email to Gavin : - the report as PDF, and a ZIP file of your code.

Recommend

Causal and Non-Causal Feature Selection for Ridge Regression Gavin Cawley School of Computing

Causal and Non-Causal Feature Selection for Ridge Regression Gavin Cawley School of Computing Sciences University of East Anglia Norwich, United Kingdom gcc@cmp.uea.ac.uk Wednesday 3 rd June 2008 Introduction Causal feature selection

246 views • 13 slides

Feature Selection: ROC and Subset Selection Theodoridis 5.5-5.7 Using ROC for Feature Selection

Feature Selection: ROC and Subset Selection Theodoridis 5.5-5.7 Using ROC for Feature Selection Hypothesis Tests Examined (e.g. t-test): Useful for discarding features But does not tell us about overlap between classes for a feature! At Left

526 views • 21 slides

Outline Reducing Dimensionality Feature Selection 1 Steven J Zeil Feature Extraction 2

Feature Selection Feature Extraction Feature Selection Feature Extraction Outline Reducing Dimensionality Feature Selection 1 Steven J Zeil Feature Extraction 2 Principal Components Analysis (PCA) Old Dominion Univ. Factor Analysis (FA)

602 views • 6 slides

Reducing Dimensionality Steven J Zeil Old Dominion Univ. Fall 2010 1 Feature Selection

Feature Selection Feature Extraction Reducing Dimensionality Steven J Zeil Old Dominion Univ. Fall 2010 1 Feature Selection Feature Extraction Outline Feature Selection 1 Feature Extraction 2 Principal Components Analysis (PCA) Factor

479 views • 36 slides

Feature Selection ZHI LI Fenys Lab October 3, 2019 What is Feature? X (Independent)

Feature Selection ZHI LI Fenys Lab October 3, 2019 What is Feature? X (Independent) Variable Predictor Covariate Face; leg; tail; Hair texture/style Impeachment Trade war Whatever you can think of.. What methods are out there?

826 views • 37 slides

Mutual Information an Adequate Tool for Feature Selection ? Benot Frnay November 15, 2013

Mutual Information an Adequate Tool for Feature Selection ? Benot Frnay November 15, 2013 Introduction What is Feature Selection ? Overview of the Presentation Example of Feature Selection: Diabetes Progression Goal : predict the diabetes

1.05k views • 64 slides

Real-time updates to signed zones using dynamic update, OpenDNSSEC and BIND views Gavin

Real-time updates to signed zones using dynamic update, OpenDNSSEC and BIND views Gavin Brown <gavin.brown@centralnic.com> ICANN 50 London PRIVATE & CONFIDENTIAL A Brief History of CentralNics DNS System 1994: Altos Series 1000

505 views • 18 slides

Evolutionary Computation for Feature Selection and Feature Construction Bing Xue School of

Evolutionary Computation for Feature Selection and Feature Construction Bing Xue School of Engineering and Computer Science Victoria University of Wellington Bing.Xue@ecs.vuw.ac.nz IEEE CIS Webinar Mon, Sep 25, 2017 2:00 PM - 3:00 PM NZDT

1.7k views • 67 slides

Feature Selection CS 760@UW-Madison Goals for the lecture you should understand the following

Feature Selection CS 760@UW-Madison Goals for the lecture you should understand the following concepts filtering-based feature selection information gain filtering Markov blanket filtering frequency pruning

686 views • 25 slides

Prof. Gavin T. L. Brown Quant-DARE, EDSW 1 IEA Research for Education Series IEA Call

31/03/2020 Prof. Gavin T. L. Brown Quant-DARE, EDSW 1 IEA Research for Education Series IEA Call no. IEA 07/09-2017 Michaelides, M. P., Brown, G., & Eklf, H., & Papanastasiou, E. C., (2019). Profiles in TIMSS Mathematics:

510 views • 18 slides

Feature Selection Richard Pospesel and Bert Wierenga Introduction Preprocessing Peaking

Feature Selection Richard Pospesel and Bert Wierenga Introduction Preprocessing Peaking Phenomenon Feature Selection Based on Statistical Hypothesis T esting Dimensionality Reduction Using Neural Networks Outlier Removal

642 views • 17 slides

Feature Selection CE-725: Statistical Pattern Recognition Sharif University of Technology

Feature Selection CE-725: Statistical Pattern Recognition Sharif University of Technology Soleymani Fall 2018 Some slides are based on Isabelle Guyons slides : Introduction to feature selection, 2007. Outline } Dimensionality

1.08k views • 34 slides

RIDGE and LASSO regularization for regression Feature selection - Some algorithms perform

RIDGE and LASSO regularization for regression Feature selection - Some algorithms perform naturally feature selection - for example Decision Trees, Boosting - Other algorithms have difficulty with correlated features - for example Naive Bayes,

801 views • 14 slides

Week 3 Video 4 Automated Feature Generation Automated Feature Selection Automated Feature

Week 3 Video 4 Automated Feature Generation Automated Feature Selection Automated Feature Generation The creation of new data features in an automated fashion from existing data features Multiplicative Interactions You have variables A

513 views • 33 slides

10601 Machine Learning Model and feature selection Model selection issues We have seen some

10601 Machine Learning Model and feature selection Model selection issues We have seen some of this before Selecting features (or basis functions) Logistic regression SVMs Selecting parameter value Prior strength

611 views • 31 slides

... Other types of activation functions ( net = w i x i ) x n w n 1 1 x i : i- th

M otivation Knowledge discovery process FEATURE SELECTION USING Interpretation ANT COLONY OPTIM IZATION: M odeling Knowledge APPLICATIONS IN HEALTH CARE Feature selection Patterns Preprocessing Reduced data Joo M . C. Sousa 1 Data

159 views • 14 slides

Principal Component Analysis (PCA) CE-717: Machine Learning Sharif University of Technology

Principal Component Analysis (PCA) CE-717: Machine Learning Sharif University of Technology Spring 2016 Soleymani Dimensionality Reduction: Feature Selection vs. Feature Extraction Feature selection Select a subset of a given feature

5.23k views • 40 slides

11/10/2019 Gavin T L Brown The University of Auckland & Ume Universitet, Sweden

11/10/2019 Gavin T L Brown The University of Auckland & Ume Universitet, Sweden Presentation to COMPASS, University of Auckland. October 2019 The ability to flourish and succeed within the environment Not fixed, not unitary, not

841 views • 11 slides

Mach Machine Le ine Learning arning Feature Space, Feature Selection Hamid R. Rabiee Jafar

Mach Machine Le ine Learning arning Feature Space, Feature Selection Hamid R. Rabiee Jafar Muhammadi Spring 2015 http://ce.sharif.edu/courses/93-94/2/ce717-1 / Agenda Agenda Features and Patterns The Curse of Size and

731 views • 27 slides

PCA & ICA CE-717: Machine Learning Sharif University of Technology Spring 2018 Soleymani

PCA & ICA CE-717: Machine Learning Sharif University of Technology Spring 2018 Soleymani Dimensionality Reduction: Feature Selection vs. Feature Extraction } Feature selection } Select a subset of a given feature set } Feature extraction }

875 views • 51 slides

Deprecating jCard in RDAP: why and how Mario Loffredo - IIT-CNR/Registro.it Gavin Brown

9 th ROW - Registration Operations Workshop - June 16 th , 2020 Deprecating jCard in RDAP: why and how Mario Loffredo - IIT-CNR/Registro.it Gavin Brown CentralNIC Registry Why to deprecate jCard in RDAP ? jCard is the most frequent

490 views • 11 slides

GAMs: Model Selection David L Miller, Eric Pedersen, and Gavin L Simpson August 6th, 2016

GAMs: Model Selection David L Miller, Eric Pedersen, and Gavin L Simpson August 6th, 2016 Overview Model selection Shrinkage smooths Shrinkage via double penalty ( select = TRUE ) Confidence intervals for smooths p values anova() AIC Model

555 views • 36 slides

Data Warehousing and Machine Learning Feature Selection Thomas D. Nielsen Aalborg University

Data Warehousing and Machine Learning Feature Selection Thomas D. Nielsen Aalborg University Department of Computer Science Spring 2008 DWML Spring 2008 1 / 16 Feature Selection When features dont help Data generated by process

578 views • 31 slides

Boosting Convergence of Timing Closure using Feature Selection in a Learning-driven Approach Que

Boosting Convergence of Timing Closure using Feature Selection in a Learning-driven Approach Que Yanghua, Harnhua Ng, Nachiket Kapre yanghua.que@ntu.edu.sg, nachiket@ieee.org Claim 2 Claim Feature Selection helps boost AUC scores for

1.09k views • 72 slides