keepin it real semi supervised learning with realistic
play

Keepin It Real: Semi-Supervised Learning with Realistic Tuning - PowerPoint PPT Presentation

Feb 02, 2024 •286 likes •1.26k views

Keepin It Real: Semi-Supervised Learning with Realistic Tuning Andrew B. Goldberg Xiaojin Zhu goldberg@cs.wisc.edu jerryzhu@cs.wisc.edu Computer Sciences Department University of Wisconsin-Madison Gap between Semi-Supervised Learning

  1. Keepin’ It Real: Semi-Supervised Learning with Realistic Tuning Andrew B. Goldberg Xiaojin Zhu goldberg@cs.wisc.edu jerryzhu@cs.wisc.edu Computer Sciences Department University of Wisconsin-Madison

  2. Gap between Semi-Supervised Learning (SSL) research and practical applications Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning

  3. Gap between Semi-Supervised Learning (SSL) research and practical applications Semi-Supervised Learning: Using unlabeled data to build better classifiers Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning

  4. Gap between Semi-Supervised Learning (SSL) research and practical applications Real World • natural language Semi-Supervised Learning: processing Using unlabeled data to • computer vision • web search & IR build better classifiers • bioinformatics • etc Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning

  5. Gap between Semi-Supervised Learning (SSL) research and practical applications Assumptions • manifold? clusters? • low-density gap? • multiple views? Real World Parameters • natural language Semi-Supervised Learning: processing • regularization? Using unlabeled data to • computer vision • graph weights? • web search & IR build better classifiers • kernel parameters? • bioinformatics • etc Model Selection • Little labeled data • Many parameters • Computational costs Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning

  6. Gap between Semi-Supervised Learning (SSL) research and practical applications Assumptions • manifold? clusters? • low-density gap? • multiple views? Wrong choices could hurt performance! Real World Parameters • natural language Semi-Supervised Learning: processing • regularization? Using unlabeled data to • computer vision How can we ensure that SSL is never worse • graph weights? • web search & IR build better classifiers • kernel parameters? than supervised learning? • bioinformatics • etc Model Selection • Little labeled data • Many parameters • Computational costs Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning

  7. O UR F OCUS Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 3

  8. O UR F OCUS • Two critical issues • Parameter tuning • Choosing which (if any) SSL algorithm to use Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 3

  9. O UR F OCUS • Two critical issues • Parameter tuning • Choosing which (if any) SSL algorithm to use • Interested in realistic settings: • Practitioner is given some new labeled and unlabeled data • Must produce the best classifier possible Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 3

  10. O UR C ONTRIBUTIONS Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  11. O UR C ONTRIBUTIONS • Medium-scale empirical study Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  12. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  13. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods • Eight less-familiar NLP tasks, three evaluation metrics Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  14. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods • Eight less-familiar NLP tasks, three evaluation metrics • Experimental protocol explores several real-world settings Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  15. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods • Eight less-familiar NLP tasks, three evaluation metrics • Experimental protocol explores several real-world settings • All parameters are tuned realistically via cross validation Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  16. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods • Eight less-familiar NLP tasks, three evaluation metrics • Experimental protocol explores several real-world settings • All parameters are tuned realistically via cross validation • Findings under these conditions: Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  17. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods • Eight less-familiar NLP tasks, three evaluation metrics • Experimental protocol explores several real-world settings • All parameters are tuned realistically via cross validation • Findings under these conditions: • Each SSL can be worse than SL on some data sets Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  18. O UR C ONTRIBUTIONS • Medium-scale empirical study • Compares one supervised learning (SL) and two SSL methods • Eight less-familiar NLP tasks, three evaluation metrics • Experimental protocol explores several real-world settings • All parameters are tuned realistically via cross validation • Findings under these conditions: • Each SSL can be worse than SL on some data sets • Can achieve agnostic SSL by using cross validation accuracy to select among SL and SSL algorithms Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 4

  19. O UTLINE • Introduce “realistic tuning” for SSL • Empirical study protocol • Data sets • Algorithms • Meta algorithm for SSL model selection • Performance metrics • Results • Conclusions Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 5

  20. SSL W ITH R EALISTIC T UNING Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  21. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  22. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  23. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  24. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating • Use default values based on heuristics/experience? Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  25. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating • Use default values based on heuristics/experience? May fail on new data Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  26. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating • Use default values based on heuristics/experience? May fail on new data • k-fold cross validation? Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  27. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating • Use default values based on heuristics/experience? May fail on new data • k-fold cross validation? Little labeled data, but best available option Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  28. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating • Use default values based on heuristics/experience? May fail on new data • k-fold cross validation? Little labeled data, but best available option • Cross validation choices: Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

  29. SSL W ITH R EALISTIC T UNING • Given labeled and unlabeled data, { ( x 1 , y 1 ) , . . . , ( x l , y l ) , x l +1 , ..., x l + u } how should you set parameters for some algorithm? • Tune based on test set performance? No, this is cheating • Use default values based on heuristics/experience? May fail on new data • k-fold cross validation? Little labeled data, but best available option • Cross validation choices: • number of folds Andrew B. Goldberg (UW-Madison), SSL with Realistic Tuning 6

Recommend

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms Avital Oliver* Augustus Odena*

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms Avital Oliver* Augustus Odena*

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms Avital Oliver* Augustus Odena* Colin Raffel* Ekin Dogus Cubuk Ian Goodfellow vs. Error rates for our reimplementation of different semi-supervised learning methods. Reported

212 views • 17 slides
CS330 Paper Presentation: October 16th, 2019 Supervised Classification Semi-Supervised

CS330 Paper Presentation: October 16th, 2019 Supervised Classification Semi-Supervised

CS330 Paper Presentation: October 16th, 2019 Supervised Classification Semi-Supervised Classification: More realistic dataset Labelled Unlabelled Semi-Supervised Classification Most biologically plausible learning regime A familiar

650 views • 46 slides
Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning. Encyclopedia of Machine Learning. Jerry Zhu, 2010 Combining Labeled and Unlabeled Data with Co- Training. Avrim Blum, Tom Mitchell. COLT

953 views • 45 slides
Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Supervised, Unsupervised, Semi-supervised Most models handled here are supervised learning

580 views • 39 slides
Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2018/ Supervised, Unsupervised, Semi-supervised Most models handled here are supervised learning

670 views • 32 slides
Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu , Jinwen Ma, Zhanxing Zhu Peking University Beijing Institute of Big Data Research June, 2019 Semi-supervised learning (SSL) Suppose we

201 views • 18 slides
Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos

Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos

Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos University of the Aegean Outline Introduction The proposed method Common n-grams SVM Semi-supervised learning Evaluation

533 views • 17 slides
Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs.

Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs.

Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs. Maria-Florina Balcan 03/25/2015 Support Vector Machines (SVMs). One of the most theoretically well motivated and practically most e ff ective

792 views • 40 slides
Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier

Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier

Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier Advisor: Dr. Rakesh Verma COMPUTER SCIENCE Motivation Semi-supervised learning(SSL) algorithms are slow when working with large-scale data.

801 views • 14 slides
Particles Competition and Cooperation in Networks for Semi-Supervised Learning Fabricio Breve

Particles Competition and Cooperation in Networks for Semi-Supervised Learning Fabricio Breve

Particles Competition and Cooperation in Networks for Semi-Supervised Learning Fabricio Breve Department of Electrical & Computer Engineering University of Alberta Seminar - October 09, 2009 Contents Introduction Semi-Supervised

863 views • 40 slides
Regularizing objective functionals in semi-supervised learning Dejan Slep cev Carnegie Mellon

Regularizing objective functionals in semi-supervised learning Dejan Slep cev Carnegie Mellon

Regularizing objective functionals in semi-supervised learning Dejan Slep cev Carnegie Mellon University February 9, 2018. . 1 / 47 References S.,Thorpe, Analysis of p-Laplacian regularization in semi-supervised learning , arxiv

742 views • 48 slides
Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824

Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824

Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative Advanced Topics Semi-supervised learning Ensemble learning Generative models Sequence prediction models Deep reinforcement

345 views • 18 slides
Classification Semi-supervised learning based on network Speakers: Hanwen Wang, Xinxin Huang, and

Classification Semi-supervised learning based on network Speakers: Hanwen Wang, Xinxin Huang, and

Classification Semi-supervised learning based on network Speakers: Hanwen Wang, Xinxin Huang, and Zeyu Li CS 249-2 2017 Winter Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions Xiaojin Zhu, Zoubin Ghahramani, John Lafferty

625 views • 45 slides
Link prediction in graph construction for supervised and semi-supervised learning Lilian Berton,

Link prediction in graph construction for supervised and semi-supervised learning Lilian Berton,

Link prediction in graph construction for supervised and semi-supervised learning Lilian Berton, Jorge Valverde-Rebaza and Alneu de Andrade Lopes Laboratory of Computational Intelligence (LABIC) University of S ao Paulo (USP) Brazil July

398 views • 20 slides
10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important question! In many cases, unlabeled data is plentiful, labeled data expensive Image classification (x=images from the web, y=image type) Text

886 views • 37 slides
Margin-based Semi-supervised Learning Using Apollonius circle MONA EMADI AND JAFAR TANHA T TC S

Margin-based Semi-supervised Learning Using Apollonius circle MONA EMADI AND JAFAR TANHA T TC S

Margin-based Semi-supervised Learning Using Apollonius circle MONA EMADI AND JAFAR TANHA T TC S 2 0 2 0 Semi-supervised learning Training data Supervised All labeled data Model learning Some labeled data Semi-supervised Model learning

749 views • 23 slides
Semi-Supervised Learning Barnabas Poczos Slides Courtesy: Jerry Zhu, Aarti Singh Supervised

Semi-Supervised Learning Barnabas Poczos Slides Courtesy: Jerry Zhu, Aarti Singh Supervised

Semi-Supervised Learning Barnabas Poczos Slides Courtesy: Jerry Zhu, Aarti Singh Supervised Learning Feature Space Label Space Goal: Optimal predictor (Bayes Rule) depends on unknown P XY , so instead learn a good prediction rule from training

1.03k views • 36 slides
Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao Han, Xiao-Ming Wu Department of Computing The Hong Kong Polytechnic University Supervised Learning Tons of labeled data A good model 2 Image

880 views • 13 slides
Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M.

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M.

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M. Mitchell Carnegie Mellon University Document Classification: Bag of Words Approach aardvark 0 about 2 all 2 Africa 1 apple 0 anxious 0 ...

748 views • 53 slides
Transfer learning and domain adaptation Semi-supervised and transfer learning Myth : you cant

Transfer learning and domain adaptation Semi-supervised and transfer learning Myth : you cant

Day 2 Lecture 5 Transfer learning and domain adaptation Semi-supervised and transfer learning Myth : you cant do deep learning unless you have a million labelled examples for your problem. Reality You can learn useful representations

183 views • 15 slides
Semi-Supervised Kernel Mean Shift Clustering A Semi-Supervised Clustering Approach Motivation:

Semi-Supervised Kernel Mean Shift Clustering A Semi-Supervised Clustering Approach Motivation:

Semi-Supervised Kernel Mean Shift Clustering A Semi-Supervised Clustering Approach Motivation: Need for Supervision Data may not form clusters in Mapping to Feature Space Constraint Vector Input Points Clustering Projection the input

307 views • 27 slides
When Semi-Supervised Learning Meets Ensemble Learning Zhi-Hua Zhou http://cs.nju.edu.cn/zhouzh/

When Semi-Supervised Learning Meets Ensemble Learning Zhi-Hua Zhou http://cs.nju.edu.cn/zhouzh/

http:/ / lam da.nju.edu.cn When Semi-Supervised Learning Meets Ensemble Learning Zhi-Hua Zhou http://cs.nju.edu.cn/zhouzh/ Email: zhouzh@nju.edu.cn LAMDA Group National Key Laboratory for Novel Software Technology, Nanjing University, China

838 views • 64 slides
A Taxonomy of Semi-Supervised Learning Algorithms Olivier Chapelle Max Planck Institute for

A Taxonomy of Semi-Supervised Learning Algorithms Olivier Chapelle Max Planck Institute for

Introduction Generative models Low density separation Graph based methods Unsupervised learning Conclusions A Taxonomy of Semi-Supervised Learning Algorithms Olivier Chapelle Max Planck Institute for Biological Cybernetics December 2005

619 views • 48 slides
NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan Nguyen Joint work with Nhat Ho, Ankit B. Patel, Anima Anandkumar , Michael I. Jordan, and Richard G. Baraniuk Deep Learning The Current

621 views • 25 slides

More recommend