strong baselines for neural semi supervised learning
play

Strong Baselines for Neural Semi-supervised Learning under Domain - PowerPoint PPT Presentation

Jun 20, 2023 •237 likes •1.37k views

Strong Baselines for Neural Semi-supervised Learning under Domain Shift Sebastian Ruder Barbara Plank Learning under Domain Shift 2 Learning under Domain Shift State-of-the-art domain adaptation approaches 2 Learning under Domain Shift

  1. Multi-task 
 Tri-training Multi-task tri-training 1. Train one model with 3 objective functions. 2. Use predictions on unlabeled data for third if two agree. y = 1 y = 1 x 1 11

  2. Multi-task 
 Tri-training Multi-task tri-training 1. Train one model with 3 objective functions. 2. Use predictions on unlabeled data for third if two agree. 3. Restrict final layers to 
 y = 1 y = 1 use different 
 x representations. 1 11

  3. Multi-task 
 Tri-training Multi-task tri-training 1. Train one model with 3 objective functions. 2. Use predictions on unlabeled data for third if two agree. 3. Restrict final layers to 
 y = 1 y = 1 use different 
 x representations. 4. Train third objective 
 1 function only on 
 pseudo labeled to 
 bridge domain shift. 11

  4. Multi-task 
 Tri-training 12

  5. Multi-task 
 Tri-training BiLSTM (Plank et al., 2016) 12

  6. Multi-task 
 Tri-training BiLSTM (Plank et al., 2016) char w 2 BiLSTM 12

  7. Multi-task 
 Tri-training BiLSTM BiLSTM (Plank et al., 2016) char char w 1 w 2 BiLSTM BiLSTM 12

  8. Multi-task 
 Tri-training BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  9. Multi-task 
 Tri-training m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  10. Multi-task 
 Tri-training m 2 m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  11. Multi-task 
 Tri-training m 2 m 3 m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  12. Multi-task 
 Tri-training m 2 m 3 m 1 m 2 m 3 m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  13. Multi-task 
 Tri-training m 2 m 3 m 2 m 3 m 1 m 2 m 3 m 1 m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  14. Multi-task 
 Tri-training L orth = ∥ W ⊤ m 1 W m 2 ∥ 2 orthogonality constraint (Bousmalis et al., 2016) F m 2 m 3 m 2 m 3 m 1 m 2 m 3 m 1 m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM 12

  15. ⃗ Multi-task 
 Tri-training L orth = ∥ W ⊤ m 1 W m 2 ∥ 2 orthogonality constraint (Bousmalis et al., 2016) F m 2 m 3 m 2 m 3 m 1 m 2 m 3 m 1 m 1 BiLSTM BiLSTM BiLSTM (Plank et al., 2016) char char char w 1 w 2 w 3 BiLSTM BiLSTM BiLSTM L ( θ ) = − ∑ i ∑ log P m i ( y | h ) + γ L orth Loss: 1,.., n 12

  16. Data & Tasks 13

  17. Data & Tasks Two tasks: Domains: 13

  18. Data & Tasks Two tasks: Domains: Sentiment analysis on Amazon reviews dataset (Blitzer et al, 2006) 13

  19. Data & Tasks Two tasks: Domains: Sentiment analysis on Amazon reviews dataset (Blitzer et al, 2006) POS tagging on SANCL 2012 dataset (Petrov and McDonald, 2012) 13

  20. Sentiment Analysis Results 82 80.25 Accuracy 78.5 76.75 75 Avg over 4 target domains VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri * result from Saito et al., (2017) 14

  21. Sentiment Analysis Results 82 80.25 Accuracy 78.5 76.75 75 Avg over 4 target domains VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri * result from Saito et al., (2017) 14

  22. Sentiment Analysis Results 82 80.25 Accuracy 78.5 76.75 75 Avg over 4 target domains VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri * result from Saito et al., (2017) 14

  23. Sentiment Analysis Results 82 80.25 Accuracy 78.5 76.75 75 Avg over 4 target domains VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri * result from Saito et al., (2017) 14

  24. Sentiment Analysis Results 82 80.25 Accuracy 78.5 76.75 75 Avg over 4 target domains VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri * result from Saito et al., (2017) ‣ Multi-task tri-training slightly outperforms tri-training, but has higher variance. 14

  25. POS Tagging Results Trained on 10% labeled data (WSJ) 89.8 89.525 Accuracy 89.25 88.975 88.7 Avg over 5 target domains Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri 15

  26. POS Tagging Results Trained on 10% labeled data (WSJ) 89.8 89.525 Accuracy 89.25 88.975 88.7 Avg over 5 target domains Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri 15

  27. POS Tagging Results Trained on 10% labeled data (WSJ) 89.8 89.525 Accuracy 89.25 88.975 88.7 Avg over 5 target domains Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri 15

  28. POS Tagging Results Trained on 10% labeled data (WSJ) 89.8 89.525 Accuracy 89.25 88.975 88.7 Avg over 5 target domains Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri 15

  29. POS Tagging Results Trained on 10% labeled data (WSJ) 89.8 89.525 Accuracy 89.25 88.975 88.7 Avg over 5 target domains Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri ‣ Tri-training with disagreement works best with little data. 15

  30. POS Tagging Results Trained on full labeled data (WSJ) 92 91.25 Accuracy 90.5 89.75 89 Avg over 5 target domains TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri * result from Schnabel & Schütze (2014) 16

  31. POS Tagging Results Trained on full labeled data (WSJ) 92 91.25 Accuracy 90.5 89.75 89 Avg over 5 target domains TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri * result from Schnabel & Schütze (2014) 16

  32. POS Tagging Results Trained on full labeled data (WSJ) 92 91.25 Accuracy 90.5 89.75 89 Avg over 5 target domains TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri * result from Schnabel & Schütze (2014) 16

  33. POS Tagging Results Trained on full labeled data (WSJ) 92 91.25 Accuracy 90.5 89.75 89 Avg over 5 target domains TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri * result from Schnabel & Schütze (2014) ‣ Tri-training works best in the full data setting. 16

  34. POS Tagging Analysis Accuracy on out-of-vocabulary (OOV) tokens 11 80 Accuracy on OOV tokens 8.25 72.5 % OOV tokens 5.5 65 2.75 57.5 0 50 Answers Emails Newsgroups Reviews Weblogs OOV tokens Src Tri MT-Tri 17

  35. POS Tagging Analysis Accuracy on out-of-vocabulary (OOV) tokens 11 80 Accuracy on OOV tokens 8.25 72.5 % OOV tokens 5.5 65 2.75 57.5 0 50 Answers Emails Newsgroups Reviews Weblogs OOV tokens Src Tri MT-Tri 17

  36. POS Tagging Analysis Accuracy on out-of-vocabulary (OOV) tokens 11 80 Accuracy on OOV tokens 8.25 72.5 % OOV tokens 5.5 65 2.75 57.5 0 50 Answers Emails Newsgroups Reviews Weblogs OOV tokens Src Tri MT-Tri 17

  37. POS Tagging Analysis Accuracy on out-of-vocabulary (OOV) tokens 11 80 Accuracy on OOV tokens 8.25 72.5 % OOV tokens 5.5 65 2.75 57.5 0 50 Answers Emails Newsgroups Reviews Weblogs OOV tokens Src Tri MT-Tri ‣ Classic tri-training works best on OOV tokens. 17

  38. POS Tagging Analysis Accuracy on out-of-vocabulary (OOV) tokens 11 80 Accuracy on OOV tokens 8.25 72.5 % OOV tokens 5.5 65 2.75 57.5 0 50 Answers Emails Newsgroups Reviews Weblogs OOV tokens Src Tri MT-Tri ‣ Classic tri-training works best on OOV tokens. ‣ MT-Tri does worse than source-only baseline on OOV. 17

  39. POS Tagging Analysis POS accuracy per binned log frequency 0.018 Accuracy delta vs. src-only baseline 0.014 0.009 0.005 0 -0.005 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Binned frequency MT-Tri Tri 18

Recommend

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Supervised, Unsupervised, Semi-supervised Most models handled here are supervised learning

580 views • 39 slides
Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2018/ Supervised, Unsupervised, Semi-supervised Most models handled here are supervised learning

670 views • 32 slides
Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning. Encyclopedia of Machine Learning. Jerry Zhu, 2010 Combining Labeled and Unlabeled Data with Co- Training. Avrim Blum, Tom Mitchell. COLT

953 views • 45 slides
NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan

NEURAL RENDERING MODEL (NRM): JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING Tan Nguyen Joint work with Nhat Ho, Ankit B. Patel, Anima Anandkumar , Michael I. Jordan, and Richard G. Baraniuk Deep Learning The Current

621 views • 25 slides
Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based

650 views • 26 slides
Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu

Tangent-Normal Adversarial Regularization for Semi-Supervised Learning Bing Yu , Jingfeng Wu , Jinwen Ma, Zhanxing Zhu Peking University Beijing Institute of Big Data Research June, 2019 Semi-supervised learning (SSL) Suppose we

201 views • 18 slides
Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos

Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos

Author Identification Using Semi-supervised Learning Ioannis Kourtis and Efstathios Stamatatos University of the Aegean Outline Introduction The proposed method Common n-grams SVM Semi-supervised learning Evaluation

533 views • 17 slides
Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs.

Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs.

Support Vector Machines (SVMs). Semi-Supervised Learning. Semi-Supervised SVMs. Maria-Florina Balcan 03/25/2015 Support Vector Machines (SVMs). One of the most theoretically well motivated and practically most e ff ective

792 views • 40 slides
Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier

Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier

Parallelizing Semi- ReDAS Lab Supervised Learning Algorithms with MapReduce Nick Gauthier Advisor: Dr. Rakesh Verma COMPUTER SCIENCE Motivation Semi-supervised learning(SSL) algorithms are slow when working with large-scale data.

801 views • 14 slides
Particles Competition and Cooperation in Networks for Semi-Supervised Learning Fabricio Breve

Particles Competition and Cooperation in Networks for Semi-Supervised Learning Fabricio Breve

Particles Competition and Cooperation in Networks for Semi-Supervised Learning Fabricio Breve Department of Electrical & Computer Engineering University of Alberta Seminar - October 09, 2009 Contents Introduction Semi-Supervised

863 views • 40 slides
Regularizing objective functionals in semi-supervised learning Dejan Slep cev Carnegie Mellon

Regularizing objective functionals in semi-supervised learning Dejan Slep cev Carnegie Mellon

Regularizing objective functionals in semi-supervised learning Dejan Slep cev Carnegie Mellon University February 9, 2018. . 1 / 47 References S.,Thorpe, Analysis of p-Laplacian regularization in semi-supervised learning , arxiv

742 views • 48 slides
Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824

Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824

Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative Advanced Topics Semi-supervised learning Ensemble learning Generative models Sequence prediction models Deep reinforcement

345 views • 18 slides
Classification Semi-supervised learning based on network Speakers: Hanwen Wang, Xinxin Huang, and

Classification Semi-supervised learning based on network Speakers: Hanwen Wang, Xinxin Huang, and

Classification Semi-supervised learning based on network Speakers: Hanwen Wang, Xinxin Huang, and Zeyu Li CS 249-2 2017 Winter Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions Xiaojin Zhu, Zoubin Ghahramani, John Lafferty

625 views • 45 slides
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms Avital Oliver* Augustus Odena*

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms Avital Oliver* Augustus Odena*

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms Avital Oliver* Augustus Odena* Colin Raffel* Ekin Dogus Cubuk Ian Goodfellow vs. Error rates for our reimplementation of different semi-supervised learning methods. Reported

212 views • 17 slides
Link prediction in graph construction for supervised and semi-supervised learning Lilian Berton,

Link prediction in graph construction for supervised and semi-supervised learning Lilian Berton,

Link prediction in graph construction for supervised and semi-supervised learning Lilian Berton, Jorge Valverde-Rebaza and Alneu de Andrade Lopes Laboratory of Computational Intelligence (LABIC) University of S ao Paulo (USP) Brazil July

398 views • 20 slides
CS330 Paper Presentation: October 16th, 2019 Supervised Classification Semi-Supervised

CS330 Paper Presentation: October 16th, 2019 Supervised Classification Semi-Supervised

CS330 Paper Presentation: October 16th, 2019 Supervised Classification Semi-Supervised Classification: More realistic dataset Labelled Unlabelled Semi-Supervised Classification Most biologically plausible learning regime A familiar

650 views • 46 slides
10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important question! In many cases, unlabeled data is plentiful, labeled data expensive Image classification (x=images from the web, y=image type) Text

886 views • 37 slides
Margin-based Semi-supervised Learning Using Apollonius circle MONA EMADI AND JAFAR TANHA T TC S

Margin-based Semi-supervised Learning Using Apollonius circle MONA EMADI AND JAFAR TANHA T TC S

Margin-based Semi-supervised Learning Using Apollonius circle MONA EMADI AND JAFAR TANHA T TC S 2 0 2 0 Semi-supervised learning Training data Supervised All labeled data Model learning Some labeled data Semi-supervised Model learning

749 views • 23 slides
Semi-Supervised Learning Barnabas Poczos Slides Courtesy: Jerry Zhu, Aarti Singh Supervised

Semi-Supervised Learning Barnabas Poczos Slides Courtesy: Jerry Zhu, Aarti Singh Supervised

Semi-Supervised Learning Barnabas Poczos Slides Courtesy: Jerry Zhu, Aarti Singh Supervised Learning Feature Space Label Space Goal: Optimal predictor (Bayes Rule) depends on unknown P XY , so instead learn a good prediction rule from training

1.03k views • 36 slides
Keepin It Real: Semi-Supervised Learning with Realistic Tuning Andrew B. Goldberg Xiaojin

Keepin It Real: Semi-Supervised Learning with Realistic Tuning Andrew B. Goldberg Xiaojin

Keepin It Real: Semi-Supervised Learning with Realistic Tuning Andrew B. Goldberg Xiaojin Zhu goldberg@cs.wisc.edu jerryzhu@cs.wisc.edu Computer Sciences Department University of Wisconsin-Madison Gap between Semi-Supervised Learning

1.26k views • 96 slides
Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao Han, Xiao-Ming Wu Department of Computing The Hong Kong Polytechnic University Supervised Learning Tons of labeled data A good model 2 Image

880 views • 13 slides
Out line Neural net wor ks Percept r on Neural Net works Supervised learning

Out line Neural net wor ks Percept r on Neural Net works Supervised learning

Out line Neural net wor ks Percept r on Neural Net works Supervised learning algorit hms f or neur al net works J uly 7, 2005 Reading: R&N Ch 20.5 CS 486/ 686 Univer sit y of Wat erloo 2 CS486/686 Lecture Slides (c)

85 views • 5 slides
Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M.

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M.

Semi-Supervised Learning and Text Analysis Machine Learning 10-701 November 29, 2005 Tom M. Mitchell Carnegie Mellon University Document Classification: Bag of Words Approach aardvark 0 about 2 all 2 Africa 1 apple 0 anxious 0 ...

747 views • 53 slides
Transfer learning and domain adaptation Semi-supervised and transfer learning Myth : you cant

Transfer learning and domain adaptation Semi-supervised and transfer learning Myth : you cant

Day 2 Lecture 5 Transfer learning and domain adaptation Semi-supervised and transfer learning Myth : you cant do deep learning unless you have a million labelled examples for your problem. Reality You can learn useful representations

183 views • 15 slides

More recommend