tdnn a two stage deep neural network for prompt
play

TDNN: A Two-stage Deep Neural Network for Prompt-independent - PowerPoint PPT Presentation

Oct 04, 2023 •182 likes •568 views

TDNN: A Two-stage Deep Neural Network for Prompt-independent Automated Essay Scoring Outline Background Method Experiments Conclusions What is Automated Essay Scoring (AES) ? Computer produces summative assessment for

  1. TDNN: A Two-stage Deep Neural Network for Prompt-independent Automated Essay Scoring

  2. Outline • Background • Method • Experiments • Conclusions

  3. What is Automated Essay Scoring (AES) ? • Computer produces summative assessment for evaluation • Aim: reduce human workload • AES has been put into practical use by ETS from 1999

  4. Prompt-specific and -Independent AES • Most existing AES approaches are prompt-specific – Require human labels for each prompt to train – Can achieve satisfying human-machine agreement • Quadradic weighted kappa (QWK) > 0.75 [Taghipour & Ng, EMNLP 2016] • Inter-human agreement: QWK=0.754 • Prompt-independent AES remains a challenge – Only non-target human labels are available

  5. Challenges in Prompt-independent AES Source Prompts Target Prompt Prompt 1: Winter Olympics Learn Predict Prompt 2: Rugby World Cup 2018 Model World Cup Prompt 3: Australian Open

  6. Challenges in Prompt-independent AES Source Prompts Target Prompt Prompt 1: Winter Olympics Unavailability of rated essays written for the target Learn Predict Prompt 2: Rugby World Cup 2018 Model World Cup prompt Prompt 3: Australian Open

  7. Challenges in Prompt-independent AES Source Prompts Target Prompt Prompt 1: Winter Olympics Learn Predict Prompt 2: Rugby World Cup 2018 Model World Cup Prompt 3: Australian Open • Previous approaches learn on source prompts – Domain adaption [Phandi et al. EMNLP 2015] – Cross-domain learning [Dong & Zhang, EMNLP 2016] – Achieved Avg. QWK = 0.6395 at best with up to 100 labeled target essays

  8. Challenges in Prompt-independent AES Source Prompts Target Prompt Prompt 1: Winter Olympics Learn Predict Prompt 2: Rugby World Cup 2018 Model World Cup Prompt 3: Australian Open Off-topic: essays written for source prompts are mostly irrelevant

  9. Outline • Background • Method • Experiments • Conclusions

  10. TDNN: A Two-stage Deep Neural Network for Prompt- independent AES • Based on the idea of transductive transfer learning • Learn on target essays • Utilize the content of target essays to rate

  11. The Two-stage Architecture • Prompt-independent stage: train a shallow model to create pseudo labels on the target prompt

  12. The Two-stage Architecture • Prompt-dependent stage: learn an end-to-end model to predict essay ratings for the target prompts

  13. Prompt-independent stage • Train a robust prompt-independent AES model • Using Non-target prompts • Learning algorithm: RankSVM for AES • Pre-defined prompt-independent features • Select confident essays written for the target prompt

  14. Prompt-independent stage • Train a robust prompt-independent AES model • Using Non-target prompts • Learning algorithm: RankSVM • Pre-defined prompt-independent features • Select confident essays written for the target prompt Predicted Scores 0 10

  15. Prompt-independent stage • Train a robust prompt-independent AES model • Using Non-target prompts • Learning algorithm: RankSVM • Pre-defined prompt-independent features • Select confident essays written for the target prompt Predicted Scores 0 4 10 Predicted ratings in [0, 4] as negative examples

  16. Prompt-independent stage • Train a robust prompt-independent AES model • Using Non-target prompts • Learning algorithm: RankSVM • Pre-defined prompt-independent features • Select confident essays written for the target prompt Predicted Scores 8 0 4 10 Predicted ratings in [8, 10] as positive examples

  17. Prompt-independent stage • Train a robust prompt-independent AES model • Using Non-target prompts • Learning algorithm: RankSVM • Pre-defined prompt-independent features • Select confident essays written for the target prompt Predicted Scores 8 0 4 10 1 0 Converted to 0/1 labels

  18. Prompt-independent stage • Train a robust prompt-independent AES model • Using Non-target prompts • Learning algorithm: RankSVM • Pre-defined prompt-independent features • Select confident essays written for the target prompt • Common sense: ≥8 is good, <5 is bad • Enlarge sample size 0 4 8 10

  19. Prompt-dependent stage • Train a hybrid deep model for a prompt- dependent assessment • An end-to-end neural network with three parts of inputs: • Word semantic embeddings • Part-of-speech (POS) taggings • Syntactic taggings

  20. Architecture of the hybrid deep model Multi-layer structure: Words – (phrases) - Sentences – Essay

  21. Architecture of the hybrid deep model Glove word embeddings

  22. Architecture of the hybrid deep model Part-of-speech taggings

  23. Architecture of the hybrid deep model Syntactic taggings

  24. Architecture of the hybrid deep model Multi-layer structure: Words – (phrases) - Sentences – Essay

  25. Architecture of the hybrid deep model

  26. Model Training • Training loss: MSE on 0/1 pseudo labels • Validation metric: Kappa on 30% non-target essays – Select the model that can best rate

  27. Outline • Background • Method • Experiments • Conclusions

  28. Dataset & Metrics • We use the standard ASAP corpus – 8 prompts with >10K essays in total • Prompt-independent AES: 7 prompts are used for training, 1 for testing • Report on common human-machine agreement metrics – Pearson’s correlation coefficient (PCC) – Spearman’s correlation coefficient (SCC) – Quadratic weighted Kappa (QWK)

  29. Baselines • RankSVM based on prompt-independent handcrafted features • Also used in the prompt-independent stage in TDNN • 2L-LSTM [Alikaniotis et al. , ACL 2016] • Two LSTM layer + linear layer • CNN-LSTM [Taghipour & Ng, EMNLP 2016] • CNN + LSTM + linear layer • CNN-LSTM-ATT [Dong et al. , CoNLL 2017] • CNN-LSTM + attention

  30. RankSVM is the most robust baseline • High variance of DNN models’ performance on all 8 prompts • Possibly caused by learning on non-target prompts • RankSVM appears to be the most stable baseline • Justifies the use of RankSVM in the first stage of TDNN

  31. Comparison to the best baseline • TDNN outperforms the best baseline on 7 out of 8 prompts • Performance improvements gained by learning on the target prompt

  32. Average performance on 8 prompts Method QWK PCC SCC Baselines RankSVM .5462 .6072 .5976 2L-LSTM .4687 .6548 .6214 CNN-LSTM .5362 .6569 .6139 CNN-LSTM-ATT .5057 .6535 .6368 TDNN TDNN(Sem) .5875 .6779 .6795 TDNN(Sem+POS) .6582 .7103 .7130 TDNN(Sem+Synt) .6856 .7244 .7365 TDNN(POS+Synt) .6784 .7189 .7322 TDNN(ALL) .6682 .7176 .7258

  33. Average performance on 8 prompts Method QWK PCC SCC Baselines RankSVM .5462 .6072 .5976 2L-LSTM .4687 .6548 .6214 CNN-LSTM .5362 .6569 .6139 CNN-LSTM-ATT .5057 .6535 .6368 TDNN TDNN(Sem) .5875 .6779 .6795 TDNN(Sem+POS) .6582 .7103 .7130 TDNN(Sem+Synt) .6856 .7244 .7365 TDNN(POS+Synt) .6784 .7189 .7322 TDNN(ALL) .6682 .7176 .7258

  34. Average performance on 8 prompts Method QWK PCC SCC Baselines RankSVM .5462 .6072 .5976 2L-LSTM .4687 .6548 .6214 CNN-LSTM .5362 .6569 .6139 CNN-LSTM-ATT .5057 .6535 .6368 TDNN TDNN(Sem) .5875 .6779 .6795 TDNN(Sem+POS) .6582 .7103 .7130 TDNN(Sem+Synt) .6856 .7244 .7365 TDNN(POS+Synt) .6784 .7189 .7322 TDNN(ALL) .6682 .7176 .7258

  35. Sanity Check: Relative Precision How the quality of pseudo examples affects the performance of TDNN? ➢ The sanctity of the selected essays, namely, the number of positive (negative) essays that are better (worse) than all negative (positive) essays. ➢ Such relative precision is at least 80% and mostly beyond 90% on different prompts ➢ TDNN can at least learn from correct 0/1 labels

  36. Conclusions • It is beneficial to learn an AES model on the target prompt • Syntactic features are useful addition to the widely used Word2Vec embeddings • Sanity check: small overlap between pos/neg examples • Prompt-independent AES remains an open problem – ETS wants Kappa>0.70 – TDNN can achieve 0.68 at best

  37. Thank you!

Recommend

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides
Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference Engine Song Han CVA group, Stanford University Apr 7, 2016 1 Intro about me and my advisor Fourth year PhD with Prof. Bill Dally at Stanford.

2.06k views • 70 slides
Deep Learning (jkim@bi.snu.ac.kr) 2015/05/7

Deep Learning (jkim@bi.snu.ac.kr) 2015/05/7

Deep Learning (jkim@bi.snu.ac.kr) 2015/05/7 1 History of Neural Network Research Neural network Deep belief net Back propagation Science

1.27k views • 108 slides
Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference Engine Song Han CVA group, Stanford University Jan 6, 2015 A few words about us Fourth year PhD with Prof. Bill Dally at Stanford.

966 views • 54 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural networks have taken over AI Tasks that are made possible by NNs, aka deep learning Tasks that were once assumed to be purely in the human domain

1.92k views • 150 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Networks: What can a network represent Deep Learning, Spring 2018 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Spring 2018 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Spring 2018 1 Recap : Neural networks have taken over AI Tasks that are made possible by NNs, aka deep learning 2 Recap : NNets and the brain In their basic form, NNets

2.14k views • 145 slides
IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK

IMPLEMENTATION OF A PARALLEL BATCH TRAINING ALGORITHM FOR DEEP NEURAL NETWORK YUPING LIN IFLYTEK LABORATORY FOR NEURAL COMPUTING FOR MACHINE LEARNING DEPARTMENT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE YORK UNIVERSITY, TORONTO NOVEMBER

211 views • 20 slides
EN. 601.467/667 Introduc3on to Human Language Technology Deep Learning I Shinji Watanabe 1

EN. 601.467/667 Introduc3on to Human Language Technology Deep Learning I Shinji Watanabe 1

EN. 601.467/667 Introduc3on to Human Language Technology Deep Learning I Shinji Watanabe 1 Todays agenda Introduction of deep neural network Basics of neural network 2 Short bio Research interests Automatic speech

1.71k views • 62 slides
How to Evaluate Efficient Deep Neural Network Approaches Vivienne Sze ( @eems_mit)

How to Evaluate Efficient Deep Neural Network Approaches Vivienne Sze ( @eems_mit)

How to Evaluate Efficient Deep Neural Network Approaches Vivienne Sze ( @eems_mit) Massachusetts Institute of Technology In collaboration with Yu-Hsin Chen, Joel Emer, Yannan Wu, Tien-Ju Yang, Google Mobile Vision Team Slides available at

692 views • 48 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides
-- Deep Neural Network or Gaussian Mixture Model? Dong Yu Microsoft Research Thanks to my

-- Deep Neural Network or Gaussian Mixture Model? Dong Yu Microsoft Research Thanks to my

Who Can Understand Your Speech Better -- Deep Neural Network or Gaussian Mixture Model? Dong Yu Microsoft Research Thanks to my collaborators: Li Deng, Frank Seide, Gang Li, Mike Seltzer, Jinyu Li, Jui-Ting Huang, Kaisheng Yao, Adam

967 views • 53 slides
Deep Convolutional Neural Network for Computer Vision Products LI XU, R&amp;D Director

Deep Convolutional Neural Network for Computer Vision Products LI XU, R&amp;D Director

Deep Convolutional Neural Network for Computer Vision Products LI XU, R&amp;D Director SenseTime Group Limited SenseTime Introduction SenseTime focuses on invention and development of computer vision and deep learning technologies. Our

1.03k views • 39 slides
Deep Learning Jiseob Kim (jkim@bi.snu.ac.kr) Artificial Intelligence Class of 2016 Spring Dept.

Deep Learning Jiseob Kim (jkim@bi.snu.ac.kr) Artificial Intelligence Class of 2016 Spring Dept.

Deep Learning Jiseob Kim (jkim@bi.snu.ac.kr) Artificial Intelligence Class of 2016 Spring Dept. of Computer Science and Engineering Seoul National University 1 History of Neural Network Research Neural network Deep belief net Back

1.3k views • 108 slides
Efficient Neural Network Compression Namhoon Lee University of Oxford 3 May 2019 A Challenge in

Efficient Neural Network Compression Namhoon Lee University of Oxford 3 May 2019 A Challenge in

Efficient Neural Network Compression Namhoon Lee University of Oxford 3 May 2019 A Challenge in Deep Learning: Overparameterization Large neural networks require: memory &amp; computations power consumption A Challenge in Deep Learning:

722 views • 49 slides
6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Monday (2/5) 30 minutes Topics: Optimization Basic neural networks Neural Network

619 views • 41 slides
Sony's deep learning software &quot;Neural Network Libraries/Console and its use cases in Sony

Sony's deep learning software &quot;Neural Network Libraries/Console and its use cases in Sony

Sony's deep learning software &quot;Neural Network Libraries/Console and its use cases in Sony Yoshiyuki Kobayashi Senior Machine Learning Researcher Sony Network Communications Inc. / Sony Corporation GTC 2018 Sony's deep learning

401 views • 21 slides
for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text

for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text

Deep Generative Modeling for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text Praveen Narayanan Ford Motor Company PROJECT DESCRIPTION Use of DNNs increasingly prevalent as a solution for many data

893 views • 53 slides
EN. 601.467/667 Introduction to Human Language Technology Deep Learning II Shinji Watanabe 1

EN. 601.467/667 Introduction to Human Language Technology Deep Learning II Shinji Watanabe 1

EN. 601.467/667 Introduction to Human Language Technology Deep Learning II Shinji Watanabe 1 Todays agenda Basics of (deep) neural network How to integrate DNN with HMM Recurrent neural network Language modeling

1.33k views • 60 slides
Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from

Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from

Neural Network Optimization 1 CS 519: Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Backpropagation learning of a network The algorithm 1. Compute a forward pass on the compute graph (DAG) from the input to all the

407 views • 16 slides
PERSISTENT AND UNFORGEABLE WATERMARKS FOR DEEP NEURAL NETWORKS Huiying Li, Emily Willson, Heather

PERSISTENT AND UNFORGEABLE WATERMARKS FOR DEEP NEURAL NETWORKS Huiying Li, Emily Willson, Heather

DNNS ARE INCREASINGLY POPULAR PERSISTENT AND UNFORGEABLE WATERMARKS FOR DEEP NEURAL NETWORKS Huiying Li, Emily Willson, Heather Zheng, Ben Y. Zhao Hey Siri Oct 30, 2019 1 2 DEEP NEURAL NETWORK (DNN) DNNS ARE HARD TO TRAIN A machine

115 views • 9 slides
Agenda Overview of deep learning Building a FAQ model with DeepLearning4J Integrating

Agenda Overview of deep learning Building a FAQ model with DeepLearning4J Integrating

Agenda Overview of deep learning Building a FAQ model with DeepLearning4J Integrating with a chatbot application Overview of deep learning AI ML Deep Learning Neural network architecture Neural network architecture Neural

469 views • 46 slides

More recommend