Finding a Jack-of-All-Trades: An Examination of Transfer Learning in - PowerPoint PPT Presentation

Finding a Jack-of-All-Trades: An Examination of Transfer Learning in Text Comprehension Kadlec, R., Bajgar, O., Hrinčár , P., Kleindienst, J. IBM Watson, Prague lab based on https://openreview.net/pdf?id=rJM69B5xx

Generalization is the key

Cloze style questions Children’s Book Test (Hill et al 2015) Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015). The Goldilocks Principle: Reading ~ 200k examples (CN+NE) Children’s Books with Explicit Memory Representations

Starting point ML Model Train Test ASReader Children’s Book BookTest (Bajgar et al, 2016) (Kadlec et al, Test 2016) 14M examples (Hill et al, 2015) CBT dev/test 2k examples Bajgar, O., Kadlec, R., & Kleindienst, J. (2016). Embracing data abundance: BookTest Dataset for Reading Comprehension. http://arxiv.org/abs/1610.00956

BookTest Trained on more data (BookTest) than the previous models! 5

Transfer learning? Train Test Children’s Book Test (Hill et al, 2015) AS BookTest (Bajgar et al, 2016) Reader 14M examples bAbI (Weston et al, 2015) Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merrienboer, B., Joulin, A., & Mikolov, T. (2015). Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.

Simple testing tasks: bAbI tasks

Simple testing tasks: bAbI tasks

Can it it generalize what it it le learned? Not really .. ... BAD! 9

Finetuning - bAbI Train Test bAbI 10 bAbI BookTest AS 100 bAbI Reader bAbI 1k

2nd Experiment: It It does better wit ith target-adjustment! 11

2nd Experiment: It It does better wit ith target-adjustment! 12

13

11 bAbI tasks mean b

Finetuning - SQuAD Train Test SQuAD BookTest (Bajgar et al, SQuAD 2016) ML dev 14M examples SQuAD (Rajpurkar et al 2016) A subset with single word answers Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text

subset SOTA is around 75% -> we are missing something, however pre-training still helps. 16

3rd Experiment: Where is is the useful knowledge? ? Part rtial pretraining Output Model parameters Input 17

18

19

Conclusions: • Pre-school helps • But it‘s not enough! • More work to be done! 20

Questions? 21