15/09/2017 Modelling word perception and comprehension across modalities Psychology in Big Question 1 PhD student : Danny Merkx Supervisors : Stefan Frank (CLS) Mirjam Ernestus (CLS) Raquel Fernandez (ILLC) Louis ten Bosch (CLS) Research objectives Develop a (cognitively plausible) vector representation of word form Apply these in a computational model that simulates spoken and written perception Investigate the interplay between form and meaning, and its role in learning and [kat] comprehension 1
15/09/2017 Challenge #1 Different modalities in a single model Current DSMs are amodal Word-perception models lack semantics and deal with one modality Written: Interactive Activation Spoken: TRACE (McClelland & Elman, 1986) (McClelland & Rumelhart, 1981) Reading is not independent from speech perception How to capture the unique perceptual constraints posed by different modalities in a vector model? Challenge #2 Complex relations between form, identity, and meaning 17 dec 1903 Meanings 20 jul 1969 14 sep 2017 Identities data data Data dates dates dates 2 dates 3 Forms [de:ts] [ deɪts ] [da:ta:] [ datə ] [ deɪtə ] 2
15/09/2017 Challenge #2 Models of the bilingual lexicon Simply running a DSM on a bilingual corpus would result in clustering by language → vectors do not reflect semantics (Most?) current work in bilingual DSMs: − Get two monolingual vector spaces − Combine such that translation equivalents receive similar vectors Artetxe, Labaka, & Agirre (ACL, 2017) No way to tell the languages apart − Not evaluated against human processing data Challenge #2 Models of the bilingual lexicon The bilingual mental lexicon: Languages are integrated but can still be told apart Psycholinguistic models of the bilingual mental lexicon: ‒ Account for response/naming times, cognate/homograph effects, interlingual priming, etc. ‒ But: small vocabularies, not trainable, no realistic semantics Costa et al. ( Cognitive Science , 2017) 3
15/09/2017 Challenge #3 Psycholinguistic evaluation How to measure the psychological accuracy of word vectors? How to evaluate representations on human data from sentence/discourse comprehension? Evaluation of statistical language models based on word surprisal: ‒ Correlate to measure of processing difficulty on naturalistic materials Challenge #3 Psycholinguistic evaluation Fit to N400 amplitude (± χ 2 ) Fit to RT (± χ 2 ) Average surprisal Average surprisal N400 size (content words only) Reading times from eye-tracking study Frank, Otten, Galli, & Vigliocco ( Brain & Frank & Thompson ( Proc. CogSci , 2012) Language , 2015) 4
15/09/2017 Challenge #3 Psycholinguistic evaluation How to measure the psychological accuracy of word vectors? How to evaluate representations on human data from sentence/discourse comprehension? Evaluating statistical language models based on word surprisal: ‒ Correlate to measure of processing difficulty on naturalistic materials ‒ Independent measure of linguistic accuracy Not (so much) is helpful for model comparison for DSMs ‒ Model variants are cognitively intepretable Psycholinguistic evaluation of DSMs Recent work Mandera, Keuleers, & Brysbaert (2017): – Compared state-of-the-art DSMs (Skipgram, CBOW) and traditional count-based DSM, on wide range of parameter values 5
15/09/2017 Psycholinguistic evaluation of DSMs Recent work Mandera, Keuleers, & Brysbaert (2017): – Compared state-of-the-art DSMs (Skipgram, CBOW) and traditional count-based DSM, on wide range of parameter values – Implicit measures (response times in semantic priming): fairly small difference between model types – Explicit norms (association, semantic relatedness): CBOW is best Rotaru, Vigliocco, & Frank (submitted): – Markov chain over semantic distance matrix (from CBOW, GloVe, LSA) simulates dynamics in semantic network 6
15/09/2017 Psycholinguistic evaluation of DSMs Recent work Mandera, Keuleers, & Brysbaert (2017): – Compared state-of-the-art DSMs (Skipgram, CBOW) and traditional count-based DSM, on wide range of parameter values – Implicit measures (response times in semantic priming): fairly small difference between model types – Explicit norms (association, semantic relatedness): CBOW is best Rotaru, Vigliocco, & Frank (submitted): – Markov chain over semantic distance matrix (from CBOW, GloVe, LSA) simulates dynamics in semantic network – This improves fit to human data (association/relatedness norms, lexical/semantic decision times and accuracies) – CBOW outperformed GloVe and LSA in almost all tests 7
15/09/2017 Psycholinguistic evaluation of DSMs Recent work Frank & Willems (2017): – Naturalistic materials: UCL corpus sentences (written) and excerpts from Dutch audiobooks (spoken) – Cosine distance (using Skipgram) between each content word and sum of previous content words Unique effects of suprisal and semantic distance Frank & Willems ( Language, Cognition and Neuroscience , in press) 8
15/09/2017 Psycholinguistic evaluation of DSMs Recent work Frank & Willems (2017): – Naturalistic materials: UCL corpus sentences (written) and excerpts from Dutch audiobooks (spoken) – Cosine distance (using Skipgram) between each content word and sum of previous content words – Explained variance in N400 and BOLD responses offers possibilities for comparing DSMs But reading times appear to be insensitive to semantic distance Reading times from two eye-tracking corpora: (no) effect of semantic distance Frank ( Proc. CogSci , 2017) Current semantic distance Previous semantic distance 0.04 0.04 UCL corpus coefficient 0.02 0.02 0 0 -0.02 -0.02 FF FP RB GP FF FP RB GP no surprisal 2-gram 3-gram 4-gram 5-gram 0.02 0.02 Dundee corpus coefficient 0.01 0.01 0 0 -0.01 -0.01 FF FP RB GP FF FP RB GP reading time measure reading time measure 9
15/09/2017 Potential pitfalls for BQ1 collaboration Different opinions about the meaning and importance of cognitively plausibility Different opinions about model evaluation: task performance versus human performance BQ1’s goal to link between neurobiology and cognition does not mean that psychology must be reduced to neuroscience Behavioural data is relevant too: Not all questions are about the brain and model comparison may be more difficult on high-dimensional (neural) data 10
Recommend
More recommend
