An introduction to computational psycholinguistics: Modeling human - PDF document

An introduction to computational psycholinguistics: Modeling human sentence processing Shravan Vasishth University of Potsdam, Germany http://www.ling.uni-potsdam.de/ ∼ vasishth vasishth@acm.org September 2005, Bochum “Background check” Who among you: 1. have some knowledge of behavioral-experiment design (e.g., factorial design)? 2. have ever carried out behavioral experiments? 3. are familiar with statistical data analysis techniques, e.g. analysis of variance, Chi-square tests, etc.? 4. have a background in programming (lisp, prolog)? 5. are familiar with connectionist models (in general)? 6. are familiar with probabilistic context free grammars? 7. are familiar with cognitive modeling in psychology? 8. have some background in parsing? know what a context-free grammar is? 9. have some background in syntactic theory (any kind)? 1

What this course is about Think about these sentences. There’s something strange about them. (1) a. The president awarded the prize died. b. After the student moved the chair broke. c. The friend of the shopkeeper who went to Japan on a holiday turned out to be very rich. d. The daughter of the colonel, who was standing by the window, looked up. (2) a. The man the dog saw died. b. The man the dog the cat bit saw died. c. The man died that the dog saw that the cat bit. 2 The problem • Natural language is rife with ambiguity and complexity. • Yet we humans (“normal” humans) have no problem processing language most of the time. Sometimes, we do. • What is the brain doing that it can take as input a sometimes ridiculously ambiguous sentence and, within a few hundred milliseconds, understand how it parses, its semantics and its implicatures and presuppositions? And generate a response immediately? • Clearly it’s doing something right. And we want to know what it is. For several reasons. • It’ll make you rich. DEMO: try the following sentence in the ERG online demo: The passenger offered a free ride in compensation for his troubles has refused to accept the offer. • It’s also an amazingly cool problem per se: what exactly is going on in the brain when we parse sentences? 3

Since we don’t care about getting rich. . . if you do, leave now before it’s too late • We’re going to look at attempts to answer this question from different perspectives: cognitive psychology, artificial intelligence, maybe even linguistic theory • All these attempts generate predictions. And these need to be evaluated, which brings up the issue of methods. Methods for testing predictions (experiment design), for measuring the responses, and analyzing them. So we’ll look at these briefly • The kinds of theories that have come up fall roughly into four baskets: discursive (“paper-pencil”; these may be specific enough to be implementable, but not necessarily), probabilistic, connectionist, and cognitive architectures (ACT-R) Note: this is a very selective tour. If I don’t mention some work it doesn’t mean it isn’t important. 4 Some central issues in real-time sentence processing • Ambiguity • Serial versus parallel parsing • Incrementality and prediction • Complexity • The underlying explanation for processing difficulty 5

Ambiguity • We’ve already seen some examples: (3) a. LEXICAL John was standing by the bank. b. HARD TO RECOVER GARDEN PATH The horse raced past the barn fell. (Bever 1970) c. EASY TO RECOVER GARDEN PATH After the student moved the chair broke. d. SYNTACTIC ATTACHMENT AMBIGUITY The daughter of the colonel who was standing by the window. • Two generalizations regarding syntactic ambiguity resolution: minimal attachment, and early/late closure (Frazier 1979): 6 Minimal attachment and late closure Minimal attachment: Choose the structurally simplest analysis (fewest additional nodes) (4) a. HARD TO RECOVER GARDEN PATH The horse raced past the barn fell. (Bever 1970) b. SYNTACTIC ATTACHMENT AMBIGUITY The daughter of the colonel who was standing by the window. Late closure: Integrate current input into current constituent (when possible). (5) After the student moved the chair broke. Note that prosody—an intonational phrase boundary after moved —plays a critical role. Even when we read we are using prosody silently (Fodor 2002). One piece of evidence for this comes from relative clause attachment ambiguity. Aside 7

Relative clause attachment ambiguities and prosody • In English there is a preference for local attachment: (6) Someone hit the maid of the actress who was on the balcony. • However, in Spanish (7) the preference is for non-local attachment to the first noun criada ((Cuetos & Mitchell, 1988), among others). (7) Alguien peg´ o a la criada de la actriz que estaba en el balc´ on someone hit dat the maid of the actress who was on the balcony ‘Someone hit the maid of the actress who was on the balcony.” • Some other languages that behave like Spanish include French (Mitchell, Cuetos, & Zagar, 1990), Italian (De Vincenzi & Job, 1993), German (Hemforth, Konieczny, & Scheepers, 1994), and Dutch (Brysbaert & Mitchell, 1996). • Hindi (Vasishth, Agnihotri, Fern´ andez, & Bhatt, 2005) shows local attachment for postnominal RCs, but no preference for participial RCs: Aside 8 Relative clause attachment ambiguities and prosody (8) a. kisii-ne usa abhinetrii-kii usa naukaraanii-ko [jo caaye pii someone-erg that actress-gen that maid-acc who tea drinking rahii-thii] maara was hit ‘Somebody hit the actress’ maid who was drinking tea.’ b. kisii-ne [caaye pii rahii] usa abhinetrii-kii usa naukaraanii-ko someone-erg tea drinking was that actress-gen that maid-acc maara hit ‘Somebody hit the actress’ maid who was drinking tea.’ In fact, there are at least six possible positions for relative clauses, and a production study (Vasishth, Fern´ andez, and F´ ery, in progress) finds that all possibilities were used by subjects producing semi-spontaneous RC constructions. Plus some others syntacticians would dismiss as ungrammatical, but that’s another story. Aside 9

Relative clause attachment ambiguities and prosody There is evidence in these languages that subjects tend to make a prosodic break before the RC in non-local attachment languages, but not in local attachment ones. In fact, in English, one can induce nonlocal attachment with a prosodic break (see also (Fern´ andez, Bradley, & Taylor, 2005; Fern´ andez, Bradley, Igoa, & Teira, 2003)). (9) The daughter of the colonel, who was standing by the window, looked up. So, even in this tiny, tiny problem, clearly the story is more complex than explanations like Minimal Attachment and Late Closure. How to answer this question? Aside 10 Some possible approaches • Give up and start work in some other area. • Ignore counterexamples (or pretend they are not there) and doggedly stick to your theory. • Turn the observation into an explanation: “Our revised theory is that local attachment is preferred unless prosody decides otherwise.” Aside 11

A saner alternative Build a precise, working process model (ideally an implementation) that (a) can be falsified, (b) generates novel (preferably surprising) and nontrivial predictions. And test them. If the model works, try to break it with new predictions. If it breaks, enhance it, regression-test it (it should not lose any coverage it has, the empirical coverage should increase monotonically), and then generate more predictions and test them. [Even in detailed process models it’s easy to re-cast the observation as the explanation, and there are all kinds of fiddle factors like free numerical parameters. They are not the solution to all our scientific problems, but they’re better in general than discursive models.] Aside 12 Some possible approaches: Discursive models Why not just stick to discursive models? It turns out it’s damn hard to figure out what the predictions are of a discursive model. A phrase one often hears when one presents a counterexample to a theory: “Well, it depends on how you construe the theory.” This is an useful and productive hedge when we are in the process of theorizing, but in the end, without an implementation (on paper or on computer) we will never commit to a position, rendering the theory unfalsifiable. Moving on to the other central issues. . . Aside 13

Memory capacity and ambiguity resolution The limited capacity of working memory has been a central motivating factor in the development of theories of ambiguity in sentence processing. Frazier suggested that Minimal Attachment and Late Closure is a reflex of a contrained capacity working memory system. Regarding Late Closure, she says (Frazier, 1979, 39): “It is a well-attested fact about human memory that the more structured the material to be remembered, the less burden the material will place on immediate memory. Hence, by allowing incoming material to be structured immediately, Late Closure has the effect of reducing the parser’s memory load.” 14 Memory capacity and ambiguity resolution Similarly, regarding Minimal Attachment (Frazier, 1979, 40): “[T]he Minimal Attachment strategy not only guarantees minimal structure to be held in memory, but also minimizes rule accessing. Hence, [Minimal Attachment is also an ‘economical’ strategy] in the sense that [it reduces] the computation and memory load of the parser.” 15