Table of contents 1. Introduction: You are already an - PDF document

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8. Plotting 9. Building linear mixed effects models Section 2: Analysis 10. Evaluating linear mixed effects models using Fisher 11. Neyman-Pearson and controlling error rates 12. Bayesian statistics and Bayes Factors 13. Validity and replicability of judgments Section 3: 14. The source of judgment effects Application 15. Gradience in judgments 106

First: Find the confounds For exercise 3, you scrutinized the design from Dillon & Hornstein (2013). They crossed EXTRACTION (+/- WH) with COMPLEMENT TYPE (naked infinitival clause NIC versus noun complement construction NCC), to create a 2x2 design: -WH, NIC: Mary heard the sneaky burglar clumsily attempt to open the door -WH, NCC: Mary heard the sneaky burglar’s clumsy attempt to open the door +WH, NIC : What did Mary hear the sneaky burglar clumsily attempt to open? +WH, NCC : What did Mary hear the sneaky burglar’s clumsy attempt to open? What potential confounds did you see in this design? Are they controlled for by the 2x2 design, or is additional norming necessary? 107

Today: Additional software and tools for the experimental syntactician Additional software: The Ibex Farm Platform(s) for deploying web-based experiments: Amazon’s Mechanical Turk (and similar) ( SEE ALSO: The Prolific Academic (specifically for academic studies) : www.prolific.ac Crowdflower (for training machine learning classifiers, usually very small studies): www.crowdflower.com Le RISC (Relais d’information sur les sciences de la cognition; free service to find French-speaking participants from metropolitan France run by CNRS): http://expesciences.risc.cnrs.fr 108

The Ibex Farm / Linger Last class, we discussed principles of Latin Square distribution, pseudo- randomization, and counterbalancing. These are necessary if you create surveys by hand (as we will). However, there are many pieces of experimental software that are designed to streamline or facilitate this process that you may find useful: The Ibex Farm : http://spellout.net/ibexfarm . Javascript-based software for deploying web-based experiments. Can create and deploy a variety of judgment-based methodologies, in addition to other psycholinguistic paradigms (e.g. self-paced reading). Pros: highly flexible platform, can be customized, can deploy experiments over the web; Cons: requires familiarity with Javascript. Linger: http://tedlab.mit.edu/~dr/Linger/. TCL/TK-based software. Similar flexibility/range of experimental methodologies to Ibex Farm. Pros: Easy to learn and deploy. Cons: cannot be used to do web-based surveys / experiments. 109

The Ibex Farm Easy to deploy judgment experiments; runs in a web- browser! 110

The Ibex Farm STEP 1: Create an account. Click here, follow instructions! 111

The Ibex Farm List of experiments created under your account. Unlimited storage (or at least I haven’t found the upper limit), and experiments stay indefinitely (but you should still back-up!) 112

Link to your experiment Directory for HTML files you would like to present (not used in class example) Directory for CSS files that control presentation parameters (font, etc). Modify at your peril! Directory for main experiment file. This is the only thing you need to edit for a basic experiment! Directory with Javascript files that implement Ibex functions. Modify at your peril! Directory that stores results. Directory that stores current Latin Square counter. 113

In this tutorial, we will only set up the data_includes file; this implements the main ‘body’ of the experiment, and is all you need for a basic Ibex set-up (though you can do much more!!). The logic is the following: 1) Set up “trial” definitions: descriptions of each possible “trial” in the experiment. Do this for all sentences we will present. 2) Tell Ibex what you want a default “trial” to look like (how many response options, etc) 3) Write instructions/practice. 4) Tell Ibex presentation order. 114

Trial definition We have coded up our experimental items in an Excel spreadsheet and given them appropriate codes, e.g.: wh.non.sh.01 Who thinks that Paul stole the necklace? To create an experimental trial from this sentence in Ibex, we have to embed it in the appropriate syntax: [["wh.non.sh",1], "AcceptabilityJudgment", {s: "Who thinks that Paul stole the necklace?"}] 115

Trial definition We have coded up our experimental items in an Excel spreadsheet and given them appropriate codes, e.g.: wh.non.sh.01 Who thinks that Paul stole the necklace? To create an experimental trial from this sentence in Ibex, we have to embed it in the appropriate syntax. Start with an open square bracket, and then we give Ibex three critical pieces of info: [["wh.non.sh",1], "AcceptabilityJudgment", {s: "Who thinks that Paul stole the necklace?"}] #1 : Item label and item set #. For experimental items, we put the item label (condition label) in quotes , and in the first position in a mini-list here. After this, [“CONDITION LABEL”,ITEM SET #] put the item set # (called group # in Ibex); no quotes here . This information allows Ibex to do automatic Latin Squaring. 116

Trial definition We have coded up our experimental items in an Excel spreadsheet and given them appropriate codes, e.g.: wh.non.sh.01 Who thinks that Paul stole the necklace? To create an experimental trial from this sentence in Ibex, we have to embed it in the appropriate syntax. Start with an open square bracket, and then we give Ibex three critical pieces of info: [["wh.non.sh",1], "AcceptabilityJudgment", {s: "Who thinks that Paul stole the necklace?"}] #2 : Type of trial (a ‘controller’ in Ibex Lingo); quotes needed. Many options “Controller name” possible; we will only use ‘AcceptabilityJudgment’ here to do acceptability judgment trials. 117

Trial definition We have coded up our experimental items in an Excel spreadsheet and given them appropriate codes, e.g.: wh.non.sh.01 Who thinks that Paul stole the necklace? To create an experimental trial from this sentence in Ibex, we have to embed it in the appropriate syntax. Start with an open square bracket, and then we give Ibex three critical pieces of info: [["wh.non.sh",1], "AcceptabilityJudgment", {s: "Who thinks that Paul stole the necklace?"}] #3 : Arguments that define critical features of this acceptability judgment trial. s = ‘sentence’ argument. After s: , put the sentence for this trial in quotes . { … arguments … } { s: “SENTENCE IN QUOTES”} 118

Trial definition We have coded up our experimental items in an Excel spreadsheet and given them appropriate codes, e.g.: wh.non.sh.01 Who thinks that Paul stole the necklace? To create an experimental trial from this sentence in Ibex, we have to embed it in the appropriate syntax. Start with an open square bracket, and then we give Ibex three critical pieces of info: [["wh.non.sh",1], "AcceptabilityJudgment", {s: "Who thinks that Paul stole the necklace?"}] You may change any part; but for a basic acceptability judgment experiment we change *only* ’s’, and condition labels! 119

Trial definition We have coded up our experimental items in an Excel spreadsheet and given them appropriate codes, e.g.: wh.non.sh.01 Who thinks that Paul stole the necklace? To create an experimental trial from this sentence in Ibex, we have to embed it in the appropriate syntax. Start with an open square bracket, and then we give Ibex three critical pieces of info: [["wh.non.sh",1], "AcceptabilityJudgment", {s: "Who thinks that Paul stole the necklace?"}] Other controllers include: “Question” (for posing a question and listing answers to choose from), “DashedAcceptabilityJudgment” (for speeded acceptability judgments), “DashedSentence” (for self-paced reading experiments) … see documentation for others! 120

Trial definition If you have fillers that will not be Latin Squared, then you do not need to define an item set. Trial definitions are therefore simpler: ["F-1F.01", "AcceptabilityJudgment", {s: "Mike prefers tennis because Jon baseball."}] Just a simple label in quotes, now! 121

The ‘items’ variable Once you’ve defined all your trial variables, you have to collect them up into a single master list called the ‘items’ variable in Ibex Lingo. To define an items variable, start by typing… var items = [ 122