Table of contents 1. Introduction: You are already an - PowerPoint PPT Presentation

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8. Plotting 9. Building linear mixed effects models Section 2: Analysis 10. Evaluating linear mixed effects models using Fisher 11. Neyman-Pearson and controlling error rates 12. Bayesian statistics and Bayes Factors 13. Validity and replicability of judgments Section 3: 14. The source of judgment effects Application 15. Gradience in judgments 28

Linguistics tends to use repeated measures Repeated Measures Independent Measures condition 2 condition 1 condition 2 condition 1 Repeated If each participants sees every condition, we call it repeated Measures: measures. It is also called a within-subjects design. Independent If each participants sees only one condition, we call it Measures: independent measures. It is also called a between-subjects design. 29

Linguistics tends to use repeated measures Repeated Measures Independent Measures Requires fewer participants Requires more participants Individual differences between Individual differences between participants is not a confound participants is a possible confound Increased statistical power Decreased statistical power Interaction of two conditions is a Interaction of two conditions is potential confound impossible 30

There are four types of items to create After you have designed your conditions, the next step is to actually make the items that will go in your experiment. The are four types of items that you will need to construct: Instruction items: These are the items that appear in your instructions. The goal there is to illustrate the task, and if necessary, anchor the response scale. Practice items: These are items that occur at the beginning of the experiment. They help to familiarize the participant with the task. They are typically not analyzed in any way. They can be marked as separate (announced) or just part of the experiment (unannounced). Experimental items: These are your treatment and control conditions. Filler items: These are items that you add to the experiment for various reasons: filling out the scale, hiding the experiment’s purpose, and balancing types of items. 31

Instruction items The number and type of instruction items depends on your task. If the task is a scale task with an odd number of points (e.g, 7-point scale), I recommend three instruction items: one at the bottom of the scale, one at the top, and one in middle. Here are three that I use. They were pre-tested in my massive LI replication study: LI-Mode LI-Mean The was insulted waitress frequently. 1 1 Tanya danced with as handsome a boy as her father. 4 4 This is a pen. 7 7 If the scale has an even number of points, you would probably just use two: the bottom and top of the scale. If the task is yes/no, you might use three: a clear yes, a clear no, and one in between. If the task is forced-choice, you might use 3 pairs: a pair with a large difference, a pair with a medium difference, and one with a small difference. 32

Practice items Practice items give participants a chance to work out any bugs before they respond to items that you actually care about (the experimental items). For scale tasks, practice items give participants a chance to see the full range of variability in acceptability, so that they can use the scale appropriately. So in scale tasks, it is important to have practice items that span the range of acceptability. Here are 9 that I have pre-tested in the LI study. One for each point on a 7-point scale, plus one more for each endpoint. LI-Mode LI-Mean She was the winner. 7 7.00 Promise to wash, Neal did the car. 1 1.31 The brother and sister that were playing all the time 4 3.91 had to be sent to bed The children were cared for by the adults and the teenagers 6 6.08 Ben is hopeful for everyone you do to attend. 2 2.00 All the men seem to have all eaten supper 5 4.92 They consider a teacher of Chris geeky. 3 3.09 It seems to me that Robert can’t be trusted. 7 6.92 There might mice seem to be in the cupboard. 1 1.25 33

Practice items For non-scale tasks, the rationale behind the practice items might be different. For yes/no tasks, you may want to give a mix of clear yes’s, clear no’s, and intermediate sentences, so that participants can sharpen their own internal boundary. For forced-choice tasks, you may want to include a mix of large differences, small differences, and medium differences, so that participants can practice identifying each size of difference. Announced practice is when you clearly indicate in the experiment that the items are practice items. This signals to the participants that it is ok to make mistakes. Announced practice is typical in psycholinguistic experiments, because it gives participants a chance to ask questions of the experimenter. Unannounced practice is when the practice items simply appear as part of the main experiment. This is appropriate if the task is relatively intuitive, such that participants won’t have questions. This is what I do with all of my judgment studies. I typically present the (unannounced) practice items in the same order for all participants. You could also counterbalance the order (more on this later). 34

Experimental items Here is a starting set of experimental items for the whether island experiment we started to construct in the previous section. Let’s use these to see the issues that arise in creating experimental items. Condition 1: non-island short Condition 2: non-island long Who __ thinks that Jack stole the car? What do you think that Jack stole __? 1. 1. Who __ thinks that Amy chased the bus? What do you think that Amy chased __? 2. 2. Who __ thinks that Dale sold the TV? What do you think that Dale sold __? 3. 3. Who __ thinks that Stacey wrote the letter? What do you think that Stacey wrote __? 4. 4. Condition 3: island short Condition 4: island long 1. Who __ wonders whether Jack stole the car? 1. What do you wonder whether Jack stole __? 2. Who __ wonders whether Amy chased the bus? 2. What do you wonder whether Amy chased __? 3. Who __ wonders whether Dale sold the TV? 3. What do you wonder whether Dale sold __? 4. Who __ wonders whether Stacey wrote the letter? 4. What do you wonder whether Stacey wrote __? 35

Experimental items - Lexically matched sets The first thing to note is that the items are created in lexically matched sets. The idea here is that the only thing you want varying between conditions is the syntactic manipulation. So, to the extent possible, you use the same lexical items in all 4 conditions. Condition 1: non-island short Condition 2: non-island long Who __ thinks that Jack stole the car? What do you think that Jack stole __? 1. 1. Who __ thinks that Amy chased the bus? What do you think that Amy chased __? 2. 2. Who __ thinks that Dale sold the TV? What do you think that Dale sold __? 3. 3. Who __ thinks that Stacey wrote the letter? What do you think that Stacey wrote __? 4. 4. Condition 3: island short Condition 4: island long 1. Who __ wonders whether Jack stole the car? 1. What do you wonder whether Jack stole __? 2. Who __ wonders whether Amy chased the bus? 2. What do you wonder whether Amy chased __? 3. Who __ wonders whether Dale sold the TV? 3. What do you wonder whether Dale sold __? 4. Who __ wonders whether Stacey wrote the letter? 4. What do you wonder whether Stacey wrote __? This helps minimize confounds in the experiment. The only lexical confound left is if the syntactic manipulation interacts with the lexical items. 36

Experimental items - variability The second thing to note is that the variability in the items is tightly controlled. In this case, I primarily varied content items, keeping functional items the same. There is a tension between variability and control. I tend to err on the side of control so that there are fewer chances for confounds. Condition 1: non-island short Condition 2: non-island long Who __ thinks that Jack stole the car? What do you think that Jack stole __? 1. 1. Who __ thinks that Amy chased the bus? What do you think that Amy chased __? 2. 2. Who __ thinks that Dale sold the TV? What do you think that Dale sold __? 3. 3. Who __ thinks that Stacey wrote the letter? What do you think that Stacey wrote __? 4. 4. Condition 3: island short Condition 4: island long 1. Who __ wonders whether Jack stole the car? 1. What do you wonder whether Jack stole __? 2. Who __ wonders whether Amy chased the bus? 2. What do you wonder whether Amy chased __? 3. Who __ wonders whether Dale sold the TV? 3. What do you wonder whether Dale sold __? 4. Who __ wonders whether Stacey wrote the letter? 4. What do you wonder whether Stacey wrote __? However, variability is also important. When items vary, you can begin to see how well the effect generalizes across lexical items. 37