Table of contents 1. Introduction: You are already an - PowerPoint PPT Presentation

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8. Plotting 9. Building linear mixed effects models Section 2: Analysis 10. Evaluating linear mixed effects models using Fisher 11. Neyman-Pearson and controlling error rates 12. Bayesian statistics and Bayes Factors 13. Validity and replicability of judgments Section 3: 14. The source of judgment effects Application 15. Gradience in judgments 46

This section is about task effects The construction of experimental items is primarily about controlling for grammar confounds and other cognitive confounds. The construction of instruction items, practice items, and filler items is primarily about controlling for task effects. The arrangement of items into an actual experiment is also primarily about controlling for task effects. instruction items practice items filler items experimental items order of items Task Effects memory parsing + + Acceptability Grammar Noise world thought 47

Assign meaningful codes to your items Before we start to manipulate our target items, let’s talk about item codes . item codes: Meaningful codes that you assign to each of your items. These will help you quickly identify the properties of each item, and will play an important role in later data analysis. Item codes should contain all of the information about an item, such as the name of its condition (if you are naming your conditions), the levels of its factors (if you have a factorial design), and the lexically-matched item-set (or lexicalization-set) number that it is. Here is how I like to create item codes: subdesign.factor1.factor2.item-set-number whether island Condition 1: non-island short short wh.non.sh.01 Who __ thinks that Jack stole the car? wh.non.sh.02 Who __ thinks that Amy stole the gold? wh.non.sh.01 wh.non.sh.03 Who __ thinks that Dale stole the pie? set 1 wh.non.sh.04 Who __ thinks that Pat stole the pen? non-island 48

Assign meaningful codes to your items wh.non.sh.01 Who __ thinks that Jack stole the car? Note that each item code is unique to that item. So wh.non.sh.02 Who __ thinks that Amy stole the gold? they are unique identifiers. wh.non.sh.03 Who __ thinks that Dale stole the pie? wh.non.sh.04 Who __ thinks that Pat stole the pen? wh.non.lg.01 What do you think that Jack stole __? However, each code captures all of the wh.non.lg.02 What do you think that Amy stole __? relationships among the wh.non.lg.03 What do you think that Dale stole __? items. wh.non.lg.04 What do you think that Pat stole __? wh.isl.sh.01 Who __ wonders whether Jack stole the car? The global design is captured in the first part, wh.isl.sh.02 Who __ wonders whether Amy stole the gold? the factors in the middle wh.isl.sh.03 Who __ wonders whether Dale stole the pie? parts, and the lexical matching in the final part. wh.isl.sh.04 Who __ wonders whether Pat stole the pen? wh.isl.lg.01 What do you wonder whether Jack stole __? Using a separator like “.” makes it easy to pull this wh.isl.lg.02 What do you wonder whether Amy stole __? information apart in wh.isl.lg.03 What do you wonder whether Dale stole __? languages like R. wh.isl.lg.04 What do you wonder whether Pat stole __? 49

Divide items into lists List: A list is a set of items that will be seen by a single participant. It is not yet ordered for presentation. Let’s assume that these are our 4 conditions. We’ve made 4 items per condition. We want each participant to see all 4 conditions, and 1 item per condition. We don’t want participants to see the same lexical material (because then they might not notice the differences). How many lists can we make? wh.non.sh.01 Who __ thinks that Jack stole the car? wh.non.lg.01 What do you think that Jack stole __? wh.non.sh.02 Who __ thinks that Amy stole the gold? wh.non.lg.02 What do you think that Amy stole __? wh.non.sh.03 Who __ thinks that Dale stole the pie? wh.non.lg.03 What do you think that Dale stole __? wh.non.sh.04 Who __ thinks that Pat stole the pen? wh.non.lg.04 What do you think that Pat stole __? wh.isl.sh.01 Who __ wonders whether Jack stole the car? wh.isl.lg.01 What do you wonder whether Jack stole __? wh.isl.sh.02 Who __ wonders whether Amy stole the gold? wh.isl.lg.02 What do you wonder whether Amy stole __? wh.isl.sh.03 Who __ wonders whether Dale stole the pie? wh.isl.lg.03 What do you wonder whether Dale stole __? wh.isl.sh.04 Who __ wonders whether Pat stole the pen? wh.isl.lg.04 What do you wonder whether Pat stole __? 50

Divide items into lists The answer is that we can create 4 List 1 List 2 List 3 List 4 lists from this design. wh.non.sh.01 wh.non.sh.02 wh.non.sh.03 wh.non.sh.04 wh.non.lg.02 wh.non.lg.03 wh.non.lg.04 wh.non.lg.01 We want each list to have all 4 wh.isl.sh.03 wh.isl.sh.04 wh.isl.sh.01 wh.isl.sh.02 conditions, but to have a different lexical item for each condition. wh.isl.lg.04 wh.isl.lg.01 wh.isl.lg.02 wh.isl.lg.03 List 1 List 2 wh.non.sh.01 Who __ thinks that Jack stole the car? wh.non.sh.02 Who __ thinks that Amy stole the gold? wh.non.lg.02 What do you think that Amy stole __? wh.non.lg.03 What do you think that Dale stole __? wh.isl.sh.03 Who __ wonders whether Dale stole the pie? wh.isl.sh.04 Who __ wonders whether Pat stole the pen? wh.isl.lg.04 What do you wonder whether Pat stole __? wh.isl.lg.01 What do you wonder whether Jack stole __? List 3 List 4 wh.non.sh.03 Who __ thinks that Dale stole the pie? wh.non.sh.04 Who __ thinks that Pat stole the pen? wh.non.lg.04 What do you think that Pat stole __? wh.non.lg.01 What do you think that Jack stole __? wh.isl.sh.01 Who __ wonders whether Jack stole the car? wh.isl.sh.02 Who __ wonders whether Amy stole the gold? wh.isl.lg.02 What do you wonder whether Amy stole __? wh.isl.lg.03 What do you wonder whether Dale stole __? 51

The analogy to Latin Squares This design is often called a Latin List 1 List 2 List 3 List 4 Square design in experimental fields. wh.non.sh.01 wh.non.sh.02 wh.non.sh.03 wh.non.sh.04 Latin Squares have been wh.non.lg.02 wh.non.lg.03 wh.non.lg.04 wh.non.lg.01 mathematical puzzles for centuries. wh.isl.sh.03 wh.isl.sh.04 wh.isl.sh.01 wh.isl.sh.02 Euler studied them using Latin characters, hence the name. wh.isl.lg.04 wh.isl.lg.01 wh.isl.lg.02 wh.isl.lg.03 Latin Square (4 letters) Latin Square Design (4 conditions) List 1 List 2 List 3 List 4 A B C D wh.non.sh 1 2 3 4 B C D A wh.non.lg 2 3 4 1 C D A B wh.isl.sh 3 4 1 2 D A B C wh.isl.lg 4 1 2 3 The number in the cells represent items numbers from the lexically-matched sets. 52

Latin Squares by hand There are a large number of possible solutions to any given Latin Square problem, but we only need one solution. Here is an algorithm that will give you a Latin Square solution every time: 1. Copy all items of one condition (they should be in a column in excel). 2. Transpose the items into a row using paste-special. 53

Latin Squares by hand There are a large number of possible solutions to any given Latin Square problem, but we only need one solution. Here is an algorithm that will give you a Latin Square solution every time: 1. Copy all items of one condition (they should be in a column in excel). 2. Transpose the items into a row using paste-special. 3. Copy all items of a second condition (again, they should be a column). 4. Transpose the items into a row using paste-special, but this time, paste them below the first row, and one cell to the right. 5. Do the same thing with the third and fourth conditions. 54

Latin Squares by hand There are a large number of possible solutions to any given Latin Square problem, but we only need one solution. Here is an algorithm that will give you a Latin Square solution every time: 1. Copy all items of one condition (they should be in a column in excel). 2. Transpose the items into a row using paste-special. 3. Copy all items of a second condition (again, they should be a column). 4. Transpose the items into a row using paste-special, but this time, paste them below the first row, and one cell to the right. 5. Do the same thing with the third and fourth conditions. 6. Now, for each row, cut the items that go past column 4, and paste them into the empty cells at the beginning of the row. For row 2, there should be 1 item to cut; for row 3 there should be 2; for row 4 there should be 3. 55

Latin Squares by hand - item codes! The previous algorithm was performed on the items themselves, not the item codes. But in order to analyze your experiment, you need to have item codes. So, you need to create a second (identical!) latin square for the item codes. 1. Copy all item codes of one condition (they should be in a column in excel). 2. Transpose the codes into a row using paste-special. 3. Copy all codes of a second condition (again, they should be a column). 4. Transpose the codes into a row using paste-special, but this time, paste them below the first row, and one cell to the right. 5. Do the same thing with the third and fourth condition codes. 6. No do the copying procedure. 56