Table of contents 1. Introduction: You are already an - PowerPoint PPT Presentation

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8. Plotting 9. Building linear mixed effects models Section 2: Analysis 10. Evaluating linear mixed effects models using Fisher 11. Neyman-Pearson and controlling error rates 12. Bayesian statistics and Bayes Factors 13. Validity and replicability of judgments Section 3: 14. The source of judgment effects Application 15. Gradience in judgments 98

Institutional Review Board (IRB) Approval In the US, before you recruit human participants, you will need approval from your university’s Institutional Review Board (the IRB). This is generally a painless process, but it can take a month or more, so you should start planning early. The process varies from institution to institution, so I can’t give you detailed instructions. But I do have some general recommendations: 1. If possible, I would suggest requesting approval for all four possible tasks, both online and offline, and for all possible languages you might study in one application. I would also request approval to test several thousand participants. This will save you time down the road. 2. Acceptability judgments generally fall under survey procedures, which means that they are exempt (category 2) . This means that they are exempt from full board review, and instead only require review by the chair of the board. This generally means that the review process will be a bit faster. (The other levels of review are “expedited”, which also doesn’t require full board review, and “full”, which is full board review) 3. Most IRBs require some sort of training before you can submit a proposal for review. So be sure to complete that before you submit your proposal. 99

Amazon Mechanical Turk If you are going to be working on US English, Amazon Mechanical Turk can be a great resource for recruiting participants. Pros: Fast! You can collect a hundred participants in an hour. More diverse than a university participant pool. Cons: Not free. You must pay participants (and Amazon). Less control over the properties of the participants. 100

AMT Sandbox The first step is to create a requester account. (AMT divides users into requesters, who post tasks, and workers, who complete them). If you want to practice using AMT without having to put up a real survey, you can use the requester’s sandbox. This is a simulated AMT environment where you can test your experiments without any risk (and without paying anything). 101

Two stages: create and manage I am going to use my real account to show you what creating an experiment looks like. There are basically two stages: the create stage, where you create your experiment, and the manage stage, where you deploy your experiment and watch the results come in. 102

Creating an experiment When you click on Create, you will see a list of all of the experiments that you’ve run in the past. This lets you easily re-use them (or edit them) if you need to. Your list will be empty (or have demos in it). But you can easily create a new one using one of the AMT HTML templates I have made available. 103

Create: Enter Properties There are three parts to creating an experiment: entering its properties, designing the layout, and then looking for errors. We start with entering the properties. The first box is where you enter information that the workers will see. I like to tell them how long I think it will take, how much I am going to pay, and any requirements that I have (that aren’t enforced by AMT - more on this soon.) 104

Create: Enter Properties The second box in “enter properties” is where you set the specific properties of this HIT (Human Intelligence Task — this is what AMT calls a task). The first box is how much you will the participant. The second is the number of participants you want to recruit per HIT. Each ordered list you have is a HIT, so you have to do some math here. If you have 8 ordered lists, and want 24 participants in your sample, then you need 3 participants per list. Since each list is a HIT, you need 3 assignments per HIT. More generally: Number of assignments per HIT = total sample size / number of ordered lists 105