JESA and Replication Studies 2015 workshop in Experimental Methods: The replicability crisis in the social sciences and how to address it Bob Slonim 27, November, 2015
Why a new journal for experimental economics? • Demand/Growth of the field – ESA established 1986 – Experimental Economics First Issue 1998 • A need for experimental methodology research • A need for replication studies
Experimental Economics 2005 ‐ 2014 300 60 Number of submissions 250 50 Number of original papers published Number of papers Acceptance rate Acceptance rate 200 40 150 30 100 20 50 10 0 0 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Year
Experimental Publications in “Top ‐ 5”: 1975 ‐ 2014 50 45 Number of published exp/al papers lab experiments field experiments 40 35 30 25 20 15 10 5 0 75-7677-7879-8081-8283-8485-8687-8889-9091-9293-9495-9697-9899-0001-0203-0405-0607-0809-1011-1213-14 Year
Growth of the ESA 2001 ‐ 2014 1100 ESA members ESA Conference Attendees 1000 900 800 700 600 500 400 300 200 100 0 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Year
JESA: Aims & Scope • JESA will focus on publishing shorter papers (original articles, methodological pieces, surveys, comments on recently published experimental papers), and article types that are important yet under ‐ represented in the experimental literature (i.e., replications, minor extensions, robustness checks, meta ‐ analyses, and good experimental designs even if obtaining null results)
JESA: Aims & Scope • The Journal of the Economic Science Association (JESA) is dedicated to advancing theoretical, empirical, methodological and policy ‐ relevant knowledge using experimental economic methods . JESA promotes research pioneering and advancing laboratory and field methods to address important economic questions that are difficult to examine using naturally occurring data . JESA is open to all areas of inquiry in economics and at the intersection of economics and other disciplines including but not limited to psychology, political science, statistics, finance, marketing, and organizational behavior.
What is a replication study?????? • “Run a study the way I think it should be run” • Consider the following seminal study conducted by Researcher O (original): – Between Subject Design – Two conditions: Control (C) and Treatment (T) – Theory ‐ > Hypothesis: • Mean choice in T greater than mean choice in C – Results show significant difference at the p < .05 level
Question 1: Sample Population • Researcher R1 (Replicator) runs the identical study and only changes the subject population: – Recruits from the general population – Recruits from a communist country – Recruits children – Recruits nurses – Recruit from Identical population, but the recruitment process puts more stress on monetary rewards, or helping science, or the time involved or the nature of tasks, … – Etc
Question 2: Information Presentation • Researcher R2 (Replicator) runs the identical study and only changes the way that payoff function is presented – Shown as an equation – Shown in a table – Shown graphically – Shown with different color schemes – Etc
Question 3: Payoffs • Researcher R3 (Replicator) runs the identical study and only changes the payoffs: – Increases all payoffs by a factor of 10 – Increases all payoffs by $1 – Identical payoffs but calls them experimental units rather than dollars – Determination of randomization (die roll, computerized, bingo cage) – Etc
Question 4: Instructions/Protocols • Researcher R4 (Replicator) runs the identical study and only changes the instruction protocols: – Minor wording changes – Read out loud – Includes review questions – Includes (different) examples – Etc
It depends on …
It depends on … Researcher Objectives If researcher had no reason to expect a different result:
It depends on … Researcher Objectives If researcher had no reason to expect a different result: Then Replication Study
It depends on … Researcher Objectives If researcher expected a different result: Then Theoretical Contribution
Designing a Replication Study • Behavioral Economics Replication Project: http://sciencepredictionmarkets.com/studies.html • “ The effect in the replication is in the same direction as in the original study, and is statistically significant with a p ‐ value smaller than 0.05. ” • Statement of hypothesis (Based on original study) • Replication sample size (Based on replication study) • Replication Power (90% and higher in this project) based on data from Original study and sample size from replication study
Experimental Economics and Replication Study Importance • Laboratory experimenters indeed replicate all the time! – To better understand why T>C, we run T and C, and then T1, T2, … to disentangle different hypotheses, to explore robustness, to stress test, … – It is part of the DNA of (lab) experimental economics • Field experiments? – Not common (rare?)
What should be replicated? • Consider how many other (published) papers have run the same or similar studies? None? Thousands? (Hello My Bayesian) – Potential opportunities with field experiments • Consider the relative importance (citations, implications/policy impact) – Consider where the paper was published • Consider the theory – E.g. cognitive dissonance • Consider the empirical analyses/original data • Consider the “smell” factor – (you may not be alone)
Thank you
Recommend
More recommend
