Designing Experiments in Political Science Delegation in - PowerPoint PPT Presentation

Designing Experiments in Political Science Delegation in Bureaucracies Michael F. Stoffel University of Konstanz Department of Politics and Public Administration UniversitätäKonstanz5äFachä9?5ä78457äKonstanz DrxäMichaeläHerrmann FachbereichäPolitik3 und Verwaltungswissenschaft Fachä9? Universitätsstraßeä,d D378464äKonstanz Telä+49ä753,ä883477, Faxä+49ä753,ä8834?dd MichaelxHerrmann@uni3konstanzxde ??x ??x ???? Dagstuhl Seminar “Empirical Evaluation for Graph Drawing” 26 to 30 January 2015

Experimental Political Science We are limited by the impossibility of experiment. Politics is an observational, not an experimental science ... A. Lawrence Lowell President of the APSA, 1910 Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Experimental Political Science Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Examples of Applications media effects (Iyengar and Kinder 1987) mobilization (Gerber and Green 2000) voting (Lodge, McGraw, and Stroh 1989) legislative and bureaucratic rules (Eavey and Miller 1984; Miller, Hammond, and Kile 1996) foreign policy decisionmaking (Geva and Mintz 1997) international negotiations (Druckman 1994) coalition bargaining (Riker 1967; Fr´ echette et al. 2003) electoral systems (Morton and Williams 1999) Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Types of Experimental Approaches Laboratory experiments, a.k.a controlled experiments Problem: sample often not representative Survey experiments Problem: treatment assignment can be corroborated Field experiments Problem: less control over experimental stimuli (Computer experiments) Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Types of Experimental Approaches Laboratory experiments, a.k.a controlled experiments Problem: sample often not representative Survey experiments Problem: treatment assignment can be corroborated Field experiments Problem: less control over experimental stimuli (Computer experiments) Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Coffee break ... Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Foundations Potential outcomes framework (Neyman, 1923) There are two different potential outcomes for the same person depending on whether or not she receives a treatment (counterfactual). Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Foundations Potential outcomes framework (Neyman, 1923) There are two different potential outcomes for the same person depending on whether or not she receives a treatment (counterfactual). Fundamental problem of causal inference (Holland 1986) We cannot simultaneously observe a person or entity in its treated and untreated states. Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Foundations Potential outcomes framework (Neyman, 1923) There are two different potential outcomes for the same person depending on whether or not she receives a treatment (counterfactual). Fundamental problem of causal inference (Holland 1986) We cannot simultaneously observe a person or entity in its treated and untreated states. Solution: Take two groups of individuals one receives the treatment, the other serves as the control calculate the mean value on the outcome variable for both groups calculate the difference between the means = ⇒ average treatment effect (ATE) Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Assumptions I Unconfoundedness Assignment to treatment and control group is independent from the potential outcomes. Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Assumptions I Unconfoundedness Assignment to treatment and control group is independent from the potential outcomes. Individuals are individual and not identical Possible characteristics of each individual may affect the assignment to the treatment (“self-selection”) the way the treatment works Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Assumptions I Unconfoundedness Assignment to treatment and control group is independent from the potential outcomes. Individuals are individual and not identical Possible characteristics of each individual may affect the assignment to the treatment (“self-selection”) the way the treatment works Random assignment (Fisher 1935) Each individual has an equal probability of being in the treatment group. = ⇒ all groups have the same individual characteristics in expectation Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Assumptions II Stable Unit Treatment Value Assumption (SUTVA) “The observation on one unit should be unaffected by the particular assignment of treatments to the other units.” (Cox, 1958) Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Assumptions II Stable Unit Treatment Value Assumption (SUTVA) “The observation on one unit should be unaffected by the particular assignment of treatments to the other units.” (Cox, 1958) Outcome of individual i does not depend on whether or not individual j was given a treatment. This it typically violated under social interference, i.e., if individuals interact (broadly defined). Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Internal and external validity Internal validity means whether the design of an experiment and the causal argumentation are correct. External validity asks whether the conclusions drawn from an experiment can be generalized to a larger population. Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Threats to internal validity Failure of randomization Social interference, a.k.a. diffusion (violation of SUTVA) Confounding variables Selection-Maturation Interaction Maturation (fatigue and learning) Repeated testing Non-compliance with experimental protocol Attrition (loss of participant), e.g., through dropout, non-response, or withdrawal Reactivity, i.e., individuals alter their performance or behavior due to the awareness that they are being observed (placebo, novelty, and Hawthorne effects) Control vs. treatment group motivation (John Henry effect) Experimenter bias To be continued ... Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Furthering internal validity Make the task sufficiently absorbing that the subject finds it more interesting to concentrate on the task at hand Pilot study, pre-tests Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Threats to external validity Sample is not representative = ⇒ confounding factors may affect the way the treatment works At the most extreme: individuals that self-select into experiments might differ from those that do not Experimental situation differs from “reality” Does the experimental stimulus resemble the “true” stimulus To be continued ... Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Extension: within-subjects design Thus far: between-subject design, i.e., each individual only receives one treatment. Now: within-subject design, a.k.a repeated-measures design, i.e., each individual receives more than one treatment. Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Extension: within-subjects design Thus far: between-subject design, i.e., each individual only receives one treatment. Now: within-subject design, a.k.a repeated-measures design, i.e., each individual receives more than one treatment. Advantages Given the same number of subjects, statistical power increases. Thus, the experiment can be run with fewer subjects and is cheaper. Eliminates differences between treatment and control group Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Extension: within-subjects design Thus far: between-subject design, i.e., each individual only receives one treatment. Now: within-subject design, a.k.a repeated-measures design, i.e., each individual receives more than one treatment. Advantages Given the same number of subjects, statistical power increases. Thus, the experiment can be run with fewer subjects and is cheaper. Eliminates differences between treatment and control group Drawbacks Being exposed to one treatment may influence the effect of another treatment (carryover effect) Fatigue and learning/practice effects Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

Extension: within-subjects design If possible, use a counterbalanced measures design; that is, test the different treatments in varying order. E.g., if there are three treatments A, B, and C, there are six possible orders to test: ABC, ACB, BAC, BCA, CAB and CBA. Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science