THE ESSENTIAL ROLE OF EXTERNAL AND CONSTRUCT VALIDITY FOR CAUSAL - PowerPoint PPT Presentation

https://www.dropbox.com/sh/ti09kj0f53e1yw2/AABmNY9SSzTDMCkeviHj_nvja?dl=0 THE ESSENTIAL ROLE OF EXTERNAL AND CONSTRUCT VALIDITY FOR CAUSAL IDENTIFICATION Kevin Esterling, School of Public Policy & Department of Political Science, UCR David Brady, School of Public Policy, UCR & WZB Berlin Social Science Center Eric Schwitzgebel, Department of Philosophy, UCR

Introductory Vignette Gold Standard Lab (GSL) Conducts an Internally-Valid RCT to Evaluate a Voter Turnout Program Well-Trained/Energetic Canvassers Visit Houses and Read Script Persuading Non-Voters to Vote in Minneapolis 90% of Treated Non-Voters Vote, Only 10% of Control Vote GSL Writes: “Exposure to the voter turnout script causes greater turnout.“ GSL Does Same RCT of Same Program in Atlanta Surprisingly, No Difference Between Treated and Control Perhaps Due to Low Trust and Bad Voting System in Atlanta vs. High Trust and Good Voting System in Minneapolis GSL Reruns RCT in Minneapolis with Inexperienced and Cheaper Canvassers This Time, No Difference Between Treated & Control

Argument & Outline We Argue Construct & External Validity Are Equally Important As Internal Validity for Causal Inference All Three Are Jointly Required – 3 Legs of a Stool Causal Inference Always Requires Semantic Communication About a Generalizable Claim Outline The Credibility Revolution 1. Conceptualize Validity and Causality 2. Essential Role of Construct Validity for Causal Inference 3. Essential Role of External Validity for Causal Inference 4.

The Credibility Revolution The Rise of Internally-Valid Quantitative Designs for Identifying “Causal Effects” Internal Validity: No Confounding of a Manipulated Cause A for Effect B in One Setting Potential Outcomes (Rubin) and Structured Causal Model (Pearl) Frameworks Lexical Priority on Internal Over Other Validities Internal Validity Has Dominated and Construct & External Have Been Subjugated in Past 30 Years Response & Critique of Prior Verbal Justifications Common in Applied Empirical Social Sciences Self-Validating Methodologies Eliminate Confoundedness and Displace Verbal Justifications

Validity and Causality Credibility Revolution Presumed Internal Validity ≈ Causal Inference Internal Validity Framed As Warrant “Local” or “Molar” or “Black Box” Causal Claim (Cook & Campbell) A Causal Claim is VALID if the Relationship In Fact Holds in the World Semantic Causal Relata = Actual Causal Relata Causal Inference: The Process of Generalizing from Evidence to Valid Claim VALIDATION Is the Epistemic Activity of Supporting the Claim With Evidence – Such That Claim is Warranted

Validity and Causality The Three Validities Correspond to Three Potential Failures in Causal Inference Internal: Fail Due to Unobserved Confounding Construct: Fail to Assign Correct Semantic Labels to Cause or/and Effect Such That Claim is False (High Quality Canvasser Was Treatment Not Script or Perhaps Turnout Misreported) External: Fail to Identify Contextual Factors Moderating Treatment Such That Claim is False (Minneapolis vs. Atlanta Trust & Voting System) Goal is to Infer a Valid Causal Generalization: Internal: Identify A Caused B IN C Construct: Correctly Label A & B External: Identify Conditions C Where A Causes B

Construct Validity Priority on Internal Validity -> Neglect of Measurement Measurement = Matter of Correlations (Cronbach) Internal Validity Advocates Assume Problem Away Exclusion Restriction: Assume Manipulation Has Effect ONLY from Exposure to the Actual Cause Stable Unit Treatment Value Assumption (SUTVA): Assume Treatment Each Unit Receives is Not Affected by Other Units Reality: Measures Bundle Active & Inert Ingredients Internal Validity Alone Abdicates How to Assign Labels One Doesn’t Even Know What One Is Talking About

Construct Validity Definition of CONSTRUCT VALIDITY: Correct Semantic Labeling of the Ontological Causal Relata BOTH A (Labeled Cause: Script) & B (Labeled Effect: Voting) Semantic Label Applies to Active NOT Inert Ingredient Active is the Necessary Component of a Sufficient Cause Script in Vignette Inert is (Hopefully) Unnecessary Component Training/Enthusiasm of Canvassers in Vignette Most Focus on Construct Validity of Outcome, But We Focus on Cause Here

Some Preliminaries for Formalization Active & Inert Ingredients of Cause (A), Effect (B), and Context (C) Formalization of Internally Valid Claim What You Claim What You Know

Construct Validity of Cause Only WEAK Construct in the Cause STRONG Construct in the Cause

External Validity External Validity Traditionally Defined As Generalizability and Extrapolation Across Settings Prioritizing Internal & Implicitly Vaguely Generalizing True External = Unreasonably High Threshold Pervasive Low “External Validity” RCTs & Experiments GSL Had Internal But Does Not Know Why Treatment Worked in 1 st Minneapolis But NOT in Atlanta Variation in Treatment Effects Across Settings Revels Causes Are Actually Treatment*Context Moderations Internal Validity = Very Limited Kind of Knowledge Only a Causal Effect for a Given Setting, and Only Knowable Retrospectively; Cartwright (2011): “it works somewhere.” Actually: It WorkED for Specific People Exposed to Specific Events in a Specific Time and Place Not “It Works Widely” or “It Will Work for Us” Internal Validity Rules Out Constant/Stable Characteristics of Settings Because Not Manipulated

External Validity Two Major Responses Don’t Really Solve Problem No Intention to Produce “Universal” Knowledge: Only Saying 1. A Causes B in C and Cannot Claim Beyond C Historicist’s Refuge: Still Do Not Understand Treatment • Effect Because Do Not Know What Contextual Factors Are Go Forth Across “Range of Settings” 2. Problematic “Simple Enumerative Induction” • Knowing What Defines “Range of Settings” Presumes • Knowledge of Relevant Contextual Factors Contextual Factors are Unknown Confounders Without a • Self-Validating Methodology to Identify “Transportability” (Pearl) Through Effect Moderation in DAG • Assumes You Know Contextual Factors “Structured Speculation” (Banerjee et al) Relies on Same • Imputed Verbal Justification Credibility Revolution Critiqued

External Validity CONTEXTUAL FACTORS: Non-Manipulated Conditions of a Setting That Augment or Undermine a Cause Cartwright: “Helping Factors” or “Countering Causes” Other Necessary Components to a Causal Effect or “Causal Field” In Which Treatment Takes Place (Oxygen -> Fire) Vignette: Trust & Voting Systems in Minneapolis vs. Atlanta Revised Definition of EXTERNAL VALIDITY: Identify How Contextual Factors Interact With Treatment to Produce Causal Effects That Generalize Across Settings Specify Which Contextual Factors Define Range of Settings Causal Inference Only Warranted With Accurate Claim of How to Generalize Causal Claim Across Settings (i.e. External Validity is Essential to Causal Inference)

External Validity WEAK External STRONG External

Conclusions Excesses & Trappings of the Credibility Revolution Revised Definitions: Construct: Correct Semantic Labeling of the Ontological Causal Relata; Active vs. Inert Ingredients External: Identify How Contextual Factors Interact With Treatment to Produce Causal Effects That Generalize Formalization Clarifies the Many Hidden Confounds Construct: Inert Ingredients External: Contextual Factors Construct & External Validity Are Equally Important As Internal Validity for Causal Inference Internal Validity Alone Cannot Warrant Causal Claim Advocate for Renewed Commitment to Construct & External Causal Inference Always Requires Semantic Communication About a Generalizable Claim

Bringing the Whole Formalization Together