Statistical Analysis of Endorsement Experiments: Measuring Support for Militant Groups in Pakistan Kosuke Imai Department of Politics Princeton University Joint work with Will Bullock and Jacob Shapiro May 13, 2011 Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 1 / 24
Motivation Survey is used widely in social sciences Validity of survey depends on the accuracy of self-reports Sensitive questions = ⇒ social desirability, privacy concerns e.g., racial prejudice, corruptions Lies and nonresponses How can we elicit truthful answers to sensitive questions? Survey methodology: protect privacy through indirect questioning Statistical methodology: efficiently recover underlying responses Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 2 / 24
Survey Methodologies for Sensitive Questions Randomized Response Technique Most extensively studied Use randomization to protect privacy Difficulties: logistics, lack of understanding among respondents List Experiments (item count technique) Use aggregation to protect privacy New multivariate regression analysis method New methods to detect and correct failures (joint with G. Blair) Difficulties: design effects, ceiling and floor effects Endorsement Experiments Use randomized endorsements to measure support levels Develop a measurement model based on item response theory Difficulties: interpretation, need for modeling Applications: Pakistanis’ support for Islamic militant groups 1 Afghanis’ support for Taliban and ISAF (joint with J. Lyall) 2 Nigerians’ support for insurgents (joint with G. Blair) 3 Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 3 / 24
Endorsement Experiments Measuring support for political actors (e.g., candidates, parties) when studying sensitive questions Ask respondents to rate their support for a set of policies endorsed by randomly assigned political actors Experimental design: Select policy questions 1 Randomly divide sample into control and treatment groups 2 Across respondents (and questions), randomly assign political 3 actors for endorsement (no endorsement for the control group) Compare support level for each policy endorsed by different actors 4 Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 4 / 24
The Pakistani Survey Experiment 6,000 person urban-rural sample in April 2009 Four militant groups: Pakistani militants fighting in Kashmir (a.k.a. Kashmiri tanzeem) Militants fighting in Afghanistan (a.k.a. Afghan Taliban) Al-Qa’ida Firqavarana Tanzeems (a.k.a. sectarian militias) Four policies: WHO plan to provide universal polio vaccination across Pakistan Curriculum reform for religious schools Reform of FCR to make Tribal areas equal to rest of the country Peace jirgas to resolve disputes over Afghan border (Durand Line) Response rate over 90% Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 5 / 24
Endorsement Experiment Questions: Example The script for the control group The World Health Organization recently announced a plan to introduce universal Polio vaccination across Pakistan. How much do you support such a plan? (1) A great deal (2) A lot (3) A moderate amount (4) A little (5) Not at all The script for a treatment group The World Health Organization recently announced a plan to introduce universal Polio vaccination across Pakistan, a policy that has received support from Al-Qa’ida. How much do you support such a plan? (1) A great deal (2) A lot (3) A moderate amount (4) A little (5) Not at all Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 6 / 24
Distribution of Responses Polio Vaccinations Curriculum Reform FCR Reforms Durand Line Firqavarana Tanzeems Al−Qaida Punjab Afghan Taliban Pakistani militant groups in Kashmir Control Group Firqavarana Tanzeems Al−Qaida Sindh Afghan Taliban Pakistani militant groups in Kashmir Control Group Firqavarana Tanzeems Al−Qaida NWFP Afghan Taliban Pakistani militant groups in Kashmir Control Group Firqavarana Tanzeems Balochistan Al−Qaida Afghan Taliban Pakistani militant groups in Kashmir Control Group 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Not At A Moderate A Great A Little A Lot All Amount Deal Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 7 / 24
Methodological Challenges and Proposed Solutions How to combine responses from multiple questions? 1 ⇒ item response theory = How to recoup loss of statistical efficiency? 2 ⇒ hierarchical modeling = How to interpret the “support”? 3 = ⇒ policy vs. valence How to select policy questions? 4 Policies should belong to a single dimension Respondents should not have strong views Should one use well-known policies?: Statistical and substantive tradeoffs Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 8 / 24
Endorsement Experiments Framework N respondents J policy questions K political actors Y ij ∈ { 0 , 1 } : response of respondent i to policy question j T ij ∈ { 0 , 1 , . . . , K } : political actor randomly assigned to endorse policy j for respondent i Y ij ( t ) : potential response given the endorsement by actor t Covariates measured prior to the treatment Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 9 / 24
The Proposed Model Quadratic random utility model (Clinton, Jackman, and Rivers): ijk ) − ζ j 1 � 2 + η ij −� ( x i + s ∗ U i ( ζ j 1 , k ) = ijk ) − ζ j 0 � 2 + ν ij −� ( x i + s ∗ U i ( ζ j 0 , k ) = x i is the ideal point and s ∗ ijk is the “influence” of endorsement The statistical model (item response theory): Pr ( Y ij = 1 | T ij = k ) = Pr ( Y ij ( k ) = 1 ) = Pr ( U i ( ζ j 1 , k ) > U i ( ζ j 0 , k )) Pr ( α j + β j ( x i + s ∗ = ijk ) > ǫ ij ) Support level: greater support ⇐ ⇒ greater prob. of Y ij = 1 � s ∗ if β j ≥ 0 ijk s ijk = − s ∗ otherwise ijk Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 10 / 24
The Proposed Model (Continued) Hierarchical modeling: indep . i δ, σ 2 N ( Z ⊤ ∼ x ) x i indep . N ( Z ⊤ i λ jk , ω 2 s ijk ∼ jk ) i . i . d . ∼ N ( θ k , Φ k ) λ jk “Noninformative” hyper prior on ( α j , β j , δ, θ k , ω 2 jk , Φ k ) Interpretation: spacial model vs. factor analysis policy vs. valence aspects of support Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 11 / 24
Quantities of Interest and Model Fitting Average support level for each militant group k Z ⊤ τ jk ( Z i ) = i λ jk for each policy j Z ⊤ κ k ( Z i ) = i θ k averaging over all policies Standardize them by dividing the (posterior) standard deviation of ideal points Bayesian Markov chain Monte Carlo algorithm Multiple chains to monitor convergence Implementation via JAGS (Plummer) Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 12 / 24
Model for the Division Level Support Ordered response with an intercept α jl varying across divisions The model specification: indep . x i ∼ N ( δ division [ i ] , 1 ) indep . N ( λ k , division [ i ] , ω 2 ∼ k ) s ijk indep . N ( µ province [ i ] , σ 2 δ division [ i ] ∼ province [ i ] ) indep . ∼ N ( θ k , province [ i ] , Φ k , province [ i ] ) λ k , division [ i ] Averaging over policies Partial pooling across divisions within each province Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 13 / 24
Estimated Division Level Support Standardized Level of Support −1.0 −0.5 0.0 0.5 1.0 Kosuke Imai (Princeton) Bahawalpur n=118 Dera Ghazi Khan n=0 Faisalabad n=313 Gujranwala n=403 Punjab Lahore n=579 Multan n=495 Rawalpindi n=208 Sargodha n=131 Hyderabad n=203 Endorsement Experiments Karachi n=473 Sindh Larkana n=311 Mirpurkhas n=0 Sukkur n=293 Bannu n=0 Dera Ismail Khan n=84 Hazara n=287 NWFP Kohat n=50 Malakand n=0 Mardan n=215 Firqavarana Tanzeems Al−Qaida Militants fighting in Afghanistan Pakistani militant groups in Kashmir Peshawar n=288 NEMP (NYU) Kalat n=103 Makran n=0 Balochistan Nasirabad n=210 Quetta n=320 14 / 24 Sibi n=67 Zhob n=61
Model with Individual Covariates Ordered response with an intercept α jl varying across divisions The model specification: indep . i δ Z , 1 ) N ( δ division [ i ] + Z ⊤ ∼ x i indep . N ( λ k , division [ i ] + Z ⊤ i λ Z k , ω 2 s ijk ∼ k ) indep . N ( µ province [ i ] , σ 2 δ division [ i ] ∼ province [ i ] ) indep . ∼ N ( θ k , province [ i ] , Φ k , province [ i ] ) λ k , division [ i ] Expands upon the division level model to include individual level covariates: gender, urban/rural, education, income Individual level covariate effects after accounting for differences across divisions Poststratification on these covariates using the census Kosuke Imai (Princeton) Endorsement Experiments NEMP (NYU) 15 / 24
Recommend
More recommend
