What papers should be published? Relevance, plausibility, validity, - PowerPoint PPT Presentation

What papers should be published? Relevance, plausibility, validity, and learning What papers should be published? Relevance, plausibility, validity, and learning Alexander Frankel Maximilian Kasy November 20, 2017 1 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction Introduction ◮ Not all empirical results get published → selection. ◮ Reasons for selection: 1. Journal decisions (“publication bias”). 2. Researcher decisions (“p-hacking”). ◮ Possible motivations for selection: 1. Statistical significance testing. 2. Novelty of results. 3. Confirmation of priors. ◮ Consequences of selection: 1. Conventional estimators are biased. 2. Conventional confidence sets don’t control size. 2 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction What is to be done? ◮ Reform proposals to mitigate selection: ◮ Pre-registration plans ◮ Hypothesis registries ◮ “Data snooping” corrections ◮ Results-blind review ◮ Journal of replication studies ◮ Journal of null results ◮ We argue: No selection is not optimal, in general. ◮ Need to be careful about specifying objective ! 3 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction Possible journal objectives 1. Validity: ◮ Conventional estimators are unbiased. ◮ Conventional confidence sets control size. 2. Relevance: ◮ Published results inform policy. ◮ Publish to maximize social welfare. 3. Plausibility: ◮ Maximize probability that published results are correct. ◮ Minimize distance of published results to truth. 4. Learning: ◮ Minimize posterior variance, ◮ given the number of publications. 4 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction Preview of results ◮ Optimal selectivity depends on objective : ◮ Validity: Don’t select on findings. ◮ Relevance and learning: Publish surprising findings. ◮ Plausibility: Publish unsurprising findings. ◮ Relevance can rationalize selection based on one-sided or two-sided testing . ◮ Dynamic relevance can rationalize publication of precise null results. 5 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction Literature ◮ Systematic replication studies: Open Science Collaboration (2015), Camerer et al. (2016) ◮ Publication bias: Ioannidis (2005), Ioannidis (2008), McCrary et al. (2016); Andrews and Kasy (2017) ◮ Reform proposals: Olken (2015), Coffman and Niederle (2015), Christensen and Miguel (2016) ◮ Economic models of publication: Glaeser (2006), Libgober (2015), Akerlof and Michaillat (2017) 6 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction Outline of talk Introduction Static model Validity Relevance Plausibility Learning Dynamic relevance Discussion and conclusion 7 / 28

What papers should be published? Relevance, plausibility, validity, and learning Static model Static model Researcher submits Journal decides Policymaker chooses welfare is whether to publish policy realized X ∼ f X | θ → D = d ( X ) → A = a ( π 1 ) → u ( A , θ ) − D · c ◮ Common prior π 0 for θ of journal and policymaker. ◮ Posterior: f X | θ ( X |· ) π J = π 0 · Journal f X ( X ) π 1 = π 0 · f X | θ ( X |· ) Policymaker if D = 1 f X ( X ) π 0 = π 0 Naive policymaker if D = 0 π 0 = π 0 · 1 − E [ d ( X ) | θ = · ] Sophisticated policymaker if D = 0 1 − E [ d ( X )] 8 / 28

What papers should be published? Relevance, plausibility, validity, and learning Validity Validity Proposition Suppose X | θ ∼ N ( θ , s ) . The following statements are equivalent: 1. Bayesian validity of naive updating: θ | d ( X ) = 0 ∼ π 0 2. Frequentist unbiasedness: E [ X | θ , D = 1 ] = θ for all θ . 3. Publication probability independent of parameter: E [ d ( X ) | θ ] is constant in θ . 4. Publication decision independent of estimate: d ( X ) does not depend on X. 9 / 28

What papers should be published? Relevance, plausibility, validity, and learning Validity Proof ◮ 4 ⇒ 1, 2, 3: immediate. ◮ 1 ⇒ 3: θ | d ( X ) = 0 ∼ π 0 · 1 − E [ d ( X ) | θ = · ] . 1 − E [ d ( X )] Equality to π 0 is equivalent to E [ d ( X ) | θ ] = ¯ d for all θ . ◮ 2 ⇒ 3: Wlog s = 1. Unbiasedness equivalent to � 0 = ( z − θ ) ϕ ( z − θ ) d ( z ) dz � ϕ ′ ( z − θ ) d ( z ) dz = − � � � = ∂ θ ϕ ( z − θ ) d ( z ) dz = ∂ θ E [ d ( Z ) | θ ] . ◮ 3 ⇒ 4: completeness of X for θ in the normal location family ⇒ If E [ d ( X ) | θ ] = ¯ d for all θ , then d ( X ) = ¯ d almost surely. 10 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance Relevance ◮ Policymaker observes ( D , D · X ) , updates prior to π 1 picks policy A = a ( π 1 ) . ◮ Common objective of journal and policymaker: maximize expectation of welfare u ( A , θ ) , net of (shadow) cost of publication D · c . ◮ Expected welfare: � U ( a , π ) = u ( a , θ ) d π ( θ ) . ◮ Optimal policy choice: a ( π ) = argmax U ( a , π ) . a 11 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance The journal’s problem ◮ Denote a d = a ( π d ) for d = 0 , 1. ◮ Journal maximizes U ( a d , π J ) − d · c . ◮ Thus decides to publish iff U ( a 1 , π J ) − U ( a 0 , π J ) > c . ◮ Notes: ◮ This presumes no commitment: Journal takes policymaker’s beliefs π 0 as given when choosing D . ◮ Therefore takes a 0 as given. ◮ π 0 depend on E [ d ( X ) | θ ] for sophisticated updating! ◮ Also, since π 1 = π J , U ( a 1 , π J ) = max U ( a , π J ) . a 12 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance No commitment = commitment = planner’s solution ◮ Three related problems: 1. No commitment: Bayes Nash optimal d ( · ) . Take a 0 as given. 2. Commitment: Pick d ( · ) ex-ante. Take a 0 as function of d ( · ) . 3. Planner’s problem: Pick ex-ante both d ( · ) and a 0 . Proposition Assuming uniqueness, all three problems have the same solution. ◮ Journal and policymaker have same objective in choosing a 0 . ◮ Equivalence of 1. Joint optimization (planner’s problem) 2. Concentrating out (commitment problem) 3. Conditionally optimizing (no commitment problem) 13 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance Canonical policy problem 1: Binary treatment, linear welfare ◮ a ∈ { 0 , 1 } and u ( a , θ ) = a · θ . ◮ Expected welfare and optimal policy: U ( a , π ) = a · E π [ θ ] a ( π ) = 1 ( E π [ θ ] > 0 ) . ◮ Return to publishing (sophisticated updating):  sign ( E [ θ | X ]) = 0   U ( a ( π 1 ) , π J ) − U ( a ( π 0 ) , π J ) = sign ( E [ θ | d ( X ) = 0 ])  | E [ θ | X ] | else .  14 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance Canonical policy problem 2: Continuous policy, quadratic welfare ◮ a ∈ R and u ( a , θ ) = − ( a − θ ) 2 . ◮ Expected welfare and optimal policy: U ( a , π ) = − Var π ( θ ) − ( a − E π [ θ ]) 2 a ( π ) = E π [ θ ] . ◮ Return to publishing (sophisticated updating): U ( a ( π 1 ) , π J ) − U ( a ( π 0 ) , π J ) = ( E [ θ | d ( X ) = 0 ] − E [ θ | X ]) 2 15 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance Normal likelihood and prior ◮ Suppose X | θ ∼ N ( θ , s 2 ) and θ ∼ N ( µ 0 , σ 2 ) . σ 2 ◮ Then, for κ = s 2 + σ 2 , θ | X ∼ N ( κ X +( 1 − κ ) µ 0 , σ 2 0 · ( 1 − κ )) . ◮ Solutions to optimal publication problems: ◮ In the binary case: “ one-sided testing ,”  � � x > c − ( 1 − κ ) µ 0 1 µ 0 < 0  κ d R , b ( x ) = � � x < − c − ( 1 − κ ) µ 0 1 µ 0 > 0 .  κ ◮ In the quadratic case: “ two-sided testing ,” | x − µ 0 | > √ d R , c ( x ) = 1 � � c / κ . ◮ Same solutions for naive and sophisticated policymaker. 16 / 28

What papers should be published? Relevance, plausibility, validity, and learning Plausibility Plausibility ◮ Assume alternatively that journals don’t want to publish wrong results or estimates far from the truth. ◮ For instance: Choose d to maximize expectation of d · ( − ( X − θ ) 2 + b ) . ◮ ⇒ Publish iff E [( X − θ ) 2 | X ] = Var ( θ | X = x )+( E [ θ | X = x ] − x ) 2 < b . ◮ With X | θ ∼ N ( θ , s 2 ) and θ ∼ N ( µ 0 , σ 2 ) , √ � � b − σ 2 0 · ( 1 − κ ) d P ( x ) = 1 | x − µ 0 | < . 1 − κ ◮ → Only publish unsurprising findings! 17 / 28

What papers should be published? Relevance, plausibility, validity, and learning Learning Learning ◮ For any publication rule d ( · ) : what is the speed of learning? ◮ What is the expected posterior variance of θ ? E [ Var ( θ | D · X , D )] = Var ( θ ) − Var ( E [ θ | D · X , D ]) = Var ( θ ) − Var ( E [ θ | X ])+ Var ( E [ θ | X ] | D = 0 ) · E [ 1 − D ] , ◮ Thus: Given publication probability E [ D ] , higher speed of learning ⇔ smaller variance Var ( E [ θ | X ] | d ( X ) = 0 ) . 18 / 28