Epistemic Diversity and Editor Decisions: A Statistical Matthew Effect Remco Heesen 1 Jan-Willem Romeijn 2 1 Faculty of Philosophy University of Cambridge remcoheesen.eu 2 Faculty of Philosophy University of Groningen www.philos.rug.nl/˜romeyn/ Formal Models of Scientific Inquiry 18 July 2017
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References The Importance of Epistemic Diversity We sometimes want to maintain cognitive diversity even in instances where it would be reasonable for all to agree that one of two theories was inferior to its rival, and we may be grateful to the stubborn minority who continue to advocate problematic ideas. (Kitcher 1990, p. 7) The history of science has been and should be a history of competing research programmes (or, if you wish, ‘paradigms’), but it has not been and must not become a succession of periods of normal science: the sooner competition starts, the better for progress. (Lakatos 1978, p. 69) ◮ Example: Peptic ulcers are caused by acid (1954–1984) ◮ Epistemic/cognitive diversity versus social diversity Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 2 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References The Role of Journals in Epistemic Diversity ◮ Important role for journals: giving (or withholding) exposure ◮ Journals (editors/peer reviewers) should promote epistemic diversity ◮ Bias in favor of monoculture is detrimental to progress Image source: www.sun.ac.za Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 3 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Editorial Biases ◮ Editors’ cognitive biases may favor established research program ◮ Confirmation bias ◮ Anchoring ◮ Editors may favor established research program due to risk aversion Image source: http://sexmahoney.blogspot.co.uk Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 4 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Statistical Biases in Peer Review Our claim: ◮ Suppose editor selects only for quality ◮ “Strictly statistical” biases in peer review ◮ Favor established research programs Image source: www.blachford.com Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 5 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References A Statistical Matthew Effect We call this a statistical Matthew effect (Merton 1968) Our argument: ◮ Information asymmetries bias “Bayesian” editor ◮ Latent quality differences bias “frequentist” editor Image source: http://theliteracywiki.wikispaces.com Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 6 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Outline Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 7 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Outline Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 8 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Quality and Uncertainty ◮ Submitted paper has latent quality q ◮ Identity of author is relevant to quality ◮ Editor’s prior for known author: π ( q | K ) ◮ Editor’s prior for unknown author: π ( q ) ◮ Distribution of quality is the same for research programs H and L ◮ Research program of author is irrelevant to quality: ◮ π ( q | K , H ) = π ( q | K , L ) and π ( q | H ) = π ( q | L ) ◮ But authors from program H more likely to be known ◮ Editor may belong to program H Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 9 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Peer Review ◮ Editor solicits reviews ◮ Reviewer report R independent of research program and identity of author (given q ) ◮ Editor updates beliefs about q ◮ Posterior for known author: π ( q | K , R ) ◮ Posterior for unknown author: π ( q | R ) Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 10 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Acceptance and Utility ◮ Editor must accept ( A ) or reject ( ¬ A ) submission ◮ Editor selects only for quality ◮ Utility of acceptance equals quality q ◮ Utility of rejection is some fixed value q ∗ = ⇒ Editor accepts if and only if posterior mean exceeds q ∗ Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 11 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Example: Normal Distributions ◮ Suppose normal distributions (Heesen forthcoming) ◮ Submissions from program H accepted at a higher rate ◮ Pr( A | H ) > Pr( A | L ) ◮ Among accepted papers, higher average quality for program H ◮ E [ q | A , H ] > E [ q | A , L ] ◮ Despite equal quality distributions, papers from program H receive more exposure and are seen to be better ◮ A statistical Matthew effect occurs Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 12 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References The General Case ◮ If knowing the author affects decision with positive probability: ◮ Higher acceptance rate or higher average quality ◮ Pr( A | H ) > Pr( A | L ) or E [ q | A , H ] > E [ q | A , L ] ◮ Proof uses Good (1967) ◮ Dilemma for the editor: decision procedure benefits program H either way ◮ A statistical Matthew effect occurs Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 13 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Discussion ◮ Due to information asymmetry, editor treats programs differently ◮ Justified? ◮ Maximum use of information given goal of selecting for quality ◮ But: epistemic diversity suffers ◮ How to prevent this? ◮ Suggestion: role of editor’s prior is unjustified ◮ Use more “frequentist” peer review process Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 14 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Outline Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 15 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References A Different Model of Peer Review ◮ As before, reviewer report R independent of research program given q ◮ Editor must accept ( A ) or reject ( ¬ A ) submission ◮ Editor accepts if and only if reviewer report exceeds q ∗ ◮ Editor still selects only for quality ◮ But no role for prior: no bias due to information asymmetry Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 16 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Latent Quality Differences ◮ In this model, problems arise from latent quality differences ◮ Plausibly, established program produces higher quality on average ◮ Novel program may have startup problems ◮ Editor need not be assumed to know this ◮ Quality follows a log-concave distribution in both programs ◮ f ( tq + (1 − t ) q ′ ) ≥ f ( q ) t f ( q ′ ) 1 − t ◮ E.g., normal, uniform, exponential, gamma ◮ Average quality in program H higher than in program L ◮ µ H > µ L ◮ No distributional assumption on R except: conditional probability of acceptance increasing in q ◮ Pr( R > q ∗ | q ) increasing in q Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 17 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Suitability and Acceptance ◮ A submission is suitable ( S ) if its quality q exceeds threshold t ◮ Peer review works better for the established program: ◮ A greater proportion of accepted papers is suitable, and suitable papers are accepted at a higher rate ◮ Pr( S | A , H ) > Pr( S | A , L ) ◮ Pr( A | S , H ) ≥ Pr( A | S , L ) (strict unless quality distribution is exponential) ◮ Proof generalizes Borsboom et al. (2008) ◮ Generalization also considers different variances Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 18 / 28
Diversity and Bias Information Asymmetry Latent Quality Differences What Can Be Done? References Why Does This Happen? Remco Heesen and Jan-Willem Romeijn Epistemic Diversity and Editor Decisions 18 July 2017 19 / 28
Recommend
More recommend
