Replication, Preregistration & Open Science Why most published - PowerPoint PPT Presentation

Replication, Preregistration & Open Science Why most published research findings are false

select random card 1 / 3 1 / 3 1 / 3 1 2 3 choose side choose side choose side 1 0 1 / 2 1 / 2 0 1 b r b r b r 0 0 1 / 3 1 / 6 1 / 6 1 / 3

P ( Card = i ) P ( Obs = j | Card = i ) P ( Obs = j | Card = i ) P ( Card = i ) P ( Card = 1 | Obs = b ) = P ( Obs = b | Card = 1) P ( Card = 1) P ( Obs = b ) P ( Obs = b | Card = 1) P ( Card = 1) = P i P ( Obs = b | Card = i ) P ( Card = i ) 1 = 2 3 = 1 3 2

hypothesis H R / R+1 1 / R+1 P ( H ) true false apply test T apply test T P ( T | H ) 1 − β β 1 − α α sign null sign null Positive predictive value P ( H = t | T = s ) = P ( T = s | H = t ) P ( H = t ) P ( T = s ) R (1 − β ) = “probability that the hypothesis is true, R (1 − β ) + α given a significant test result”

Positive predictive value P ( H = t | T = s ) = P ( T = s | H = t ) P ( H = t ) P ( T = s ) R (1 − β ) = “probability that the hypothesis is true, R (1 − β ) + α given a significant test result” example: P ( H = t | T = s ) = 0 . 8 R = 1 , β = 0 . 2 , α = 0 . 05 0 . 85 ≈ 0 . 94

hypothesis H R / R+1 1 / R+1 true false apply test T apply test T β 1 − β 1 − α α sign null sign null p -hacking ::: combination of design/presentation/analysis p -hack p -hack factors that favor a significant test result beyond the normal 1 − u u 1 − u u alpha level sign null sign null

Positive predictive value p -hacking ::: combination of R (1 − β ) + u β R P ( H = t | T = s ) = design/presentation/analysis R (1 − β ) + u β R + α + u (1 − α ) factors that favor a significant test result beyond the normal alpha level example: R = 1 , β = 0 . 2 , α = 0 . 05

hypothesis H R / R+1 1 / R+1 true false p -fishing ::: reporting at least apply test T apply test T one significant test results 1 − (1 − α ) n β n (1 − α ) n 1 − β n from n (equally powered) studies sign null sign null

Positive predictive value R (1 − β n ) p -fishing ::: reporting at least P ( H = t | T = s ) = one significant test results R + 1 − (1 − α ) n − R β n from n (equally powered) studies example: R = 1 , β = 0 . 2 , α = 0 . 05