Really? Using the nullabor package to learn if what we see is - PowerPoint PPT Presentation

Really? Using the nullabor package to learn if what we see is really there Di Cook, Monash University Joint with Hadley Wickham, Heike Hofmann, Niladri Roy Chowdhury, Mahbub Majumder Photo by Lyn Cook

Outline Why? lineup, rorschach functions null generating mechanisms p-values metrics WOMBAT 2016, Melbourne, Australia 2 − 20

“Biomass really looks related to nitrogen deposition, but none of my tests show a significant relationship!” Ecologist colleague 400 800 biomass g m-1 200 600 400 800 200 600 12 3 6 9 12 9 12 3 6 9 12 3 N deposition g m-1 y-1

“These four species of wasps have very different gene expression patterns” Published paper 2010

“Is it possible that the pollsters are systematically biased?” Our US election monitoring

Why inference? Plots of data allow us to uncover the unexpected, but it needs to be calibrated against what might be seen by chance, if there really is no underlying pattern Classical statistical inference allows computing probabilities of this being a likely value of a statistic if there really is no structure WOMBAT 2016, Melbourne, Australia 6 − 20

Inference Once you see it, its too late You cannot legitimately test for significance of structure WOMBAT 2016, Melbourne, Australia 7 − 20

nullabor Lineup protocol: Plots your data among a field of “null” plots Puts it in the context of what it might look like if there is really no structure Encrypts the location of the data plot WOMBAT 2016, Melbourne, Australia 8 − 20

1 2 3 4 5 800 600 400 200 6 7 8 9 10 800 600 400 biomass g m-1 200 11 12 13 14 15 800 600 400 200 16 17 18 19 20 800 600 400 200 > decrypt("fg0t DARA up iYzuRuYp Fl") 3 6 9 12 3 6 9 12 3 6 9 12 3 6 9 12 3 6 9 12 N deposition g m-1 y-1 [1] "True data in position 10"

1 2 3 4 5 ● ● ● ● ● ● 4 ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● − 2 ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 4 ● ● ● ● ● ● ● ● ● ● ● − 6 ● 6 7 8 9 10 ● ● ● ● 4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● 2 ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Group ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● − 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 4 ● ●● ● ● ● F ●● ● ● ● ● ● ● − 6 ● LD2 G ● 11 12 13 14 15 ● ● ● ● Q ● ● ● ● ● ● ● ● 4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● W ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 2 ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 6 16 17 18 19 20 ● ● ● ● ● ● ● ● ● 4 ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● − 6 ● − 10 − 5 0 5 − 10 − 5 0 5 − 10 − 5 0 5 − 10 − 5 0 5 − 10 − 5 0 5 LD1

1 2 3 4 5 ● ● ● ● ● 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ● 0 ● − 5 ● ● ● ● ● ● ● 6 7 8 9 10 ● ● ● ● ● 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 0 Difference in % − 5 ● ● ● ● ● ● 11 12 13 14 15 ● ● ● ● ● 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 0 ● − 5 ● ● ● 16 17 18 19 20 ● ● ● ● ● 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 0 > decrypt("fg0t DARA up iYzuRuYp Q") − 5 ● ● ● ● ● RasmussenGallup Other Fox RasmussenGallup Other Fox RasmussenGallup Other Fox RasmussenGallup Other Fox RasmussenGallup Other Fox [1] "True data in position 5"

nullabor functions lineup: Generates a lineup using one of the given null generating mechanisms pvisuals: Compute p-values after showing to impartial jurers distmet: empirical distribution of distance between data plot and null plots WOMBAT 2016, Melbourne, Australia 12 − 20

Demo WOMBAT 2016, Melbourne, Australia 13 − 20

Table 1: Comparison of visual inference with traditional hypothesis testing. Mathematical Inference Visual Inference H 0 : µ 1 = µ 2 vs H a : µ 1 � = µ 2 H 0 : µ 1 = µ 2 vs H a : µ 1 � = µ 2 Hypothesis 200 Conc (mg/kg) label 150 y 1 − ¯ ¯ y 2 site A T ( y ) = T ( y ) = Test Statistic 100 site B q n 1 + 1 1 s 50 n 2 site A site B Site 1 2 3 4 5 200 150 100 50 6 7 8 9 10 200 150 100 Conc (mg/kg) 50 label f T ( y ) ( t ); f T ( y ) ( t ); Sampling Distribution site A 11 12 13 14 15 site B 200 150 100 50 ! t n ! 1 ( ! 2 ) 0 t n ! 1 ( ! 2 ) 16 17 18 19 20 200 150 100 50 site A site B site A site B site A site B site A site B site A site B Site Reject H 0 if observed T is extreme observed plot is identifiable