Is our knowledge of ecology becoming more precise and accurate over - PowerPoint PPT Presentation

Is our knowledge of ecology becoming more precise and accurate over time? A meta-analysis of meta-analyses Jeremy Fox University of Calgary Lab: foxlabcalgary.wordpress.com Blog: dynamicecology.wordpress.com

The big vague motivating question Is progress in ecology more Or this? like this?

The “decline effect”: effect size estimates decrease in magnitude over time Autistic vs. non-autistic neurology Implicit bias (Rødgaard et al. 2019): (Charlesworth & Banaji 2019): Ego depletion (Vidallo 2019):

The decline effect in ecology and evolution Status signaling in male house sparrows EEB effect sizes estimates typically are (Sánchez-Tojár et al. 2018) negatively correlated with publication year (Jenions & Moller 2001)

Why might we expect decline effects? Because unusual things get noticed. ZSL

The much more specific questions that I’ll actually answer 1. Do effect size estimates tend to decline in magnitude as more studies are published? 2. Does the precision of the estimated mean effect size typically improve as more studies are published? 3. Is sampling error or effect size estimates declining? 4. How much heterogeneity is there among ecological effect size estimates, and how does it change as more studies are conducted?

Methods overview: meta-analysis of meta-analyses • Meta-analysis: statistically summarizes different estimates of the “same” effect • Various measures of effect size • Hedge’s d , log(response ratio), etc. • Cumulative random effects meta-analysis • Random effects: variation in effect size among studies (papers), within studies, and due to sampling error • Cumulative: add in one study at a time, in order of publication, recalculate the meta-analysis • Look for patterns across the meta-analyses

1. Are decline effects widespread?

No widespread decline effects r =0.07 25 2000 20 Absolute value of weighted effect size 1500 15 Frequency 1000 10 500 5 0 0 1995 2000 2005 2010 2015 -1.0 -0.5 0.0 0.5 1.0 Study year Corr(abs(weighted effect size), study year) • Same results whether or not you weight by the inverse of the sampling variance • Same results when using untransformed effect sizes instead of absolute values • The extreme correlations are mostly from meta-analyses with very few studies • See the backup slides for additional analyses

2. Do our estimates of mean effect size get more precise as more studies are published?

Precision of the cumulative mean effect size usually (not always) increases over time 30 25 • In 13% of meta-analyses, precision 20 decreases on avg. as more studies Frequency are performed! 15 • Often due to increasing among- study heterogeneity 10 5 0 -1.0 -0.5 0.0 0.5 1.0 Cor(95% c.i. width, final study year)

3. Does sampling error tend to decline over time?

No general trend for sampling error to decrease (or increase) over time 50 Sampling variance of effect size 8 r =0.07 40 6 30 Frequency 4 20 2 10 0 0 1995 2000 2005 2010 2015 -1.0 -0.5 0.0 0.5 1.0 Study year Corr(sampling variance, study year)

4. How large are among- and within-study heterogeneity? And how do they change as more studies are conducted? • Typically ~85% of the variance in effect size • confirms Senior et al. 2016 • Heterogeneity becomes an increasingly important source of variation as more studies are conducted

Summary • Don’t worry about the decline effect • Instead, worry about heterogeneity: • Within and among studies • Often keeps precision low, can prevent it from improving much as more studies are conducted • Anecdotally, we usually can’t explain much of it with moderator variables

Ecologists often study small effects (and if they don’t look small, it’s often because they’re estimated imprecisely) 4 30 25 3 20 se(mean effect size) Frequency 15 2 10 1 5 0 0 -2 0 2 4 6 -2 0 2 4 6 Mean effect size Mean effect size • 59% of mean effect sizes are significantly different from 0

A slight excess of both the decline effect, and its opposite? Real data Permuted data 25 40 20 30 15 Frequency Frequency 20 10 10 5 0 0 -1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0 Corr(abs(weighted effect size), study year) Corr(abs(weighted effect size), permuted study year)

Early studies often are not representative of later studies Only 71% of final means fall within the 95% c.i. from the first two studies 1.5 40 Mean effect size (95% c.i.) 1.0 30 Estimated mean effect size from Frequency first two studies, ± 1.96 s.e. 0.5 20 10 0.0 Final mean effect size estimate is 1.07 s.e. below the estimate from 0 the first two studies ( z =-1.07) -0.5 -10 -5 0 5 10 0 10 20 30 40 z score of final mean effect size Years since first study • Probably because no two studies are representative of the others!

Slight excess of both increases and decreases in sampling error over time? 50 50 40 40 30 30 Frequency Frequency 20 20 10 10 0 0 -1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0 Corr(sampling variance, study year) Corr(sampling variance, permuted study year)

How does precision of the estimated mean effect size change over time as more studies are conducted? Cumulative mean effect size 1.0 1.8 r =-0.95 1.6 95% c.i. width 0.5 1.4 (95% c.i.) 1.2 0.0 1.0 -0.5 0.8 0.6 -1.0 1985 1990 1995 2000 2005 2010 2015 10 15 20 25 30 35 Years since first study Final year Cumulative mean ean effect 0.40 1.0 r =0.71 0.35 0.8 95% c.i. width size (95% c.i.) 0.30 0.6 0.25 0.4 0.20 0.15 0.2 0.10 0.0 4 6 8 10 12 14 2006 2008 2010 2012 2014 Years since first study Final year

Variation in effect size estimates is mostly among- and within-study heterogeneity, not sampling error 50 40 Frequency 30 20 10 0 0 20 40 60 80 100 Total heterogeneity (%) • Confirms Senior et al. 2016 • Heterogeneity increases with # of studies in the meta-analysis

Heterogeneity becomes a more important source of variation in effect size as more studies are conducted 100 80 Total heterogeneity (%) 60 40 20 0 0 100 200 300 Studies • In 80% of meta-analyses, % of variation due to heterogeneity increases over time as more studies are conducted

How can we address heterogeneity? • Lower our expectations? • Maybe we should be happy to explain any heterogeneity? • Ask narrower questions? • Probably not • Narrower question  fewer studies  reduced precision • If you don’t know the sources of heterogeneity, you don’t know which narrow question to ask • Narrow questions often of narrow interest • Distributed experiments? • Learn to love heterogeneity? • Increasing heterogeneity = diversifying our study subjects? • Seek other sorts of generalizations besides those provided by meta-analyses • Fox (in press), Philosophical Topics