bets the dangers of selection bias in early analyses of
play

BETS: The dangers of selection bias in early analyses of the - PowerPoint PPT Presentation

Feb 03, 2023 •211 likes •629 views

BETS: The dangers of selection bias in early analyses of the coronavirus disease (COVID-19) pandemic Qingyuan Zhao Statistical Laboratory, University of Cambridge September 24, 2020 @ Ohio State University Manuscript: arXiv:2004.07743

  1. BETS: The dangers of selection bias in early analyses of the coronavirus disease (COVID-19) pandemic Qingyuan Zhao Statistical Laboratory, University of Cambridge September 24, 2020 @ Ohio State University Manuscript: arXiv:2004.07743 (forthcoming in The Annals of Applied Statistics ) Slides: http://www.statslab.cam.ac.uk/~qz280/ .

  2. Collaborators Nianqiao (Phyllis) Ju Sergio Bacallado Rajen Shah PhD student at Harvard Stats Lab, Cambridge Stats Lab, Cambridge And many thanks to... Cindy Chen, Yang Chen, Yunjin Choi, Hera He, Michael Levy, Marc Lipsitch, James Robins, Andrew Rosenfeld, Dylan Small, Yachong Yang, Zilu Zhou, and many other who have provided helpful suggestions. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 1 / 49

  3. COVID-19 is personal for everyone Me and my parents, all grew up in in Wuhan, China. (September 7, 2019) Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 2 / 49

  4. Wuhan Lockdown (January 23, 2020) Before the lockdown After the lockdown Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 3 / 49

  5. The beginning of this project On January 29, I heard from my parents that a close relative was just diagnosed with “viral pneumonia”. This prompted me to start looking into the data available at the time. However, epidemiological data from Wuhan are very unreliable! Some anecdotal evidence Inadequate testing: The relative of mine could not get a RT-PCR test till mid-February, when she was already recovering. False negative test: Her first test was negative. A few days later she was tested again and the result came back positive. Insufficient contact tracing: Her husband who also showed COVID symptoms quickly recovered and was never tested. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 4 / 49

  6. Insufficient testing in Wuhan A change of diagnostic criterion on February 12 led to a huge spike of cases. Solution: Using cases “exported” from Wuhan This has two benefits: Testing and contact tracing were intensive in other locations. 1 Detailed case reports (instead of mere case counts) are often available. 2 This design was first used by Neil Ferguson’s team in Imperial College, who estimated on January 17 that there might be already over 1,700 cases in Wuhan. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 5 / 49

  7. Our first analysis Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 6 / 49

  8. A puzzling comparison Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 7 / 49

  9. Which one is correct? 1,000,000 United States United States Italy Spain France United Kingdom 10,000 Spain Italy Germany France United Kingdom 100,000 Iran Belgium Iran Turkey Germany Netherlands Belgium Canada Netherlands Brazil Switzerland Brazil Turkey Russia Portugal Sweden Total deaths Total cases Austria 1,000 Canada Switzerland Israel India Ireland Sweden Peru South Korea 10,000 Portugal Japan Ecuador Chile Poland Romania Norway Indonesia Czech Republic Denmark Australia Pakistan MexicoAustria Ireland Mexico Saudi Arabia India Philippines Malaysia Romania Ecuador United Arab Emirates Indonesia Algeria Philippines Denmark Serbia Poland Panama Qatar Belarus Dominican Republic UkraineLuxembourg Finland Singapore Peru South Korea Colombia Thailand Dominican Republic South Africa Argentina Egypt Russia Egypt Greece Czech Republic Algeria Moldova Morocco Hungary Croatia Iceland Colombia Morocco Norway Hungary Bahrain Israel Japan Estonia Iraq Kuwait Pakistan Argentina Kazakhstan Ukraine Greece Uzbekistan Armenia Azerbaijan Slovenia 100 Chile Panama Bosnia and Herzegovina New Zealand Lithuania Bangladesh Serbia MalaysiaIraq 1,000 Saudi Arabia Luxembourg Finland Australia Slovenia Singapore 100 10 0 20 40 60 0 20 40 Days since 100 cases Days since 10 deaths In countries most hard hit by COVID-19, the total cases and deaths grew about 100 times in the first 20 days (doubling time: 20 / log 2 (100) = 3 . 01 days). Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 8 / 49

  10. How can the results be so different? Spoilers... Similar data and model were used in these two studies, with one crucial difference: The Lancet study did not take into account the travel ban. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 9 / 49

  11. Rest of the talk Overview of selection bias 1 Dataset 2 Model 3 Why some early analyses were severely biased? 4 Conclusions 5 Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 10 / 49

  12. Bias (i): Under-ascertainment This may occur if symptomatic patients did not seek healthcare or could not be diagnosed. Susceptible studies: All studies using cases confirmed when testing is insufficient. Direction of bias: Varied, depending on the pattern of under-ascertainment and parameter of interest. Solution: Use carefully considered and planned study designs. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 12 / 49

  13. Bias (ii): Non-random sample selection Cases included in the study are not representative of the population. Susceptible studies: All studies, as detailed information of COVID-19 cases is sparse, but especially those without clear inclusion criteria. Direction of bias: Varied. Solution: Follow a protocol for data collection with a clearly defined sample inclusion criterion. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 13 / 49

  14. Bias (iii): Travel ban Outbound travel from Wuhan was banned from January 23, 2020 to April 8, 2020. Susceptible studies: Studies that analyze cases exported from Wuhan. Direction of bias: Under-estimation of epidemic growth and infection-to-recovery time. Solution: Derive tailored likelihood functions to account for travel restrictions. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 14 / 49

  15. Bias (iv): Epidemic growth Patients were more likely to be infected towards the end of their exposure period. Susceptible studies: Studies that treat infections as uniformly distributed over the exposure period. Direction of bias: Over-estimation of the incubation period. Solution: Derive tailored likelihood functions to account for epidemic growth. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 15 / 49

  16. Bias (v): Right-truncation Cases confirmed after a certain time are excluded from the dataset. Susceptible studies: Studies that only use cases detected early in an epidemic. Direction of bias: Under-estimation of the incubation period. Solution: Collect all cases that meet a selection criterion, do not end data collection 1 prematurely; Derive tailored likelihood functions to correct for right-truncation. 2 Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 16 / 49

  17. Recap Types of bias in COVID-19 analyses (i) Under-ascertainment. (ii) Non-random sample selection. (iii) Travel ban. (iv) Epidemic growth. (v) Right-truncation. Keys to avoid the selection bias Carefully design the study and adhere to the sample inclusion criterion. 1 Start from a generative model and derive likelihood functions that adjust for 2 sample selection. Qingyuan Zhao (Stats Lab, Cambridge) BETS on COVID-19 Sep 14, 2020 17 / 49

Recommend

BETS: The dangers of selection bias in early analyses of the coronavirus disease (COVID-19)

BETS: The dangers of selection bias in early analyses of the coronavirus disease (COVID-19)

BETS: The dangers of selection bias in early analyses of the coronavirus disease (COVID-19) pandemic Qingyuan Zhao Statistical Laboratory, University of Cambridge May 5, 2020 @ YSPH Biostatistics Seminar Manuscript: arXiv:2004.07743 Slides:

555 views • 43 slides
Making Generative Classifiers Robust to Selection Bias Andrew Smith Charles Elkan November

Making Generative Classifiers Robust to Selection Bias Andrew Smith Charles Elkan November

Making Generative Classifiers Robust to Selection Bias Andrew Smith Charles Elkan November 30th, 2007 Outline What is selection bias? Types of selection bias. Overcoming learnable bias with weighting. Overcoming bias with

736 views • 59 slides
Review Selection bias, overfitting Bias v. variance v. residual Bias-variance tradeoff

Review Selection bias, overfitting Bias v. variance v. residual Bias-variance tradeoff

Review Selection bias, overfitting Bias v. variance v. residual Bias-variance tradeoff 1 n=1 Cramr-Rao bound n=4 0.8 n=30 0.6 CDF of max of n samples of 0.4 N( =2, 2 =1) 0.2 [representing error estimates for n

517 views • 28 slides
Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection

Variable Selection Variable Selection Variable selection bias Bias in Ensemble Bias in Ensemble Methods Methods Variable selection Variable selection Variable Selection Bias in Classification bias bias rpart Trees and Ensemble Methods

183 views • 6 slides
Early SUSY analyses with ATLAS Giacomo Polesello INFN, Sezione di Pavia Early analyses at the

Early SUSY analyses with ATLAS Giacomo Polesello INFN, Sezione di Pavia Early analyses at the

. Early SUSY analyses with ATLAS Giacomo Polesello INFN, Sezione di Pavia Early analyses at the LHC The LHC will start producing high-energy collisions in the next months Large uncertainties on the data-taking parameters: Energy (ies)

634 views • 40 slides
Using Cost-Benefit Analyses to Promote Using Cost-Benefit Analyses to Promote y y the Early

Using Cost-Benefit Analyses to Promote Using Cost-Benefit Analyses to Promote y y the Early

Using Cost-Benefit Analyses to Promote Using Cost-Benefit Analyses to Promote y y the Early Childhood Agenda the Early Childhood Agenda Rob Bradham Senior Vice President of Business Development p and Government Affairs Greater Richmond

521 views • 16 slides
Dose selection Dose selection in early paediatric development in early paediatric development

Dose selection Dose selection in early paediatric development in early paediatric development

Dose selection Dose selection in early paediatric development in early paediatric development Oscar Della Pasqua Clinical Pharmacology Modelling & Simulation Leiden/Amsterdam Center for Drug Research Division of Pharmacology Outline

578 views • 29 slides
How Non Ignorable is the Selection Bias in Non Probability Samples? An Illustration of New

How Non Ignorable is the Selection Bias in Non Probability Samples? An Illustration of New

11/8/2018 How Non Ignorable is the Selection Bias in Non Probability Samples? An Illustration of New Measures using a Large Genetic Study on Facebook Brady T. West 1 Phil Boonstra 2 Roderick J.A. Little 1,2 Jingwei Hu 1 Fernanda Alvarado

382 views • 14 slides
So you think you have all the data? Causes and consequences of selection bias David J. Hand

So you think you have all the data? Causes and consequences of selection bias David J. Hand

So you think you have all the data? Causes and consequences of selection bias David J. Hand Imperial College London and Winton Capital Management 16 th September 2016 1 BACKGROUND The promise of big data: Out with every theory of human behavior,

402 views • 38 slides
Identification of Causal Effect in the Presence of Selection Bias Juan D. Correa Jin Tian Elias

Identification of Causal Effect in the Presence of Selection Bias Juan D. Correa Jin Tian Elias

Identification of Causal Effect in the Presence of Selection Bias Juan D. Correa Jin Tian Elias Bareinboim AAAI Honolulu, 2019 Challenge 1: Confounding Bias Age Whats the causal effect of Exercise on Cholesterol ? What about

297 views • 15 slides
Selection and estimation in exploratory subgroup analyses a proposal Gerd Rosenkranz, Novartis

Selection and estimation in exploratory subgroup analyses a proposal Gerd Rosenkranz, Novartis

Selection and estimation in exploratory subgroup analyses a proposal Gerd Rosenkranz, Novartis Pharma AG, Basel, Switzerland EMA Workshop, London, 07-Nov-2014 Purpose of this presentation Proposal for exploratory subgroup analyses

290 views • 12 slides
BIAS What Is Bias? Bias can be defined as favoring one side, position, or belief being

BIAS What Is Bias? Bias can be defined as favoring one side, position, or belief being

BIAS What Is Bias? Bias can be defined as favoring one side, position, or belief being partial or prejudiced Where have you seen bias? Bias vs. Propaganda Bias is prejudice; a preconceived judgment or an opinion formed

717 views • 29 slides
An Interval Estimation Approach to Selection Bias in Observational Studies Matt Tudball 1 with

An Interval Estimation Approach to Selection Bias in Observational Studies Matt Tudball 1 with

An Interval Estimation Approach to Selection Bias in Observational Studies Matt Tudball 1 with Rachael Hughes 1 , Kate Tilling 1 , Qingyuan Zhao 2 and Jack Bowden 1 1 MRC Integrative Epidemiology Unit, University of Bristol 2 Department of

282 views • 11 slides
ECON 626: Applied Microeconomics Lecture 1: Selection Bias and the Experimental Ideal

ECON 626: Applied Microeconomics Lecture 1: Selection Bias and the Experimental Ideal

ECON 626: Applied Microeconomics Lecture 1: Selection Bias and the Experimental Ideal Professors: Pamela Jakiela and Owen Ozier Potential Outcomes Do Hospitals Make People Healthier? Your health status is: excellent, very good, good, fair, or

1.27k views • 94 slides
A Modeling Approach to Compensate for Nonresponse and Selection Bias in Surveys? Tien-Huan (Amy)

A Modeling Approach to Compensate for Nonresponse and Selection Bias in Surveys? Tien-Huan (Amy)

A Modeling Approach to Compensate for Nonresponse and Selection Bias in Surveys? Tien-Huan (Amy) Lin, Ismael Flores Cervantes Westat JSM 2019 Denver, Colorado July 27-August 1 The views presented in this paper are those of the

646 views • 13 slides
Using Hierarchical Models to Calibrate Selection Bias Douglas Rivers Stanford University and

Using Hierarchical Models to Calibrate Selection Bias Douglas Rivers Stanford University and

Using Hierarchical Models to Calibrate Selection Bias Douglas Rivers Stanford University and YouGov February 26, 2016 Margins of error We agree that margin of sampling error in surveys has an accepted meaning and that this measure is not

564 views • 20 slides
Adjusting for selection bias in case control studie S.Geneletti, S.Richardson, N.Best Department

Adjusting for selection bias in case control studie S.Geneletti, S.Richardson, N.Best Department

References Adjusting for selection bias in case control studie S.Geneletti, S.Richardson, N.Best Department of Epidemiology and Public Health, Imperial College 24/07/2008 References OUTLINE 1. Examples 2. Hypospadias Study 3. What is a DAG? 4.

801 views • 77 slides
Sneaky Brain Our brain is playing tricks Cognitive Bias We create our own "subjective

Sneaky Brain Our brain is playing tricks Cognitive Bias We create our own "subjective

Sneaky Brain Our brain is playing tricks Cognitive Bias We create our own "subjective social reality" Confirmation Bias Halo Effect Hindsight Bias Wason selection task If a card has a D on one side, it has a 3 on the other side.

425 views • 20 slides
Principle of case control studies Part II Selection of case and control Recall bias

Principle of case control studies Part II Selection of case and control Recall bias

Principle of case control studies Part II Selection of case and control Recall bias Advantage and limitation of case control study Piyanit Tharmaphornpilas MD, MPH The I nternational Field Epidemiology Training Program, Thailand Who

607 views • 12 slides
Equity & Excellence: Hidden Bias Implicit Bias Inherent Bias

Equity & Excellence: Hidden Bias Implicit Bias Inherent Bias

Equity & Excellence: Hidden Bias Implicit Bias Inherent Bias ____________________________________________ Vincent R. De Lucia Educator-in-Residence Director of Mandatory Training & PD New Jersey School Boards Association Anderson

426 views • 41 slides
DANGERS OF SECONDARY CRASHES By Bill Fuller DANGERS OF SECONDARY CRASHES Danger to Motorists

DANGERS OF SECONDARY CRASHES By Bill Fuller DANGERS OF SECONDARY CRASHES Danger to Motorists

DANGERS OF SECONDARY CRASHES By Bill Fuller DANGERS OF SECONDARY CRASHES Danger to Motorists Can be more severe than original incident Can occur as a result of traffic slowing quickly : Around a curve Beyond hill crest Larger

207 views • 7 slides
Multiplicity and Estimation P.Bauer Medical University of Vienna London, November 2012

Multiplicity and Estimation P.Bauer Medical University of Vienna London, November 2012

Multiplicity and Estimation P.Bauer Medical University of Vienna London, November 2012 Selection bias Reporting bias Bias at admission Methods of estimation to reduce bias Multiple testing Multiple confidence intervals

668 views • 23 slides
bad decision bias? 2 5 6 Affinity Bias: What can we do? 2. 3. 1. Seek

bad decision bias? 2 5 6 Affinity Bias: What can we do? 2. 3. 1. Seek

bad decision bias? 2 5 6 Affinity Bias: What can we do? 2. 3. 1. Seek outputs Compare rather than Be aware performance compentenci not people es 7 9 10 11 12 Unconscious bias refers to a

292 views • 15 slides
Decision-making Bias in Instance Matching Model Selection Mayank Kejriwal, Daniel P . Miranker

Decision-making Bias in Instance Matching Model Selection Mayank Kejriwal, Daniel P . Miranker

Decision-making Bias in Instance Matching Model Selection Mayank Kejriwal, Daniel P . Miranker Acknowledgements: US National Science Foundation, Microsoft Research Instance Matching 50+ year old Artificial Intelligence problem When do

231 views • 19 slides

More recommend