A public health approach to appraising evidence: Meta-tool for - PowerPoint PPT Presentation

A public health approach to appraising evidence: Meta-tool for Quality Appraisal (MetaQAT) Laura Rosella Carolyn Bowman Beata Pach June 16, 2016

Agenda • Discuss quality appraisal and evidence in public health • Development of PHO MetaQAT • Overview of the meta-tool • Applied exercise PublicHealthOntario.ca 2

Appraisal within review process Idea or request • Steps to systematically review evidence Review question • Different types of reviews • All follow same core process Search • Level of detail and rigour of the different steps differ between types Select • Examples: Appraise • Systematic review, literature review, scoping review, rapid review Extract and analyze data Synthesize Some iteration is required Report PublicHealthOntario.ca 3

Quality (critical) appraisal “Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness , and its value and relevance in a particular context .” • To be blunt – judiciously assessing quality of evidence is challenging • Methods and content expertise required • Informs public health decisions – requires a judicious and transparent process • The “why” is often as important as the outcome PublicHealthOntario.ca 4 Burls A. What is critical appraisal? 2 nd ed. [Internet]. Newmarket, UK: Hayward Group; 2009 [cited 2016 Jun 14]. Available from: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/what_is_critical_appraisal.pdf

Don’t be so negative! • Appraisal should be about understanding • Both strengths and weaknesses • We need to make decisions, important to use evidence to fullest extent possible • Understand how evidence can be used appropriately PublicHealthOntario.ca 5

Subjective is not a bad word • Judgments are made based on appraisal principles • Using the same principles, similar issues will be identified • No one right answer, that doesn’t mean anything goes! • “Document the information on which judgements are based” Sterne J, Higgins J, Reeves B, editors. A Cochrane risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-NRSI). Version 1.0.0 [Internet]. London: Cochrane Collaboration; 2014 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info PublicHealthOntario.ca 6 Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M, et al. The risk of bias in non-randomized studies – of interventions (ROBINS-I) assessment tool [Internet]. London: Cochrane Collaboration; 2016 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info

Transparency • Importance of documentation – if others know what you appraised and why, they can ‘appraise your appraisal’ • Transparency is key : another person should be able to understand why you made your assessment • Think of it as a way that one can audit your analysis of the evidence – akin to auditing code following a statistical analysis PublicHealthOntario.ca 7

Skills and knowledge needed Setting Application Validity Methods concepts Quality Topic Documentation Appraisal These traits can be covered by a team PublicHealthOntario.ca 8

Skills and knowledge needed Setting Application Validity Methods concepts Quality Topic Documentation Appraisal These traits can be covered by a team PublicHealthOntario.ca 9

Internal and external validity Internal Validity External Validity • Elwood: measure of how • The process of generalizing the easily a difference in an findings of the study to the outcome between two population from which the groups can be attributed sample was drawn (or even to the effects of an beyond that population to a exposure or intervention more universal statement) • L. Green: “ the essence of • Without internal validity a study rigor” cannot have external validity • L. Green “ the essence of relevance” Ellwood M. Critical appraisal of epidemiological studies and clinical trials. New York: Oxford University Press; 2007. PublicHealthOntario.ca 10 Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126-153.

Internal validity can be influenced by... • Study design • Measurement properties of the variables • Study recruitment/response rates • Selection pressures • Sampling strategy • Setting • Investigators/funding • Among others factors… PublicHealthOntario.ca 11

Current approach to assessing risk of bias - RCTs Principles for assessing risk of bias: 1. Do not use quality scales 2. Focus on internal validity 3. Assess the risk of bias in trial results, not reporting or other issues 4. Assessments of bias require judgment 5. Choose domains to be assessed based on both theoretical and empirical considerations 6. Focus on risk of bias in data as presented in the review rather than as originally reported 7. Report outcome specific evaluations of risk of bias Higgins J, Altman D, Gotzsche P, Juni P, Moher D, Oxman A, et al; Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane PublicHealthOntario.ca Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. Available from: 12 http://www.bmj.com/content/bmj/343/bmj.d5928.full.pdf Figure 1 used under license terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Current approach to assessing risk of bias – non-randomized studies Sterne J, Higgins J, Reeves B, editors. A Cochrane risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-NRSI). Version 1.0.0 PublicHealthOntario.ca [Internet]. London: Cochrane Collaboration; 2014 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info 13 Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M, et al. The risk of bias in non-randomized studies – of interventions (ROBINS-I) assessment tool [Internet]. London: Cochrane Collaboration; 2016 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info

Assessing risk of bias says nothing about the intended use of the evidence • Application of the evidence is important in public health • Context matters – where the evidence was generated, where it is going to be applied • Importance of different factors when appraising evidence should be guided by the intended application PublicHealthOntario.ca 14

Evidence for Public Health Questions • Evidence-based medicine evidence-based public health Different types of Application to new Rethink what is questions and areas appraised and why context • What is the best evidence to answer your question? • Best available evidence Harder T, Takla A, Rehfuess E, Sanchez-Vivar A, MtysiakKlose D, Echmanns T, et al. Evidence-based decision-making in infectious diseases epidemiology, PublicHealthOntario.ca 15 prevention and control: matching research questions to study designs and quality appraisal tools. BMC Med Res Methodol. 2014;14:69. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-14-69.pdf

Beware the evidence pyramid • Important concept in evidence-based medicine • Pyramid developed to approach evidence for effectiveness of clinical interventions – based on internal validity of study designs • Describes best evidence to answer a particular kind of question PublicHealthOntario.ca 16 SUNY Downstate Medical Center, Medical Research Library of Brooklyn. Guide to research methods: the evidence pyramid [Internet]. Brooklyn, NY: SUNY Downstate Medical Center; 2004 [cited 2016 Jun 14]. Available from: http://library.downstate.edu/EBM2/2100.htm

Critical appraisal and the evidence hierarchy What’s better? • A large and rigorous prospective cohort study or a small poorly conducted randomized control trial? • A systematic review of dozens of well-conducted case control studies or one large randomized control trial? • Large and rigorous randomized controlled trial done in Japan or a time-series analysis in Calgary? (assuming you live in Toronto) • A well conducted study that’s ‘lower’ on the evidence hierarchy may actually be better...every study needs to be interpreted and assessed in its own right PublicHealthOntario.ca 17

Points to remember 1. Quality appraisal should be a transparent process 2. Appraisal involves making considered judgments 3. Overall quality appraisal ≠ internal validity only 4. The question and the context should guide your judgments PublicHealthOntario.ca 18

TOOLS, TOOLS, EVERYWHERE PublicHealthOntario.ca 19

Tools to guide appraisal • Many tools exist • Challenges: • Most not validated • Most from evidence-based medicine perspective – clinical effectiveness focus • Many use numeric scales or scores – not valid, not recommended by Cochrane Collaboration • Many tools appropriate for specific contexts • One topic, project, or study design PublicHealthOntario.ca 20