Evidence-informed Health Policy: NASHP Pre-conference Origins of - PDF document

11/17/2017 October 23, 2017 Evidence-informed Health Policy: NASHP Pre-conference Origins of the EiHP Workshop • Original workshops were developed as a joint project between CEbP and the Milbank Memorial Fund in 2009 • Meant to equip policymakers in state and local government with knowledge, skills, and attitudes for understanding and applying research evidence • Adapted in 2014 to be delivered in different formats, timeframes, and audiences 1

11/17/2017 Acknowledgements • Martha Gerrity and Mark Gibson • Jane Beyer • Milbank Memorial Fund • NASHP and PCORI • Pew-MacArthur Results First Initiative To start • Introductions (of a sort) • Ground rules • Overview of the session 2

11/17/2017 Ground rules • Please don’t hesitate to ask questions • Three hours is a long time, so: – We’ll be generous with the breaks, including to get lunch at noon – We understand if you need to step out • Participation in the polls and other activities will make this more interesting and useful Overview of session • 10:30-10:45 Introduction, ground rules, overview • 10:45-11:25 Defining EiHP and Evidence basics • 11:25-11:30 Break • 11:30-12:00 Summarizing evidence and Common pitfalls • 12:00-12:20 Break for lunch • 12:20-1:10 Examples from the evidence • 1:10-1:30 Wrap-up, discussion, and questions 3

11/17/2017 What is evidence? • For our purposes, evidence comes from research that: – Is intended to test the validity of a claim – Uses reproducible methods – Collects and interprets data using tests to distinguish between chance and true effects – Can be scrutinized by peers and the public – Falsifiable! • More simply stated by W. Edwards Deming: “In God we trust. All others must bring data.” What is EiHP? • An approach to health policy decisions that is informed by the best and most complete available research evidence • A structured way to use research to better understand what works, recognizing that: – Not all studies are created equal – Some studies may not be relevant to policymaking – Transparency in identifying and applying studies is important 4

11/17/2017 What is EiHP? • Systematic process in which relevant research is: – Identified in an unbiased fashion – Critically interpreted to understand its quality – Combined to provide a better estimate of the real effects – Applied appropriately to policymaking – Re-assessed when new information becomes available Why should I use EiHP? • When tackling complex issues, it’s useful to have a sense of: – What we know – What we don’t know – What we merely believe 5

11/17/2017 Why should I use EiHP? • An analogy from medicine – When there is a delay in adopting an effective therapy or discarding an ineffective or harmful therapy, lives are at stake – This hazard is magnified when we consider programs and policies that affect the lives of many more people Why should I use EiHP? • EiHP improves the chances that a policy or investment will achieve the desired ends – And reduces the likelihood that a failed policy will have to be abandoned in the future • EiHP can be a starting point for engaging stakeholders with divergent views • When done deliberately and transparently, EiHP can increase public confidence in the policymaking process • Consider the alternatives: – Anecdotes – Opinion – Intuition 6

11/17/2017 Understanding Evidence The origins of epidemiology 7

11/17/2017 The origins of epidemiology… and public health policy https://upload.wikimedia.org/wikipedia/commons/c/cb/John_Snow_memorial_and_pub.jpg Health in the 20 th Century • Vaccination • Safer, healthier foods • Motor-vehicle safety • Maternal and prenatal care • Safer workplaces • Family planning • Control of infectious • Fluoridation disease • Decline in deaths • Tobacco control from heart attack and policies stroke CDC. (1999). MMWR, 48(12):241-243. 8

11/17/2017 Asking the right question • Having a standard way for framing questions you hope to answer with the evidence is critical – A research tool for assessing which studies are relevant – An exercise in establishing, a priori, what types of research and outcomes would influence the policy decision being contemplated – Agreement about desired outcomes and a process for reviewing and summarizing the evidence can help build consensus Asking the right question • PICO(TS+) – Population – Intervention – Comparison – Outcome – (Timing, Setting, Policy context) 9

11/17/2017 Asking the right question: Population • Demographics • Conditions • Geography Asking the right question: Intervention • Drug, device, or procedure • Diagnostic test • New methods of organizing or delivering care • Systems or process changes • Policy changes 10

11/17/2017 Asking the right questions: Comparison • Status quo • Placebo • Sham procedure • Alternate treatment Asking the right question: Outcomes • Health or wellbeing (most important outcomes) • Surrogate measures • System performance • Process measures 11

11/17/2017 Asking the right question: PICO example Population: Adults with serious mental illness Intervention: Assertive community treatment Comparator: Usual care Outcomes: Psychiatric hospitalization, emergency dept use, homelessness, psychiatric symptoms, medication adherence The “reverse” PICO • When there are disagreements about the meaning or applicability of a study, it can help to reverse the PICO process – May clarify whether the study is really answering the question you are interested in 12

11/17/2017 The Challenge of Using Evidence • There are an estimated 24 million studies in PubMed, each a potential piece of evidence • Studies often reach conflicting results • It’s easy to pick and choose the evidence that best supports a given position • How do you know what evidence is most accurate and reliable? Why are some studies “good” and some studies “bad”? • Some studies are not designed to fairly answer the question they pose • Studies can be biased to favor certain results, intentionally or unintentionally • Conflict of interest can result in a bias toward favorable results • It’s time consuming and takes some technical sophistication to sort through studies to assess quality and summarize results 13

11/17/2017 The essence of epidemiology • How do you explore the relationship between an exposure and an outcome ? – Hypothesize and observe – Hypothesize and experiment The evidence hierarchy Murad et al. (2016). Evidence-Based Medicine Published Online First: 23 June 2016. doi:10.1136/ebmed-2016-110401 14

11/17/2017 The evidence hierarchy Murad et al. (2016). Evidence-Based Medicine Published Online First: 23 June 2016. doi:10.1136/ebmed-2016-110401 Case series or reports • Simply describes a set of cases and their outcomes • Often used for rare conditions, or when a treatment or test is very new • Usually represent the experience of a single center • Should not be used to establish effectiveness of a treatment • Be especially wary of non-consecutive case series (meaning that the authors picked out only the cases they wanted to describe) 15

11/17/2017 The risk of case reports • Porter and Jick, 1981: • Addiction is rare in people treated with narcotics The risk of case reports Leung, et al. (2017) NEJM. 376;22. 16

11/17/2017 Case-control studies Exposed Cases Unexposed Exposed Controls Unexposed Past Start of study Case-control example Large class High test scores Small class Large class Low test scores Small class Past Start of study 17

11/17/2017 Case-control studies • Advantages: – Quick and inexpensive – Particularly good for investigating rare outcomes – Dynamic populations • Disadvantages: – Recall bias – Cases and controls may not be representative (selection bias) – Confounding Prospective cohort studies Exposed Outcomes Healthy Population Unexposed Outcomes Start of study Future 18

11/17/2017 Prospective cohort example Small class Test scores Healthy Population Large class Test scores Start of study Future Prospective cohort studies • Advantages: – Eliminate recall bias – Can examine multiple outcomes – Allows estimation of incidence of outcome – Better at detecting long-term harms than other studies • Disadvantages – Expensive and take a long time – Loss to follow-up (attrition bias) – Still subject to confounding 19

11/17/2017 Retrospective cohort studies Exposed Outcomes Unexposed Outcomes Start of study Past Retrospective cohort example Small class Test scores Large class Test scores Start of study Past 20

11/17/2017 Retrospective cohort studies • Advantages – Much easier to do than prospective cohorts • Disadvantages – Only as good as the data set being used • Claims data and diagnosis codes are often unreliable – Data about confounding factors could be missing and can’t be adjusted for Randomized controlled trial (RCT) Intervention Outcomes Population of Interest Control Outcomes Start of study Future 21