Assessing the Quality of Observational studies in ILCOR Eddy Lang - PowerPoint PPT Presentation

Assessing the Quality of Observational studies in ILCOR Eddy Lang Russell Griffin Associate Professor Science and Medicine Advisor University of Calgary American Heart Association Michael Sayre Peter Morley Professor, Emergency Medicine Associate Professor University of Washington University of Melbourne

Overview • Observational studies (non-RCTs) in evidence hierarchy • Assessing risk of bias in observational studies • GRADE risk of bias tool • Pooling data from observational studies

Evidence pyramid

Quality assessment criteria Quality of Lower if… Higher if… Study evidence design Study limitations High Large effect (e.g., RR 0.5) RCTs  (design and execution) Very large effect (e.g., RR 0.2) Inconsistency Moderate Evidence of dose-response gradient Observational Indirectness Low All plausible confounding… studies  …would reduce a Imprecision Very low demonstrated effect …would suggest a spurious Publication bias effect when results show no effect 4

It’s not fair! • Observational studies may be best available • RCTs not feasible ? ethical • Large observational studies > RCTs

“Categories” of quality (1)    Further research is very unlikely to change our High confidence in the estimate of effect       Further research is likely to have an important impact on     Moderate our confidence in the estimate of effect and may change  the estimate    Further research is very likely to have an important       Low impact on our confidence in the estimate of effect and is likely to change the estimate        Very low Any estimate of effect is very uncertain  6

Conceptualizing quality (2)    We are very confident that the true effect lies close to High that of the estimate of the effect.       We are moderately confident in the estimate of effect:     Moderate The true effect is likely to be close to the estimate of  effect , but possibility to be substantially different.    Our confidence in the effect is limited: The true effect       Low may be substantially different from the estimate of the effect.    We have very little confidence in the effect estimate:     Very low The true effect is likely to be substantially different from the estimate of effect.  7

RCT start high, obs. data start low 1. Risk of bias Grade down Outcome Critical P 2. Inconsistency High 3. Indirectness I Outcome Critical Moderate 4. Imprecision Low C Outcome Important 5. Publication Very low O Outcome Not bias Summary of findings Grade up 1. Large effect & estimate of effect 2. Dose for each outcome response 3. Confounders Systematic review Guideline development Formulate recommendations : Rate • For or against (direction) overall quality of evidence • Strong or weak (strength) across outcomes based on lowest quality By considering:  Quality of evidence of critical outcomes  Balance benefits/harms  Values and preferences • “We recommend using…” • “We suggest using…” Revise if necessary by considering: • “We recommend against using…”  Resource use (cost) • “We suggest against using…”

Newcastle-Ottawa Quality Assessment Scale: Cohort Studies • Selection (4) • Comparability (1) • Outcome (3) – A study can be awarded a maximum of one star for each numbered item within the Selection and outcome categories. A maximum of two stars can be given for Comparability

Selection 1. Representativeness of the exposed cohort a) truly representative of the average ___________ (describe) in the community  b) somewhat representative of the average ___________ in the community  c) selected group of users eg nurses, volunteers d) no description of the derivation of the cohort In the case of mortality 2. Selection of the non exposed cohort a) drawn from the same community as the exposed cohort  studies, outcome of b) drawn from a different source interest is still the presence c) no description of the derivation of the non exposed cohort of a disease/ incident, rather than death; that is a 3. Ascertainment of exposure a) secure record (eg surgical records)  statement of no history of b) structured interview  disease or incident earns a c) written self report star d) no description 4. Demonstration that outcome of interest was not present at start of study a) yes  b) no

Comparability 1. Comparability of cohorts on the basis of the design or analysis a) study controls for ___________ (select the most important factor)  b) study controls for any additional factor (This criteria could be modified to indicate specific control for a second important factor.) 

Outcome 1. Assessment of outcome a) independent blind assessment  b) record linkage  c) self report d) no description 2. Was follow up long enough for outcomes to occur a) yes (select an adequate follow up period for outcome of interest)  b) no 3. Adequacy of follow up of cohorts a) complete follow up - all subjects accounted for  b) subjects lost to follow up unlikely to introduce bias - small number lost - > ___ % (select an adequate %) follow up, or description of those lost)  c) follow up rate < ___% (select an adequate %) and no description of those lost d) no statement

Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Ever Use) Cohort Studies Selection Comparability Outcome Lauritzen / 83 Wilson / 85 Petitti / 87 Henderson / 91 Lafferty / 94 Folsom / 95 Ettinger / 96 Wolf / 96 0.01 0.1 1 10

The GRADE approach to RoB

Key Questions for Body of Obs Studies • Do I preserve the LoE at low for this outcome? • Do I downgrade to very low? • Are upgrade criteria present?

RoB in Observational Studies • 1. Failure to develop and apply appropriate eligibility criteria (inclusion of control population) • Under- or overmatching in case-control studies • Selection of exposed and unexposed in cohort studies from different populations

RoB in Observational • 2. Flawed measurement of both exposure and outcome • Differences in measurement of exposure (e.g., recall bias in case-control studies) • Differential surveillance for outcome in exposed and unexposed in cohort studies

RoB in observational • 3. Failure to adequately control confounding • Failure of accurate measurement of all known prognostic factors • Failure to match for prognostic factors and/or lack of adjustment instatistical analysis • 4. Incomplete follow-up

Reasons to upgrade? • Large effect size – OR > 2.0 or < 0.5 = increase to moderate – OR > 4.0 or < 0.2 = increase to high • Dose – response ? Time factor • All possible confounding supports conclusions

RoB incons Large effect

Pooling observational studies • Included studies • Similar PICOs • RevMan

Questions?