Surveillance system evaluation Potjaman Siriarayapon, MD, FETP, - PowerPoint PPT Presentation

Surveillance system evaluation Potjaman Siriarayapon, MD, FETP, DrPH

Why do we evaluate surveillance Why do we evaluate surveillance system? system?

Number of acute diarrhea cases in 2002 compare with median 5 years (1997-2001), Akat - Umnuay District , Sakonnakorn 200 180 161 150 Median 5 yrs 122 2002 100 97 96 80 78 70 66 61 45 50 35 48 23 17 18 18 14 0 Jan Feb Mar Apr May June July Aug Sep Oct Mov Dec Source : Akat-Umnuay Health Office, Sakonnakorn

Evaluation Systematic investigation of the merit (quality), worth (cost-effectiveness), or significant (importance) of an object

Questions to be answered? Questions to be answered? • What will be evaluated? • What aspects of the program will be considered? • What standard must be reached for considered successful program?

Questions to be answered? Questions to be answered? • What evidence will be used to indicate how the program has performed? • What conclusion regarding program performance are justified? • How will the lessons learned from the inquiry be used to improve public health effectiveness?

Program evaluation Process Output Input Man Detail step Outcome Effect Impact Money of working Management (Context)

Framework for program evaluation in public health Engage Stakeholders Ensure use Describe and share the program lessons learned Focus the Justify evaluation conclusions design Gather credible evidence

Purpose of evaluating public health surveillance system To ensure that problems of public health importance are being monitored efficiently and effectively effectively efficiently

Engage the stakeholders in the evaluation • Those involved in program operations • Those served or effected by the program • Primary user of the evaluation

Describe the surveillance system to be evaluated • Describe the public health importance of the disease (or health-related event) under surveillance • Describe the purpose and operation of the system • Describe the resources used to operate the system

A. Public health importance • Disease frequency (number of cases/death, incidence, prevalence, mortality rate) • Severity (case-fatality ratio, hospitalization rate, disability rate) • Preventability • Public interest

B. Purpose and operation of the surveillance system (1) • List the purpose and objectives of the system • Describe the planned uses of the data from the system • Describe the disease under surveillance, including case definition • Cite any legal authority for the data collection

B. Purpose and operation of the surveillance system (2) • Where in the organization the system resides • Level of integration with other systems • Flow chart • Component of the system – Population under surveillance – Period of time of data collection – Collected what data and how – …

C. Resources used to operate the surveillance system • Funding source • Personnel • Other resources – Travel – Training – Supplies – Computer and other equipment – Related services (e.g., mail, telephone, internet, laboratory support, hardware & software maintenance)

Focus the evaluation design • Determining the specific purpose of the evaluation • Identifying stakeholders • Considering what will be done with the result • Determining standards for assessing the performance of the system

Gathering credible evidence • Usefulness • System attribute – Simplicity - Sensitivity – Flexibility - Predictive value positive – Data quality - Representativeness – Acceptability - Timeliness – Stability

About public health surveillance • Surveillance for action!!! • How? – Early detection of abnormal event – Estimated magnitude of morbidity & mortality – Detect trend – Permit assessment of preventive & control measures – Stimulate research intended to lead to prevention and control

Usefulness Usefulness • Describing the “actions taken” as a result of the data from the surveillance system • Should begin with review objectives of the system and assess whether the utilization of the system meet the objectives

Sensitivity Sensitivity Can be considered at 2 levels • Proportion of cases detect by the surveillance system • Ability to detect outbreak,

Comparing data from surveillance system with the real situation Real situation Non-case Case Case A B Surveillance system Non-case C D Sensitivity = A A+C

Sensitivity of suspected avian influenza reported in Nakonnayok province by district, 2005 90.0% 80.0% 76.9% 76.9% 80.0% 70.0% 60.0% 50.0% 50.0% รอยละ 40.0% 30.0% 20.0% 10.0% 0.0% 0.0% เมือง บานนา องครักษ มศว ปากพลี โรงพยาบาล

Predictive value positive (PVP) Predictive value positive (PVP) • Proportion of reported cases of Disease A that really be the Disease A • PVP provide different perspectives to tell how well the system is functioned

Comparing data from surveillance system with the real situation Real situation Non-case Case Case A B Surveillance PVP = A A+B system Non-case C D

Predictive value positive of suspected avian influenza cases Suspected AI cases reported to central level 87 Met case definition 8 Not met case definition 95 Total PVP of suspected Avian influenza cases = 87/95 = 91.58%

Representativeness Representativeness • Represent disease occurrence over time and distribution • These characteristic generally related to time, place, person

Attack rate of severe diarrhea cases by sub - district , Akat Umnuay District ( per 100,000 population )

Timeliness Timeliness • Reflect speed between steps in a public health surveillance system • Should be evaluated in terms of availability of information to control the disease

Simplicity Simplicity • Refer to both its structure and ease of operation • Surveillance system should be as simple as possible while still meeting their objectives

Flexibility Flexibility A flexible public health surveillance system can adapt to changing information needs on operating conditions with little additional time, personnel or allocated funds.

Data quality Data quality Reflect completeness and validity of the data from public health surveillance

Acceptability Acceptability Willingness of persons and organizations to participate in the surveillance system

Stability Stability Refer to reliability (ability to collect, manage, and provide data properly without failure) and availability (ability to be operational when needed) of the public health surveillance system

Example of different surveillance Example of different surveillance evaluation evaluation • Main surveillance system • HIV/AIDS surveillance system • EWARS

Thank you for your attention Thank you for your attention