Assessing EARS’ Ability to Locally Detect the 2009 H1N1 Pandemic Ron Fricker, Katie Hagen, Krista Hanni, Susan Barnes, and Kristy Michie January 9, 2011
Early Aberration Reporting System • EARS designed as a drop-in surveillance system • But also used by local public health departments for routine health surveillance – Including Monterey County Health Department (MCHD) Clinic 3 Clinic 4 Clinic 5 Clinic 2 Clinic 6 Clinic 1 2
MCHD’s “Dose Report” 3
Research Goal • Assess how well the Early Aberration Reporting System (EARS) can detect known outbreaks (seasonal ILI and H1N1) – Including two EARS variants implemented by the Monterey County Health Department – Compared to an alternative CUSUM-based methodology 4
Deriving Daily Syndrome Counts • County clinics and hospital ERs send chief complaint data to MCHD on daily basis – Weekdays less holidays – Examples: 5
Deriving Daily Syndrome Counts • Text matching searches for terms in the data to derive symptoms – E.g., existence of word “flu” results in classifying an individual with the flu symptom • Symptoms then used to determine whether to classify an individual with a syndrome • MCHD has used three definitions for ILI syndrome: 6
Out of 153,696 total records… ILI Definition Counts 9,093 total ILI Base Case 5,154 added 291 deleted Detection 13,956 total ILI (53% ↑) Expanded Algorithms 51 added 8,410 deleted 734 total ILI (92% ↓) Restricted 7
Daily Count Time Series 8
Early Aberration Reporting System • EARS’ detection algorithms: • Sample statistics calculated from Y t ( ) Y t ( ) previous 7 days’ data 1 C t ( ) 1 • Signal when C 1 > 3 s t ( ) 1 • Sample statistics calculated from Y t ( ) Y t ( ) 3 C t ( ) 7 days’ of data prior to 2 day lag 2 s t ( ) • Signal when C 2 > 3 3 t 2 C t ( ) max 0, C i ( ) 1 • Signal when C 3 > 2 3 2 i t • Often referred to as CUSUMs, but not true • In SPC parlance, C 1 and C 2 are Shewhart variants 9
CUSUM on Adaptive Regression Forecast Errors • Adaptive regression: regress a sliding baseline of observations on time relative to current observation – I.e. regress on Y t Y t n ( 1),..., ( ) n ,...,1 • Calculate standardized residuals from one day ahead ˆ forecast, , where Z t ( ) R t ( ) / Y ˆ ˆ ˆ R t ( ) Y t ( ) ( n 1) 0 1 j and • CUSUM: S t ( ) max 0, ( S t 1) Z t ( ) k where a signal is generated if S ( t )> h 10
Three CUSUMs Evaluated • We looked at the performance of three CUSUMs based on choices of k and h : – Smaller k : Can detect smaller increases in mean – Larger h : Fewer false positive signals (i.e., larger ATFS) but slower to signal 11
Determining the Outbreak Periods 12
Baseline ILI Definition Results 13
Expanded ILI Definition Results 14
Restricted ILI Definition Results 15
Quantifying Performance • Metrics: – Sensitivity : # outbreak days with signal / # outbreak days – Specificity : # non-outbreak days without signal / # non-outbreak days – Average delay : • d average time to signal from start of outbreak period 1 • d average time to signal from earliest signal 2 • Results: 16
Conclusions: ILI Definitions • With EARS algorithms: – Baseline definition did not perform well – Expanded definition performed even worse – Restricted definition gave best performance • With CUSUM algorithms – Choice of ILI definition seems less critical • Hard to conclude CUSUM performed better under one definition than the others • Suggests CUSUM is robust – a good characteristic since connection of definition to arbitrary outbreak type tenuous • More syndrome definition research warranted – “Low hanging fruit” to improve performance 17
Conclusions: Algorithms • Consider incorporating (real) CUSUM detection algorithms into EARS – SPC literature shows that CUSUM methods can detect smaller mean increases than Shewhart methods (such as C1 and C2) • For routine surveillance applications, at least incorporate methods that use more historical data 18
Recommend
More recommend
