Sliding-Window Aggregation in Worst-Case Constant Time Martin - PowerPoint PPT Presentation

Sliding-Window Aggregation in Worst-Case Constant Time Martin Hirzel, IBM Research AI 30 October 2017 Dagstuhl Seminar on Big Stream Processing Systems

Streaming Engines Telco Medical Science Finance , Streaming engine Insights Actions Martin Hirzel, IBM Research AI 2 ibmstreams.github.io

Productivity Challenge Telco Medical Science Finance , Streaming engine High-level programming experience Insights Actions SPL: An Extensible Language for Distributed Martin Hirzel, IBM Research AI 3 Stream Processing [TOPLAS'17]

Performance Challenge Telco Medical Science Finance , Parallel algorithms f ( x 1 ) || f ( x 2 ) Streaming engine High-level programming experience Incremental algorithms f ( x ) ± f (∆) Insights Actions A Catalog of Stream Processing Martin Hirzel, IBM Research AI 4 Optimizations [CSUR'14]

Sliding-Window Aggregation Oldest Youngest a. max : 6 2 6 3 5 3 State evict Query b. max : 6 6 3 5 3 Step insert 1 Time c. max : 6 6 3 5 3 1 evict d. max : 5 3 5 3 1 In general: insert 4 • Any associative e. max : 5 3 5 3 1 4 aggregation operation ⊕ evict (not just max ⇒ f. max : 5 5 3 1 4 sum, geoMean, Bloom, ,) insert 2 • Any interleaving of g. max : 5 5 3 1 4 2 insert and evict evict (not just alternating ⇒ 3 1 4 2 h. max : 4 variable-sized windows) insert 7 3 1 4 2 7 i. max : 7 General Incremental Sliding- Martin Hirzel, IBM Research AI 5 Window Aggregation [VLDB'15]

Sliding-Window Aggregation in Worst-Case Constant Time De-Amortized Banker’s Aggregator (DABA): Every insert , evict , and query invokes the associative ⊕ operation at most O (1) times. F L R A B E (front) (left) (right) (accum) (back) (end) vals aggs (| l F | = 0 and | l B | = 0) or (| l L | = | l R | and | l L | + | l R | + | l A | + 1 = | l F | − | l B |) Low-Latency Sliding-Window Aggregation in Worst- Martin Hirzel, IBM Research AI 6 Case Constant Time [DEBS’17] (best paper)