recursive lattice search hierarchical heavy hitters
play

Recursive Lattice Search: Hierarchical Heavy Hitters Revisited - PowerPoint PPT Presentation

Jan 14, 2024 •354 likes •807 views

Recursive Lattice Search: Hierarchical Heavy Hitters Revisited Kenjiro Cho IIJ Research Laboratory IMC17 November 2, 2017 Hierarchical Heavy Hitters (HHHs) identifying significant clusters across multiple planes - exploiting underlying

  1. Recursive Lattice Search: Hierarchical Heavy Hitters Revisited Kenjiro Cho IIJ Research Laboratory IMC’17 November 2, 2017

  2. Hierarchical Heavy Hitters (HHHs) • identifying significant clusters across multiple planes - exploiting underlying hierarchical IP address structures - e.g., (src, dst) address pairs • (1.2.3.4, *) → one-to-many: e.g., scanning • (*, 5.6.7.8) → many-to-one: e.g., DDoS • (1.2.3.0/24, 4.5.6.0/28) → subnet-to-subnet - can be extended to higher dimensions (e.g., 5-tuple) • powerful tool for traffic monitoring/anomaly detection 2

  3. Unidimensional HHH • an HHH: an aggregate with count c ≥ φ N - φ : threshold N : total input (e.g., packets or bytes) • HHHs can be uniquely identified by depth-first tree traversal - aggregating small nodes until it exceeds the threshold 0.0.0.0/0 0.0.0.0/0 10.1/16 192.168/16 10.1/16 192.168/16 192.168.3/24 192.168.3/24 10.1.1/24 10.1.2/24 10.1.1/24 10.1.2/24 3 10.1.1.4 10.1.2.5 10.1.1.4 10.1.2.5

  4. Multi-dimensional HHH sum of prefix lengths 0,0 0 8,0 0,8 • each node has multiple parents 8 8,8 0,16 16,0 - many combinations for aggregation 16 8,16 24,0 0,24 16,8 - much harder than one-dimension 24 16,16 24,8 8,24 32,0 0,32 • search space for 2-dimensional IPv4 addrs 32 16,24 24,16 8,32 32,8 - 5×5=25 for bytewise aggregation 40 32,16 24,24 16,32 - 33×33=1089 for bitwise aggregation 48 src: 1.2.3.4 dst: 5.6.7.8 32,24 24,32 56 32,32 [1.2.3.4/32,5.6.0.0/16] [1.2.3.0/24,5.6.7.0/24] [1.2.0.0/16,1.2.3.4/32] 64 Lattice for IPv4 prefix length pair [1.2.3.4/32,5.6.7.0/24] [1.2.3.0/24,5.6.7.8/32] with 8-bit granularity [1.2.3.4/32,5.6.7.8/32] 4

  5. Challenges • performance - bitwise aggregation is costly • operational relevance - ordering: e.g., [32, *] and [16, 16] - broad and redundant aggregates: (e.g., 128/4 and 128/2) • re-aggregation - useful for interactive analysis (for zoom-in/out) 5

  6. Contributions into a tractable one, by revisiting the commonly accepted definition • new efficient HHH algorithm for bitwise aggregation - matches operational needs, supports re-aggregation • open-source tool and open datasets • more broadly, transforming the existing hard problem 6

  7. Various HHH definitions • discounted HHH ⬅ we also employ this - exclude descendant HHHs’ counts for concise outputs c i ′ = ∑ j c j ′ where { j ∈ child(i) | c j ′ < φ N } • rollup rules: how to aggregate counts to parent - overlap rule: allows double-counting to detect all possible HHHs - split rule: preserves counts ⬅ we use a simple first-found split rule • aggregation ordering - sum of prefix lengths ⬅ we’ll revisit this ordering 7

  8. Previous algorithms • elaborate structures - cross-producting, grid-of-trie, rectangle-search • theoretical analyses - streaming approximation algorithms w/ error bounds • all the existing methods are bottom-up • our algorithm: top-down, deterministic - no elaborate structure, no approximation, no parameter 8

  9. HHH revisited • key idea: redefine child(i) to allow space partitioning - child(i) : from bin-tree to quadtree [16, 16] [16, 16] (1.2/16, 5.6/16) [16, 16] [24, 16] [16, 24] [24, 24] [24, 16] [16, 24] (1.2.0/24, 5.6/16) (1.2/16, 5.6.0/24) (1.2.1/24, 5.6/16) (1.2/16, 5.6.1/24) (1.2.3/24, 5.6/16) (1.2/16, 5.6.3/24) ... ... bottom-up aggregation top-down space partitioning 9

  10. Z-order [Morton1966] dimensions • a space filling curve 0,0 - by bit-interleaving (l 0 , l 1 ) 8,0 0,8 8,8 0,16 16,0 8,16 24,0 16,8 0,24 • prefers the largest value across 24,8 8,24 16,16 32,0 0,32 16,24 24,16 8,32 32,8 24,24 32,16 16,32 • looks different from standard Z-curve 32,24 24,32 - [0..32] doesn’t have full 5-bit space 32,32 0,0 0,16 16,0 16,16 0,32 32,0 32,32 0,8 8,0 8,8 0,24 8,16 8,24 16,8 24,0 24,8 16,24 24,16 24,24 8,32 16,32 24,32 32,8 32,16 32,24 (I) upper (II) right (III) left (IV) lower (V) right (VI) left - makes /32 higher in the order sub-area sub-area sub-area sub-area bottom edge bottom edge 10

  11. Recursive spatial partitioning • (VI) left-bottom edge • (V) right-bottom edge • (IV) lower quadrant • (III) left quadrant • (II) right quadrant • (I) upper quadrant 0,0 • visit regions from (VI) to (I) recursively 8,0 0,8 - 2 bottom edges 8,8 0,16 16,0 (I) 8,16 24,0 0,24 16,8 24,8 16,16 8,24 32,0 0,32 - 4 quadrants (III) (II) 16,24 24,16 32,8 8,32 32,16 16,32 24,24 (VI) (VI) (V) (V) (IV) 32,24 24,32 32,32 11

  12. Recursive Lattice Search (RLS) • idea: recursively subdivide aggregates by Z-order • pros - recurse only for flows ≥ thresh - sub-division needs only parent’s sub-flows - /32 becomes higher in the order • cons - bias for the first dimension 12

  13. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  14. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  15. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  16. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  17. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  18. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  19. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  20. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  21. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  22. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  23. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  24. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  25. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  26. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  27. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  28. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  29. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  30. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  31. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  32. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  33. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  34. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  35. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  36. RLS Illustrated 0,0 10 inputs, HHH ≥ 2 4 HHHs extracted 16,0 0,16 16,16 32,0 0,32 32,16 16,32 32,32 13

  37. Evaluation (in the paper) • differences due to different definitions • but requires more memory (as a non-streaming algo) • ordering bias: (src, dst) vs (dst, src) ➡ negligible • comparison with Space-Saving: to illustrate differences - outputs ➡ much more compact - speed ➡ 100 times faster for bitwise aggregation 14

  38. meet operational needs Implementations: RLS in agurim • agurim: open-source tool • 2-level HHH - main-attribute (src-dst adds), sub-attritbute (ports) • protocol specific heuristics - change depth of recursions by protocol knowledge to • online processing by exploiting multi-core CPU 15

  39. agurim Web UI http://mawi.wide.ad.jp/~agurim/ 16

  40. Summary • Recursive Lattice Search algorithm for HHH - revisit the definition of HHH, apply Z-ordering - propose an efficient HHH algorithm • open-source tool and open datasets from 2013 http://mawi.wide.ad.jp/~agurim/about.html 17

  41. evaluation in detail • simulation: code from SpaceSaving [Mitzenmacher2012] - quick hack to port agurim’s RLS - input: a mawi packet trace from 2016-10-20 • order sensitivity: (src,dst) vs. (dst,src) - very similar outputs: not sensitive to the order • comparing with SS (streaming algorithm, overlap rollup) - different definitions: just to illustrate major differences - outputs: comparable, except nodes in upper lattice - performance: 100x faster for bit-wise aggregation! 18

Recommend

Distributed Private Heavy Hitters Justin Hsu, Sanjeev Khanna, Aaron Roth University of

Distributed Private Heavy Hitters Justin Hsu, Sanjeev Khanna, Aaron Roth University of

Distributed Private Heavy Hitters Justin Hsu, Sanjeev Khanna, Aaron Roth University of Pennsylvania July 11, 2012 Hsu, Khanna, Roth (UPenn) Distributed Private Heavy Hitters July 11, 2012 1 / 18 A motivating problem: Website referrals A

1k views • 33 slides
Finding Interesting Correlations with Conditional Heavy Hitters Katsiaryna Mirylenka (University

Finding Interesting Correlations with Conditional Heavy Hitters Katsiaryna Mirylenka (University

Finding Interesting Correlations with Conditional Heavy Hitters Katsiaryna Mirylenka (University of Trento) Themis Palpanas (University of Trento) Graham Cormode (AT&amp;T Labs) Divesh Srivastava (AT&amp;T Labs) Streaming Data Processing

483 views • 19 slides
Search Algorithms 15-110 - Monday 2/24 Learning Objectives Trace over recursive functions

Search Algorithms 15-110 - Monday 2/24 Learning Objectives Trace over recursive functions

Search Algorithms 15-110 - Monday 2/24 Learning Objectives Trace over recursive functions that use multiple recursive calls with Towers of Hanoi Recognize linear search on lists and in recursive contexts Use binary search when reading

465 views • 30 slides
If w p f fxtsnz En l list then added to s If x F Lto 4 Ee 20 E 0.01 Suppose on list

If w p f fxtsnz En l list then added to s If x F Lto 4 Ee 20 E 0.01 Suppose on list

Hitters Heavy stream of Its e g IP addresses a ay ay 64 each 2 n c ex ai n For any ten true t f let ett tires in appears 9,92 at a ffrequency Goal f allelts Fa return set x large using of t space fxt Tz heavy

194 views • 5 slides
Communication Complexity in Locally Private Distribution Estimation and Heavy Hitters ICML 2019,

Communication Complexity in Locally Private Distribution Estimation and Heavy Hitters ICML 2019,

Communication Complexity in Locally Private Distribution Estimation and Heavy Hitters ICML 2019, Long Beach June 11th, 2019 Jayadev Acharya, Cornell University Ziteng Sun, Cornell University Distribution Learning [ k ] = { 0 , 1 , 2 , ...,

359 views • 23 slides
Recursive po2DFA: Hierarchical Automata for FO-definable Languages Simoni S. Shah Joint work

Recursive po2DFA: Hierarchical Automata for FO-definable Languages Simoni S. Shah Joint work

Recursive po2DFA: Hierarchical Automata for FO-definable Languages Simoni S. Shah Joint work with Paritosh K. Pandya Tata Institute of Fundamental Research, Mumbai 9 Feb, 2015 S. S. Shah Recursive PO2DFA Overview Recursive po2dfa and its

626 views • 25 slides
Analytic computations of an effective lattice theory for heavy quarks Jonas R. Glesaaen Mathias

Analytic computations of an effective lattice theory for heavy quarks Jonas R. Glesaaen Mathias

Analytic computations of an effective lattice theory for heavy quarks Jonas R. Glesaaen Mathias Neuman, Owe Philipsen Lattice Conference 2015 - July 16th 1 The Effective Theory Results 2 3 Conclusion Heavy QCD Phase Diagram T [ MeV ]

629 views • 42 slides
CountMin and Count Sketches Lecture 10 February 14, 2019 Chandra (UIUC) CS498ABD 1 Spring

CountMin and Count Sketches Lecture 10 February 14, 2019 Chandra (UIUC) CS498ABD 1 Spring

CS 498ABD: Algorithms for Big Data, Spring 2019 CountMin and Count Sketches Lecture 10 February 14, 2019 Chandra (UIUC) CS498ABD 1 Spring 2019 1 / 18 Heavy Hitters Problem Heavy Hitters Problem: Find all items i such that f i &gt; m / k

786 views • 35 slides
Recursive Regularization for Large-scale Classification with Hierarchical and Graphical

Recursive Regularization for Large-scale Classification with Hierarchical and Graphical

Motivation Related Work Proposed Model Optimization Experiments Recursive Regularization for Large-scale Classification with Hierarchical and Graphical Dependencies Siddharth Gopal Yiming Yang Carnegie Mellon Univeristy 12th Aug 2013

569 views • 53 slides
Charmed hadrons from heavy ion collisions Hadron Interactions and Polarizations from Lattice QCD,

Charmed hadrons from heavy ion collisions Hadron Interactions and Polarizations from Lattice QCD,

Charmed hadrons from heavy ion collisions Hadron Interactions and Polarizations from Lattice QCD, Quark Model, and heavy ion collisions March 26 th 2019 Yukawa Institute for Theoretial Physics, Kyoto University Sungtae Cho Kangwon National

492 views • 24 slides
Large N c behaviour of an effective lattice theory for heavy dense QCD The 37 th International

Large N c behaviour of an effective lattice theory for heavy dense QCD The 37 th International

CRC - TR Large N c behaviour of an effective lattice theory for heavy dense QCD The 37 th International Symposium on Lattice Field Theory Owe Philipsen, Jonas Scheunert Goethe-Universit at Frankfurt am Main Institut f ur Theoretische

313 views • 17 slides
Recursive Lexicographical Search: Finding all Markov Perfect Equilibria in Directional Dynamic

Recursive Lexicographical Search: Finding all Markov Perfect Equilibria in Directional Dynamic

Introduction: Computing All Markov Perfect Equilibria PART I : Solving the Leapfrogging Model with RLS PART II: Recursive Lexicographical Search in General DDGs Conclusions Recursive Lexicographical Search: Finding all Markov Perfect Equilibria

1.73k views • 143 slides
HPS Heavy Photon Search A Proposal to Search for Massive Photons in Hall B at Jefferson

HPS Heavy Photon Search A Proposal to Search for Massive Photons in Hall B at Jefferson

HPS Heavy Photon Search A Proposal to Search for Massive Photons in Hall B at Jefferson Laboratory John Jaros for the Heavy Photon Search Collaboration Jefferson Laboratory PAC37 January 11, 2011 HPS Collaboration 16 Institutions, 58 and

540 views • 38 slides
Search for heavy right-handed gauge bosons decaying into boosted heavy neutrinos with the ATLAS

Search for heavy right-handed gauge bosons decaying into boosted heavy neutrinos with the ATLAS

Search for heavy right-handed gauge bosons decaying into boosted heavy neutrinos with the ATLAS detector at s = 13 TeV Debarati Roy on behalf of the ATLAS Collaboration ILHC-ICTP2019 1 Standard Model (SM) and beyond Left Right Symmetric

425 views • 13 slides
Study of QCD Phase Diagram by Heavy Ion Experiments and Lattice QCD Atsushi Nakamura Far

Study of QCD Phase Diagram by Heavy Ion Experiments and Lattice QCD Atsushi Nakamura Far

Study of QCD Phase Diagram by Heavy Ion Experiments and Lattice QCD Atsushi Nakamura Far Eastern Federal University &quot;Theory of Hadronic Matter under Extreme Conditions, Lab. of Theor. Phys. , Moscow April 18, 2016 Plan of the

1.09k views • 92 slides
New perspectives for heavy flavour physics from the lattice A Rainer Sommer LPHA Collaboration

New perspectives for heavy flavour physics from the lattice A Rainer Sommer LPHA Collaboration

New perspectives for heavy flavour physics from the lattice A Rainer Sommer LPHA Collaboration DESY, A Research Centre of the Helmholtz Association Les rencontres de Moriond, March 2009 Rainer Sommer New perspectives for heavy flavour

638 views • 41 slides
Lattice optimization for low charge Lattice optimization for low charge state heavy ion operation

Lattice optimization for low charge Lattice optimization for low charge state heavy ion operation

Lattice optimization for low charge Lattice optimization for low charge state heavy ion operation state heavy ion operation state heavy ion operation state heavy ion operation Collimation concepts for beam ions Collimation concepts for beam

396 views • 22 slides
Lattice QCD &amp; the search for BSM physics in beauty Matthew Wingate DAMTP, University of

Lattice QCD &amp; the search for BSM physics in beauty Matthew Wingate DAMTP, University of

Lattice QCD &amp; the search for BSM physics in beauty Matthew Wingate DAMTP, University of Cambridge Outline Quark flavour Peering through the glue to study electroweak symmetry breaking Lattice QCD Uniting the gauge theory,

711 views • 42 slides
Heavy Urban Search and Rescue (HUSAR), Emergency Exercises &amp; Joint Emergency Preparedness

Heavy Urban Search and Rescue (HUSAR), Emergency Exercises &amp; Joint Emergency Preparedness

Heavy Urban Search and Rescue (HUSAR), Emergency Exercises &amp; Joint Emergency Preparedness Program (JEPP) Funding Agenda Update on the Johnsons Landing Heavy Urban Search and Rescue Deployment Overview of Upcoming Training

333 views • 30 slides
Hierarchical Pattern Discovery in Stochastic Lattice Systems Markos Katsoulakis University of

Hierarchical Pattern Discovery in Stochastic Lattice Systems Markos Katsoulakis University of

Hierarchical Pattern Discovery in Stochastic Lattice Systems Markos Katsoulakis University of Crete &amp; FORTH, Heraklion, Greece, and University of Massachusetts, Amherst, USA Joint work with: -Petr Plech a c, Mathematics, University

484 views • 31 slides
Heavy hadron interactions from Lattice QCD Daniel Mohler Wien, Sep 14th, 2017 Daniel Mohler

Heavy hadron interactions from Lattice QCD Daniel Mohler Wien, Sep 14th, 2017 Daniel Mohler

Heavy hadron interactions from Lattice QCD Daniel Mohler Wien, Sep 14th, 2017 Daniel Mohler (HIM) Heavy hadron interactions from Lattice QCD Wien, Sep 14th, 2017 1 / 29 Observation of a doubly-charmed ++ cc by LHCb Roel Aaij et al. , LHCb

869 views • 42 slides
Heavy Quark Diffusion from the Lattice Viljami Leino Technische Universitt Mnchen, t30f In

Heavy Quark Diffusion from the Lattice Viljami Leino Technische Universitt Mnchen, t30f In

Heavy Quark Diffusion from the Lattice Viljami Leino Technische Universitt Mnchen, t30f In collaboration with: Nora Brambilla, Saumen Datta, Miguel Escobedo, Peter Petreczky, Antonio Vairo, and Peter Vander Griend From Euclidean spectral

731 views • 29 slides
Quark composition and color structure of heavy-heavy mesons and tetraquarks Asia-Pacific

Quark composition and color structure of heavy-heavy mesons and tetraquarks Asia-Pacific

Quark composition and color structure of heavy-heavy mesons and tetraquarks Asia-Pacific Symposium for Lattice Field Theory Marc Wagner Goethe-Universit at Frankfurt, Institut f ur Theoretische Physik mwagner@itp.uni-frankfurt.de

678 views • 18 slides
Informed Search and Exploration Sections 3.5 and 3.6 Ch. 03 p.1/47 Outline Best-first

Informed Search and Exploration Sections 3.5 and 3.6 Ch. 03 p.1/47 Outline Best-first

Informed Search and Exploration Sections 3.5 and 3.6 Ch. 03 p.1/47 Outline Best-first search A search Heuristics, pattern databases IDA search (Recursive Best-First Search (RBFS), MA and SMA search) Ch. 03 p.2/47

1.29k views • 47 slides

More recommend