VAST A Unified Platform for Interactive Network Forensics Matthias - PowerPoint PPT Presentation

VAST A Unified Platform for Interactive Network Forensics Matthias Vallentin 1 , 2 Vern Paxson 1 , 2 Robin Sommer 2 , 3 1 UC Berkeley 2 International Computer Science Institute (ICSI) 3 Lawrence Berkeley National Laboratory (LBNL) March 17, 2016 USENIX NSDI 1 / 28

Omnipresent Data Breaches 2 / 28

Breach Timeline Detection Compromise Forensics Time 3 / 28

Breach Timeline Detection Compromise Time 3 / 28

Breach Timeline Detection ? Compromise Time 3 / 28

Network Forensics — Characteristics 4 / 28

Network Forensics — Characteristics Organization 4 / 28

Network Forensics — Characteristics 4 / 28

Network Forensics — Characteristics ? 4 / 28

Network Forensics — Characteristics Interactive data exploration ◮ Iterative query refinement ◮ High-dimensional search ? 4 / 28

Network Forensics — Characteristics Interactive data exploration ◮ Iterative query refinement ◮ High-dimensional search Disparate data access ◮ Temporal ◮ Spatial ? 4 / 28

Network Forensics — Characteristics Interactive data exploration ◮ Iterative query refinement ◮ High-dimensional search Disparate data access ◮ Temporal ◮ Spatial Massive data volumes ? ◮ 50–100K events/sec ◮ 10s TBs/day 4 / 28

Log Example — Bro Connection Log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path conn #open 2016-01-06-15-28-58 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_.. #types time string addr port addr port enum string interval count count string bool bool count string 1258531.. Cz7SRx3.. 192.168.1.102 68 192.168.1.1 67 udp dhcp 0.163820 301 300 SF - - 0 Dd 1 329 1 328 (empty) 1258531.. CTeURV1.. 192.168.1.103 137 192.168.1.255 137 udp dns 3.780125 350 0 S0 - - 0 D 7 546 0 0 (empty) 1258531.. CUAVTq1.. 192.168.1.102 137 192.168.1.255 137 udp dns 3.748647 350 0 S0 - - 0 D 7 546 0 0 (empty) 1258531.. CYoxAZ2.. 192.168.1.103 138 192.168.1.255 138 udp - 46.725380 560 0 S0 - - 0 D 3 644 0 0 (empty) 1258531.. CvabDq2.. 192.168.1.102 138 192.168.1.255 138 udp - 2.248589 348 0 S0 - - 0 D 2 404 0 0 (empty) 1258531.. CViJEOm.. 192.168.1.104 137 192.168.1.255 137 udp dns 3.748893 350 0 S0 - - 0 D 7 546 0 0 (empty) 1258531.. CSC2Hd4.. 192.168.1.104 138 192.168.1.255 138 udp - 59.052898 549 0 S0 - - 0 D 3 633 0 0 (empty) 1258531.. Cd3RNm1.. 192.168.1.103 68 192.168.1.1 67 udp dhcp 0.044779 303 300 SF - - 0 Dd 1 331 1 328 (empty) 1258531.. CEwuIl2.. 192.168.1.102 138 192.168.1.255 138 udp - - - - S0 - - 0 D 1 229 0 0 (empty) 1258532.. CXxLc94.. 192.168.1.104 68 192.168.1.1 67 udp dhcp 0.002103 311 300 SF - - 0 Dd 1 339 1 328 (empty) 1258532.. CIFDQJV.. 192.168.1.102 1170 192.168.1.1 53 udp dns 0.068511 36 215 SF - - 0 Dd 1 64 1 243 (empty) 1258532.. CXFISh5.. 192.168.1.104 1174 192.168.1.1 53 udp dns 0.170962 36 215 SF - - 0 Dd 1 64 1 243 (empty) 1258532.. CQJw4C3.. 192.168.1.1 5353 224.0.0.251 5353 udp dns 0.100381 273 0 S0 - - 0 D 2 329 0 0 (empty) 1258532.. ClfEd43.. fe80::219:e3ff:fee7:5d23 5353 ff02::fb 5353 udp dns 0.100371 273 0 S0 - - 0 D 2 369 0 0 1258532.. C67zf02.. 192.168.1.103 137 192.168.1.255 137 udp dns 3.873818 350 0 S0 - - 0 D 7 546 0 0 (empty) 1258532.. CG1FKF1.. 192.168.1.102 137 192.168.1.255 137 udp dns 3.748891 350 0 S0 - - 0 D 7 546 0 0 (empty) 1258532.. CNFkeF2.. 192.168.1.103 138 192.168.1.255 138 udp - 2.257840 348 0 S0 - - 0 D 2 404 0 0 (empty) 1258532.. Cq4eis4.. 192.168.1.102 1173 192.168.1.1 53 udp dns 0.000267 33 497 SF - - 0 Dd 1 61 1 525 (empty) 1258532.. CHpqv31.. 192.168.1.102 138 192.168.1.255 138 udp - 2.248843 348 0 S0 - - 0 D 2 404 0 0 (empty) 1258532.. CFoJjT3.. 192.168.1.1 5353 224.0.0.251 5353 udp dns 0.099824 273 0 S0 - - 0 D 2 329 0 0 (empty) 1258532.. Cc3Ayyz.. fe80::219:e3ff:fee7:5d23 5353 ff02::fb 5353 udp dns 0.099813 273 0 S0 - - 0 D 2 369 0 0 5 / 28

Existing Solutions MapReduce (Hadoop) ✓ Scalability ✗ Batch-oriented: no iterative, exploratory analysis 6 / 28

Existing Solutions MapReduce (Hadoop) ✓ Scalability ✗ Batch-oriented: no iterative, exploratory analysis In-Memory Cluster Computing (Spark) ✓ Efficient & complex analysis ✗ Thrashing when working set does not fit in aggregate memory 6 / 28

Contribution VAST V isibility A cross S pace and T ime 7 / 28

Contribution VAST V isibility A cross S pace and T ime Architecture ◮ Performance : concurrent & modular design ◮ Scaling : intra-machine & inter-machine ◮ Typing : strong & rich 7 / 28

Contribution VAST V isibility A cross S pace and T ime Architecture ◮ Performance : concurrent & modular design ◮ Scaling : intra-machine & inter-machine ◮ Typing : strong & rich Implementation ◮ Composition : high-level bitmap indexing framework ◮ Adaptation : fine-grained component flow-control ◮ Asynchrony : finite state machines for query execution 7 / 28

Outline 1. Architecture 2. Implementation 3. Evaluation

VAST Architecture — Single Machine 8 / 28

VAST Architecture — Single Machine node archive 10.0.0.1 10.0.0.254 53/udp 10.0.0.2 10.0.0.254 80/tcp source importer exporter sink index 8 / 28

VAST Architecture — Ingestion 10.0.0.1 53/udp 10.0.0.2 80/tcp … generate event batch source meta type 10.0.0.1 53/udp meta type 10.0.0.2 80/tcp 9 / 28

VAST Architecture — Ingestion 10.0.0.1 53/udp assign IDs 10.0.0.2 80/tcp … generate event batch source importer meta type 10.0.0.1 53/udp meta type 10.0.0.2 80/tcp 9 / 28

VAST Architecture — Ingestion archive compress batch 10.0.0.1 53/udp assign IDs 10.0.0.2 80/tcp … generate event batch source importer meta type 10.0.0.1 53/udp meta type 10.0.0.2 80/tcp 9 / 28

VAST Architecture — Ingestion archive compress batch 10.0.0.1 53/udp assign IDs 10.0.0.2 80/tcp … generate event batch source importer meta type 10.0.0.1 53/udp meta type 10.0.0.2 80/tcp index 9 / 28

VAST Architecture — Ingestion archive compress batch 10.0.0.1 53/udp assign IDs 10.0.0.2 80/tcp … append data to bitmap index generate event batch type source importer 10.0.0.1 53/udp 10.0.0.2 80/tcp meta type 10.0.0.1 53/udp meta type 10.0.0.2 80/tcp index 9 / 28

VAST Architecture — Index index meta index partition partition partition 10 / 28

VAST Architecture — Index index meta index partition partition partition conn 10.0.0.2 53/udp 8.8.4.4 53/udp “dns” indexer 10 / 28

VAST Architecture — Querying exporter X in 10.0.0.0/8 || X == 80/tcp 11 / 28

VAST Architecture — Querying exporter X in 10.0.0.0/8 || X == 80/tcp index 11 / 28

VAST Architecture — Querying lookup bit vectors from partitions exporter X in 10.0.0.0/8 X in 10.0.0.0/8 || X == 80/tcp _ index X == 80/tcp 11 / 28

VAST Architecture — Querying archive locate & ship event batch for ID lookup bit vectors from partitions exporter X in 10.0.0.0/8 X in 10.0.0.0/8 || X == 80/tcp _ index X == 80/tcp 11 / 28

VAST Architecture — Querying archive decompress locate & ship batch event batch for ID candidate check lookup bit vectors from partitions exporter X in 10.0.0.0/8 X in 10.0.0.0/8 || X == 80/tcp _ index X == 80/tcp 11 / 28

VAST Architecture — Querying archive decompress locate & ship batch event batch for ID candidate check lookup bit vectors from partitions exporter sink X in 10.0.0.0/8 X in 10.0.0.0/8 || X == 80/tcp _ index X == 80/tcp 11 / 28

VAST Architecture — Querying archive decompress locate & ship batch event batch for ID candidate check lookup bit vectors from partitions exporter sink X in 10.0.0.0/8 meta type 10.0.0.1 53/udp X in 10.0.0.0/8 meta type 10.0.0.2 80/tcp || X == 80/tcp render results _ 10.0.0.1 53/udp index 10.0.0.2 80/tcp … X == 80/tcp 11 / 28

VAST Architecture — Distributed 12 / 28

Outline 1. Architecture 2. Implementation 3. Evaluation

Indexing Basics — Tree Indexes 13 / 28

Indexing Basics — Composition ( ) _ _ 14 / 28

Indexing Basics — Inverted Index A B C D 1 0 2 2 3 5 4 4 5 8 6 9 0 1 2 3 4 5 6 7 8 9 15 / 28

Indexing Basics — Bitmap Index A B C D 0 0 1 0 0 1 1 0 0 0 2 0 0 1 1 3 1 0 0 0 1 0 1 0 4 5 0 1 1 0 0 1 2 3 4 5 6 7 8 9 16 / 28

VAST A Unified Platform for Interactive Network Forensics Matthias - PowerPoint PPT Presentation

VAST A Unified Platform for Interactive Network Forensics Matthias Vallentin 1 , 2 Vern Paxson 1 , 2 Robin Sommer 2 , 3 1 UC Berkeley 2 International Computer Science Institute (ICSI) 3 Lawrence Berkeley National Laboratory (LBNL) March 17, 2016

IAMI / VAST, Vietnam Grid enabled Medical Informatics In Vietnam (VAST/IAMI, formerly VAST/

RBINS VAST GTI collaboration A 10 years+ success story! RBINS and 2 VAST institutions:

VAST CHALLENGE 2017 Bianca Barnucz & Stephanie Wegscheidl OVERVIEW VAST Challenge

Nov. 5-16, 2007, IoIT (VAST) ACGRID Denis Perret-Gallix IoIT (VAST) Hanoi IN2P3/CNRS Nov. 5

VAST s 52nd Annual Meeting 2019 Meeting Description & Overview Health of the

VAST: Visibility Across Space and Time Architecture & Usage Matthias Vallentin

WORK IN THE GIG ECONOMY Huma Humans a ns as a s a Se Service rvice @JeremiasPrassl VAST

North-East Visualization and Analytics Center (NEVAC) VAST Grand Challenge Award: Data

U N I V E R S A L S T O R A G E I N T R O D U C T I O N M A Y 2 0 1 9 VAST COMPANY TI MELI NE

A Presentation by About Avalon AVALON GROUP Avalon has a vast experience of almost three

1 Examples of protein functionality: Enzymatic catalysis vast majority of reactions

D I S C O V E R A vast choice of activities, excursions and adventures awaits Almyra guests, all

FOUR|PARK MANAGEMENT, LLC An Introduction and Guided Discussion EXECUTIVE SUMMARY VAST

How Will We Populate the Semantic Web on a Vast Scale? Tom M. Mitchell Weam AbuZaki, Justin

the private and multidisciplinary Services public sectors backgrounds Vast experience and

1 Ocean Green Energy MICROALGAE : A VAST SOURCE OF PRODUCTIVE ORGANISMS A MIXOTHOPHIC

Disclosure Mark Ashley DAC Beachcroft LLP Disclosure: Overview A vast topic! What is

Motivations Automated Text Mining and Vast Quantities of Text Available Knowledge Discovery

Introduction Vast transistor budgets, but .... Poor interconnect scaling Pressure to

VAST MIGRATIONS AND THE COLLAPSE OF POPULATIONS: GLOBALISATION, CLIMATE CHANGE AND CLIMATE

Welfare Analysis for Behavioral Models Vast evidence on people holding wrong beliefs and

Anaerobic and Aerobic digestion 1 Introduction Organic matter is the vast array of carbon

B Y 1 About At Ohh Deer we work with a vast catalogue of talented artists but we wanted the

EXERCISES EXERCISES Important Perfectly safe for the vast majority of people Those with

VAST A Unified Platform for Interactive Network Forensics Matthias - PowerPoint PPT Presentation

VAST A Unified Platform for Interactive Network Forensics Matthias Vallentin 1 , 2 Vern Paxson 1 , 2 Robin Sommer 2 , 3 1 UC Berkeley 2 International Computer Science Institute (ICSI) 3 Lawrence Berkeley National Laboratory (LBNL) March 17, 2016

IAMI / VAST, Vietnam Grid enabled Medical Informatics In Vietnam (VAST/IAMI, formerly VAST/

RBINS VAST GTI collaboration A 10 years+ success story! RBINS and 2 VAST institutions:

VAST CHALLENGE 2017 Bianca Barnucz &amp; Stephanie Wegscheidl OVERVIEW VAST Challenge

Nov. 5-16, 2007, IoIT (VAST) ACGRID Denis Perret-Gallix IoIT (VAST) Hanoi IN2P3/CNRS Nov. 5

VAST s 52nd Annual Meeting 2019 Meeting Description &amp; Overview Health of the

VAST: Visibility Across Space and Time Architecture &amp; Usage Matthias Vallentin

WORK IN THE GIG ECONOMY Huma Humans a ns as a s a Se Service rvice @JeremiasPrassl VAST

North-East Visualization and Analytics Center (NEVAC) VAST Grand Challenge Award: Data

U N I V E R S A L S T O R A G E I N T R O D U C T I O N M A Y 2 0 1 9 VAST COMPANY TI MELI NE

A Presentation by About Avalon AVALON GROUP Avalon has a vast experience of almost three

1 Examples of protein functionality: Enzymatic catalysis vast majority of reactions

D I S C O V E R A vast choice of activities, excursions and adventures awaits Almyra guests, all

FOUR|PARK MANAGEMENT, LLC An Introduction and Guided Discussion EXECUTIVE SUMMARY VAST

How Will We Populate the Semantic Web on a Vast Scale? Tom M. Mitchell Weam AbuZaki, Justin

the private and multidisciplinary Services public sectors backgrounds Vast experience and

1 Ocean Green Energy MICROALGAE : A VAST SOURCE OF PRODUCTIVE ORGANISMS A MIXOTHOPHIC

Disclosure Mark Ashley DAC Beachcroft LLP Disclosure: Overview A vast topic! What is

Motivations Automated Text Mining and Vast Quantities of Text Available Knowledge Discovery

Introduction Vast transistor budgets, but .... Poor interconnect scaling Pressure to

VAST MIGRATIONS AND THE COLLAPSE OF POPULATIONS: GLOBALISATION, CLIMATE CHANGE AND CLIMATE

Welfare Analysis for Behavioral Models Vast evidence on people holding wrong beliefs and

Anaerobic and Aerobic digestion 1 Introduction Organic matter is the vast array of carbon

B Y 1 About At Ohh Deer we work with a vast catalogue of talented artists but we wanted the

EXERCISES EXERCISES Important Perfectly safe for the vast majority of people Those with

VAST CHALLENGE 2017 Bianca Barnucz & Stephanie Wegscheidl OVERVIEW VAST Challenge

VAST s 52nd Annual Meeting 2019 Meeting Description & Overview Health of the

VAST: Visibility Across Space and Time Architecture & Usage Matthias Vallentin