introduction
play

Introduction Harsh realities of network analytics netbeam - PowerPoint PPT Presentation

Sep 29, 2023 •227 likes •568 views

Introduction Harsh realities of network analytics netbeam Demo Technology Stack Alternative Approaches Lessons Learned 2 ESnet Data, Analytics and Visualization Architecture 3 The Harsh Realities of Network

  2. Introduction ● Harsh realities of network analytics netbeam ● ● Demo ● Technology Stack ● Alternative Approaches Lessons Learned ● 2

  3. ESnet Data, Analytics and Visualization Architecture 3

  4. The Harsh Realities of Network Analytics 1. It’s a mess 3. There’s always more ● Your data isn’t neat and tidy ● More devices & more telemetry 2. Things change 4. It’s never really done ● Time and money are limited ● What you need today may not be what you need tomorrow. 4

  5. Coping strategies 1. It’s a mess 3. There’s always more ● Design knowing things won’t ● Rely on the cloud for scaling be tidy 2. Things change 4. It’s never really done ● “What” not “How” ● Keep raw data to keep your options open 5

  6. netbeam Network Analytics in Google Cloud Three Pillars 1. Real time analytics ○ Low latency, incomplete 2. Offline analytics ○ High latency, complete 3. Flexible data model ○ Changing needs? Recompute from raw data! Secret sauce: Apache Beam 6

  7. What is Apache Beam? 1. The Beam Programming Model SDKs for writing Beam pipelines 2. 3. Runners for existing distributed processing backends Apache Apex ○ ○ Apache Flink Apache Spark ○ ○ Google Cloud Dataflow Local runner for testing ○ 7 Slide courtesy of the Apache Beam Project

  8. The Evolution of Apache Beam Colossus BigTable PubSub Dremel Google Cloud Dataflow Spanner Megastore Millwheel Flume Apache Beam MapReduce 8 Slide courtesy of the Apache Beam Project

  9. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream Processing) avro Bigtable Apache Beam BigQuery BigQuery (realtime) (Batch Processing) (immutable) (historical) API Client 9

  10. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Google Pubsub ● Uses Python outside Align/rates of Google Cloud to poll devices and write Bigtable Rollups BigQuery BigQuery (realtime) to Pubsub topic 5m, 1h, 1d avg (immutable) (historical) ● Code within Google Cloud subscribes to Percentiles topic to process data ... API Client 10

  11. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Stream processing ● Subscribes to Bigtable Rollups BigQuery BigQuery (realtime) Pubsub topic 5m, 1h, 1d avg (immutable) (historical) Percentiles ... API Client 11

  12. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Stream processing ● Subscribes to Bigtable Rollups BigQuery BigQuery (realtime) Pubsub topic 5m, 1h, 1d avg (immutable) (historical) ● Raw data is written to BigQuery Percentiles ... API Client 12

  13. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Stream processing ● Subscribes to Bigtable Rollups BigQuery BigQuery (realtime) Pubsub topic 5m, 1h, 1d avg (immutable) (historical) ● Raw data is written to BigQuery Percentiles ● Real time ... transformed data (e.g. aligned data rates) written to Bigtable API ● Writes and makes use of meta data in BigTable (not shown) Client 13

  14. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Cloud Bigtable ● Like HBase Align/rates ● Write to cells in rows, indexed by keys Bigtable Rollups BigQuery BigQuery (realtime) ● We write 1 day of 5m, 1h, 1d avg (immutable) (historical) data to a single row (columns are the time Percentiles of day, key is metric ... and day) ● Fast access to row by key, can serve data from here API ● Store one year Client 14

  15. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● BigQuery ● Data warehousing Align/rates solution ● Cheap storage, SQL Bigtable Rollups BigQuery BigQuery (realtime) access, but not 5m, 1h, 1d avg (immutable) (historical) suitable for real-time access Percentiles ● Allows SQL queries ... for ad hoc investigation ● We store our source of truth here API Client 15

  16. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● BigQuery ● Data warehousing Align/rates solution ● Cheap storage, SQL Bigtable Rollups BigQuery BigQuery (realtime) access, but not 5m, 1h, 1d avg (immutable) (historical) suitable for real-time access Percentiles ● Allows SQL queries ... for ad hoc investigation ● We store our source of truth here API ● Also store historical data (7 years), imported via avro files Client 16

  17. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) 5m, 1h, 1d avg (immutable) (historical) Percentiles ... API Client 17

  18. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) ● Recalculate Bigtable 5m, 1h, 1d avg (immutable) (historical) data each night from source of truth in Percentiles BigQuery ... API Client 18

  19. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) ● Recalculate Bigtable 5m, 1h, 1d avg (immutable) (historical) data each night from source of truth in Percentiles BigQuery ... ● Process Bigtable rows into new rows of 5min, 1 hr and 1 day aggregations API Client 19

  20. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) ● Recalculate Bigtable 5m, 1h, 1d avg (immutable) (historical) data each night from source of truth in Percentiles BigQuery ... ● Process Bigtable rows into new rows of 5min, 1 hr and 1 day aggregations API ● Additional pre-computed views e.g. percentiles for traffic distribution over a month Client 20

  21. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● API ● Currently runs on Align/rates App Engine ● Node.js Bigtable Rollups BigQuery BigQuery (realtime) ● Serves data out of 5m, 1h, 1d avg (immutable) (historical) Bigtable ● Timeseries data is Percentiles served as ‘tiles’, each ... tile is one row ● Would like to use Cloud Endpoints Dataserver API and provide a gRPC (node.js) service ● Looking forward to grpc-web solution Client 21

  22. Use case example: Historical Trends 22

  23. Use case example: Historical Trends Per-day SNMP collection Interface totals BigQuery Stream to BQ system Bigtable Old SNMP system avro BigQuery (historical) Per-month totals Bigtable rows Dataserver API (node.js) Jan 1 Jan 2 ... Dec31 snmp-daily::2017-08::$interface 1.8 Pb 1.9 Pb ... 3.1 Pb Client Jan 1991 Feb 1991 ... Sep 2017 snmp-monthly-totals 23 28 Gb 29 Gb ... 56 Pb

  24. Use case: real time anomaly detection Generates avg for each interface over the past 3 months for that hour/day SNMP collection Baseline BigQuery Stream to BQ system generation Bigtable Anomaly detection Compares baseline to real time values to generate current deviation from normal Dataserver API (node.js) Mon Mon Mon Sun ... 12am 1am 2am 11pm baseline::5m::avg::$interface 2.1 1.9 0.3 ... 0.5 Client iface-1 iface-2 ... iface-n anomaly::5m::avg +0.1 +2.0 ... -1.5 24

  25. Use case example: Percentiles 25

  26. Use case example: Percentiles Daily rollups 5m avg SNMP collection Stream to Bigtable Bigtable system Percentiles Bigtable rows Dataserver API 1 2 ... 8640 (node.js) rollup-month-5m::2017-08::$interface::in 6Gbps 5Gbps ... 2Gbps Client 1 pct 2 pct ... 99 pct 26 percentiles::2017-08::$interface::in 0.1 Gbps 0.3 Gbps ... 22.1Gbps

  27. Example: Computing Total Traffic # Python Beam SDK pipeline = beam.Pipeline('DirectRunner') (pipeline | 'read' >> ReadFromText('./example.csv') | 'csv' >> beam.ParDo(FormatCSVDoFn()) | 'ifName key' >> beam.Map(group_by_device_interface) | 'group by iface' >> beam.GroupByKey() | 'compute rate' >> beam.FlatMap(compute_rate) | 'timestamp key' >> beam.Map(lambda row: (row['timestamp'], row['rateIn'])) | 'group by timestamp' >> beam.GroupByKey() | 'sum by timestamp' >> beam.Map(lambda rates: (rates[0], sum(rates[1]))) | 'format' >> beam.Map(lambda row: '{},{}'.format(row[0], row[1])) | 'save' >> beam.io.WriteToText('./total_by_timestamp')) pipeline.run() Full code available at: http://x1024.net/blog/2017/05/chinog-flexible-network-analytics-in-the-cloud/ 27

Recommend

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

TRAINING COURSE ON SOCIAL PROTECTION & FORMALIZATION TRINIDAD AND TOBAGO MARCH 15, 2017 INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION Design of the NIS Assistance from the

547 views • 23 slides
INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

Patterns of variation in the expression of case and agreement Andrs Brny Leiden University Centre for Linguistics 9 November 2019, BaSIS Workshop a.barany@hum.leidenuniv.nl INTRODUCTION Introduction 2/42 INTRODUCTION

1.37k views • 42 slides
Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures Definition and concepts Process and product properties Principles SE Approaches Motivation Software and the economy The economies of

1.03k views • 49 slides
Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1 https://www.cdc.gov/std/stats17/infographic.htm Introduction Epidemiology in Western Colorado CPHE Surveillance Data Report, May 2018 Chlamydia Epidemiology in Western

977 views • 39 slides
DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every meeting (2) Some ideas of diagrams for section 7.1 (Introduction) of TP. Giles Barr 25 Jan 2018, Pembroke College On DUNE, the reality is there is a

474 views • 10 slides
INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

ADVANCED COMPUTER SECURITY INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT IS SECURITY? A systems security policies describe What the system is supposed to do Store and provide access

479 views • 16 slides
Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout World Wide

387 views • 14 slides
14 Introduction Introduction Bad guys can put malware into Example: hosts

14 Introduction Introduction Bad guys can put malware into Example: hosts

Introduction Introduction source Encapsulation ISO/OSI reference model message M application segment H t H t M transport network datagram H n H n H t M presentation: allow applications to frame link H l H n H t M interpret

583 views • 4 slides
Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout

322 views • 8 slides
Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Introduction Zynq Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL Processing System Processor Peripherals Data Buses AXI Bus Conclusion BTE5380 - Embedded Systems Octobre 2014 Andreas Habegger

1.81k views • 30 slides
2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04 Project References S E 5 PART.1 SMILE5 Introduction M I L Introduction STORION-SMILE5 Residential Energy Storage System Inverter 5kW Output Input

1.06k views • 25 slides
Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an application server? Why use an application server? Course introduction Services provided by CICS How CICS applications are defined Scaling CICS to meet

1.84k views • 146 slides
Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic building blocks Node Links (relationship between nodes) Spatial information Network data

584 views • 9 slides
Introduction Introduction (1) Main argument of the paper: universities are not only

Introduction Introduction (1) Main argument of the paper: universities are not only

Le jugement des pairs comme outil de management des universits Peer-review as a management tool Christine Musselin (CSO, Sciences Po et CNRS) Avril 2013, Lirrsistible ascension du capitalisme acadmique Introduction Introduction (1)

590 views • 28 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of the brand new JA-100 alarm system Visions Basic architecture System components Setting Help us with final validations You are the FIRST ones

1.69k views • 111 slides
1 Introduction Introduction Communication System Baseband an adjective that describes

1 Introduction Introduction Communication System Baseband an adjective that describes

Introduction Introduction Physical Layer Mobile network Global ISP Provides the means to transmit bits Basic Concepts of from sender to receiver, Data Transmission Defines the mechanical, electrical, and Home network timing

463 views • 7 slides
Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

INFOGR Computer Graphics Jacco Bikker - April-July 2015 - Lecture 1: Introduction Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical Details Assignments Field Study State of

869 views • 62 slides
Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Virtual and Augmented Reality Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction Who am I? What do I do here? 2 About me Short research bio: - 2014 - finished PhD at FMFI UK - 2014 2016 -

1.3k views • 39 slides
PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience for your presentation. It tells them what your presentation is about, why it is important, and finally, what to listen for in your speech. Get the

505 views • 3 slides
How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

Introduction to Beamer Hafida Benhidour How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer Introduction to Beamer Hafida Benhidour Department of computer science King Saud University November

732 views • 49 slides
Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Shenzhen Cuilu jewelry Co., Ltd

147 views • 14 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

735 views • 34 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

558 views • 18 slides

More recommend