Towards a Methodology for Benchmarking Edge Processing Frameworks - PowerPoint PPT Presentation

Towards a Methodology for Benchmarking Edge Processing Frameworks Pedro Silva, Alexandru Costan, Gabriel Antoniu Inria, IRISA France Invited Talk, BenchCouncil’19, Denver, November 2019

Data Shifts to the Edge By 2022 Gartner predicts that 75% of enterprise-generated data will be created and processed outside of the data center and cloud infrastructures compared with 10% today. Source: Smarter with Gartner, What Edge Computing Means for Infrastructure and Operations, October 3, 2018 Extract from: BullSequana Edge positioning paper (Atos) 2

3

4

Why Edge Processing? EDGE Advantages q Easier access to data DATA q Bandwidth saving FOG q Privacy q High potential parallelism DATA CLOUD / DC 5

Edge Processing Tools q Custom software EDGE q Generic frameworks DATA q Apache Edgent q Amazon Greengrass FOG q Azure Stream Analytics q IBM Watson IoT DATA q Intel IoT q Oracle Edge Analytics q … CLOUD / DC 6

Edge Processing Tools Are Great! J EDGE DATA FOG DATA CLOUD / DC 7

How Great? EDGE DATA What is their performance? FOG Under which conditions? Do they integrate well with my app? DATA CLOUD / DC 8

We Need Benchmarking! EDGE DATA Goal: Understand performance FOG DATA CLOUD / DC 9

Benchmarking: Questions q Are the cost models precise? EDGE q What is the impact of networking on the performance? q How do my algorithms react to real-time scenarios? q How does my hybrid approach compare to a fully centralized solution? FOG q SILVA, P., COSTAN A. and ANTONIU, G., Towards a Methodology for Benchmarking Edge Processing Frameworks. 1st Workshop on Parallel AI and Systems for the CLOUD Edge (PAISE workshop collocated with IPDPS 2019). 10

Benchmarking Platform: Objectives q Benchmark complete scenarios q Control network characteristics q Control framework configuration parameters q Control Edge, Fog and Cloud infrastructures 11

Benchmarking Edge Processing: Related Work q TPCx-IoT q Created for hardware benchmarking q Fog oriented q Academic benchmarks q Difficult to reproduce q Lack of a clear methodology (metrics, workloads, parameters) q Not focused on the tools 12

Benchmarking Edge Processing Tools q Edge/Fog data processing tools q Processing performance workload q Supported programming languages q Connectivity data metrics q Development easiness transmission q Use cases processing q Overall application performance q Viability on different infrastructure configurations … 13

Benchmarking Edge Processing Tools: Zoom … … Cloud … Fog 14 Edge

Benchmarking Edge Processing Tools : Parameters Network : Network : Bandwidth Bandwidth Workloads : Loss Loss CCTV Latency Latency NYC Taxi … EEW … Fog : Cloud Edge : Cloud : MQTT server Processing Kafka + + processing tools Flink tools … Fog 15 Edge

Benchmarking Edge Processing Tools: Metrics Latency Edge to Fog Latency Fog to Cloud Processing Latency … Each component has a resource utilization log. … Cloud Throughput Throughput … Fog 16 Edge

Benchmarking Platform: Implementation q Experiment manager Python / q Configures the infrastructure Execo / EnosLib q Deploys frameworks/tools Grid5K q Deploys applications and manages their Infrastructure executions Experiment Manager q Monitors resource usage Bare q Gathers metrics and logs enoslib VMs / Containers Metal q Edge+Fog+Cloud processing management app q Wrappers/interfaces Edge Fog Cloud stack q Metric generation, configuration, connection 17

Earthquake Early Warning Systems (EEW) Warning broadcaster Seismometer Data center P-wave Data upload 18

Earthquake Early Warning Systems (EEW) Data Warning q Deem: hierarchical and distributed ML algorithm q Enables the usage of multiple types of sensors … … q Enables the deployment on … less powerful networks q Enables local decision Scientific Intermediate machines with Broadcasting users Centralized data center making. Instruments computing capabilities q FAUVEL, K. ; BALOUEK-THOMERT, D. ; MELGAR, D. ; SILVA, P., SIMONET, A. ; ANTONIU G. ; Deem: local decision COSTAN, A ; MASSON, V ; PARASHAR, M. ; RODERO, I. ; TERMIER, A. A Distributed Multi-Sensor Machine Learning Approach to Earthquake Early Warning. Just accepted at AAAI 2020. Deem: global decision q SILVA, P., BALOUEK-THOMERT, D.; FAUVEL, K. ; MELGAR, D. ; SIMONET, A. ; ANTONIU G. ; COSTAN, A ; MASSON, V ; PARASHAR, M. ; RODERO, I. ; TERMIER, A A hybrid Fog and Cloud computing based approach for Earthquake Early Warning Systems. (In preparation.) 19

EEW: Fog-Based Infrastructure q Thousands of producers q High load on Fog and Cloud q Objectives q Reduction of network costs q Reduction of Cloud costs q Easier network reconfiguration (intelligent fog nodes) 20

Next Steps q Improve the benchmark prototype q Experiment with the EEW scenario q Integrate extra scenarios and use cases (e.g., DL-based) Thank you!