Sensitivity Analysis of Network Performance Models Final talk for - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Sensitivity Analysis of Network Performance Models Final talk for the Bachelor’s Thesis by Niklas Beck advised by Max Helm, Henning Stubbe Wednesday 17 th June, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Background Sensitivity Analysis (SA) • determines sensitivity of parameters • effect of input parameters on the output of a model • purposes: • model validation • investigating model behavior • model optimization Niklas Beck — SA 2

Background Sensitivity Analysis (SA) model-based formula-based local • systematically evaluating model • uses network formula • change one parameter after another • differentiation of network formula global • simultaneous variation of multiple input parameters • full exploration of complete input space (to a feasible granularity) Niklas Beck — SA 3

Background Network Calculus (NC) Model Niklas Beck — SA 4

Background NC Analysis Methods • Total Flow Analysis (TFA) • Seperate Flow Analysis (SFA) • Pay Multiplexing Only Once Analysis (PMOOA) • Tandem Matching Analysis (TMA) Niklas Beck — SA 5

Related Work Sensitivity Analysis for Map/Map/1 Queues [2] • queueing models (QM) are very similar to NC models • parameters (e.g. service times (QM) - service curve (NC)) • performance measures (e.g. number of customers in the system (QM) - backlog bound (NC)) • only local sensitivity analysis was performed • investigated only one specific queuing model This thesis: local and global SA performed with 16 different network calculus models Niklas Beck — SA 6

Implementation Model Creation • NC models are built with a deterministic network calculator (DiscoDNC [1]) • restricted to feed-forward networks • tandem model • random model • generate an Internet-like random graph with a customized Barabási-Albert algorithm • custom extension of DiscoDNC needed to parse and build NC model Niklas Beck — SA 7

Random Internet-like Network Graph Niklas Beck — SA 8

Random Internet-like Network Graph Niklas Beck — SA 9

Random Internet-like Network Graph Niklas Beck — SA 10

Implementation Local SA • model-based • runs model with altering input parameters • isolated evaluation • formula-based • differentiation with SymPy Global SA • model independent open source Python library: SALib [3] • SALib implements varius methods for global SA • Sobol Sensitivity Analysis [5] • Fourier Amplitude Sensitivity Test (FAST) [4] • computes sensitivity indices from model outputs All results can be automatically plotted with MatPlotLib Niklas Beck — SA 11

Evaluation Model Parameters Parameter Description Abbreviation Number of Network Nodes n Service Curve Rate scr Service Curve Latency scl Usage of Maximum Service Curve umsc Maximum Service Curve Rate mscr Maximum Service Curve Latency mscl Arrival Curve Rate acr Arrival Curve Burst acb Number of Cross-Traffic Flows xf Niklas Beck — SA 12

Local Sensitivity Values: Tandem Network - Delay Bound Evaluation Sensitivity Value (Delay Bound) 10 12 0 2 4 6 8 0.0164 0.01 n 0.01 0.01 -6.7e-09 -2.8e-09 scr -1.4e-09 -1.4e-09 8.519 6.5 scl 5.75 5.75 0.0 mscr 0.0 0.0 0.0 5.2e-08 2e-08 acr 1e-08 1e-08 1.5e-06 8.7e-07 acb 3.7e-07 3.75e-07 0.022 Niklas Beck — SA 0.0157 xf TMA PMOOA SFA TFA 0.0073 0.0073 13

Evaluation Local Sensitivity Analysis: Tandem Network - Delay Bound 1.2 TFA TFA 0.16 SFA SFA 1.0 PMOOA PMOOA TMA TMA 0.14 0.8 Delay Bound Delay Bound 0.12 0.6 0.10 0.4 0.08 0.2 0.06 0.0 0 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Number of Network Nodes Service Curve Rate 1e8 TFA TFA 4 SFA SFA 0.8 PMOOA PMOOA TMA TMA 3 0.6 Delay Bound Delay Bound 2 0.4 1 0.2 0 0.0 0.1 0.2 0.3 0.4 0.5 0 2 4 6 8 Service Curve Latency Number of Cross-Traffic Flows Niklas Beck — SA 14

Evaluation Local Sensitivity Analysis: Random Network - Delay Bound 12 TFA TFA 0.8 SFA SFA 10 0.7 PMOOA PMOOA TMA TMA 0.6 8 Delay Bound Delay Bound 0.5 6 0.4 4 0.3 2 0.2 0.1 0 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.2 0.4 0.6 0.8 1.0 1e5 Service Curve Latency Arrival Curve Burst • inaccuracy of curve computation in DiscoDNC • computational error of a single server node leads to flawed performance bounds • worst case: could lead to false results for local SA Niklas Beck — SA 15

Evaluation Global Sensitivity Indices: Tandem Network - Delay Bound 1.0 1.0 First First Total Total 0.8 0.8 0.5773 0.5489 0.5109 0.4864 Sensitivity Index Sensitivity Index 0.6 0.6 0.4179 0.3959 0.3278 0.3147 0.4 0.4 0.1818 0.1659 0.1529 0.2 0.2 0.0433 0.0338 0.0187 0.0184 0.045 0.0094 0.0113 0.0107 0.037 0.0077 0.0075 0.0032 0.0026 0.0014 0.0022 0.0026 0.006 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 n scr umsc mscr acr n scr umsc mscr acr scl mscl acb xf scl mscl acb xf Total Flow Analysis Seperate Flow Analysis Niklas Beck — SA 16

Global Sensitivity Indices: Tandem Network - Backlog Bound Evaluation Sensitivity Index 0.0 0.2 0.4 0.6 0.8 1.0 0.2138 TFA first- and total-order indices n 0.4004 0.0 scr 0.0 0.226 scl 0.4184 umsc 0.0 0.0 mscr 0.0 0.0 mscl 0.0 0.0 0.2094 acr 0.3916 0.0 acb 0.0 0.0553 Total First xf 0.1231 Sensitivity Index 0.0 0.1 0.2 0.3 0.4 0.5 n-scr 0.005721 n-scl 0.072762 n-umsc 0.005715 n-mscr 0.005721 TFA second-order indices n-mscl 0.005735 n-acr 0.074852 n-acb 0.005728 n-xf 0.022218 scr-scl 0.0 scr-umsc 0.0 scr-mscr 0.0 scr-mscl 0.0 scr-acr 0.0 scr-acb 0.0 scr-xf 0.0 scl-umsc 0.000956 scl-mscr 0.00095 scl-mscl 0.000959 scl-acr 0.067554 scl-acb 0.000947 scl-xf 0.021081 umsc-mscr 2.1e-05 umsc-mscl 2.2e-05 Niklas Beck — SA umsc-acr 2.9e-05 umsc-acb 2.1e-05 umsc-xf 5e-06 mscr-mscl 0.0 mscr-acr 0.0 mscr-acb 0.0 mscr-xf 0.0 mscl-acr 4.5e-05 mscl-acb 2.9e-05 Second mscl-xf 2.2e-05 acr-acb -0.00261 acr-xf 0.015113 acb-xf 5.4e-05 17

Conclusion • sensitivity analysis performed • local (formula-based - model-based) • global (Sobol - FAST) • 16 different network calculus models investigated • performance measure (delay bound - backlog bound) • analysis methods (TFA, SFA, PMOOA, TMA) • real-world applicability through Internet-like network topologies • comprehensive sensitivity analysis Python framework provided Niklas Beck — SA 18

Bibliography [1] S. Bondorf and J. B. Schmitt. The DiscoDNC v2 – a comprehensive tool for deterministic network calculus. In Proc. of the International Conference on Performance Evaluation Methodologies and Tools , ValueTools ’14, pages 44–49, December 2014. [2] A. Heindl. Sensitivity analysis for map/map/1 queues. pages 235–244, 01 2004. [3] J. Herman and W. Usher. Salib: An open-source python library for sensitivity analysis. Journal of Open Source Software , 2(9):97, 2017. [4] A. Saltelli, S. Tarantola, and K. P .-S. Chan. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics , 41(1):39–56, 1999. [5] I. Sobol. Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and Computers in Simulation , 55(1):271 – 280, 2001. The Second IMACS Seminar on Monte Carlo Methods. Niklas Beck — SA 19

Appendix Local Sensitivity Analysis: Tandem Network - Backlog Bound 1e5 1e5 TFA 1.3 SFA 4 PMOOA 1.2 TMA 1.1 Backlog Bound Backlog Bound 3 TFA 1.0 SFA PMOOA 2 0.9 TMA 0.8 1 0.7 0.6 0 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1e8 Number of Network Nodes Service Curve Rate 1e6 1e5 TFA TFA 8 5 SFA SFA PMOOA PMOOA 7 TMA TMA 4 6 Backlog Bound Backlog Bound 5 3 4 2 3 1 2 1 0 0.0 0.1 0.2 0.3 0.4 0.5 0 2 4 6 8 Service Curve Latency Number of Cross-Traffic Flows Niklas Beck — SA 20

Appendix Local Sensitivity Analysis: Random Network - Backlog Bound 1e5 1e7 1.2 4.0 TFA TFA SFA SFA PMOOA 1.0 PMOOA 3.5 TMA TMA 3.0 0.8 Backlog Bound Backlog Bound 2.5 0.6 2.0 0.4 1.5 0.2 1.0 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.0 0.1 0.2 0.3 0.4 0.5 1e8 Service Curve Rate Service Curve Latency 1e5 1e6 TFA 1.0 TFA 5 SFA SFA PMOOA PMOOA 0.8 TMA TMA 4 Backlog Bound Backlog Bound 0.6 3 2 0.4 1 0.2 0 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 0.0 0.2 0.4 0.6 0.8 1.0 Arrival Curve Rate 1e6 Arrival Curve Burst 1e5 Niklas Beck — SA 21

Global Sensitivity Indices: Random Network - Delay Bound Appendix Sensitivity Index 0.0 0.2 0.4 0.6 0.8 1.0 1.2 SFA first- and total-order indices 0.0408 scr 0.0833 0.9046 scl 0.9252 umsc 0.0 0.0 0.0 mscr 0.0 mscl 0.0 0.0 0.009 acr 0.0387 0.0 acb Total First 0.0 Sensitivity Index 0.0 0.1 0.2 0.3 0.4 0.5 scr-scl 0.020407 scr-umsc 0.003284 scr-mscr 0.003278 SFA second-order indices scr-mscl 0.003273 scr-acr 0.031051 scr-acb 0.003286 scl-umsc -0.000543 scl-mscr -0.00054 scl-mscl -0.00054 scl-acr 0.000625 scl-acb -0.000515 umsc-mscr -5e-06 umsc-mscl -5e-06 Niklas Beck — SA umsc-acr -4e-06 umsc-acb -5e-06 mscr-mscl 0.0 mscr-acr 0.0 mscr-acb 0.0 mscl-acr -1e-06 Second mscl-acb -2e-06 acr-acb -0.001688 22