NFV Testing Jose Lausuch (Ericsson) Trevor Cooper (Intel) Agenda - PowerPoint PPT Presentation

NFV Testing Jose Lausuch (Ericsson) Trevor Cooper (Intel)

Agenda NFV Testing with Open Source Projects Role of CI Performance Testing

Intro to OPNFV

OPNFV – testing and integration project Additions Enhancements

OPNFV – testing and integration project SDN PLATFORM COMPONENTS SFC Fault management BPGPVN FEATURES IPv6 Hugepages

OPNFV – testing and integration project OSS/BSS NFV orchestration NFV apps Cloud Management Infrastructure Virtualization Compute Storage Network

NFV Testing with Open Source Projects

Test Ecosystem in OPNFV

Types of testing Functional Performance Stress Security Upgradability Infrastructure Network Network Features Storage Storage Backup&Restore Components Compute Compute VNF Virtual layer … . MANO Traffic gen. . … … .

The concept of scenarios scenario X scenario Y scenario Z SFC

The role of CI Continuous Integration

Infrastructure – Distributed Pharos Labs 4/3/2017 LF Infra Collab

Pharos Architecture

Role of CI Google Cloud Docker Docker Hub Storage Hub Pull Down Pull Down Download Functest Image Yardstick Image ISO Labs connected 24x7 using JNLP Initiate Deploy Functest Yardstick Run for certain Scenario Jumphost Pharos POD (SUT) 4/3/2017 LF Infra Collab

CI workflow Google Cloud Storage Upload ISO Poll changes Patch Verified Clone repo +1/-1 Build yes new Build changes? Build Build Server Patchset Push Image Verification no Docker Hub

Working with upstream communities contribute contribute contribute contribute pull from master, deploy, test and/or verify patchset, post feedback

NFV Performance Testing 1. Meanings 2. Approaches 3. Tools 4. Examples 5. Ambitions “When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” … Lord Kelvin

Performance Testing Infrastructure Meanings of NFV Performance Testing Test Frameworks Test Dashboards Test Databases How do vendor A & B compare? Traffic Profiles Test Methodologies • Can the platform deliver required networking performance? KPIs / Metrics • Evaluating switching, acceleration technology, hardware platform, NFVI, VNF, full solution? Traffic Generators What Tools and Methods are needed? • We have relied on trusted proprietary tools • Traditional network performance methodologies are useful but not sufficient NFV => general purpose compute elements in a heterogeneous compute environment • Benchmarking virtual network functions is different to physical functions • System configurations are complex and few deployments are the same (test infrastructures also differ) Performance Testing Tools and Methods are not just for Developers and Test-Experts

COVERAGE SPEED ACCURACY RELIABILITY SCALABILITY Approaches to NFV Performance Testing Activation Operation System Under Test, Workloads and Stimuli De-activation • VNF / NFVI IETF NFV Benchmarking Internet Drafts • Pre-deployment / In-service Data Centre Benchmarking Methodology draft-ietf-bmwg-dcbench-methodology-02 • Topology / SW versions / configurations Data Centre Benchmarking Terminology draft-ietf-bmwg-dcbench-terminology-05 • Data Plane vs Control Plane + Data Plane Benchmarking Methodology for SDN Controller Performance draft-ietf-bmwg-sdn-controller-benchmark-meth-01 • Component / sub-system / platform / network service Terminology for Benchmarking SDN Controller Performance draft-ietf-bmwg-sdn-controller-benchmark-term-01 • Benchmarking VNFs and Their Infrastructure Traffic profiles, Workloads draft-ietf-bmwg-virtual-net-02 Considerations for Benchmarking High Availability of NFVI draft-kim-bmwg-ha-nfvi-01 • Deployment automation vs control Benchmarking Methodology for EVPN draft-kishjac-bmwg-evpntest-00 Test Objectives VNF Benchmarking Methodology draft-rosa-bmwg-vnfbench-00 • How Fast? How much Latency? How much Capacity? Benchmarking Virtual Switches in OPNFV draft-vsperf-bmwg-vswitch-opnfv-02 • How many Sessions? ETSI NFV Test Specifications • How Quick is Recovery? NFV Pre-deployment Testing (Spec) http://www.etsi.org/technologies-clusters/technologies/nfv Test Methods and Metrics NFV Interoperability Testing Methodology (Spec) • Throughput, back2back, frame-loss Use cases & recommendations for VNF Snapshot (Spec) • Packet and Frame Delay Distribution NFV Issue Tracker – to report feedback on NFV DRAFTs http://nfvwiki.etsi.org/index.php?title=NFV_Issue_Tracker • Scalability according to number of flows, active ports, etc. Path-implementations among NS Endpoints & VNFs https://docbox.etsi.org/isg/nfv/open/drafts/ • Stream Type with realistic profiles / flows: L2, L3, L4 Capture of VNF data and VNF/VNFC snapshots DevOps and CI/CD - test procedures and metrics • Control and Data Path Coupling Interop test guidelines VNF, MANO, VIM-NFVI • CPU and Memory Consumption Key performance NFVI metrics at different layers • “Soak” tests capture transient changes in performance

NFV Performance Testing Tools Test Infrastructure • https://wiki.opnfv.org/display/pharos • https://wiki.opnfv.org/display/pharos/Community+Labs • Video Tutorial: Introduction to Pharos Test Frameworks • Network Service Performance (MANO) • VNF Characterization (Control + Data Plane) • NFVI Performance (Data Plane) • Components and Sub-system Performance Traffic Generator • Staging Manager and Bottleneck Analysis IXIA (physical and virtual) Proprietary SW/HW Spirent (physical and virtual) Proprietary SW/HW • Analytics (post-process test results from CI/CD pipeline) Xena Proprietary SW/HW • Integrated statistics / events monitoring Moongen Open Source SW TREX Open Source SW Traffic Generators Pktgen Open Source SW LAN Tornado RFC 2544 Proprietary SW • Vary in measurement capability and suitability Seagull Open Source SW • Vary in complexity and cost PROX Open Source SW

NFV Test Tools - VSPERF Modular test framework, combining traffic generation, virtual switches, VNFs, and network configuration test cases IETF Internet Draft : https://tools.ietf.org/html/draft-ietf-bmwg-vswitch-opnfv-01 Test-cases : configure and program vSwitch, deploy and terminate VNFs, execute a traffic generator, modify a VSPERF configuration, execute external commands, etc. http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/user/userguide/teststeps.html Metrics : Throughput, Packet delay, Packet delay variation, Packet loss, Burst behavior, Packet re-ordering, Packet correctness, Availability and capacity of the DUT http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/developer/requirements/vswitchperf_ltd.html Results • OPNFV community lab resources https://wiki.opnfv.org/display/pharos/VSPERF+in+Intel+Pharos+Lab+-+Pod+12 • Scenario example http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/developer/results/scenario.html • CI daily results: https://build.opnfv.org/ci/view/vswitchperf/ https://build.opnfv.org/ci/view/vswitchperf/job/vswitchperf-daily-danube/lastBuild/console • Visualization (proto): http://testresults.opnfv.org/proto/index-vsperf.html Speed Accuracy Reliability Scale Activation • Address Learning Rates • Flow • RFC2544. • RFC2889. and Caching Capacities Addition System AddressCaching • Packet Processing RecoveryTime Capacity • RFC2544. Latencies and Variations ResetTime Operation • RFC2544.PacketLossRatio • Frame • RFC2889. • Scalability.RFC25 • RFC2544.PacketLossRate FrmMod Filtering Soak 44.0PacketLoss • RFC2889. • MemoryBandwid • RFC2544.BackToBack Frames SoakFrame th.RFC2544.0Pac • RFC2889.MaxForwarding Rate Modifcation ketLoss.Scalability • RFC2889.ForwardPressure • PacketDelay • RFC2889.BroadcastFrame Forwarding Variation. RFC3393. • RFC2889.BroadcastFrame Latencytest Soak • CPU.RFC2544.0PacketLoss • RFC2544.WorstN-BestN • InterPAcketDelayVariation RFC5481

NFV Test Tools - Yardstick Yardstick is used in for verifying the OPNFV infrastructure KPIs and some OPNFV features • Test configurations in YAML … parsed and converted into an internal model … converted into a Heat template and deployed • Scenario runners execute commands in VMs with SSH • Output written as json records to a file or influxdb, results shown with Grafana Test Cases : http://docs.opnfv.org/en/latest/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html Metrics : http://docs.opnfv.org/en/latest/submodules/yardstick/docs/testing/user/userguide/02-methodology.html#metrics Results : http://testresults.opnfv.org/grafana/

Test Example - TCP VM Performance 24.90 1.43 COLORADO (FUEL) LATEST (KOLLA) DANUBE 25

Test Example – vSwitch Flow Rules • OpenStack deployed with Fuel/Apex uses single logical tenant network (two tenant networks possible with DevStack/Kolla) • Two OVS bridges by default. Br-prv is outward facing bridge. Br-int is inner bridge where VMs are logically connected • VLAN segmentation means additional modify VLAN ID required to and from VMs • OVS flow rules are created by OpenStack Neutron • OVS flow action NORMAL - FDB table updated per packet