How to use OPNFV Testing tools to test/validate an Openstack - PowerPoint PPT Presentation

How to use OPNFV Testing tools to test/validate an Openstack provided by a vendor

Speakers Morgan Richomme Valentin Boucher NFV architect at Orange Labs Software engineer at Kontron Canada OPNFV ambassador OPNFV Functest committer: ONAP & OPNFV Functest https://wiki.opnfv.org/display/functest committer

Agenda ● OPNFV testing overview • OPNFV Testing chain • OPNFV Testing tools ● Reusing those tools • Kontron • Orange

OPNFV Testing Overview Testing chain and tools …..

OPNFV CI complexity Addons / Features SFC BGPVPN DPDK FDio SDN Controller OpenDayLight ONOS OpenContrail Installer / Deployer APEX Compass Daisy Fuel Joid XCI Linux Lab/POD Intel Ericsson Huawei Foundation

OPNFV Testing chain Scenario HA/noHA, SDN, DPDK etc.. Test result Version Master, Fraser Test Report Deploy Functest, Test api, Installer project Yardstick etc reporting Deployer APEX, Compass, Daisy, Fuel and JOID Pod LF, Huawei, Intel etc ..

OPNFV Testing ecosystem

Functional testing -> Functest Functest ● Answer the question: “ Does my infrastructure simply work as expected? ” ● Provide tests from basic connectivity (<1s) to VNF testing (>1h) ● Leverage Upstream test suites from the integrated components (OpenStack/Rally & Tempest, Kubernetes/Default test suites, OpenDaylight, …) ● Allow “A la carte” test combination through rich, light and modular Alpine based framework, adapted to CI/CD chains ● PTL: C.Ollivier, Orange

Official Release validation Target validation Full stack Functest testcases for OpenStack Gating Complexity

Performance testing -> Yardstick & cie Yardstick ● Answer the question: “ Qualify my NFVI and give me valuable figures? ” ● Qualify Compute, Network and Storage domains through elementary tests ● Leverage OPNFv community to build baseline and define acceptable thresholds for Telco Cloud solution ● Allow lab owner to compare their Telco Cloud KPI with the other OPNFV labs ● In line with ETSI NFV ● PTL: R. Brattain, Intel

Yardstick More than 50 available tests covering different domains (compute, network, storage, HA, IPv6, KVM, ..) Src: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html

Yardstick TC_012 post processing Src: Alassane Samba (Orange) https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results

Yardstick testing capacity • Generic test cases • Compute • Storage • Network • HA test cases • Service HA • Node HA • Function test cases • SRIOV • Dpdk • Hugepages

Certification -> Dovetail Dovetail ● Answer to question: “ Do these products conform to the interfaces, behaviors and functionalities defined by OPNFV? ” ● Main objective: reduce VNF onboarding cost ● Do not implement test cases in their own code base but reuse the test cases from the OPNFV test projects (Functest, Yardstick etc..) ● Allow vendors to use the OPNFV verified logo ● PTL: Georg Kunz, Ericsson

Dovetail – Test scope Whole OPNFV testsuite OPNFV Verified 2018.01 Future .. Mandatory test cases ● OpenStack API tests (205 tests) Future test case candidates ● Layer 2 packet forwarding (2 tests) ● OpenStack control service HA (8 tests) ● Neutron Trunk ports ● BGPVPN Tempest tests ● SDN Controller HA Optional test cases ● VNF lifecycle management ● …. ● IPv6 tenant networks (25 tests) ● BGPVPNs (4 tests) ● Fundamental VIM capabilities (30 tests)

Certification workflow 1. Submission of participation form 2. Testing 3. Submission of results 4. Notification of reviewers 5. Review of test results 6. Grant of use of program marks

Reuse those tools

Context Kontron communications unit (Canada): part ODM, part integrator SYMKLOUD hw and solution portfolio: integrate open source with high density converged platforms + cloud servers + white box switches Used for: NFVi at Edge + Core, SDN, DC clouds for the telco and video/broadcast, and cloud providers We make it easy: removed complexities to integrate software (CI/CD open source or not) on Symkloud hw; plus can ‘mix-n-match’ composable modules (GPU, FPGA, Crypto, Storage). Ramp up services faster and with lower risk: easy to Configure, Deploy and Manage services on our platforms (automated switch configuration and deployment, integrate addons,etc.)

Problematics How to test/validate that our deployment of Openstack or Kubernetes on SymKloud is fine ? ● Different SymKloud Platforms ○ Different type of switches (40Gb, 100Gb/SDN-ready etc..) ○ Different CPU (12, 16 .. cores) ○ Servers with/without GPU or FPGA etc… ● Differents software type and version (OpenStack Newton/Queens, Kubernetes 1.9/1.10 etc..) ● Differents add-ons (6wind network acceleration for OpenStack, OpenStack Ironic support etc..) More features can be added in the future (New deployer, SDN controller, New hardware)

Current CI Scenario HA/noHA, ironic, 6wind etc.. Version Arnold, Britney, Test Develop etc.. Deploy Report Homemade MAAS/Juju test suite Testrail using Behave Cloud software OpenStack, Kubernetes Hardware

Experimental CI Kontron Scenario OPNFV tools HA/noHA, ironic, 6wind etc.. Test result Version Arnold, Britney, Test Test Report Develop etc.. Deploy Test api, Homemade Functest, MAAS/Juju reporting test suite Yardstick Testrail Cloud software OpenStack, Kubernetes Hardware

Reporting

Our goals Why reuse those tools in addition of our internal QA test ? • Our QA test suite has a more diverse test coverage than the OPNFV tools ( not only focusing on the NFVI+VIM test but also installer, add-ons, configuration and hardware test) • The OPNFV test suite has more tests related to NFV so it completes our test coverage • The goal is to be better and close to the customer target usage • We ensure that next Dovetail verification will work if all opnfv tests pass on our platform.

The Orange CI/CD view (Preproduction)

CI/CD viewn ONAP OpenLAB Bare Metal O/S Openstack BIFROST / management OPNFV XCI Infrastructure Installation Kubespray Kolla Ansible Test Functest Xtesting API (Healthchecks/moke/vnf // Run daily smoke/conformance) ONAP Installation ONAP Heat Test ONAP Heat (Community lab) DB ONAP OOM Run daily Functest Xtesting/onap-robot

OPNFV Orange Integration Center Installers 5: Apex, Compass, Daisy, Fuel, Joid Preproduction: 1 (RedHat Director) Orange ONAP OpenLab: 1 Kolla Ansible Scenarios Mores than 30 Preproduction: 1 OpenStack + Contrail Orange ONAP OpenLab: 2 OpenStack / Kubernetes + ONAP Tests Tests (automation): relying on several test Preproduction: Functest + Yardstick projects: Orange ONAP OpenLab: Functest + ONAP ● Functest* for functional testing Robot test suites ● Yardstick* for performance testing ● Qtip for bench HW ● Storeperf for ceph ● Vsperf for Vswitch qualification ● Bottleneck to optimize Yardstick runs * used for release validation

Reuse of OPNFV tooling for ONAP Reuse of Functest Xtesting framework to onboard ONAP robot tests • Only 3 files needed (Robot natively supported) • Docker size from 1 Go to 100Mo (Alpines) • Inherit Test reporting capabilities • Consistent view to manage Infra and orchestration testing (reuse DB, Test API)

Reuse of OPNFV tooling for ONAP ● Test framework ● Test API/Test DB ● CI/CD methodology

Conclusions • OPNFV testing tool is useful, evolutive , promoting Upstream First (Openstack/ODL testing projects), it can easily complete existing vendor suites with a good focus on NFV • Automation is key : Xtesting framework is very well adapted to this context (light (Alpines), simple, modular) • Tooling is easily reusable out of OPNFV context (ONAP, Openstack/Kubernetes - non OPNFV - solutions) including frameworks, DB, test API • OPNFV release/Dovetail criteria should be stronger as test frameworks offer more than what is used as release criteria: infrastructure trustability is key for Telco Cloud/NFV adoption

Thank You!

How does Functest work?

Yardstick Architecture • User Interface • Main Framework • Context • Scenario • Runner • Dispatcher • Plugin • Container Manager • Services

OPNFV Test result API Result of each run in OPNFV test API name: "cloudify", version: "4.0", orchestrator status: "PASS", duration: 349 case_name: "cloudify_ims", pod_name: "huawei-pod2", name: "clearwater" version: "fraser", descriptor_version: "122", installer: "compass", VNF version: "107", scenario: "os-odl_l3-nofeature-ha", status: "PASS", start_date: "2018-04-18 04:58:10", duration: 1330 stop_date: "2018-04-18 05:38:50", criteria: “PASS” , name: "clearwater-live-test”, build_tag: "jenkins-functest-baremetal-22", version: "1.0", details: { } VNF Test status: "PASS", duration: 273, result: {failures: 0, skipped: 30, passed: 46} http://artifacts.opnfv.org/releng/docs/testapi.html