functest in depth
play

FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 - PowerPoint PPT Presentation

Sep 29, 2023 •123 likes •641 views

FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 jose.lausuch@ericsson.com Agenda What is Functest? Test cases Repository structure CI & Automation Brahmaputra Release 09/11/2015 Functest in Depth, OPNFV Summit 2015 2

  1. FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 jose.lausuch@ericsson.com

  2. Agenda What is Functest? Test cases Repository structure CI & Automation Brahmaputra Release 09/11/2015 Functest in Depth, OPNFV Summit 2015 2

  3. OPNFV Projects ecosystem OCTOPUS (CI) RELENG INSTALLER Projects FEATURE Projects TEST Projects Fuel VPN Functest Foreman SFC Yardstick JOID SDN Contr. Qtip Compass Resource mgmt. VPerf … … … Testbed Testbed Testbed Testbed Testbed PHAROS 09/11/2015 Functest in Depth, OPNFV Summit 2015 3

  4. What is Functest? 09/11/2015 Functest in Depth, OPNFV Summit 2015 4

  5. What we do… Base System Functionality Testing Focus on verifying the OpenStack deployment and the SDN Controllers Have a full integration and automation mechanism Provide comprehensive testing methodology 09/11/2015 Functest in Depth, OPNFV Summit 2015 5

  6. ETSI NFV Reference WE DON’T: Performance tests HW tests Frameworks 09/11/2015 Functest in Depth, OPNFV Summit 2015 6

  7. The gang… 09/11/2015 Functest in Depth, OPNFV Summit 2015 7

  8. The project… source: Bitergia (http://projects.bitergia.com/opnfv) date: November 2015 09/11/2015 Functest in Depth, OPNFV Summit 2015 8

  9. Functest Testcases Arno 09/11/2015 Functest in Depth, OPNFV Summit 2015 9

  10. Functest Testcases vPing test case Create machines and verify connectivity ODL test case Robot framework, ODL functional testing Rally bench tests Benchmark the OpenStack deployment Tempest test OpenStack native tests (100+ smoke-tests) 09/11/2015 Functest in Depth, OPNFV Summit 2015 10

  11. Functest Testcases vPing test case Create machines and verify connectivity 09/11/2015 Functest in Depth, OPNFV Summit 2015 11

  12. vPing test case VM1 VM2 provisional Neutron network 09/11/2015 Functest in Depth, OPNFV Summit 2015 12

  13. Functest Testcases ODL test case Robot framework, ODL functional testing 09/11/2015 Functest in Depth, OPNFV Summit 2015 13

  14. ODL test case Dashboard/API (Horizon) Networking Compute Storage ….. (Neutron) ML2 Plug-in Northbound API ODL – Neutron REST API OVSDB 09/11/2015 Functest in Depth, OPNFV Summit 2015 OVS Nodes 14

  15. ODL test cases Borrowed from ODL repository 4 basic test cases: Check ODL Restconf Create/Delete/Check Network Create/Delete/Check Subnet Create/Delete/Check Port 09/11/2015 Functest in Depth, OPNFV Summit 2015 15

  16. Functest Testcases Rally bench tests Benchmark the OpenStack deployment 09/11/2015 Functest in Depth, OPNFV Summit 2015 16

  17. Rally Rally is a benchmarking tool that answers the question: “How does OpenStack work at scale?”. 09/11/2015 Functest in Depth, OPNFV Summit 2015 17

  18. Rally - scenario example Runners: • Serial : launch scenario N times in one thread • Constant : launch scenario N times in M parallel threads • RPS : generates N threads per second M times Context: • Contexts needed for this scenario (tenants/users) Success criteria (SLA) • You can define a maximum time per iteration • It will skip the test if 1 iteration fails 09/11/2015 Functest in Depth, OPNFV Summit 2015 18

  19. Rally - output 09/11/2015 Functest in Depth, OPNFV Summit 2015 19

  20. Functest Testcases Tempest test OpenStack native tests (100+ smoke-tests) 09/11/2015 Functest in Depth, OPNFV Summit 2015 20

  21. Tempest test Tempest is a set of integration tests to be run against a live OpenStack cluster Rally performs the tempest installation and generation of tempest.conf 09/11/2015 Functest in Depth, OPNFV Summit 2015 21

  22. Functest Structure today 09/11/2015 Functest in Depth, OPNFV Summit 2015 22

  23. Directory structure Dockerfile and automation scripts SDN Controllers test suites Format results to dashboard ready and other utils Openstack related tests (Rally, Tempest) vIMS test case vPing test case Main configuration script. Installs needed tools (Rally, Robot, …) Configuration parameters 09/11/2015 Functest in Depth, OPNFV Summit 2015 23

  24. Configuration file Configuration yaml file Predefined tested values Can be given as an input to Functest Possible to specify commit IDs of the used repositories Versioning control 09/11/2015 Functest in Depth, OPNFV Summit 2015 24

  25. tion CI & Aut 09/11/2015 Functest in Depth, OPNFV Summit 2015 25

  26. CI & Automation Functest Jenkins job Triggered automatically after a successful deployment One macro per test case You can execute a single test From Jenkins 09/11/2015 Functest in Depth, OPNFV Summit 2015 26

  27. CI & Automation Daily verification of OPNFV platform JENKINS BUILD DEPLOYMENT FUNCTEST New Features YARDSTICK …. 09/11/2015 Functest in Depth, OPNFV Summit 2015 27

  28. Execution of tests Target POD Don’t worry sir, I’ll do it for you jjob Jumphost 09/11/2015 Functest in Depth, OPNFV Summit 2015 28

  29. Brahmaputra release 09/11/2015 Functest in Depth, OPNFV Summit 2015

  30. Functest in Brahmaputra release Dockerization Test Result collection Dashboard Virtual IMS test case + SDN Controllers tests Feature projects 09/11/2015 Functest in Depth, OPNFV Summit 2015 30

  31. Docker -ization 09/11/2015 Functest in Depth, OPNFV Summit 2015 31

  32. Functest Docker $docker run -t opnfv/functest $dir/run_test.sh 09/11/2015 Functest in Depth, OPNFV Summit 2015 32

  33. Functest Docker Without Docker With Docker • Have to install needed libraries in • Needed libraries preinstalled in your jumphost the docker image • Maintain in the documentation • Update image when there is a the needed libraries and new software dependency and dependencies push to the OPNFV Docker Hub • Possible OS dependencies • No OS dependencies • Not portable • Portable 09/11/2015 Functest in Depth, OPNFV Summit 2015 33

  34. Execution of tests Don’t worry sir, Target POD I’ll do it for you Docker Hub jjob Jumphost 09/11/2015 Functest in Depth, OPNFV Summit 2015 34

  35. HSS CSCF MRF PCRF vIMS test case 09/11/2015 Functest in Depth, OPNFV Summit 2015 35

  36. vIMS test case Full automation of a core Open Source vIMS solution on OPNFV from Clearwater Deployment of an orchestrator (Cloudify) to manage the VNF Run suite of signaling tests on vIMS SIP Successfully tested from Jenkins on LinuxFoundation POD2 09/11/2015 Functest in Depth, OPNFV Summit 2015 36

  37. vIMS test case 1) Deploy the orchestrator 2) Create the tenant 3) Create the VNF (10 VMs) 4) Launch test suite (more than 100 signaling tests) 5) Collect results 6) Clean-up 09/11/2015 Functest in Depth, OPNFV Summit 2015 37

  38. SDN Controllers 09/11/2015 Functest in Depth, OPNFV Summit 2015 38

  39. SDN Controllers Extension of OpenDaylight test cases Integration of ONOS test framework Open Contrail test 09/11/2015 Functest in Depth, OPNFV Summit 2015 39

  40. Test results collection 09/11/2015 Functest in Depth, OPNFV Summit 2015 40

  41. Test results collection Test projects Functest Yardstick QTIP VPerf Bottlenecks … REST API Mongo DB 09/11/2015 Functest in Depth, OPNFV Summit 2015 41

  42. Test results collection - API Filter per test case – http://<DB_Public_IP>/results?case=vPing Filter per pod – http://<DB_Public_IP>/results?pod=opnfv-jump-2 Filter per project – http://<DB_Public_IP>/results?project=yardstick Combination of filters – http://<DB_Public_IP>/results?project=functest&case=Tempest&pod=opnfv-jump-2 09/11/2015 Functest in Depth, OPNFV Summit 2015 42

  43. Dashboard 09/11/2015 Functest in Depth, OPNFV Summit 2015 43

  44. Dashboard Show all the results using simplistic graphs Possible to filter per project/test case/… Comparison of previous results with recent ones … 09/11/2015 Functest in Depth, OPNFV Summit 2015 44

  45. Dashboard Draft 09/11/2015 Functest in Depth, OPNFV Summit 2015 45

  46. Beyond Brahmaputra New functional test (vCPE, vHGW, vCDN, …) Results analytics Better Rally usage More SDN controllers test integration Broaden test coverage (security, O&M, …) 09/11/2015 Functest in Depth, OPNFV Summit 2015 46

  47. Lessons learned in Arno Need for stable labs some weeks before the release Need to run the tests on different PODs from CI Complex troubleshooting Functest cannot solve all problems 09/11/2015 Functest in Depth, OPNFV Summit 2015 47

  48. Challenges … Different OpenStack installers Different SDN controllers … Many features Lots of combinations to be tested !! 09/11/2015 Functest in Depth, OPNFV Summit 2015 48

  49. Summary First test project in OPNFV Validate a fresh OPNFV deployment Umbrella for testing projects Umbrella for feature projects We don’t bite, come to us with questions  09/11/2015 Functest in Depth, OPNFV Summit 2015 49

  50. 09/11/2015 Functest in Depth, OPNFV Summit 2015 50

  51. Thank you! 09/11/2015 Functest in Depth, OPNFV Summit 2015 51

Recommend

Tsinghua University Monocular Depth-Pose Prediction [R, t] Depth and Pose RGB PoseNet

Tsinghua University Monocular Depth-Pose Prediction [R, t] Depth and Pose RGB PoseNet

Wang Zhao, Shaohui Liu, Yezhi Shu, Yong-Jin Liu Tsinghua University Monocular Depth-Pose Prediction [R, t] Depth and Pose RGB PoseNet Fails to Generalize! All Drift ! Depth estimation in Indoor environments with Visual Odometry with

610 views • 15 slides
RGBD Tutorial 14210240041 Gu Pan Image RGB YUV Lab Depth Image RGB image Depth image Each pixel in

RGBD Tutorial 14210240041 Gu Pan Image RGB YUV Lab Depth Image RGB image Depth image Each pixel in

RGBD Tutorial 14210240041 Gu Pan Image RGB YUV Lab Depth Image RGB image Depth image Each pixel in depth image shows the distance to camera Device Kinect Kinect2 (we use) SoftKinetic Leapmotion Kinect Depth camera developed by

480 views • 29 slides
Depth from HDR: Depth Induction or Increased Realism? P. Vangorp 1 R. K. Mantiuk 2 B. Bazyluk 3

Depth from HDR: Depth Induction or Increased Realism? P. Vangorp 1 R. K. Mantiuk 2 B. Bazyluk 3

Depth from HDR: Depth Induction or Increased Realism? P. Vangorp 1 R. K. Mantiuk 2 B. Bazyluk 3 K. Myszkowski 1 R. Mantiuk 3 S. J. Watt 2 H.-P. Seidel 1 1 MPI Informatik 2 Bangor University 3 West Pomeranian 3 University of Technology Depth from

466 views • 34 slides
Depth &amp; Complexity Framework - Understanding the pieces ELP Depth and Complexity Resource

Depth &amp; Complexity Framework - Understanding the pieces ELP Depth and Complexity Resource

Depth &amp; Complexity Framework - Understanding the pieces ELP Depth and Complexity Resource Page: https://www.jtayloreducation.com/tagtelp/ J Taylor Education, 2016 Depth &amp; Complexity Framework Understanding the parts Think Like A

254 views • 22 slides
Tillage Depth Control (TDC) Benefits of Tillage Depth Control Is the definition of precision

Tillage Depth Control (TDC) Benefits of Tillage Depth Control Is the definition of precision

Tillage Depth Control (TDC) Benefits of Tillage Depth Control Is the definition of precision agriculture! Increased efficiency Automating time consuming tasks Enables less skilled workers to perform the task better Accurate

353 views • 11 slides
WHAT DEPTH ? Drillers Depth Determination Harald Bolt 1 45 th General Meeting March 17 th ,

WHAT DEPTH ? Drillers Depth Determination Harald Bolt 1 45 th General Meeting March 17 th ,

WHAT DEPTH ? Drillers Depth Determination Harald Bolt 1 45 th General Meeting March 17 th , 2017 The Hague, The Netherlands Wellbore Positioning Technical Section Speaker Information Harald Bolt What Depth ? March 17, 2017

266 views • 24 slides
Unsupervised Monocular Depth Estimation CNN Robust to Training Data Diversity Valery

Unsupervised Monocular Depth Estimation CNN Robust to Training Data Diversity Valery

Unsupervised Monocular Depth Estimation CNN Robust to Training Data Diversity Valery Anisimovskiy, Andrey Shcherbinin, 15 May, 2020 Sergey Turko and Ilya Kurilin Problem Statement: Depth Sensors limitations IR depth sensor Stereo camera

671 views • 21 slides
Depth Sensing and Deep Learning: Grasping and Segmenting 3D Objects from Real Depth Images using

Depth Sensing and Deep Learning: Grasping and Segmenting 3D Objects from Real Depth Images using

Depth Sensing and Deep Learning: Grasping and Segmenting 3D Objects from Real Depth Images using Synthetic Data Mike D e Dan aniel elczuk uk, Jeffrey Mahler, Matthew Matl, Saurabh Gupta, Andrew Lee, Andrew Li, Vishal Satish, Bill DeRose,

774 views • 73 slides
A Near-Optimal Depth-Hierarchy Theorem for Small-Depth Multilinear Circuits Nutan Limaye Compuer

A Near-Optimal Depth-Hierarchy Theorem for Small-Depth Multilinear Circuits Nutan Limaye Compuer

A Near-Optimal Depth-Hierarchy Theorem for Small-Depth Multilinear Circuits Nutan Limaye Compuer Science and Engineering Department, Indian Institute of Technology, Bombay, (IITB) India. Joint work with Suryajith Chillara, Christian Engels,

2.53k views • 191 slides
3D Modeling with Depth Sensors Andreas Geiger, Torsten Sattler Spring 2017

3D Modeling with Depth Sensors Andreas Geiger, Torsten Sattler Spring 2017

3D Modeling with Depth Sensors Andreas Geiger, Torsten Sattler Spring 2017 http://www.cvg.ethz.ch/teaching/3dvision/ 3D Modeling with Depth Sensors Todays class Obtaining depth maps / range images unstructured light

617 views • 47 slides
Week 5 Kullmann Analysing BFS Depth-first search Depth-first search Analysing DFS

Week 5 Kullmann Analysing BFS Depth-first search Depth-first search Analysing DFS

CS 270 Algorithms Oliver Week 5 Kullmann Analysing BFS Depth-first search Depth-first search Analysing DFS Analysing BFS Dags and 1 topological sorting Detecting Depth-first search 2 cycles Analysing DFS 3 Dags and topological

482 views • 23 slides
Aron Yu Nov 2, 2012 1 Depth Image Body Parts 3D Joint Est. 2 Image Credit: Shotton et al.

Aron Yu Nov 2, 2012 1 Depth Image Body Parts 3D Joint Est. 2 Image Credit: Shotton et al.

Aron Yu Nov 2, 2012 1 Depth Image Body Parts 3D Joint Est. 2 Image Credit: Shotton et al. Real-Time Human Pose Recognition in Parts from Single Depth Images Released: Nov 4, 2010 Color: 640 x 480@ 32 bits Depth: 640 x 480 @

880 views • 36 slides
Defense in Depth: In Depth Presented by: Chelsea H. Komlo About me - Software engineer, privacy

Defense in Depth: In Depth Presented by: Chelsea H. Komlo About me - Software engineer, privacy

Defense in Depth: In Depth Presented by: Chelsea H. Komlo About me - Software engineer, privacy and security engineer - HashiCorp, ThoughtWorks, Tor - Worked in 5 countries and two languages About this talk - NOT how to do security -

564 views • 32 slides
LEP500 Etch Depth Monitor Real-time, in-situ plasma etch depth monitoring and end point control

LEP500 Etch Depth Monitor Real-time, in-situ plasma etch depth monitoring and end point control

LEP500 Etch Depth Monitor Real-time, in-situ plasma etch depth monitoring and end point control plus co-linear wafer vision system Base Configuration Etch Depth Monitoring LEP500 Recessed Window Laser Spot on Plasma Plasma Patterned Wafer

299 views • 25 slides
LEP400 Etch Depth Monitor Real-time, in-situ plasma etch depth monitoring and end point control

LEP400 Etch Depth Monitor Real-time, in-situ plasma etch depth monitoring and end point control

LEP400 Etch Depth Monitor Real-time, in-situ plasma etch depth monitoring and end point control plus co-linear wafer vision system Base Configuration Etch Depth Monitoring LEP400 Recessed Window Laser Spot on Plasma Plasma Patterned Wafer

266 views • 22 slides
Understand the principles behind Webbs Depth of Knowledge criteria Know how these

Understand the principles behind Webbs Depth of Knowledge criteria Know how these

Learning Targets Understand the principles behind Webbs Depth of Knowledge criteria Know how these principles can be applied to improve the alignment and rigor of instruction and assessment with the standards Realize how Depth of

475 views • 23 slides
Depth from Stereo Sanja Fidler CSC420: Intro to Image Understanding 1 / 12 Depth from Two

Depth from Stereo Sanja Fidler CSC420: Intro to Image Understanding 1 / 12 Depth from Two

Depth from Stereo Sanja Fidler CSC420: Intro to Image Understanding 1 / 12 Depth from Two Views: Stereo All points on projective line to P map to p Figure: One camera Sanja Fidler CSC420: Intro to Image Understanding 2 / 12 Depth from Two

968 views • 43 slides
Learning Lear ning Center Centers s In In-Depth Depth : Building Policy Around

Learning Lear ning Center Centers s In In-Depth Depth : Building Policy Around

Cincinnatis Community Learning Lear ning Center Centers s In In-Depth Depth : Building Policy Around Cincinnatis Community School Initiative Annie e Bogensc nschut utz, Director of Training and Development, Community Learning

465 views • 17 slides
Our Moral Origins Part 1/3 Objects Number Depth Geometry Objects Number Depth Geometry

Our Moral Origins Part 1/3 Objects Number Depth Geometry Objects Number Depth Geometry

Our Moral Origins Part 1/3 Objects Number Depth Geometry Objects Number Depth Geometry Other Minds? Knowledge Intention Morality Theory of Mind Egocentricism Three Mountains Task Mean Monkey Task False Belief Task

507 views • 22 slides
Learning Approaches to Estimate Depth from RGB Lecture 5 What will we learn - Latest Approaches

Learning Approaches to Estimate Depth from RGB Lecture 5 What will we learn - Latest Approaches

Learning Approaches to Estimate Depth from RGB Lecture 5 What will we learn - Latest Approaches to Depth Estimation based on Machine Learning (DNNs) Why do we need new approaches? Paper1 -&gt; CNNs for Depth estimation Paper2

634 views • 17 slides
Depth-First Search A B D E C Outline and Reading Definitions (6.1) Subgraph

Depth-First Search A B D E C Outline and Reading Definitions (6.1) Subgraph

Depth-First Search A B D E C Outline and Reading Definitions (6.1) Subgraph Connectivity Spanning trees and forests Depth-first search (6.3.1) Algorithm Example Properties Analysis Applications of

551 views • 15 slides
Wo o l l y Mammo t h Th eat r e Lighting Depth Canopy Lobby Theatre Office Electrical Depth

Wo o l l y Mammo t h Th eat r e Lighting Depth Canopy Lobby Theatre Office Electrical Depth

Location: Washington DC Owner: The Woolly Mammoth Theatre Company Architect: Mcinturff Architects Size: 31,600 SF Total Stories: 3 Cost: $8,000,000 Over view Wo o l l y Mammo t h Th eat r e Lighting Depth Canopy Lobby Theatre Office

766 views • 33 slides
Human Perception of Depth Lecture 5 Machine Depth Perception Multi-view / Stereo Motion

Human Perception of Depth Lecture 5 Machine Depth Perception Multi-view / Stereo Motion

Human Perception of Depth Lecture 5 Machine Depth Perception Multi-view / Stereo Motion Humans? Stereo (binocular disparity) Motion (&amp; motion perspective) What else? Occlusion Height in Visual Field Relative

118 views • 11 slides
Boolean Circuit Depth (II) Yijia Chen Fudan University A Quick Recap Definition The depth d ( C

Boolean Circuit Depth (II) Yijia Chen Fudan University A Quick Recap Definition The depth d ( C

Boolean Circuit Depth (II) Yijia Chen Fudan University A Quick Recap Definition The depth d ( C ) of a circuit C is the length of the longest path from the output node to an input node. The size L ( F ) of a formula F is the number of its input

1.69k views • 168 slides

More recommend