Automated OSCAR testing with Automated OSCAR testing with Linux- - PowerPoint PPT Presentation

Introduction (finally) Introduction (finally) � The automated testing should not replace The automated testing should not replace � completely the quality insurance process of the completely the quality insurance process of the OSCAR project. OSCAR project. � It has to be considered like a It has to be considered like a « «simulation simulation» » of a of a � cluster installation with OSCAR using only one cluster installation with OSCAR using only one physical computer. physical computer. � This being said, we believe that this testing This being said, we believe that this testing � infrastructure will allow the OSCAR project to infrastructure will allow the OSCAR project to «release early, release often release early, release often» » and, as a and, as a « conseguence, to aument the functionality and conseguence, to aument the functionality and general quality of the project. general quality of the project.

Introduction (finally) Introduction (finally) � The objective of this article is to present an The objective of this article is to present an � automated testing infrastructure that will ( can ) be automated testing infrastructure that will ( can ) be used for OSCAR development. used for OSCAR development.

Introduction (finally) Introduction (finally) � The objective of this article is to present an The objective of this article is to present an � automated testing infrastructure that will ( can ) be automated testing infrastructure that will ( can ) be used for OSCAR development. used for OSCAR development. � The main goal of this testing infrastructure is to The main goal of this testing infrastructure is to � reduce the development time used for testing the reduce the development time used for testing the software. software.

Introduction (finally) Introduction (finally) � The objective of this article is to present an The objective of this article is to present an � automated testing infrastructure that will (can) automated testing infrastructure that will (can) be used for OSCAR development. be used for OSCAR development. � The main goal of this testing infrastructure is to The main goal of this testing infrastructure is to � reduce the development time used for testing the reduce the development time used for testing the software. software. � We took advantage of a stable and mature We took advantage of a stable and mature � virtualization technology that is distribution virtualization technology that is distribution independent and support several architectures. independent and support several architectures.

System overview System overview

System overview System overview Before exposing the virtual testing Before exposing the virtual testing infrastructure, we'll briefly present the real infrastructure, we'll briefly present the real structure of the files and directories of the structure of the files and directories of the host system (Files organization Files organization), explain how ), explain how host system ( the sources of OSCAR are « «manipulated manipulated» » the sources of OSCAR are (From suversion to tarball From suversion to tarball) and, finally, ) and, finally, ( present the design utilized (Design overview Design overview) ) present the design utilized (

System overview System overview HOST HOST � Files organization � Files organization / / vservers vservers home home etc etc

System overview System overview � Files organization � Files organization / / vservers vservers oscarserver oscarserver oscarnode1 oscarnode1 oscarnode1 oscarnode1 oscarnode1 oscarnode1 oscarnode2 oscarnode2 ... ... oscarnode n oscarnode n

System overview System overview � Files organization � Files organization / / vservers vservers oscarserver oscarserver opt opt oscar oscar var var etc etc

System overview System overview � Files organization � Files organization / / oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc

System overview System overview / / � Files organization � Files organization oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc

System overview System overview / / � Files organization � Files organization oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc home home mdk10.0 mdk10.0 rhel3.0 rhel3.0

System overview System overview / / � Files organization � Files organization oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc tftpboot tftpboot rpm rpm home home mdk10.0 mdk10.0 rhel3.0 rhel3.0

System overview System overview / / � Files organization � Files organization oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc tftpboot tftpboot rpm rpm home home mdk10.0 mdk10.0 rhel3.0 rhel3.0

System overview System overview / / � Files organization � Files organization oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc tftpboot tftpboot rpm rpm home home mdk10.0 mdk10.0 rhel3.0 rhel3.0

System overview System overview / / � Files organization � Files organization oscarserver oscarserver vservers vservers / / opt opt oscar oscar var var etc etc tftpboot tftpboot rpm rpm home home mdk10.0 mdk10.0 rhel3.0 rhel3.0

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver opt opt oscar oscar var var var var lib lib sytemimager sytemimager

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver opt opt oscar oscar var var var var lib lib sytemimager sytemimager oscarimage oscarimage

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver oscarserver oscarserver opt opt oscar oscar var var var var lib lib sytemimager sytemimager oscarimage oscarimage oscarnode1 oscarnode1

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver oscarserver oscarserver opt opt oscar oscar var var var var lib lib sytemimager sytemimager oscarimage oscarimage oscarnode1 oscarnode1 oscarnode2 oscarnode2

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver oscarserver oscarserver oscarnode1 oscarnode1 oscarnode2 oscarnode2 etc etc

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver oscarserver oscarserver oscarnode1 oscarnode1 oscarnode2 oscarnode2 etc etc vservers vservers

System overview System overview / / � Files organization � Files organization vservers vservers oscarserver oscarserver oscarserver oscarserver oscarnode1 oscarnode1 oscarnode2 oscarnode2 etc etc vservers vservers oscarserver.conf oscarserver.conf oscarserver.sh oscarserver.sh oscarnode1.conf oscarnode1.conf oscarnode1.sh oscarnode1.sh

System overview System overview

System overview System overview � From subversion to tarball � From subversion to tarball

System overview System overview � From subversion to tarball � From subversion to tarball ➔ Once released, OSCAR is a Once released, OSCAR is a « «tarball tarball» »: : a software file a software file ➔ containing all the necessary files and binaries to install an containing all the necessary files and binaries to install an OSCAR cluster OSCAR cluster

System overview System overview � From subversion to tarball � From subversion to tarball ➔ Once released, OSCAR is a Once released, OSCAR is a « «tarball tarball» »: : a software file a software file ➔ containing all the necessary files and binaries to install an containing all the necessary files and binaries to install an OSCAR cluster OSCAR cluster ➔ Developers are using a subversion repository to track the Developers are using a subversion repository to track the ➔ changes to the source code and an automated suite of changes to the source code and an automated suite of programs (autoconf, automake, make) is used to build the programs (autoconf, automake, make) is used to build the tarball tarball

System overview System overview � From subversion to tarball � From subversion to tarball ➔ While it's possible to test directly from the subversion While it's possible to test directly from the subversion ➔ repository, this is not what a regular OSCAR user will use: he repository, this is not what a regular OSCAR user will use: he will use the tarball available will use the tarball available

System overview System overview � From subversion to tarball � From subversion to tarball ➔ While it's possible to test directly from the subversion While it's possible to test directly from the subversion ➔ repository, this is not what a regular OSCAR user will use: he repository, this is not what a regular OSCAR user will use: he will use the tarball available will use the tarball available ➔ So, we build the tarball from the sources, copy it to the So, we build the tarball from the sources, copy it to the ➔ main vserver and use it for install OSCAR main vserver and use it for install OSCAR

System overview System overview � Design overview � Design overview

System overview System overview � Design overview � Design overview ➔ A A « «standard standard» » script ( script (script A script A) is used ) is used to create a vserver to create a vserver ➔

System overview System overview � Design overview � Design overview ➔ A A « «standard standard» » script ( script (script A script A) is used ) is used to create a vserver to create a vserver ➔ ➔ This script is generic and can be used to create any Linux This script is generic and can be used to create any Linux- - ➔ VServer system (Fedora, RHAS, Mandrake, Debian)* VServer system (Fedora, RHAS, Mandrake, Debian)* * At the time of this paper * At the time of this paper

System overview System overview � Design overview � Design overview ➔ A A « «standard standard» » script ( script (script A script A) is used ) is used to create a vserver to create a vserver ➔ ➔ This script is generic and can be used to create any Linux This script is generic and can be used to create any Linux- - ➔ VServer system (Fedora, RHAS, Mandrake, Debian)* VServer system (Fedora, RHAS, Mandrake, Debian)* ➔ This is the starting point of the automated testing process: This is the starting point of the automated testing process: ➔ the creation, from scratch, of a new Linux- -VServer that VServer that the creation, from scratch, of a new Linux will become the OSCAR master node will become the OSCAR master node * At the time of this paper * At the time of this paper

System overview System overview � Design overview � Design overview ➔ Three additional scripts will be used to test OSCAR: Three additional scripts will be used to test OSCAR: ➔ * At the time of this paper * At the time of this paper

System overview System overview � Design overview � Design overview ➔ Three additional scripts will be used to test OSCAR: Three additional scripts will be used to test OSCAR: ➔ � script B script B: to adapt the original vserver configurations : to adapt the original vserver configurations � to suit OSCAR and make a copy of the to suit OSCAR and make a copy of the subversion repository to the new vserver subversion repository to the new vserver * At the time of this paper * At the time of this paper

System overview System overview � Design overview � Design overview ➔ Three additional scripts will be used to test OSCAR: Three additional scripts will be used to test OSCAR: ➔ � script B script B: to adapt the original vserver configurations : to adapt the original vserver configurations � to suit OSCAR and make a copy of the to suit OSCAR and make a copy of the subversion repository to the new vserver subversion repository to the new vserver � script C script C: to initiate the installation process : to initiate the installation process � * At the time of this paper * At the time of this paper

System overview System overview � Design overview � Design overview ➔ Three additional scripts will be used to test OSCAR: Three additional scripts will be used to test OSCAR: ➔ � script B script B: to adapt the original vserver configurations : to adapt the original vserver configurations � to suit OSCAR and make a copy of the to suit OSCAR and make a copy of the subversion repository to the new vserver subversion repository to the new vserver � script C script C: to initiate the installation process : to initiate the installation process � � script D script D: to create the client nodes : to create the client nodes � * At the time of this paper * At the time of this paper

➔ The numbers of The numbers of ➔ scripts as well as scripts as well as the user interaction the user interaction System model design System model design will be reduced in will be reduced in the future versions the future versions in order to achieve in order to achieve a better a better automatization of automatization of the testing process the testing process

➔ Create the vserver Create the vserver 1 ➔ 1 (script A script A) with ) with ( OSCAR specifics OSCAR specifics (script B script B) ) (

➔ Start the vserver Start the vserver ➔ master: master: 2 2 vserver oscarserver start vserver oscarserver start

➔ Change to the Change to the ➔ vserver master: vserver master: 3 3 vserver oscarserver enter vserver oscarserver enter

➔ Launch Launch ➔ install_cluster install_cluster 4 4

➔ Wizard: step 0 Wizard: step 0 ➔ 5 5

➔ Wizard: step 1 Wizard: step 1 ➔ 6 6

➔ Wizard: step 2 Wizard: step 2 ➔ 7 7

➔ Wizard: step 3 Wizard: step 3 ➔ 8 8

➔ Wizard: step 4 Wizard: step 4 ➔ 9 9

➔ Wizard: step 5 Wizard: step 5 ➔ 10 10

➔ Return to the host Return to the host ➔ system system exit exit 11 11

➔ Create the vservers Create the vservers ➔ clients from the clients from the oscarimage (script script oscarimage ( D) ) D 12 12

➔ Start each one of Start each one of ➔ the clients the clients 13 13

➔ Return to the Return to the ➔ vserver master vserver master (oscarserver) (oscarserver) 14 14

➔ Wizard: step 7 Wizard: step 7 ➔ 15 15

➔ Wizard: step 8 Wizard: step 8 ➔ 16 16

System overview System overview � Actual implementation � Actual implementation

System overview System overview � Actual implementation � Actual implementation ➔ Can be separated in two major steps: Can be separated in two major steps: ➔

System overview System overview � Actual implementation � Actual implementation ➔ Can be separated in two major steps: Can be separated in two major steps: ➔ � one for creating the main vserver that will host the one for creating the main vserver that will host the � OSCAR server (step 1 step 1) ) OSCAR server (

System overview System overview � Actual implementation � Actual implementation ➔ Can be separated in two major steps: Can be separated in two major steps: ➔ � one for creating the main vserver that will host the one for creating the main vserver that will host the � OSCAR server (step 1 step 1) ) OSCAR server ( � a second to install OSCAR ( a second to install OSCAR (step 2 step 2) ) �

System overview System overview � Actual implementation � Actual implementation ➔ There should not be big problems regarding the first one, There should not be big problems regarding the first one, ➔ once the respective RPMs are available in the default once the respective RPMs are available in the default location and the system is ready to host vservers the system is ready to host vservers location and

System overview System overview � Actual implementation � Actual implementation ➔ There should not be big problems regarding the first one, There should not be big problems regarding the first one, ➔ once the respective RPMs are available in the default once the respective RPMs are available in the default location and the system is ready to host vservers the system is ready to host vservers location and ➔ The testing will really begin in the second setp The testing will really begin in the second setp ➔

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node ➔ The main The main « «make make- -vserver vserver» » script ( script (A A) can be considered as ) can be considered as ➔ a function requiring as input parameters hostname hostname, , a function requiring as input parameters domainname and and ip address ip address of the OSCAR server you of the OSCAR server you domainname want to generate. This « «function function» » will return a fully will return a fully want to generate. This functional Linux- -VServer VServer functional Linux

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node ➔ The main The main « «make make- -vserver vserver» » script ( script (A A) can be considered as ) can be considered as ➔ a function requiring as input parameters hostname hostname, , a function requiring as input parameters domainname and and ip address ip address of the OSCAR server you of the OSCAR server you domainname want to generate. This « «function function» » will return a fully will return a fully want to generate. This functional Linux- -VServer VServer functional Linux ➔ The option 'TYPE=OSCAR' will trigger additional The option 'TYPE=OSCAR' will trigger additional ➔ threatment of the generated vserver. This will call script B threatment of the generated vserver. This will call script B to modify the original vserver so that it can be used to to modify the original vserver so that it can be used to build and install OSCAR (capabilities, ssh permissions, build and install OSCAR (capabilities, ssh permissions, /tftpboot, etc) /tftpboot, etc)

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node ➔ Those scripts also generate two configuration files for the Those scripts also generate two configuration files for the ➔ vservers: vservers:

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node ➔ Those scripts also generate two configuration files for the Those scripts also generate two configuration files for the ➔ vservers: vservers: ➔ « «vserver_name vserver_name» ».conf .conf: data specific to the vserver (hostname, : data specific to the vserver (hostname, ➔ dommainame, ip address and capabilities) dommainame, ip address and capabilities)

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node ➔ Those scripts also generate two configuration files for the Those scripts also generate two configuration files for the ➔ vservers: vservers: ➔ « «vserver_name vserver_name» ».conf .conf: data specific to the vserver (hostname, : data specific to the vserver (hostname, ➔ dommainame, ip address and capabilities) dommainame, ip address and capabilities) ➔ « «vserver_name vserver_name» ».sh .sh: contains : contains « «hooks hooks» » to scripts that should to scripts that should ➔ be executed pre or post- -start of the Linux start of the Linux- -VServer and pre or VServer and pre or be executed pre or post post- -stop stop post

/ / vservers vservers oscarserver oscarserver oscarserver oscarserver oscarnode1 oscarnode1 oscarnode2 oscarnode2 etc etc vservers vservers oscarserver.conf oscarserver.conf oscarserver.sh oscarserver.sh oscarnode1.conf oscarnode1.conf oscarnode1.sh oscarnode1.sh

oscarserver.conf oscarserver.conf IPROOT=192.168.17.1 IPROOT=192.168.17.1 #IPROOTMASK= #IPROOTMASK= #IPROOTBCAST= #IPROOTBCAST= IPROOTDEV=eth1 IPROOTDEV=eth1 ONBOOT=yes ONBOOT=yes S_HOSTNAME=oscarserver S_HOSTNAME=oscarserver S_DOMAINNAME=test S_DOMAINNAME=test S_NICE= S_NICE= S_FLAGS="lock nproc" S_FLAGS="lock nproc" ULIMIT="- ULIMIT=" -HS HS - -u 1000" u 1000" S_CAPS="CAP_NET_RAW CAP_MKNOD CAP_CHOWN CAP_SYS_TIME CAP_SYS_RESOURCE" OURCE" S_CAPS="CAP_NET_RAW CAP_MKNOD CAP_CHOWN CAP_SYS_TIME CAP_SYS_RES

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node In order to allow OSCAR to be installed inside a Linux- - In order to allow OSCAR to be installed inside a Linux VServer, specific capabilities capabilities has to be given to this has to be given to this VServer, specific vserver: vserver:

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node In order to allow OSCAR to be installed inside a Linux- - In order to allow OSCAR to be installed inside a Linux VServer, specific capabilities capabilities has to be given to this has to be given to this VServer, specific vserver: vserver: ● CAP_MKNOD CAP_MKNOD: : permit the privileged aspects of mknod permit the privileged aspects of mknod ●

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node In order to allow OSCAR to be installed inside a Linux- - In order to allow OSCAR to be installed inside a Linux VServer, specific capabilities capabilities has to be given to this has to be given to this VServer, specific vserver: vserver: ● CAP_MKNOD CAP_MKNOD: : permit the privileged aspects of mknod permit the privileged aspects of mknod ● ● CAP_CHOWN CAP_CHOWN: : change the ownership and group change the ownership and group ●

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node In order to allow OSCAR to be installed inside a Linux- - In order to allow OSCAR to be installed inside a Linux VServer, specific capabilities capabilities has to be given to this has to be given to this VServer, specific vserver: vserver: ● CAP_MKNOD CAP_MKNOD: : permit the privileged aspects of mknod permit the privileged aspects of mknod ● ● CAP_CHOWN CAP_CHOWN: : change the ownership and group change the ownership and group ● ● CAP_SYS_TIME CAP_SYS_TIME: : allow manipulation of the system clock allow manipulation of the system clock ●

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node In order to allow OSCAR to be installed inside a Linux- - In order to allow OSCAR to be installed inside a Linux VServer, specific capabilities capabilities has to be given to this has to be given to this VServer, specific vserver: vserver: ● CAP_MKNOD CAP_MKNOD: : permit the privileged aspects of mknod permit the privileged aspects of mknod ● ● CAP_CHOWN CAP_CHOWN: : change the ownership and group change the ownership and group ● ● CAP_SYS_TIME CAP_SYS_TIME: : allow manipulation of the system clock allow manipulation of the system clock ● ● CAP_SYS_RESOURCE CAP_SYS_RESOURCE: : override resource limits, quota, override resource limits, quota, ● reserved space on fs, ... reserved space on fs, ...

System overview System overview / / � Files organizaton � Files organizaton vservers vservers oscarserver oscarserver oscarserver oscarserver oscarnode1 oscarnode1 oscarnode2 oscarnode2 etc etc vservers vservers oscarserver.conf oscarserver.conf oscarserver.sh oscarserver.sh oscarnode1.conf oscarnode1.conf oscarnode1.sh oscarnode1.sh

oscarserver.sh oscarserver.sh #!/bin/sh #!/bin/sh case $1 in case $1 in pre- pre -start) start) # Ajouter les actions de pré # Ajouter les actions de pr é- -d dé émarrage ici marrage ici mount mount -- --bind /home/mdk10.0 /vservers/oscarserver/tftpboot/rpm bind /home/mdk10.0 /vservers/oscarserver/tftpboot/rpm vproc vproc - -e /proc/net/unix e /proc/net/unix vproc - -e /proc/net e /proc/net vproc vproc - -e /proc/net/dev e /proc/net/dev vproc vproc - -e /dev/null e /dev/null vproc vproc - -e /proc/net/route e /proc/net/route vproc vproc - -e /proc/sys e /proc/sys vproc vproc - -e /proc/sys/* e /proc/sys/* vproc ;; ;; post post- -start) start) # Ajouter les actions de post- # Ajouter les actions de post -d dé émarrage ici marrage ici ;; ;; pre- pre -stop) stop) # Ajouter les actions de pr # Ajouter les actions de pré é- -arrêt ici arrêt ici ;; ;; post- -stop) stop) post # Ajouter les actions de post- -arrêt ici arrêt ici # Ajouter les actions de post umount /vservers/oscarserver/tftpboot/rpm umount /vservers/oscarserver/tftpboot/rpm ;; ;; *) *) echo $0 pre- echo $0 pre -start start echo $0 pre echo $0 pre- -stop stop echo $0 post echo $0 post- -start start echo $0 post- echo $0 post -stop stop ;; ;; esac esac

Actual implementation Actual implementation � Generation of the OSCAR master node � Generation of the OSCAR master node ➔ The pre The pre- -start actions include binding the distribution start actions include binding the distribution ➔ RPM's repository to /tftpboot/rpm and giving to the RPM's repository to /tftpboot/rpm and giving to the vserve access to several entries in the /proc filesystem filesystem vserve access to several entries in the /proc (using vproc) (using vproc)