Motivation Applications with non-functional requirements in Grid - PDF document

ZeliGrid: an architecture to deploy applications with dynamic non-functional aspects on computational grids Rodrigo Souza Granja e Alexandre Sztajnberg DICC/IME e PEL/FEN - UERJ Rio de Janeiro, RJ, Brazil {rodrigos, alexszt}@ime.uerj.br 1 Motivation • Applications with non-functional requirements in Grid environments – Local resources (ex.: CPU, memory) – Communication resources (ex.: BW, latency) • To provide the required dynamic support we need – to describe non-functional requirements and policies – adaptation and reconfiguration mechanisms – monitoring mechanisms – management of the policies for resource use 2

Goals • Develop the support for applications with non-functional requirements in a grid environment – Based on the QoS contracts from the CR-RIO architecture • Integrate largely used tools – Globus: security, remote job execution and management – NWS: resource monitoring – LDAP: directory service • Develop the additional modules – Contract interpretation – Contract management – Reconfiguration mechanisms 3 Globus Toolkit • Middleware – Grid basic services (v 2.4) – MDS – Directory service • GRIS and GIIS – GRAM – Remote job execution and management – GSI – Security and authentication – Java CoG Kit – Java API – JNDI – Java Directory access API 4

Network Weather Service • Resource monitoring and forecasting distributed system • Active and static measurement – Operating System information ( uptime and vmstat ) – Probes • CPU, latency, bandwidth • Information available through LDAP – SLDAP server 5 Proposed architecture ZeliGrid user ZeliGrid user Remote jobs management Resource management Experiment control and Support architecture Implementation Contract rules User Remote jobs management Experiment control and Resource management Support architecture Implementation Contract rules User reconfiguration module interpretation module control module module reconfiguration module interpretation module control module module Interface Interface • Provides the QoS contract • Developed with Java • Globus architecture • Provides the QoS contract • Developed with Java • Globus architecture API API • Configures the ZeliGrid • Uses the functions • Access to the Globus 2.4 • Uses GRAM functions to • Gather GIIS and NWS/ • Reads and parses QoS • Integrated with NWS • Configures the ZeliGrid • Uses the functions • Access to the Globus 2.4 • Gather GIIS and NWS/ • Uses GRAM functions to • Reads and parses QoS • Integrated with NWS CoG kit / JNDI CoG kit / JNDI parameters • Provides basic services services using API CoG kit provided by the other SLAPD measurements start, stop and monitor contracts parameters • Provides basic services services using API CoG kit provided by the other start, stop and monitor SLAPD measurements contracts Control / Control / • Access directory services • Save these values • Can have up to 2 profiles modules to execute and remote jobs • Monitoring • Access directory services • Save these values • Can have up to 2 profiles modules to execute and remote jobs • Monitoring Configuration Configuration MCRE MCRE • Directory • Gather the QoS contract • Provides two functions to using JNDI API reconfigure the application • Directory • Gather the QoS contract • Provides two functions to using JNDI API reconfigure the application • Remote job control • Remote job control • Graphic Interface • Loads user provided start remote jobs from MIRC to determinate • Graphic Interface • Loads user provided start remote jobs from MIRC to determinate Contract Contract • Shared resources • Shared resources configurations (application configuration, apt machine set � Individual Job (application configuration, configurations apt machine set � Individual Job MIRC MIRC MGR MGR � Several jobs (same job • Reconfigure remote jobs visualization and control) � Several jobs (same job • Reconfigure remote jobs visualization and control) Apt Apt Queries Queries multiple machines) multiple machines) machines machines • Can be extended • Can be extended SLAPD/GIIS SLAPD/GIIS Available Available MGRP MGRP machines machines All hosts measurements All hosts measurements Start job request Start job request NWS / GRIS NWS / GRIS Individual resources measurements Individual resources measurements GRAM GRAM Resources Resources Job submission Job submission 6

Contract Example 01 [profile1] 02 osName=Linux 03 bandWidth>78 04 cpuAvail>0.6 05 cpuSpeed>2000 06 freeMemory>200 07 [profile2] 08 osName=Linux 09 bandWidth>50 10 cpuAvail>0.5 11 cpuSpeed>1800 12 freeMemory>100 7 Sequence diagram User MGR MIRC MCRE MGRP Interface Start execution Save attributes Search Save contract Read contract Load configuration Prepare execution Apt machines request Search contract Contract Apt machines Start Jobs Start remote execution Machines in use Verify in use machines Search for free apt Cancel Start reconfiguration machine Job Apt Machine Cancel job request Job start request Verify in use Start Job machines 8

Deployment <<Protocol>> {Type=LDAP} <<GlobusServer>> <<Protocol>> <<GlobusServer>> <<UserMachine>> :MDSDirectory {Type=LDAP’} <<UserMachine>> :MDSDirectory :ComponentsManagement :ComponentsManagement <<GridMachine>> << GlobusService>> <<GridMachine>> :Zeligrid GIIS :Resources :Zeligrid :Resources Contract management <<MIRC>> Contract management <<MIRC>> <<GlobusService>> Resource management <<MGR>> <<LDAPServer>> Resource management <<MGR>> GRIS <<LDAPServer>> Jobs management <<MGRP>> Jobs management <<MGRP>> :SlapdNWSDirectory Experiment control <<MCRE>> :SlapdNWSDirectory Experiment control <<MCRE>> <<ServiçoNws>> NWS sensors << LDAPService>> Servidor SLAPD << GlobusService>> GRAM gatekeeper << GlobusService>> << GlobusService>> <<NWSServer>> Proxy <<NWSServer>> Proxy <<UserApplication>> :NWSDirectory :NWSDirectory Applications << NWSService>> NWS name server <<Protocol>> {Type=GramProtocol} << NWSService>> NWS memory server 9 Resources • Nodes that execute the application jobs • Runs: – GRIS instance • Gather static information (OS variables) – NWS Sensors • Active probe measurements • CPU availability • Bandwidth between each slave and the master – Gram Gatekeeper 10

Information Servers • May be deployed in distinct machines • MDS directory (GIIS) – Concentrate and deliver GRIS measurements • NWS SLAPD directory – Deliver NWS measurements trough LDAP • NWS servers – Memory server – Name server 11 User machine • Controls the architecture • Where ZeliGrid is deployed – MIRC – MGR – MCRE – MGRP • Must have an authenticated proxy – Access Globus secure services 12

Main window Tool Bar Menu Bar Information trees Remote Jobs Messages Menu Bar Toolbar • Job Status • Hosts attributes • Status Bar • Experiment • 5 buttons • Contracts • Log messages • Host status • Search Hosts • Open contract • Job Status • Exit • Start Remote Jobs • Help • Date and time • Refresh • Profile • Help • Stop Information trees • Stdout • About • Exit • StdIn Log messages Status Bar 13 Configuration If true, finished jobs will be restarted Time between two Host status verification Master application execution parameters Time between reconfiguration attempts Slapd NWS and MDS (GIIS) address Nws measurement expiration time 14

Experiment • Linux Slackware 10 / Ethernet 10/100 network • Master-slave software architecture • Qualitative tests – Verify if the mechanisms behaved as expected – Measurements made by the log messages analysis – External applications created to consume resources (CPU and memory) forcing reconfigurations 15 Results • Jobs are deployed on available machines that best fits the QoS contract requirements • During operation the monitoring mechanisms could consistently detect contract violations • Reconfiguration mechanisms were able to – relocate remote jobs to other machines – switch to the 2 nd profile, according to the specified profiles 16

Future works • New reconfigurations mechanisms – State persistent • More complex contracts – Closer to CR-RIO’s semantics • Use ZeliGrid to deploy other applications 17 Contacts • Rodrigo Souza Granja – rodrigos@ime.uerj.br – http://www.ime.uerj.br/~rodrigos • Alexandre Sztajnberg – alexszt@ime.uerj.br – http://www.ime.uerj.br/~alexszt • Apoio – PIBIC CNPq e UERJ – Projeto CARAVELA (Proj. GIGA / RNP-CPqD) 18