Motivation Applications with non-functional requirements in Grid - - PDF document

▶
motivation
SMART_READER_LITE
LIVE PREVIEW

Motivation Applications with non-functional requirements in Grid - - PDF document

Dec 22, 2023 306 likes •420 views

ZeliGrid: an architecture to deploy applications with dynamic non-functional aspects on computational grids Rodrigo Souza Granja e Alexandre Sztajnberg DICC/IME e PEL/FEN - UERJ Rio de Janeiro, RJ, Brazil {rodrigos, alexszt}@ime.uerj.br 1

slide-1
SLIDE 1

1

ZeliGrid: an architecture to deploy applications with dynamic non-functional aspects on computational grids

Rodrigo Souza Granja e Alexandre Sztajnberg

DICC/IME e PEL/FEN - UERJ Rio de Janeiro, RJ, Brazil {rodrigos, alexszt}@ime.uerj.br

2

Motivation

  • Applications with non-functional requirements in

Grid environments

– Local resources (ex.: CPU, memory) – Communication resources (ex.: BW, latency)

  • To provide the required dynamic support we

need

– to describe non-functional requirements and policies – adaptation and reconfiguration mechanisms – monitoring mechanisms – management of the policies for resource use

slide-2
SLIDE 2

3

Goals

  • Develop the support for applications with non-functional

requirements in a grid environment

– Based on the QoS contracts from the CR-RIO architecture

  • Integrate largely used tools

– Globus: security, remote job execution and management – NWS: resource monitoring – LDAP: directory service

  • Develop the additional modules

– Contract interpretation – Contract management – Reconfiguration mechanisms

4

Globus Toolkit

  • Middleware – Grid basic services (v 2.4)

– MDS – Directory service

  • GRIS and GIIS

– GRAM – Remote job execution and management – GSI – Security and authentication – Java CoG Kit – Java API – JNDI – Java Directory access API

slide-3
SLIDE 3

5

Network Weather Service

  • Resource monitoring and forecasting distributed

system

  • Active and static measurement

– Operating System information (uptime and vmstat) – Probes

  • CPU, latency, bandwidth
  • Information available through LDAP

– SLDAP server

6

Support architecture

  • Globus architecture
  • Integrated with NWS
  • Provides basic services
  • Monitoring
  • Directory
  • Remote job control
  • Shared resources

Support architecture

  • Globus architecture
  • Integrated with NWS
  • Provides basic services
  • Monitoring
  • Directory
  • Remote job control
  • Shared resources

Contract rules interpretation module

  • Reads and parses QoS

contracts

  • Can have up to 2 profiles

Contract rules interpretation module

  • Reads and parses QoS

contracts

  • Can have up to 2 profiles

Resource management control module

  • Gather GIIS and NWS/

SLAPD measurements

  • Save these values
  • Gather the QoS contract

from MIRC to determinate apt machine set Resource management control module

  • Gather GIIS and NWS/

SLAPD measurements

  • Save these values
  • Gather the QoS contract

from MIRC to determinate apt machine set User

  • Provides the QoS contract
  • Configures the ZeliGrid

parameters User

  • Provides the QoS contract
  • Configures the ZeliGrid

parameters Remote jobs management module

  • Uses GRAM functions to

start, stop and monitor remote jobs

  • Provides two functions to

start remote jobs

Individual Job Several jobs (same job multiple machines)

Remote jobs management module

  • Uses GRAM functions to

start, stop and monitor remote jobs

  • Provides two functions to

start remote jobs

Individual Job Several jobs (same job multiple machines)

Experiment control and reconfiguration module

  • Uses the functions

provided by the other modules to execute and reconfigure the application

  • Loads user provided

configurations

  • Reconfigure remote jobs
  • Can be extended

Experiment control and reconfiguration module

  • Uses the functions

provided by the other modules to execute and reconfigure the application

  • Loads user provided

configurations

  • Reconfigure remote jobs
  • Can be extended

Implementation

  • Developed with Java
  • Access to the Globus 2.4

services using API CoG kit

  • Access directory services

using JNDI API

  • Graphic Interface

(application configuration, visualization and control) Implementation

  • Developed with Java
  • Access to the Globus 2.4

services using API CoG kit

  • Access directory services

using JNDI API

  • Graphic Interface

(application configuration, visualization and control)

Start job request All hosts measurements Individual resources measurements

Proposed architecture

Contract Available machines Job submission Apt machines Queries API

SLAPD/GIIS NWS / GRIS Resources MIRC GRAM MGR MGRP MCRE

Control / Configuration

CoG kit / JNDI Interface ZeliGrid user

Start job request All hosts measurements Individual resources measurements Contract Available machines Job submission Apt machines Queries API

SLAPD/GIIS NWS / GRIS Resources MIRC GRAM MGR MGRP MCRE

Control / Configuration

CoG kit / JNDI Interface ZeliGrid user

slide-4
SLIDE 4

7

Contract Example

01 [profile1] 02

  • sName=Linux

03 bandWidth>78 04 cpuAvail>0.6 05 cpuSpeed>2000 06 freeMemory>200 07 [profile2] 08

  • sName=Linux

09 bandWidth>50 10 cpuAvail>0.5 11 cpuSpeed>1800 12 freeMemory>100

8

Start execution

Sequence diagram

User MGR MIRC MCRE MGRP Interface

Search Save attributes Read contract Save contract Prepare execution Load configuration Apt machines request Search contract Contract Apt machines Start remote execution Machines in use Cancel job request Verify in use machines Start reconfiguration Search for free apt machine Apt Machine Job start request Verify in use machines Start Jobs Cancel Job Start Job

slide-5
SLIDE 5

9

Deployment

<<Protocol>> {Type=LDAP} <<Protocol>> {Type=LDAP’} <<Protocol>> {Type=GramProtocol}

<<GridMachine>> :Resources <<GridMachine>> :Resources

<<ServiçoNws>> NWS sensors << GlobusService>> GRAM gatekeeper <<UserApplication>> Applications <<GlobusService>> GRIS

<<UserMachine>> :ComponentsManagement <<UserMachine>> :ComponentsManagement

<< GlobusService>> Proxy << GlobusService>> Proxy :Zeligrid Contract management <<MIRC>> Resource management <<MGR>> Jobs management <<MGRP>> Experiment control <<MCRE>> :Zeligrid Contract management <<MIRC>> Resource management <<MGR>> Jobs management <<MGRP>> Experiment control <<MCRE>>

<<GlobusServer>> :MDSDirectory <<GlobusServer>> :MDSDirectory

<< GlobusService>> GIIS

<<NWSServer>> :NWSDirectory <<NWSServer>> :NWSDirectory

<< NWSService>> NWS name server

<<LDAPServer>> :SlapdNWSDirectory <<LDAPServer>> :SlapdNWSDirectory

<< LDAPService>> Servidor SLAPD << NWSService>> NWS memory server

10

Resources

  • Nodes that execute the application jobs
  • Runs:

– GRIS instance

  • Gather static information (OS variables)

– NWS Sensors

  • Active probe measurements
  • CPU availability
  • Bandwidth between each slave and the master

– Gram Gatekeeper

slide-6
SLIDE 6

11

Information Servers

  • May be deployed in distinct machines
  • MDS directory (GIIS)

– Concentrate and deliver GRIS measurements

  • NWS SLAPD directory

– Deliver NWS measurements trough LDAP

  • NWS servers

– Memory server – Name server

12

User machine

  • Controls the architecture
  • Where ZeliGrid is deployed

– MIRC – MGR – MCRE – MGRP

  • Must have an authenticated proxy

– Access Globus secure services

slide-7
SLIDE 7

13

Main window

Menu Bar Information trees Tool Bar Toolbar

  • 5 buttons
  • Search Hosts
  • Start
  • Refresh
  • Stop
  • Exit

Menu Bar

  • Experiment
  • Open contract
  • Exit
  • Help
  • Help
  • About

Information trees

  • Hosts attributes
  • Contracts

Remote Jobs

  • Job Status
  • Host status
  • Job Status
  • Date and time
  • Profile
  • Stdout
  • StdIn

Remote Jobs Messages

  • Status Bar
  • Log messages

Status Bar Log messages 14

Configuration

Slapd NWS and MDS (GIIS) address Master application execution parameters Nws measurement expiration time Time between reconfiguration attempts

If true, finished jobs will be restarted

Time between two Host status verification

slide-8
SLIDE 8

15

Experiment

  • Linux Slackware 10 / Ethernet 10/100 network
  • Master-slave software architecture
  • Qualitative tests

– Verify if the mechanisms behaved as expected – Measurements made by the log messages analysis – External applications created to consume resources (CPU and memory) forcing reconfigurations

16

Results

  • Jobs are deployed on available machines that

best fits the QoS contract requirements

  • During operation the monitoring mechanisms

could consistently detect contract violations

  • Reconfiguration mechanisms were able to

– relocate remote jobs to other machines – switch to the 2nd profile, according to the specified profiles

slide-9
SLIDE 9

17

Future works

  • New reconfigurations mechanisms

– State persistent

  • More complex contracts

– Closer to CR-RIO’s semantics

  • Use ZeliGrid to deploy other applications

18

Contacts

  • Rodrigo Souza Granja

– rodrigos@ime.uerj.br – http://www.ime.uerj.br/~rodrigos

  • Alexandre Sztajnberg

– alexszt@ime.uerj.br – http://www.ime.uerj.br/~alexszt

  • Apoio

– PIBIC CNPq e UERJ – Projeto CARAVELA (Proj. GIGA / RNP-CPqD)

slide-10
SLIDE 10

19

Integration steps

  • Implantação do Globus

– Geração, autenticação e aplicação de certificados – Ativação dos serviços e configuração

  • Implantação do NWS

– Integração com o SLAPD – Acesso aos dados através do servidor LDAP

  • Acesso do Globus ao repositório LDAP

– JNDI

20

Implementation

  • Developed in Java

– Architecture configuration interface – Remote jobs visualization – Resources state monitoring – Remote jobs reconfiguration – Remote jobs reallocation