A Multi-agent Approach for Semantic Resource Allocation Jorge - PowerPoint PPT Presentation

A Multi-agent Approach for Semantic Resource Allocation Jorge Ejarque , Raül Sirvent and Rosa M. Badia Grid Computing and Clusters Group - Barcelona Supercomputing Center (BSC) Arti ficial Intelligence Research Institute - Spanish National Research Council (IIIA-CSIC) IEEE CloudCom 2010 December 3, 2010 Indianapolis

Outline  Motivation  Previous Work. – Resource Allocation Ontology – Centralized Semantic Scheduling  Distributed Approach – Multi-agent distribution  Implementation and Evaluation  Conclusions and Future Work 2

Motivation  SP has to allocate the user’s tasks in their available resources.  Resources are heterogeneous and can be supplied by different providers.  SP users specify requirements in different SLA terms which must be satisfied.  Users and providers use their own terms. There is a need of semantic interoperability Clouds Service Users Grids Service Provider Clusters 3

Motivation  SP Infrastructure can change – New resources can be added or removed from the SP infrastructure  Resource can fail – Performance Degradation  Resource Allocation must be adapted according to customer and provider preferences  Agent technologies provide: – Adaptation to infrastructure changes – Distributed resource allocation according to several customer and provider preferences 4

Previous work  Centralized approach Stores semantic descriptions of system elements Semantic Metadata Repository 1. Get Resource 0. Resource description Candidates registration 4. Update scheduling Create executions Resource Manager C l i e n t M a n a g e r Manage the 3. Get descriptions environments, customer’s tasks execute customers requests C l i e n t M Resource Manager a n a g e r tasks 2. Schedule 5. Execute Job Job Semantic Scheduler Allocates resources on the different customer’s tasks (semantic descriptions) 5

Resource Allocation Ontology  Based on Grid Resource Ontology  Extension to describe basic resource allocation concepts – Resources properties, Collections, Abstract task (resource requirements, time constraints) and Process (abstract task allocation), Actors, Providers, etc. 6

Semantic Scheduler  Rule-based semantic scheduling 7

Decentralized motivation  Scalability  Reduce single point of failure  Customer and providers has different interests and they should be taken into account in the allocation process. – Different rules depending on the customers and providers or resource types 8

Distributed Architecture  Distributed approach Resource Allocation Rules Semantic Metadata Repository Resource Allocation Ontology Agent Platform Private Clouds Job Agent Container Res. Agent Container Resource Provisioning Client Manager Scheduler Scheduler Res. Man Jobs . . . . . . Scheduler Scheduler Jobs Res. Man Clusters . . . . . . Job Agent Container Res. Agent Container Resource Provisioning Client Manager Grids Scheduler Scheduler Res. Man Jobs . . . . . . Scheduler Scheduler Public Clouds Jobs Res. Man EC2, ... 9

System Agents  Distributed resource allocation – Scheduling functionalities distributed across the system agents – Based on a negotiation between the system agents. (CNP) – Agents platforms can be distributed across multiple hosts  Two parts – Client Management (Job Agents)  Management of jobs  Resource selection – Resource Provisioning (Resource Agents)  Management of resources  Resource allocation proposals  Belief-Desire-Intention agent modeling – Define data, goals and plans for each agent 10

Job Agent  Semantic annotation of tasks.  Initiate the negotiation and select the best proposal for the customer interests.  Adapt the task allocation to the system events. (Rescheduling)  React to status changes (Fault tolerance) Belief ( Status) Goal Plans Requested Get Resources Annotate task description Negotiate allocation Scheduled Find Better Allocation Negotiate allocation Running Monitor execution Check performance Update requirements Increase resources Suspended Recover Suspended Evaluate work done Update requirements Negotiate allocation Stopped Recover Stopped Negotiate allocation Non Scheduled Recover Non Scheduled Negotiate allocation Cancelled Cancel Job Remove allocation 11

Resource Agent  Semantic annotation of resource descriptions  Make proposal according to resource interests  Create execution environments and execute tasks by means of a resource management interface  React to resource events (Failures) Belief Goal Plans Scheduled Jobs Monitor Scheduled Jobs Check scheduled job Perform execution Running Jobs Monitor Running Jobs Check running jobs Notify status changes Status Failed Recover Failure Try local rescheduling Notify affected jobs agents Status Running Register Resource Annotate resource description 12

Distributed Resource Allocation  Resource allocation negotiation sequence SELECT DISTINCT ?id ?host WHERE { ?host tech:isProvidedBy ?rm . ?rm biz:actor_name ?id . ?host tech:containsResource ?proc . ?proc rdf:type gro:Processing . ?proc gro:hasResourceProperty ?memory . ?memory rdf:type tech:MemoryCapacity . 1. Select candidate resources ?memory tech:hasValue ?mem_cap . [Evaluate_Deadline: FILTER (?mem_cap >= 1024ˆˆxsd:int) . (?job rdf:type gro:Execute_Task), 2. Send call for resource allocation ?host tech:containsResource ?image . (?job gro:hasActionState tech:requested), to selected resource agents ?image rdf:type tech:Image . [Earliest_Start_Date: (?job tech:hasDeadline ?deadline), ?image tech:containsResource gro:Software_XX } (?job rdf:type gro:Execute_Task), (?job gro:incarnatedToProcess ?incarnation), 3. Evaluate requirements, (?job gro:hasActionState tech:requested), (?incarnation gro:hasPlannedEnd ?job_end), discard imposible allocations (?job gro:incarnatedToProcess ?incarnationA), lessThan(?deadline, ?job_end), (resource full, license terms) (?incarnationA gro:hasPlannedStart ?stDateA), -> and apply other provider's rules (?job gro:incarnatedToProcess ?incarnationB), drop(3,4)] (?incarnationB gro:hasPlannedStart ?stDateB), lessEqual(?stDateA, ?stDateB), [Used_Hosts_First: 4. Send back proposed resource allocations -> (?job rdf:type gro:Execute_Task), remove(4,5)] (?job gro:hasActionState tech:requested), (?job gro:incarnatedToProcess ?incarnationA), 5. Apply customer selection rules [Priority_AtoB: (?incarnationA gro:usesResource ?candidateA), (?job rdf:type gro:Execute_Task), (?candidateA gro:containsResources ?HostA), 6. Accept selected proposal (?job gro:hasActionState tech:requested), hasValue(?HostA gro:isUsedBy), (?job gro:incarnatedToProcess ?incarnationA), (?job gro:incarnatedToProcess ?incarnationB), (?incarnationA gro:usesResource ?candidateA), (?incarnationB gro:usesResource ?candidateB), (?candidateA gro:containsResources ?resourceA), (?candidateB gro:containsResources ?HostB), (?resourceA tech:isProvidedBy biz:ResourceProviderA), noValue(?HostB, gro:isUsedBy) (?job gro:incarnatedToProcess ?incarnationB), -> (?incarnationB gro:usesResource ?candidateB), drop(6,7)] (?candidateB gro:containsResources ?resourceB), (?resourceB tech:isProvidedBy biz:ResourceProviderB) -> 13 remove(6,7)]

Implementation  System agents: Jadex BDI Engine  Semantic Capabilities: Jena 2 Framework – RDF, OWL APIs – Jena Rule Engine (Scheduler Module) – SPARQL queries 14

Evaluation  Qualitative evaluation – BDI Agents facilitate coordination of complex tasks (job and resource management) – Resource Allocation Classes mapped to different schemas  EC2 – Instances -> Host Class – Instance Storage, Memory, ECU -> Storage, Processing Resource  GLUE schema – ComputeElementType -> Host Class – CE properties -> Storage, Processing Resource properties  Ganglia schema – HostType-> Host Class – Ganglia metrics -> Resource properties 15

Evaluation  Centralized vs Decentralized allocation time 16

Evaluation  Centralized vs. Distributed 17

Evaluation  Different agent deployment configuration 18

Conclusions  Resource allocation framework which combines several technologies – Semantics for unifying data sources – Multi-agents for adapting to a changing environment – Rule scheduling promising approach  Negotiation combines customer and providers preferences – Different rule examples  Evaluation – Annotations from different schemas – Agents introduce a negotiation overhead – Trade-off between number of resources per agent and agents per machine . 19

Future Work  Improve performance reducing overheads  Exploit the benefits of rule-based resource allocation – Dynamic scheduling policies with rules  Introduce semantics in the resource management interface interoperability – Not only in the resource description 20

Thank you for your attention. Questions? jorge.ejarque@bsc.es