User Tools and Languages for Graph-based Grid Workflows User Tools - PowerPoint PPT Presentation

User Tools and Languages for Graph-based Grid Workflows

User Tools and Languages for Graph-based Grid Workflows Global Grid Forum 10 – Berlin, Germany Grid Workflow Workshop Andreas Hoheisel (andreas.hoheisel@first.fraunhofer.de) Fraunhofer Institute for Computer Architecture and Software Technology – Berlin, Germany

Outline Fraunhofer Resource Grid Grid Job Orchestration Petri nets Grid Job Builder Workflow Enactment Dynamic Workflow Refinement Fault management Parameter Studies Hoheisel_2004_GGF10-Workflow-workshop Grid Job Handler Conclusions and future work Page 3

Fraunhofer Resource Grid (FhRG) Ob Objectives jectives Development and implementation of a stable and robust Grid infrastructure Software layer on top of Globus to enable fast realizations of distributed applications (computational science & engineering) Integration of available resources Hoheisel_2004_GGF10-Workflow-workshop Open Source Software: http://www.eXeGrid.net/ Page 4

n User Run Simple Run Assemble/ Solve Globus Job Grid Job Run Grid Job Problem Web Services Web Portal l GResourceDL GResourceDL 1 Task Mapping Repository GResourceDL (SOAP) <hardware> Java Application <software> <data> n GResourceDL Grid Job Builder <user> (SOAP) GJobDL (SOAP) Web Service Hoheisel_2004_GGF10-Workflow-workshop m 1 Grid Job Handler (Meta) Scheduler RSL (GRAM), GridFTP MDS, LDAP Globus 2.4 Page 5

Hoheisel_2004_GGF10-Workflow-workshop Page 6

Hoheisel_2004_GGF10-Workflow-workshop Grid Job Orchestration Page 7

What is a Grid Job? Grid job id job Composition of Grid resources forming grid applications Grid resource id resource Software, hardware, data, (people) Atomic job omic job Single task, indivisible component of a Grid job Execution of a software component with input data Future: Invocation of a WebService method call Component Model Componen Model Loosely coupled software components Executables that read input files and write output files Communication via files and GridFTP GJ GJobDL DL Description of Grid jobs on an abstract level Independent from Grid infrastructure Hoheisel_2004_GGF10-Workflow-workshop Connecting software components and data Based on Petri Nets XML Page 8

Different Grid Workflow Approaches Inherent model Inherent model Workflow is defined inside the software components (e.g. MPI, CORBA, Cactus ) Externa External mo mode del Workflow is defined on top of software components Complete view of workflow (e.g. scripts or graphs) Scripts Scripts GridAnt, JPython (XCAT) Combined scripts + graphs Combined scripts + graphs WSFL, XLANG, BPEL4WS, UNICORE, GSFL Gra Graphs hs DAG: Condor DAGman, Symphony Hoheisel_2004_GGF10-Workflow-workshop Petri net: GJobDL (Fraunhofer Resource Grid) Page 9

ERAMAS – Pollutant Transport in the Atmosphere: Accident → Source → Atmospheric Transport → Exposure Input data dw m Output data dw m start Diagnostic Windfield Model Output data Input data evaporation evaporation User Interface Monte Carlo Evaporation Model Output data Input data Accident-specific data Lagrange Model Lagrange Lagrange (Location, Pollutant,...) Preprocessing Ouput data Input data Risk assessment Risk assessment Risk Assessment Model Hoheisel_2004_GGF10-Workflow-workshop Input data Visualization (GIS -> html) end Visualization Page 10

Why Petri Nets? Prob Problem lem Description of complex workflows of grid jobs DAG DAG Directed Acyclic Graph (see Condor, Cactus, UNICORE) no loops A PARENT A CHILD B C PARENT B C CHILD D B C D Petri Ne Petri Nets ts Graphical flow control of discrete systems Directed graph, can be made Turing complete Hoheisel_2004_GGF10-Workflow-workshop Page 11

Petri Nets Places Places Files, buffers, control places Transitions Transitions Software components, control transitions Arcs from places to transitions cs from places to transitions (Place is input place of transition) Arcs from transitions to cs from transitions to places places (Place is output place of transition) Tokens Tokens Data, State (done, failed) Rule Rule A transition is activated if all input places are filled with tokens and all output places have not reached their maximum capacity of tokens Hoheisel_2004_GGF10-Workflow-workshop Ref Refinement nement A single part of a Petri Net can be replaced by a sub Petri Net Description of state scription of state A Petri Net describes workflow and state of a system Page 12

Petri Nets (from van der Aalst und Kumar, 2000 ) Task Sequence Choice Hoheisel_2004_GGF10-Workflow-workshop Page 13

Hoheisel_2004_GGF10-Workflow-workshop Condition Petri Nets Page 14

Petri Nets Parallel execution with synchronization Hoheisel_2004_GGF10-Workflow-workshop Page 15

Petri Nets Parallel execution without synchronization Hoheisel_2004_GGF10-Workflow-workshop Page 16

Petri Nets Wait all with time out Hoheisel_2004_GGF10-Workflow-workshop Page 17

Petri Nets Wait any with time out Hoheisel_2004_GGF10-Workflow-workshop Page 18

Hoheisel_2004_GGF10-Workflow-workshop While … do Petri Nets Page 19

Hoheisel_2004_GGF10-Workflow-workshop Grid Job Builder /// Page 20

Hoheisel_2004_GGF10-Workflow-workshop Workflow Enactment Page 21

Dynamic Workflows Ref Refinement nement The Grid Job Handler supplements the workflow during runtime by introducing additional tasks File File tran transfer sfer A GridFTP task may be introduced automatically to transfer files that are not available on the remote computer transfer task A B A GridFTP idFTP B Software dep Software deploy oyment ment A software deployment task may be introduced to install software components on a remote computer Hoheisel_2004_GGF10-Workflow-workshop deployment task transfer transfer unpack unpack A A A.tar.gz A.tar.gz A.tar.gz A.tar.gz Page 22

Implicit Fault Management Grid middlewa id middleware Fault management that is included in the Grid middleware Petri Net refinemen Petri Net refinement Fault management tasks are introduced automatically if the submission or execution of an atomic task fails re-schedule max. 3 times count < 4 count < 4 failed failed failed Hoheisel_2004_GGF10-Workflow-workshop A A Page 23

Explicit Fault Management Petri Net workflow model Petri Net workflow model The user defines the fault management explicitly by including user-defined fault management tasks in the Petri Net of his Grid job failed failed failed B begin end A done done done C Hoheisel_2004_GGF10-Workflow-workshop Page 24

Parameter Study n = total number of runs (e.g. 10,000) p = number of parallel runs (e.g. 128) all submitted end Post all Post al els else els else Pre Pre Calc Calc Post Post begin Pre all Pre all Pre Pre Calc Calc Post Post if count<n if count<n if count<n-1 if count<n-1 Pre Pre Calc Calc Post Post Pre Pre Calc Calc Post Post Token generator Token consumer Hoheisel_2004_GGF10-Workflow-workshop p parallel runs Page 25

Application flow of Grid Job Handler • Read the GJobDL document • Create Petri Net from this job description • Verify the Petri Net (well-formedness, liveliness, deadlocks, pits, ...) • Start the Grid Job (own thread) • Collect all activated transitions • Evaluate conditions Invoke resource mapping � repository, (meta-)scheduler • • Refine the Petri N Refine the Petri Net � insert GridFTPs, fault management, etc. if necessary Hoheisel_2004_GGF10-Workflow-workshop • Create and submit atomic jobs using grid middleware (e.g. GRAM) • The transition fires, if atomic job is “done“ or has “failed“. Page 26

Hoheisel_2004_GGF10-Workflow-workshop Page 27

Hoheisel_2004_GGF10-Workflow-workshop Page 28

Hoheisel_2004_GGF10-Workflow-workshop Page 29

Hoheisel_2004_GGF10-Workflow-workshop Page 30

Hoheisel_2004_GGF10-Workflow-workshop /// /// Parameter Study /// Page 31

Hoheisel_2004_GGF10-Workflow-workshop Future Work Conclusions and Page 32

Conclusions Description of w Description of workf rkflow ow External, graph-based workflow definition GJobDL uses Petri Nets instead of directed acyclic graphs to model workflow of Grid jobs Petri Petri Ne Nets ts Easy orchestration of complex workflows, including conditions and loops Dynamic wo mic workflow mod flow model Petri Nets can be refined and modified during runtime Adding new tasks to the workflow, e.g.: transfer tasks software deployment tasks fault management tasks implicit � automatic, included in Grid middleware Fault Fault Management Management Hoheisel_2004_GGF10-Workflow-workshop explicit � user-defined, included in workflow model Page 33

Future Work Now: one transition � one executable Tight coupling sche Tight coupling scheme? me? Future: one transition � one method call (?) � Grid Service (OGSA), Web Service Simulation of Petri Nets Simulation of Petri Nets Prediction of Petri Nets for advanced reservation of resources ( � scheduler) based on software und hardware benchmarks Fault management Fault management Workflow check pointing, recovery of grid jobs Hoheisel_2004_GGF10-Workflow-workshop Page 34