HOCs: Higher-Order Components for Grids Jan Dnnweber (with Martin - PowerPoint PPT Presentation

HOCs: Higher-Order Components for Grids Jan Dünnweber (with Martin Alt, Jens Müller and Sergei Gorlatch) University of Münster, Germany 1

F ROM HPC P ROGRAMMING TO G RID P ROGRAMMING ➜ Low-level programming model remains a challenge in the HPC world (e.g., MPI with explicit send-recv ) ➜ Difficult to manage large numbers of processors ➜ Grids connect several HPC systems together, thereby making the task of application programmers even more challenging ➜ Grid-specific tasks are added: • heterogeneous processing nodes • heterogeneous, dynamic interconnects ➜ Grid middleware deals with these tasks: • Grid Services extend Web Services to become stateful and transient • Web Services allow for remote procedure calls via the Internet (passing proxies and firewalls), using SOAP via HTTP F ROM HPC P ROGRAMMING TO G RID P ROGRAMMING 2

✞ ✝ ✁ ✠ ✠ ✠ ✝ ✝ ✝ ✝ ✝ ✞ ✁ ✞ ✞ ✞ ✞ � ✟ ✟ ✟ ✟ ✁ ✁ ✟ � � � � � � � � � � � ✁ � � � ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✟ P ROGRAMMING G RIDS WITH M ODERN M IDDLEWARE Grid Middleware Grid Servers ✆✄✆✄✆✄✆✄✆✄✆✄✆✄✆✄✆✄✆✄✆✄✆ ✂✄✂✄✂✄✂✄✂✄✂✄✂✄✂✄✂✄✂✄✂✄✂ ☎✄☎✄☎✄☎✄☎✄☎✄☎✄☎✄☎✄☎✄☎✄☎ hosted by request Service Service Factory P T T H / P A O S create create create deliver Result Grid Service Grid Service Instance Instance Application Client provide Grid Service Service Instance ➜ Middleware should take care of resources ➜ What should the programmer take care of? P ROGRAMMING G RIDS WITH M ODERN M IDDLEWARE 3

E XAMPLE : P ROGRAMMER ’ S T ASKS IN G LOBUS • writing GWSDL interface definitions • writing service implementations (OGSI in GT 3) • writing GWSDD deployment configuration • packaging Grid Application Archive (GAR) • deploying the services remotely GSDD GT 3 OGSI OGSA GWSDL GAR GWSDD WSRF Deployment GWSDL GT 3 Descriptor CODE CODE Service Globus GAR Interfaces Implementation Developer Grid Application Archive E XAMPLE : P ROGRAMMER ’ S T ASKS IN G LOBUS 4

A N E XAMPLE GWSDL- INTERFACE DEFINITION <?xml version="1.0" encoding="UTF-8"?> <wsdl:definitions name="MasterService" targetNamespace="http://org.gridhocs/Master" xmlns:gwsdl="http://www.gridforum.org/namespaces/2003/03/gridWSDLExtensions" ... <!-- more namespace declarations --> xmlns="http://schemas.xmlsoap.org/wsdl/"> <wsdl:import location="schema/ogsi/ogsi.gwsdl" namespace="http://www.gridforum.org/namespaces/2003/03/OGSI"/> <wsdl:types> <schema targetNamespace="http://org.gridhocs/Master" xmlns="http://www.w3.org/2001/XML <import namespace="http://schemas.xmlsoap.org/soap/encoding/"/> <complexType name="ArrayOf_xsd_double"> <complexContent> <restriction base="soapenc:Array"> <attribute ref="soapenc:arrayType" wsdl:arrayType="xsd:double[]"/> </restriction> </complexContent> </complexType> ... <!-- more parameter type declarations --> </element> </schema> </wsdl:types> <wsdl:message name="configureRequest"> <wsdl:part name="in0" type="impl:ArrayOf_xsd_string"/> </wsdl:message> <wsdl:message name="configureResponse"> <part name="parameters" element="impl:void"/> </wsdl:message> <!-- more message declarations --> <gwsdl:portType name="MasterPortType" extends="ogsi:GridService ogsi:NotificationSink"> <wsdl:operation name="configure" parameterOrder="in0"> <wsdl:input message="impl:configureRequest"/> <wsdl:output message="impl:configureResponse"/> <fault name="Fault" message="ogsi:FaultMessage"/> </wsdl:operation> ... <!-- more operation declarations --> </gwsdl:portType> </wsdl:definitions> A N E XAMPLE GWSDL- INTERFACE DEFINITION 5

I DEA : G RID P ROGRAMMING WITH C OMPONENTS ➜ Select and parameterize suitable components with distributed implementation in form of Grid Services ➜ Express application by composing components Note: Program’s parameters are programs ⇒ higher-order select Component Repository Divide & Pipeline Map Conquer HOC HOC HOC Farm Scan HOC HOC application specific code, e.g., worker parameter for an image processing farm write Java Grid Expert Application Developer mobile code parameter I DEA : G RID P ROGRAMMING WITH C OMPONENTS 6

G RID P ROGRAMMING USING C OMPONENTS ➜ Components are developed by Grid experts capable of writing efficient parallel code for the target machines ➜ The component repository is packaged with neccesary confi- guration files (WSDL, WSDD) in a GAR file deployed remotely select Component Repository Divide & Pipeline Map Conquer HOC HOC HOC Farm Scan HOC HOC stored in write Java provides GAR Grid Expert Application Developer contains: GWSDL, GT3−Code, GWSDD mobile code parameter G RID P ROGRAMMING USING C OMPONENTS 7

✂ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✂ ✁ ✂ ✂ ✂ ✄ ✂ ✂ ✂ ✁ ✁ ✄ � ✄ ✄ ✄ ✄ ✄ ✄ ✄ � � � � � � � � � ✂ G RID P ROGRAMMING USING C OMPONENTS ➜ For e.g., Farm applications the application programmer only needs to provide two parameters: application specific code for the master and the workers ➜ A distributed evaluator implementation in the form of Grid Services incl. all needed configuration files will be provided by the component developer. ? ? GWSDL CODE GWSDD Service Deployment Interfaces Descriptor Master Worker Parameter Parameter GT 3 CODE Application Grid Farm−Service Developer Expert Implementation G RID P ROGRAMMING USING C OMPONENTS 8

A N E XAMPLE : F ARM -HOC E VALUATION ➀ client obtains Farm Service instance, ➁ Farm service obtains worker services ➂ , distributes calculations ➃ and resassembles result ➄ client 1 Farm−HOC Factory 2 5 Farm−HOC create 1) Farm Service ... 3 3 4 4 4 Worker Worker Factory Factory 2) Worker 2) Worker 2) Worker 2) Worker ... ... Service Service Service Service create create A N E XAMPLE : F ARM -HOC E VALUATION 9

HOC S : H IGHER -O RDER C OMPONENTS FOR G RIDS ➜ Servers provide architecture-tuned implementations of composable and reusable components e.g., Farms, Pipes, ... ➜ Clients access components via common remote interfaces e. g. the FarmHOC: public interface FarmHOC { public void setMaster(Master m); public void setWorker(Worker w); public void configureGrid(String[] availableServers); public double [] compute( double [] input); } ➜ known HOC interface (e.g. FarmHOC) is implemented hardware-specific in server-side HOC-library HOC S : H IGHER -O RDER C OMPONENTS FOR G RIDS 10

P ARAMETERS FOR HOC S : M OBILE C ODE U NITS ➜ Code parameters are hardware-independent, application-specific, and implement interfaces like e.g., public interface Worker { public double [] compute( double [] input); } public interface Master { public double [][] split( double [] input, int numWorkers); public double [] join( double [][] input); } P ARAMETERS FOR HOC S : M OBILE C ODE U NITS 11

S OLUTION : T HE HOC C ODE B ASE We introduce Code Mobility Mechanisms to the OGSA by providing two extensions: ➜ The HOC Code base, an OGSA-compliant Grid Service, that allows for the upload and download of code in the form of primitive byte data. ➜ An HOC Remote Class Loader, that replaces the default class loader and connects to the Code Base when needed. farm("WorkerID",...) client Farm HOC store("WorkerID", ...) Remote Class Loader CodeBase Service retrieve("WorkerID") S OLUTION : T HE HOC C ODE B ASE 12

G RID A PPLICATION P ROGRAMMING USING HOC S Programming Grid Applications with HOCs can now be outlined as follows: Grid hosts Server Y HOC repository A C B A Client HOC1 HOC2 local code ... 2 instantiation A of HOC1 HOC1( ) A,B B ... Server X 4 Server Z code for A 3 CodeBase A HOC repository Service code for B A B 1 C B B C Remote A D B Class Loader A C HOC2 HOC3 G RID A PPLICATION P ROGRAMMING USING HOC S 13