Uniform access to heterogeneous Uniform access to heterogeneous grid infrastructures with grid infrastructures with ISGC 2008, Taiwan Sylvain Reynaud
Plan Plan � Introduction � Infrastructure heterogeneity – Interoperability solutions – Security – What is JSAGA ? – Data management – Execution management � Middleware heterogeneity � Transport data to/from – Security worker nodes – Data management – Execution management � Conclusion Uniform access to heterogeneous grid infrastructures with JSAGA 2
Introduction Introduction Interoperability solutions Interoperability solutions � The medium term solution (GIN) – Forward jobs to other grid infrastructures EGEE • Adapt existing components user • Or use CE as a gateway � The long term solution (OMII-Europe) – Adapt components to set common interfaces • Standard (Open Grid Forum) • Use common mechanism (e.g. for security) � Client-side adaptors and translators solution – Meta-schedulers: GridWay, GridSAM… regional grids regional grids – Client API: GAT, CoG-Kit… …and cluster cluster Uniform access to heterogeneous grid infrastructures with JSAGA 3
Introduction Introduction What is JSAGA ?... A simple example What is JSAGA ?... A simple example job desc. EGEE RUGBI JSAGA JSAGA gLite Globus plug-ins plug-ins staging JDL RSL graph delegate selection & files staging WMS SRM WMS SRM input data GridFTP GridFTP LCG- -CE CE LCG- -CE CE WS- -GRAM GRAM LCG LCG WS WS- -GRAM GRAM WS firewall job job Uniform access to heterogeneous grid infrastructures with JSAGA 4
Plan Plan � Introduction � Infrastructure heterogeneity – Interoperability solutions – Security – What is JSAGA ? – Data management – Execution management � Middleware heterogeneity � Transport data to/from – Security worker nodes – Data management – Execution management � Conclusion Uniform access to heterogeneous grid infrastructures with JSAGA 5
Hide middleware heterogeneity Hide middleware heterogeneity SAGA as the uniform interface SAGA as the uniform interface COMMANDS COMMANDS SAGA API SAGA API JSAGA ENGINE JSAGA ENGINE jsaga-context-init jsaga-context-info Context Security Mgr Security Mgr jsaga-context-destroy jsaga-cat security jsaga-cp jsaga-ls jsaga-logical jsaga-mkdir File Data Service Data Service jsaga-mv jsaga-rm jsaga-rmdir protocol jsaga-test Job Service Job Service jsaga-job-run monitor jsaga-job-submit control Job jsaga-job-monitor jsaga-job-getoutput Simple API for Grid Application Non-standard API � � – object-oriented API – service-oriented API – to enable end-users to … – to enable engine to provide SAGA develop easily grid applications capabilities as efficiently as possible • << functional packages for fundamental • With low-level or optimized plug-in methods programming capabilities >> • Plug-ins expose supported methods & techno • Plug-ins throw homogeneous exceptions Uniform access to heterogeneous grid infrastructures with JSAGA 6
Hide middleware heterogeneity Hide middleware heterogeneity SAGA as the uniform interface SAGA as the uniform interface Implementation focuses on… … Implementation focuses on JSAGA ENGINE JSAGA ENGINE � Modularity and efficiency Security Mgr Security Mgr – see next slides… � Software quality security – plug-ins are validated with a reusable test-suite Data Service Data Service � Lightness protocol – minimize external dependencies Job Service Job Service • resolve libraries version conflicts monitor • regenerate stubs control • remove useless dependencies – install only what you need… Uniform access to heterogeneous grid infrastructures with JSAGA 7
Hide middleware heterogeneity Hide middleware heterogeneity Security Security Security Mgr Security Mgr Data Service Data Service Security Mgr Security Mgr Job Service Job Service Uniform access to heterogeneous grid infrastructures with JSAGA 8
done Hide middleware heterogeneity Hide middleware heterogeneity construction planned Security Security � Data and execution plug-ins /etc/grid-security/certificates CA – declare • supported security plug-ins � Security plug-ins /tmp/x509up_u_$UID – declare pass proxy proxy proxy phrase path • attributes usage rules • attributes default values ~/.globus/user*.pem – engine set the context instance cert key p12 to first matching usage rule jsaga-context-init jsaga-context-info jsaga-context-destroy InMemCred Login / pwd G. RFC820 G. Legacy MyProxy Globus VOMS X509 PKI Uniform access to heterogeneous grid infrastructures with JSAGA 9
Hide middleware heterogeneity Hide middleware heterogeneity Data management Data management Security Mgr Security Mgr Data Service Data Service Data Service Data Service Job Service Job Service Uniform access to heterogeneous grid infrastructures with JSAGA 10
done Hide middleware heterogeneity Hide middleware heterogeneity construction planned Data management Data management � The engine enables caching – opened connections Data Service Data Service – information about entries gsiftp – content of entries through cache � The engine chooses the most gsiftp v1 gsiftp v2 dCache appropriate plug-in methods to efficiently provide the requested P functionality Logical files h y s i c – e.g. third-party transfer a l f i l e s catalog rbyteio cache gsiftp https mail http sftp rns srb file zip ftp tar lfn Uniform access to heterogeneous grid infrastructures with JSAGA 11
done Hide middleware heterogeneity Hide middleware heterogeneity construction planned Data management Data management � The engine enables caching – opened connections Data Service Data Service – information about entries srm – content of entries through cache SAGA � The engine chooses the most Data Service Data Service appropriate plug-in methods to efficiently provide the requested P functionality Logical files h y s i c – e.g. third-party transfer a l f i l e s catalog rbyteio cache gsiftp https mail http sftp srm rns srb file zip ftp tar lfn Uniform access to heterogeneous grid infrastructures with JSAGA 12
Hide middleware heterogeneity Hide middleware heterogeneity Execution management Execution management Security Mgr Security Mgr Data Service Data Service Job Service Job Service Job Service Job Service Job control Job control Job monitoring Job monitoring Uniform access to heterogeneous grid infrastructures with JSAGA 13
done Hide middleware heterogeneity Hide middleware heterogeneity construction planned Execution management Execution management � Monitoring and job control are Monitoring Plug-in interfaces: independent querying / listening – e.g. monitor with gatekeeper, gLite- individual job / list of jobs / filtered jobs query listen query LB or gsiftp LIST command individual individual filtered job job jobs � Monitoring strategy of the engine depends on – monitoring capabilities of the plug-in getStatus waitFor – monitoring activities of the user SAGA user interface: getStatus / waitFor • current, on going and past activities Job control Job control Job monitoring Job monitoring gatekeeper gLite-WMS gatekeeper unicore6 unicore6 gLite-LB wsgram gsiftp-ls wsgram remote gridvm gridvm cream cream PBS local ssh … Uniform access to heterogeneous grid infrastructures with JSAGA 14
Hide middleware heterogeneity Hide middleware heterogeneity Execution management Execution management COMMANDS COMMANDS SAGA API SAGA API JSAGA ENGINE JSAGA ENGINE jsaga-context-init jsaga-context-info Context Security Mgr Security Mgr jsaga-context-destroy jsaga-cat security jsaga-cp jsaga-ls jsaga-logical jsaga-mkdir File Data Service Data Service jsaga-mv jsaga-rm jsaga-rmdir protocol jsaga-test Job Service Job Service jsaga-job-run monitor jsaga-job-submit control Job jsaga-job-monitor jsaga-job-getoutput SAGA- SAGA -like + like + jobs Collection Mgr Collection Mgr file jsaga-jobcollection-run expression jsaga-jobcollection-submit language Job Collection jsaga-jobcollection-monitor jsaga-jobcollection-getoutput Uniform access to heterogeneous grid infrastructures with JSAGA 15
Hide middleware heterogeneity Hide middleware heterogeneity Execution management Execution management Security Mgr Security Mgr Data Service Data Service Collection Mgr Collection Mgr Job Service Job Service Collection Mgr Collection Mgr Uniform access to heterogeneous grid infrastructures with JSAGA 16
done Hide middleware heterogeneity Hide middleware heterogeneity construction planned Execution management Execution management • Resources requirements � JSDL + extensions – Candidate tags – Job collection • Data staging • Parametric job – Alternatives (e.g. local/remote) • Termination rules – Pre/post processing • Retry rules – Elapsed time – Elapsed time – % jobs per status – Status – Individual jobs • Selected resources • Job identification – Identifier • Application – Information – POSIX application • Middleware specific extension – Script application JSDL + ext. SAGA RSL-2 RSL-4 JDL Uniform access to heterogeneous grid infrastructures with JSAGA 17
Recommend
More recommend
