Grid-enabled for Digital Archives: The Development and Applications - PowerPoint PPT Presentation

Grid-enabled for Digital Archives: The Development and Applications of SRM- SRB interface for DataGrid Services Wei-Long UENG Academia Sinica Grid Computing Taiwan

Outline • Introduction to the National Digital Archive Program Data Grid Services • Deployment of Data Grid as the Information Infrastructure • Interoperation – SRM-SRB interface implementation and application • Summary

Goals of NDAP • Preserve national cultural collections • Popularize fine cultural holdings • Revitalize cultural heritage and cultural development • Invigorate cultural, content and value-added industries • Enhance research, education and learning • Promote knowledge and information sharing • Improve literacy, creativity and quality of life • Embrace international communities and collaboration Cultural, Academic, Socio- economic & Educational (CASE) Values

Digital Archives in Taiwan • Three Levels – Archive Level • high resolution, for preservation purposes • accessible on a case-by-case basis – Open-Market Level • medium resolution, for value-added, commercial purposes • accessible for a fee by membership or by licensing agreement – Public Information Access Level • low resolution, for educational purposes • accessible to the public free of charge

Information Infrastructure • Digital archives/libraries are widely recognized as a crucial component of a global information infrastructure • Pursue research and development efforts on many aspects of digital archive/library technologies: – (1) the establishment of standardized information reference guidelines for digital content creation, storage, and processing – (2)the development of common and application-specific information processing infrastructure and tools

Requirements • Long-Term Preservation and Data Creation – preserving ability to read (physically) and understand (logically) • Full Spectrum and Precise Metadata in Collection, Object and Management Level • Workflow Support: Digital Information Life-Cycle – Create--> Content Analysis & Annotation--> IPR Protection --> Repurposing-->Multi-modal/Integrative Search --> Archive • Data Exploration across Institutional and Disciplinary Domains • Petabyte Scale Storage Management with Performance

Requirements • Long-Term Preservation and Data Creation – preserving ability to read (physically) and understand (logically) • Full Spectrum and Precise Metadata in Collection, Object and Management Level • Workflow Support: Digital Information Life-Cycle A New Information Infrastructure is Required! – Create--> Content Analysis & Annotation--> IPR Protection --> Repurposing-->Multi-modal/Integrative Search --> Archive • Data Exploration across Institutional and Disciplinary Domains • Petabyte Scale Storage Management with Performance

Why Grid is needed in multiple projects • To conduct R&D and integration tasks to help digitize and network the collections and resources of different institutes or multiple projects. • A Data Grid Services is necessary to provide virtualized middleware services for sharing data across distributed, heterogeneous data resources in different administrative and security domains.

Why Grid infrastructure for NDAP in general is required • Digital Archives in Taiwan demand reliable storage systems for persistent digital objects, well-organized information structure for effective content management, efficient and accurate information retrieval mechanisms, and flexible services for variant users needs. • Grid technology enlightened a viable solution for long- term preservation and processing diversified heterogeneous Petabyte scale digital archives. • Data Grid aims to set up a computational and data- intensive grid of resources for data analysis. It requires coordinated resource sharing, collaborative processing and analyzing on huge amounts of data produced and stored by many institutions.

Workflow of Digital Archives

Middleware Deployment • The SRB system in Taiwan is used for the long-term preservation of the digital contents produced by the digital archives projects. • The system was deployed by the Academia Sinica Grid Computing Centre (ASGC) in early 2004. • Constituted from 8 sites in different institutes.

System Architecture

Interoperation • With the nature of Grid, the most effective way to share data resources is to integrate the data sources and data grids. • SRM for SRB interface is to make the popular SRB Data Grid System interoperable with the EGEE infrastructure. • Support the standard SRM services for SRB 14

Why SRM • SRM is an unique interface for accessing diffident backend storages for diffident middleware. • Easy to develop applications to adapt different backend storages. • Provide space and file management on the storage system. • SRM is the web service interface and the implementation usually depends on the backend storage technology. • Grid middleware needs to access files with an uniform interface 15

Concept 16

Concept 16

Architecture Overview Users/applications File transfer (gridftp) SRM API Web Service Core Auxiliary Filecatalog Data server management Gridftp/management API SRB+DSI 17

• User Interface 18

• User Interface • Gridftp/ management commands • SURL 18

• User Interface • Gridftp/ management commands • SURL • Hostname:t-ap51.grid.sinca.edu.tw • Info: AMGA server • Host information 18

• Hostname: t-ap20.grid.sinica.edu.tw • User Interface • Info: SRB server (SRB-DSI installed) • Return some information • Gridftp/ management commands • SURL • Hostname:t-ap51.grid.sinca.edu.tw • Info: AMGA server • Host information 18

• Hostname: t-ap20.grid.sinica.edu.tw • User Interface • Info: SRB server (SRB-DSI installed) • Return some information • Gridftp/ management commands • SURL • Hostname:t-ap51.grid.sinca.edu.tw • Info: AMGA server • TURL • Host information • Hostname: fct01.grid.sinica.edu.tw • The end point: httpg://fct01.grid.sinica.edu.tw:8443/axis/services/srm 18 • Info: SRM interface

List the content of a SRB directory [sary357@fct01 isgc2008]$ sh SrmLs.sh Please input the SURL you'd like to list the content:srm://fct01.grid.sinica.edu.tw:8443/axis/services/srm?/AS/home/sary357.ASGC/sary3571/ *************** SrmLs ********************** Status code: SRM_SUCCESS Explanation: null ========================================================== ===== The individual status and result ========= The status code:SRM_SUCCESS The explanation:null File name:/AS/home/sary357.ASGC/sary3571 The Size:0 The File Type:DIRECTORY The owner of this file/directory:sary357 The owner permission of this file/directory:RW ************** The sub dir and files ***************** File name:/AS/home/sary357.ASGC/sary3571/steps2.html1 CheckSumType: The Size:9 The File Type:FILE The owner of this file/directory:sary357 The owner permission of this file/directory:RW File name:/AS/home/sary357.ASGC/sary3571/20080409 The Size:0 The File Type:DIRECTORY 19 The owner of this file/directory:sary357

Putting a file sary357@fct01 isgc2008]$ sh SrmPut.sh Please input the local file name you'd like to put:/tmp/testFile1.txt Please input the SURL you'd like to put:srm://fct01.grid.sinica.edu.tw:8443/axis/services/srm?/AS/home/sary357.ASGC/sary3571/testFile1.txt ***************SrmPrepareToPut*********************** Token:1207660686127 Status code: SRM_SUCCESS Explanation: null =============================================================== URI: srm://fct01.grid.sinica.edu.tw:8443/axis/services/srm?/AS/home/sary357.ASGC/sary3571/testFile1.txt | The status code: SRM_SPACE_AVAILABLE | TURL: gsiftp://t-ap20.grid.sinica.edu.tw:2811/AS/home/sary357.ASGC/sary3571/testFile1.txt **********Try to upload the data****************** Upload TURL: gsiftp://t-ap20.grid.sinica.edu.tw:2811/AS/home/sary357.ASGC/sary3571/testFile1.txt start.... Upload TURL: gsiftp://t-ap20.grid.sinica.edu.tw:2811/AS/home/sary357.ASGC/sary3571/testFile1.txt end...... *************** SrmPutDone ********************* Status code: SRM_SUCCESS Explanation: null 20

Getting a file [sary357@fct01 isgc2008]$ sh SrmGet.sh Please input the SURL you'd like to get:srm://fct01.grid.sinica.edu.tw:8443/axis/services/srm?SFN=/AS/home/sary357.ASGC/111.jpg Please input the local file name you'd like to store:/tmp/111.jpg *************** SrmPrepareToGet ********************** The status: Status code: SRM_SUCCESS Explanation: null ======================================================== The individual result: The SURL: srm://fct01.grid.sinica.edu.tw:8443/axis/services/srm?SFN=/AS/home/sary357.ASGC/111.jpg the status code:SRM_FILE_PINNED the explanation of this SURL:null The TURL:gsiftp://t-ap20.grid.sinica.edu.tw:2811/AS/home/sary357.ASGC/111.jpg ************* Got the TURL **************************** ************* Download file *************************** download from TURL: gsiftp://t-ap20.grid.sinica.edu.tw:2811/AS/home/sary357.ASGC/111.jpg start.... download from TURL: gsiftp://t-ap20.grid.sinica.edu.tw:2811/AS/home/sary357.ASGC/111.jpg end...... And the file name after downloading is /tmp/111.jpg. *************** SrmReleaseFiles ******************* Status code: SRM_SUCCESS Explanation: null 21 ==========================================================