Migrating from Grid to Cloud: Migrating from Grid to Cloud: Migrating from Grid to Cloud: Migrating from Grid to Cloud: Case Study from GEO Grid Case Study from GEO Grid Kyoung-Sook Kim Data Science Research Group Information Technology Research Institute
Interoperability and Integration Interoperability and Integration Geo* Contents Applications Individual interfaces & protocols Geology Satellite image Satellite image for information exchange for information exchange It is not easy to • Access and (re)use • • Manage and control Manage and control Environment & Energy GIS data • Integrate geospatial contents and services • Cooperate with different Cooperate with different organizations http:// www.geofabrik.de /data/ shapefiles.html Disaster Response High cost of application High cost of application Sensor data Sensor data development and management 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 2
GEO Grid: Overview GEO Grid: Overview *Global Earth Observation e- Science infrastructure Science infrastructure for global earth observation systems with heterogeneous geospatial datasets Applications Geo* Contents Standard Standard - based interoperability & Integration Standard Standard - based interoperability & Integration Geology Geology Satellite image - Ease - of - use interfaces (search, access, process data) - Low integration efforts Web Web Web Web WMS WMS WFS WFS Services Services WPS WCS CS -W SOS Environment & Energy GIS data http:// www.geofabrik.de /data/ shapefiles.html Disaster Response Disaster Response • • Security management Security management Sensor data Sensor data • Scalability of resources (computing and storage ) 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 3
GEO Grid: GEO Grid: Overview Overview e- Science infrastructure Science infrastructure for global earth observation systems with heterogeneous geospatial datasets Applications Geo* Contents Standard Standard Standard - based interoperability & Integration Standard - based interoperability & Integration Geology Geology Satellite image - Ease - of - use interfaces (search, access, process data) - Low integration efforts Web Web Web Web WMS WMS WFS WFS Services Services WPS WCS CS -W SOS Environment & Energy GIS data http:// www.geofabrik.de /data/ shapefiles.html Disaster Response Disaster Response Sensor data Sensor data Grid Computing Technologies & Grid Computing Technologies & Standards for sharing of geographically distributed resources for sharing of geographically distributed resources and controlling resource sharing rules 2014/5/15 Geospatial World F orum 2014 @ Geneve , Switzerland 4
GEO Grid: Overview GEO Grid: Overview e- Science infrastructure Science infrastructure for global earth observation systems with heterogeneous geospatial datasets Applications Geo* Contents Standard Standard Standard - based interoperability & Integration Standard - based interoperability & Integration Geology Geology Satellite image - Ease - of - use interfaces (search, access, process data) - Low integration efforts Web Web Web Web WMS WMS WFS WFS Services Services WPS WCS CS -W SOS Environment & Energy GIS data http:// www.geofabrik.de /data/ shapefiles.html Disaster Response Disaster Response Sensor data Sensor data Grid Computing Technologies & Grid Computing Technologies & Heterogeneous Heterogeneous Distributed Distributed Security and Metadata Storage and Distributed and User Management Grids Database Parallel Management Management (LOD/RDF) (LOD/RDF) Federation Federation Federation Processing Processing Processing Grid and Cloud Infrastructure Grid and Cloud Infrastructure 2014/5/15 Geospatial World F orum 2014 @ Geneve , Switzerland 5
GEO GEO Grid: Grid: System Architecture System Architecture Outcomes Outcomes Outcomes Outcomes Decision Decision Share Share Sci - Tech Sci - Tech Making Making Disaster Disaster Business Business R&D R&D Support Support Support Support Information Information Information Information QuiQuake QuiQuake ASTER Flood Flux Science Hotspot Application Application DEM Simulation Monitoring DCP Detection Others Services Services Services Services OGC standard web services OGC standard web services WMS WCS CS -W WPS SOS WFS Heterogeneous Heterogeneous Security and Grid and Cloud Grid and Cloud and Distributed Computing Grid Storage Grids User Infrastructure Infrastructure Database and Cloud Management Federation GeoSpatial Resources GeoSpatial Resources GeoSpatial GeoSpatial Resources Resources Our HPC cluster Our HPC cluster Data Archive Services Geological Map of 40m 40m 40m Japan 30m 30m 30m 20m 20m 20m ASTER satellite archive 10m 10m 10m ~= 2million scenes Satellite Geological -0.01 -0.01 -0.01 - 0.05 - 0.1 - 0.05 - 0.1 - 0.05 - 0.1 ~= 1PB -0.2 -0.2 -0.2 CfreeVG10,SRTM, Other -0.5 -0.5 -0.5 Data Data sensors sensors MODIS, JERS -1 geospatial info geospatial info PALSAR,LANDSAT PALSAR,LANDSAT resources 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 6
An An Example Example of GEO Grid of GEO Grid Public Services Public Services • QuiQuake (https://gbank.gsj.jp/QuiQuake/index.en.html) • • Quick estimation system for Quick estimation system for earthquake maps triggered by observation records • Provides wide - ranging and detailed (250m - grid) strong ground motion maps for quick disaster response. • Produces the results soon after the occurrence of an earthquake . • Archiving all data of past earthquake occurrences. 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 7
QuiQuake QuiQuake Realization Realization in GEO Grid in GEO Grid Share Share Disaster Disaster Information Information Information Information Decision Decision Sensor Data Making Making Support Support Provide the results as OGC services and formats High Performance Performance Data Computing Integration Geospatial Data Grid and Cloud Grid and Cloud Application Application GeoSpatial Resources GeoSpatial Resources Outcome Outcome Infrastructure Infrastructure Services Services 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 8
GEO Grid Security GEO Grid Security • GSI (Grid Security Infrastructure) – S ecure communication (authenticated and confidential ) between Grid elements – Security across organizational boundaries – Security across organizational boundaries – ”Single sign - on" for users of the Grid in multiple resources and/or sites • • VO (Virtual Organization) VO (Virtual Organization) Management A dynamic group of individuals, groups, or organizations who define organizations who define the conditions and rules for sharing resources 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 9
Migration from Grid to Cloud Migration from Grid to Cloud 2011 T ohoku E arthquake GE O Grid at AIS T (~200km away) was damaged by the earthquake. Cannot operate GEO Grid for 2 months by recovery works & subsequent power Cannot operate GEO Grid for 2 months by recovery works & subsequent power restriction request. BUT s atellite images were very important to understand situations and provide useful information for helping the disaster response activities. provide useful information for helping the disaster response activities. 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 10
ASTER Comparison Before and After Tsunami ASTER Comparison Before and After Tsunami • AS TE R : – Japanese sensor (optical, 15m) on NASA Terra satellite satellite – Can create 3D model by making stereo - matching – AIST can produce true - color images. Mouth of Kitakami - river 11
GEO Grid Disaster GEO Grid Disaster Responses Responses • Rapid Evacuation : – Moved our minimum environment to oversea volunteer servers • Using VM/Cloud technology – – Changed our daily network ( data transfer ) workflow to use these servers Changed our daily network ( data transfer ) workflow to use these servers • To get/receive latest images from the satellite(ASTER) ALOS/PALSAR TDRS Until March 11 - Data providing - Portal 70 GB/day (ASTER) AIST JAXA 360 GB/day (PALSAR) NASA ERSDAC - Archive (tape, B - ray) - Archive (on - Disk) - Processing - WMS • • AS AS TER data: NAS TER data: NAS A A ERSDAC ERS DAC AIS AIS T T (AIST: processing , WMS, portal site, and data archive) • P ALS AR data: J AXA ERS DAC AIST 2014/5/15 Geospatial World F orum 2014 @ Geneve , Switzerland 12
What We DID What We DID Terra/ASTER ALOS/PALSAR Data Flow and Services TDRS from March 11 to April 20 - Portal Google (AIST) JAXA NASA ERSDAC Orkney - Processing - WMS - WMS • • ASTER data: NASA AS TE R data: NAS A ERSDAC E R S DAC (AIS (AIST) T) (Orkney: processing and WMS (Orkney: processing and WMS , Google: portal site) , Google: portal site) • P ALS AR data: J AXA E R S DAC (AIS T) 2014/5/15 Geospatial World F orum 2014 @ Geneve, S witzerland 13
Recommend
More recommend
