Data-centric Computing for Earth Observation . . Comments and - PowerPoint PPT Presentation

. . Lizhe Wang June 22nd, 2012 Chinese Academy of Sciences Center for Earth Observation and Digital Earth (CEODE) Lizhe Wang Data-centric Computing for Earth Observation . . Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation

. . Lizhe Wang . . Data-intensive computing for RS image processing Multi-datacenter computing for Earth Observation . . . Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . Contents 1 Data-centric Computing for EO: General Discussion 2 Current Work 3 Comments and Discussion

. . airborne & spaceborne RS data Laboratory of Digital Earth Sciences research on geo-spatial information science, remote sensing, and scientific platform for Digital Earth . . Center for Spatial Data . . . . . Lizhe Wang processing, distribution and archiving of airborne RS data . Comments and Discussion . . . . Data-centric Computing for EO: General Discussion Current Work . acquisition, processing and storage of . Satellite Remote Sensing Center receiving, archiving and processing RS data from satellites (home & abroad) Airborne Remote Sensing Center Data-centric Computing for Earth Observation . About CEODE

. . Lizhe Wang . . Data flow in remote sensing engineering emitted from aircraft or satellites) means of propagated signals (e.g. electromagnetic radiation use of aerial sensor technologies to detect objects on Earth by Remote sensing . . . Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . Remote sensing methodology: data-centric system acquisition → transmission → processing → archiving → distribution → visualization

. 50GB EO Resolution EO Platform Data Set 1000m Meteorological satellite 1 0.5GB 100m EO-1 10m . SPOT 5TB 1m Quickbird 0.5PB 0.1m GeoEye 50PB Lizhe Wang . . 1MB/day . . . . . Data-centric Computing for EO: General Discussion Current Work Comments and Discussion . . . Sources 10TB/Day Ground sensors 10,000 5TB/flight Airborne RS Data generation 100 High-resolution satellite No. Sources Data-centric Computing for Earth Observation . How big? 10 6 Pixels/km 2 10 2 10 4 10 6 10 8

. 5 700 (MB) 80.6173 Geometric correction 80 700 (MB) 36.08247 Gridding of AIRS Data 500 (GB) Radiometric correction 1.36 AWI from MODIS Data 1 500 (GB) 2.5 . . Lizhe Wang 80 896.9981 . . . . . . Data-centric Computing for EO: General Discussion Current Work Comments and Discussion . 305 (MB) . . Processing Process No. Data size Throughput (MB/S) NDVI 32 Data-centric Computing for Earth Observation . How intensive?

. . Lizhe Wang . . Parameter assessment and model development Complex processing algorithms: pixel based, region based, and global based Highly unstructured with various metadata information social information Multiple dimensions: time, space, multiple spectrum, geographical information, Multiple sources: various sensors, various resolutions, various regions . . . Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . How complex?

. Multi-datacenter computing for Earth Observation Lizhe Wang Data-intensive computing for RS image processing Multi-datacenter computing for Earth Observation . Data-intensive computing for RS image processing Comments and Discussion . Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . Current work

. ground-borne) and quantitative RS production system Lizhe Wang continues and space continues 40 types of comment-featured remote sensing products: time publish PB-level data processing: data fusion, production, sharing and 6 satellite data centers management, service oriented Production system: industry standards, event driven, workflow Objective and common-featured RS production system Sub-project: integrated EO from multiple satellite datacenters Integrated earth observation (satellite, airborne and . . Data-intensive computing for RS image processing Multi-datacenter computing for Earth Observation Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . Project background and objective

. Comments and Discussion Lizhe Wang . . Data-intensive computing for RS image processing Multi-datacenter computing for Earth Observation Data-centric Computing for Earth Observation Current Work Data-centric Computing for EO: General Discussion . . . . . Software architecture Product Product Product Product Functionalities search order publish download Metadata Workflow Order aggregation management management Multi-datacenter level Algorithm management/query/publish Data scheduling/query/search Metadata Monitoring & Task scheduling management security Datacenter level Algorithm deployment & task execution Data movement/storage/download

. Metadata management: extraction, Lizhe Wang Production system management, automatic deployment standardized development, metadata Algorithm/executables management: download Data management: move, index, search, aggregation, query, publish . . Data-intensive computing for RS image processing Multi-datacenter computing for Earth Observation Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . Key challenges

. Research project funded by Chinese Academy of Sciences: Lizhe Wang Rumtime framework Optimized parallel file system Remote Sensing RS-GPPS: Generic Parallel Programming Skeletons from processing Objective: high-performance data-intensive RS image Engineering project: Parallel Image Process System (PIPS) Collaboration with German DFG project 2011 – 2014 Data-centric Computing for Earth Observation . . Data-intensive computing for RS image processing Multi-datacenter computing for Earth Observation Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . . Project background and objective

. . . . . . Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing . Lizhe Wang Data-centric Computing for Earth Observation . RS image processing algorithms

. . . . . . Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing . Lizhe Wang Data-centric Computing for Earth Observation . Parallel programming with RS-GPPS

. . . . . . Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Multi-datacenter computing for Earth Observation Data-intensive computing for RS image processing . Lizhe Wang Data-centric Computing for Earth Observation . The Architecture of RS-GPPS

. Big? It depends! Lizhe Wang NoSQL: No RDBMS: Yes Database: Workflow, dataflow: Yes Hadoop/HDFS: No MPI/OpenMP/PGAS Fine-grained parallelism: Programming: Intensive? Yes! 10-100 TB: a global EO problem . 10 GB: 1 RS image Data set: . Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . Data-centric Computing for Earth Observation . Experiences and comments for RS applications

. Current problem: Lizhe Wang Programming model Fine-grained vs. coarse-grained memory Shared memory vs. distributed Road map: Runtime system Performance Data-centric Computing for Earth Observation . . Comments and Discussion Current Work Data-centric Computing for EO: General Discussion . . . . . Discussion

. . . . . . Data-centric Computing for EO: General Discussion Current Work Comments and Discussion Thank you! Happy Duanwu Festival! Contact: Lizhe.Wang@gmail.com Lizhe Wang Data-centric Computing for Earth Observation