The European Commission’s science and knowledge service Joint Research Centre
The European Open Science Cloud for the Earth Observation Science Community (EOSC4EOSC). EGI Conference 2019 (Federated Data Management) Guido Lemoine (guido.lemoine@ec.europa.eu)
Rationale The European Union’s Copernicus program is the global leader in Earth Observation for environmental monitoring. Data from 4 different sensors types (7 operational platforms) are provided under a full, free and open data license. Currently data rate is ~ 6 PB/year. Includes Sentinel-1 and -2 high resolution (10 m) global coverage, with 6- and 5-day revisit. Lower resolution S-3 and S-5P with near-daily coverage. Data accessed by 1000s of users via ESA ScienceHub, downloaded 10-fold. Leading to dramatic scale up in data uptake, spatially, temporally and spectrally.
Rationale (2) An obvious “Big Data Analytics” domain, but poorly served by European capacity Currently, BDA (much) better served by AWS and Google Earth Engine . European solutions tends to be fragmented, across thematic use, national audiences, user sectors. No concerted science community support. European solutions are not state-of-the-art, don’t scale well, are not user friendly. Europe does not have a consolidated plan to maintain the full data archive !
Continental scale crop classification Using European Sentinel-1 data Using European LUCAS 2018 survey data On Google Earth Engine...
EOSC for EO Science Community Become the permanent host of the on-line long term archive of Sentinel data Closely coupled with cloud computing infrastructure Federate existing initiatives (CollGS, DIAS, TEP) into a sustainable platform Benefit from expertise in other EOSC Big Data user communities for cutting edge structured data access, massive parallel processing, and intelligent APIs Leverage leading European Open Source solutions in geospatial data analysis into a scalable computing infrastructure, with novel renumeration concepts Bring the added value of EO data into wider [science] community of [non-space] users, through open science data sharing and education.
EOSC4EOSC technical requirements Optimize access to S3 object storage for 20Kx20K UInt16 or Float32 images E.g. Cloud Optimized GeoTIFF, multi-resolution tile pyramids, data cubes (?) Need to handle map projections “on the fly” Integrate with large vectors data sets (vector tiles, Big Query) APIs that hide cloud resource marshaling, provide access to extendable libraries of core and advanced processing routines Orchestration of massive parallel workflow, across image scales (kubernetes?) Support interactive analysis & visualization, background batch processes
5x DIAS Each DIAS hosts (a part of) the Sentinel archive Typically on S3 type storage With OpenStack compute resources With a “business” focus, incl. 3rd party exploitation JRC assists EU Member States to onboard DIAS for CAP monitoring
S1 provides consistency (~200 images per year) 500K agricultural parcels with declared practice Machine learning applied to S1 time series Splits 95-5 “outliers” Developed in GEE, now migrated to the Copernicus DIAS A EuroGEOSS demonstrator (+EOSC)
Manager node hosts database that provides S3 endpoints and parcel Vector extracts run on worker nodes vector and stores extracts. Uses python3+gdal+boto3 image. Uses postgresql/postgis image. Each worker has access to S3 store.
Towards EOSC4EOSC A vision document was recently drafted. Sets out goals and identifies key actors. A first estimate of storage and compute costs (up till 2027). Proposes technical solutions, practical steps. Currently under discussion.
Thanks! Any questions? Contact guido.lemoine@ec.europa.eu
Recommend
More recommend
