Data Preservation l’esperienza di ESA e il caso studio in EVER-EST ISMAR 05 July 2016 ESA UNCLASSIFIED – For Official Use
Data lifecycle: sensed data needs to be acquired… Level 0 001 4100 002 4102 003 4102 004 6144 005 6150 006 7168 007 6146
...to be combined and processed to get this Processing/c Processing ombining Level 0 Level 1 Level 2 3
...or even further to get this Scientific Publications
...through complex processing schemes Algorithm Manual
1. EO Heritage Data : 2. � � � the health record of our planet � I don’t feel very well Since when are you 1 feeling like this ? I don’t remember but luckily someone has Earth Medical 2 preserved and updated History my health records; here they are � Great!! We have 3 new instruments today but knowing your past is crucial. Let me consult with my colleagues … 4 We identified the cause and 5 have the right treatment for you !!! Thank you !!! 6 I feel much better now !!! Heritage Data Programme (LTDP+) � Heritage Data Programme (LTDP+) � � � � preserve and valorize the past to better shape the future � � � preserve and valorize the past to better shape the future
ESA UNCLASSIFIED – For Official Use Heritage Data Long Term Preservation Programme Objectives 1. ESA EO heritage missions data & knowledge holdings preservation � Secure state-of-art preservation of ESA EO heritage data assets and associated information holdings relying on state of art technologies, systems and standards, and ensuring data authenticity and understandability in the long term (Authority Entity) � with the assumption/awareness that one can never anticipate the exploitation value of the data asset in the future 2. Coordination and cooperation with MS on data preservation � establish an harmonised approach in cooperation with data owners in the MS to ensure European EO data preservation � coordination with non-European actors 3. European interest heritage data assets preservation � prevent loss of non-ESA EO data at risk and judged of European long-term interest
Heritage Data: Valorise the past to better understand the future Spot 5 map ESA UNCLASSIFIED – For Official Use
ESA Long Time Data Series (excerpt) Covered by Covered by LTDP LTDP+
Data Management Plan Discoverability DMP-1: METADATA FOR Curation DISCOVERY DMP-9: DATA REVIEW AND REPROCESSING Accessibility DMP-2 : ONLINE ACCESS DMP-10: PERSISTENT AND RESOLVABLE IDENTIFIERS Usability DMP-3: DATA ENCODING Preservation DMP-7: DATA PRESERVATION DMP-4: DATA DOCUMENTATION DMP-8: DATA AND METADATA DMP-5: DATA TRACEABILITY VERIFICATION DMP-6: DATA QUALITY-CONTROL ESA UNCLASSIFIED – For Official Use
Digital Objects Preservation Generic definition: Digital Preservation is the management and maintenance of digital objects so they can be accessed and used by future users. In Earth Observation Context Digital Objects: Data Records : these include raw data and/or Level-0 data, higher-level products, browse images, auxiliary and ancillary data, calibration and validation data sets, and descriptive metadata; Associated Knowledge : this includes all the Tools used in the Data Records generation, quality control, visualization and value adding, and all the Information needed to make the Data Records understandable and usable by the Designated Community. ESA UNCLASSIFIED – For Official Use
Data Records Preservation ESA UNCLASSIFIED – For Official Use
DATA Records Preservation Data in different formats, quality, accuracy, etc… Heritage Data Programme: Ensures preservation of data and knowledge • Ensure data discoverability and accessibility • Valorise and aligns old data to new data to generate time series •
Data Preservation Heritage Missions SEASAT ESA DATA HOLDINGS RECOVERY • Data recovered from an LTO Tape in 2012 (received from DLR), information needed to understand and open data retrieved from paper documents. • Processor created starting from JERS-1 SAR one: alignment of SEASAT, ALOS and JERS-1 SAR L-band data. Full reprocessing campaign, metadata catalogue, web page creation with all information for users. Data archiving and dissemination. • JERS-1 (1992-1998) SAR and OPS will follow in Q2 2016. Comparison of images from Seasat, ERS-2 and Sentinel-1 to map etreat of two large glaciers in southeast Greenland over a 36-year period.
SEASAT SAR data recovery: � Greenland glaciers as seen by 3 generations of radar missions � � � Seasat L-band SAR ESA data holdings (European coverage Jul-Oct 1978) fully recovered and accessible online for the first time ever. SEASAT, ALOS and JERS-1 data aligned to generate a long time data series in L-band. Comparison of images from Seasat on 16 August 1978, ERS-2 on August 1996 and Sentinel-1 on 20 August 2014 to map the retreat of two large glaciers in southeast Greenland over a 36-year period. Retreat has been estimated at a rate of about 180 m/yr (top glacier) and 61.5 m/yr (bottom glacier)
Heritage datasets NOAA AVHRR • 1987-2011 Nimbus-7 • Coastal Zone Color Scanner ( CZCS ) • Nov 1978 – Jun 1986 Marine Observation Satellites MOS-1 & MOS-1b • MESSR (Multi-Spectral Electronic Self-Scanning Radiometer) • VTIR (Visible and Thermal Infrared Radiometer) • Feb 1987 – Nov 1995 & Feb 1990 – April 1996 Adeos-1 • OCTS (Ocean Colour and Temperature Scanner) • Aug 1996 – Jun 1997 SPOT-1 & SPOT-2 • Feb 1986 – Dec 1990 & Jan 1990 – Jun 2009 • HRV (High Resolution Visible) & HRVIR (High Resolution Visible IR) Priority order being confirmed with CCI representatives
Preserving Bits: Technology Timeline Archives evolution Robotic Libraries
Preserving Bits: Technology Timeline Archive Media evolution CCT 140MB --------- � T10000 8TB
Discoverability: Outputs Timeline Discoverability evolution On-request access (via phone or fax). Paper catalogue quicklook to check availability, followed by generation of image on paper and physical delivery (e.g. courier) Web Catalogues Data Retrieval Time: from days to seconds
Accessibility: Outputs Timeline Data Delivery Format evolution Film / Printouts Paper / Photographic Paper Digital Images Meteosat 1984 Sentinel -1 First ERS image http://esamultimedia.esa.int/docs/corporate/ESA50_slideshow.pdf
Accessibility: Outputs Timeline Media dissemination evolution DLT CCT Exabyte HDD/CD/DVD CCT 140MB --------- � Unlimited
From Selected Scientist to Crowd Science Outputs Timeline Usability evolution Selected scientist Selected and expert users (restricted access and usability) Medium and Large registered Designated community Restricted Communities Open & Free on TEP- Thematic Exploitation Platform Any user can access the data Open & Free on App & Scientific Cloud
Associated Knowledge Preservation ESA UNCLASSIFIED – For Official Use
Associated Knowledge elements Software/Tools: Software Applications : SW related “IT Infrastructure” : • • Data Product generation Compiler • • Quality control Programming language • • Product visualization Storage system • • Value adding Operative System • • Libraries • Databases • Information : Documentation Workflows • • Images Bi directional links • • Metadata file (information on creation, Schemas • • access rights, restrictions, preservation Email • history, and rights management) Multimedia (Video/Audio) • ESA UNCLASSIFIED – For Official Use
Information Preservation ESA UNCLASSIFIED – For Official Use
Information Format for Digital Preservation Text documents (often MS Word, Excel Files, txt, etc.) can From various • Standards and be preserved as: Publications Sources PostScript, PDF, DSSSL, RTF, ASCII, SGML, TIFF, CGM • PostScript, PDF, RTF are proprietary � PDF , DSSSL, SGML not (yet?) widely used � PDF/A, CGM has multiple variants in use � FITS Images can be preserved as: • Loss of Quality JPEG, JPEG2000 • TIFF , Lossless compression TIFF, PBM, PNG, FITS. • FITS Metadata can be preserved as: • ASCII, the most durable format for metadata because it is • widespread, backwards compatible when used with Unicode (superset of ASCII), and utilizes human- ASCII, readable characters, not numeric codes. XML For higher functionality, SGML or XML should be used. • Multimedia can be preserved as: • AVI, QuickTime, MPEG, WMV, MJ2. • MJ2 ESA UNCLASSIFIED – For Official Use
Preservation metadata: PREMIS 1. Descriptive metadata (domain specific EAD for documents, FITs, SAFE, etc, ISAR, ISAD) 2. Administrative (including rights and permissions) 3. Technical (physical needed for implementing most static preservation functions) 4. Structural 5. Documenting digital provenance (history) 6. Documenting relationship in the preservation repository 7. Collaborative 8. Generated during the life cycle of the asset to be preserved
Data Stewardship Best Practices/Guidelines ESA UNCLASSIFIED – For Official Use
Data Stewardship Best Practices Japan Spaceborne Spaceborne USA Brazil Earth Observation Earth Observation Europe Other CEOS Spaceborne, Spaceborne, airborne, and in- airborne, and in- CEOS CEOS CEOS situ situ ESA LTDP Earth Observation Earth Observation Team and CEOS-WGISS GSCB DSIG LTDP WG GEO ? ? NEEDS OGC CCSDS
Recommend
More recommend
