Towards Structure-Aware Earth System Data Management Jakob Lttgau, - PowerPoint PPT Presentation

Towards Structure-Aware Earth System Data Management Towards Structure-Aware Earth System Data Management Jakob Lüttgau, Julian Kunkel, Bryan N. Lawrence, Sandro Fiore, Huang Hua Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 1 / 5

Towards Structure-Aware Earth System Data Management How to manage Earth System Data? What to optimize? Write throughput on generation? Avoid transformation? Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 2 / 5

Towards Structure-Aware Earth System Data Management Data Representations Different views to the same data. Suboptimal serialization on storage. Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 3 / 5

Towards Structure-Aware Earth System Data Management Middleware for Earth System Data Adaptively choose backends. Discriminate by data, metadata and access type. Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 4 / 5

Towards Structure-Aware Earth System Data Management Summary, Status and Outlook � Architectures likely to become more heterogeneous � Systems prohibitively complex for manual optimization Status � Reports and design documents publicly available: http://esiwace.eu (WP4, Deliverables 4.1 and 4.2) � Prototype to demonstrate viability of adaptive tier selection Outlook � Open development of middleware, licensed under LGPL: http://github.com/ESiWACE � Backends being developed for Object Storage and MongoDB � NVM backends as hardware becomes more widely available Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 5 / 5

The ESiWACE project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 675191 Disclaimer: This material reflects only the author’s view and the EU-Commission is not responsible for any use that may be made of the information it contains Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 6 / 5