320454 Big Data Project A Instructor: Peter Baumann email: - PowerPoint PPT Presentation

320473 Databases & Web Applications Lab 320454 Big Data Project A Instructor: Peter Baumann email: p.baumann@jacobs-university.de tel: -3178 office: room 88, Research 1 320302 Databases & Web Applications (P. Baumann)

Big Science Data [OGC Ocean Science Interoperability Experiment; image source: Mbari] Databases & Web Applications Lab | Big Data Project (P. Baumann) 2

OGC Coverage Types «FeatureType»  Coverage = digital representation of Abstract Coverage space/time varying phenomenon Grid Coverage • n-D MultiSolid Coverage Rectified GridCoverage MultiSurface Referenceable Coverage MultiCurve GridCoverage Coverage MultiPoint Coverage Databases & Web Applications Lab | Big Data Project (P. Baumann) 3

Facing the Coverage Deluge sensor feeds [OGC SWE] coverage server Databases & Web Applications Lab | Big Data Project (P. Baumann) 4 4

Taming the Coverage Deluge sensor feeds [OGC SWE] coverage server Databases & Web Applications Lab | Big Data Project (P. Baumann) 5 5

Let’s Take a Closer Look... t  Divergent access patterns for ingest and retrieval  Server must mediate between access patterns Databases & Web Applications Lab | Big Data Project (P. Baumann) 6

Our Research  Large-Scale Scientific Information Services (L-SIS) Research Group  flexible, scalable services on massive multi-dimensional scientific data • Particular focus: n-D arrays • Massive = multi- TB … multi -PB per object  Results: • rasdaman array DBMS (www.rasdaman.org), demo at www.earthlook.org • Geoservice standards: OGC WCS suite, http://external.opengeospatial.org/twiki_public/CoveragesDWG/WebHome • ISO 9075 SQL Part 15: MDA (under work) Databases & Web Applications Lab | Big Data Project (P. Baumann) 7

rasdaman: Scalable Array Analytics  „ raster data man ager“: Array Database = SQL + n-D arrays • select img.green[x0:x1,y0:y1] > 130 from LandsatArchive as img  “tile streaming” architecture: scaling from laptop to cloud rasdaman Web visitors www.rasdaman.org Databases & Web Applications Lab | Big Data Project (P. Baumann) 8

Use Case: Satellite ImageTime Series [Diedrich et al 2001] Databases & Web Applications Lab | Big Data Project (P. Baumann) 9

Ea Eart rthServe Server  Big Earth Data Analytics  Up to 130 TB databases for all Earth sciences + planetary science • EU FP7-INFRA, 3 years, 5.85 mEUR  Platform: rasdaman; strictly open standards Cryospheric Airborne Atmospheric Geology Oceanography Planetary Science Science Science marine model runs + Science geological models in-situ data landcover mapping high-altitude drones climate variables Mars geology Databases & Web Applications Lab | Big Data Project (P. Baumann) 10

Database Visualization select encode( struct { red: (char) s.image.b7[x0:x1,x0:x1], green: (char) s.image.b5[x0:x1,x0:x1], blue: (char) s.image.b0[x0:x1,x0:x1], alpha: (char) scale( d.elev, 20 ) }, "image/png" ) from SatImage as s, DEM as d [JacobsU, Fraunhofer 2012; [data courtesy BGS, ESA] [JacobsU, Fraunhofer; data courtesy BGS, ESA] Databases & Web Applications Lab | Big Data Project (P. Baumann) 11

Parallel / Distributed Query Processing  ad-hoc federation  mixed hardware Dataset D select max((A.nir - A.red) / (A.nir + A.red)) - max((B.nir - B.red) / (B.nir + B.red)) Dataset C - max((C.nir - C.red) / (C.nir + C.red)) - max((D.nir - D.red) / (D.nir + D.red)) from A, B, C, D Dataset A Dataset B Databases & Web Applications Lab | Big Data Project (P. Baumann) 12

Secured Archive Integration First-ever direct, ad-hoc mix from protected NASA & ESA services in OGC WCS/WCPS Web client (EarthServer + CobWeb) Databases & Web Applications Lab | Big Data Project (P. Baumann) 13

Demo Databases & Web Applications Lab | Big Data Project (P. Baumann) 14

Next: On-Board Query Intelligence [OPS-SAT: ESA CubeSat] Democratize direct data access [imagery courtesy ESA, NASA] Databases & Web Applications Lab | Big Data Project (P. Baumann) 15

Summary  Project work • embedded in international projects & collaborations  Present  Publish Databases & Web Applications Lab | Big Data Project (P. Baumann) 16

Big Picture  320302 Databases and Web Applications • Fall lecture, undergrad + grad • Advanced course in spring: Information Architectures  320473 Databases and Web Applications Lab • Lab, grad  320454 Big Data Project A • Project, grad New meeting slot: Tue 09:45, Research 1, room 88 Databases & Web Applications Lab | Big Data Project (P. Baumann) 17

Project Task  Pick a topic • http://www.faculty.jacobs-university.de/pbaumann/iu- bremen.de_pbaumann//Courses/ResearchTopics/  Perform task – planful:  Spec document 20% -- Sep 26 Oct 03  Prototype 1: breakthrough implementation 20% -- Oct 17  Prototype 2: ready for benchmark 20% -- Oct 31  Benchmark results 20% -- Nov 14  Publication 10% -- Nov 28  Prototype 3: ready for handover 10% -- Dec 05 Databases & Web Applications Lab | Big Data Project (P. Baumann) 18

Resources  rasdaman website • www.rasdaman.org  demo • www.earthlook.org  Our publications • http://www.faculty.jacobs-university.de/pbaumann/iu-bremen.de_pbaumann//pubs.php  Instructor: • p.baumann@...  ...and the rasdaman team Databases & Web Applications Lab | Big Data Project (P. Baumann) 19

Main Evaluation Criteria  complete wrt. requirements  Solid engineering • bug-free, project & code documentation, coding quality, ...  user-friendliness and appealing look&feel  complexity (in absolute terms and in comparison to other teams' work)  Good writeup • Specification, documentation, paper  (no particular order) Databases & Web Applications Lab | Big Data Project (P. Baumann) 20

Project/Lab Topics  http://www.faculty.jacobs-university.de/pbaumann/iu- bremen.de_pbaumann/teaching.php  -> course list -> list of topics Databases & Web Applications Lab | Big Data Project (P. Baumann) 21