SPATIAL SEARCHES IN ASTRONOMY DATABASES MULTI-DIMENSIONAL INDEXING - PowerPoint PPT Presentation

Tamás Budavári (Johns Hopkins University) 7/16/2012

SPATIAL SEARCHES IN ASTRONOMY DATABASES MULTI-DIMENSIONAL INDEXING FOR SIMULATIONS AND OBSERVATIONS Tamás Budavári (Johns Hopkins University) 7/16/2012

Storing Simulations 3 Tamás Budavári  Millennium Run (MPA)  10 billion particles, 64 snapshots  FoF groups and merger trees  Millennium XXL  300 billion particles  MultiDark – Bolshoi  Turbulence simulations (JHU)  1024 4 grid, 27TB ISSAC at HiPACC 7/16/2012

Storing Simulations 4 Tamás Budavári  Millennium Run (MPA)  10 billion particles, 64 snapshots  FoF groups and merger trees  Millennium XXL  300 billion particles  MultiDark – Bolshoi  Turbulence simulations (JHU)  1024 4 grid, 27TB Kai Bürger (TUM, JHU) ISSAC at HiPACC 7/16/2012

Observing Simulations 5 Tamás Budavári  Comparison to real observations  Lots of spatial searches  In the database? ISSAC at HiPACC 7/16/2012

Sky Coverage 6 Tamás Budavári  For precise window function  Virtual surveys ISSAC at HiPACC 7/16/2012

Outline 7 Tamás Budavári  Query shapes in SQL  Indexing with space-filling curve  Combine for spatial searches  Periodic boxes  Celestial sphere ISSAC at HiPACC 7/16/2012

Databases 8 Tamás Budavári  Which one to use depends on the task  Sqlite, MySQL, PostGRES, DB2, Oracle, SQL Server  Free “express versions” of the big ones, too  Customization is a must  There is always something missing  Extend by loading your libraries ISSAC at HiPACC 7/16/2012

Query Shapes Tamás Budavári 9  IShape interface TopoPoint Contains(Point p); TopoShape GetTopo(Box b); Box GetBoundingBox();  Geometric primitives  Sphere, Box, Cone… ISSAC at HiPACC 7/16/2012

Query Shapes Tamás Budavári 10  IShape interface TopoPoint Contains(Point p); TopoShape GetTopo(Box b); Box GetBoundingBox();  Geometric primitives  Sphere, Box, Cone… ISSAC at HiPACC 7/16/2012

Query Shapes Tamás Budavári 11  IShape interface TopoPoint Contains(Point p); TopoShape GetTopo(Box b); Box GetBoundingBox();  Composites  Intersect, Union, Difference… ISSAC at HiPACC 7/16/2012

Query Shapes Tamás Budavári 12  In SQL  UDT ISSAC at HiPACC 7/16/2012

Query Shapes Tamás Budavári 13  Generic  UDT  Boolean  Methods ISSAC at HiPACC 7/16/2012

Query Shapes Tamás Budavári 14  Generic  UDT  Boolean  Methods ISSAC at HiPACC 7/16/2012

Indexing Tables Tamás Budavári 15  Better performance of queries  Instantaneous range searches  Fast JOINs  Syntax CREATE INDEX ix_Name ON Table (X ASC , …) INCUDE (V, …) ISSAC at HiPACC 7/16/2012

Multi-Dimensional Tamás Budavári 16  Map the space to a simple index  Different kinds of Space-Filling Curves  Morton’s Z -curve  Peano-Hilbert Curve ISSAC at HiPACC 7/16/2012

Peano-Hilbert Curve 17 Tamás Budavári  Hierarchical space filling ISSAC at HiPACC 7/16/2012

Peano-Hilbert Curve 18 Tamás Budavári  Hierarchical space filling ISSAC at HiPACC 7/16/2012

Peano-Hilbert Curve 19 Tamás Budavári  Hierarchical space filling ISSAC at HiPACC 7/16/2012

Peano-Hilbert Curve 20 Tamás Budavári  Hierarchical space filling ISSAC at HiPACC 7/16/2012

Also others… Tamás Budavári 21  Morton Z-order  Simple bit interleave  Etc…  Which one to use?  Statistical analyses  Correlation fn ISSAC at HiPACC 7/16/2012

Divide and Conquer Tamás Budavári 22 ISSAC at HiPACC 7/16/2012

Covers for Shapes 23 Tamás Budavári  Inside approximation  Outside overshoot ISSAC at HiPACC 7/16/2012

Covers for Shapes 24 Tamás Budavári  Inside approximation  Outside overshoot  They are Key ranges ISSAC at HiPACC 7/16/2012

Covers for Shapes 25 Tamás Budavári  Inside approximation  Outside overshoot Key between 0 and 3  They are Key ranges ISSAC at HiPACC 7/16/2012

Covers for Shapes 26 Tamás Budavári  Inside approximation  Outside overshoot Key between 0 and 3  They are Key ranges Key between 0 and 7 ISSAC at HiPACC 7/16/2012

Covers for Shapes 27 Tamás Budavári  Inside approximation  Outside overshoot Key between 0 and 3  They are Key ranges Key between 0 and 7 Key between 0 and 3 or Key between 8 and 11 ISSAC at HiPACC 7/16/2012

Covers for Shapes 28 Tamás Budavári  Inside approximation  Outside overshoot Key between 0 and 3  They are Key ranges Key between 0 and 7 Key between 0 and 3 or Key between 8 and 11 ISSAC at HiPACC 7/16/2012

Periodic Boundaries 29 Tamás Budavári  Infinite with periodicity  Have to search all boxes ISSAC at HiPACC 7/16/2012

Periodic Boundaries 30 Tamás Budavári  Infinite with periodicity  Have to search all boxes ISSAC at HiPACC 7/16/2012

Searching in SQL Tamás Budavári 31  Key filter  By Cover  ShiftX,-Y,-Z  Where? ISSAC at HiPACC 7/16/2012

Real! Tamás Budavári 32  E.g., ISSAC at HiPACC 7/16/2012

Online Interfaces Tamás Budavári 33  Largest simulations  Search and visualize  10 billion+ objects and growing…  Indra 512 simulations  Coming soon at JHU ISSAC at HiPACC 7/16/2012

Millennium XXL Tamás Budavári 34 ISSAC at HiPACC

Web Services Tamás Budavári 35  Programming interfaces  Execute SQL queries  Most flexible  Inject probes in simulations  Turbulence  Cosmology ISSAC at HiPACC 7/16/2012

Sky Coverage 36 ISSAC at HiPACC 7/16/2012

No Sky Coverage? 37 Tamás Budavári

Spherical Geometry 38 Tamás Budavári A B A B Green area: A  (B- ε ) should find B if it contains an A and not masked Yellow area: A  (B± ε ) is an edge case may find B if it contains an A. 7/16/2012

Approaches to Consider 39 Tamás Budavári  Pixel maps  Sensitivity, etc…  Equations of shapes  Spherical “vector graphics”  And beyond… ISSAC at HiPACC 7/16/2012

An Observation 40 Tamás Budavári  FITS header with WCS  Image dimensions map to the geometry  More exposures?  No common pixel coordinate-system  Overlapping areas ISSAC at HiPACC 7/16/2012

Common Pixels 41 Tamás Budavári  Pre-defined pages of an atlas  Standard in cartography  Image pyramids of hierarchical pixels  Including HTM, Igloo, HEALPix, SDSSPix , etc…  Always approximate! 7/16/2012

Practical Implementation 42 Tamás Budavári  Looking at Terapixels  We know how to work with images  Now have commodity Internet  We have cheap hard-drives WorldWideTelescope.org Sky in G  gle Earth  Integrated catalogs for efficiency  How about more surveys? ISSAC at HiPACC 7/16/2012

Drawing with Equations 43 Tamás Budavári  Working with 3D normal vectors  Benefits include  No wraparound  No projections  No singularities ISSAC at HiPACC 7/16/2012

Drawing with Equations 44 Tamás Budavári  Direct 3D approach  Halfspace  Circle/Cap  Convex  Simple shapes  Region  Unions of convexes  Patches on the sphere ISSAC at HiPACC 7/16/2012

Point in Region Test 45 Tamás Budavári   Halfspace: one side of a plane ( , ) n c      Inside, when n x c  Convex: a collection of halfspaces  Inside, when inside all halfspaces  Region: a collection of convexes  Inside, when inside any convex ISSAC at HiPACC 7/16/2012

Shape Operations 46 Tamás Budavári  Intersection  Concat halfspace lists  Union  Concat convex lists  Unique coverage  Analytic area  Boolean algebra ISSAC at HiPACC 7/16/2012

Difference of Convexes is a Region Tamás Budavári 47  The set of Regions is closed for the Boolean ops ISSAC at HiPACC 7/16/2012

Simplification 48 Tamás Budavári  Eliminate redundant halfspaces  First handle trivial combinations of constraints  Then solve geometry on the surface  Derive Roots, Arcs, Patches  Eliminate redundant convexes  Some trivial cases, but …  Make convexes disjoint  Unique coverage, area, etc.  Stitch together convexes  When possible ISSAC at HiPACC 7/16/2012

SphericalLib .NET 49 Tamás Budavári  C# code  10k lines  OS independent (Windows, Un*x w/ Mono)  Documentation via Sandcastle  Great performance!  Sloan Digital Sky Survey in 10s (13× larger than USA in area) ISSAC at HiPACC 7/16/2012

Numerical Imprecision 50 Tamás Budavári  Double precision calculations  Lots of tricks from Graphics Gems ɛ  IEEE 754 standard  Degeneracy  When are two vectors the same?  Spatial resolution limit  Roughly 30 cm on Earth ISSAC at HiPACC 7/16/2012

51 Tamás Budavári Sky coverage of the Sloan Digital Sky Survey’s 5 th Data Release and the Galaxy Evolution Explorer’s 2 nd Public Release

Region in SQL Tamás Budavári 52 ISSAC at HiPACC 7/16/2012