PRACTICAL ANALYTICS Tams Budavri / The Johns Hopkins University - PowerPoint PPT Presentation

PRACTICAL ANALYTICS Tamás Budavári / The Johns Hopkins University 7/19/2012

Statistics Tamás Budavári  Of numbers  Of vectors  Of functions  Of trees ISSAC at HIPACC 7/19/2012

Statistics Tamás Budavári  Description, modeling, inference, machine learning  Bayesian / Frequentist / Pragmatist ? Supervised Unsupervised Discrete Classification Clustering Continuous Regression Dimensional Reduction ISSAC at HIPACC 7/19/2012

What’s Large? Tamás Budavári  VOLUME  Say >100TB today but tomorrow? Moving target…  COMPLEXITY  The raw dataset are simple unlike their derivatives  DEFINITION?  Large when you cannot apply the “usual” tools ISSAC at HIPACC 7/19/2012

Data LARGE !! ISSAC at HIPACC 7/19/2012

Data LARGE !! ISSAC at HIPACC 7/19/2012

Large? Tamás Budavári  Sample size ISSAC at HIPACC 7/19/2012

Large? Tamás Budavári  Sample size ISSAC at HIPACC 7/19/2012

Large? Tamás Budavári  Dimensions  Ratio of surface/volume grows all points are lonely in high dimensions ISSAC at HIPACC 7/19/2012

ISSAC at HIPACC 7/19/2012

Keeping Up? Tamás Budavári  Image processing  Catalog extraction  O ( n )  What is difficult?  O ( n log n )  O ( n 2 ), … Worse w/ Moore’s law ISSAC at HIPACC 7/19/2012

Fundamental Challenges Tamás Budavári  Cross-identification of sources  To assemble multicolor catalogs  Drop-outs from sky coverage  To constrain fluxes not detected  Constraining physical properties  To interpret the data ISSAC at HIPACC 7/19/2012

Cross-Identification From long-tail science to the largest experiments ISSAC at HIPACC 7/19/2012

Recording Observations Tamás Budavári  Astronomers drew it…  Now kids do it on SkyServer #1 by Haley  ISSAC at HIPACC

Multicolor Universe Tamás Budavári ISSAC at HIPACC 7/19/2012

Eventful Universe Tamás Budavári ISSAC at HIPACC 7/19/2012

Cross-Identification One of the most fundamental analysis steps ISSAC at HIPACC 7/19/2012

What is the Right Question? Tamás Budavári  Cross-identification is a hard problem  Computationally, Scientifically & Statistically  Need symmetric n -way solution  Need reliable quality measure  Same or not?  Distance threshold? Maximum likelihood? ISSAC at HIPACC 7/19/2012

Tabletop Astronomy Tamás Budavári  Imagine the observed sky has only 6 pixels  One object : one die  Observing : rolling a die  Locality : die is loaded  Sky : a bag of dice ISSAC at HIPACC 7/19/2012

Model Comparison: Same or Not? Tamás Budavári  Crossmatch : draw two dice with replacement  Same or not?  Bayes Factor is the ratio of the  Likelihood of “Same”  Likelihood of “Not”  Likelihood of a hypothesis?  Sum over all possibilities ISSAC at HIPACC 7/19/2012

Model Comparison: Same or Not? Tamás Budavári  Crossmatch : draw two dice with replacement  Same or not?  Bayes Factor is the ratio of the  Likelihood of “Same”  Likelihood of “Not”  Likelihood of a hypothesis?  Sum over all possibilities ISSAC at HIPACC 7/19/2012

Model Comparison: Same or Not? Tamás Budavári  Model for loaded dice is matrix of probabilities  E.g., loaded toward l =1  Etc. for l =2…6  2-way case  Same:  Not:  n -way: same ISSAC at HIPACC 7/19/2012

Model Comparison: Same or Not? Tamás Budavári  Model for loaded dice is matrix of probabilities  E.g., loaded toward l =1  Etc. for l =2…6  2-way case  Same:  Not:  n -way: same ISSAC at HIPACC 7/19/2012

Model Comparison: Same or Not? Tamás Budavári  Model for loaded dice is matrix of probabilities  E.g., loaded toward l =1  Etc. for l =2…6  2-way case  Same:  Not:  n -way: same ISSAC at HIPACC 7/19/2012

Celestial Sphere Tamás Budavári  Continuous functions  General formalism  Accuracy is a density fn on sky ISSAC at HIPACC 7/19/2012

Modeling the Astrometry Tamás Budavári  Astrometric precision  A simple function  Where on the sky?  Anywhere really… ISSAC at HIPACC 7/19/2012

Same or Not? Tamás Budavári OR  The Bayes factor SAME  H: all observations of the same object at m NOT  K: might be from separate objects at { m i } ISSAC at HIPACC 7/19/2012

Same or Not? Tamás Budavári OR  The Bayes factor SAME  H: all observations of the same object at m NOT  K: might be from separate objects at { m i } ISSAC at HIPACC 7/19/2012

Same or Not? Tamás Budavári OR  The Bayes factor SAME  H: all observations of the same object at m NOT  K: might be from separate objects at { m i } ISSAC at HIPACC 7/19/2012

Same or Not? Tamás Budavári OR  The Bayes factor SAME  H: all observations of the same object at m On the sky Astrometry NOT  K: might be from separate objects at { m i } ISSAC at HIPACC 7/19/2012

Same or Not? Tamás Budavári OR  The Bayes factor SAME  H: all observations of the same object at m On the sky Astrometry NOT  K: might be from separate objects at { m i } ISSAC at HIPACC 7/19/2012

Analytic Results Tamás Budavári  Normal distribution  Flat and spherical  Gauss and Fisher  2-way results ISSAC at HIPACC

Normal Distribution Tamás Budavári  Astrometric precision:  Fisher distribution:  Analytic results:  For high accuracies: ISSAC at HIPACC 7/19/2012

Wikipedia: Interpretation Tamás Budavári ISSAC at HIPACC 7/19/2012

Probability of a Match Same or not? ISSAC at HIPACC 7/19/2012

From Priors to Posteriors Tamás Budavári  Bayes factor is the connection ISSAC at HIPACC 7/19/2012

From Priors to Posteriors Tamás Budavári  Posterior probability from prior & Bayes factor  Prior probability of a match  Like dice in a bag: 1/ N and N 1  n  In general? ISSAC at HIPACC 7/19/2012

From Priors to Posteriors Tamás Budavári  Different selections  Nearby / Distant  Red / Blue  But only 1 number ISSAC at HIPACC

Self-Consistent Estimates Tamás Budavári  Prior has an unknown fudge-factor  Educated guess TB & Szalay (2008)  Or solve for it: ISSAC at HIPACC 7/19/2012

Simulations Tamás Budavári  Mock objects  With correct clustering  U 01 values as properties 0 1  Simulated sources  Subsets: N 1 N 2  Overlap: N ★ ISSAC at HIPACC 7/19/2012

Simulations Tamás Budavári  Mock objects  With correct clustering  U 01 values as properties 0 1  Simulated sources  Subsets: N 1 N 2  Overlap: N ★ ISSAC at HIPACC 7/19/2012

Simulations Tamás Budavári  Quality  Multiple matches Explained by simple model of point sources! Heinis, TB, Szalay (2009) ISSAC at HIPACC 7/19/2012

Proper Motion Tamás Budavári  Same hypotheses but different parameters  Just need  prior to integrate Sources from SDSS ISSAC at HIPACC 7/19/2012

Proper Motion Tamás Budavári  Same hypotheses but different parameters  Just need  prior to integrate Kerekes, TB+ (2010) Sources from SDSS ISSAC at HIPACC 7/19/2012

Matching Events Tamás Budavári  Streams of events in time and space  E.g., thresholded peaks in signal-to-noise (1) (x) (2) ISSAC at HIPACC 7/19/2012

Dropouts from Sky Coverage ISSAC at HIPACC 7/19/2012

Drawing with Equations Tamás Budavári TB, Szalay & Fekete (2010) r = 0.6  r = 0.5  ISSAC at HIPACC 7/19/2012

Matching in Practice ISSAC at HIPACC 7/19/2012

Open SkyQuery Tamás Budavári  Following our 1 st prototype  Successful  Not bayesian  Limitations ISSAC at HIPACC 7/19/2012

SkyQuery – The 3 rd Generation Tamás Budavári  Dynamic federation of astronomy databases  Query the collection as if they were one  The 3 rd gen tool coming this summer  Cluster of machines running partitioned jobs  Proper probabilistic exec with variable errors ISSAC at HIPACC 7/19/2012

SkyQuery Tamás Budavári  Almost pure standard SQL ISSAC at HIPACC 7/19/2012

SkyQuery Tamás Budavári  Almost pure standard SQL ISSAC at HIPACC 7/19/2012

SkyQuery Tamás Budavári  Almost pure standard SQL ISSAC at HIPACC 7/19/2012

SkyQuery Tamás Budavári  Almost pure standard SQL  Added XMATCH  Verifiable  Flexible ISSAC at HIPACC 7/19/2012

Tamás Budavári ISSAC at HIPACC 7/19/2012

 HST Crossmatch Catalog RELEASE AT AAS Tamás Budavári  SQL pipeline  Astrometric TB & Lubow (2012) correction  Subpixel precision ISSAC at HIPACC

 HST Crossmatch Catalog RELEASE AT AAS Tamás Budavári  FoF groups  Possible chains  Bayesian model selection  Chainbreaker ISSAC at HIPACC

 HST Crossmatch Catalog RELEASE AT AAS Tamás Budavári  Lots of matching sources during HST’s long life TB & Lubow (2012) TB & Lubow (2012) ISSAC at HIPACC