Scalable gapfilling in spatio- temporal remote sensing data @ReinhardFurrer, I-Math/ICS, UZH NZZ.ch JSM, 2019/07/30
Joint work with Florian Gerber and with contributions of several others URPP Global Change and Biodiversity 175529 2
Global Change and Biodiversity ◮ Species abundance plot scale measurements from the International Tundra Experiment (ITEX) � Shannon biodiversity index on site and plot scale ◮ MODIS/Landsat NDVI images and ASTER elevation data � characterization of the landscape heterogeneity Source: F. Gerber 3
Visual example Source: Gerber et al (2018), TGRS 4
Visual example Source: Gerber et al (2018), TGRS 5
Spatial statistics: prediction � Y ( s ) = Z ( s ) + ε ( s ) , s ∈ D ⊂ R d � Spatial process: . Predict Z ( s 0 ) given y ( s 1 ) , . . . , y ( s n ). Minimize mean squared prediction error (over all linear unbiased predictors) 6
Spatial statistics: prediction � Y ( s ) = Z ( s ) + ε ( s ) , s ∈ D ⊂ R d � Spatial process: . Predict Z ( s 0 ) given y ( s 1 ) , . . . , y ( s n ). Minimize mean squared prediction error (over all linear unbiased predictors) � Best Linear Unbiased Predictor: � � � � − 1 obs BLUP = Cov Z ( s predict ) , Y ( s obs ) Var Y ( s obs ) Z ( s 0 ) = c T Σ − 1 y � (one spatial process, no trend, known covariance structure; otherwise almost the same) 6
Issues of basic, classical kriging Z ( s 0 ) = c T Σ − 1 y = Cov(pred , obs) · Var(obs) − 1 · obs � ◮ “Simple” spatial interpolation . . . . . . on paper or in class! 7
Issues of basic, classical kriging Z ( s 0 ) = c T Σ − 1 y = Cov(pred , obs) · Var(obs) − 1 · obs � ◮ “Simple” spatial interpolation . . . . . . on paper or in class! ◮ BUT: 1. Complex mean structure 2. Unknown parameters 3. Non-stationary covariances 4. Large spatial fields 5. Space-time data on the sphere 7
Methods for large spatial datasets ◮ Sparse Covariance methods: — Spatial Partitioning Heaton — Covariance Tapering Furrer ◮ Sparse Precision methods: — Lattice Kriging Nychka — Multiresolution Approximations Katzfuss — Stochastic Partial Differential Equations Lindgren — Periodic Embedding Guinness — Nearest Neighbor Processes Datta ◮ Low rank approximation: — Fixed Rank Kriging Zammit-Mangion — Predictive Processes Finley ◮ Algorithmic approaches: — Local Approximate Gaussian Processes Gramacy — Metakriging Guhaniyogi 8
Arctic NDVI data MODIS NDIV data (satellite product MOD13A1, NDVI = NIR − R NIR + R ) 9
Kriging is smoothing 10
Kriging is smoothing 10
Interpolation using gapfill 11
gapfill : ranking of the images Day of the year 161 177 193 2004 NDVI 0.8 ● 2005 0.6 Year 0.4 2006 0.2 2007 low high r = 1 2 3 4 5 6 7 8 9 10 11 12 12
gapfill : quantile regression Date: 193 doy 2004 177 doy 2006 177 doy 2005 193 doy 2006 193 doy 2005 Score: 0.65 0.71 0.77 0.88 0.91 Rank: 8 9 10 11 12 q : 0.64 0.12 0.77 ˆ NA NA 13
Interpolation using gapfill Day of the year 145 161 177 193 2004 NDVI 0.8 2005 0.6 0.4 2006 0.2 2007 14
gapfill : prediction uncertainties Day of the year Day of the year 145 161 177 193 145 161 177 193 2004 2004 interval NDVI length 0.8 2005 2005 0.8 0.6 0.6 Year Year 0.4 0.4 2006 2006 0.2 0.2 2007 2007 data and predictions uncertainties 15
gapfill : location 16
gapfill : comparison RMSE × 10 3 17
gapfill : uncertainties (l) Uncertainty contribution from the indicated four steps of the gapfill procedure. (m) Average width of the 90% prediction intervals (40% missing values). (r) Average interval widths and coverage rate per day of the year. 18
Underlying statistical model Z [ a, s, x, y ] is locally g [ a, s ] + Y [ x, y ] + ε [ a, s, x, y ] ◮ g [ a, s ] “arbitrary” ◮ Y ( s ) “some” process ◮ ε “some” noise 19
Visual example II Source: Gerber et al (2018), TGRS 20
Visual example II Source: Gerber et al (2018), TGRS 21
Visual example II Add an overall mean field to model, similar as in IMA (interpolation of mean anomalies) approach Militino et al., TGRS (2019). 22
Summary gapfill Implementation: Intuition: conceptual Model: distribution free Uncertainties: resampling type Practicality: competitive 23
Brute force kriging c T Σ − 1 y Software to exploit the sparse structure spam for : ◮ an R package for sparse matrix algebra ◮ storage economical and fast ◮ versatile, intuitive and simple See Furrer et al. (2006) JCGS; Furrer, Sain (2010) JSS 24
Brute force kriging c T Σ − 1 y Software to exploit the sparse structure spam64 for : ◮ an R package for sparse matrix algebra ◮ storage economical and fast ◮ versatile, intuitive and simple See Furrer et al. (2006) JCGS; Furrer, Sain (2010) JSS ◮ R objects have at most 2 31 elements (almost) ◮ R does not ‘have’ 64-bit integers: stored as doubles ◮ 64-bit exploitation consists of type conversions between front-end R and pre-compiled code Gerber, M¨ osinger, Furrer (2017) CaGeo Gerber, M¨ osinger, Furrer (2018) Software X 24
Summary spam64 gapfill Implementation: Intuition: statistical conceptual Model: frequentist based distribution free Uncertainties: formal resampling type Practicality: play ground competitive 25
References (some, alphabetical) Furrer Sain (2010) spam: A sparse matrix R package with emphasis on MCMC methods for Gaussian Markov random fields JSS 36 1–25 Furrer et al (2017) spam: Sparse Matrix algebra . R package version 2.2-0 Furrer et al (2017) spam64: 64-Bit Extension of the SPArse Matrix R Package ’spam’ . R package version 2.2-0 Gerber Moesinger Furrer (2017) Extending R Packages to Support 64-bit Compiled Code: An Illustration with spam64 and GIMMS NDVI3g Data Comput Geosci 104 109-119 Gerber et al (2018) Predicting Missing Values in Spatio-Temporal Remote Sensing Data. IEEE TGRS 56(5) 2841-2853 Gerber Moesinger Furrer (2017) dotCall64: An Efficient Interface to Compiled C/C++ and Fortran Code Supporting Long Vectors SoftwareX 7 217-221 Heaton et al (2017) A Case Study Competition among Methods for Analyzing Large Spatial Data arXiv :1710.05013 Porcu Alegria Furrer (2018) Modeling Temporally Evolving and Spatially Globally Dependent Data International Statistical Review 86 344–377 Complete list at: www.math.uzh.ch/furrer/research/publications.shtml 26
Recommend
More recommend
