SPECTRAL AND MORPHING ENSEMBLE KALMAN FILTERS AND APPLICATIONS Jan Mandel, Jonathan D. Beezley, Loren Cobb, Ashok Krishnamurthy, University of Colorado Denver Adam K. Kochanski University of Utah Krystof Eben, Pavel Jurus, and Jaroslav Resler Czech Academy of Sciences 31st International Symposium on Forecasting, Praha, June 29, 2011
ACKNOWLEDGEMENTS This work was partially supported by the U.S. National Science Foundation grant AGS-0835579, National Institutes of Health grant 1RC1 LM01641-01, National Institute of Standards and Technology Fire Research grant 60NANB7D6144, and the Academy of Sciences of the Czech Republic project M100300904. A part of this work was done during visits of Jan Mandel and Jonathan Beezley at the Department of Nonlinear Modeling, Institute for Computer Science, Czech Academy of Sciences. The hospitality of the Institute during those visits is gratefully acknowledged.
EXAMPLE 1: WILDFIRE SIMULATION Simulation of FireFlux experiment (Clements et al., 2007) with WRF 3.3 and SFIRE 2011. Simulation setup by Adam Kochanski, visualization in VAPOR by Bedrich Sousedik. http://openwfm.org
E XAMPLE 2: A N E PIDEMIC IN S WEDISTAN • Pandemic influenza in a population with no immunity. • Human Development Index set to HDI = 0.30 (similar to Congo). • No air traffic, dirt roads between cities. • Initial epidemic has two epicenters. • Entire country has only one IDP camp for the ill and displaced (homeless). • No public health efforts except near the IDP camp. Author: Loren Cobb Funding: Nat’l Defence College of Sweden and the US Joint Staff (J8 Directorate) Project: STRATMAS II Date: 2001
EXAMPLE 3: RAIN FORECASTING Project Medard, Department of Nonlinear Modeling, Institute for Informatics, Czech Academy of Sciences, http://www.medard-online.cz
WHAT DO THESE THREE PROBLEMS HAVE IN COMMON? • Strongly nonlinear • Solutions exhibit coherent, moving features: fire line, epidemic wave, weather front • Data assimilation: ingesting data while the simulation is running • standard Bayesian amplitude corrections often give nonphysical solutions • position corrections needed in addition to amplitude • Also, in wildfire and epidemic, statistical variability causes secondary fires and epidemics, which keep growing, do not dissipate
DATA ASSIMILATION • The state-space model: the model state is a probability density p ( u ) • Simulation updates the probability distribution by mapping it forward in time. • Data given with an error estimate, as data likelihood p ( u|d ) : the probability density of the state u given data d • Bayesian update of the state: p analysis ( u ) = const p ( u|d ) p ( u ) • If all is Gaussian, in particular p ( u|d ) = const exp( - ( d-Hu ) T R -1 ( d-Hu ) /2 ), we get new state mean U analysis = U + QH T ( HQH T +R ) -1 ( d-HU ) • This formula is also used in the non-Gaussian case anyway (also justified as least squares or maximum likelihood) • Now the game is to estimate the state covariance Q
COVARIANCE ESTIMATION • The classical Kalman filter evolves the state covariance Q assuming the model is linear. • Optimal statistical interpolation (OSI) takes a human guess for the covariance Q • Ensemble Kalman filter (EnKF) is a confluence of two techniques: • Ensemble forecasting a.k.a. scenarios - the ensemble members are independent simulations: if it rains in 50% of the members we say that the probability of rain is 50% • Estimation of the state covariance Q from the ensemble (e.g., by sample covariance ) • But this gives the EnKF analysis covariance too small. Fix: • randomly perturb the data, or • deterministically move of ensemble members to ensure correct sample covariance (adjustment EnKF, square root EnKF,...)
SOME OF THE CHALLENGES • Large state vector (easily, in GB), large ensembles needed (often, 100s) • Covariance estimates • the state consists of several random fields on a physical domain: the covariance function between locations drops off with their distance (sample covariance with a small sample does not; it needs tapering ) • but the covariance function also changes with the location in the domain; it needs localization • Non-gaussian probability distributions - the Holy Grail of data assimilation • here, the locations of coherent features may have approximately gaussian distribution but the value of the field at a fixed location surely does not
SO WHAT IS NEW AND DIFFERENT IN THIS TALK? • Optimal statistical interpolation by Fast Fourier Transform (FFT) - needs just a laptop instead of a supercomputer. Automatic tapering. • Morphing - position corrections, not just amplitude, makes distributions close to gaussian. • FFT EnKF - approximate the covariance by the diagonal of the sample covariance in the frequency space . Then, small ensembles are enough. • Wavelet EnKF - provides also an automatic localization.
COVARIANCE ESTIMATION BY THE FFT APPROXIMATION BY THE DIAGONAL IN THE FREQUENCY SPACE Given covariance Ensemble of 5 random functions Sample covariance FFT estimation From Mandel et al. (2010b)
MORPHING • Moving coherent features: need also position correction • Replace the state by an extended state: deformation of a reference field + a residual • by automatic registration: multiscale optimization, also related to advection field found in radar analysis • run the EnKF on the extended states: • probability distributions closer to gaussian • recover ensemble members from the deformation and the residual fields • basically, replace linear combinations by morphs : • Intermediate states in the middle created automatically from a linear combination of deformation fields and residual fields • tricky: the right kind of combination to avoid ghosting
OPTIMAL STATISTICAL INTERPOLATION BY FFT • Use negative power of Laplace operator for the covariance • Green’s function drops off from the diagonal, cheap FFT implementation • Combined with morphing, example: rain fields Forecast Data Standard analysis Analysis with morphing Duplicate storm center
OPTIMAL STATISTICAL INTERPOLATION WITH MORPHING AND MISSING RADAR DATA Forecast OPERA radar data Analysis The storm line direction as well as the overall precipitation intensity are interpolated between the forecast and the data
MORPHING ENKF FOR A WILDFIRE SIMULATION Truth Forecast Analysis - standard EnKF breaks down Analysis - morphing EnKF OK
COVARIANCE ESTIMATION BY WAVELETS Wavelets are basis functions localized both in space. We use orthogonal wavelets; wavelet transform of a function consists of the coefficients of its expansion in a wavelet basis . Wavelet EnKF uses diagonal approximation of covariance: • Apply the wavelet transform to ensemble members • Compute the diagonal of the sample covariance in the wavelet space • Apply the inverse wavelet transform Diagonal approximation of covariance in frequency and wavelet space has been An example of a wavelet: Coiflet 25 justified for weather fields in other data assimilation contexts (Deckmyn and Berre 2005,..)
ORTHOGONAL WAVELET BASIS Wavelet value r e b m u n t e l e v a W Obtained by shift and argument scaling of a single function, the mother wavelet Organized in octaves (the argument scaling): multiresolution structure
FFT AND WAVELET COVARIANCE ESTIMATIONS, 2 VARIABLES Covariance, sample of 1000 Variable 1, sample of 5 Variable 2, sample of 5 Covariance, sample of 5 FFT estimation, sample of 5 Wavelet estimation, sample of 5 Estimation by FFT results in a distribution that is homogeneous in space, smearing the distribution across the domain. Wavelet estimation keeps the spatial structure, while filtering out spurious long-distance correlations.
DATA ASSIMILATION FOR EPIDEMIC SPREAD • The standard S-I-R model (susceptible, infected, recovered) with a spatial diffusion added to model infection spread in space • Key: the probability distribution of cases in an epidemic distribution is Poisson. Use the Poisson distribution, not gaussian, also to randomize the data in the EnKF, then all falls in place. • Go back to the EnKF roots: statistical ensemble for the forecast and uncertainty estimation, but use the assumed covariance i.e., optimal statistical estimation
Bayesian Tracking using OSI Time 10 Truth Forecast Analysis
Bayesian Tracking using OSI Time 20 Truth Forecast Analysis
Bayesian Tracking using OSI Time 30 Truth Forecast Analysis
Bayesian Tracking using OSI Time 40 Truth Forecast Analysis
Bayesian Tracking using OSI Time 50 Truth Forecast Analysis
Recommend
More recommend
