Design for screening, emulation and calibration http://personalpages.manchester.ac.uk/staff/alexis.boukouvalas/ Joint work with Dan Cornford, Milan Stehl´ ık, J.P. Gosling and H. Maruri-Aguilar beamer-aston-logo Design for screening, emulation and calibration 1/39
Overview Design questions permeate many aspects of computer experiments. Sequential screening → Identifying important simulator parameters. Parameter estimation with Heteroscedastic Gaussian Process models. → Learning the parameters of the surrogate model. Calibration. → How well can the simulator describe the observed system? What design to use in each case to minimize number of simulator evaluations needed? beamer-aston-logo Design for screening, emulation and calibration 2/39
Introduction Screening Identify important inputs to the simulator. Emulation A statistical model to the computer code simulator. Typically modelled as a Gaussian Process. Design Where to evaluate the simulator? Use criterion to minimize predictive variance, parameter uncertainty? Standard optimal design theory does not apply to GPs as it assumes independent homoscedastic errors. Calibration beamer-aston-logo Understanding the simulator parameter space. Design for screening, emulation and calibration 3/39
Screening: Overview of sequential approach High-dimensional input spaces may require more simulator runs to train and validate the emulator. Screening the simulator’s inputs for non-linear effects on the output rather than distinguishing between negligible and active effects. Based on the elementary effects method for screening (Morris, 1991). Utilises a threshold value to separate the inputs with linear and non-linear effects. Sequential to keep the number of simulator runs down to a minimum. beamer-aston-logo Design for screening, emulation and calibration 4/39
Screening review Cheap simulators Simulator-based functional ANOVA. e.g. Sobol’ indices where an additive model of first and higher order effects used. Expensive simulators Supersaturated design: use fewer model runs than input dimensions by making assumptions on the number of active inputs or the type of effects on the response e.g. monotonicity of the model output with respect to the inputs → sequential bifurcation (Kleijnen, 2009). Response surface methods → a surrogate model is utilised to approximate the simulator response. e.g. Savitsky et al. (2011) uses a Gaussian-process with a screening prior to encapsulate the assumptions of effect beamer-aston-logo sparsity. Design for screening, emulation and calibration 5/39
Elementary effects method: Morris (1991) 1 Based on calculation of elementary effects (EE) for each input variable: EE i ( x ) = f ( x + ∆ e i ) − f ( x ) . ∆ 2 Use One-At-a-Time (OAT) trajectory designs. Morris Design with five trajectories beamer-aston-logo Design for screening, emulation and calibration 6/39
Elementary effects method 1 Compute sample moments for each input factor 2 Constant or linear effect will have constant EE i and the σ i zero. 3 Linear scaling: Number of simulator runs R ( K + 1) for K factors. 4 Number of trajectories typically small to minimize simulator evaluations, e.g. Morris (1991) used R = 3 although values between 10 and 50 also used (Campolongo et al., 2004, 2007). 5 Initial points for trajectories random from the input space grid (Morris, 1991) or space-filling criterion, maximising the minimum distance between many trajectories (Campolongo et al., 2007). beamer-aston-logo Design for screening, emulation and calibration 7/39
Randomly rotated simplices 1 Caveat: design points may fall on top of each other when projected into lower dimensions. Reduces number of available runs after screening out unimportant factors. 2 Screen with randomly rotated simplices (Pujol, 2009) but suboptimal effect calculation. beamer-aston-logo Design for screening, emulation and calibration 8/39
Sequential Elementary effects method 1 Based on calculation of elementary effects (EE) for each input variable: EE i ( x ) = f ( x + ∆ e i ) − f ( x ) . ∆ 2 Use One-At-a-Time (OAT) trajectory designs, increasing by one trajectory at each stage. 3 If variance of EE exceeds Morris Design with five trajectories threshold, remove factor from consideration. beamer-aston-logo Design for screening, emulation and calibration 9/39
Sequential Elementary effects method 1 Create space-filling design of M starting points, ordering according to the biggest distance between points. 2 Create one-at-a-time for current active factors and run the simulator at those points. 3 Compute elementary effects and their sample moments. 4 Remove factors σ i > σ 0 . These have non-linear effects and should be kept for downstream analysis. 5 Got to step 2 unless all factors have been removed as non-linear or reach maximum number of trajectories. 6 All factors remaining have constant or linear effects. beamer-aston-logo Design for screening, emulation and calibration 10/39
Thresholding solely on the variance of the elementary effects σ i 1 Linear effects of factors may be removed from the simulator output at a preprocessing stage or during the emulation phase. 2 Linear effects may be incorporated in the mean function of a Gaussian Process emulator while omitted from the covariance specification. beamer-aston-logo Design for screening, emulation and calibration 11/39
Heuristic selection of variance threshold σ 0 1 No natural units for the selection of the variance threshold. 2 Specify the threshold in terms of deviation for each factor from a simple regression line. Y ( x i ) = ax i + b + ǫ i where ǫ i ∼ (0 , γ i ). 3 Then threshold is : � 2 γ χ 2 σ 0 = 0 . 99 , R − 1 ∆ 2 ( R − 1) where χ 2 0 . 99 , R − 1 is the 99% quantile of a chi-squared distribution with R − 1 degrees of freedom. 4 Adaptive threshold as it depends on number of trajectories used. beamer-aston-logo Design for screening, emulation and calibration 12/39
Synthetic data 1 Synthetic test function introduced in Morris (1991) 2 Factors x 1 , . . . , x 7 have a non-linear effect on the function output while factors x 8 , x 9 , x 10 have a linear effect and factors x 11 , . . . , x 20 have negligible effect. 3 A threshold value of γ = 2 . 6 was used, corresponds to around 0.005% of the range of the response y. 4 Results: 210 function evaluations R = 10 for the batch EE procedure, 150 function evaluations required on average for the sequential approach. 5 That is an average savings of 28% of simulator runs. beamer-aston-logo Design for screening, emulation and calibration 13/39
Synthetic data results 1 Six of the seven factors with non-linear effects are identified at the first iteration: beamer-aston-logo Design for screening, emulation and calibration 14/39
Synthetic data results: Sobol’ 1 Sobol’ sensitivity analysis method with 220 runs to compute first-order and total indices results in large 95% confidence intervals. 2 Many more runs required to reduce confidence interval. 3 Screening methods such as the EE method, can be utilised prior to a more detailed sensitivity analysis in order to minimise the number of model runs. beamer-aston-logo Design for screening, emulation and calibration 15/39
Rabies model 1 Quantify risk of disease re-introduction by modelling the raccoon dog vector species interactions with red fox. 2 Non-spatial and disease propagation is calculated solely with respect to population dynamics. 3 γ = a factor has near-linear effect if the output varies no more than ∼ 5% from linear. 4 Encapsulates both the internal variability of the stochastic model and our prior definition of a near-linear effect. 5 Model has 13 free parameters and R = 20 → 280 simulator evaluations for EE design was previously used (Singer and Kennedy, 2008) . 6 Sequential approach required 102 runs ∼ 61% savings. beamer-aston-logo Design for screening, emulation and calibration 16/39
Rabies model results Two factors found to have linear effect. Solid line denotes path from beamer-aston-logo previous value of EE samples moments for each factor. Design for screening, emulation and calibration 17/39
Screening summary 1 Identify inputs with non-linear effects with a minimum number of trajectories. 2 Significant computational savings compared to the batch approach on both synthetic data and a real-world simulator. 3 Utilise an easily interpretable variance value γ specified on the simulator output space. 4 Use space-filling design for starting trajectory points with a maximum size or use a low discrepancy space filling sequence. beamer-aston-logo Design for screening, emulation and calibration 18/39
Random Output Simulators Stochastic Simulator A mapping that produces random output given a fixed set of inputs. Gaussian Process Approximation In addition to having a finite number of simulator runs, uncertainty due to stochastic simulator. Our assumed observational model is: y i ( x i ) = t i ( x i ) + ǫ ( x i ) beamer-aston-logo Design for screening, emulation and calibration 19/39
Recommend
More recommend
