multi index monte carlo and multi index stochastic
play

Multi-Index Monte Carlo and Multi-Index Stochastic Collocation Ra - PowerPoint PPT Presentation

Mar 14, 2024 •25 likes •1.57k views

Multi-Index Monte Carlo and Multi-Index Stochastic Collocation Ra ul Tempone Alexander von Humboldt Professor, RWTH Aachen, Germany and KAUST, Saudi Arabia. Course material developed together with Fabio Nobile (CSQI-MATHICSE, Ecole

  1. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Variance Reduction techniques – Control variates Does control variates pay in terms of computational work? Assume that the work to generate the pair ( X j , Y j ) is a factor (1 + θ ) times the work to generate Y j . Method # samples Computat. Work � � 2 Var [ Y ] � � 2 Var [ Y ] C C MC TOL TOL � � 2 Var [ Z ( β ∗ )] � � 2 Var [ Z ( β ∗ )] C C MC + Cont.Var. (1 + θ ) TOL TOL We see that the strategy only pays if Var [ Y ] > (1 + θ ) Var [ Z ( β ∗ )] i.e. θ 1 > (1 + θ )(1 − ( ρ XY ) 2 ) iff ( ρ XY ) 2 > 1 + θ. Obs: As said before, we need to use some knowledge of Y to find a sufficiently correlated X , i.e. that it has a relatively small θ and a large ρ XY . . . Question: Can we use the structure of SDEs and random PDEs for this purpose? 7 / 101

  2. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) Outline Variance Reduction techniques – Control variates Multi Level Monte Carlo (MLMC) Main idea and complexity analysis Further optimization of MLMC Normality of MLMC estimator CMLMC Optimal hierarchies Conclusions Multi-Index Monte Carlo General Framework Choosing the Multi-Index Set in MIMC Main Theorem Numerical Results Conclusions Multi-index Stochastic Collocation Complexity analysis Infinite dimensional case (parametric regularity) 8 / 101

  3. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis The Multi Level Monte Carlo idea Consider again a random vector y ( ω ) = ( y 1 ( ω ) , . . . , y N ( ω )) with density ρ ( y ) : Γ → R + , a Hilbert-space valued function u ( y ) : Γ → V , u ∈ L 2 ρ (Γ; V ), solution of a differential problem a Lipshitz functional Q : V → R , with Q (0) = 0. Goal: Compute E [ Q ( u ( y ( ω )))] In practice the solution u ( y ( ω )) can not be computed exactly and we will approximate the differential problem by some discretization scheme. Let h be the discretization parameter and u h ( y ( ω )) the approximate solution such that u h ( y ( ω )) → u ( y ( ω )) (in the proper norm) as h → 0. 9 / 101

  4. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis The Multi Level Monte Carlo idea Consider again a random vector y ( ω ) = ( y 1 ( ω ) , . . . , y N ( ω )) with density ρ ( y ) : Γ → R + , a Hilbert-space valued function u ( y ) : Γ → V , u ∈ L 2 ρ (Γ; V ), solution of a differential problem a Lipshitz functional Q : V → R , with Q (0) = 0. Goal: Compute E [ Q ( u ( y ( ω )))] In practice the solution u ( y ( ω )) can not be computed exactly and we will approximate the differential problem by some discretization scheme. Let h be the discretization parameter and u h ( y ( ω )) the approximate solution such that u h ( y ( ω )) → u ( y ( ω )) (in the proper norm) as h → 0. 9 / 101

  5. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis The Multi Level Monte Carlo idea Introduce a sequence of finer and finer discretizations h 0 > h 1 > . . . , h L and assume that mesh size h L achieves the desired accuracy ..... h 0 h 1 h L Notation : g ( ω ) = Q ( u ( y ( ω ))) (output quantity) g ℓ ( ω ) = Q ( u h ℓ ( y ( ω ))), ℓ = 0 , . . . , L . Goal : compute E [ g L ] Idea : write the above expectation as a telescopic sum L � E [ g L ] = E [ g 0 ] + E [ g ℓ − g ℓ − 1 ] ℓ =1 10 / 101

  6. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis The Multi Level Monte Carlo idea Introduce a sequence of finer and finer discretizations h 0 > h 1 > . . . , h L and assume that mesh size h L achieves the desired accuracy ..... h 0 h 1 h L Notation : g ( ω ) = Q ( u ( y ( ω ))) (output quantity) g ℓ ( ω ) = Q ( u h ℓ ( y ( ω ))), ℓ = 0 , . . . , L . Goal : compute E [ g L ] Idea : write the above expectation as a telescopic sum L � E [ g L ] = E [ g 0 ] + E [ g ℓ − g ℓ − 1 ] ℓ =1 10 / 101

  7. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Multi Level Monte Carlo estimator MLMC estimator M 0 M ℓ � � L � A := 1 1 g 0 ( ω 0 ( g ℓ − g ℓ − 1 )( ω ℓ j ) + j ) M 0 M ℓ j =1 ℓ =1 j =1 with { ω ℓ j , j = 1 , . . . , M ℓ } independent samples of iid replica on each level. Notice that E [ A ] = E [ g L ]. Mean Square error (MSE) E [( A − E [ g ]) 2 ] = E [( A − E [ g L ]) 2 ] + ( E [ g L − g ]) 2 � �� � � �� � Variance (statistical error) Bias (discret. error) Variance of the estimator . Thanks to the independent samples among levels: L � Var [ A ] = E [( A − E [ g L ]) 2 ] = Var [ g 0 ] Var [ g ℓ − g ℓ − 1 ] + M 0 M ℓ ℓ =1 Key point : Since Var [ g ℓ − g ℓ − 1 ] gets smaller and smaller for ℓ large, one can take M ℓ smaller and smaller � Only few samples on the fine grid h L . 11 / 101

  8. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Multi Level Monte Carlo estimator MLMC estimator M 0 M ℓ � � L � A := 1 1 g 0 ( ω 0 ( g ℓ − g ℓ − 1 )( ω ℓ j ) + j ) M 0 M ℓ j =1 ℓ =1 j =1 with { ω ℓ j , j = 1 , . . . , M ℓ } independent samples of iid replica on each level. Notice that E [ A ] = E [ g L ]. Mean Square error (MSE) E [( A − E [ g ]) 2 ] = E [( A − E [ g L ]) 2 ] + ( E [ g L − g ]) 2 � �� � � �� � Variance (statistical error) Bias (discret. error) Variance of the estimator . Thanks to the independent samples among levels: L � Var [ A ] = E [( A − E [ g L ]) 2 ] = Var [ g 0 ] Var [ g ℓ − g ℓ − 1 ] + M 0 M ℓ ℓ =1 Key point : Since Var [ g ℓ − g ℓ − 1 ] gets smaller and smaller for ℓ large, one can take M ℓ smaller and smaller � Only few samples on the fine grid h L . 11 / 101

  9. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Multi Level Monte Carlo estimator MLMC estimator M 0 M ℓ � � L � A := 1 1 g 0 ( ω 0 ( g ℓ − g ℓ − 1 )( ω ℓ j ) + j ) M 0 M ℓ j =1 ℓ =1 j =1 with { ω ℓ j , j = 1 , . . . , M ℓ } independent samples of iid replica on each level. Notice that E [ A ] = E [ g L ]. Mean Square error (MSE) E [( A − E [ g ]) 2 ] = E [( A − E [ g L ]) 2 ] + ( E [ g L − g ]) 2 � �� � � �� � Variance (statistical error) Bias (discret. error) Variance of the estimator . Thanks to the independent samples among levels: L � Var [ A ] = E [( A − E [ g L ]) 2 ] = Var [ g 0 ] Var [ g ℓ − g ℓ − 1 ] + M 0 M ℓ ℓ =1 Key point : Since Var [ g ℓ − g ℓ − 1 ] gets smaller and smaller for ℓ large, one can take M ℓ smaller and smaller � Only few samples on the fine grid h L . 11 / 101

  10. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Multi Level Monte Carlo estimator MLMC estimator M 0 M ℓ � � L � A := 1 1 g 0 ( ω 0 ( g ℓ − g ℓ − 1 )( ω ℓ j ) + j ) M 0 M ℓ j =1 ℓ =1 j =1 with { ω ℓ j , j = 1 , . . . , M ℓ } independent samples of iid replica on each level. Notice that E [ A ] = E [ g L ]. Mean Square error (MSE) E [( A − E [ g ]) 2 ] = E [( A − E [ g L ]) 2 ] + ( E [ g L − g ]) 2 � �� � � �� � Variance (statistical error) Bias (discret. error) Variance of the estimator . Thanks to the independent samples among levels: L � Var [ A ] = E [( A − E [ g L ]) 2 ] = Var [ g 0 ] Var [ g ℓ − g ℓ − 1 ] + M 0 M ℓ ℓ =1 Key point : Since Var [ g ℓ − g ℓ − 1 ] gets smaller and smaller for ℓ large, one can take M ℓ smaller and smaller � Only few samples on the fine grid h L . 11 / 101

  11. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Variance reduction: MLMC Recall: With Monte Carlo we have to satisfy 1 1 Var [ g ] ≤ TOL 2 . Var [ A MC ] = Var [ g L ] ≈ M L M L Main point: MLMC reduces the variance of the deepest level using samples on coarser ( less expensive) levels! Var [ A MLMC ] = 1 Var [ g 0 ] M 0 L � 1 Var [∆ g ℓ ] ≤ TOL 2 . + M ℓ ℓ =1 Observe: Level 0 in MLMC is usually determined by both stability and accuracy, i.e. Var [∆ g 1 ] << Var [ g 0 ] ≈ Var [ g ] < ∞ . 12 / 101

  12. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Optimal choice of M ℓ Let C 0 : cost of generating one realization of g 0 C ℓ : cost of generating one realization of g ℓ − g ℓ − 1 , ℓ > 0 V 0 = Var [ g 0 ] V ℓ = Var [ g ℓ − g ℓ − 1 ], ℓ > 0 Then L L � � V ℓ Total work: W MLMC = M ℓ C ℓ , Total variance: Var [ A ] = . M ℓ ℓ =0 ℓ =0 Find optimal { M ℓ } to minimize the cost at a fixed variance level L L � � M − 1 ℓ V ℓ ≤ TOL 2 min M ℓ C ℓ subject to { M ℓ } ℓ =0 ℓ =0 If we replace (relaxation) M ℓ by real variables, the optimal solution is � L � � V ℓ M ℓ = TOL − 2 V j C j C ℓ j = 0 13 / 101

  13. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Optimal choice of M ℓ Let C 0 : cost of generating one realization of g 0 C ℓ : cost of generating one realization of g ℓ − g ℓ − 1 , ℓ > 0 V 0 = Var [ g 0 ] V ℓ = Var [ g ℓ − g ℓ − 1 ], ℓ > 0 Then L L � � V ℓ Total work: W MLMC = M ℓ C ℓ , Total variance: Var [ A ] = . M ℓ ℓ =0 ℓ =0 Find optimal { M ℓ } to minimize the cost at a fixed variance level L L � � M − 1 ℓ V ℓ ≤ TOL 2 min M ℓ C ℓ subject to { M ℓ } ℓ =0 ℓ =0 If we replace (relaxation) M ℓ by real variables, the optimal solution is � L � � V ℓ M ℓ = TOL − 2 V j C j C ℓ j = 0 13 / 101

  14. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Optimal choice of M ℓ Let C 0 : cost of generating one realization of g 0 C ℓ : cost of generating one realization of g ℓ − g ℓ − 1 , ℓ > 0 V 0 = Var [ g 0 ] V ℓ = Var [ g ℓ − g ℓ − 1 ], ℓ > 0 Then L L � � V ℓ Total work: W MLMC = M ℓ C ℓ , Total variance: Var [ A ] = . M ℓ ℓ =0 ℓ =0 Find optimal { M ℓ } to minimize the cost at a fixed variance level L L � � M − 1 ℓ V ℓ ≤ TOL 2 min M ℓ C ℓ subject to { M ℓ } ℓ =0 ℓ =0 If we replace (relaxation) M ℓ by real variables, the optimal solution is � L � � V ℓ M ℓ = TOL − 2 V j C j C ℓ j = 0 13 / 101

  15. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: write the Lagrangian function L � L � V j TOL 2 − � � L ( M 0 , . . . , M L , λ ) = M ℓ C ℓ − λ M j ℓ =0 j = 0 Then ∂ L ∂ M ℓ = C ℓ − λ V ℓ = 0 , M 2 ℓ � λ V ℓ hence M ℓ = C ℓ . Replacing in the constraint gives L L � √ V j C j � = TOL 2 , λ = TOL − 2 � � = ⇒ V j C j λ j =0 j = 0 In practice, one should take the ceiling of the real value M ℓ (important if M ℓ < 1).   � � L � V ℓ   TOL − 2 Optimal sample sizes: M ℓ = V j C j   C ℓ   j = 0   2 L L � � � Optimal work: W MLMC ≤ TOL − 2   + V j C j C ℓ j = 0 ℓ = 0 14 / 101

  16. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: write the Lagrangian function L � L � V j TOL 2 − � � L ( M 0 , . . . , M L , λ ) = M ℓ C ℓ − λ M j ℓ =0 j = 0 Then ∂ L ∂ M ℓ = C ℓ − λ V ℓ = 0 , M 2 ℓ � λ V ℓ hence M ℓ = C ℓ . Replacing in the constraint gives L L � √ V j C j � = TOL 2 , λ = TOL − 2 � � = ⇒ V j C j λ j =0 j = 0 In practice, one should take the ceiling of the real value M ℓ (important if M ℓ < 1).   � � L � V ℓ   TOL − 2 Optimal sample sizes: M ℓ = V j C j   C ℓ   j = 0   2 L L � � � Optimal work: W MLMC ≤ TOL − 2   + V j C j C ℓ j = 0 ℓ = 0 14 / 101

  17. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: write the Lagrangian function L � L � V j TOL 2 − � � L ( M 0 , . . . , M L , λ ) = M ℓ C ℓ − λ M j ℓ =0 j = 0 Then ∂ L ∂ M ℓ = C ℓ − λ V ℓ = 0 , M 2 ℓ � λ V ℓ hence M ℓ = C ℓ . Replacing in the constraint gives L L � √ V j C j � = TOL 2 , λ = TOL − 2 � � = ⇒ V j C j λ j =0 j = 0 In practice, one should take the ceiling of the real value M ℓ (important if M ℓ < 1).   � � L � V ℓ   TOL − 2 Optimal sample sizes: M ℓ = V j C j   C ℓ   j = 0   2 L L � � � Optimal work: W MLMC ≤ TOL − 2   + V j C j C ℓ j = 0 ℓ = 0 14 / 101

  18. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: write the Lagrangian function L � L � V j TOL 2 − � � L ( M 0 , . . . , M L , λ ) = M ℓ C ℓ − λ M j ℓ =0 j = 0 Then ∂ L ∂ M ℓ = C ℓ − λ V ℓ = 0 , M 2 ℓ � λ V ℓ hence M ℓ = C ℓ . Replacing in the constraint gives L L � √ V j C j � = TOL 2 , λ = TOL − 2 � � = ⇒ V j C j λ j =0 j = 0 In practice, one should take the ceiling of the real value M ℓ (important if M ℓ < 1).   � � L � V ℓ   TOL − 2 Optimal sample sizes: M ℓ = V j C j   C ℓ   j = 0   2 L L � � � Optimal work: W MLMC ≤ TOL − 2   + V j C j C ℓ j = 0 ℓ = 0 14 / 101

  19. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Complexity analysis (error versus cost) [Giles 2008], [Cliffe, Giles, Scheichl and Teckentrup 2011] For a problem in R d ( d -dimensional), assume 1. h ℓ = h 0 δ ℓ , 0 < δ < 1 (geometric meshes) 2. | E [ g − g ℓ ] | ≤ C w h α ℓ (weak convergence rate) 3. E [( g − g ℓ ) 2 ] ≤ C s h β ℓ (strong convergence rate) 4. C ℓ = C c h − d γ ( γ = 3 for direct solver and full matrix; γ ≈ 1 for ℓ optimal solver with linear complexity) Notice that from assumption 3. we have ℓ − 1 , with C v = 2 C s ( δ β + 1) 5. V ℓ ≤ C v h β Indeed V ℓ = Var [ g ℓ − g ℓ − 1 ] ≤ E [( g ℓ − g ℓ − 1 ) 2 ] ≤ 2 E [( g − g ℓ ) 2 ]+2 E [( g − g ℓ − 1 ) 2 ] ≤ 2 C s ( δ β +1) h β ℓ − 1 Moreover one always has β ≤ 2 α (typically β = 2 α for PDEs with smooth random coefficients) Indeed by Cauchy-Schwarz β α ≥ β 1 | E [ g − g ℓ ] | ≤ E [( g − g ℓ ) 2 ] 2 ≤ C s h 2 ℓ , hence 2 15 / 101

  20. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Complexity analysis (error versus cost) [Giles 2008], [Cliffe, Giles, Scheichl and Teckentrup 2011] For a problem in R d ( d -dimensional), assume 1. h ℓ = h 0 δ ℓ , 0 < δ < 1 (geometric meshes) 2. | E [ g − g ℓ ] | ≤ C w h α ℓ (weak convergence rate) 3. E [( g − g ℓ ) 2 ] ≤ C s h β ℓ (strong convergence rate) 4. C ℓ = C c h − d γ ( γ = 3 for direct solver and full matrix; γ ≈ 1 for ℓ optimal solver with linear complexity) Notice that from assumption 3. we have ℓ − 1 , with C v = 2 C s ( δ β + 1) 5. V ℓ ≤ C v h β Indeed V ℓ = Var [ g ℓ − g ℓ − 1 ] ≤ E [( g ℓ − g ℓ − 1 ) 2 ] ≤ 2 E [( g − g ℓ ) 2 ]+2 E [( g − g ℓ − 1 ) 2 ] ≤ 2 C s ( δ β +1) h β ℓ − 1 Moreover one always has β ≤ 2 α (typically β = 2 α for PDEs with smooth random coefficients) Indeed by Cauchy-Schwarz β α ≥ β 1 | E [ g − g ℓ ] | ≤ E [( g − g ℓ ) 2 ] 2 ≤ C s h 2 ℓ , hence 2 15 / 101

  21. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Complexity analysis (error versus cost) [Giles 2008], [Cliffe, Giles, Scheichl and Teckentrup 2011] For a problem in R d ( d -dimensional), assume 1. h ℓ = h 0 δ ℓ , 0 < δ < 1 (geometric meshes) 2. | E [ g − g ℓ ] | ≤ C w h α ℓ (weak convergence rate) 3. E [( g − g ℓ ) 2 ] ≤ C s h β ℓ (strong convergence rate) 4. C ℓ = C c h − d γ ( γ = 3 for direct solver and full matrix; γ ≈ 1 for ℓ optimal solver with linear complexity) Notice that from assumption 3. we have ℓ − 1 , with C v = 2 C s ( δ β + 1) 5. V ℓ ≤ C v h β Indeed V ℓ = Var [ g ℓ − g ℓ − 1 ] ≤ E [( g ℓ − g ℓ − 1 ) 2 ] ≤ 2 E [( g − g ℓ ) 2 ]+2 E [( g − g ℓ − 1 ) 2 ] ≤ 2 C s ( δ β +1) h β ℓ − 1 Moreover one always has β ≤ 2 α (typically β = 2 α for PDEs with smooth random coefficients) Indeed by Cauchy-Schwarz β α ≥ β 1 | E [ g − g ℓ ] | ≤ E [( g − g ℓ ) 2 ] 2 ≤ C s h 2 ℓ , hence 2 15 / 101

  22. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Complexity analysis (error versus cost) Theorem [Cliffe-Giles-Scheichl-Teckentrup 2011] Under the assumptions 1-4 above, if 2 α ≥ min( β, d γ ), the computational work required to approximate E [ g ] with MLMC with accuracy 0 < TOL < 1 / e in mean square sense, that is E [( A − E [ g ]) 2 ] ≤ TOL 2 is bounded as follows:  TOL − 2 , for β > d γ,   TOL − 2 log 2 ( TOL ) , for β = d γ, W MLMC ≤ C   TOL − 2 − ( d γ − β ) /α , for β < d γ, Remark : standard MC has corresponding complexity W MC ∝ TOL − 2 − d γ/α . 16 / 101

  23. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Complexity analysis (error versus cost) Theorem [Cliffe-Giles-Scheichl-Teckentrup 2011] Under the assumptions 1-4 above, if 2 α ≥ min( β, d γ ), the computational work required to approximate E [ g ] with MLMC with accuracy 0 < TOL < 1 / e in mean square sense, that is E [( A − E [ g ]) 2 ] ≤ TOL 2 is bounded as follows:  TOL − 2 , for β > d γ,   TOL − 2 log 2 ( TOL ) , for β = d γ, W MLMC ≤ C   TOL − 2 − ( d γ − β ) /α , for β < d γ, Remark : standard MC has corresponding complexity W MC ∝ TOL − 2 − d γ/α . 16 / 101

  24. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: Enforce a MSE ≤ TOL 2 as Bias constraint: | E [ g − g L ] | 2 ≤ 1 Variance constraint: Var [ A ] ≤ 1 2 TOL 2 , 2 TOL 2 From the Bias constraint we get √ � � 2C w h α 0 TOL − 1 ) log( 1 L ( TOL ) = ∼ log δ TOL α α log δ − 1 and setting ˜ C v = C v h β 0 and ˜ C c = C c h − d γ , the total work is (disregarding the term 0 � L j =0 C j ) 2   L � C c δ j β − d γ W MLMC ≤ TOL − 2 � C v ˜ ˜ 2   j = 1 � 2 C c δ j β − d γ Case β > d γ : W MLMC ≤ TOL − 2 �� ∞ � ˜ C v ˜ ≤ C TOL − 2 2 j = 1 Case β = d γ : W MLMC ≤ C TOL − 2 L ≤ C TOL − 2 (log TOL − 1 ) 2 Case β < d γ : W MLMC ≤ C TOL − 2 δ L β − d γ ≤ C TOL − 2 − d γ − β 2 α If, moreover, α ≥ min( β, d γ ) then the one sample work/level, � L j =0 C j , is negligible with respect to the TOL dependent one. 17 / 101

  25. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: Enforce a MSE ≤ TOL 2 as Bias constraint: | E [ g − g L ] | 2 ≤ 1 Variance constraint: Var [ A ] ≤ 1 2 TOL 2 , 2 TOL 2 From the Bias constraint we get √ � � 2C w h α 0 TOL − 1 ) log( 1 L ( TOL ) = ∼ log δ TOL α α log δ − 1 and setting ˜ C v = C v h β 0 and ˜ C c = C c h − d γ , the total work is (disregarding the term 0 � L j =0 C j ) 2   L � C c δ j β − d γ W MLMC ≤ TOL − 2 � C v ˜ ˜ 2   j = 1 � 2 C c δ j β − d γ Case β > d γ : W MLMC ≤ TOL − 2 �� ∞ � ˜ C v ˜ ≤ C TOL − 2 2 j = 1 Case β = d γ : W MLMC ≤ C TOL − 2 L ≤ C TOL − 2 (log TOL − 1 ) 2 Case β < d γ : W MLMC ≤ C TOL − 2 δ L β − d γ ≤ C TOL − 2 − d γ − β 2 α If, moreover, α ≥ min( β, d γ ) then the one sample work/level, � L j =0 C j , is negligible with respect to the TOL dependent one. 17 / 101

  26. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis Proof: Enforce a MSE ≤ TOL 2 as Bias constraint: | E [ g − g L ] | 2 ≤ 1 Variance constraint: Var [ A ] ≤ 1 2 TOL 2 , 2 TOL 2 From the Bias constraint we get √ � � 2C w h α 0 TOL − 1 ) log( 1 L ( TOL ) = ∼ log δ TOL α α log δ − 1 and setting ˜ C v = C v h β 0 and ˜ C c = C c h − d γ , the total work is (disregarding the term 0 � L j =0 C j ) 2   L � C c δ j β − d γ W MLMC ≤ TOL − 2 � C v ˜ ˜ 2   j = 1 � 2 C c δ j β − d γ Case β > d γ : W MLMC ≤ TOL − 2 �� ∞ � ˜ C v ˜ ≤ C TOL − 2 2 j = 1 Case β = d γ : W MLMC ≤ C TOL − 2 L ≤ C TOL − 2 (log TOL − 1 ) 2 Case β < d γ : W MLMC ≤ C TOL − 2 δ L β − d γ ≤ C TOL − 2 − d γ − β 2 α If, moreover, α ≥ min( β, d γ ) then the one sample work/level, � L j =0 C j , is negligible with respect to the TOL dependent one. 17 / 101

  27. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Main idea and complexity analysis (Shocking) comments on the MLMC rates Let us focus on two particular cases: Fastest convergence rate, β > d γ. Here the convergence rate is TOL − 2 , which is the same of Monte Carlo sampling when the cost to sample each realization is fixed . This means that we do not feel the effect of the discretization in the rates! Smooth noise, β = 2 α and β < d γ. Here the resulting convergence rate is TOL − d γ/α , which is the complexity of solving just one realization in the deepest level! 18 / 101

  28. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Further optimization of MLMC MLMC: Beyond geometric hierarchies The previous result is based on geometric discretizations, i.e. h ℓ = h 0 δ ℓ . Questions: Can we construct non-geometric discretization MLMC hierarchies? Do they reduce the computational work of MLMC even further? [HvSST] “Implementation and analysis of an adaptive multilevel Monte Carlo algorithm” by H. Hoel, E. von Schwerin, A. Szepessy and R. Tempone, Monte Carlo Methods and Applications. Volume 20, Issue 1, pp. 1–41, November 2013. [ANvST00] “Optimization of mesh hierarchies for Multilevel Monte Carlo” , by A.-L Haji-Ali, F. Nobile, E. von Schwerin and R. Tempone. Stochastic Partial Differential Equations: Analysis and Computations, Vol. 4, Issue 1, Pages 76–112, 2016 [NANvST01] “A Continuation Multilevel Monte Carlo” , by N. Collier, A.-L Haji-Ali, F. Nobile, E. von Schwerin and R. Tempone. BIT Numer. Math., 55, pp. 399–432, 2015. 19 / 101

  29. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Further optimization of MLMC Our goal is to compute E [ g ] where g = g ( u ). Here g is either a bounded linear functional or a Lips- chitz functional with respect to u , and u solves a stochastic equation. Example: for x ∈ D = [0 , 1] d , −∇ · ( a ( x ; ω ) ∇ u ( x ; ω )) = f ( x ; ω ) u ( x ; ω ) = 0 for x ∈ ∂ D , and � g ( u ) = κ ( x ) u ( x ) d x , D for sufficiently smooth a , f , κ . 20 / 101

  30. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Further optimization of MLMC Following the standard MLMC approach, we introduce a hierarchy of L + 1 meshes defined by decreasing mesh sizes { h ℓ } L ℓ =0 and we denote the approximation of g using mesh size h ℓ by g ℓ . We then write the MLMC estimator as M 0 L M ℓ � � � A = 1 1 g 0 ( ω 0 , m ) + ( g ℓ ( ω ℓ, m ) − g ℓ − 1 ( ω ℓ, m )) . (2) M 0 M ℓ m =1 ℓ =1 m =1 We assume the following models: � � � � � ≈ Q W h q 1 � E [ g ℓ − g ] ℓ , (3a) Var [ g ℓ − g ℓ − 1 ] := V ℓ ≈ Q S h q 2 ℓ − 1 , (3b) Work per sample of level ℓ := W ℓ ≈ h − d γ . (3c) ℓ for some positive constants Q W , Q S , q 1 , q 2 , d and γ . 21 / 101

  31. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Further optimization of MLMC Examples for q 1 , q 2 : q 1 = q 2 = 1 for an SDE with Euler-Maruyama approximation. In our example: a PDE with smooth random coefficients and for piecewise linear or piecewise bilinear continuous finite element approximations we have q 1 = 2 and q 2 = 4. Examples for γ : γ = 1 for an SDE with Euler-Maruyama approximation. In our PDE example: d = 3 and γ = 3 for a naive Gaussian elimination implementation. Moreover, Using an Iterative solver has γ ≈ 1 while using Direct solver has γ ≈ 1 . 5. χ = q 2 η = q 1 We define: and d γ . d γ In our PDE example: χ ≈ 1 . 34 for iterative solver and χ ≈ 0 . 89 for direct solver. 22 / 101

  32. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Further optimization of MLMC Examples for q 1 , q 2 : q 1 = q 2 = 1 for an SDE with Euler-Maruyama approximation. In our example: a PDE with smooth random coefficients and for piecewise linear or piecewise bilinear continuous finite element approximations we have q 1 = 2 and q 2 = 4. Examples for γ : γ = 1 for an SDE with Euler-Maruyama approximation. In our PDE example: d = 3 and γ = 3 for a naive Gaussian elimination implementation. Moreover, Using an Iterative solver has γ ≈ 1 while using Direct solver has γ ≈ 1 . 5. χ = q 2 η = q 1 We define: and d γ . d γ In our PDE example: χ ≈ 1 . 34 for iterative solver and χ ≈ 0 . 89 for direct solver. 22 / 101

  33. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Further optimization of MLMC Problem (Optimization of computational work) Given L ∈ N and θ ∈ (0 , 1) , find H = ( { h ℓ } L ℓ =0 , { M ℓ } L ℓ =0 ) ∈ R L +1 × R L +1 + + such that � L M ℓ W ( H ) = , (4a) h d γ ℓ ℓ =0 is minimized while satisfying the constraints Q W h q 1 L ≤ (1 − θ ) TOL , (4b) � θ TOL � 2 L h q 2 � V 0 ℓ − 1 + Q S ≤ , (4c) M 0 M ℓ C α ℓ =1 for a confidence parameter, C α , such that Φ( C α ) = 1 − α 2 ; here, 0 < α ≪ 1 and Φ is the cumulative distribution function of a standard normal random variable. 23 / 101

  34. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Normality of MLMC estimator Generalized CLT Theorem (Lindeberg-Feller CLT) For each n, let X n , m , for 1 ≤ n ≤ m, be independent random variables (not necessarily identical). Denote n � a n = X n , m , m =1 Y n , m = X n , m − E [ X n , m ] , n � � � s 2 Y 2 n = . E n , m m =1 Suppose the following Lindeberg condition is satisfied for all ǫ > 0 : � n � � n →∞ s − 2 Y 2 lim E n , m 1 | Y n , m | >ǫ s n = 0 . (5) n m =1 Then, � a n − E [ a n ] � n →∞ P lim ≤ z = Φ( z ) . s n 24 / 101

  35. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Normality of MLMC estimator On normality of MLMC estimator As presented in [NANvST01] , [HvSST] has similar results for SDE. Lemma ([Collier, Haji-Ali, N., von Schwerin, Tempone, 2014]) Consider the MLMC estimator A given by � L � M ℓ G ℓ ( ω ℓ, m ) A = , M ℓ ℓ =0 m =1 where G ℓ ( ω ℓ, m ) denote as usual i.i.d. samples of the random variable G ℓ = g ℓ − g ℓ − 1 . The family of random variables, ( G ℓ ) ℓ ≥ 0 , is also assumed independent. Denote Y ℓ = | G ℓ − E [ G ℓ ] | and assume the following C 1 β − q 3 ℓ ≤ E [ Y 2 ℓ ] for all ℓ ≥ 0 , (6a) E [ Y 2+ δ ] ≤ C 2 β − τℓ for all ℓ ≥ 0 , (6b) ℓ for some β > 1 and strictly positive constants C 1 , C 2 , q 3 , δ and τ . 25 / 101

  36. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Normality of MLMC estimator Lemma (cont.) Choose the number of samples on each level M ℓ to satisfy, for q 2 > 0 and a strictly positive sequence { H ℓ } ℓ ≥ 0 � L � � M ℓ ≥ β − q 2 ℓ TOL − 2 H − 1 H ℓ for all ℓ ≥ 0 . (7) ℓ ℓ = 0 Moreover, choose the number of levels L to satisfy � � � TOL − 1 � 0 , c log L ≤ max + C (8) log β for some constants C, and c > 0 . Finally, denoting p = (1 + δ/ 2) q 3 + ( δ/ 2) q 2 − τ, if we have that either p > 0 or c < δ/ p, then � � A − E [ A ] TOL → 0 P lim � ≤ z = Φ ( z ) . Var [ A ] 26 / 101

  37. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Normality of MLMC estimator QQ Plot: Experiment verification of normality of MLMC estimator TOL = 0 . 005 TOL = 0 . 00707107 Direct Iterative 1 . 0 1 . 0 0 . 8 0 . 8 Normal CDF Normal CDF 0 . 6 0 . 6 0 . 4 0 . 4 0 . 2 0 . 2 0 . 0 0 . 0 0 . 0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 0 . 0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 Empirical CDF Empirical CDF This plot shows that the density of the normalized statistical error is well approximated by a standard normal density. 27 / 101

  38. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC Error splitting Using the asymptotic normality of the MLMC estimator, we can require separately Bias [ A ] ≈ C w h α L ≤ (1 − θ ) TOL , � θ TOL � 2 L � Var [ A ] ≈ V 0 V ℓ + ≤ , M 0 M ℓ c δ ℓ =1 to guarantee a total error smaller than TOL with probability ≥ 1 − δ . θ ∈ (0 , 1) is a free parameter chosen to minimize the computational work. The Bias constraint determines the maximum level L to use Minimization of work under variance constraint leads to optimal sample sizes { M ℓ } L ℓ =0 To find optimal L and { M ℓ } ℓ one needs good estimates of the rates ( α, β, γ ), constants ( C w , C v , C c ) and variances { V ℓ } L l =0 . 28 / 101

  39. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC Error splitting Using the asymptotic normality of the MLMC estimator, we can require separately Bias [ A ] ≈ C w h α L ≤ (1 − θ ) TOL , � θ TOL � 2 L � Var [ A ] ≈ V 0 V ℓ + ≤ , M 0 M ℓ c δ ℓ =1 to guarantee a total error smaller than TOL with probability ≥ 1 − δ . θ ∈ (0 , 1) is a free parameter chosen to minimize the computational work. The Bias constraint determines the maximum level L to use Minimization of work under variance constraint leads to optimal sample sizes { M ℓ } L ℓ =0 To find optimal L and { M ℓ } ℓ one needs good estimates of the rates ( α, β, γ ), constants ( C w , C v , C c ) and variances { V ℓ } L l =0 . 28 / 101

  40. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC Error splitting Using the asymptotic normality of the MLMC estimator, we can require separately Bias [ A ] ≈ C w h α L ≤ (1 − θ ) TOL , � θ TOL � 2 L � Var [ A ] ≈ V 0 V ℓ + ≤ , M 0 M ℓ c δ ℓ =1 to guarantee a total error smaller than TOL with probability ≥ 1 − δ . θ ∈ (0 , 1) is a free parameter chosen to minimize the computational work. The Bias constraint determines the maximum level L to use Minimization of work under variance constraint leads to optimal sample sizes { M ℓ } L ℓ =0 To find optimal L and { M ℓ } ℓ one needs good estimates of the rates ( α, β, γ ), constants ( C w , C v , C c ) and variances { V ℓ } L l =0 . 28 / 101

  41. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC A “continuation” adaptive algorithm (CMLMC) [Haji-Ali, Nobile, Tempone 2014] Choose a sequence of decreasing tolerances: TOL 0 > TOL 1 > . . . > TOL K = TOL and an initial guess of the rates ( α (0) , β (0) , γ (0) ), constants ( C (0) w , C (0) v , C (0) c ) and variances { V ℓ } L (0) ℓ =0 , FOR k = 1 , . . . , K Based on rates ( α ( k − 1) , β ( k − 1) , γ ( k − 1) ), constants ( C ( k − 1) , C ( k − 1) , C ( k − 1) ) and variances { V ℓ } L ( k − 1) w v c ℓ =0 compute optimal ( L ( k ) , θ ( k ) ) = s.t. C w h α arg min Work ( L , θ ) , L ≤ (1 − θ ) TOL k θ ∈ (0 , 1) L ( k − 1) ≤ L ≤ Lmax ℓ } L ( k ) compute optimal { M ( k ) ℓ =0 to satisfy TOL k . ℓ } L ( k ) run MLMC with L ( k ) , { M ( k ) ℓ =0 update rates ( α ( k ) , β ( k ) , γ ( k ) ), constants ( C ( k ) w , C ( k ) , C ( k ) ) and v c variances { V ℓ } L ( k ) ℓ =0 based on the new simulations performed k = k + 1 end FOR 29 / 101

  42. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC A “continuation” adaptive algorithm (CMLMC) [Haji-Ali, Nobile, Tempone 2014] Choose a sequence of decreasing tolerances: TOL 0 > TOL 1 > . . . > TOL K = TOL and an initial guess of the rates ( α (0) , β (0) , γ (0) ), constants ( C (0) w , C (0) v , C (0) c ) and variances { V ℓ } L (0) ℓ =0 , FOR k = 1 , . . . , K Based on rates ( α ( k − 1) , β ( k − 1) , γ ( k − 1) ), constants ( C ( k − 1) , C ( k − 1) , C ( k − 1) ) and variances { V ℓ } L ( k − 1) w v c ℓ =0 compute optimal ( L ( k ) , θ ( k ) ) = s.t. C w h α arg min Work ( L , θ ) , L ≤ (1 − θ ) TOL k θ ∈ (0 , 1) L ( k − 1) ≤ L ≤ Lmax ℓ } L ( k ) compute optimal { M ( k ) ℓ =0 to satisfy TOL k . ℓ } L ( k ) run MLMC with L ( k ) , { M ( k ) ℓ =0 update rates ( α ( k ) , β ( k ) , γ ( k ) ), constants ( C ( k ) w , C ( k ) , C ( k ) ) and v c variances { V ℓ } L ( k ) ℓ =0 based on the new simulations performed k = k + 1 end FOR 29 / 101

  43. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC A “continuation” adaptive algorithm (CMLMC) [Haji-Ali, Nobile, Tempone 2014] Choose a sequence of decreasing tolerances: TOL 0 > TOL 1 > . . . > TOL K = TOL and an initial guess of the rates ( α (0) , β (0) , γ (0) ), constants ( C (0) w , C (0) v , C (0) c ) and variances { V ℓ } L (0) ℓ =0 , FOR k = 1 , . . . , K Based on rates ( α ( k − 1) , β ( k − 1) , γ ( k − 1) ), constants ( C ( k − 1) , C ( k − 1) , C ( k − 1) ) and variances { V ℓ } L ( k − 1) w v c ℓ =0 compute optimal ( L ( k ) , θ ( k ) ) = s.t. C w h α arg min Work ( L , θ ) , L ≤ (1 − θ ) TOL k θ ∈ (0 , 1) L ( k − 1) ≤ L ≤ Lmax ℓ } L ( k ) compute optimal { M ( k ) ℓ =0 to satisfy TOL k . ℓ } L ( k ) run MLMC with L ( k ) , { M ( k ) ℓ =0 update rates ( α ( k ) , β ( k ) , γ ( k ) ), constants ( C ( k ) w , C ( k ) , C ( k ) ) and v c variances { V ℓ } L ( k ) ℓ =0 based on the new simulations performed k = k + 1 end FOR 29 / 101

  44. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC A “continuation” adaptive algorithm (CMLMC) [Haji-Ali, Nobile, Tempone 2014] Choose a sequence of decreasing tolerances: TOL 0 > TOL 1 > . . . > TOL K = TOL and an initial guess of the rates ( α (0) , β (0) , γ (0) ), constants ( C (0) w , C (0) v , C (0) c ) and variances { V ℓ } L (0) ℓ =0 , FOR k = 1 , . . . , K Based on rates ( α ( k − 1) , β ( k − 1) , γ ( k − 1) ), constants ( C ( k − 1) , C ( k − 1) , C ( k − 1) ) and variances { V ℓ } L ( k − 1) w v c ℓ =0 compute optimal ( L ( k ) , θ ( k ) ) = s.t. C w h α arg min Work ( L , θ ) , L ≤ (1 − θ ) TOL k θ ∈ (0 , 1) L ( k − 1) ≤ L ≤ Lmax ℓ } L ( k ) compute optimal { M ( k ) ℓ =0 to satisfy TOL k . ℓ } L ( k ) run MLMC with L ( k ) , { M ( k ) ℓ =0 update rates ( α ( k ) , β ( k ) , γ ( k ) ), constants ( C ( k ) w , C ( k ) , C ( k ) ) and v c variances { V ℓ } L ( k ) ℓ =0 based on the new simulations performed k = k + 1 end FOR 29 / 101

  45. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC A “continuation” adaptive algorithm (CMLMC) [Haji-Ali, Nobile, Tempone 2014] Choose a sequence of decreasing tolerances: TOL 0 > TOL 1 > . . . > TOL K = TOL and an initial guess of the rates ( α (0) , β (0) , γ (0) ), constants ( C (0) w , C (0) v , C (0) c ) and variances { V ℓ } L (0) ℓ =0 , FOR k = 1 , . . . , K Based on rates ( α ( k − 1) , β ( k − 1) , γ ( k − 1) ), constants ( C ( k − 1) , C ( k − 1) , C ( k − 1) ) and variances { V ℓ } L ( k − 1) w v c ℓ =0 compute optimal ( L ( k ) , θ ( k ) ) = s.t. C w h α arg min Work ( L , θ ) , L ≤ (1 − θ ) TOL k θ ∈ (0 , 1) L ( k − 1) ≤ L ≤ Lmax ℓ } L ( k ) compute optimal { M ( k ) ℓ =0 to satisfy TOL k . ℓ } L ( k ) run MLMC with L ( k ) , { M ( k ) ℓ =0 update rates ( α ( k ) , β ( k ) , γ ( k ) ), constants ( C ( k ) w , C ( k ) , C ( k ) ) and v c variances { V ℓ } L ( k ) ℓ =0 based on the new simulations performed k = k + 1 end FOR 29 / 101

  46. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC An adaptive algorithm (Continuation-MLMC) A critical issue is how to estimate variances V ℓ on fine levels where very few simulations are run. In [Haji-Ali, N., Tempone 2014] we have proposed a Bayesian update: prior model V ℓ ≈ C v h β ℓ , 30 / 101

  47. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC An adaptive algorithm (Continuation-MLMC) A critical issue is how to estimate variances V ℓ on fine levels where very few simulations are run. In [Haji-Ali, N., Tempone 2014] we have proposed a Bayesian update: prior model V ℓ ≈ C v h β ℓ , 30 / 101

  48. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC [NANvST01] , Given a hierarchy { h ℓ } L ℓ =0 and the previous models with certain estimates, it is easy to find some optimal parameters Splitting parameter is given in terms of h L as θ = 1 − Q W h q 1 TOL . L Optimal number of samples in R � L � � 2 � � � � C α V ℓ M ∗ ℓ = V ℓ W ℓ . (9) θ TOL W ℓ ℓ =0 Of course, M ℓ ∈ N . We can take M ℓ = ⌈ M ∗ ℓ ⌉ ≤ M ∗ ℓ + 1 . � L � L � L M ∗ Total work = M ℓ W ℓ ≤ ℓ W ℓ + W ℓ . ℓ =0 ℓ =0 ℓ =0 However, under certain conditions (Recall 2 q 1 ≥ min( q 2 , d γ )) the first term dominates the second one as TOL → 0 . Hence, we can consider M ℓ ≈ M ∗ ℓ to get the same work estimate, approximately. Optimal L ∈ N can be found with brute-force optimization in some limited range. 31 / 101

  49. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC We propose a Continuation MLMC algorithm that, given a hierarchy, solves the given approximation problem for a sequence of decreasing tolerances, ending with the desired one. The sequence is chosen such that the total work is dominated by the last iteration. 1: Compute with an initial hierarchy. 2: Estimate problem parameters { V ℓ } L ℓ =0 , Q S , Q W , q 1 and q 2 . 3: Set i = 0. 4: repeat Find optimal integer L . 5: Generate hierarchy { h ℓ } L ℓ =0 for TOL i . 6: Using TOL i and the variance estimates { V ℓ } L ℓ =0 and opti- 7: mal θ , compute { M ℓ } L ℓ =0 according to (9). Compute with the resulting MLMC hierarchy. 8: Estimate problem parameters, { V ℓ } L ℓ =0 , Q S , Q W , q 1 and q 2 . 9: Estimate the total error. 10: Set i = i + 1 11: 12: until Total error estimate is less than TOL 32 / 101

  50. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC Estimating q 1 , q 2 , Q S and Q W is done using a Bayesian approach that allows assuming a prior on q 1 and q 2 . Moreover, samples from multiple levels are used to estimate these parameters. Estimating V ℓ for ℓ > 0 is also done using a Bayesian approach with the models (3) as priors. For r 1 > 1 and r 2 > 1 (e.g. r 1 = 2 and r 2 = 1 . 1) we choose � r i E − i r − 1 2 TOL i < i E , 1 TOL i = r i E − i r − 1 i ≥ i E , 2 TOL 2 � � − log( TOL )+log( r 2 )+log( TOL max ) Here i E = , imposes log( r 1 ) TOL 0 = TOL max for some maximum tolerance. Moreover TOL i E = r − 1 2 TOL . 33 / 101

  51. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC Error plot of CMLMC TOL Direct 0 Iterative 0 1 2 10 − 1 10 − 1 2 1 0 1 0 1 3 2 0 0 0 0 3 10 − 2 3 10 − 2 10 − 3 Error Error 10 − 3 10 − 4 10 − 4 10 − 5 10 − 6 10 − 5 10 − 3 10 − 2 10 − 1 10 − 3 10 − 2 10 − 1 TOL TOL The algorithm was run with C α = 2 so that the bound holds with 95% confidence. 34 / 101

  52. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC Total work of CMLMC Reference Algorithm Direct Iterative 10 3 10 3 10 2 10 2 Time (sec.) Time (sec.) 10 1 10 1 10 0 10 0 10 − 1 10 − 1 10 − 3 10 − 2 10 − 1 10 − 3 10 − 2 10 − 1 TOL TOL Reference lines are TOL − 2 . 25 and TOL − 2 , respectively. This is consistent with Corollary 7. 35 / 101

  53. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC SMLMC( � CMLMC M = 3 , θ = 0 . 5 ) CMLMC ( θ = 0 . 5 ) SMLMC( � M = 25 , θ = 0 . 5 ) Direct Iterative 8 7 7 6 Normalized running time Normalized running time 6 5 5 4 4 3 3 2 2 1 1 0 0 10 − 3 10 − 2 10 − 1 10 − 3 10 − 2 10 − 1 TOL TOL Improvement in running time due to better choice of splitting parameter, θ . 36 / 101

  54. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – CMLMC SMLMC( � CMLMC with reuse M = 3 , θ = 0 . 5 ) with reuse SMLMC( � CMLMC without reuse M = 3 , θ = 0 . 5 ) without reuse Direct Iterative 7 8 7 6 Normalized running time Normalized running time 6 5 5 4 4 3 3 2 2 1 1 0 0 10 − 3 10 − 2 10 − 1 10 − 3 10 − 2 10 − 1 TOL TOL Reusing samples in CMLMC does not significantly improve running, since the work is dominated by the work of the last iteration. 37 / 101

  55. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Optimal hierarchies Optimal hierarchies The optimal hierarchies are presented in [ANvST00] . Theorem (On the optimal hierarchies when χ = 1) For any fixed L ∈ N , with χ = 1 , the optimal sequences { h ℓ } L ℓ =0 and { M ℓ } L ℓ =0 in Problem 1 are given by � (1 − θ ) TOL � 1 q 1 , h ℓ = β ( ℓ − L ) for l = 0 , 1 , 2 , . . . , L , Q W and the optimal choice of the splitting parameter is � − 1 � 1 + 1 1 θ (1 , η, L ) = as L → ∞ . → 1 2 η L + 1 For this case the optimal number of levels, L , satisfies asymptotically log TOL − 1 = 1 L + 1 lim 2 η . TOL → 0 38 / 101

  56. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Optimal hierarchies Theorem (On the optimal hierarchies when χ � = 1) For any fixed L ∈ N , with χ � = 1 , the optimal sequences, { h ℓ } L ℓ =0 in Problem 1 are given by 1 − χℓ +1 1 − χ L +1 � V 0 χℓ − χ L � (1 − θ ) TOL � 1 � 1 d γ 1 − χ L +1 q 1 h ℓ ( θ, L ) = Q W Q S � L ( 1 − χℓ +1 ) − ℓ ( 1 − χ L +1 ) � χ L +1 − χℓ +1 − 1 2 + 1 − χ L +1 1 − χ L +1 d γ 1 − χ · χ , where the optimal choice of the splitting parameter is � − 1 � − 1 � � 1 + 1 1 − χ 1 + 1 − min( χ, 1) θ ( χ, η, L ) = → as L → ∞ . 1 − χ L +1 2 η 2 η For this case the optimal number of levels, L , satisfies asymptotically 1 χ − 1 L + 1 log ( TOL − 1 ) ≤ max { 1 , χ } L + 1 χ − 1 log χ ≤ lim inf log ( TOL − 1 ) ≤ lim sup log χ . 2 η 2 η TOL → 0 TOL → 0 39 / 101

  57. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Optimal hierarchies Corollary (On the asymptotic work with optimal hierarchies) For the these optimal hierarchies and using asymptotic upper bounds on L , the total computational complexity W ( H ) � → C 0 , as TOL ց 0 for 0 < χ < 1 � 1 + 1 − χ TOL − 2 2 η W ( H ) TOL − 2 (log( TOL )) 2 → C 1 , as TOL ց 0 for χ = 1 , W ( H ) TOL − 2 → C 2 , as TOL ց 0 for χ > 1 , with known constants of proportionality, � � 1 � 1 − χ � � 1+ 1 − χ � � 2 � � 2 � 2 η − 2 χ 2 η C 0 = C 2 η α Q S Q χ 1 − χ 1 + , W 2 η 1 − χ � 1 � 2 C 1 = C 2 α Q S exp(2) , 2 η � χ − 1 � � 1 � � χ � χ 2 ( χ − 1) − 2 . C 2 = C 2 χ α V 0 χ Q χ − 1 S 40 / 101

  58. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Optimal hierarchies For χ � = 1, the hierarchies are not geometric in general � V 0 � − ( χ − 1) χℓ � (1 − θ ) TOL � ( χ − 1) χℓ +1 1 − χ + ( L +1) χℓ +1 h ℓ +1 d γ ( χ 1+ L − 1) q 1( χ L +1 − 1) 2 1 d γ ( χ L +1 − 1 ) = χ . h ℓ Q S Q W However, we argue that for a nearby L ∈ R (not necessarily optimal, or applicable) that these hierarchies become geometric. Moreover, for geometric hierarchies h ℓ = h 0 β ℓ (not necessarily nested) with � 2 d γ (1 − χ ) , χ if χ ∈ R + \ { 1 } , � � β = − 2 exp , if χ = 1 , q 2 � � 1 q 2 χ 1 d γ (1 − χ ) , if χ > 1, the asymptotic computational V 0 and h 0 = Q S complexity is the same as the computational complexity of the optimized hierarchies. 41 / 101

  59. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Optimal hierarchies Real-valued optimized hierarchies Real-valued geometric hierarchies Integer-valued hierarchies integer-valued geometric hierarchies Local integer optimization χ = 1 . 33333333333 χ = 0 . 888888888889 1 . 4 1 . 3 1 . 3 Normalized Work Normalized Work 1 . 2 1 . 2 1 . 1 1 . 1 1 . 0 1 . 0 0 . 9 0 . 9 10 − 7 10 − 6 10 − 5 10 − 4 10 − 3 10 − 2 10 − 1 10 − 7 10 − 6 10 − 5 10 − 4 10 − 3 10 − 2 10 − 1 TOL TOL The number of levels in each hierarchy was chosen using brute-force optimization. 42 / 101

  60. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi Level Monte Carlo (MLMC) – Conclusions Conclusions Showed normality of MLMC estimator under certain conditions through the use of Lindeberg central limit theorem. We use this in the formulation of our MLMC algorithm and the work optimization problem to control the error in weak sense. Computational saving through better tolerance splitting between bias and statistical error contributions. A more stable continuation MLMC algorithm to learn problem parameters with a small overhead. In CMLMC, reusing samples does not introduce significant computational savings. We show that geometric hierarchies, which are not integer subdivisions, are near-optimal. We derive the MLMC computational complexity with known rates and multiplicative constants. The knowledge of these constants may be used to improve the algorithm further. 43 / 101

  61. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo Outline Variance Reduction techniques – Control variates Multi Level Monte Carlo (MLMC) Main idea and complexity analysis Further optimization of MLMC Normality of MLMC estimator CMLMC Optimal hierarchies Conclusions Multi-Index Monte Carlo General Framework Choosing the Multi-Index Set in MIMC Main Theorem Numerical Results Conclusions Multi-index Stochastic Collocation Complexity analysis Infinite dimensional case (parametric regularity) 44 / 101

  62. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo Variance reduction: MLMC 45 / 101

  63. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo Variance reduction: further potential 45 / 101

  64. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework MIMC Estimator Consider discretization parameters possibly different in each direction h i ,α i = h i , 0 β − α i i with β i > 1. For a multi-index α = ( α i ) d i =1 ∈ N d , we denote by g α the approximation of g calculated using a discretization defined by α . For i = 1 , . . . , d , define the first order difference operators � g α if α i = 0 , ∆ i g α = g α − g α − e i if α i > 0 , and construct the first order mixed difference � � � ⊗ d ( − 1) | j | g α − j ∆ g α = i =1 ∆ i g α = j ∈{ 0 , 1 } d with | j | = � d i =1 j i . Requires 2 d evaluations of g on different grids. 46 / 101

  65. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework MIMC Estimator Consider discretization parameters possibly different in each direction h i ,α i = h i , 0 β − α i i with β i > 1. For a multi-index α = ( α i ) d i =1 ∈ N d , we denote by g α the approximation of g calculated using a discretization defined by α . For i = 1 , . . . , d , define the first order difference operators � g α if α i = 0 , ∆ i g α = g α − g α − e i if α i > 0 , and construct the first order mixed difference � � � ⊗ d ( − 1) | j | g α − j ∆ g α = i =1 ∆ i g α = j ∈{ 0 , 1 } d with | j | = � d i =1 j i . Requires 2 d evaluations of g on different grids. 46 / 101

  66. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework MIMC Estimator Consider discretization parameters possibly different in each direction h i ,α i = h i , 0 β − α i i with β i > 1. For a multi-index α = ( α i ) d i =1 ∈ N d , we denote by g α the approximation of g calculated using a discretization defined by α . For i = 1 , . . . , d , define the first order difference operators � g α if α i = 0 , ∆ i g α = g α − g α − e i if α i > 0 , and construct the first order mixed difference � � � ⊗ d ( − 1) | j | g α − j ∆ g α = i =1 ∆ i g α = j ∈{ 0 , 1 } d with | j | = � d i =1 j i . Requires 2 d evaluations of g on different grids. 46 / 101

  67. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework Left: Tensor domain, cylinder. Center: Non-tensor domain immersed in a tensor box. Right: Non-tensor domain with a structured mesh. 47 / 101

  68. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework Example: Computing g α in d = 2 α 1 For α = ( α 1 , α 2 ), we have ∆ g ( α 1 ,α 2 ) = ∆ 2 (∆ 1 g ( α 1 ,α 2 ) ) = ∆ 2 ( g α 1 ,α 2 − g α 1 − 1 ,α 2 ) = g α 1 ,α 2 − g α 1 − 1 ,α 2 α 2 − g α 1 ,α 2 − 1 + g α 1 − 1 ,α 2 − 1 . Notice that in general, ∆ g α requires 2 d eval- uations of g at different discretization pa- rameters, the largest work of which corre- sponds precisely to the index appearing in ∆ g α , namely α . 48 / 101

  69. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework MIMC Estimator Then, assuming that E [ g α ] → E [ g ] as α i → ∞ for all i = 1 , . . . , d , it is not difficult to see that � � E [ g ] = E [∆ g α ] ≈ E [∆ g α ] α ∈ N d α ∈I where I ⊂ N d is a properly chosen index set. As in MLMC, appoximating each term by independent MC samplers, the MIMC estimator can be written as M α � � 1 A MIMC = ∆ g α ( ω α , m ) M α α ∈I m =1 with properly chosen sample sizes ( M α ) α ∈I . 49 / 101

  70. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework MIMC Estimator Then, assuming that E [ g α ] → E [ g ] as α i → ∞ for all i = 1 , . . . , d , it is not difficult to see that � � E [ g ] = E [∆ g α ] ≈ E [∆ g α ] α ∈ N d α ∈I where I ⊂ N d is a properly chosen index set. As in MLMC, appoximating each term by independent MC samplers, the MIMC estimator can be written as M α � � 1 A MIMC = ∆ g α ( ω α , m ) M α α ∈I m =1 with properly chosen sample sizes ( M α ) α ∈I . 49 / 101

  71. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework MIMC Estimator Then, assuming that E [ g α ] → E [ g ] as α i → ∞ for all i = 1 , . . . , d , it is not difficult to see that � � E [ g ] = E [∆ g α ] ≈ E [∆ g α ] α ∈ N d α ∈I where I ⊂ N d is a properly chosen index set. As in MLMC, appoximating each term by independent MC samplers, the MIMC estimator can be written as M α � � 1 A MIMC = ∆ g α ( ω α , m ) M α α ∈I m =1 with properly chosen sample sizes ( M α ) α ∈I . 49 / 101

  72. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – General Framework Our objective is to build an estimator A = A MIMC where P ( |A − E [ g ] | ≤ TOL ) ≥ 1 − ǫ (10) for a given accuracy TOL and a given confidence level determined by 0 < ǫ ≪ 1. We instead impose the following, more restrictive, two constraints: Bias constraint: | E [ A − g ] | ≤ (1 − θ ) TOL , (11) Statistical constraint: P ( |A − E [ A ] | ≤ θ TOL ) ≥ 1 − ǫ. (12) For a given fixed θ ∈ (0 , 1). Moreover, motivated by the asymptotic normality of the estimator, A , we approximate (12) by � θ TOL � 2 Var [ A ] ≤ . (13) C ǫ Here, 0 < C ǫ is such that Φ( C ǫ ) = 1 − ǫ 2 , where Φ is the cumulative distribution function of a standard normal random variable. 50 / 101

  73. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Given variance and work estimates we can already optimize for the optimal number of samples M ∗ α ∈ R to satisfy the variance constraint (13) �� � � 2 � � � C ǫ V α M ∗ α = V α W α . θ TOL W α α ∈I Taking M ∗ α ≤ M α ≤ M ∗ α + 1 such that M α ∈ N and substituting in the total work gives � 2 �� � 2 � � � C ǫ Work( I ) ≤ V α W α + W α . θ TOL α ∈I α ∈I � �� � Min. cost of I Observe:The work now depends on the index set, I , only. 51 / 101

  74. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Choosing the Index set I – Tensor Product sets An obvious choice of I is the Full Tensor α 1 index-set I ( L ) = { α ∈ N d : α i ≤ L i for i ∈ { 1 · · · d }} for some L ∈ R d . L 1 It turns out, unsurprisingly, that Full Tensor (FT) index-sets impose restrictive conditions on the weak rates w i and yield sub-optimal α 2 L 2 complexity rates. � g − � � Remark : The Bias = E α ∈I L g α = E [ g − g L ] corresponds to the discretization error on a (possibly anisotropic) full tensor grid of level L = ( L 1 , . . . , L d ). 52 / 101

  75. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Choosing the Index set I – Tensor Product sets An obvious choice of I is the Full Tensor α 1 index-set I ( L ) = { α ∈ N d : α i ≤ L i for i ∈ { 1 · · · d }} for some L ∈ R d . L 1 It turns out, unsurprisingly, that Full Tensor (FT) index-sets impose restrictive conditions on the weak rates w i and yield sub-optimal α 2 L 2 complexity rates. � g − � � Remark : The Bias = E α ∈I L g α = E [ g − g L ] corresponds to the discretization error on a (possibly anisotropic) full tensor grid of level L = ( L 1 , . . . , L d ). 52 / 101

  76. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Choosing the Index set I – Tensor Product sets An obvious choice of I is the Full Tensor α 1 index-set I ( L ) = { α ∈ N d : α i ≤ L i for i ∈ { 1 · · · d }} for some L ∈ R d . L 1 It turns out, unsurprisingly, that Full Tensor (FT) index-sets impose restrictive conditions on the weak rates w i and yield sub-optimal α 2 L 2 complexity rates. � g − � � Remark : The Bias = E α ∈I L g α = E [ g − g L ] corresponds to the discretization error on a (possibly anisotropic) full tensor grid of level L = ( L 1 , . . . , L d ). 52 / 101

  77. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC MIMC general analysis framework Question: How do we find optimal index set I for MIMC? � I⊂ N d Work ( I ) min such that Bias = E α ≤ (1 − θ ) TOL , α / ∈I Assumption: MIMC work is not dominated by the work to compute a single sample corresponding to each α . � Then, minimizing equivalently Work ( I ), the previous min problem can be recast into a knapsack problem with profits defined for each multi-index α . The corresponding α profit is P α = Bias contribution E α √ V α W α Work contribution = 53 / 101

  78. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Bias for MIMC What is the Bias in a MIMC approximation? � � � � Bias = E g − g α = E [ g α ] α ∈I α / ∈I It corresponds to the discretization error on a sparse grid by the so called Combination Technique [Griebel-Schneider-Zenger ’91] (from [Burgartz-Griebel, Acta Num. ’04]) 54 / 101

  79. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Bias for MIMC What is the Bias in a MIMC approximation? � � � � Bias = E g − g α = E [ g α ] α ∈I α / ∈I It corresponds to the discretization error on a sparse grid by the so called Combination Technique [Griebel-Schneider-Zenger ’91] (from [Burgartz-Griebel, Acta Num. ’04]) 54 / 101

  80. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC MIMC general analysis framework Define the total error associated with an index-set I as � E ( I ) = E α α / ∈I and the corresponding total work estimate as � � W ( I ) = V α W α . α ∈I Then we can show the following optimality result with respect to E ( I ) and W ( I ), namely: Lemma (Optimal profit sets) The index-set I ( ν ) = { α ∈ N d : P α ≥ ν } √ V α W α is optimal in the sense that any other index-set, ˜ E α for P α = I , with smaller work, W (˜ I ) < W ( I ( ν )) , leads to a larger error, E (˜ I ) > E ( I ( ν )) . 55 / 101

  81. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC MIMC general analysis framework Once the shape of I is determined, we find I ( TOL ) to be the minimum set of the family that satisfies the bias constraint � E ( I ( TOL )) = E α ≤ ( 1 − θ ) TOL α / ∈I ( TOL ) yielding the corresponding computational work   2 � � 2 � � C ǫ   � TOL − ( 2 + p ) V α W α θ TOL α ∈I ( TOL ) with p ≥ 0 and possibly some multiplicative log factors in the above estimate. To get sharper complexity results we need particular, problem dependent, assumptions, as we see next. 56 / 101

  82. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Particular Assumptions for MIMC For every α , we assume the following � d i =1 β − α i w i Assumption 1 (Bias) : E α = | E [∆ g α ] | � i � d i =1 β − α i s i Assumption 2 (Variance) : V α = Var [∆ g α ] � , i � d i =1 β α i γ i Assumption 3 (Work) : W α = Work(∆ g α ) � , i For positive constants γ i , w i , s i ≤ 2 w i and for i = 1 . . . d . �� d � � � i =1 β α i γ i Work(MIMC) = M α W α � M α . i α ∈I α ∈I 57 / 101

  83. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Remark on product rate particular assumptions The Assumptions 1 & 2 in MIMC that the rates for each α are products of 1D rates � d � d i =1 β − α i w i i =1 β − α i s i E α ∝ , V α ∝ i i are stronger than the corresponding assumptions in MLMC. They imply existence of mixed derivatives of the solution of the PDE (and possibly the solution of the adjoint problem associated to the functional Ψ), as opposed to standard derivatives for MLMC. 58 / 101

  84. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Remark on product rate particular assumptions The Assumptions 1 & 2 in MIMC that the rates for each α are products of 1D rates � d � d i =1 β − α i w i i =1 β − α i s i E α ∝ , V α ∝ i i are stronger than the corresponding assumptions in MLMC. They imply existence of mixed derivatives of the solution of the PDE (and possibly the solution of the adjoint problem associated to the functional Ψ), as opposed to standard derivatives for MLMC. 58 / 101

  85. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Defining the optimal index-set for MIMC Under Assumptions 1-3 (assuming that they hold as sharp estimates and not just as upper bounds) we have � d − ( w i + γ i − si i =1 log( β i )( w i + γ i − si ) α i = e − � d ) α i = e − C δ � d i =1 δ i α i P α = i =1 β 2 2 i δ i = log( β i )( w i + γ i − s i � d ) log( β j )( w j + γ j − s j 2 with , and C δ = ) . C δ 2 j =1 Observe that 0 < δ i ≤ 1, since s i ≤ 2 w i and γ i > 0. Moreover, � d δ i = 1. i =1 Then, for any L ∈ R , the optimal index-set can be written as � d I δ ( L ) = { α ∈ N d : α · δ = α i δ i ≤ L } . (14) i =1 59 / 101

  86. Monte Carlo, Multilevel Monte Carlo, Multi-Index Monte Carlo Ra´ ul Tempone Multi-Index Monte Carlo – Choosing the Multi-Index Set in MIMC Defining the optimal index-set for MIMC Under Assumptions 1-3 (assuming that they hold as sharp estimates and not just as upper bounds) we have � d − ( w i + γ i − si i =1 log( β i )( w i + γ i − si ) α i = e − � d ) α i = e − C δ � d i =1 δ i α i P α = i =1 β 2 2 i δ i = log( β i )( w i + γ i − s i � d ) log( β j )( w j + γ j − s j 2 with , and C δ = ) . C δ 2 j =1 Observe that 0 < δ i ≤ 1, since s i ≤ 2 w i and γ i > 0. Moreover, � d δ i = 1. i =1 Then, for any L ∈ R , the optimal index-set can be written as � d I δ ( L ) = { α ∈ N d : α · δ = α i δ i ≤ L } . (14) i =1 59 / 101

Recommend

Bayesian parameter estimation using Multilevel and multi-index Monte Carlo Kody Law joint with

Bayesian parameter estimation using Multilevel and multi-index Monte Carlo Kody Law joint with

SMC 2 S(ML)MC 2 MLMC BIP OBIP Coupling PMCMC PMC(ML)MC Numerics MIMC Summary Bayesian parameter estimation using Multilevel and multi-index Monte Carlo Kody Law joint with A. Jasra (NUS), K. Kamatani (Osaka), Y. Xu (NUS*), &amp; Y.

701 views • 53 slides
Multilevel and Multi-index Monte Carlo methods for the McKean-Vlasov equation Abdul-Lateef

Multilevel and Multi-index Monte Carlo methods for the McKean-Vlasov equation Abdul-Lateef

Multilevel and Multi-index Monte Carlo methods for the McKean-Vlasov equation Abdul-Lateef Haji-Ali ul Tempone Ra Heriot-Watt University, United Kingdom Alexander von Humboldt Professor, RWTH Aachen, Germany, KAUST, Saudi

791 views • 60 slides
Multilevel and Multi-index Monte Carlo methods for the McKean-Vlasov equation ul Tempone

Multilevel and Multi-index Monte Carlo methods for the McKean-Vlasov equation ul Tempone

Multilevel and Multi-index Monte Carlo methods for the McKean-Vlasov equation ul Tempone Abdul-Lateef Haji-Ali Ra Mathematical Institute, University of Oxford, United Kingdom King Abdullah University of Science and Technology

684 views • 65 slides
IGA-based Multi-Index Stochastic Collocation for Uncertainty Quantification J. Beck 1 , L.

IGA-based Multi-Index Stochastic Collocation for Uncertainty Quantification J. Beck 1 , L.

IGA-based Multi-Index Stochastic Collocation for Uncertainty Quantification J. Beck 1 , L. Tamellini 2 , R. Tempone 1 , 3 1 King Abdullah University of Science and Technology, Saudi Arabia 2 CNR-IMATI, Italy 3 Alexander von Humboldt Professor in

691 views • 65 slides
Monte-Carlo tree search for Monte-Carlo tree search for multi-player, no-limit multi-player,

Monte-Carlo tree search for Monte-Carlo tree search for multi-player, no-limit multi-player,

Monte-Carlo tree search for Monte-Carlo tree search for multi-player, no-limit multi-player, no-limit Texas hold'em poker Texas hold'em poker Guy Van den Broeck Should I bluff? Deceptive play Should I bluff? Is he bluffing? Opponent

1.14k views • 112 slides
ACCURATE EXTRACTION OF TISSUE PARAMETERS FOR MONTE CARLO SIMULATIONS USING MULTI-ENERGY CT Arthur

ACCURATE EXTRACTION OF TISSUE PARAMETERS FOR MONTE CARLO SIMULATIONS USING MULTI-ENERGY CT Arthur

ACCURATE EXTRACTION OF TISSUE PARAMETERS FOR MONTE CARLO SIMULATIONS USING MULTI-ENERGY CT Arthur Lalonde and Hugo Bouchard Dpartement de physique, Universit de Montral THE IMPORTANCE OF MC IN RT Monte Carlo (MC) simulations offer many

575 views • 24 slides
Unifying Orthogonal Monte Carlo Methods From Kacs Random Walks To Hadamard Multi Rademachers

Unifying Orthogonal Monte Carlo Methods From Kacs Random Walks To Hadamard Multi Rademachers

Unifying Orthogonal Monte Carlo Methods From Kacs Random Walks To Hadamard Multi Rademachers Krzysztof Choromanski, Mark Rowland Wenyu Chen, Adrian Weller The Phenomenon of Orthogonal Monte Carlo Estimators Estimation task: Applications:

247 views • 23 slides
Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits Martin Zhang joint work with:

Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits Martin Zhang joint work with:

Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits Martin Zhang joint work with: David Tse &amp; James Zou Stanford University Problem | Monte Carlo Multiple Hypothesis Testing SNP 1 SNP 2 SNP m Problem | Monte Carlo Multiple

526 views • 19 slides
Meet Genetic Algorithms in Monte Carlo: Optimised Placement of Multi-Service Applications in the

Meet Genetic Algorithms in Monte Carlo: Optimised Placement of Multi-Service Applications in the

IEEE INTERNATIONAL CONFERENCE ON EDGE COMPUTING JULY 8-13, 2019, MILAN, ITALY Meet Genetic Algorithms in Monte Carlo: Optimised Placement of Multi-Service Applications in the Fog Antonio Brogi, Stefano Forti Carlos Guerrero, Isaac Lera

876 views • 24 slides
Simulation Monte Carlo Monte Carlo simulation Outcome of a single stochastic simulation run

Simulation Monte Carlo Monte Carlo simulation Outcome of a single stochastic simulation run

Simulation Monte Carlo Monte Carlo simulation Outcome of a single stochastic simulation run is always random A single instance of a random variable Goal of a simulation experiment is to get knowledge about the distribution of the

427 views • 18 slides
Multi-Indexed Files : Outline ! Introduction ! Inverted Files ! Multilist Files rasitjutrakul

Multi-Indexed Files : Outline ! Introduction ! Inverted Files ! Multilist Files rasitjutrakul

Multi-Indexed Files : Outline ! Introduction ! Inverted Files ! Multilist Files rasitjutrakul Organization Index by key #1 Index by key #1 Data File Index by key #2 Data File Index by key #2 Index by key #3 Index by key #3 rasitjutrakul

672 views • 9 slides
Equilibrium Study of Spin-Glass Models: Monte Carlo Simulation and Multi-variate Analysis Koji

Equilibrium Study of Spin-Glass Models: Monte Carlo Simulation and Multi-variate Analysis Koji

Equilibrium Study of Spin-Glass Models: Monte Carlo Simulation and Multi-variate Analysis Koji Hukushima Department of Basic Science, University of Tokyo mailto:hukusima@phys.c.u-tokyo.ac.jp July 1115, 2003 In collaboration with Prof. Y.

226 views • 20 slides
Monte Carlo Methods Guojin Chen Christopher Cprek Chris Rambicure Monte Carlo Methods 1.

Monte Carlo Methods Guojin Chen Christopher Cprek Chris Rambicure Monte Carlo Methods 1.

Monte Carlo Methods Guojin Chen Christopher Cprek Chris Rambicure Monte Carlo Methods 1. Introduction 2. History 3. Examples Introduction Monte Carlo methods are stochastic techniques. Monte Carlo method is very

1.27k views • 68 slides
Monte Carlo Simulation technique S. B. Santra Department of Physics Indian Institute of

Monte Carlo Simulation technique S. B. Santra Department of Physics Indian Institute of

Monte Carlo Simulation technique S. B. Santra Department of Physics Indian Institute of Technology Guwahati Introduction What is Monte Carlo (MC)? Monte Carlo method is a common name for a wide variety of stochastic techniques. These

3.05k views • 48 slides
Monte Carlo methods for sampling-based Stochastic Optimization Gersende FORT LTCI CNRS &amp;

Monte Carlo methods for sampling-based Stochastic Optimization Gersende FORT LTCI CNRS &amp;

Monte Carlo methods for sampling-based Stochastic Optimization Monte Carlo methods for sampling-based Stochastic Optimization Gersende FORT LTCI CNRS &amp; Telecom ParisTech Paris, France Joint works with B. Jourdain, T. Leli` evre, G.

608 views • 36 slides
Index Theory and Structural Analysis for multi-mode DAE Systems Albert Benveniste Benot

Index Theory and Structural Analysis for multi-mode DAE Systems Albert Benveniste Benot

Index Theory and Structural Analysis for multi-mode DAE Systems Albert Benveniste Benot Caillaud Khalil Ghorbal (Inria, Rennes) Marc Pouzet (ENS, Paris) Hilding Elmqvist (Mogram, Lund) Martin Otter (DRL, Munich) December 6, 2016 1 / 34

877 views • 69 slides
Monte Carlo Approximation of Monte Carlo Filters Adam M. Johansen et al. Collaborators Include:

Monte Carlo Approximation of Monte Carlo Filters Adam M. Johansen et al. Collaborators Include:

Monte Carlo Approximation of Monte Carlo Filters Adam M. Johansen et al. Collaborators Include: Arnaud Doucet, Axel Finke, Anthony Lee, Nick Whiteley 7th January 2014 Introduction Monte Carlo Approximationof Monte Carlo Filters Approximating

1.46k views • 31 slides
Other Covered Units An environmental investigation revealed the index unit in a multi- unit

Other Covered Units An environmental investigation revealed the index unit in a multi- unit

EBLL Response in Public Housing: Other Covered Units An environmental investigation revealed the index unit in a multi- unit property has lead- based paint hazards What now? RESPONDING TO EBLLs: Other Covered Units Perform a risk

435 views • 12 slides
Monte Carlo simulations of the 2D Ising model Stochastic sampling of spin configurations to

Monte Carlo simulations of the 2D Ising model Stochastic sampling of spin configurations to

Monte Carlo simulations of the 2D Ising model Stochastic sampling of spin configurations to estimate Spin configurations configurations; can sample very small fraction for large N Trivial Monte Carlo sampling fails at low T, because the sum is

332 views • 15 slides
Other Covered Units An environmental investigation revealed the index unit in a multi- unit

Other Covered Units An environmental investigation revealed the index unit in a multi- unit

EBLL Response in HCV Units: Other Covered Units An environmental investigation revealed the index unit in a multi- unit property has lead- based paint hazards What now? RESPONDING TO EBLLs: Other Covered Units Perform a risk

150 views • 11 slides
28 April 2017 Contents What is Digital Literacy Multi-Dimensional Digital Literacy Index

28 April 2017 Contents What is Digital Literacy Multi-Dimensional Digital Literacy Index

HSRC Seminar Team: Krish Chetty (HSRC), LIU Qigun (AIF), LI Wenwei (AIF), CHEN Fang (AIF) 28 April 2017 Contents What is Digital Literacy Multi-Dimensional Digital Literacy Index Need for a Standardised Definition Data Collection

392 views • 25 slides
Multi-Dimensional Indexing: FFT Use a transform to generate an index pattern that still

Multi-Dimensional Indexing: FFT Use a transform to generate an index pattern that still

Multi-Dimensional Indexing: FFT Use a transform to generate an index pattern that still contains all values in the range (0 to N-1). Size N n takes on values of n: 0, 1, 2, 3, ... N-2, N-1 or N, 1+N, 2+N, 3+N, ... 2N-2, 2N-1

1.04k views • 56 slides
Experimental design for multi-level data: Improving our approach to power analysis using Monte

Experimental design for multi-level data: Improving our approach to power analysis using Monte

Experimental design for multi-level data: Improving our approach to power analysis using Monte Carlo simulation-based parameter recovery estimation Chadwick, S. 1 , &amp; Davies, R. 1 International Multilevel Conference 2019 1 Department of

657 views • 30 slides
Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Gersende

Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Gersende

Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Stochastic approximation-based algorithms, when the Monte Carlo bias does not vanish Gersende Fort Institut de Math ematiques de Toulouse CNRS Toulouse,

559 views • 29 slides

More recommend