Randomness in reduced order modeling Akil Narayan 1 1 Department of - PowerPoint PPT Presentation

Randomness in reduced order modeling Akil Narayan 1 1 Department of Mathematics, and Scientific Computing and Imaging (SCI) Institute University of Utah February 7, 2020 ICERM A. Narayan (U. Utah – SCI) Randomization and ROM

Randomness is your friend Many things that are difficult to accomplish with deterministic optimization/algorithms can be accomplished ˚ with randomization. A. Narayan (U. Utah – SCI) Randomization and ROM

Randomness is your friend Many things that are difficult to accomplish with deterministic optimization/algorithms can be accomplished ˚ with randomization. ˚ : with “high” probability A. Narayan (U. Utah – SCI) Randomization and ROM

Randomness is your friend Many things that are difficult to accomplish with deterministic optimization/algorithms can be accomplished ˚ with randomization. ˚ : with “high” probability We’ll consider three examples of this in ROM: RBM for elliptic PDE’s Sparse approximation Measure atomization/discretization A. Narayan (U. Utah – SCI) Randomization and ROM

Randomness is your friend Many things that are difficult to accomplish with deterministic optimization/algorithms can be accomplished ˚ with randomization. ˚ : with “high” probability We’ll consider three examples of this in ROM: RBM for elliptic PDE’s Sparse approximation Measure atomization/discretization A. Narayan (U. Utah – SCI) Randomization and ROM

❊ ❊ ❊ ❊ Why is randomness helpful? Intuition is straightforward and simplistic: Let X be a random variable. Let p X m q m ě 1 be iid copies of X . Law of large numbers: M ÿ S : “ X m Ñ ❊ r X s . m “ 1 A. Narayan (U. Utah – SCI) Randomization and ROM

❊ ❊ Why is randomness helpful? Intuition is straightforward and simplistic: Let X be a random variable. Let p X m q m ě 1 be iid copies of X . Law of large numbers: M ÿ S : “ X m Ñ ❊ r X s . m “ 1 Furthermore, this convergence is quantitative through the Central limit theorem: 0 , σ 2 p X q ˆ ˙ S p M q ´ ❊ r X s „ N . M In other words, S concentrates around ❊ r X s . A. Narayan (U. Utah – SCI) Randomization and ROM

Why is randomness helpful? Intuition is straightforward and simplistic: Let X be a random variable. Let p X m q m ě 1 be iid copies of X . Law of large numbers: M ÿ S : “ X m Ñ ❊ r X s . m “ 1 Furthermore, this convergence is quantitative through the Central limit theorem: 0 , σ 2 p X q ˆ ˙ S p M q ´ ❊ r X s „ N . M In other words, S concentrates around ❊ r X s . This statement is quite powerful: S provides an estimator for ❊ r X s , without knowing ❊ r X s . Convergence is essentially independent of distribution of X . Convergence rate is independent of dimension of X . A. Narayan (U. Utah – SCI) Randomization and ROM

Examples of concentration Concentration in general plays an important role in computing estimates: Monte Carlo (CLT) estimates Chebyshev inequalities (bounds on mass away from the mean) Hoeffding inequalities (bounds on deviation of iid sums from the mean) Chernoff bounds (bounds on deviation of spectrum) Concentration of measure (bounds on deviation of random functions) Today: We’ll see a particular Chernoff bound in action. A. Narayan (U. Utah – SCI) Randomization and ROM

Chernoff bound applications We will see how randomization and Chernoff bounds can be applied to: RBM for elliptic PDE’s Sparse approximation Measure atomization/discretization Before discussing ROM, let’s present the Chernoff bound. A. Narayan (U. Utah – SCI) Randomization and ROM

❊ Matrix law of large numbers Let G P ❘ N ˆ N be a Gramian matrix that is an iid sum of symmetric rank-1 matrices. I.e., let X P ❘ N have distribution µ on ❘ N , and define M G : “ 1 ÿ X m X T m , M m “ 1 where t X m u m ě 1 are iid copies of X . Chernoff bounds make quantitative statements about the spectrum of G that depend on the distribution of X . A. Narayan (U. Utah – SCI) Randomization and ROM

Matrix law of large numbers Let G P ❘ N ˆ N be a Gramian matrix that is an iid sum of symmetric rank-1 matrices. I.e., let X P ❘ N have distribution µ on ❘ N , and define M G : “ 1 ÿ X m X T m , M m “ 1 where t X m u m ě 1 are iid copies of X . Chernoff bounds make quantitative statements about the spectrum of G that depend on the distribution of X . For large M , we expect that M Ò8 p G q j,k Ý Ñ ❊ r X j X k s . A. Narayan (U. Utah – SCI) Randomization and ROM

Matrix law of large numbers Let G P ❘ N ˆ N be a Gramian matrix that is an iid sum of symmetric rank-1 matrices. I.e., let X P ❘ N have distribution µ on ❘ N , and define M G : “ 1 ÿ X m X T m , M m “ 1 where t X m u m ě 1 are iid copies of X . Chernoff bounds make quantitative statements about the spectrum of G that depend on the distribution of X . For large M , we expect that M Ò8 p G q j,k Ý Ñ ❊ r X j X k s . For simplicity, in all that follows we assume that the components of X are uncorrelated, of unit variance, so that M Ò8 G Ý Ñ I A. Narayan (U. Utah – SCI) Randomization and ROM

Matrix Chernoff bounds The proximity of G to I , as a function of M , is determined by K : “ sup } X } 2 , X which is assumed finite. A. Narayan (U. Utah – SCI) Randomization and ROM

Matrix Chernoff bounds The proximity of G to I , as a function of M , is determined by K : “ sup } X } 2 , X which is assumed finite. Theorem ( [Cohen, Davenport, Leviatan 2012] ) Assume that ˆ 1 ˙ log M Á K M δ 2 log . ǫ Then, ” ı ď Pr p σ min p G q ă 1 ´ δ q p σ max p G q ą 1 ` δ q ď ǫ. A. Narayan (U. Utah – SCI) Randomization and ROM

Matrix Chernoff bounds The proximity of G to I , as a function of M , is determined by K : “ sup } X } 2 , X which is assumed finite. Theorem ( [Cohen, Davenport, Leviatan 2012] ) Assume that ˆ 1 ˙ log M Á K M δ 2 log . ǫ Then, ” ı ď Pr p σ min p G q ă 1 ´ δ q p σ max p G q ą 1 ` δ q ď ǫ. What can we do with G ? Form least-squares approximations using X . Remarks: The δ ´ 2 dependence is “CLT-like”. K is the only thing that depends on the distribution of X . A. Narayan (U. Utah – SCI) Randomization and ROM

The induced distribution It turns out that K can be quite large (or infinite) for practical situations. A fix for this utilizes importance sampling . In particular, define ˜ ¸ N 1 ÿ x 2 d ρ p x q : “ d µ p x q , n N n “ 1 where µ is the distribution of X . ρ is a probability measure on ❘ N , and is frequently called the induced distribution . A. Narayan (U. Utah – SCI) Randomization and ROM

A (more) optimal Chernoff bound In practical scenarios, the induced distribution ρ can also be sampled from without too much effort. More importantly, we can get a (much) better Chernoff bound here. Let p Y m q m ě 1 P ❘ N be iid samples from ρ . We need to weight the Gramian so that we produce an unbiased estimate: M F : “ 1 w m : “ d µ ÿ w m Y m Y T d ρ p Y m q m , M m “ 1 A. Narayan (U. Utah – SCI) Randomization and ROM

A (more) optimal Chernoff bound In practical scenarios, the induced distribution ρ can also be sampled from without too much effort. More importantly, we can get a (much) better Chernoff bound here. Let p Y m q m ě 1 P ❘ N be iid samples from ρ . We need to weight the Gramian so that we produce an unbiased estimate: M F : “ 1 w m : “ d µ ÿ w m Y m Y T d ρ p Y m q m , M m “ 1 This results in the (better) Chernoff bound ” ı ď p σ min p F q ă 1 ´ δ q p σ min p F q ą 1 ` δ q ď ǫ, Pr with the much more reasonable assumption ˆ 1 ˙ log M Á N M δ 2 log . ǫ This Chernoff bound will be a seed for achieving model reduction. A. Narayan (U. Utah – SCI) Randomization and ROM

Example 1: RBM (for elliptic problems) A. Narayan (U. Utah – SCI) Randomization and ROM

Reduced basis methods For the parameterized problem, ˜ 8 ¸ ÿ ´ ∇ ¨ µ j a j p x q ∇ u “ b, j “ 1 with µ P r´ 1 , 1 s 8 , recall that RBM (essentially) iteratively computes } u p µ q ´ P j ´ 1 p u p µ qq} , arg max µ A. Narayan (U. Utah – SCI) Randomization and ROM

Reduced basis methods For the parameterized problem, ˜ 8 ¸ ÿ ´ ∇ ¨ µ j a j p x q ∇ u “ b, j “ 1 with µ P r´ 1 , 1 s 8 , recall that RBM (essentially) iteratively computes } u p µ q ´ P j ´ 1 p u p µ qq} , arg max µ If (any truncation of) µ is high-dimensional, this is an expensive optimization, even if the objective is easy to evaluate. There’s a bigger problem: the arg max is typically taken over a discrete µ grid. If µ is high-dimensional, how can we certify error without densely sampling? A. Narayan (U. Utah – SCI) Randomization and ROM

Reduction feasibility Some analysis gives us a strategy to proceed: if the t a j u d j “ 1 satisfies an ℓ p summability condition, 8 ÿ } a j } p L 8 ă 8 , p ă 1 , j “ 1 then there is an N -dimensional downward-closed polynomial space P N in the variable µ such that s : “ 1 p ´ 1 › ď N ´ s , › › › u p µ q ´ Proj P N u p µ q sup 2 . µ There are constructive algorithms to essentially identify P N , [Cohen, Devore, Schwab 2011] . In particular, once P N is identified, this approximation can be obtained by µ -least-squares approximation. A. Narayan (U. Utah – SCI) Randomization and ROM