Stochastic algorithms for trace estimation and inverse problems - PowerPoint PPT Presentation

. Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions . Uri Ascher Department of Computer Science University of British Columbia May 2014 . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 1 / 47

Motivation Many data sets . PDE inverse problems & many data sets . Nonlinear parameter function estimation problems involving partial differential equation (PDE) constraints Many measurements for obtaining credible reconstructions Applications Electromagnetic data inversion in mining exploration Seismic data inversion in oil exploration Direct current (DC) resistivity Electrical impedance tomography (EIT) Diffuse optical tomography (DOT) Quantitative photoacoustic tomography (QPAT) . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 2 / 47

Motivation Many data sets . Inverse problem . After discretization and for our problems of interest: d i = F i ( m ) + ε i i = 1 , 2 , . . . s F i ( m ) = P i u i = P i G ( m ) q i Calculating “ G ( m ) q i ” for each i is costly! d i ∈ R l is the measurement obtained in the i th experiment F i is the known forward operator for the i th experiment m ∈ R l m is the sought-after model ε i is the noise incurred in the i th experiment s is the total number of experiments u i ∈ R l u is the i th field q i ∈ R l u is the i th source G − 1 is a square matrix discretizing the PDE with the BC P i ia the projection matrix for the i th experiment . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 3 / 47

Motivation Many data sets . Merit function . Assume ε i ∼ N (0 , ˜ σ I ) Maximum Likelihood (ML) data misfit: φ ( m ) = ∥ F ( m ) − D ∥ 2 F F ∈ R l × s with F i ( m ) for i th column D ∈ R l × s with d i for i th column Maximum a Posteriori (MAP) merit function: φ R ,β ( m ) = φ ( m ) + β R ( m ) R ( m ) can be implicit (e.g. dynamic regularization) Seek to reduce merit function (not much) below the noise level. . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 4 / 47

Motivation Many data sets . Numerical example: many data sets . [van den Doel, Ascher & Haber, 2013] Forward problem: ∇ · ( σ ∇ u i ) = q i , x ∈ Ω , i = 1 , . . . , s , ∂ u i ∂ν | ∂ Ω = 0 . Take Ω to be a square. Construct s data sets, choosing different current patterns: q i ( x ) = δ x , p L − δ x , p R , iL iR where p i L L and p i R R are located on the left and right boundaries resp. Place each at √ s different, equidistant locations, 1 ≤ i L , i R ≤ √ s . . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 5 / 47

Motivation Many data sets . L2G vs L1G results (3% noise) . Left: true. Center: s = 4 , L2G. Right: s = 4 , L1G. . . . . . . . . . . . . . . . . . . . . Left: true. Center: s = 64 , L2G. Right: s = 64 , L1G. .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 6 / 47

Motivation Many data sets . L2G vs L1G results (1% noise) . Decrease noise to 1%, increase √ s : now there is enough information for L1G to shine. 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 Left: s = 1024 , L2G. Right: s = 1024 , L1G. . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 7 / 47

Motivation Many data sets Model reduction and stochastic approximation . . Need to solve s PDEs just to evaluate forward operator F or misfit function! Proceed with caution . Consider framework of iterative method (e.g. Gauss-Newton for a Tikhonov regularization): at k th iteration, given current iterate m k , solve for correction µ k and set m k +1 = m k + µ k . Want to make do with fewer data set combinations at each iteration: s k ≪ s at iteration k . Can we apply stochastic/deterministic model reduction ? . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 8 / 47

Motivation Outline . Outline . Motivation Many measurements Implicit matrix trace estimation Stochastic/deterministic model reduction Adaptive selection of sample size Numerical experiments: EIT/DC resistivity inverse problem Conclusions . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 9 / 47

Implicit matrix trace estimation Trace estimators . Trace? what trace? . Let B ( m ) = F ( m ) − D ∈ R l × s Matrix A = B T B is implicit symmetric positive semi-definite (SPSD); can only carry out matvec products A ∗ v with this s × s matrix. Let w be drawn from distribution such that E ( ww T ) = I . Then φ ( m ) = ∥ B ( m ) ∥ 2 F = tr ( B T B ) = E ( w T A w ) Approximating expectation ⇔ Approximating the trace Monte-Carlo approximation n tr ( A ) ≈ 1 ∑ w T i A w i n i =1 Note we can obtain exact trace using n = s samples with w i scaled i th column of identity; but we want n ≪ s . Link notation with our application : n = s k . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 10 / 47

Implicit matrix trace estimation Trace estimators . Which probability distribution? . Can choose from several: Gaussian: normal distribution Hutchinson: Rademacher distribution – for w = ( w 1 , . . . , w s ) T , Pr ( w j = 1) = Pr ( w j = − 1) = 1 / 2 Random subset (RS) of scaled columns of the identity matrix. Selling features: Hutchinson (1990) showed that Rademacher has smaller variance than normal distribution. This made the Hutchinson choice popular in industry, too. RS applies to general matrices, unlike the other two. On the other hand, for SPSD matrices including in our motivating example, RS is somewhat slower than the other two. . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . .. . . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 11 / 47

Implicit matrix trace estimation Trace estimators . Probabilistic bounds . [Roosta-Khorasani & Ascher, 2013; Avron & Toledo, 2011] following [Achlioptas, 2001] , want n smallest for a desired quality, look for probabilistic bounds. Given pair ( ε, δ ) Probability distribution ∆ Want lower bound on n such that | tr n ( ∆ ( A ) − tr ( A ) | ≤ ε tr ( A ) ) ≥ 1 − δ Pr n ∆ ( A ) = 1 ∑ tr n w T i A w i n i =1 with w i from probability distribution ∆ . . . . . . . . . . . . . . . . . . . . .. . .. . .. . .. . .. . .. . .. . .. . . .. . .. . .. .. . .. . .. . .. . .. . .. . .. . .. . .. . Fred Roosta & Kees van den Doel (UBC) Stochastic algorithms for trace estimation and inverse problems involving many PDE inversions May 2014 12 / 47