How to Cope with the Curse of Dimensionality ? Henryk Wo - PowerPoint PPT Presentation

Henryk Wo´ zniakowski Curse of Dimensionality How to Cope with the Curse of Dimensionality ? Henryk Wo´ zniakowski University of Warsaw and Columbia University 30 Years of IMSM, 1

Henryk Wo´ zniakowski Curse of Dimensionality Curse of Dimensionality ε error demand d the number of variables n ( ε, d ) the minimal cost Many problems suffer from the curse of dimensionality n ( ε, d ) ≥ c (1 + C ) d for all d = 1 , 2 , . . . with c > 0 and C > 0 . 30 Years of IMSM, 2

Henryk Wo´ zniakowski Curse of Dimensionality IBC IBC = Information-Based Complexity • IBC is the branch of computational complexity that studies continuous mathematical problems. • Typically, such problems are defined on spaces of functions of d variables. Often d is large. • Typically, the available information is given by finitely many function values. Therefore it is partial, costly and often noisy. 30 Years of IMSM, 3

Henryk Wo´ zniakowski Curse of Dimensionality Multivariate Integration for Korobov Spaces r = { r j } with 1 ≤ r 1 ≤ r 2 ≤ · · · 1 -periodic f : [0 , 1] → C , f ( r j − 1) abs. cont, f ( r j ) ∈ L 2 H r j : � 1 � 1 2 � � 2 � � � f � 2 � f ( r j ) ( t ) � � = f ( t ) d t + d t � � � � H rj � � � 0 0 For d ≥ 1 , H d,r = H r 1 ⊗ H r 2 ⊗ · · · ⊗ H r d Usually, it is assumed that r j ≡ r 30 Years of IMSM, 4

Henryk Wo´ zniakowski Curse of Dimensionality Multivariate Integration For f ∈ H d,r we want to approximate � I d ( f ) := [0 , 1] d f ( t ) d t ≈ A n ( f ) • Algorithms: with x j ∈ [0 , 1] d A n ( f ) = φ n ( f ( x 1 ) , f ( x 2 ) , . . . , f ( x n )) • Minimal Worst Case Error: e ( n, d ) = inf sup | I d ( f ) − A n ( f ) | A n � f � Hd,r ≤ 1 • Information Worst Case Complexity: n ( ε, d ) = min { n | e ( n, d ) ≤ ε } 30 Years of IMSM, 5

Henryk Wo´ zniakowski Curse of Dimensionality Theorem Let r j ≡ r . Then there exists c r > 0 and C r > 0 such that n ( ε, d ) > c r (1 + C r ) d Based on Hickernell+W [2001] and Novak+W[2001], see also Sloan+W[2001] Multivariate integration for Korobov space with arbitrarily smooth functions suffers from the curse of dimensionality 30 Years of IMSM, 6

Henryk Wo´ zniakowski Curse of Dimensionality How to cope with the curse of dimensionality • switch to spaces with increased smoothness with respect to successive variables • switch to weighted spaces, i.e., groups of variables are of varying importance • switch to a more lenient setting, i.e, from the worst case setting to the randomized or average case setting 30 Years of IMSM, 7

Henryk Wo´ zniakowski Curse of Dimensionality Increasing Smoothness Still the worst case setting and unweighted spaces with r 1 ≤ r 2 ≤ · · · . But we now allow to increase r j Let ln k R := lim sup r k k →∞ Theorem If R < 2 ln 2 π then • no curse with p := max( r − 1 n ( ε, d ) ≤ C ε − p (1+ p/ 2) • 1 , R/ ln 2 π ) < 2 , i.e., strong polynomial tractability Based on Papageorgiou+W [09], Kuo, Wasilkowski+W[09] 30 Years of IMSM, 8

Henryk Wo´ zniakowski Curse of Dimensionality Weighted Spaces Major research activities in last 20 years... In particular, for r j ≡ r and γ = { γ j } , redefine H r j ,γ j with � 1 � 1 2 � � 1 2 � � � f � 2 � f ( r j ) ( t ) � � = f ( t ) d t + d t � � H rj ,γj � � γ j � � � 0 0 For d ≥ 1 , H d,r = H r 1 ,γ 1 ⊗ H r 2 ,γ 2 ⊗ · · · ⊗ H r d ,γ d 30 Years of IMSM, 9

Henryk Wo´ zniakowski Curse of Dimensionality Theorem • Gnewuch+W[08] � d j =1 γ j lim d →∞ = 0 iff no curse, d • Hickernell+W[01] � d j =1 γ j lim sup d →∞ < ∞ iff polynomial tractability, ln d i.e., n ( ε, d ) ≤ C d q ε − p • Hickernell+W[01] � ∞ j =1 γ j < ∞ iff strong polynomial tractability, i.e., n ( ε, d ) ≤ C ε − p 30 Years of IMSM, 10

Henryk Wo´ zniakowski Curse of Dimensionality More Lenient Settings From Worst Case Setting to • Randomized Setting • Average Case Setting Average Case Setting ≤ Randomized Setting 30 Years of IMSM, 11

Henryk Wo´ zniakowski Curse of Dimensionality Randomized Setting • Algorithms: A n,ω ( f ) = φ n,ω ( f ( x 1 ,ω ) , f ( x 2 ,ω ) , . . . , f ( x n ( ω ) ,ω )) for a random ω • Minimal Randomized Error: E | I d ( f ) − A n,ω ( f ) | 2 � 1 / 2 � e ( n, d ) = inf sup A n � f � Hd,r ≤ 1 • Information Randomized Complexity: n ( ε, d ) = min { n | e ( n, d ) ≤ ε } 30 Years of IMSM, 12

Henryk Wo´ zniakowski Curse of Dimensionality Monte Carlo Algorithm n A n,ω ( f ) = 1 � f ( x j,ω ) n j =1 with iid with uniform distribution over [0 , 1] d x j,ω • n ( ε, d ) ≤ ε − 2 • no curse and strong polynomial tractability 30 Years of IMSM, 13

Henryk Wo´ zniakowski Curse of Dimensionality Conclusions • Many multivariate problems suffer from the curse of dimensionality in the worst case setting • We may sometimes break the curse of dimensionality by – switching to spaces with increased smoothness with respect to successive variables – switching to weighted spaces, i.e., groups of variables are of varying importance – switching to a more lenient setting, i.e, from the worst case setting to the randomized or average case setting 30 Years of IMSM, 14

Henryk Wo´ zniakowski Curse of Dimensionality Book More can be found in Tractability of Multivariate Problems Erich Novak and Henryk Wo´ zniakowski • Volume I: Linear Information (2008) • Volume II: Standard Information for Functionals (2010) • Volume III: Standard Information for Operators (2012) 30 Years of IMSM, 15