Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization From greedy approximation to greedy optimization Vladimir Temlyakov December 10, 2013 Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Introduction 1 Greedy approximation in Hilbert spaces 2 Greedy approximation in Banach spaces 3 Greedy algorithms for convex optimization 4 Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Toy example Let Ψ := { ψ } ∞ k =1 be an orthonormal basis for a Hilbert space H . For any f ∈ H there is a convergent (in H ) orthogonal expansion ∞ � f = � f , ψ k � ψ k . k =1 A classical way of approximation of f is to take a partial sum n � S n ( f , Ψ) := � f , ψ k � ψ k . k =1 For the error we have ∞ � f − S n ( f , Ψ) � 2 = � |� f , ψ k �| 2 . k = n +1 Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization m -term approximation In nonlinear approximation we use the m -term approximation � � f , ψ k � ψ k , | Λ | = m . k ∈ Λ It is clear that the optimal (from the point of view of the error) choice of Λ is the set of m biggest in absolute value coefficients � f , ψ k � . We can realize this choice by picking the biggest coefficients one by one. This results in the reordering (greedy reordering) of the orthogonal expansion: ∞ � f = � f , ψ k i � ψ k i , |� f , ψ k 1 �| ≥ |� f , ψ k 2 �| ≥ . . . . i =1 Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Major questions of greedy approximation 1 Let instead of an orthonormal basis Ψ we have a redundant system D . How to approximate with regard to D ? Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Major questions of greedy approximation 1 Let instead of an orthonormal basis Ψ we have a redundant system D . How to approximate with regard to D ? 2 How to work in a Banach space X instead of a Hilbert space H ? Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Notations We begin with the case where approximation takes place in a Banach space X equipped with a norm � · � := � · � X . We formulate our approximation problem in the following general way. Definition (Dictionary) We say a set of functions D from X is a dictionary if each g ∈ X has norm one ( � g � X = 1) and the closure of Span D coincides with X . We let Σ m ( D ) denote the collection of all functions (elements) in X which can be expressed as a linear combination of at most m elements of D . Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization m -sparse elements Thus each function s ∈ Σ m ( D ) can be written in the form � s = c g g , Λ ⊂ D , #Λ ≤ m , g ∈ Λ where the c g are real numbers. In some cases, it may be possible to write an element from Σ m ( D ) in this form in more than one way. The space Σ m ( D ) is not linear: the sum of two functions from Σ m ( D ) is generally not in Σ m ( D ). Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Examples Perhaps the first example of approximation involving dictionaries was considered by E. Schmidt in 1907, who considered the approximation of functions f ( x , y ) of two variables in L 2 ([0 , 1] 2 ) by functions of the form m � B m ( x , y ) = c j u j ( x ) v j ( y ) . j =1 This approximation problem can be seen as an m -term approximation with regard to the dictionary Π = { g : g ( x , y ) = u ( x ) v ( y ); u , v ∈ L 2 ([0 , 1]) , � u � L 2 = � v � L 2 = 1 } . Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization One more example Another approximation problem of this type which is well known in statistics is the projection pursuit regression problem. The problem is to approximate in L 2 a given multivariate function f ∈ L 2 by a sum of ridge functions, i.e. by m � x , ω j ∈ R d , W m ( x ) = r j ( � ω j , x � ) , j = 1 , . . . , m , j =1 where r j , j = 1 , . . . , m , are univariate functions. Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization More examples Another example, from signal processing, uses the Gabor functions g a , b ( x ) := e iax e − bx 2 and approximates a univariate function by linear combinations of the elements { g a , b ( x − c ) : a , c ∈ R , b > 0 } . Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Best m -term approximation For a function f ∈ X we define its best m -term approximation error σ m ( f , D ) X := s ∈ Σ m ( D ) � f − s � X . inf We concentrate on an important problem of finding good methods of m -term approximation in the case of general dictionary D and on studying their efficiency. Let us begin this discussion in the special case of a Hilbert space with the inner product �· , ·� . We define first the Weak Greedy Algorithm (WGA) in Hilbert space H . We describe this algorithm for a general dictionary D . Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization WGA Let a sequence τ = { t k } ∞ k =1 , 0 ≤ t k ≤ 1, be given. WGA We define f τ 0 := f . Then for each m ≥ 1, we inductively define: 1 ϕ τ m ∈ D is any satisfying |� f τ m − 1 , ϕ τ |� f τ m �| ≥ t m sup m − 1 , g �| ; g ∈D Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization WGA Let a sequence τ = { t k } ∞ k =1 , 0 ≤ t k ≤ 1, be given. WGA We define f τ 0 := f . Then for each m ≥ 1, we inductively define: 1 ϕ τ m ∈ D is any satisfying |� f τ m − 1 , ϕ τ |� f τ m �| ≥ t m sup m − 1 , g �| ; g ∈D 2 f τ m := f τ m − 1 − � f τ m − 1 , ϕ τ m � ϕ τ m ; Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization WGA Let a sequence τ = { t k } ∞ k =1 , 0 ≤ t k ≤ 1, be given. WGA We define f τ 0 := f . Then for each m ≥ 1, we inductively define: 1 ϕ τ m ∈ D is any satisfying |� f τ m − 1 , ϕ τ |� f τ m �| ≥ t m sup m − 1 , g �| ; g ∈D 2 f τ m := f τ m − 1 − � f τ m − 1 , ϕ τ m � ϕ τ m ; m ( f , D ) := � m 3 G τ j =1 � f τ j − 1 , ϕ τ j � ϕ τ j . Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Historical comment In the case t k = 1, k = 1 , . . . the WGA is called Pure Greedy Algorithm (PGA). The PGA was proposed by J.H. Friedman and W. Stuetzle in 1981 for the ridge dictionary. We note that in a particular case t k = t , k = 1 , 2 , . . . , the WGA was considered by L. Jones (1987) (also for the ridge dictionary). The WGA provides for each f ∈ H an expansion into a series (greedy expansion) ∞ � c j ( f ) ϕ τ c j ( f ) := � f τ j − 1 , ϕ τ f ∼ j , j � . j =1 In general it is not an expansion into an orthogonal series but it has some similar properties. Vladimir Temlyakov From greedy approximation to greedy optimization
Introduction Greedy approximation in Hilbert spaces Greedy approximation in Banach spaces Greedy algorithms for convex optimization Parseval’s formula The coefficients c j ( f ) of an expansion are obtained by the Fourier formulas with f replaced by the residuals f τ j − 1 . It is easy to see that m � 2 = � f τ m − 1 � 2 − | c m ( f ) | 2 . � f τ There are convergence results for the greedy expansion and, therefore, from the above equality we get for this expansion an analog of the Parseval formula for orthogonal expansions: ∞ � f � 2 = � | c j ( f ) | 2 . j =1 Vladimir Temlyakov From greedy approximation to greedy optimization
Recommend
More recommend
