Optimality and Support Projection Algorithm for Sparsity Constrained - PowerPoint PPT Presentation

Optimality and Support Projection Algorithm for Sparsity Constrained Minimization Lili Pan †‡ , Naihua Xiu † , Shenglong Zhou † † Department of Applied Mathematics, Beijing Jiaotong University ‡ School of Science, Shandong University of Technology Sept 2014

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Outline Introduction 1 Optimality Conditions (I) 2 Optimality Conditions (II) 3 Gradient Support Projection Algorithms 4 Numerical Experiments 5 Summary 6 L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 2 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Introduction Introduction In this talk, we mainly consider the nonlinear minimization with sparse and nonnegative constraints. By discussing tangent cone and normal cone of sparse constraint, we give the first necessary optimality conditions, α -Stability, T-Stability and N-Stability, and the second necessary and sufficient optimality conditions for the nonlinear problem. By adopting Armijo-type stepsize rule, we present a gradient support projection algorithmic framework for the problem and establish its full convergence and computational complexity under mild conditions. By doing some numerical experiments, we show the excellent performance of the new algorithm for the least squares without and with noise. L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 3 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Introduction Introduction Model Representation Sparsity and Nonnegativity Constrained Nonlinear Optimization min f ( x ) , s.t. � x � 0 ≤ s , x ≥ 0 . (1) where f ( x ) : R N → R is a continuously differentiable or twice differentiable function, � x � 0 is the l 0 -norm of x . The special case of problem (1) min � Ax − b � 2 s . t . � x � 0 ≤ s , x ≥ 0 , (2) where A ∈ R M × N , b ∈ R M , s < M < N and � · � is l 2 -norm. L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 4 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Model (I) Introduction We study the first and second order optimality conditions of the following model min f ( x ) , s.t. � x � 0 ≤ s . (3) Let S � { x ∈ R N | � x � 0 ≤ s } . Support Projection y ∈ R N | y i = x i , i ∈ I s ( x ); y i = 0 , i / � � P S ( x ) = ∈ I s ( x ) . where I s ( x ) := { j 1 , j 2 , · · · , j s } ⊆ { 1 , 2 , · · · , N } of indices of x with i ∈ I s ( x ) | x i | ≥ max min ∈ I s ( x ) | x i | . i / L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 5 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Tangent Cone and Normal Cone Optimality Conditions (I) Definition of Bouligand Tangent Cone For any nonempty set Ω ⊆ R N , its Bouligand Tangent Cone T B Ω ( x ) , and corresponding Normal Cone N B Ω ( x ) at point x ∈ Ω are defined as: k →∞ x k = x , λ k ≥ 0 , k = 1 , � � ∃ { x k } ⊂ Ω , lim � � T B d ∈ R N Ω ( x ) := , � k →∞ λ k ( x k − x ) = d 2 , · · · , such that lim � � d ∈ R N | � d , z � ≤ 0 , ∀ z ∈ T B N B Ω ( x ) := � Ω ( x ) � , L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 6 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Tangent Cone and Normal Cone Optimality Conditions (I) Definition of Clarke Tangent Cone The Clarke Tangent Cone T C Ω ( x ) and corresponding Normal Cone N C Ω ( x ) at point x ∈ Ω are defined as: k →∞ x k = x ,  ∀ { x k } ⊂ Ω , ∀ { λ k } ⊂ R + with lim  �    �  k →∞ y k = d T C d ∈ R N lim k →∞ λ k = 0 , ∃ { y k } such that lim Ω ( x ) := , � � and x k + λ k y k ∈ Ω , k ∈ N  �    d ∈ R N | � d , z � ≤ 0 , ∀ z ∈ T C N C Ω ( x ) := � Ω ( x ) � . L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 7 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Tangent Cone and Normal Cone Optimality Conditions (I) Bouligand Tangent Cone of Sparse Set Theorem For any x ∈ S and letting Γ = supp ( x ) , the Bouligand tangent cone and corresponding normal cone of S at x are � T B S ( x ) = span { e i , i ∈ Υ ⊇ Γ , | Υ | ≤ s } (4) Υ � span { e i , i / ∈ Γ } , if | Γ | = s N B S ( x ) = (5) { 0 } , if | Γ | < s where e i ∈ R N is a vector whose the ith component is one and others are zeros, span { e i , i ∈ Γ } denotes the subspace of R N spanned by { e i , i ∈ Γ } , and supp ( x ) = { i ∈ { 1 , · · · , N } | x i � = 0 } . L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 8 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary Tangent Cone and Normal Cone Optimality Conditions (I) Clarke Tangent Cone of Sparse Set Theorem For any x ∈ S and letting Γ = supp ( x ) , then the Clarke tangent cone and corresponding normal cone of S at x are { d ∈ R N | supp ( d ) ⊆ Γ } = span { e i , T C S ( x ) = i ∈ Γ } (6) N C S ( x ) = span { e i , i / ∈ Γ } . (7) L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 9 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary α -Stability, N -Stability and T -Stability Optimality Conditions (I) α -Stability, N -Stability and T -Stability Definition For real number α > 0, a vector x ∗ ∈ S is called an α -stationary point, N ♯ -stationary point and T ♯ -stationary point of ( 3 ) if it respectively satisfies the relation P S ( x ∗ − α ∇ f ( x ∗ )) , x ∗ α − stationary point: ∈ (8) N ♯ − stationary point: ∇ f ( x ∗ ) + N ♯ S ( x ∗ ) , 0 ∈ (9) T ♯ − stationary point: �∇ ♯ S f ( x ∗ ) � , 0 = (10) where ∇ ♯ S f ( x ∗ ) = arg min { � x + ∇ f ( x ∗ ) � | x ∈ T ♯ S ( x ∗ ) } , ♯ ∈ { B , C } stands for the sense of Bouligand tangent cone or Clarke tangent cone. L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 10 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary α -Stability, N -Stability and T -Stability Optimality Conditions (I) Relationship of the Three Kinds of Stability Theorem Under the concept of Bouligand tangent cone, for model ( 3 ) and α > 0 , if the vector x ∗ ∈ S satisfies � x ∗ � 0 = s, then ⇒ N B − stationary point ⇐ ⇒ T B − stationary point ; α − stationary point = if the vector x ∗ ∈ S satisfies � x ∗ � 0 < s, then ⇒ N B − stationary point ⇐ ⇒ T B − stationary point α − stationary point ⇐ ⇒ ∇ f ( x ∗ ) = 0 . ⇐ L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 11 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary α -Stability, N -Stability and T -Stability Optimality Conditions (I) Relationship of the Three Kinds of Stability � x ∗ � 0 = s � x ∗ � 0 < s  α – stationary point = 0 , i ∈ Γ  | ( ∇ f ( x ∗ )) i | ∇ f ( x ∗ ) = 0 x ∗ ∈ P S ( x ∗ − α ∇ f ( x ∗ )) 1 ≤ α M s ( | x ∗ | ) , i / ∈ Γ ,   N B – stationary point = 0 , i ∈ Γ  ( ∇ f ( x ∗ )) i ∇ f ( x ∗ ) = 0 −∇ f ( x ∗ ) ∈ N B S ( x ∗ ) ∈ R , i / ∈ Γ ,   T B – stationary point = 0 , i ∈ Γ  ( ∇ f ( x ∗ )) i ∇ f ( x ∗ ) = 0 ∇ B S f ( x ∗ ) = 0 ∈ R , i / ∈ Γ ,  L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 12 / 38

Introduction Optimality Conditions (I) Optimality Conditions (II) Gradient Support Projection Algorithms Numerical Experiments Summary α -Stability, N -Stability and T -Stability Optimality Conditions (I) Relationship of the Three Kinds of Stability Theorem Under the concept of Clarke tangent cone, we consider the problem ( 3 ) . For α > 0 , if x ∗ ∈ S then ⇒ N C − stationary point ⇐ ⇒ T C − stationary point . α − stationary point = L Pan, S Zhou, N Xiu Optimality and Support Projection Algorithm for Sparsity Constrained Minimization May 2014 13 / 38