Sparse and TV Kaczmarz solvers and the linearized Bregman method Dirk Lorenz, Frank Schöpfer, Stephan Wenger, Marcus Magnor, March, 2014 Sparse Tomo Days, DTU
Motivation Split feasibility problems Sparse Kaczmarz and TV-Kaczmarz Application to radio interferometry March, 2014 Dirk Lorenz Linearized Bregman Page 2 of 25
Motivation Split feasibility problems Sparse Kaczmarz and TV-Kaczmarz Application to radio interferometry March, 2014 Dirk Lorenz Linearized Bregman Page 3 of 25
Underdetermined systems Seeking solutions of linear systems Ax = b . Kaczmarz iteration: a T r ( k ) x k − b r ( k ) x k + 1 = x k − a r ( k ) � a r ( k ) � 2 2 a T r : r -th row of A , r ( k ) : control sequence. Amounts to iterative projection onto hyperplane defined by r ( k ) -th equation. When initialized with 0: Converges to solution of min � x � 2 2 such that Ax = b . March, 2014 Dirk Lorenz Linearized Bregman Page 4 of 25
Aiming at sparse solutions? March, 2014 Dirk Lorenz Linearized Bregman Page 5 of 25
Aiming at sparse solutions? Iterate a T r ( k ) x k − b r ( k ) x k + 1 = x k − a r ( k ) � a r ( k ) � 2 2 March, 2014 Dirk Lorenz Linearized Bregman Page 5 of 25
Aiming at sparse solutions? Iterate a T r ( k ) x k − b r ( k ) S λ z k + 1 = z k − a r ( k ) � a r ( k ) � 2 2 x k + 1 = S λ ( z k + 1 ) − λ λ March, 2014 Dirk Lorenz Linearized Bregman Page 5 of 25
Aiming at sparse solutions? Iterate a T r ( k ) x k − b r ( k ) S λ z k + 1 = z k − a r ( k ) � a r ( k ) � 2 2 x k + 1 = S λ ( z k + 1 ) − λ λ Theorem [L, Schöpfer, Wenger, Magnor 2014]: The sequence x k , when initialized with x 0 = 0, converges to the solution of min λ �·� 1 + 1 2 �·� 2 2 such that Ax = b . March, 2014 Dirk Lorenz Linearized Bregman Page 5 of 25
Aiming at sparse solutions? Iterate a T r ( k ) x k − b r ( k ) S λ z k + 1 = z k − a r ( k ) � a r ( k ) � 2 2 x k + 1 = S λ ( z k + 1 ) − λ λ Theorem [L, Schöpfer, Wenger, Magnor 2014]: The sequence x k , when initialized with x 0 = 0, converges to the solution of min λ �·� 1 + 1 2 �·� 2 2 such that Ax = b . Two interesting things: 1. Very similar to Kaczmarz. Other “minimum- J -solutions” possible? 2. Very similar to linearized Bregman iteration (replace first equation by z k + 1 = z k − t k A T ( Ax k − b ) ) March, 2014 Dirk Lorenz Linearized Bregman Page 5 of 25
Aiming at sparse solutions? Iterate a T r ( k ) x k − b r ( k ) S λ z k + 1 = z k − a r ( k ) � a r ( k ) � 2 2 x k + 1 = S λ ( z k + 1 ) − λ λ Theorem [L, Schöpfer, Wenger, Magnor 2014]: The sequence x k , when initialized with x 0 = 0, converges to the solution of min λ �·� 1 + 1 2 �·� 2 2 such that Ax = b . Two interesting things: 1. Very similar to Kaczmarz. Other “minimum- J -solutions” possible? 2. Very similar to linearized Bregman iteration (replace first equation by z k + 1 = z k − t k A T ( Ax k − b ) ) Approach: “Split feasibility problems” will answer the first and explain the second point. In a nutshell: Adapt the notion of “projection” to new objective. March, 2014 Dirk Lorenz Linearized Bregman Page 5 of 25
Motivation Split feasibility problems Sparse Kaczmarz and TV-Kaczmarz Application to radio interferometry March, 2014 Dirk Lorenz Linearized Bregman Page 6 of 25
Convex and split feasibility problems Convex feasibility problem (CFP): Find x , such that x ∈ C i , i = 1 , . . . N C C i convex , projecting onto C i “easy” March, 2014 Dirk Lorenz Linearized Bregman Page 7 of 25
Convex and split feasibility problems Split feasibility problem (SFP): Find x , such that x ∈ C i , i = 1 , . . . N C , A i x ∈ Q i , i = 1 , . . . , N Q C i , Q i convex, A i linear, projecting onto C i and Q i “easy” Constraints “split into two types” March, 2014 Dirk Lorenz Linearized Bregman Page 7 of 25
Convex and split feasibility problems Split feasibility problem (SFP): Find x , such that x ∈ C i , i = 1 , . . . N C , A i x ∈ Q i , i = 1 , . . . , N Q C i , Q i convex, A i linear, projecting onto C i and Q i “easy” Constraints “split into two types” Alternating projections: x k + 1 = P C i ( x k ) i = ( k mod N C ) + 1 “control sequence” March, 2014 Dirk Lorenz Linearized Bregman Page 7 of 25
Convex and split feasibility problems Split feasibility problem (SFP): Find x , such that x ∈ C i , i = 1 , . . . N C , A i x ∈ Q i , i = 1 , . . . , N Q C i , Q i convex, A i linear, projecting onto C i and Q i “easy” Constraints “split into two types” Alternating projections: x k + 1 = P C i ( x k ) i = ( k mod N C ) + 1 “control sequence” [1933 von Neumann (two subspaces), 1962 Halperin (several subspaces), Dijkstra, Censor, Bauschke, Borwein, Deutsch, Lewis, Luke…] March, 2014 Dirk Lorenz Linearized Bregman Page 7 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } x k + 1 = P C i ( x k ) for a constraint u ∈ C i x k + 1 = P H k ( x k ) for a constraint A i x ∈ Q i March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } x k + 1 = P C i ( x k ) for a constraint u ∈ C i x k + 1 = P H k ( x k ) for a constraint A i x ∈ Q i March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } x k + 1 = P C i ( x k ) for a constraint u ∈ C i x k + 1 = P H k ( x k ) for a constraint A i x ∈ Q i March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } x k + 1 = P C i ( x k ) for a constraint u ∈ C i x k + 1 = P H k ( x k ) for a constraint A i x ∈ Q i March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } x k + 1 = P C i ( x k ) for a constraint u ∈ C i x k + 1 = P H k ( x k ) for a constraint A i x ∈ Q i Converges to feasible point. March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Tackling split feasibility problems Projecting onto { x | Ax ∈ Q } too expensive Construct a separating hyperplane: For a given x k : Set w k = Ax k − P Q ( Ax k ) Project onto H k = { x | � A T w k , x � ≤ � A T w k , x k � − � w k � 2 } x k + 1 = P C i ( x k ) for a constraint u ∈ C i x k + 1 = P H k ( x k ) for a constraint A i x ∈ Q i Converges to feasible point. E.g.: Q = { b } : x k + 1 = x k + t k A T ( Ax k − b ) � minimum norm solution of Ax = b March, 2014 Dirk Lorenz Linearized Bregman Page 8 of 25
Towards sparse solutions with generalized projections D : X × X → R abstract “distance function” P C ( x ) = argmin y ∈ C D ( x , y ) March, 2014 Dirk Lorenz Linearized Bregman Page 9 of 25
Towards sparse solutions with generalized projections D : X × X → R abstract “distance function” P C ( x ) = argmin y ∈ C D ( x , y ) D ( x , y ) = � x − y � 2 � orthogonal projection March, 2014 Dirk Lorenz Linearized Bregman Page 9 of 25
Towards sparse solutions with generalized projections D : X × X → R abstract “distance function” P C ( x ) = argmin y ∈ C D ( x , y ) D ( x , y ) = � x − y � 2 � orthogonal projection J : X → R convex, z ∈ ∂ J ( x ) D z ( x , y ) = J ( y ) − J ( x ) − � z , y − x � Bregman distance � Bregman projection March, 2014 Dirk Lorenz Linearized Bregman Page 9 of 25
Towards sparse solutions with generalized projections D : X × X → R abstract “distance function” P C ( x ) = argmin y ∈ C D ( x , y ) D ( x , y ) = � x − y � 2 � orthogonal projection J : X → R convex, z ∈ ∂ J ( x ) D z ( x , y ) = J ( y ) − J ( x ) − � z , y − x � Bregman distance � Bregman projection J : R n → R continuous, α -strongly convex ( = ⇒ ∇ J ∗ is 1 / α -Lipschitz) March, 2014 Dirk Lorenz Linearized Bregman Page 9 of 25
Recommend
More recommend
