Projection methods: convergence and counterexamples 4 January 2019 - PowerPoint PPT Presentation

Projection methods: convergence and counterexamples 4 January 2019 Hangzhou Dianzi University Vera Roshchina School of Mathematics and Statistics UNSW Sydney v.roshchina@unsw.edu.au Based on joint work with Hong-Kun Xu , Roberto Cominetti and Andrew Williamson .

The method of alternating projections C 2 C 1

The method of alternating projections Let H be a Hilbert space, with inner product �· , ·� and norm � · � . For any closed convex set C ⊆ H and any x ∈ H there exists a unique point P C ( x ) ∈ C such that � x − P C ( x ) � = inf y ∈ C � x − y � . Given two closed convex sets C 1 , C 2 ⊆ H and x 0 ∈ H , let x 1 = P C 1 ( x 0 ) , x 2 = P C 2 ( x 1 ) , x 3 = P C 1 ( x 2 ) , x 4 = P C 2 ( x 3 ) , . . . . . . x 2 k +1 = P C 1 x 2 k , x 2 k +2 = P C 2 x 2 k +1 , . . . . . .

Convergence Let M 1 and M 2 be closed affine subspaces of H , M = M 1 ∩ M 2 . Theorem 1 (von Neumann 1933) . For each x ∈ H n →∞ � ( P M 2 P M 1 ) n ( x ) − P M ( x ) � = 0 . lim von Neumann, Functional Operators-Vol. II. The Geometry of Orthogonal Spaces, Annals of Math. Studies, 1950 (reprint of 1933 lectures). Theorem 2 (Bregman 1965) . For C = C 1 ∩ C 2 � = ∅ , where C 1 , C 2 ⊆ H are closed convex sets, the sequence of alternating projections converges weakly to a point in C . Bregman, The method of successive projection for finding a common point of convex sets, Sov. Math. Dokl., 1965. The question of whether convergence is always strong remained open until 2004, despite many works on sufficient conditions.

Counterexample of Hundal Theorem 3 (Hundal 2004) . There exist a Hilbert space H , closed convex sets C 1 , C 2 ⊂ H with intersection C 1 ∩ C 2 = { 0 } and a starting point x 0 such that n →∞ � ( P C 2 P C 1 ) n ( x 0 ) � > 0 . lim In a separable Hilbert space with an orthonormal basis { e i } ∞ i =1 , let C 1 = { x | � x, e 1 � ≤ 0 } , C 2 = cone { p ( t ) | t ≥ 0 } , p ( t ) = e ⌊ t ⌋ +2 cos( f ( t )) + e ⌊ t ⌋ +3 sin( f ( t )) + e 1 h ( t ) , t ≥ 0 , f ( t ) = π h ( t ) = e − 100 t 3 2( t − ⌊ t ⌋ ) , Hundal, An alternating projection that does not converge in norm. Nonlinear Anal. 2004.

Rate of convergence x 0 x 0 x 0 x 0

Angles between subspaces The Friedrichs angle between two closed linear subspaces M 1 and M 2 is α ∈ [0 , π 2 ] such that ( B H is a unit ball, M = M 1 ∩ M 2 ) c = cos α = sup |� x, y �| . x ∈ M 1 ∩ M ⊥ ∩ B H y ∈ M 2 ∩ M ⊥ ∩ B H Theorem 4 (Aronszajn, 1950) . For each x ∈ H and n ≥ 1 � ( P M 2 P M 1 ) n ( x ) − P M ( x ) � ≤ c 2 n − 1 � x � . We have c < 1 iff M 1 + M 2 is closed; in this case the method of alternating projections converges linearly. Aronszajn, Theory of reproducing kernels, Trans. Amer. Math. Soc., 1950. The constant is the smallest possible Kayalar and Weinert, Error bounds for the method of alternating projections, Math. Control Signals Systems, 1988. Generalisations to several sets Reich and Zalas, The optimal error bound for the method of simultaneous projections, J. Approx. Theory, 2017

What if c = 1? Theorem 5 (Bauschke, Borwein and Lewis) . For two closed affine subspaces M 1 , M 2 ∈ H exactly one of the alternatives holds. (1) M 1 + M 2 is closed. Then for each x the alternating projections converge linearly to P M 1 ∩ M 2 ( x ) with a rate c 2 . (2) M 1 + M 2 is not closed. Then for any sequence of positive real numbers 1 > λ 1 ≥ λ 2 ≥ · · · ≥ λ n → 0 there exists a point x λ ∈ H such that � ( P M 2 P M 1 ) n ( x λ ) − P M ( x λ ) � ≥ λ n ∀ n ∈ N . Bauschke, Borwein, and Lewis, The method of cyclic projections for closed convex sets in Hilbert space, Contemporary Mathematics, 1997. Bauschke, Deutsch, Hundal, Characterizing arbitrarily slow convergence in the method of alternating projections. Int. Trans. Oper. Res., 2009.

Special properties and convergence Regularity and the existence of Slater points Gubin, Polyak, Raik, The method of projections for finding the common point of convex sets, USSR Comput. Math. Math. Phys., 1967. Symmetry Bruck, Reich, Nonexpansive projections and resolvents of accretive operators in Banach spaces, Houston J. Math., 1977. Reich, A limit theorem for projections, Linear and Multilinear Algebra, 1983. Semialgebraic structure Borwein, Li, Yao, Analysis of the convergence rate for the cyclic projection algo- rithm applied to basic semialgebraic convex sets. SIAM J. Optim. 24, 498–527 (2014) Drusvyatskiy, Li, Wolkowicz, A note on alternating projections for ill-posed semidef- inite feasibility problems. Math. Program. 162 (2017), 537–548.

What if the problem is infeasible? Assume that C 1 , C 2 ∈ H are convex and closed, but possibly C 1 ∩ C 2 = ∅ . Define the distance between C 1 and C 2 as � y − x � . dist( C 1 , C 2 ) = inf x ∈ C 1 y ∈ C 2 The following sets may be empty, P 1 = { x ∈ C 1 | dist( x, C 2 ) = dist( C 1 , C 2 ) } , P 2 = { y ∈ C 2 | dist( y, C 1 ) = dist( C 1 , C 2 ) } . C 2 C 2 P 2 v v P 1 C 1 C 1

The displacement vector and convergence Define the displacement vector v = P C 2 − C 1 (0) , where C 2 − C 1 is the Minkowski difference, C 2 − C 1 = { y − x, x ∈ C 1 , y ∈ C 2 } . For the alternating projections we have x 2 k − x 2 k +1 → v, x 2 k +2 − x 2 k +1 → v. If P 1 and P 2 are empty, then � x n � → ∞ . Otherwise x 2 k +1 ⇀ ¯ x ∈ P 1 , x 2 k ⇀ ¯ y ∈ P 2 , and ¯ y − ¯ x = v . Bauschke, Borwein, On the Convergence of yon Neumann’s Alternating Projec- tion Algorithm for Two Sets, Set-Valued Analysis, 1993.

A helpful illustration C 2 C 1

What about more than two sets? For m ≥ 2 sets we can generalise alternating projections starting from x 0 ∈ H , and projecting cyclically onto each of the sets. For three sets C 1 , C 2 , C 3 , x 1 = P C 1 ( x 0 ) , x 2 = P C 2 ( x 1 ) , x 3 = P C 3 ( x 2 ) , x 4 = P C 1 ( x 3 ) , · · · u 0 u 1 u 2 C 2 u 5 C 1 u 4 u 3 u 6 C 3

There is no variational characterisation Under mild assumptions (e.g. one of the sets is bounded) cyclic projections converge weakly either to a point in the intersection C 1 ∩ C 2 ∩ · · · ∩ C m or to a fixed cycle if the intersection is empty. Bruck, Reich, Nonexpansive projections and resolvents of accretive operators in Banach spaces. Houston J. Math., 1977. Recall that for two sets this cycle realises the distance between the sets; however, for m ≥ 3 there is no function Φ : H m → R such that for any collection of compact convex sets C 1 , C 2 , . . . , C m ⊂ H the limit cycles are precisely the solutions to the minimisation problem min Φ( x 1 , x 2 , . . . , x m ) . x i ∈ C i Baillon, Combettes, Cominetti, There is no variational characterization of the cy- cles in the method of periodic projections. J. Funct. Anal., 2012.

Under-relaxed projections Fix α ∈ (0 , 1] and instead of P C ( x ) consider R ( x ) = (1 − α ) x + αP C ( x ) . true u projection C under-relaxed projection This leads to under-relaxed alternating and cyclic projections.

Under-relaxed projections C 2 C 1 C 3 Iterations for α = 0 . 75 and α = 0 . 35 (shown in red).

Two special limits Fix α ∈ (0 , 1] and instead of P C ( x ) consider R ( x ) = (1 − α ) x + α ( P C ( x ) − x ) . The under-relaxed cyclic projections converge weakly to a fixed cy- cle iff such a cycle exists (e.g. when one of the sets is bounded). Bruck, Reich, Nonexpansive projections and resolvents of accretive operators in Banach spaces. Houston J. Math., 1977. Consider the limit of such α -cycles as α ↓ 0 , or alternatively vary α , letting α k ↓ 0 , � k ∈ N α k = + ∞ .

De Pierro’s conjecture Conjecture 1. The least squares solution m � x − x i � 2 � S = Arg min min x i ∈ C i x ∈H i =1 exists iff both limits exist and solve this least squares problem. De Pierro, From parallel to sequential projection methods and vice versa in convex feasibility: results and conjectures, Stud. Comput. Math., 2001. The conjecture is true for affine subspaces of R n , Censor, Eggermont, Gordon, Strong underrelaxation in Kaczmarz’s method for in- consistent systems. Numer. Math., 1983. closed affine subspaces satisfying a metric regularity condition, Bauschke, Edwards, A conjecture by De Pierro is true for translates of regular sub- spaces, J. Nonlinear Convex Anal., 2005. and sets satisfying a certain geometric condition. Baillon, Combettes, Cominetti, Asymptotic behavior of compositions of under- relaxed nonexpansive operators, J. Dyn. Games, 2014.

A misleading example C 1 = co { ( − 2 , 2 , 1) , ( − 2 , 2 , − 1) } , C 2 = co { (2 , 2 , 1) , (2 , 2 , − 1) } , C 3 = { ( x, y, z ) | x 2 + y 2 ≤ 1 , | z | ≤ 1 } , �� 0 , 5 � � S = : | z | ≤ 1 3 , z . C 3 C 3 u 0 z 0 =0.5 S C 1 u 0 z 0 =-0.5 C 1 S C 2 C 2 Under-relaxed projections for α = 0 . 5 and different starting points.

Counterexample C 1 = co { ( − 2 , 2 , 1) , ( − 2 , 2 , − 1) } , C 2 = co { (2 , 2 , 1) , (2 , 2 , − 1) } , p k = (cos t k , sin t k , ( − 1) k ) . C 3 = co { p k | k ∈ N } , Here { t k } is increasing, t 1 = π 4 and t k → π 2 as k → ∞ . p 2 p 4 C 1 C 3 p 3 p 1 C 2

Counterexample For this three-set system the limits described earlier do not exist, how- ever, the least-squares problem has a solution. p 2 p 4 C 1 C 3 p 3 p 1 C 2 Cominetti, Roshchina, Williamson, A counterexample to De Pierro’s conjecture on the convergence of under-relaxed cyclic projections, Optimization, 2018.