CS475/CM375 Lecture 8: Oct 6, 2011 Iterative Methods Reading: [Saad] - PDF document

05/10/2011 CS475/CM375 Lecture 8: Oct 6, 2011 Iterative Methods Reading: [Saad] Chapt 4 CS475/CM375 (c) 2011 P. Poupart & J. Wan 1 SOR Iteration • Successive Over Relaxation (SOR) ��� � �� � � � and � � � • Weighted average of � � ��� � � � � � ∑ � ∑ � �� � � � �� � � ��� � 1 � � � � � � ��� ��� � � � �� �� � � 1 SOR � ��������������� � � 1 �������������� � � 1 � • Select � � � � � � • For a suitably chosen � (>1), SOR can be much better than GS CS475/CM375 (c) 2011 P. Poupart & J. Wan 2 1

05/10/2011 Convergence Analysis • Q1: Under what conditions does the iteration converge? • Q2: If the iteration converges, how fast is it? • Def: � is an eigenvalue and � an eigenvector of � if �� � �� ( � � 0 ) • Def: the spectral radius of � , ���� , is the largest absolute value of the eigenvalues of � CS475/CM375 (c) 2011 P. Poupart & J. Wan 3 Convergence Analysis • Theorem: the iterative method � ��� � � � � � �� � � �� � is convergent for any � � and � if and only if � � � � �� � � 1 • � � � �� � is called the iteration matrix • ��� � � �� �� is called the rate of convergence CS475/CM375 (c) 2011 P. Poupart & J. Wan 4 2

05/10/2011 Minimization Formulation • Assume � is SPD. Consider the functional: � � � � �� � � � � � ∈ � � � � ≡ • Theorem: The solution of �� � � is equivalent to the solution of the minimization problem: min � ���� CS475/CM375 (c) 2011 P. Poupart & J. Wan 5 Minimization Formulation • Proof: the minimizer satisfies � � � � 0 i.e. �� �� � � 0 � � ∑ � �� � � � � � ∑ � � � � – Note � � � �� � �� �� � � – Hence CS475/CM375 (c) 2011 P. Poupart & J. Wan 6 3

05/10/2011 Minimization Formulation • Since ���� is convex, then local min = global min • Picture: CS475/CM375 (c) 2011 P. Poupart & J. Wan 7 Search Directions • Idea: minimize � � along the direction � � 0 • Let � � � current approximation • Define � ��� � � � � �� • Determine � by min ��� ��� � along � � ��� � � ��� – i.e., min CS475/CM375 (c) 2011 P. Poupart & J. Wan 8 4

05/10/2011 Search Directions • Let � � ≡ � � � � �� � • Hence 0 � � � � � and � � CS475/CM375 (c) 2011 P. Poupart & J. Wan 9 Search Directions • Notes � is SPD ⟹ � � �� � 0 1. 2. What is the optimal search direction � ? • Picture: CS475/CM375 (c) 2011 P. Poupart & J. Wan 10 5

05/10/2011 Steepest Descent • Local optimal direction • Consider � � � � � � �� � � � � �� � �� • Then � � 0 � � � � � � � � changes in � at � � in the direction � • Idea: make �′�0� as negative as possible by varying � CS475/CM375 (c) 2011 P. Poupart & J. Wan 11 Steepest Descent • Assume � � � 1 . Then � � � � �′�0� is max if � � � steepest descent � � � � � � � � �′�0� is min if � � � � steepest descent � � � � � � (where � � � � �� ) � � � CS475/CM375 (c) 2011 P. Poupart & J. Wan 12 6

05/10/2011 Steepest Descent • Steepest descent method: � ��� � � � � � � � � � � � step length � � � � � ( � � � � � � /� � �� ) • The optimal � � � � � � �� � • Also � ��� � CS475/CM375 (c) 2011 P. Poupart & J. Wan 13 Steepest Descent • Algorithm Given � � , compute � � � � � �� � for � � 0,1,2, … � � � � � � � � � � � �� � � ��� � � � � � � � � � ��� � � � � � � �� � end CS475/CM375 (c) 2011 P. Poupart & J. Wan 14 7

05/10/2011 Steepest Descent • Notes 1. Only 1 matrix ‐ vector product ��� � � per iteration 2. “Nonlinear” iterative method: � ��� � � � � � � �� � �� � � i.e., � � � � � � � � � CS475/CM375 (c) 2011 P. Poupart & J. Wan 15 8