Computational Optimization Constrained Optimization Algorithms - PowerPoint PPT Presentation

Computational Optimization Constrained Optimization Algorithms

Same basic algorithms Repeat � Determine descent direction � Determine step size � Take a step Until Optimal But now must consider feasibility, e.g. Pick a feasible descent directions Choose step size to maintain feasibility Need dual multipliers to check optimality

Builds on Prior Approaches Unconstrained: Lots of approaches min f x ( ) Linear Equality Constrained: Convert to unconstrained min f x ( ) and solve = s t . . Ax b

Prior Approaches (cont) Linear Inequality Constrained: Identify active constraints min f x ( ) Solve subproblems ≥ s t . . Ax b Nonlinear Inequality Constrained: Linearize constraints min f x ( ) Solve subproblems g x ≥ s t . . ( ) 0

Feasible Direction Methods Use feasible descent directs. 1 + Consider min x Qx ' c x ' 2 = s t . . Ax b ˆ If we have feasible point x = + then all are feasible. ˆ x x Zp

Reduced Problem 1 ˆ + + + + ˆ ˆ min ( x Zp Q x )' ( Zp ) c x '( Zp ) p 2 � 1 1 + + + + ˆ ˆ ˆ ˆ min p Z QZp ' ' ( Qx c Zp )' xQx c x ' p 2 2 � 1 + + ˆ min p Z QZp ' ' ( Qx c Zp )' p 2 = Z QZ ' : Reduced Hessian + + = ˆ Z Q x '( ( Zp ) c ) : Reduced Gradient

Reduced Newton’s Equation For general problems ∇ = 2 Z ' f x Z ( ) : Reduced Hessian ∇ = Z ' f x ( ) : Reduced Gradient Reduced Newton Equations ∇ = ∇ 2 Z ' f x Z v ( ) - ' Z f x ( ) − ⇒ = − ∇ ∇ 2 1 v ( Z ' f x Z ( ) ) Z ' f x ( ) in reduced space − ⇒ = = − ∇ ∇ 2 1 p Zv Z Z ( ' f x Z ( ) ) Z ' f x ( ) in original spac e Then apply usual Newton’s Method

Reduced Steepest Descent Use reduced steepest descent direction = − ∇ v Z ' f x ( ) in reduced space ⇒ = = − ∇ d Zv ZZ ' f x ( ) in original space Convergence rate depends on condition number of reduced Hessian ∇ ≤ ∇ 2 2 Cond Z ( ' f x Z ( ) ) Cond Z Z Cond ( ' ) ( f x ( )) KEY POINT – Construct Z to have good conditioning!

Ways to Compute Z Projection Method (orthogonal and non-orthogonal) Variable Reduction QR Factorization What are they and how do they effect conditioning? How can you use them to compute multipliers?

Projection Method Find closest feasible point − = 1 2 min c x Ax 0 x 2 Compute a KKT point: This is optimal projection since problem is convex

Projection Method − = 1 min c x Ax 0 x 2 ⇒ λ = − + λ 1 L x ( , ) c x ( Ax )' 2 ( ) ∇ λ = − − + λ = L x ( , ) c x A ' 0 ⇒ x KKT are − = Ax b 0 − − λ = Ac Ax AA ' 0 ( ) − ⇒ λ = ⇒ λ = 1 AA ' Ac AA ' Ac ( ) − ⇒ = − 1 x c A ' AA ' Ac ( ) ⎡ − ⎤ ⇒ = − 1 x I A ' AA ' A c ⎣ ⎦ ( ) ⎡ ⎤ − = − 1 Projection Matrix P I A ' AA ' A ⎣ ⎦

Project method The matrix ( ) − = − 1 Z I A ' AA ' A is a basis for the null space matrix of A. Check ( ) ⎡ ⎤ − = − = − = 1 AZ A I A ' AA ' A A A 0 ⎣ ⎦

Get Lagrangian Multipliers for free! The matrix ( ) ( ) − − = 1 = 1 = A A ' AA ' where AA AA ' AA ' I r r is the right inverse matrix for A. For general problems = min ( ) . . f x s t Ax b λ = ∇ ' * A f x ( *) r

Let’s try it For ⎡ ⎤ = + + + 1 2 2 2 2 min f x ( ) x x x x ⎣ ⎦ 2 1 2 3 4 + + + = s t . . x x x x 1 1 2 3 4 Projection matrix − 1 ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 0 0 0 1 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 0 1 0 0 1 1 ⎢ ⎥ ( ) ⎢ ⎥ ⎢ ⎥ [ ] ⎢ ⎥ [ ] − = − 1 = − Z I A ' AA ' A 1 1 1 1 1 1 1 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 0 0 1 0 1 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ 0 0 0 1 1 ⎣ 1 ⎦ ⎡ ⎤ 3 1 1 1 − − − ⎢ ⎥ 4 4 4 4 1 3 1 1 ⎢ ⎥ − − − = ⎢ 4 4 4 4 ⎥ 1 1 3 1 − − − ⎢ ⎥ 4 4 4 4 ⎢ 1 1 1 3 ⎥ ⎣ ⎦ − − − 4 4 4 4

Solve FONC for Optimal Point FONC ⎡ ⎤ x ⎡ ⎤ 1 ⎢ ⎥ 1 ⎢ ⎥ ⎢ ⎥ x 1 ⎢ ⎥ ∇ − λ = = λ ⎢ ⎥ 2 f x ( ) A ' ⎢ ⎥ ⎢ ⎥ 1 x ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ 1 ⎢ ⎥ x ⎣ ⎦ 4 + + + = x x x x 1 1 2 3 4

Check Optimality Conditions ⎡ ⎤ ⎡ ⎤ 3 1 1 1 = 1 − − − x * [1111]/ 4 ⎢ ⎥ ⎢ ⎥ 4 4 4 4 For 1 3 1 1 1 ⎢ ⎥ ⎢ ⎥ − − − 1 ∇ = ∇ = = f x ( *) [1111]/ 4 4 4 4 4 Z f x ( *) * 0 ⎢ ⎥ ⎢ ⎥ 1 1 3 1 1 4 − − − ⎢ ⎥ ⎢ ⎥ 4 4 4 4 ⎢ ⎥ ⎢ 1 1 1 3 ⎥ ⎣ ⎦ 1 ⎣ ⎦ − − − 4 4 4 4 = Ax * b Using Lagrangian 1 ( ) − = 1 = A A ' AA ' [1111]' 4 r λ = ∇ = A f x ( *) 1/ 4 r Clearly ∇ = λ f x ( *) A '

You try it = 1 min f x ( ) x Cx ' 2 = s t . . Ax b For − − − ⎡ ⎤ 0 13 6 3 ⎢ ⎥ − − ⎡ ⎤ ⎡ ⎤ 13 23 9 3 2 1 2 1 2 ⎢ ⎥ = = = ⎢ ⎥ ⎢ ⎥ C A b ⎢ ⎥ − − − − ⎣ ⎦ ⎣ ⎦ 6 9 12 1 1 1 3 1 3 ⎢ ⎥ − ⎢ ⎥ ⎣ ⎦ 3 3 1 3 Find projection matrix Confirm optimality conds are Z’Cx*=0, Ax* = b Find x* Compute Lagrangian multipliers Check Lagrangian form of the multipliers.

Projected Gradient Method Algorithm for min f(x) s.t. Ax=b Let Z=I-A’(AA’) -1 A x 0 given g =gradf(x 0 ) While not optimal d=-Z’g x i+1 = x i+ α d (use linesearch) g =gradf(x i+1 ) i=i+1

Projected Gradient Method Equivalent to steepest descent on f(x+Zp) where x is some feasible point. So convergence rate depends on condition number of this problem The condition number is less than equal condition of Hessian of f * Compute Z’HZ for last example and check it’s conditioning:

Lagrangian Multipliers Each null space method yields a corresponding calculation of the right inverse. Thus each yields calculation of corresponding dual multipliers.

Variable Reduction Method Let A=[B N] A is m by n B is m by m assume m < n ⎡ − ⎤ − 1 B N = ⎢ ⎥ Z ⎣ ⎦ I is a basis matrix for null space of A ⎡ ⎤ ⎡ ⎤ − − 1 1 B B [ ] = ⇒ = = + ⎢ ⎥ ⎢ ⎥ A AA B N I 0 r r ⎣ ⎦ ⎣ ⎦ 0 0

Try on our example Take for example first two columns for B ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 2 1 2 1 2 1 2 1 [ ] = = = ⎢ ⎥ A ⎢ ⎥ B N ⎢ ⎥ ⎢ ⎥ − − ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ 1 1 3 1 ⎣ 1 1 3 1 ⎦ Then − − ⎡ ⎤ ⎡ ⎤ 1 2 1 1 ⎢ ⎥ ⎢ ⎥ − − 4 3 1 2 ⎢ ⎥ ⎢ ⎥ = = Z A ⎢ ⎥ ⎢ ⎥ r 1 0 0 0 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ 0 1 0 0 Condition number of Z’ CZ = 158 better but not great

QR Factorization Use Gram-Schmidt algorithm to make orthogonal factorize A’=QR with Q orthogonal and R upper triangular ⎡ ⎤ R [ ] = = 1 ⎢ ⎥ A ' QR Q Q 1 2 ⎣ ⎦ 0 ∈ × ∈ × ∈ × − ∈ × where A m n Q , n m Q , n ( n m R ), m m 1 2 1 = = Q R − T Z Q A 2 r 1 1

QR on problem Use matlab command QR [Q R ] = qr(A’) Q2 = Q(:,3:4) Cond(Q2’*C*Q2) = 9.79

Next Problem Consider Next Hardest Problem min f x ( ) ≥ s t . . Ax b How could we adapt gradient projection or other linear equality constrained problem to this problem?