Line search via interpolation and SDP Etienne de Klerk , Gamal - PowerPoint PPT Presentation

Line search via interpolation and SDP Etienne de Klerk † , Gamal Elabwabi, Dick den Hertog † Tilburg University Workshop Advances in Continuous Optimization, Iceland, June, 2006 Line search via interpolation and SDP – p. 1/21

Outline Lagrange interpolation – an overview; Minimizing Lagrange interpolants using SDP; Extracting minimizers; Numerical results; Discussion. Line search via interpolation and SDP – p. 2/21

Basic idea Line search problem for given f : [ − 1 , 1] → I R: x ∈ [ − 1 , 1] f ( x ) min Approximate f by its Lagrange interpolant using the Chebyshev nodes of the first kind; Minimize the interpolant using semidefinite programming (SDP); Extract global minimizers of the interpolant. Line search via interpolation and SDP – p. 3/21

Lagrange interpolation The Lagrange interpolating polynomial of degree n of a function f : [ − 1 , 1] → I R, say L n ( f ) with respect to given nodes x i ∈ [ − 1 , 1] ( i = 0 , . . . , n ) , is the unique polynomial given by: n � L n ( f )( x ) := f ( x i ) l n,i ( x ) , i =0 where x − x j � l n,i ( x ) := x i − x j j � = i is called the fundamental Lagrange polynomial . NB: l n,i ( x j ) = δ ij . Line search via interpolation and SDP – p. 4/21

Remainder formula Suppose that f ∈ C n +1 [ − 1 , 1] and let L n ( f ) be given as before. Then for any x ∈ [ − 1 , 1] , one has n ( x − x i ) f ( n +1) ( ζ ( x )) � f ( x ) − L n ( f )( x ) = for some ζ ( x ) ∈ [ − 1 , 1] ( n + 1)! i =0 � f ( n +1) � ∞ , [ − 1 , 1] ≤ , 2 n ( n + 1)! if � (2 i + 1) π � x i = cos i = 0 , . . . , n, 2( n + 1) i.e. x i ’s ≡ Chebyshev nodes of the first kind . Line search via interpolation and SDP – p. 5/21

Interpolation error for analytic f If f is analytic on and inside an ellipse E in the complex plane with foci ± 1 and axis lengths 2 L and 2 l , then � � ( l + L ) − n � f − L n ( f ) � ∞ , [ − 1 , 1] ≤ max z ∈E | f ( z ) | 1 Example: if f ( x ) = 1+ x 2 then the poles ± i determine that l = 1 √ √ 2) − n ) . and therefore L = 2 . Convergence rate: O ((1 + If f is analytic (holomorphic) on all of C , then even faster � for all c > 0 . � 1 convergence: o c n Line search via interpolation and SDP – p. 6/21

Summary: convergence rates � f − L n ( f ) � ∞ , [ − 1 , 1] type f � � log n f ∈ C r [ − 1 , 1] O n r � � 1 f analytic on E ⊇ [ − 1 , 1] O ( l + L ) n � 1 � ∀ c > 0 f holomorphic on C o c n Consequently, one has x ∈ [ − 1 , 1] L n ( f ) → f ≡ min x ∈ [ − 1 , 1] f as n → ∞ min with the same rates of convergence as in the table. Line search via interpolation and SDP – p. 7/21

Minimizing an interpolant via SDP Let (an interpolant) p ∈ I R [ x ] (real univariate polynomial) be given. := x ∈ [ − 1 , 1] p ( x ) min p = max { τ : p ( x ) − τ ≥ 0 , ∀ x ∈ [ a, b ] } . Idea by Nesterov: rewrite the nonnegativity condition as a LMI using an old theorem by Lucács. Yu. Nesterov, Squared functional systems and optimization problems. In H. Frenk, K. Roos, T. Terlaky and S. Zhang (eds.), High Performance Optimization . Dordrecht, Kluwer Academic Press, 405–440, 2000. Line search via interpolation and SDP – p. 8/21

Lucács theorem Let p be of degree 2 m . Then p is nonnegative on an interval [ a, b ] iff: p ( x ) = q 2 1 ( x ) + ( x − a )( b − x ) q 2 2 ( x ) , for some polynomials q 1 of degree m and q 2 of degree m − 1 . Moreover, if the degree of p is 2 m + 1 then p is nonnegative on [ a, b ] if and only if it has the following representation: p ( x ) = ( x − a ) q 2 1 ( x ) + ( b − x ) q 2 2 ( x ) , for some polynomials q 1 and q 2 of degree m . Line search via interpolation and SDP – p. 9/21

Gram matrix representation If q is sum-of-squares polynomial of degree 2 m , then q ( x ) = B m ( x ) T XB m ( x ) for some matrix X � 0 of order m + 1 , where B m is a basis for polynomials of degree ≤ m , like B m ( x ) T = [1 x x 2 . . . x m ] . For stable computation we use instead the Chebyshev basis : B m ( x ) T = [ T 0 ( x ) T 1 ( x ) T 2 ( x ) . . . T m ( x )] , where T j ( x ) := cos( j arccos x ) ( j = 0 , 1 , . . . ) are the Chebyshev polynomials . Line search via interpolation and SDP – p. 10/21

Sampling approach Assume degree ( p ) = 2 m . Using the Gram matrix representation and the Lucács theorem, the condition p ( x ) − τ ≥ 0 , ∀ x ∈ [ a, b ] can now be reformulated as: p ( x ) − τ = q 1 ( x ) + ( x − a )( b − x ) q 2 ( x ) , q 1 , q 2 are sums-of-squares B m ( x ) T X 1 B m ( x ) + ( x − a )( b − x ) B m − 1 ( x ) T X 2 B m − 1 ( x ) , = where X 1 , X 2 are positive semidefinite matrices. We rewrite this as an SDP using the ‘sampling’ approach by Löfberg-Parrilo. Basic idea: two univariate polynomials of degree at most 2 m are identical iff their function values coincide at 2 m distinct points. Line search via interpolation and SDP – p. 11/21

Sampling approach (ctd.) We had p ( x ) − τ = B m ( x ) T X 1 B m ( x )+( x − a )( b − x ) B m − 1 ( x ) T X 2 B m − 1 ( x ) . Since degree ( p ) = 2 m this is the same as requiring p ( x i ) − τ = B m ( x i ) T X 1 B m ( x i )+( x i − a )( b − x i ) B m − 1 ( x i ) T X 2 B m − 1 ( x i ) for each interpolation node x i ( i = 0 , . . . , 2 m ) . J. Löfberg and P. Parrilo, From coefficients to samples: a new approach to SOS optimization, IEEE Conference on Decision and Control , December 2004. Line search via interpolation and SDP – p. 12/21

Final SDP formulation Since p interpolates f at the x i ’s, i.e. p ( x ) = L 2 m ( f )( x ) , we may replace p ( x i ) by f ( x i ) . x ∈ [ a,b ] L 2 m ( f )( x ) = max min τ,X 1 ,X 2 τ subject to f ( x i ) − τ = B m ( x i ) T X 1 B m ( x i )+( x i − a )( b − x i ) B m − 1 ( x i ) T X 2 B m − 1 ( x i ) , for ( i = 0 , . . . , 2 m ) , where X 1 � 0 , X 2 � 0 , the x i ’s are the Chebychev nodes, and B m the Chebychev basis. Line search via interpolation and SDP – p. 13/21

Advantages of the SDP formulation We do not have to compute the coefficients of the Lagrange interpolant; The constraints of the SDP problems involve only rank one matrices , and this is exploited by the SDP solver DSDP; Extracting a global minimizer of the interpolant using the method by Lasserre-Henrion and Laurent-Jibetean only involves an eigenvalue problem of order the number of global minimizers of the interpolant (very small in practice); D. Henrion and J.B. Lasserre. Detecting global optimality and extracting solutions in GloptiPoly. In D. Henrion, A. Garulli (eds), Positive Polynomials in Control, LNCIS, 312 , Springer Verlag, 2005. D. Jibetean and M. Laurent. Semidefinite approximations for global unconstrained polynomial optimization. SIOPT , 16 (2), 490–514, 2005. Line search via interpolation and SDP – p. 14/21

P 5 Test functions Function f [ a, b ] max x ∈ [ a,b ] f ( x ) Global maximizer(s) 6 x 6 + 52 25 x 5 − 39 − 1 80 x 4 1 [-1.5,11] 29,763.233 10 10 x 3 + 79 20 x 2 + x − − 71 1 10 − sin x − sin 10 3 x 2 [2.7,7.5] 1.899599 5.145735 -6.7745761 P 5 k =1 k sin(( k + 1) x + k ) 3 [-10,10] 12.03124 -0.491391 5.791785 (16 x 2 − 24 x + 5) e − x 4 [1.9,3.9] 3.85045 2.868034 ( − 3 x + 1 . 4) sin 18 x 5 [0,1.2] 1.48907 0.96609 ( x + sin x ) e − x 2 6 [-10,10] 0.824239 0.67956 − sin x − sin 10 3 x 7 [2.7,7.5] 1.6013 5.19978 − ln x + 0 . 84 x − 3 - 7.083506 k =1 k cos(( k + 1) x + k ) 8 [-10,10] 14.508 -0.800321 5.48286 Line search via interpolation and SDP – p. 15/21

Test functions (ctd.) Function f [ a, b ] max x ∈ [ a,b ] f ( x ) Global maximizer(s) − sin x − sin 2 3 x 9 [3.1,20.4] 1.90596 17.039 x sin x 10 [0,10] 7.91673 7.9787 2.09439 8 2 cos x + cos 2 x 11 [-1.57,6.28] 1.5 4.18879 < π : − sin 3 x − cos 3 x 12 [0,6.28] 1 4.712389 √ x 2 / 3 + (1 − x 2 ) 1 / 3 1 / 2 13 [0.001,0.99] 1.5874 e − x sin 2 πx 14 [0,4] 0.788685 0.224885 ( − x 2 + 5 x − 6) / ( x 2 + 1) 15 [-5,5] 0.03553 2.41422 − 2( x − 3) 2 − e − x 2 / 2 16 [-3,3] -0.0111090 3 -3 − x 6 + 15 x 4 − 27 x 2 − 250 17 [-4,4] -7 3 − ( x − 2) 2 , x ≤ 3 ; 18 [0,6] 0 2 − 2 ln( x − 2) − 1 , otherwise. Line search via interpolation and SDP – p. 16/21

Test functions (ctd.) Function f [ a, b ] max x ∈ [ a,b ] f ( x ) Global maximizer(s) sin 3 x − x − 1 19 [0,6.5] 7.81567 5.87287 ( x − sin x ) e − x 2 20 [-10,10] 0.0634905 1.195137 Posed as maximization problems. P. Hansen, B. Jaumard, and S-H. Lu. Global optimization of univariate Lipschitz functions : II. New algorithms and computational comparisons. Mathematical Programming , 55 , 273-292, 1992. Line search via interpolation and SDP – p. 17/21

Relative errors For the 20 test functions, we used DSDP to compute the relative errors: | ¯ f − ¯ L n ( f ) | 1 + | ¯ f | in the table below, where ¯ f := max x ∈ [ a,b ] f ( x ) , etc. Number of Chebyshev nodes: n = 20 , 30 , . . . , 90 . f \ n 20 30 40 50 60 70 80 90 1 4.18e-9 2.29e-10 5.30e-9 7.45e-9 5.79e-9 2.44e-9 1.02e-8 1.04e-8 2 1.01e-7 1.06e-7 1.18e-7 1.16e-7 1.18e-7 1.18e-7 1.18e-7 1.19e-7 3 1.29e-2 7.50e-2 4.76e-2 5.52e-2 1.53e-2 8.50e-5 3.15e-8 4.94e-8 4 1.40e-7 1.34e-7 1.40e-7 1.39e-7 1.43e-7 1.43e-7 1.35e-7 1.43e-7 5 2.80e-5 5.38e-8 1.01e-6 1.01e-6 1.01e-6 1.01e-6 1.01e-6 1.01e-6 6 2.98e-1 7.54e-2 5.70e-3 1.01e-3 2.15e-4 1.45e-5 6.04e-7 2.16e-7 Line search via interpolation and SDP – p. 18/21