Nonlinear Optimization Practical Advice for Non-linear Least Square Problems Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se December 10, 2007 c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Function call structure Problem independent Problem dependent 1 Calculate a starting approxima- tion x 0 . Repeat for k = 0 , . . . 2 Calculate residual F ( x k ) and Ja- 3 cobian J ( x k ) . 4 Calculate termination criteria. Calculate search direction p k . 5 Compute F ( x k + α k p k ) . 6 Calculate step length α k . 6 Calculate x k + 1 = x k + α k p k . 7 c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Toy problem: Circle fitting y i ] T , i = 1 , 2 , . . . , m , “find the circle p i = [˜ x i , ˜ Given a number of points ˜ that fits the points best in the least squares sense”. p 2 ˜ p 1 p 2 ˜ p 1 r ( c x , c y ) p 3 ˜ p 3 p 4 ˜ p 4 c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Objective function Step 1: Decide what to minimize. Minimize the squared sum of Euclidian distances between each mea- p i and the closest point p i on the circle, i.e. sured point ˜ m f ( x ) = 1 � p i − ˜ p i � 2 . � 2 i = 1 c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Models and parameters Step 2: Formulate a mathematical model of the object and determine the unknowns. Step 2a: Formulate a local model for each “term” of the least squares sum. A point ( x i , y i ) on a circle with center ( c x , c y ) and radius r is modelled � x i � c x as � cos θ i � � � G i ( x ) = + r = y i c y sin θ i for some “phase angle” θ i . Step 2b: Formulate a global model describing a vector with all terms of the least squares sum. G 1 ( x ) G 2 ( x ) G ( x ) = . . . . G m ( x ) c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Models and parameters Step 2c: Determine which parameters to put in the vector of unknowns, and in what order. c x c y r x = , θ 1 . . . θ m where c x , c y , r are the “global” parameters, corresponding to the circle we wish to find and the θ i are “local” parameters, corresponding to one point each. Step 2d: Implement the model function. Step 2e: Verify the model and implementation by calculating G ( x ) for realistic values of x . c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
The residual Step 3a: Implement the residual F ( x ) as “model minus data”. p 1 ˜ p 2 ˜ F ( x ; d ) = G ( x ) − d , where d = contain the “measurements”. . . . p m ˜ Step 3b: Reality check. Verify that F ( x ∗ ; G ( x ∗ )) = 0 . c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
The Jacobian Step 4a: Derive an analytical expression for the Jacobian. Use symbolic tools, e.g. Maple, if necessary. Step 4b: Implement the Jacobian. − r sin θ 1 1 0 cos θ 1 r cos θ 1 0 1 sin θ 1 . . . ... J ( x ; d ) = . . . . . . . − r sin θ m 1 0 cos θ m r cos θ m 0 1 sin θ m Step 4c: Compare the implemented Jacobian with a numerical approximation. � F ( x + ǫ 1 ) − F ( x − ǫ 1 ) F ( x + ǫ n ) − F ( x − ǫ n ) � J ( x ; d ) ≈ , . . . 2 ǫ 2 ǫ � T , ǫ n = � T . � � ǫ 1 = ǫ 0 . . . 0 0 . . . 0 ǫ Step 4d: Verify that J ( x ) has full rank. c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Convergence check on synthetic, error-free data Step 5: Tests on perfect data. Use a realistic x ∗ and generate “measurements” without any errors, i.e. d = G ( x ∗ ) . Step 5a: Call the optimization method with x 0 = x ∗ and verify that it returns x ∗ as the solution after maximum 1 iteration. Step 5b: Generate starting approximations x 0 as pertubations of the true solution x ∗ . Verify convergence from a reasonable large region. c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Pertubation sensitivity of solution Step 5: Tests on data with known errors. Step 5a: Use a realistic x ∗ and generate “measurements” with an added measurement error, i.e. d = G ( x ∗ ) + ε, ε ∈ N ( 0 , σ 2 ) . where σ 2 is chosen to give errors of a reasonable size. Step 5b: Solve the optimization problem with x 0 = x ∗ . Call the x . solution to the new problem ˆ x with the Step 5c: Compare the solution of the perturbed problem ˆ solution of the original problem x ∗ . x from x ∗ . Step 5d: Repeat steps 5a-5c and analyze the deviation of ˆ Step 5e: Repeat steps 5a-5d with x 0 � = x ∗ , i.e. perturb the starting approximation also. c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Starting approximation calculation Step 6a: Construct a function for the starting approximation that (optimally) only relies on measurements. y i − c y 0 y i − c y 0 ) 2 , θ 0 i = tan − 1 ˜ � c x 0 = ˜ x i , c y 0 = ˜ y i , r 0 = x i − c x 0 ) 2 + (˜ (˜ x i − c x 0 ˜ Step 6b: Check the quality of the starting approximating function on error-free data. Step 6b.1: Generate realistic measurements d without any errors. Step 6b.2: Use the function in Step 6a to determine x 0 from d and x . solve the optimization problem. Call the solution ˆ x with x ∗ . Step 6b.3: Compare ˆ Step 6c: Repeat steps 6b.1 to 6b.3 on data with realistic errors. c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Sensitivity analysis “Truth” “Real world” 1 Start with a “true” parameter vector x ∗ = [ c x , c y , r , θ 1 , . . . , θ m ] . Calculate points p i on the circle. 2 3 Generate simulated measure- p i = p i + ǫ i , ǫ i ∈ N ( 0 , σ 2 ) . ments ˜ 4 Construct a starting approxima- tion x 0 from the measurements. 5 Solve the parameter estimation x . problem. Call the solution ˆ x for re- Study the deviation of ˆ 6a peated simulations. Determine precision, repeatability. 6b Compare the true parameter vector x ∗ with the estimated ˆ x . Determine accuracy. c � 2007 Niclas Börlin, CS, UmU Practical Advice for Non-linear Least Square Problems
Recommend
More recommend
