Exact Scalable Sensitivity Analysis for the Next Release Problem Mark Harman ∗ Jens Krinke ∗ Inmaculada Medina-Bulo † Francisco Palomo-Lozano † Jian Ren ‡ Shin Yoo ∗ ∗ CREST, University College London (UK) † UCASE, Universidad de Cádiz (Spain) ‡ Beihang University (China) Talk at the Best of RESG Research 2014 British Computer Society
The Next Release Problem Companies constantly release new products or versions 1 Requirements for a release can be of different kinds 2 Mandatory requirements vs. optional requirements Decision-makers only control optional requirements Release success depends on both kinds of requirements Requirements are subject to constrains 3 Policy Time-to-market Dependencies Budget Requirements have expected estimations 4 Costs Revenues Which optional requirements to implement? 5 F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 2 / 19
Industrial example: Motorola Motorola identified 40 requirements for a new product A small number of requirements were identified as mandatory Any release of the product must include at least these A fix amount of budget is devoted to this task Requirements can become mandatory by different reasons Core product functionality Dependence to other requirements Decision-makers faced 35 independent requirements Costs were estimated Revenues were based in customer assessments There are 2 35 possible choices > 34000 million possibilities > 1000 million possibilities with half the total cost as budget F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 3 / 19
Formalisation The Next Release Problem (NRP) Several variants defined by Bagnall et al. Simplest NRP is a classical knapsack problem Maximising revenues under budget constrains Budget is “known” a priori Optimal solutions exist and may not be unique The Knapsack Problem and the NRP Knapsack problems first studied in the 1950s Basic variants of these problems are well understood Most variants are hard computational problems Simplest NRP is hard, but “weaker” than other hard problems Different algorithms have been devised F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 4 / 19
Approximation algorithms Guarantees Different guarantees for errors depending on the algorithm 1 No guarantee Relative error Absolute error Most heuristic methods are “no guarantee” algorithms 2 Local search Genetic algorithms Simulated annealing Facts Guaranteeing absolute error for NRP is as hard as exactness 1 Relative error approximation algorithms exist 2 F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 5 / 19
Greedy algorithms Strategies Increasing Cost Order ICO DRO Decreasing Revenue Order DRD Decreasing Revenue Density DRO and DRD can suffer from unbounded error DRD with postprocessing is at worst 50% below the optimal Advantages Very fast and easy to implement 1 Do not impose artificial restrictions to their inputs 2 A natural basis for better algorithms, for example local search 3 F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 6 / 19
Sensitivity analysis Estimation is not an exact science 1 Requirements are affected by estimation errors 2 Costs are invariably underestimated Revenues are usually more predictable Budgets are subjected to cuts (nowadays even more true) 3 Complex products go through different budgeting scenarios 4 Sensitivity analysis helps to identify the hot-spots 5 The Big Question Q: How do you make sure that the observed variation stems from inherent sensitivity and not from the optimisation algorithm when you use an approximation algorithm? A: You cannot! F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 7 / 19
Sensitivity analysis of the Motorola data 21 perturbations (from -50% to +50% with 5% steps) 1 Positive perturbations show the effect of underestimations 2 Approximate Sensitivity Analysis (PIAC = 5%) Exact Sensitivity Analysis (PIAC = 5%) 3 0.0 3 0.0 10 10 3 3 10 10 3 3 10 10 3 3 0.3 0.1 10 10 3 3 20 20 3 3 20 20 2 2 20 0.6 20 0.2 2 2 30 30 3 3 40 40 1 1 40 2 40 2 0.9 0.3 40 2 40 2 50 2 50 2 50 2 50 2 1.2 0.4 60 3 60 3 70 3 70 3 Revenues Revenues Costs 80 1 Costs 80 1 80 3 1.5 80 3 0.5 100 1 100 1 100 3 100 3 100 3 100 3 1.8 0.6 110 3 110 3 120 1 120 1 150 3 150 3 1 2.1 1 0.7 180 180 1 1 200 200 3 3 200 200 2 2 230 2.4 230 0.8 1 1 300 300 1 1 300 300 3 3 400 400 2.7 0.9 3 3 500 500 2 2 1000 1000 2 2 1000 1000 1 3.0 1 1.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 3 0 0 0 4 4 6 6 7 7 8 9 0 2 2 4 9 0 3 4 7 1 9 1 6 2 2 2 0 4 0 5 3 0 0 0 4 4 6 6 7 7 8 9 0 2 2 4 9 0 3 4 7 1 9 1 6 2 2 2 0 4 5 2 4 5 2 4 1 1 4 5 6 7 7 7 9 9 0 0 5 5 5 6 6 8 8 9 2 2 3 3 4 6 7 8 9 0 1 2 6 7 7 1 1 4 5 6 7 7 7 9 9 0 0 5 5 5 6 6 8 8 9 2 2 3 3 4 6 7 8 9 0 1 2 6 7 7 1 1 1 1 3 3 3 3 3 3 3 3 4 4 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 1 1 1 1 3 3 3 3 3 3 3 3 4 4 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 Budgets Budgets F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 8 / 19
Challenges Problem Requirements are uncertain 1 Precise sensitivity analysis is key to decision making 2 Approximate solutions disable precise sensitivity analysis 3 Solution Exact sensitivity analysis based on exact algorithms Issues Efficiency 1 Scalability 2 Interactions 3 F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 9 / 19
Exact algorithms Facts Many algorithms have been devised in the last 50 years 1 Advanced algorithms are difficult to implement and test 2 For many algorithms space can be as critical as time 3 The curse of computational complexity on the NRP 4 No matter how good our exact algorithms are, they will always behave bad for an infinite number of instances, according to the current state of our knowledge (i.e., as long as P �= NP ) Approach Focus in “simple” algorithms and data structures 1 Algorithms have to be efficient just for the application domain 2 F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 10 / 19
Dynamic programming algorithms Bellman’s equation for KP 0 if n = 1 ∧ c 1 > B r 1 if n = 1 ∧ c 1 ≤ B z ( n , B ) = z ( n − 1 , B ) if n > 1 ∧ c n > B max { z ( n − 1 , B ) , z ( n − 1 , B − c n ) + r n } if n > 1 ∧ c n ≤ B Nemhauser-Ullmann’s algorithm (NU) As values of z are computed, they are saved for later reuse 1 Solution is recovered from the values of z previously saved 2 In 2003, Beier and Vöcking proved very interesting properties 3 NU solves a random KP in expected polynomial time This is so under quite general conditions Strongly correlated instances are provably harder for NU F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 11 / 19
Correlation Highly-correlated instances are reported hard in the literature 1 We control Pearson’s correlation during instance generation 2 Random requirements ( ρ =0 . 00 ) 60000 50000 40000 Revenue 30000 20000 10000 10000 20000 30000 40000 50000 Cost F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 12 / 19
Scalability experiments 50 different datasets with 315 NRP random instances each 1 15 problem sizes from 100 to 1500, with steps of 100 21 correlation degrees from 0% to 100%, every 5% Significant # of experiments: 15 750 instances solved 2 Experiments performed in just one core of a 1000 £ machine 3 Intel Core i7 2.67 GHz CPU with 12 GiB RAM C++ on GNU/Linux Results show that a polynomial model explains well our times 4 NU behaves polynomially in the number of requirements Except when costs and revenues are very highly correlated Assumptions High correlations between cost and revenue are uncommon 1 This is particularly true in the case of optional requirements 2 F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 13 / 19
Interactions One at a time Requirements are perturbed in isolation No interactions are taken into account k at a time At most k simultaneous perturbations are considered Interactions are taken into account Very expensive F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 14 / 19
Trade-off Second order interactions No interactions for a first analysis Interactions between pairs of requirements are then regarded F. Palomo-Lozano (UCA) Exact Scalable SA for the NRP British Computer Society (2014) 15 / 19
Recommend
More recommend
