Modified Process and Technology Objective Functions List of 2.Search with 1.Exploration with Worst-Case + Critical Surrogate Dimensionality Scenarios Controller Regression Partitions Modeling Reduction Domain Model Tree Expert (Simulink) Visualization of the 8-dimension space using regression trees Dimensionality reduction to identify Surrogate modeling the significant variables to predict the objective (Elementary Effect Analysis) function and speed up the search (Machine learning) 30
Dimensionality Reduction ∗ 10 − 2 • Sensitivity Analysis: 0.6 Elementary Effect Analysis Sample Standard Deviation ( ) S δ i (EEA) FD ID Cal5 Cal3 • Identify non-influential inputs in computationally 0.4 costly mathematical models • Requires less data points than other techniques 0.2 Cal6 • Observations are simulations generated Cal4 during the Exploration step 0.0 • Compute sample mean Cal1,Cal2 and standard deviation for ∗ 10 − 2 each dimension of the distribution of elementary -0.6 -0.4 -0.2 0.0 0.2 effects Sample Mean ( ) δ i 31
Elementary Effects Analysis Method ü Imagine function F with 2 inputs, x and y: Elementary Effects Y for X for Y B2 F(A1)-F(A) F(A2)-F(A) F(B1)-F(B) F(B2)-F(B) ∆ y F(C1)-F(C) F(C2)-F(C) B1 ∆ x B … … A2 C2 ∆ y ∆ y A1 ∆ x A C1 ∆ x C X 32
Visualization in Inputs & Configuration Space All Points Count 1000 Mean 0.007822 Std Dev 0.0049497 FD>=0.43306 FD<0.43306 Count 574 Count 426 Mean 0.0059513 Mean 0.0103425 Std Dev 0.0040003 Std Dev 0.0049919 Cal5>=0.020847 Cal5>0.020847 ID<0.64679 ID>=0.64679 Count 201 Count Count 182 Count 373 244 Mean Mean Mean 0.0134555 Mean 0.0047594 0.0081631 0.0080206 Std Dev Std Dev Std Dev 0.0052883 Std Dev 0.0034346 0.0040422 0.0031751 Cal5>=0.014827 Cal5<0.014827 131 Count 70 Count 0.0068185 Mean 0.0106795 Mean Regression Tree 0.0023515 Std Dev 0.0052045 Std Dev 33
Surrogate Modeling During Search • Goal: To predict the value of the objective functions within a critical partition, given a number of observations, and use that to avoid as many simulations as possible and speed up the search B A 34
Surrogate Modeling During Search • Any supervised learning or Real Function statistical technique Fitness providing fitness predictions Surrogate Model with confidence intervals 1. Predict higher fitness with high confidence: Move to new position, no simulation 2. Predict lower fitness with high confidence: Do not move to new position, no simulation x 3. Low confidence in prediction: Simulation 35
Experiments Results (RQ1) ü The best regression technique to build Surrogate models for all of our three objective functions is Polynomial Regression with n = 3 ü Other supervised learning techniques, such as SVM Mean of R 2 /MRPE values for different surrogate modeling techniques LR ER PR(n=2) PR(n=3) R 2 / MRPE R 2 / MRPE R 2 / MRPE R 2 / MRPE F sm 0.66/ 0.0526 0.44/ 0.0791 0.95/ 0.0203 0.98/ 0.0129 0.78/ 0.0295 0.85/ 0.0247 0.85/ 0.0245 0.49/ 1.2281 F r F st 0.22/ 1.2519 0.26/ 0.2043 0.46/ 0.1755 0.54/ 0.1671 36
Experiments Results (RQ2) ü Dimensionality reduction helps generate better surrogate models for Smoothness and Responsiveness requirements Mean Relative Prediction Errors 0.08 0.05 0.3 0.06 0.04 (MRPE Values) 0.04 0.03 0.2 0.02 0.02 0.1 0.0 0.01 DR No DR DR No DR DR No DR Smoothness( ) F sm F st Responsiveness( ) F r Stability( ) 37
Experiments Results (RQ3) ü For responsiveness, the search with SM was 8 times faster After 200 seconds After 300 seconds After 3000 seconds Search Output 0.168 Values 0.164 0.160 SM NoSM NoSM SM NoSM SM ü For smoothness, the search with SM was much more effective After 800 seconds After 2500 seconds After 3000 seconds 0.235 Search Output 0.230 Values 0.225 0.220 0.215 38 SM NoSM SM NoSM SM NoSM
Experiments Results (RQ4) ü Our approach is able to identify critical violations of the controller requirements that had neither been found by our earlier work nor by manual testing. MiL-Testing MiL-Testing different configurations fixed configurations Manual MiL-Testing Stability 2.2% deviation - - Smoothness 24% over/undershoot 20% over/undershoot 5% over/undershoot Responsiveness 170 ms response time 80 ms response time 50 ms response time 39
A Taxonomy of Automotive Functions Computation Controlling Transforming Calculating State-Based Continuous State machine unit convertors calculating positions, Closed-loop controllers duty cycles, etc controllers (PID) Different testing strategies are required for different types of functions 40
Open-Loop Controllers Engaging • No feedback loop -> no automated [ ¬ ( vehspd = 0) ∧ oracle time > 2] OnMoving OnSlipping • No plant model: Much quicker time + +; time + +; simulation time ctrlSig := f ( time ) ctrlSig := g ( time ) • Mixed discrete-continuous behavior: [ time > 4] [( vehspd = 0) ∧ OnCompleted Simulink stateflows time > 3] time + +; ctrlSig := 1 . 0 • The main testing cost is the manual analysis of output signals • Goal: Minimize test suites CtrlSig • Challenge: Test selection On • Entirely different approach to testing 41 Off
Selection Strategies Based on Search • Input diversity • White-box Structural Coverage S3 • State Coverage t • Transition Coverage • Output Diversity • Failure-Based Selection Criteria • Domain specific failure patterns S3 • Output Stability • Output Continuity t 42
Failure-based Test Generation • Search: Maximizing the likelihood of presence of specific failure patterns in output signals • Domain-specific failure patterns elicited from engineers Instability Discontinuity 1.0 1.0 CtrlSig Output CtrlSig Output 0.5 0.75 0.50 0.0 0.25 -0.5 0.0 -1.0 0.0 1.0 2.0 0.0 1.0 2.0 Time Time 4 3
Summary of Results • The test cases resulting from state/transition coverage algorithms cover the faulty parts of the models • However, they fail to generate output signals that are sufficiently distinct from the oracle signal, hence yielding a low fault revealing rate • Output-based algorithms are more effective 44
Automated Testing of Driver Assistance Systems Through Simulation Reference: R. Ben Abdessalem et al., "Testing Advanced Driver Assistance Systems Using Multi-Objective Search and Neural Networks”, ACM ESEC/FSE 2016 45
Pedestrian Detection Vision System (PeVi) • The PeVi system is a camera-based collision-warning system providing improved vision 46
Testing DA Systems • Testing DA systems requires complex and comprehensive simulation environments – Static objects: roads, weather, etc. – Dynamic objects: cars, humans, animals, etc. • A simulation environment captures the behavior of dynamic objects as well as constraints and relationships between dynamic and static objects 47
Approach Specification Documents (Simulation Environment and Pe Vi System) (1) Development of Requirements and domain models Requirements Domain model model (2) Generation of Test specifications test case specification Static Dynamic [ranges/values/ [ranges/ resolution] resolution] 48
PeVi and Environment Domain Model « enumeration » « enumeration » Weather Road SceneLight - weatherType: - roadType: RT - intensity: Real Condition RT 1 1 1 Condition - fog - curved - rain - straight - snow - ramped RoadSide * - normal Test Scenario Test Scenario Test Scenario Object Position - simulationTime: - simulationTime: - simulationTime: - x: Real Real Real Real Trees - y: Real - timeStep: Real - timeStep: Real - timeStep: Real * Parked Pedestrian Pedestrian 1 Cars Collision - x0: Real - x0: Real Output 1 1 - state: Boolean Trajectory - y0: Real - y0: Real 1 1 Camera Vehicle Vehicle - AWA - θ : Real - θ : Real 1 Sensor - v0: Real - v0: Real Detection - v0: Real - v0: Real - field of view: 1 1 « positioned » - state: Boolean 1 Real 1 Dynamic 1 1 1 Object PeVi PeVi PeVi « uses » Static inputs Dynamic inputs Outputs 49
Requirements Model The NiVi system shall detect any person located in the Acute Warning Area of a vehicle <<trace>> <<trace>> pos y2 pos y1 trajectory Trajectory Human 1 * pos x1 pos x2 human 1 1 appears Speed Path Profile 1 1.. * AWA * sensor 1 Car/Motor/ has 1 Sensors posx1, posx2 Truck/Bus Path Slot 1 Warning has awa * posy1, posy2 Segment 50
MiL Testing via Search Meta-heuristic Search Generate (multi-objective) scenarios Simulator + NiVi Fixed during Search Manipulated by Search Car Simulator PeVi (speed) Environment Settings (Roads, weather, Human Simulator Detection Collision vehicle type, etc.) (initial position, or not? or not? speed, orientation) 51
Test Case Specification: Static (combinatorial) Type of Road Type of vehicle Type of actor Situation 1 Straight Car Male Situation 2 Straight Car Child Situation 3 Straight Car Cow Situation 4 Straight Truck Male Situation 5 Straight Truck Child Situation 6 Straight Truck Cow Situation 7 Curved Car Male Situation 8 Curved Car Child Situation 9 Curved Car Cow Situation 10 Curved Truck Male Situation 11 Curved Truck Child Situation 12 Curved Track Cow Situation 13 Ramp Car Male Situation 14 Ramp Car Child Situation 15 Ramp Car Cow Situation 16 Ramp Truck Male Situation 17 Ramp Truck Child Situation 18 Ramp Truck Cow Situation 19 Straight Car+ Cars in parking Male Situation 20 Car + buildings 5 2
Test Case Specification: Dynamic pathCar : Path StartPointX = 10 segmentCar : Path Segment StartPointY = 50.125 Length = 100 StartPointZ =0.56 Type = Straight StartAngle = 0 car : Actor MaxSpeedLimit = 100 End Angle = 0 Position X=10 trajectoryCar : Trajectory Length = 100 Position Y= 50.125 Start locationX = 10 Position Z = 0.56 Start locationY = 50.125 OrientationHeading = 0 Start locationZ = 0.56 profileCar : Speed slotCar : Slot Acceleration = 0 Orientation = 0 Profile ID MaxWalkingSpeed =100 UniqueId startCar : StartState Time = 0 Collision Speed = 60.66 MinTTC=0.3191 pathPerson : Path StartPointX = 74 segmentPerson : Path Segment StartPointY = 37.72 Length = 60 StartPointY = 0 Type = Straight StartAngle = 93.33 MaxSpeedLimit = 14 person :Actor End Angle = 0 PositionX= 74 Length = 60 trajectoryPerson : Trajectory Position Y= 37.72 Start locationX = 74 Position Z = 0 Start locationY = 37.72 OrientationHeading = 93.33 profilePerson : Start locationZ = 0 slotPerson : Slot Acceleration = 0 Speed Profile ID Orientation = 0 MaxWalkingSpeed =14 UniqueId height=1.75 startPerson : StartState 53 Time = 0 Speed = 12.59
Choice of Surrogate Model • Neural networks (NN) have been trained to learn complex functions predicting fitness values • NN can be trained using different algorithms such as: – LM: Levenberg-Marquardt – BR: Bayesian regularization backpropagation – SCG: Scaled conjugate gradient backpropagation R 2 (coefficient of determination) indicates how well • data fit a statistical model Computed R 2 for LM, BR and SCG è BR has the • highest R 2 54
Multi-Objective Search • Input space: car-speed, person- speed, person-position (x,y), person-orientation pos y2 • Search algorithm need objective pos y1 or fitness functions for guidance • In our case several independent pos x1 pos x2 functions could be interesting: – Minimum distance between car and pedestrian – Minimum distance between pedestrian and AWA – Minimum time to collision • NSGA II algorithm 55
Pareto Front O 1 Pareto front x Individual A Pareto dominates individual B if A is at least as good as B Dominated by x in every objective and better than B in at least one objective. O 2 • A multi-objective optimization algorithm must achieve: • Guide the search towards the global Pareto-Optimal front. • Maintain solution diversity in the Pareto-Optimal front. 56
MO Search with NSGA-II Non-Dominated Selection based on rank Sorting and crowding distance Size: N Size: 2*N Size: 2*N • Based on Genetic Algorithm • N: Archive and population size • Non-Dominated sorting: Solutions are ranked according to how far they are from the Pareto front, fitness is based on rank. • Crowding Distance: Individuals in the archive are being spread more evenly across the front (forcing diversity) • Runs simulations for close to N new solutions 57
Pareto Front Results 58
Pareto Front Projection 59
Simulation Scenario Execution • Straight road with parking • The person appears in the AWA, but is not detected 60
Improving Time Performance • Individual simulations take on average more than 1min • It takes 10 hours to run our search-based test generation (≈ 500 simulations) • We use surrogate modeling to improve the search • Neural networks are used to predict fitness values within a confidence interval • During the search, we use prediction values & confidence intervals to run simulations only for the solutions likely to be selected 61
Search with Surrogate Models NSGA II Selection based on rank Non-Dominated and crowding distance Sorting Size: 2*N Size: 2*N Size: N Original Algorithm Our Algorithm - Runs simulations for all - Uses prediction values new solutions & intervals to run simulations only for the solutions likely to be selected 62
Results – Surrogate Modeling by NSGAII and NSGAII-SM 1.00 0.75 HV 0.50 0.25 NSGAII-SM (mean) NSGAII (mean) 0.00 10 50 100 150 Time (min) 63
Results – Random Search by RS and NSGAII-SM 1.00 0.75 HV 0.50 0.25 NSGAII-SM (mean) RS (mean) 0.00 10 50 100 150 Time (min) (c) HV values for worst runs of NSGAII, 64
Results – Worst Runs NSGAII-SM and RS 1.00 0.75 HV 0.50 0.25 NSGAII-SM NSGAII RS 0.00 10 50 100 150 Time (min) 65
Minimizing CPU Shortage Risks During Integration References: • S. Nejati et al., ‘‘Minimizing CPU Time Shortage Risks in Integrated Embedded Software’’, in 28th IEEE/ACM International Conference on Automated Software Engineering (ASE 2013), 2013 • S. Nejati, L. Briand, “Identifying Optimal Trade-Offs between CPU Time Usage and Temporal Constraints Using Search”, ACM International Symposium on Software Testing and Analysis (ISSTA 2014), 2014 66
Automotive: Distributed Development 67
Software Integration 68
Stakeholders Car Makers Integrator • Develop software optimized for • Integrate car makers software their specific hardware with their own platform • Provide integrator with runnables • Deploy final software on ECUs and send them to car makers 69
Different Objectives Integrator Car Makers • Objective: Effective usage of • Objective: Effective execution and CPU time synchronization of runnables • Max CPU time used by all the • Some runnables should execute runnables should remain as low simultaneously or in a certain order as possible over time 70
An overview of an integration process in the automotive domain sw runnables AUTOSAR Models Glue AUTOSAR Models sw runnables 71
CPU time shortage • Static cyclic scheduling: predictable, analyzable • Challenge – Many OS tasks and their many runnables run within a limited available CPU time • The execution time of the runnables may exceed their time slot • Goal – Reducing the maximum CPU time used per time slot to be able to • Minimize the hardware cost • Reduce the probability of overloading the CPU in practice • Enable addition of new functions incrementally 40ms ✗ (a) 5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms ✔ (b) 5ms 10ms 15ms 20ms 25ms 30ms 35ms 72 72
Using runnable offsets (delay times) Inserting runnables’ offsets 40ms ✗ 5ms 10ms 15ms 20ms 25ms 30ms 35ms 5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms ✔ Offsets have to be chosen such that the maximum CPU usage per time slot is minimized, and further, the runnables respect their period the runnables respect their time slot the runnables satisfy their synchronization constraints 73 73
Without optimization 5.34ms 5.34ms CPU time usage (ms) 5 ms Time CPU time usage exceeds the size of the slot (5ms) 74
With Optimization CPU time usage (ms) 5 ms 2.13ms Time CPU time usage always remains less than 2.13ms, so more than half of each slot is guaranteed to be free 75
Single-objective Search algorithms Hill Climbing and Tabu Search and their variations Solution Representation a vector of offset values: o0=0, o1=5, o2=5, o3=0 Tweak operator o0=0, o1=5, o2=5, o3=0 à o0=0, o1=5, o2=10, o3=0 Synchronization Constraints offset values are modified to satisfy constraints Fitness Function max CPU time usage per time slot 76
Summary of Problem and Solution Optimization Explicit Time Model while satisfying for real-time embedded systems synchronization/temporal constraints Search 10^27 an industrial case study with a meta-heuristic single objective large search space search algorithms 77
Search Solution and Results - The objective function is the max CPU usage of a 2s-simulation of runnables - The search modifies one offset at a time, and updates other offsets only if timing constraints are violated - Single-state search algorithms for discrete spaces (HC, Tabu) Case Study: an automotive software system with 430 runnables, search space = 10^27 2.13 ms 5.34 ms 78 Optimized offset assignment 78 Running the system without offsets
Comparing different search algorithms Best CPU usage (ms) Time to find Best CPU usage (s) 79 79
Comparing our best search algorithm with random search Lowest max CPU usage values computed by HC within 70 ms Lowest max CPU usage values computed by Random Comparing average behavior of Random and HC in computing over 100 different runs within 70 ms over 100 different runs (a) lowest max CPU usage values within 70 s and over 100 different runs (b) (c) (a) HC Random Average 80 80
Trade-off between Objectives Car Makers Integrator r 3 r 1 r 2 r 0 Execute r 0 to r 3 close to one another. Minimize CPU time usage 1 slot 30ms 4ms 0ms 5ms 10ms 15ms 20ms 25ms 2 slots 3ms 0ms 5ms 10ms 15ms 20ms 25ms 30ms 3 slots 2ms 0ms 5ms 10ms 15ms 20ms 25ms 30ms 81
Trade-off curve 1 Boundary Trade Offs 21 # of slots Interesting Solutions 3 14 2 12 1.56 2.04 1.45 CPU time usage (ms) 82
Multi-objective search • Multi-objective genetic algorithms (NSGA II) • Pareto optimality • Supporting decision making and negotiation between stakeholders CPU Time Usage-NSGAII & CPU Time Usage-Random 3.0 Objectives: Max CPU Time Usage (ms) • (1) Max CPU time NSGA-II(25,000) 2.5 • (2) Maximum time Random(25,000) A slots between “dependent” tasks 2.0 C 1.5 1.45 B 12 10 15 20 25 30 35 40 45 83 Total Number of Time Slots Number of Slots-NSGAII & Number of Slots-Random
Trade-Off Analysis Tool A list of solutions: - objective 1 (CPU usage) Search - objective 2 (# of slots) - vector of group slots Input.csv: - vector of offsets - runnables - Periods - CETs Visualization/ - Groups - # of slots per Query Analysis groups - Visualize solutions - Retrieve/visualize simulations - Visualize Pareto Fronts - Apply queries to the solutions 84
Conclusions - Search algorithms to compute offset values that reduce the max CPU time needed - Generate reasonably good results for a large automotive system and in a small amount of time - Used multi-objective search à tool for establishing trade-off between relaxing synchronization constraints and maximum CPU time usage 85 85
Schedulability Analysis and Stress Testing References: • S. Di Alesio et al., “Worst-Case Scheduling of Software Tasks – A Constraint Optimization Model to Support Performance Testing, Constraint Programming (CP), 2014 • S. Di Alesio et al. “Combining Genetic Algorithms and Constraint Programming to Support Stress Testing”, ACM TOSEM, 25(1), 2015 86
Real-time, concurrent systems (RTCS) • Real-time, concurrent systems (RTCS) have concurrent interdependent tasks which have to finish before their deadlines • Some task properties depend on the environment, some are design choices • Tasks can trigger other tasks, and can share computational resources with other tasks • How can we determine whether tasks meet their deadlines? 87
Problem • Schedulability analysis encompasses techniques that try to predict whether all (critical) tasks are schedulable, i.e., meet their deadlines • Stress testing runs carefully selected test cases that have a high probability of leading to deadline misses • Stress testing is complementary to schedulability analysis • Testing is typically expensive, e.g., hardware in the loop • Finding stress test cases is difficult 88
Finding Stress Test Cases is Difficult j0, j1 , j2 arrive at at0 , at1 , at2 and must finish before dl0 , dl1 , dl2 j0 j1 j2 j0 j1 j2 at0 at0 0 0 at2 1 1 at1 2 2 at2 3 3 at1 dl2 4 4 T 5 5 dl0 dl0 dl1 dl2 6 6 T 7 7 dl1 8 8 9 9 J1 can miss its deadline dl1 depending on when at2 occurs! 89
Challenges and Solutions • Ranges for arrival times form a very large input space • Task interdependencies and properties constrain what parts of the space are feasible • We re-expressed the problem as a constraint optimisation problem • Constraint programming (e.g., IBM CPLEX) 90
Constraint Optimization Constraint Optimization Problem Static Properties of Tasks (Constants) OS Scheduler Behaviour (Constraints) Dynamic Properties of Tasks (Variables) Performance Requirement (Objective Function) 91
Process and Technologies UML Modeling (e.g., INPUT MARTE) System Design Design Model (Time and Concurrency Information) System Platform Constraint Optimization Optimization Problem Deadline Misses (Find arrival times that maximize the Analysis chance of deadline misses) Constraint Programming (CP) OUTPUT Solutions Stress Test Cases (Task arrival times likely to lead to deadline misses) 92
Context System monitors gas leaks and fire in oil extraction platforms Drivers (Software-Hardware Interface) Control Modules Alarm Devices Real-Time Operating System (Hardware) Multicore Architecture 93
Challenges and Solutions • CP effective on small problems • Scalability problem: Constraint programming (e.g., IBM CPLEX) cannot handle large input spaces (CPU, memory) • Solution: Combine metaheuristic search and constraint programming – metaheuristic search (GA) identifies high risk regions in the input space – constraint programming finds provably worst-case schedules within these (limited) regions – Achieve (nearly) GA efficiency and CP effectiveness • Our approach can be used both for stress testing and schedulability analysis (assumption free) 94
Combining GA and CP Fig. 3: Overview of GA+CP: the solutions , and in the initial population of GA evolve into 95
Process and Technologies UML Modeling (e.g., INPUT MARTE) System Design Design Model (Time and Concurrency Information) System Platform Constraint Optimization Optimization Problem Deadline Misses (Find arrival times that maximize the Analysis chance of deadline misses) Genetic Constraint Algorithms Programming (GA) (CP) OUTPUT Solutions Stress Test Cases (Task arrival times likely to lead to deadline misses) 96
V&V Topics Addressed by Search • Many projects over the last 15 years • Design-time verification – Schedulability – Concurrency – Resource usage • Testing – Stress/load testing, e.g., task deadlines – Robustness testing, e.g., data errors – Reachability of safety or business critical states, e.g., collision and no warning – Security testing, e.g., XML and SQLi injections 97
Publicity! • Chunhui Wang et al., “System Testing of Timing Requirements based on Use Cases and Timed Automata”. Session R09 @ ICST 2017, Tuesday, 2 pm • Sadeeq Jan et al., “A Search-based Testing Approach for XML Injection Vulnerabilities in Web Applications”. Session R11 @ ICST 2017, Thursday 11 am 98
General Pattern: Using Metaheuristic Search Objective Function Problem = fault model n Model = system or n environment Search to optimize n objective function(s) Metaheuristics n Scalability: A small part n Search of the search space is traversed Spac e Model: Guidance to n worst case, high-risk scenarios across space Reasonable modeling n effort based on standards or extension Heuristics: Extensive n Search empirical studies are required Technique 99
General Pattern: Using Metaheuristic Search Objective Simulator Function Model simulation can be n time consuming Makes the search n impractical or ineffective Surrogate modeling n Search based on machine Spac e learning Simulator dedicated to n search Search Technique 100
Recommend
More recommend
