Computational Red Teaming to Investigate Failure Patterns in Medium - PowerPoint PPT Presentation

8th EUROCONTROL Innovative Research Workshop Computational Red Teaming to Investigate Failure Patterns in Medium Term Conflict Detection (MTCD) S. Alam, H.A. Abbass, C.J. Lokan M. Ellejmi and S. Kirby DSARC EUROCONTROL University of New South Wales, Brétigny-sur-Orge, Paris, France Australian Defence Force Academy, Canberra, Australia

Outline • Medium Term Conflict Detection (MTCD) • Red teaming concept • Evolving conflict scenarios • Evaluation framework • Results and conclusions

Why MTCD? MTCD is a Planning Tool MTCD Functions: • Calculation of aircraft trajectories (look-ahead time) • Monitoring an aircraft’s progress against the trajectory • Detection of conflicting trajectories • Presentation of this information from 8 to 20 minutes ahead

Why MTCD? • Moves away from current reactive form of air traffic control to more pro-active control • Safety - at a planning level finds the conflict that might be missed • Early conflict detection with less uncertainty leading to optimum resolution • Re-balance sector team workload - improve efficiency in sector team • Improves traffic awareness • Provides future workload indication

Why MTCD? Are there any conflicts How far apart coming? Have we these two will missed any be? conflict?

MTCD Field Trails • Amsterdam ACC • Maastricht UAC • U.S. FAA’s User Request Evaluation Tool • Rome ACC Earlier conflict detection. Better insight into conflict problem geometry by: - Display of minimum distance - Information on aircraft position. High rate of nuisance alerts. Instability over time in the predicted closest point of approach.

Research Questions • To identify patterns in conflict characteristics that lead to False Alerts and Missed Detects • When detecting early is too early (False Alerts) and when detecting late is too late (Missed Detects)?

Red Teaming • A defence concept of studying a problem by anticipating adversary behaviours • Playing the Devil's advocate • Provides a wider and deeper understanding of potential adversary options and behaviour that can expose potential vulnerabilities in a system Simulation Red teaming to MTCD evaluation Set of Set of Conflict MTCD Adversary Defenders Scenarios Algorithm Behaviours

Probing Methods in MTCD Fixed Threshold Conflict Method • The time to and distance of Closest Point of Approach (T2CPA and CPA, respectively) are first computed for each potential conflict pair • Then these two thresholds (CPA and T2CPA) are used to recognise the event of a conflict. • If the CPA time is within the look-ahead window (8-20 minutes) and if the CPA distance is less than the separation minimum the aircraft pair is tentatively declared conflicting Covariance Method • Error ellipse path uncertainty regions are computed at the T2CPA together with the CPA distance for each potential conflict pair • The error ellipses are based on covariance calculations obtained by modelling surveillance errors and aircraft path following errors • An intruder is tentatively conflicting if the predicted uncertainty ellipse at a time point intersects the separation standard circle around own ship

Conflict Scenario Planning Scenarios = Air Traffic Samples What's good about it? What's bad about it? • Can provide robust feedback • Low rate of conflicts • Use of realistic air traffic data • Induced conflicts are pre-scripted • Preserve real world errors and • Evaluation is not rigorous features How to overcome these problems? • Repetition – Replication – Evolution • Events should unfold themselves as the scenario progresses • Scenarios with varying degrees of complexity

Conflict Scenario Planning Generating conflict scenarios through Genetic Algorithms Vertical separation distance Horizontal separation distance Conflict angle Intruder geometry Own ship geometry Turn angle Aircraft Performance Atmospheric & Database Wind Data Conflict Generation Module Airspace Aircraft Database Configuration Flight Plans of Two Aircraft

Evaluation Methodology Conflict Conflict Conflict Characteristics Characteristics Characteristics Scenario 1 Scenario 2 Scenario n Evaluating MTCD by MTCD (Fixed Threshold) generating increasingly Fast Time Air Traffic Simulator MTCD complex conflict (Covariance) Objective Values scenarios using feedback Missed Detects False Alerts from the process itself! Rank based selection of scenarios Report Scenario Crossover of scenarios Set Scenarios with high Mutation of conflict failures (missed detects Extract Scenario characteristics Characteristics & False Alerts) and their Conflict variants are repeatedly Characteristics New Analysis Population of Scenarios fed back in the process. MTCD Algorithm Evaluation No Yes Max. Generation

Evaluation Metrics & Results Conflict Detection Conflicts Conflicts Performance Measures predicted by predicted by the algorithm the algorithm Reliability False Missed Valid Alerts Alerts Detects Missed Detects False Alerts Actual Actual Valid Alerts conflicts conflicts MTCD Evaluation Covariance Method performs slightly better that Fixed Fixed Threshold Covariance Method Threshold method as it takes False Alerts 4.6% 3 . 6% into consideration the inherent uncertainty in CNS Missed Detects 0 . 8% 0 . 8%

Results (Covariance Method ) Sensitivity of the probe method to Conflict Alert Window - False Alerts - Time to CPA for conflict alerts that led to False Alerts Duration of conflict alerts that led to False Alerts in Covariance Method in Covariance method (8–20 minutes window) • From 8 to 15 minutes (34% of conflicts) there is For a large number of conflict flights the a linear relationship between the False Alerts conflict duration is small, i.e., the conflict was and conflict alert time. flagged and it was subsequently removed. From 16-20 minutes ( 66% of conflicts ) the rate • of False Alerts increases.

Results (Covariance Method ) Sensitivity of the probe method to Conflict Alert Window - False Alerts - Eliminating the flight pairs that Conflict alerts with duration have conflict duration less than greater than 30 seconds that led to False Alerts in 30 seconds reduces the False Covariance method. Alerts by more than 19% Eliminating the flight pairs that Conflict alerts with duration have conflict duration less than greater than 60 seconds that led to False Alerts in 60 seconds reduces the False Covariance method. Alerts by more than 34%

Results (Covariance Method ) Sensitivity of the probe methods to Conflict Alert Window - Missed Detects - 1200 1200 1200 Missed Detects Missed Detects Missed Detects 0 0 0 8 20 8 20 8 20 Time /minutes Time /minutes Time /minutes Valid Alerts that did not Valid Alerts that were Valid Alerts that were continue in the time window discontinued before 9 discontinued before 10 and led to Missed Detects 8-20 minutes in conflict alert minutes in conflict alert minutes window window and led to Missed window and led to Missed Detects Detects • Reducing the threshold window from 8–20 minutes to 9–20 minutes reduces the Missed Detects by 75.1%. • Further reducing the threshold window to 10–20 minutes reduces the Missed Detects by another 58.9%.

Results (Conflict Characteristics ) Flight conflict geometry of False Alerts . Flight conflict geometry of Missed Detects . Key CL – climb CR – cruise DS – descent Both MTCD algorithms are more vulnerable to cruise-cruise conflicts

Results (Conflict Characteristics ) Conflict angle of flight pairs that generated False Alerts for Conflict angle of flight pairs that generated Missed Detect for Fixed Threshold and Covariance method. Fixed Threshold and Covariance method. Both MTCD algorithms are susceptible to generating False Alerts and Missed Detects when the own ship and intruder have wider conflict angles (90–180 degrees)

Conclusions MTCD Covariance method • Reducing the conflict alert threshold window to 8–15 minutes can reduce up to 66% of False Alerts Eliminating the conflict alerts with conflict duration of less than 30 seconds can further • reduce the False Alerts by 19% Raising the lower end of the conflict alert threshold window from 8 to 9 minutes can • reduce the Missed Detect rate by 75.1% Other conclusions • The MTCD Fixed Threshold method results are not shown, but lead to the same conclusions • Both MTCD algorithms are more vulnerable to ‘cruise-cruise’ conflicts • Monitoring or delaying the alert of conflicts with wider convergence angles (90–180 degrees) can also reduce False Alerts and Missed Detects in both methods • The two MTCD algorithms have similar vulnerabilities • Need to confirm the effect of the suggested changes with further work

Any questions?