Computational Red Teaming to Investigate Failure Patterns in Medium - - PowerPoint PPT Presentation

Computational Red Teaming to Investigate Failure Patterns in Medium Term Conflict Detection (MTCD)

• S. Alam, H.A. Abbass, C.J. Lokan

DSARC University of New South Wales, Australian Defence Force Academy, Canberra, Australia

• M. Ellejmi and S. Kirby

EUROCONTROL Brétigny-sur-Orge, Paris, France

8th EUROCONTROL Innovative Research Workshop

Outline

• Medium Term Conflict Detection (MTCD)
• Red teaming concept
• Evolving conflict scenarios
• Evaluation framework
• Results and conclusions

MTCD is a Planning Tool

MTCD Functions:

• Calculation of aircraft trajectories (look-ahead time)
• Monitoring an aircraft’s progress against the trajectory
• Detection of conflicting trajectories
• Presentation of this information from 8 to 20 minutes ahead

Why MTCD?

Why MTCD?

• Moves away from current reactive form of air traffic

control to more pro-active control

• Safety - at a planning level finds the conflict that might be

missed

• Early conflict detection with less uncertainty leading to
• ptimum resolution
• Re-balance sector team workload - improve efficiency in

sector team

• Improves traffic awareness
• Provides future workload indication

Why MTCD?

Have we missed any conflict? Are there any conflicts coming? How far apart these two will be?

MTCD Field Trails

• Amsterdam ACC
• Maastricht UAC
• U.S. FAA’s User Request Evaluation Tool
• Rome ACC

Earlier conflict detection. Better insight into conflict problem geometry by:

• Display of minimum distance
• Information on aircraft position.

High rate of nuisance alerts. Instability over time in the predicted closest point of approach.

• To identify patterns in conflict characteristics that

lead to False Alerts and Missed Detects

• When detecting early is too early (False Alerts)

and when detecting late is too late (Missed Detects)?

Research Questions

Red Teaming

• A defence concept of studying a problem by anticipating adversary

behaviours

• Playing the Devil's advocate
• Provides a wider and deeper understanding of potential adversary options

and behaviour that can expose potential vulnerabilities in a system

Set of Adversary Behaviours Set of Defenders Conflict Scenarios MTCD Algorithm

Simulation Red teaming to MTCD evaluation

Probing Methods in MTCD

Fixed Threshold Conflict Method

• The time to and distance of Closest Point of Approach (T2CPA and CPA,

respectively) are first computed for each potential conflict pair

• Then these two thresholds (CPA and T2CPA) are used to recognise the event
• f a conflict.
• If the CPA time is within the look-ahead window (8-20 minutes) and if the CPA

distance is less than the separation minimum the aircraft pair is tentatively declared conflicting Covariance Method

• Error ellipse path uncertainty regions are computed at the T2CPA together

with the CPA distance for each potential conflict pair

• The error ellipses are based on covariance calculations obtained by modelling

surveillance errors and aircraft path following errors

• An intruder is tentatively conflicting if the predicted uncertainty ellipse at a time

point intersects the separation standard circle around own ship

Conflict Scenario Planning

What's good about it? What's bad about it?

• Low rate of conflicts
• Induced conflicts are pre-scripted
• Evaluation is not rigorous
• Can provide robust feedback
• Use of realistic air traffic data
• Preserve real world errors and

features

Scenarios = Air Traffic Samples

• Repetition – Replication – Evolution
• Events should unfold themselves as the scenario progresses
• Scenarios with varying degrees of complexity

How to overcome these problems?

Aircraft Database Aircraft Performance Database Airspace Configuration Atmospheric & Wind Data Vertical separation distance Horizontal separation distance Conflict angle Intruder geometry Own ship geometry Turn angle Conflict Generation Module Flight Plans of Two Aircraft

Conflict Scenario Planning

Generating conflict scenarios through Genetic Algorithms

Scenario 1 Fast Time Air Traffic Simulator Missed Detects False Alerts Scenario 2 Scenario n MTCD (Fixed Threshold) Objective Values Rank based selection of scenarios Crossover of scenarios Mutation of conflict characteristics New Population

• f Scenarios
• Max. Generation

No Yes Report Scenario Set MTCD (Covariance) Extract Scenario Characteristics Conflict Characteristics Analysis MTCD Algorithm Evaluation

Evaluation Methodology

Conflict Characteristics Conflict Characteristics

Evaluating MTCD by generating increasingly complex conflict scenarios using feedback from the process itself! Scenarios with high failures (missed detects & False Alerts) and their variants are repeatedly fed back in the process.

Conflict Characteristics

Conflict Detection Performance Measures Reliability Missed Detects False Alerts Valid Alerts

Conflicts Conflicts predicted by predicted by the algorithm the algorithm Actual Actual conflicts conflicts

Evaluation Metrics & Results

0.8% 0.8% Missed Detects 3.6% 4.6% False Alerts Covariance Method Fixed Threshold MTCD Evaluation

Covariance Method performs slightly better that Fixed Threshold method as it takes into consideration the inherent uncertainty in CNS

False Alerts Missed Detects Valid Alerts

Results (Covariance Method )

Sensitivity of the probe method to Conflict Alert Window

Time to CPA for conflict alerts that led to False Alerts in Covariance method (8–20 minutes window)

• From 8 to 15 minutes (34% of conflicts) there is

a linear relationship between the False Alerts and conflict alert time.

• From 16-20 minutes (66% of conflicts) the rate
• f False Alerts increases.

Duration of conflict alerts that led to False Alerts in Covariance Method

For a large number of conflict flights the conflict duration is small, i.e., the conflict was flagged and it was subsequently removed.

• False Alerts -

Results (Covariance Method )

• False Alerts -

Sensitivity of the probe method to Conflict Alert Window

Conflict alerts with duration greater than 30 seconds that led to False Alerts in Covariance method. Conflict alerts with duration greater than 60 seconds that led to False Alerts in Covariance method.

Eliminating the flight pairs that have conflict duration less than 30 seconds reduces the False Alerts by more than 19% Eliminating the flight pairs that have conflict duration less than 60 seconds reduces the False Alerts by more than 34%

Results (Covariance Method )

Sensitivity of the probe methods to Conflict Alert Window

• Missed Detects -

Valid Alerts that did not continue in the time window and led to Missed Detects 8-20 minutes window Valid Alerts that were discontinued before 9 minutes in conflict alert window and led to Missed Detects Valid Alerts that were discontinued before 10 minutes in conflict alert window and led to Missed Detects

• Reducing the threshold window from 8–20 minutes to 9–20 minutes reduces

the Missed Detects by 75.1%.

• Further reducing the threshold window to 10–20 minutes reduces the

Missed Detects by another 58.9%.

1200 8 20 Missed Detects Time /minutes 1200 8 20 Missed Detects Time /minutes 1200 8 20 Missed Detects Time /minutes

Results (Conflict Characteristics )

Both MTCD algorithms are more vulnerable to cruise-cruise conflicts

Flight conflict geometry of False Alerts. Flight conflict geometry of Missed Detects. Key CL – climb CR – cruise DS – descent

Results (Conflict Characteristics )

Both MTCD algorithms are susceptible to generating False Alerts and Missed Detects when the own ship and intruder have wider conflict angles (90–180 degrees)

Conflict angle of flight pairs that generated False Alerts for Fixed Threshold and Covariance method. Conflict angle of flight pairs that generated Missed Detect for Fixed Threshold and Covariance method.

Conclusions

MTCD Covariance method

• The MTCD Fixed Threshold method results are not shown, but lead to the same

conclusions

• Both MTCD algorithms are more vulnerable to ‘cruise-cruise’ conflicts
• Monitoring or delaying the alert of conflicts with wider convergence angles (90–180

degrees) can also reduce False Alerts and Missed Detects in both methods

• The two MTCD algorithms have similar vulnerabilities
• Need to confirm the effect of the suggested changes with further work

Other conclusions

• Reducing the conflict alert threshold window to 8–15 minutes can reduce up to 66% of

False Alerts

• Eliminating the conflict alerts with conflict duration of less than 30 seconds can further

reduce the False Alerts by 19%

• Raising the lower end of the conflict alert threshold window from 8 to 9 minutes can

reduce the Missed Detect rate by 75.1%

Any questions?