Thesis Defense Slides Data August 2016 CITATIONS READS 0 47 1 - PDF document

Collision Avoidance System Elements IF (ITF.A LT G.ZTHR) IF (ITF.A LT G.ZTHR) IF (ITF.A LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN SET ZHIT; THEN SET ZHIT; THEN SET ZHIT; ELSE CLEAR ZHIT; ELSE CLEAR ZHIT; ELSE CLEAR ZHIT; ELSE IF (ITF.ADOT GE P.ZDTHR) ELSE IF (ITF.ADOT GE P.ZDTHR) ELSE IF (ITF.ADOT GE P.ZDTHR) THEN CLEAR ZHIT THEN CLEAR ZHIT THEN CLEAR ZHIT ELSE ELSE ELSE ITF.TAUV = -ITF.A/ITF.ADOT; Sensor Resolution ITF.TAUV = -ITF.A/ITF.ADOT; ITF.TAUV = -ITF.A/ITF.ADOT; IF (ITF.TAUV LT TVTHR AND Measurements Advisory IF (ITF.TAUV LT TVTHR AND IF (ITF.TAUV LT TVTHR AND ((ABS(ITF.VMD) LT G.ZTHR) OR ((ABS(ITF.VMD) LT G.ZTHR) OR ((ABS(ITF.VMD) LT G.ZTHR) OR (ITF.TAUV LT ITF.TRTRU)) (ITF.TAUV LT ITF.TRTRU)) (ITF.TAUV LT ITF.TRTRU)) THEN SET ZHIT THEN SET ZHIT THEN SET ZHIT ELSE CLEAR ZHIT ELSE CLEAR ZHIT ELSE CLEAR ZHIT IF (ZHIT EQ $TRUE AND IF (ZHIT EQ $TRUE AND IF (ZHIT EQ $TRUE AND ABS(ITF.ZDINT) GT P.MAXZDINT ABS(ITF.ZDINT) GT P.MAXZDINT ABS(ITF.ZDINT) GT P.MAXZDINT THEN CLEAR ZHIT THEN CLEAR ZHIT THEN CLEAR ZHIT Surveillance Advisory Logic Display • Intruder detection • Alert criteria • Aural annunciation • Position tracking • Advisory selection • Advisory display M.S Thesis Defense- 15 AJW 12/1/14

Markov Decision Process (MDP) Framework for sequential decision problems • State space 1 +1 +5 – Set of all possible states 0.1 A B • Action space – Set of all possible actions 0.6 • Dynamic model – State transition probabilities 0.4 A A 0.9 • Reward model 3 2 B B – Reward for making transitions 0.7 0.3 Objective is to maximize reward -10 Both TCAS and ACAS X selected system states by hand and have not quantified the utility of each state M.S Thesis Defense- 16 AJW 12/1/14

Dynamic Programming (DP) DP is an efficient way to solve an MDP Expected value    ( , ) ( , ) ( ' | , ) ( ' ) Q s a R s a P s s a V s ' s  ( ) max ( , ) V s Q s a a • DP is an iterative process for computing the expected value when starting from each state • Best action can be derived directly from expected value M.S Thesis Defense- 17 AJW 12/1/14

ACAS X Independent Actions Advisory Separation Collision + = Logic Provision Avoidance Horizontal Actions Vertical Actions Meets SAA Independent Actions Horizontal and vertical logics implemented independently due to “curse of dimensionality” and memory constraints. Capability gap of a joint horizontal and vertical action set logic exists. M.S Thesis Defense- 18 AJW 12/1/14

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 19 AJW 12/1/14

Quantifying Potential MDPs Objective: Instead of selecting states by hand, develop a quantitative approach for selecting states to enable joint horizontal and vertical action space • The aircraft avoidance problem has two simple questions – What is the risk of collision with another aircraft? – If the risk is sufficient, what is the optimal action to minimize risk? • MDP must characterize problem with regards to NMAC* – Safety is easily defined numerically and aircraft agonistic – Operational constraints are more abstract and aircraft specific • Develop simulation-based framework to explore MDPs – Generate MDPs without coding new dynamics – Identify the memory efficiency for each state – Assess potential of space aggregation *Lost of separation ± 500 ft horizontal and ± 100 ft vertically M.S Thesis Defense- 20 AJW 12/1/14

Conceptualizing Collision Avoidance Safety Component • Numerically defined by three parameters – NMAC is used as a surrogate for collisions – NMAC is a binary state and defined the encounter geometry at the closet point of approach (CPA) – A larger time to CPA is associated with smaller risk of collision Parameter NMAC Definition Unit Time to CPA 0 Seconds Horizontal miss distance at CPA ±500 Feet Vertical miss distance at CPA ±100 Feet • Safety component is aircraft independent – NMAC definition doesn’t change if flying manned or unmanned – Since numeric, very easy to encode into MDP M.S Thesis Defense- 21 AJW 12/1/14

Conceptualizing Collision Avoidance Operational Component • UASs integration invalidates many historical assumptions – Operational constraints now widely vary between aircraft – TCAS assumes only manned aircraft with specific dynamics • Operational component isn’t easily quantified numerically – A specific state can’t be designated as “bad” or “good” – Component decomposes into three questions 1. How much does each action reduce the safety risk? – Quantifies the relationship between the action and separation 2. What is the smallest time to initiate each action? – Should only select an action when the NMAC risk is sufficient 3. What is the expected time to complete each action? – Short alerts appear unneeded and long alerts impair the mission M.S Thesis Defense- 22 AJW 12/1/14

Conceptualizing Collision Avoidance • Problem composed of safety and operational elements – The safety component quantifies the collision risk – The operational component quantifies selecting an action to minimize the risk • 30-60s time window to “escape” dangerous and avoid collision – Assumes ATC separation has failed and will provide no assistance – Validated using ownship’s vertical rate 𝒊 𝒑 and NMAC definition* = 𝟐𝟏𝟏 ft = 𝟔𝟏𝟏ft 𝒖 𝒘 𝒊 𝒑 𝒖 𝒘 𝒊 𝒑 ft s ft s 𝒊 𝒑 𝒊 𝒑 Don’t want to just narrowly 𝒖 𝒘 𝟑𝟔. 𝟏 ft s 𝒖 𝒘 𝟑𝟔. 𝟏 ft s = 𝟓 s = 𝟑𝟏s prevent an NMAC 𝒖 𝒘 𝟐𝟑. 𝟔 ft s 𝒖 𝒘 𝟐𝟑. 𝟔 ft s = 𝟗 s = 𝟓𝟏 s *Lost of separation ± 500 ft horizontal and ± 100 ft vertically M.S Thesis Defense- 23 AJW 12/1/14

Quantifying Potential MDPs Information Theory and NMAC Entropy • Shannon’s entropy characterizes the randomness (uncertainty) of an event given a state – Entropy is superadditive, enabling independent analysis of states – Identify states with large amount of information about NMAC events 𝑰 𝒀 = − 𝑸 𝒚 𝒋 𝐦𝐩𝐡 𝟑 𝑸 𝒚 𝒋 𝒋 • Aircraft avoidance is already quantified by an event, NMAC – NMAC is a binary state where NMAC = true is considered very “bad” – Entropy has sporadically been applied to aviation safety systems 𝑰 = 𝑸 NMAC = 𝟐 𝐦𝐩𝐡 𝟑 𝑸 𝑶𝑵𝑩𝑫 = 𝟐 + 𝑸 NMAC = 𝟏 𝐦𝐩𝐡 𝟑 𝑸 NMAC = 𝟏 M.S Thesis Defense- 24 AJW 12/1/14

Quantifying Potential MDPs Memory and Matrix Representation • A full (dense) matrix allocates MATLAB Memory memory for each element 10000 – TCAS and ACAS X current Full Sparse representations 1000 Megabytes Required • A sparse matrix only stores 100 nonzero elements and indices – No operations on zero elements 10 – Potential for exponential memory savings over dense matrices 1 0.1 • Sparsity corresponds to loosely coupled systems 0.01 – Typical of physical dynamics 0 25 50 75 100 Percentage of nonzero elements M.S Thesis Defense- 25 AJW 12/1/14

Simulation Framework The simulation framework facilitates the generation of MDPs based on Monte Carlo simulations. MDPs can be generated and evaluated using any combination of states without code change*. • Contrasts current ACAS X approach of calculating dynamics and determining transitions during optimization – Thesis approach represents dynamics as a large transition matrix – New approach leverages higher fidelity simulation dynamics • Pair memory with NMAC entropy to measure computational efficiency of states and MDPs – Compare different states and discretizations – “Is a fine discretization of X better than a course discretization of Y?” *Given computational memory constraints M.S Thesis Defense- 26 AJW 12/1/14

Simulation Framework Encounter Models • Encounter model describe the nominal encounter situation without a collision avoidance system – Based on continuous radar data feed from the U.S. Air Force 84 th Radar Evaluation Squadron (RADES) (~15 GB of data/day) – Represented as a probabilistic Bayesian network • Encounter models not used for MDP dynamics due to limited number of variables – Intended use to describe aircraft’s general behavior – Limited by curse of dimensionality (7-15 states depending on model) • Sample models to build Monte Carlo simulation and record all aircraft states at each time step in simulation – Recorded significantly more data than traditional MIT LL analyses – Enabled new perspective on relationship between aircraft states M.S Thesis Defense- 27 AJW 12/1/14

Simulation Framework Associate Arrays and D4M • Leverages the Dynamic Distributed Dimensional Data Model (D4M) to process Monte Carlo data into MDP transition matrices – Uniform mathematical framework based on associate arrays – Doesn’t require a priori knowledge of data D4M Feature Description and Benefits Row store Processing efficient, enables constant time look up Sparsity Memory efficient, only stores non-empty columns Unlimited columns Processing not limited by discretizations or number of states High performance Efficient processing through parallel and distributed architecture • Identify potential high quality MDPs based on NMAC entropy and memory requirements – Optimize MDP and evaluate feasibility (shape optimization) – Determine if NMAC entropy is a good surrogate for algorithm feasibility and performance M.S Thesis Defense- 28 AJW 12/1/14

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 29 AJW 12/1/14

MDP Formulation • The MDP requires an action space, state space, and costs • Actions – Explore the relationship between actions and control inputs – Identify differences between manned and unmanned performances • States – Quantify both the horizontal and vertical axes – Uniquely represent different encounter geometries • Costs – Deter behavior that increases the risk of NMAC – Minimize alert duration and rate with respect to safety M.S Thesis Defense- 30 AJW 12/1/14

Action Space Variety of Potential Actions Depending upon the aircraft, there are a wide variety of actions • Issue a waypoint to α ft • Climb to an altitude of α ft • Implement a pitch rate of Y deg • Climb at X ft s s • Cause a pitch acceleration of Z ft • Climb to α ft at X ft s 𝒕 𝟑 Some controls produce more deterministic behavior than others M.S Thesis Defense- 31 AJW 12/1/14

Action Space • Evaluate two different action spaces to represent different aircraft capabilities – For simplicity, only vertical rates differ between spaces – A common action space across all aircraft isn’t feasible nor practical Manned Action Space UAS Action Space Command Description Command Description COC Clear of conflict COC Clear of conflict CL1500 Climb at 1500 ft/min CL750 Climb at 750 ft/min DE1500 Descend at 1500 ft/min DE750 Descend at 750 ft/min L3 Left turn at 3 deg/s L3 Left turn at 3 deg/s R3 Right turn at 3 deg/s R3 Right turn at 3 deg/s M.S Thesis Defense- 32 AJW 12/1/14

State Space • State space must contain sufficient information to represent how each action influences the state-transition probabilities • Decompose into a vertical axis and horizontal plane – Treat each space independently – Consider state as part of an angle / magnitude vector perspective • Aviation systems are typically a mix of coordination systems M.S Thesis Defense- 33 AJW 12/1/14

TCAS / ACAS Xa State Space 𝑻 = 𝜠𝒊, 𝒊 𝒑 , 𝒊 𝒋 , 𝝊 𝒊 , 𝒕 𝑺𝑩 Intruder 𝒊 𝒋 𝜠𝒊 𝒕 𝑺𝑩 𝒊 𝒑 Ownship 𝝊 𝒊 Vertical States Other States State Name State Name 𝜠𝒊 𝝊 𝒊 Relative altitude Time to no horizontal separation 𝒊 Ownship vertical rate 𝒑 𝒕 𝑺𝑩 Ownship RA state 𝒊 Intruder vertical rate 𝒋 M.S Thesis Defense- 34 AJW 12/1/14

ACAS Xu State Space 𝑻 = 𝑺 𝒊 , 𝜾, 𝚾, 𝐖 𝐩 , 𝐖 𝐣 , 𝝊 𝒘 , 𝑻 𝑺𝑩 𝚾 𝑻 𝒋 𝑻 𝒑 𝒕 𝑺𝑩 Intruder 𝒔 𝒊 𝜾 Ownship Position States Other States State Name State Name 𝒔 𝒊 𝑾 𝒑 Range to intruder Ownship speed 𝜾 𝑾 𝒋 Intruder bearing Intruder speed 𝚾 𝒕 𝑺𝑩 Intruder relative heading Ownship RA state 𝝊 𝒘 Time to no vertical separation * Not illustrated on horizontal plane M.S Thesis Defense- 35 AJW 12/1/14

Space Aggregation • An aggregate feature is a metamodel that combines multiple states to reduce memory requirements – Often generated through simple operations (i.e. sum) over states – 𝝊 is an aggregate feature that combines range and range rate 𝝊 𝒊 = 𝒔 𝒊 𝒔 𝒊 𝝊 𝒘 = 𝚬𝐢 𝚬𝐢 • Feature-based hard aggregation constructs a nonlinear (piecewise constant) feature-based architecture – Partition space based on “similar” features between states – Associates original state with unique aggregate state / subset – Alternatives associate aggregate states with original space or both Feature Generate Extraction MDP States States Features Aggregate States *Illustration reproduced from Dynamic Programming and Optimal Control: M.S Thesis Defense- 36 AJW 12/1/14 Approximate Dynamic Programming by Dimitri P. Bertsekas

Space Aggregation Potential Memory Reduction 𝒊 𝒑 = −𝑫 𝒊 𝒋 = 𝟏 𝒊 𝒋 = −𝑫 𝒊 𝒋 = 𝑫 𝒊 𝒑 = 𝟏 𝒊 𝒑 = 𝑫 𝒊 𝒑 + 𝒊 𝒋 = 𝟏 𝒊 𝒑 + 𝒊 𝒋 = 𝟏 𝒊 𝒑 + 𝒊 𝒋 = 𝟏 𝒊 𝒑 , 𝒊 𝒋 requires 𝒊 𝒑 , 𝒊 𝒑 + 𝒊 𝒋 requires 3 X 3 = 9 elements 3 X 1 = 3 elements 𝒊 𝒑 ∈ −𝑫, 𝟏, 𝑫 𝒊 𝒑 ∈ −𝑫, 𝟏, 𝑫 𝒊 𝒋 ∈ −𝑫, 𝟏, 𝑫 𝒊 𝒑 + 𝒊 𝒋 ∈ 𝟏 M.S Thesis Defense- 37 AJW 12/1/14

Costs • Costs defined as indicator functions multiplied by some constant – ACAS X has leveraged advance cost techniques such as surrogate models and online costs – Alert behavior is “controlled” via the costs • Costs remained simple to focus on state and action objectives – Reversals and strengthens not included – Additional costs will be required for operational suitability Cost When to apply NMAC For states associated with an NMAC Horizontal Alert Alerting for a left or right turn Vertical Alert Alerting for a climb or descend COC Neither a horizontal or vertical alert is issued M.S Thesis Defense- 38 AJW 12/1/14

Costs NMAC Horizon • Develop alerting region via simulation – Define a large “good” region compared to a small “bad” region – Leverage understanding of collision avoidance dynamics • Quantify NMAC risk through a “NMAC horizon” binary variable – 𝝃 𝒖 𝑻 𝟐 if NMAC occurs within 𝒖 seconds 𝟏 otherwise – Ability to assign an NMAC cost for spaces without 𝚬𝐢, 𝐬 𝐢 NMAC Horizon Approach ACAS X Approach Very “good” region 𝜠𝒊 𝜠𝒊 Very “bad” region 𝒔 𝒊 𝒔 𝒊 NMAC NMAC M.S Thesis Defense- 39 AJW 12/1/14

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 40 AJW 12/1/14

Implementation 1. Monte Carlo Simulations 2. Simulation Processing 3. Calculate State-Transitions 4. Generate Cost Matrix State 5. Optimize Policy t 6. Evaluate Policy M.S Thesis Defense- 41 AJW 12/1/14

Monte Carlo Simulations CASSATT* • Developed to help certify TCAS Version 7.1 and currently used in FAA, DoD, and DHS studies • Implements a point-mass dynamic model Simulated 500,000 correlated encounters and recorded 52 simulation states each second *Collision Avoidance System Safety Assessment Tool (CASSATT) M.S Thesis Defense- 42 AJW 12/1/14 MS-57023B

Simulation Processing and State-Transition Generation Simulation produced ≈ 6,970,000,000 numeric doubles* Previous processing capability supported ≈ 13,000,000 doubles • Process raw simulation files into associative arrays for D4M – One time processing of 13,404 files with ≈10,000 lines per file – Stored in MATLAB .mat file and require from 0.09 — 1.2 Megabytes 1. Generated a sorted matrix with all unique state combinations and a kd-tree for each individual state 2. Preallocate state-transition matrix as an empty sparse matrix. 3. Filter relevant states using D4M and the nearest-neighbor discretization point for each simulation state are determined 4. Index into state-transition using nearest-neighbor indices Update state-transition matrix *To author’s knowledge: largest individual aircraft M.S Thesis Defense- 43 AJW 12/1/14 simulation data set ever generated

Generate Cost Matrix 𝑻 = 𝜠𝒊 NMAC Cost Matrix • Inefficient to define costs and allocate nonzero elements for states not observed in state- transition matrix 𝜠𝒊 𝒕′ (ft) • Using nonzero linear index record to efficiently assign costs – Ability to implement probabilistic costs using nonzero index record – Identifying indexes requires the most memory and computation 𝜠𝒊 𝒕 (ft) • By using linear indexing, cost Naïve NMAC Cost Structure matrix generation requires up to State-Transition observations a few minutes M.S Thesis Defense- 44 AJW 12/1/14

Optimize and Evaluate Policy • Optimization via DP-policy iteration using Inra MDP toolbox Optimization Parameter Value Max Iterations 100 Discount value 0.9 # of CPUs 1 • Policy post-processed dilated and eroded – Minimal processing using diamond morphological structuring element sized less than 1% of state space – Addressed noise or lack of individual elements in state-transition • Policy implemented as a look-up table in CASSATT and evaluated via Monte Carlo simulation – Safety metrics driven by NMAC – Operational metrics driven be alert rate M.S Thesis Defense- 45 AJW 12/1/14

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 46 AJW 12/1/14

Transition Matrix Generation • Transition matrix represents the dynamics from state s s to state s ’ – Research success dependent upon generation of sufficient transition matrices from simulation data – Must be smooth without discontinuities • 𝜠𝒊 𝒕 distribution validates ability to create transition matrices Low probability of extreme 𝜠𝒊 Altitude is independent with opposite vertical rate sense of turn sense 𝜠𝒊 𝒕 Distribution* 10 − 2 Log(Pr(s)) 10 − 4 10 − 6 − 10,000 − 7500 − 5000 − 2500 0 2500 5000 7500 10,000 𝜠𝒊 𝒕 (ft) COC DES1500 CL1500 L3 R3 *Manned action space M.S Thesis Defense- 47 AJW 12/1/14

Transition Matrix Generation 𝜠𝒊 𝒕, 𝒕′ Sparsity • 𝜠𝒊 𝒕, 𝒕′ distribution* clearly illustrates sparsity principle – Sparsity decreases as discretization becomes more course – Most states are at least 95% sparse given a reasonable discretization 10,000 Observed transition from Monte Carlo simulation 5000 𝜠𝒊 𝒕′ (ft) NMAC 0 region − 5000 Unrealistic to transition from 𝜠𝒊 ≥ 𝟔𝟏𝟏𝟏 to 𝜠𝒊 ≤ −𝟔𝟏𝟏𝟏 in one second − 10,000 − 10,000 − 5000 0 5000 10,000 𝜠𝒊 𝒕 (ft) *Manned action space M.S Thesis Defense- 48 AJW 12/1/14

Range 𝒔 𝒕 𝜠𝒊 𝒔 𝒊 If 𝜠𝒊 ≤ 𝝑, then 𝒔 𝒕 ∼ 𝒔 𝒊 NMAC Entropy 24 21 Negible difference Entropy (bits) between 𝒔 𝒊 , 𝒔 𝒕 18 15 12 Horizontal plane 9 provides more information 6 than vertical axis 𝚬𝐢 3 0 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 10 10 # of discretized points 𝒔 𝒊 𝒔 𝒕 𝚬𝒊 M.S Thesis Defense- 49 AJW 12/1/14

Range 𝚬𝐢, 𝐬 𝒊 • Fine uniform discretizations NMAC Entropy are not feasible – Basic 𝚬𝐢, 𝐬 𝒊 spaces can use up to 100GB of memory • Increasing discretization of 𝐬 𝒊 has greater effect on NMAC entropy but also on memory required Memory Required (MB) • Finer discretizations of 𝜠𝒊 have severe diminishing returns – Aircraft rarely deviate from cruising altitude M.S Thesis Defense- 50 AJW 12/1/14

Range Rates A single aircraft can’t directly control a rate, it is subject to both aircraft NMAC Entropy Basic vertical rate 15 sense isn’t sufficient Like range states, negible Entropy (bits) difference between 𝒔 𝒊 , 𝒔 𝒕 (i.e. simply knowing 𝚬𝐢 is changing isn’t very useful) 10 5 0 10 2 10 3 10 4 10 5 10 6 10 7 10 8 −𝟒𝟏𝟏𝟏, 𝟑𝟘𝟘𝟘, 𝚬𝐢 = Memory Required (MB) … , 𝟑𝟘𝟘𝟘, 𝟒𝟏𝟏𝟏 Memory (MB) 10 1 10 − 1 10 − 3 10 2 10 3 10 4 10 5 10 6 10 7 10 8 # of discretized points 𝒔 𝒊 𝒔 𝒕 𝚬𝐢 𝒔 𝒕 = spherical range rate 𝚬𝐢 = vertical range rate M.S Thesis Defense- 51 AJW 12/1/14 𝒔 𝒊 = horizontal range rate

Simple 𝝊 Utility of a 𝝊 state depends upon on the discretized edges NMAC Entropy 9 Similar to range and range rate, Entropy (bits) no difference between 6 spherical and horizontal 3 Extending edge from 50s to 100s significantly 0 improves NMAC entropy 10 2 10 3 10 4 Memory Required (MB) Memory (MB) 10 0 Space is very dense, 10 -1 memory requirements 10 -2 agnostic of different 𝝊 states and discretizations 10 -3 10 2 10 3 10 4 # of discretized points 𝝊 𝒕,𝟔𝟏 𝝊 𝒊,𝟔𝟏 𝝊 𝒘,𝟔𝟏 𝝊 𝒕,𝟐𝟏𝟏 𝝊 𝒊,𝟐𝟏𝟏 𝝊 𝒘,𝟐𝟏𝟏 𝝊 𝒕 = 𝒔 𝒕 𝒔 𝒕 𝝊 𝒊 = 𝒔 𝒊 𝒔 𝒊 𝝊 𝒘 = 𝚬𝐢 M.S Thesis Defense- 52 𝚬𝐢 AJW 12/1/14

Simple 𝝊 NMAC Entropy vs. Probability NMAC Entropy of a given 𝝊 𝒊 (𝒕) to 𝝊 𝒊 (𝒕′) Low 𝝊 𝒊 (𝒕) values are likely to transition to · 10 − 2 𝝊 𝒊 𝒕 = 𝟐𝟏 s similar low 𝝊 𝒊 (𝒕′) values where randomness of 2 bits 𝝊 𝒊 𝒕 = 𝟘𝟏 s an NMAC is smaller 1 0 0 10 20 30 40 50 60 70 80 90 100 Large 𝝊 𝒊 (𝒕) values 𝝊 𝒊 𝒕′ provide less information about transitioning Probability of transitioning from 𝝊 𝒊 (𝒕) to 𝝊 𝒊 (𝒕′) to an NMAC Probability 10 -4 10 -7 Probability of transitioning to lower 𝝊 𝒊 increases 10 -10 with the value of 𝝊 𝒊 (𝒕) 0 10 20 30 40 50 60 70 80 90 100 𝝊 𝒊 𝒕′ 𝟏 s 𝟐𝟏 s 𝟑𝟏 s 𝟒𝟏 s 𝟓𝟏 s 𝟔𝟏 s 𝟕𝟏 s 𝟖𝟏 s 𝟗𝟏 s 𝟘𝟏 s 𝝊 𝒕 = 𝒔 𝒕 𝒔 𝒕 𝝊 𝒊 = 𝒔 𝒊 𝒔 𝒊 𝝊 𝒘 = 𝚬𝐢 M.S Thesis Defense- 53 𝚬𝐢 AJW 12/1/14

Angles Angular states are very memory intensive due to wrap around NMAC Entropy 12 Angular states provide Entropy (bits) significant entropy 9 but with large memory requirements 6 3 𝜾 𝒕 yields the least 0 10 4 10 5 NMAC entropy but doesn’t require the Memory Required (MB) Memory (MB) least memory 10 1 10 0 10 -1 Angular states have 10 -2 no edge due to 10 4 10 5 𝟑𝝆 wrap around # of discretized points 𝜾 𝒕 𝚬𝝎 𝝎 𝑺 𝜠𝝎 = relative heading 𝜾 𝒕 = inclination angle M.S Thesis Defense- 54 AJW 12/1/14 𝝎 𝑺 = angle of resultant

Angles Inclination Angle 𝜾 𝒕 𝜠𝒊 𝒔 𝒕 , 𝜾 𝒕 Distribution • Inclination angle ( θ s ) is not a good 180 state for an aircraft avoidance MDP 135 – Negligible change between 𝜠𝒊 states when 𝒔 𝒕 is sufficiently (deg) 90 large 𝒔 𝒕 ≥ 𝟕𝟏𝟏𝟏 ft 45 • Generates some of the lowest NMAC entropy overall 0 0 1500 3000 4500 6000 • Mathematical intuition matches 𝒔 𝒕 (ft) NMAC entropy result − 6000 − 3000 0 3000 6000 Small 𝒔 𝒕 Large 𝒔 𝒕 𝜾 𝒕 = 𝜹 𝜠𝒊 = 𝑫 𝜾 𝒕 ≪ 𝜹 𝜠𝒊 = 𝑫 M.S Thesis Defense- 55 AJW 12/1/14

Airspeed Difficult to identify “similar” features of airspeed states NMAC Entropy 15 𝑺 yields Entropy (bits) 12 greatest entropy Aggregating operation has little effect 9 6 𝒘 𝒑 𝒘 𝒋 ∗ shows that 3 complicated isn’t 0 necessary better 10 1 10 2 10 3 10 4 10 5 10 6 10 7 Memory Required Memory (MB) 10 1 𝑺 requires 10 -1 more memory 10 -3 10 1 10 2 10 3 10 4 10 5 10 6 10 7 # of discretized points 𝒘 𝒑 ∗ 𝒘 𝒑 𝒘 𝒋 𝒘 𝒑 𝒘 𝒑 + 𝒘 𝒋 𝒘 𝒑 − 𝒘 𝒋 𝒘 𝒑 − 𝒘 𝒋 𝑺 𝒘 𝒑 + 𝒘 𝒋 𝒘 𝟏 = ownship′s airspeed ∗ 𝒘 𝒑 𝒘 𝒋 𝒘 𝒑 𝒘 𝒋 <𝟐 − 𝒘 𝒋 𝒘 𝒑 M.S Thesis Defense- 56 = 𝟐 𝒘 𝒑 𝒘 𝒋 + 𝟐 𝒘 𝒑 𝒘 𝒋 AJW 12/1/14 𝒘 𝒋 = intruder′s airspeed ≥𝟐

Vertical Rates Significant potential for adopting aggregate features NMAC Entropy Aggregate features 6 Entropy (bits) yield greater NMAC entropy 3 No difference between 𝒊 𝒑 and 𝒊 𝒋 indicates that an individual perspective 0 isn’t as useful 10 2 10 3 10 4 10 5 as a “system” view Memory Required (MB) Memory (MB) 10 -1 Aggregate features 10 -2 require less memory 10 -3 given discretization 10 2 10 3 10 4 10 5 # of discretized points 𝒊 𝒑 𝒊 𝒑 𝒊 𝒋 𝒊 𝒑 + 𝒊 𝒋 𝒊 𝒑 + 𝒊 𝒋 𝒊 𝒋 𝒊 𝒑 = ownship′s vertical rate 𝒊 𝒋 = intruder′s vertical rate M.S Thesis Defense- 57 AJW 12/1/14

Vertical Rates 𝚬𝐢, 𝒊 𝒑 vs. 𝚬𝐢, 𝒊 𝒑 + 𝒊 𝒋 NMAC Entropy Aggregate feature preserves more NMAC entropy as 𝚬𝐢 becomes more course Memory requirements grow faster than NMAC entropy. Range grows from Memory Required (MB) (8 bits, 𝝑 MB) to (12 bits, 24 MB) Similar memory requirements for 𝚬𝐢, 𝒊 𝒑 and 𝚬𝐢, 𝒊 𝒑 + 𝒊 𝒋 𝚬𝐢 = vertical separation 𝒊 𝒑 + 𝒊 𝒋 = vertical rates sum M.S Thesis Defense- 58 AJW 12/1/14

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 59 AJW 12/1/14

Manned 𝚬𝐢, 𝐬 𝐢 DES1500 CL1500 L3 R3 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟓𝟔 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟒𝟏 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟐𝟔 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 10 -2 M.S Thesis Defense- 60 AJW 12/1/14 0 0 . 2 0 . 4 0 . 6 0 . 8 1 1 . 2 1 . 4 1 . 6 1 . 8

Manned 𝚬𝐢, 𝐬 𝐢 NMAC Risk Across 𝐬 𝒊 CL1500 R3 2500 2500 1250 1250 𝝃 𝟓𝟔 𝜠𝒊 𝜠𝒊 0 0 -1250 -1250 -2500 -2500 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 2500 2500 As horizon Horizontal 1250 1250 shrinks, so does 𝝃 𝟒𝟏 𝜠𝒊 𝜠𝒊 maneuvers reduce 0 0 the region of risk along 𝐬 𝒊 better -1250 -1250 NMAC risk -2500 -2500 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 2500 2500 1250 1250 𝝃 𝟐𝟔 𝜠𝒊 𝜠𝒊 0 0 -1250 -1250 -2500 -2500 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 10 -2 M.S Thesis Defense- 61 AJW 12/1/14 0 0 . 2 0 . 4 0 . 6 0 . 8 1 1 . 2 1 . 4 1 . 6 1 . 8

Manned 𝚬𝐢, 𝐬 𝐢 NMAC Risk Across 𝚬𝐢 DES1500 CL1500 L3 R3 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟓𝟔 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 NMAC risk is dependent Risk along the vertical axis upon the vertical action. is agnostic to the type of Climbing when 𝚬𝐢 ≤ 𝟏 or horizontal maneuver descending when 𝚬𝐢 ≥ 𝟏 is more risky 10 -2 M.S Thesis Defense- 62 AJW 12/1/14 0 0 . 2 0 . 4 0 . 6 0 . 8 1 1 . 2 1 . 4 1 . 6 1 . 8

U 𝐁𝐓 𝚬𝐢, 𝐬 𝐢 DES750 CL750 L3 R3 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟓𝟔 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟒𝟏 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 2500 2500 2500 2500 1250 1250 1250 1250 𝝃 𝟐𝟔 𝜠𝒊 𝜠𝒊 𝜠𝒊 𝜠𝒊 0 0 0 0 -1250 -1250 -1250 -1250 -2500 -2500 -2500 -2500 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 · 10 4 · 10 4 10 -2 M.S Thesis Defense- 63 AJW 12/1/14 0 0 . 2 0 . 4 0 . 6 0 . 8 1 1 . 2 1 . 4 1 . 6 1 . 8

Comparing Manned and UAS NMAC Risk Horizons of 𝚬𝐢, 𝐬 𝐢 • Compare risk between action spaces using convex hulls of 𝝃 𝒖 • UAS’s limited performance significantly changes risk region – Larger 𝚬𝐢 values less risky…takes longer for an NMAC to transpire – Vertical actions have less of an effect on risk along 𝚬𝐢 …intruder can usually maneuver vertically greater 𝝃 𝟒𝟏 : R3 / R3 𝝃 𝟒𝟏 : CL1500 / CL750 2500 2500 1250 1250 𝜠𝒊 (ft) 𝜠𝒊 (ft) 0 0 -1250 -1250 -2500 -2500 0 1 2 3 4 5 0 1 2 3 4 5 · 10 4 · 10 4 𝒔 𝒊 (ft) 𝒔 𝒊 (ft) Manned action set (CL1500, R3) UAS action set (CL750, R3) M.S Thesis Defense- 64 AJW 12/1/14

Adding States 𝚬𝐢, 𝐬 𝐢 can’t describe • (0 , 1) 𝒛 encounter geometries – Corresponds to a large and 𝝆 𝟑 unacceptable alerting region 𝝆 𝟓 𝟒𝝆 𝟓 𝟘𝟏° 𝟐𝟒𝟔° 𝟓𝟔° • Adding states increases ( − 1 , 0) NMAC entropy (1 , 0) 𝝆 𝟐𝟗𝟏° 𝟏° 𝟑𝝆 𝒚 – More information about NMAC risk 𝟑𝟑𝟔° 𝟒𝟐𝟔° – Change alerting behavior 𝟖𝝆 𝟓 𝟔𝝆 𝟓 𝟑𝟖𝟏° 𝟒𝝆 𝟑 • Complete MDP must include angular and rate states (0 , − 1) M.S Thesis Defense- 65 AJW 12/1/14

Adding States NMAC Entropy Gain • Select states to add to state space based on NMAC entropy gain – Adding states will increase NMAC entropy – Select state with reasonable tradeoff between gain and increase in required memory – Approach can be automated but was not • Gain for states were similar between action sets 𝒔 𝒊 = −𝟔𝟏𝟏, −𝟓𝟏𝟏, … , 𝟔𝟏𝟏 𝚬𝝎 = 𝟏°, 𝟒°, … , 𝟒𝟕𝟏° Manned Action Set (±1500, 3 °) High gain 4 with reasonable High gain memory increase Gain (bits) 3 but significant Marginal memory increase 2 information gain 𝜾 𝒕 = 𝟏°, 𝟒°, … , 𝟐𝟗𝟏° 1 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 Memory Increase Factor Horizontal Vertical 𝒔 𝒊 = horizontal range rate 𝜾 𝒕 = inclination angle M.S Thesis Defense- 66 AJW 12/1/14 𝜠𝝎 = relative heading °)

Adding States UAS Range rate 𝒔 𝒊 example 𝚬𝐢, 𝐬 𝐢 , 𝒔 𝒊 • can identify if encounter is becoming riskier – Even course discretizations of 𝒔 𝒊 are useful – Lacks information to fully describe geometry CL750 R3 2500 2500 If aircraft are moving 1250 1250 𝒔 𝒊 = 𝜠𝒊 𝜠𝒊 away from each other and 0 0 𝟑𝟔𝟏, 𝟔𝟏𝟏 an NMAC, then risk is low -1250 -1250 -2500 -2500 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 2500 2500 Risk is significantly greater 1250 1250 𝒔 𝒊 = 𝜠𝒊 𝜠𝒊 if aircraft are moving 0 0 −𝟔𝟏𝟏, −𝟑𝟔𝟏 towards each other -1250 -1250 -2500 -2500 0 1 2 3 4 0 1 2 3 4 𝒔 𝒊 𝒔 𝒊 · 10 4 · 10 4 10 -2 M.S Thesis Defense- 67 AJW 12/1/14 0 0 . 2 0 . 4 0 . 6 0 . 8 1 1 . 2 1 . 4 1 . 6 1 . 8

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 68 AJW 12/1/14

Metrics • Risk Ratio*: the relative benefit of equipping with the system – The primary safety metric used for aviation safety evaluations – 5NMAC risk ratio is the relative benefit protecting a volume 5X an NMAC – Risk ratio components: • Unresolved: NMAC occurs with and without SAA • Induced: NMAC induced by SAA that would not occur otherwise < 1 System increases safety P(NMAC with Sense and Avoid | Encounter) Risk Ratio = > 1 System creates safety hazard P(NMAC without Sense and Avoid | Encounter) = 1 System has no net affect on safety • Alert Duration: the average alert duration – Consistently long alerts are unacceptable for operational use – Conservative logics will alert when risk is relatively low with long durations A logic and corresponding MDP is considered feasible if the NMAC risk ratio is less than 0.1 with an average alert duration of 60 or less seconds M.S Thesis Defense- 69 AJW 12/1/14 • –

Baseline 𝚬𝐢, 𝐬 𝐢 Policy Generation 𝚬𝐢 = −𝟑𝟏𝟏𝟏, −𝟐𝟘𝟔𝟏, … , 𝟐𝟘𝟔𝟏, 𝟑𝟏𝟏𝟏 𝒔 𝒊 = 𝟏, 𝟔𝟏𝟏, … , 𝟐𝟘𝟔𝟏𝟏, 𝟑𝟏𝟏𝟏𝟏 • Optimal policy strongly favors vertical maneuvers – Challenging to determine the effect of a horizontal maneuver – Reducing horizontal maneuver costs increases the number of horizontal maneuvers Policy defined by 𝝃 𝟒𝟏 and equal alert costs Very “long” alert behavior along 𝒔 𝒊 axis 𝚬𝐢 = vertical separation M.S Thesis Defense- 70 AJW 12/1/14 𝒔 𝒊 = horizontal range

Baseline 𝚬𝐢, 𝐬 𝐢 Evaluation 𝝃 𝒖 Horizon (s) 5NMAC Risk Ratio NMAC Risk Ratio Mean Alert Duration (s) 15 0.3851 0.2474 25 30 0.1607 0.0177 150 45 0.0665 0.0181 220 • A 𝚬𝐢, 𝐬 𝐢 MDP produces a conservative logic • At reasonable NMAC risk horizons, can protect against most NMAC encounters but with unacceptable alerting behavior • Reducing time horizon to 15s and relying on “last - second” maneuvers minimizes alert duration but is unsafe – Resolves many “easy” NMAC encounters – Last-second maneuvers also protect larger 5NMAC volume 𝚬𝐢 = vertical separation M.S Thesis Defense- 71 AJW 12/1/14 𝒔 𝒊 = horizontal range

𝚬𝐢, 𝐬 𝐢 , 𝒔 𝒊 Policy Generation 𝚬𝐢 = −𝟑𝟏𝟏𝟏, −𝟐𝟘𝟔𝟏, … , 𝟐𝟘𝟔𝟏, 𝟑𝟏𝟏𝟏 𝒔 𝒊 = 𝟏, 𝟔𝟏𝟏, … , 𝟐𝟘𝟔𝟏𝟏, 𝟑𝟏𝟏𝟏𝟏 𝒔 𝒊 = −𝟔𝟏𝟏, −𝟓𝟏𝟏, … , 𝟓𝟏𝟏, 𝟔𝟏𝟏 • Inclusion of 𝒔 𝒊 significantly changes alerting behavior – Still favors vertical maneuvers (lack of angular information) – Policy requires 1.146 MB of memory Policy defined by 𝝃 𝟒𝟏 and equal alert costs Even coarse 𝒔 𝒊 discretizations substantially change the alerting behavior 𝚬𝐢 = vertical separation 𝒔 𝒊 = horizontal range rate M.S Thesis Defense- 72 AJW 12/1/14 𝒔 𝒊 = horizontal range

𝚬𝐢, 𝐬 𝐢 , 𝒔 𝒊 Evaluation 𝝃 𝒖 Horizon (s) 5NMAC Risk Ratio NMAC Risk Ratio Mean Alert Duration (s) 15 0.2134 0.1843 14 30 0.1356 .1025 33 45 0.0733 .0562 50 • MDP produces feasible collision avoidance logic • Demonstrates that defining the alert volume, an operational component, as a function of 𝝃 𝒖 , a safety component, is feasible • Additional states required for operational use – Lack of angular states limits the ability to characterize encounter geometry – Last-second maneuvers also protect larger 5NMAC volume 𝚬𝐢 = vertical separation 𝒔 𝒊 = horizontal range rate M.S Thesis Defense- 73 AJW 12/1/14 𝒔 𝒊 = horizontal range

𝚬𝐢, 𝐬 𝐢 , 𝒔 𝒊 , 𝜷, 𝒊 𝒑 + 𝒊 𝒋 Policy Generation 𝚬𝐢 = −𝟑𝟏𝟏𝟏, −𝟐𝟘𝟔𝟏, … , 𝟐𝟘𝟔𝟏, 𝟑𝟏𝟏𝟏 𝒔 𝒊 = 𝟏, 𝟔𝟏𝟏, … , 𝟐𝟘𝟔𝟏𝟏, 𝟑𝟏𝟏𝟏𝟏 𝒔 𝒊 = −𝟔𝟏𝟏, −𝟓𝟏𝟏, … , 𝟓𝟏𝟏, 𝟔𝟏𝟏 𝜷 = −𝟐𝟗𝟏, −𝟐𝟒𝟔, … , 𝟐𝟒𝟔, 𝟐𝟗𝟏 𝒊 𝒑 + 𝒊 𝒋 = −𝟔𝟏, 𝟑𝟔, … , 𝟑𝟔, 𝟔𝟏 • MDP with 1,035,045 unique combinations of five geometric states and one policy state – State-transition matrix has 1,071,318,152,025 elements – Evaluated optimal policy requires 41 MB of memory – State-transition matrix can require tens of GBs of memory • Inclusion of angular states leads to more horizontal maneuvers considered optimal but policies still favors vertical maneuvers 𝚬𝐢 = vertical separation 𝒔 𝒊 = horizontal range rate 𝒊 𝒑 + 𝒊 𝒋 = M.S Thesis Defense- 74 AJW 12/1/14 𝒔 𝒊 = horizontal range 𝜷 = bearing of intruder vertical rates sum

𝚬𝐢, 𝐬 𝐢 , 𝒔 𝒊 , 𝜷, 𝒊 𝒑 + 𝒊 𝒋 Evaluation 𝝃 𝒖 Horizon (s) 5NMAC Risk Ratio NMAC Risk Ratio Mean Alert Duration (s) 15 0.2031 .1758 20 30 0.1256 .0834 36 45 0.0937 .0651 55 • Feasibility demonstrates the utility of “system” perspective states and benefits of not decomposing into 𝝊 states • Regardless of additional states, “last - second” logics defined by 𝝃 𝟐𝟔 not safety feasible • Continue to add states via NMAC entropy approach but difficult to optimize with reasonable discretizations – Tradeoff between state flexibility and efficiency 𝚬𝐢 = vertical separation 𝒔 𝒊 = horizontal range rate 𝒊 𝒑 + 𝒊 𝒋 = M.S Thesis Defense- 75 AJW 12/1/14 𝒔 𝒊 = horizontal range 𝜷 = bearing of intruder vertical rates sum

• History of Aircraft Avoidance • MDP for Collision Avoidance • Problem Conceptualization • MDP Formulation • Implementation • Entropy and Memory Results • NMAC Horizon Results • Optimization and Evaluation • Conclusion M.S Thesis Defense- 76 AJW 12/1/14

Summary of Contributions • Applied an information theoretic approach to the aircraft avoidance problem – Quantified the information a state provides about a potential NMAC – Defined the inherent risk for each element in a state space • Demonstrated an alternative simulations-based approach to solving collision avoidance MDPs – Provides more flexibility because dynamics are observed, not coded – Produced similar alerting regions as ACAS X • Prototyped a joint horizontal and vertical action space MDPs – Illustrated preference for vertical maneuvers – Added states to baseline MDP to reach operational feasibility; safety feasibility can be achieved with a very simple state space ( 𝚬𝐢, 𝐬 𝐢 ) M.S Thesis Defense- 77 AJW 12/1/14

What went wrong • Extremely large state-transition matrices are memory intensive – Even with sparse matrices, 100’s GBs of memory could be required – Memory limits constrained the size of the MDP state space • Monte Carlo simulations and evaluations reliant on super computing architecture and systems – System maintenance led to weeks of no ability to calculating NMAC entropy, NMAC horizon, or evaluate policies – Code speed optimization took longer than anticipated (ever 0.1 second counts!) • Difficult to elegantly encourage horizontal maneuvers – No priori analysis on relationship between different maneuvers – Unable to develop aggregate feature that quantifies angular information at reduced memory requirements M.S Thesis Defense- 78 AJW 12/1/14

Future Work • Prototype simulation-based MDPs as a traditional MDP DP and solve for dynamics during optimization – No longer need to hold state-transition matrix in memory – Will lead to faster optimization times • Solve continuous, not discrete, MDP – Potential to overcome fine angular discretization problem – More representative of real world • Transition associative array processing to aviation safety assessments – More advanced safety metrics possible since entire simulation can now be processed – Identify specific state combinations that logics are robust or susceptible to M.S Thesis Defense- 79 AJW 12/1/14

Acknowledgments MIT Lincoln Laboratory Boston University Dr. Michael Owen* Dr. David Castañón* Dr. Mykel Kochenderfer Dr. Ioannis Paschalidis* Robert Klaus Dr. John Baillieul Richard Williams Dr. Mario Cabodi Vijay Gadepally Dr. Michael Caramanis Randal Guendel Dr. Prakash Ishwar Christine Parry Dr. Eric Schwartz Chansup Byun Dr. Ari Trachtenberg Gregory Hogan Austin Alexander Gregg Shoults Cali Ann Stephens Dr. Paul Breimyer Andrew Weinert Assistant Technical Staff Surveillance Systems Email: andrew.weinert@ll.mit.edu Phone: (781) 981-0986 M.S Thesis Defense- 80 *M.S Thesis Committee AJW 12/1/14

Supplemental Slides Index • Enhanced TCAS II Algorithm • Performance Metrics • Angular Rate States • Encounter Model Categories • Simulation States • Bayesian Networks • Simulation Processing Details • Encounter Model Overview • State-Transition Gen. Details • Coordination • Policy Generation with Noise • ACAS X Input • Policies with Horizontal • ACAX X Tuning Maneuvers • ACAS X Optimization • Baseline SOC • ACAS X Feedback • ACAS X Usage • ACAS X Modifications M.S Thesis Defense- 81 AJW 12/1/14

Enhanced TCAS II Algorithm Bendix Corporation Prototype 𝑻 = 𝒔 𝒊 , 𝑻 𝒑 , 𝐓 𝐣 , 𝝉 𝐄 , 𝑻 𝑺𝑩 𝒕 𝑺𝑩 𝟒𝝉 𝑬 NMAC 𝑻 𝒋 𝑻 𝒑 Ownship Intruder 𝒔 𝒊 = 𝒕 𝒑 + 𝒕 𝒋 𝝊 𝒊 Primary States Secondary States State Name State Name 𝒔 𝒊 𝝉 𝑪 Straight line range Angular rate SD 𝝉 𝑬 𝝊 𝒊 Horizontal miss estimate Horizontal tau 𝑻 𝒑 𝒕 𝑺𝑩 Ownship speed Ownship RA state 𝑻 𝒋 Intruder speed 𝟑 𝝉 𝑪 𝝉 𝑬 = 𝒕 𝒑 + 𝒕 𝒋 𝝊 𝒊 M.S Thesis Defense- 82 AJW 12/1/14

Angular Rate States 𝜠𝝎 = 𝟏° 𝝎 𝑺 = 𝟏° 𝝎 𝟏 = 𝑫° 𝝎 𝒋 = 𝑫° 𝜷 = −𝟐𝟗𝟏° 𝜠𝝎 = 𝟏° 𝜠𝝎 = 𝟏° 𝝎 𝑺 = 𝟏° 𝝎 𝑺 = 𝟏° 𝜷 = 𝟘𝟏° 𝜷 = −𝟘𝟏° 𝜠𝝎 = relative heading 𝜠𝝎 = 𝟏° 𝝎 𝑺 = angle of resultant 𝝎 𝑺 = 𝟏° 𝜷 = bearing 𝜷 = 𝟏° Difficult to represent encounter type with relative angular states. Bearing ( 𝜷 ) is advantageous because it quantifies the quadrant M.S Thesis Defense- 83 AJW 12/1/14

Simulation States • Airspeed • East position Rate • Airspeed acceleration • Heading angle • Altitude • North position • Altitude rate of change • North position rate • Bank angle • Pitch angle • Body-fixed angular vector • Pitch rate • Commanded vertical rate • Turn rate • Command turn rate • Vertical acceleration • Commanded longitude • Vertical rate acceleration • Yaw angle • East position • Yaw rate M.S Thesis Defense- 84 AJW 12/1/14

Simulation Processing Implementation Details Simulation produced ≈ 6,970,000,000 numeric doubles Largest individual aircraft simulation data set ever generated Store raw simulation result in 13,404 files with ≈10,000 lines 1. per file – File limit required for load balancing – Raw data files ranged in size from 0.18 — 2.56 Megabytes 2. Process raw files into associative arrays for D4M – An associative array is represented by a triple store of row keys, column keys, and a value – Stored in MATLAB .mat file and require from 0.09 — 1.2 Megabytes – Row key concentration of encounter set id and Monte Carlo id – Column key concentration of time, state name, and state value – Value key is always one M.S Thesis Defense- 85 AJW 12/1/14

Calculate State-Transition Implementation Details 1. Generated a matrix containing all unique state combinations – Structure is provided by sorting rows in ascending order by state – A kd-tree is generated for each individual state as well 2. Preallocate state-transition matrix as an empty sparse matrix. – A linear index matrix that records every nonzero element in the state-transition is also preallocated 3. Filter relevant states using D4M and the nearest-neighbor discretization point for each simulation state are determined – Index into state-transition using nearest-neighbor indices – Searching many single state kd-trees is faster than a large kd-tree 4. Update state-transition matrix – For UAS set, filter out observations with vertical rates greater than representative UAS vertical rates M.S Thesis Defense- 86 AJW 12/1/14

Policy Generation Simulation Noise • Optimization with observed dynamics can be noisy – Resulting policy may not be completely smooth – Smoothing that doesn’t change the character of the policy required • Mitigated through state-transition matrix smoothing or minimal dilation / erosion of policy “Holes in the policy” M.S Thesis Defense- 87 AJW 12/1/14

Policies with Horizontal Maneuvers • Without angular information, little incentive to select horizontal maneuvers – Horizontal cost needs to be significantly less than the vertical cost to encourage horizontal maneuvers • Inclusion on angular information can produce similar policies • Horizontal maneuvers “grow out” from the NMAC region when changing the cost M.S Thesis Defense- 88 AJW 12/1/14

Baseline 𝚬𝐢, 𝐬 𝐢 Evaluation: SOC Curve • A system operating characteristic (SOC) curve illustrates the relationship between safety and operational components • Since relationship between vertical and horizontal alert costs was unknown, randomly assigned alert costs during optimization and evaluated policy NMAC Risk Ratio 0 . 3 0.2 0 . 1 0 20 40 60 80 100 120 140 160 180 200 220 240 Alert Duration (s) Equal alert cost Random alert cost M.S Thesis Defense- 89 AJW 12/1/14

Performance Metrics Operational Validation Plan Meets/Exceeds Example Metrics Requirements ✓ Safety Risk ratio (unresolved, induced) ✓ Overall alert rate ✓ Corrective alert rate Disruptive alerts in normal operations ✓ (500’, 1,000’, parallels, 3NM) Operational Location distribution Suitability ✓ (airspace, airports, altitude) ✓ Operation distribution (air carriers vs. business jets) ✓ Altitude change for corrective alerts ✓ Crossing alert rate ✓ Reversal rate Performance metrics, definitions, ✓ Strengthening rate and required exit criteria ✓ Magnitude of vertical rate change Acceptability ✓ Yo-yo advisory sequences ✓ Alert timing, duration, termination ✓ Complex alert sequences And more… … M.S Thesis Defense- 90 AJW 12/1/14

Encounter Model Categories MIT LL Models Aircraft of interest • Correlated (cooperative) Discrete 1200/VFR – Prior U.S. model needed to be code updated, captures RVSM Intruder aircraft – Assumes ATC involvement Discrete code • Uncorrelated 1200-code (noncooperative surrogate) 1200/VFR – First model to capture encounters between VFR aircraft Noncooperative – Assumes no ATC involvement Conventional • Uncorrelated (unconventional aircraft) Noncooperative – Models vehicles unlikely to Unconventional carry transponders – Assumes no ATC involvement Conventional: General Aviation typical of 1200-code aircraft Unconventional: balloons, gliders, ultralights ,… Encounter model describe the nominal encounter situation without the SS/CA system M.S Thesis Defense- 91 AJW 12/1/14

Bayesian Networks • Bayesian networks graphically represent a set of random variables and their conditional dependencies • Nodes represent variables, arcs symbolize dependencies • Count each observed occurrence to build probability tables • Used to specify encounter initial conditions Conditional Probability Table P(Airspeed | Airspace, Altitude Layer) … M.S Thesis Defense- 92 AJW 12/1/14

Model Development Overview Correlated Model Airspace Statistics Feature Encounter Extraction Database Radar tracker i.e., Class D airspace 1200 ft AGL Radar Fusion 3 deg/s turn tracker tracker 1500 ft/min climb Models encounters between two aircraft 80 kt airspeed ~800,000 encounters; 16 variables 0 kt/s acceleration Radar tracker Uncorrelated Model Models nominal 134 radar sites VFR flight Feature ~15+ GB per day VFR Track ~100,000 VFR flight Extraction hours Database 6 variables Unconventional Models unconventional Model aircraft — e.g., GPS GPS Post- Feature paragliders, tracker Processing Extraction balloons, skydivers ~100,000 flight hours 5 variables M.S Thesis Defense- 93 AJW 12/1/14

Airspace Model Publications, Briefings, and Users Documented in Academic Journals and Conferences AIAA Journal of Guidance, Control, and Dynamics March 2010 USA/Europe Air Traffic Management Research July 2009 and Development Seminar Lincoln Laboratory Journal December 2008 8th Integrated Communications, Navigation and May 2008 Surveillance Conference AIAA Guidance, Navigation, and Control August 2008 Conference and Exhibit AUVSI Unmanned Systems North America August 2010 Briefed to and Leveraged by Standards Organizations RTCA SC-203 (Unmanned Aircraft/SAA) NATO Flight in Non-Segregated Airspace (FINAS) FAA UAS SAA Workshop Used by Industry, Government, R&D Centers, Academia NGC, Honeywell, General Atomics, AeroVironment Industry Utopia Compression, Bihrle, MTSI, SELEX Galileo • Model reports publically available Government/ AFRL, SIMAF, MITRE, JHU/APL, EDA MIDCAS • Data tables, software for sampling models, R&D Centers and density model available upon request Academia UND, MIT M.S Thesis Defense- 94 AJW 12/1/14

Collision Avoidance Coordination Overview Messages sent by Mode S interrogation/reply (currently) or ADS-B broadcast (future) • Slightly different sensor measurements can result in both aircraft selecting the same direction • Coordination messages are sent to instruct other aircraft to not select the same direction • In case of a maneuver conflict, master aircraft overrides • The following scenarios are to be assessed when evaluating coordination: – Both aircraft respond – Cross link disabled, both respond – Only own aircraft responds, but cross link is still enabled – All of the above with switching slave/master M.S Thesis Defense- 95 AJW 12/1/14

ACAS X Methodology Expert Input Computer Human Expertise Optimization Evaluation Tuning Meets design Requirements Logic Parameters goals? Feedback and Tuning Process Considerations Example Metric Performance Goals Pr(NMAC with system) Risk Ratio = • Safer than TCAS Safety Pr(NMAC without system) Level Alert frequency, rates, • Fewer alerts and less Operational ~500 feet and types in standard disruptive than TCAS Suitability visual encounters Level-off Advisory Sequences “Climb, Climb… • Pilots accept alerts User Descend, Descend NOW… Climb, Climb NOW” (or lack thereof) Acceptability M.S Thesis Defense- 96 AJW 12/1/14

ACAS X Methodology Tunable Parameters Computer Human Expertise Optimization Evaluation Tuning Meets design Requirements Logic Parameters goals? Feedback and Tuning Process Online Parameters Offline Cost Function Influences alerting Influences alerting behavior based on through computer operational context optimization • C switch : cost of switching advisory • C conflict : vertical conflict • C restart : cost of restarting advisory • C alert : initial alert • C reversal : reversal • C inhibit : cost of issuing descends • C clear of coflict : terminate alert (reward) below certain altitudes • σ ddh : nominal vertical acceleration • dh max : max achievable vertical speed • E[T init ]: mean initial pilot response delay • E[T subs ]: mean subsequent pilot response M.S Thesis Defense- 97 • … AJW 12/1/14

ACAS X Methodology Logic Optimization Computer Human Expertise Evaluation Optimization Tuning Meets design Requirements Logic Parameters goals? Feedback and Tuning Process Look-Up Table Online Parameters Optimization Algorithm Probabilistic Dynamic Model 0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 0 0 0 0 1 1 1 0 1 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 1 1 1 0 1 1 0 0 0 1 1 0 1 1 0 0 1 1 1 1 0 0 1 1 1 0 1 1 1 0 1 1 0 1 1 0 1 0 1 0 0 1 0 0 0 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 1 0 0 1 0 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 0 1 1 0 1 0 0 0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 0 0 0 1 0 1 1 1 0 0 1 0 0 0 Offline Costs 1 1 1 0 1 1 0 0 0 1 0 1 1 1 0 0 1 0 0 0 Alert Types (and simulation framework) M.S Thesis Defense- 98 AJW 12/1/14

ACAS X Methodology Feedback and Tuning Process Computer Human Expertise Optimization Evaluation Meets design Tuning Requirements Logic Parameters goals? Feedback and Tuning Process Evaluation Results vs. Metrics Feedback for Additional Tuning  Most Wanted Risk ratio is better than TCAS List 1) 2)  3) Issues too many disruptive alerts Desired behavior 4) 5) provided to team for further tuning  Alert sequences are not acceptable  Does not alert in visual procedures 8+ logic evaluations since June 2011 M.S Thesis Defense- 99 AJW 12/1/14