Context MCDA strategy Time integration Experimentation Concluding remarks and future work Integrating a temporal component into multi-criteria majority-rule sorting models Application to the cyber-defense context Arthur VALKO Director : Patrick Meyer Supervisors : Alexandru-Liviu Olteanu, David Brosset November 22, 2018 Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 1 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Outline Context 1 Decision problem MCDA strategy 2 Operational constraints Model choice Time integration 3 Hierarchical model Learning Process Experimentation 4 Test platform Experimentation 5 Concluding remarks and future work Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 2 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Decision problem MCDA & cyber-defense : ship example Ship captain Attacks Decision maker Mission Actions Functionalities Criteria Alternatives Preferences Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 3 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Decision problem Dashboard for the decision maker Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 4 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Operational constraints Operational constraints The ship captain does not necessarily need a ranking of the actions, but rather a qualitative evaluation of each of them. → He / she wishes to have the final word! → sorting algorithm Evaluation scales of the criteria are heterogeneous and have a strong meaning for the decision maker. He / she (cyber defender) does not trust black boxes . → High readability of the decision recommendation required → outranking method Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 5 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Model choice MR-Sort Sorting outranking model Various extensions possible to increase expressiveness Output easy to read and to explain Indirect learning process Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 6 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Model choice Further need Consequences of actions might vary with time on the various criteria T T-4 T-3 T-2 T-1 T+1 T+2 T+3 T+4 T+5 ? Good evaluation 5/5 4/5 4/5 5/5 5/5 Bad 99% 60% 30% 60% 35% evaluation T+1 T+2 T+3 T+4 T+5 T+1 T+2 T+3 T+4 T+5 5/5 5/5 5/5 3/5 5/5 15% 15% 99% 50% 99% Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 7 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Time integration Context 1 Decision problem MCDA strategy 2 Operational constraints Model choice Time integration 3 Hierarchical model Learning Process Experimentation 4 Test platform Experimentation 5 Concluding remarks and future work Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 8 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Time integration Multiple options : Increase the number of criteria ( J · T ) One "time"-criterion per time step Loss of readability for DMs The DM may have difficulties to understand the output preference model Time aggregation Loss of information (intra- and inter-criterion) Our proposal : hierarchical approach ( H ) Time structure conservation Better readability for DMs Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 9 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Hierarchical model Hierarchical model T+1 T+2 T+3 T+4 T+5 1/5 3/5 5/5 5/5 5/5 30% 50% 99% 99% 99% criteria weights G good criteria weights G good criteria weights G good criteria weights G good criteria weights G good alternative alternative alternative alternative alternative separation separation separation separation separation pro fi le pro fi le pro fi le pro fi le pro fi le majority majority bad majority bad B bad majority bad majority bad threshold : 5 B threshold : 5 B threshold : 6 alternative threshold : 7 B threshold : 5 B alternative alternative alternative alternative T+1 T+2 T+3 T+4 T+5 Good criteria weights G good alternative evaluation separation pro fi le Bad majority bad B threshold : 6 alternative evaluation Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 10 / 19
fi fi fi fi fi fi Context MCDA strategy Time integration Experimentation Concluding remarks and future work Learning Process Learning process T T-4 T-3 T-2 T-1 T+1 T+2 T+3 T+4 T+5 ? 5/5 4/5 4/5 5/5 5/5 99% 60% 30% 60% 35%
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Learning Process The MR-Sort hierarchical learning algorithm Mixed-Integer Program : comparison with MR-Sort J · T Increase of number of profiles and number of majority thresholds Increase of number of variables (integer ones) Increase of number of constraints Study : Learning time as a function of problem size Inferred model quality as a function of problem size Cross-analysis between and MR-Sort with an increase of the number of criteria (MR-Sort J · T ) and the hierarchical approach (MR-Sort H ). Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 12 / 19
fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi fi Context MCDA strategy Time integration Experimentation Concluding remarks and future work Test platform generator Test platform Model Mixed integer program T+1 T+2 T+3 T+4 T+5 T+1 T+2 T+3 T+4 T+5 0/5 0/5 2/5 4/5 5/5 45% 40% 30% 60% 75% 5/5 5/5 5/5 3/5 5/5 15% 15% 99% 50% 99% Models generator Random MR-Sort J · T or MR-Sort H models Select number of criteria ( | J | ), time steps ( | T | ) and categories ( k ) Alternatives generator Select number of criteria | J | , time steps | T | and the number of examples Assign alternatives with one of generated models Learn preference parameters via MR-Sort J · T and MR-Sort H models Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 13 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation Experimentation Parameters : Number of criteria : { 3 , 5 , 7 } , 3 time steps Number of categories k : { 3 , 5 } Number of assignment examples : { 10 , 20 , 30 , 50 , 70 } Classification accuracy tests Comparison between inferred model and the original one with 10 000 new alternatives Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 14 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation Learning times Global learning time on the set of test MR-Sort H models are more difficult to learn (often > 1h) Important standard deviation MR-Sort J · T shorter learning time (0.2 - 10 seconds) Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 15 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation Classification accuracy with original model, MR-Sort J · T Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 16 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Experimentation Classification accuracy with original model, MR-Sort H Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 17 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Concluding remarks and future work Hierarchical model : Long learning times Adds a time component into the decision-making process. Decision recommendation decomposition for better explanation. Good classification accuracy Future work : Apply the model to a real-world case Meta-heuristic learning method Automatic explanation of recommendations Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 18 / 19
Context MCDA strategy Time integration Experimentation Concluding remarks and future work Thank you for your attention. Any questions? Arthur Valko Chaire cyber navals DA2PL ’2018 November 22, 2018 19 / 19
Recommend
More recommend
