Applied Quantitative Cyber Risk Analysis Michael Rich, OSCP, CISSP Director of IT Security, Infrastructure & Operations Motion Picture Industries Pension & Health Care Plan
| 2 | Disclaimer for those reading from the ISACA link My talks are image and slide-build heavy. So they don’t “print” well. Sorry about that.
| 3 | Agenda Seek Beyond Your Interest – “Risk Analysis: Don’t Just Trust Your Gut” – Evan Wheeler @ BSidesLA 2016 The Idea: – What is a Risk? – The Calibration of the Experts – Monte Carlo Risk simulation – A Cyber Risk Model Example The Application: – Risk Decomposition – Gedanken Experiments – “The SHOCKING truth about probability they don’t want you to know!!!” – Snowflakes and Monte Carlo – Equivalent Life Event Probabilities Now What?
| 4 | The Idea
| 5 | What is a Risk? An event that has some chance of happening and causes effects we don’t want. Qualitative Analysis Quantitative Analysis Source credit: https://www.risklens.com/blog/4-steps-to-a-smarter-risk-heat-map
| 6 | What is a Risk? Probability of Occurrence Log-normal distribution example – Numerically-expressed probability – Can be a range to express uncertainty i.e.: 9-14% chance Impact (Loss) – Numerically expressed range: Upper bound Lower bound 90% confidence – Used with a log-normal distribution 5% values are < Lower bound 5% of values are > Upper bound Black Swans!
| 7 | Log Normal – In Real Life Image from Blackline.com
| 8 | What is a Risk? Estimated over given time period A basic risk: – Tomorrow, if a traffic incident occurs during my morning commute, I will be delayed – Probability of occurrence: 30% – Impact (90% confidence): 5 – 60 minute delay from normal commute time
| 9 | Subjective Range Estimation AKA The Calibration of the Experts What is the stated capacity of Wembley Stadium in London? The Equivalent Bet: for 1000 Imperial Credits would you rather Win nothing – See if the answer is in your interval – Spin the dial? This slide covered on purpose so we don’t ruin the fun at the event!! Win it all Capacity: 90,000
| 10 | Monte Carlo Simulation Iterate over probability of occurrence and generate random impacts Many times (100K+) Example: Probability: 30% Impact, Upper bound: 60 Impact, Lower bound: 5 Number Trials 10001 Trial Delay 1 0 2 14.55244 3 17.37702 4 16.64968 5 0 6 0 7 0 8 0 9 0 10 49.68741
| 11 | Sim Results and the Loss Exceedance Curve
| 12 | Reducing Loss Exceedance Curves Curves are pretty, but I need a number! – Ranking – Comparison – Mitigation effectiveness In insurance world: – Average Annual Loss = Premium – “Area under the curve” For Commute: – Average Event Impact – 6.8 minutes…. But… 241 Minute MAX impact
| 13 | Methodology Demonstration – The Shared Home Computer Cost chosen as impact only for purposes of this example Banking Trojan Clumsy Cat Probability 5% Probability 5% Max Impact $3000 Max Impact $25,000 ($35,000) Min Impact $750 Min Impact $500 Amazon Spree Ransomware Probability 30% Probability 10% Risks over next 6 months Max Impact $750 Max Impact $3000 Min Impact $150 Min Impact $200 Creepy Spyware Probability 2% Max Impact $2000 ($5000) Min Impact $300
| 15 | Simulation Results (100K iterations) Use Case: Ranking Risks Total Expected Average $638 Loss Banking Trojan $317 Amazon Spree $112 Ransomware $110 Clumsy Cat $80 Creepy Spyware $19
| 16 | The Application
| 17 | Risk Decomposition Break your risk effects down into chunks – Measureable and observable – Company dependent Manpower Costs – Business Departments – Leadership Remediation Costs – IR Retainer – Legal – Hardware – Software
| 18 | Risk Decomposition Active? Active? Active? LB LB LB Time UB Time UB Hardware Cost UB PSC Cap Security Cap LB LB Cap $/Hr UB $/Hr UB Active? Cap Cap LB Active? Active? Software Cost UB LB LB Time UB Time UB Cap Retirements Cap IT Ops Cap Active? LB LB LB $/Hr UB $/Hr UB IR Retainer Cap Cost UB Cap Active? Active? Cap LB LB Active? UB Time Time UB LB Cap Accounting IT Leadership Cap Legal LB LB Cost UB $/Hr UB $/Hr UB Cap Cap Cap
| 19 | Gedanken Experiments
| 20 | The ONE SHOCKING Truth About Probability Aggregate probability is a bitch… 2 times in 120 days, I escalated a security event to the CIO What are the odds I have to escalate an issue any given day: – Odds: 2/120 – Probability [Odds/(1+Odds)]: 1.64% What is the probability (p) I’ll have an event in the next 6 months I have to escalate? Well: – Probability (p-not) of it not happening [1-p]: 98.4% – Probability of it not happening for 120 consecutive days [(p-not)^120]: 14.4% – Probability of an escalated event in 120 days [1-(not happening)]: 85.6%
| 21 | Is Monte Carlo a Precious Snowflake? (Sensitivity Analysis) 3 independent variables. How sensitive is the Average Event Loss? Probability Lower Bound Upper Bound
| 22 | Monte Carlo IS a Precious Snowflake.. Probably
| 23 | Ooof.. It’s Even Worse Than I Thought
| 24 | Handling the Snowflake Must include uncertainty in your probability estimate (i.e. a range) Instead of 1% probability, let’s make it 0.5% - 1.5% (50% error bar) Test AEL($) 1% Fixed $72 1% +/- .5% $70
| 25 | Beta Distribution Single: $71.79 Uniform: $71.15 Beta: $71.63 Test EAL ($) 1% fixed $71.79 1% +/- 0.5% $71.15 1% Beta $71.63
| 26 | Some More Experiments Test EAL ($) 5% fixed $367 5% +/- 4% $355 5% Beta $356
| 27 | Some More Experiments Test EAL ($) 5% fixed $350 5% +/- 4% $349 4% +/- 3% $293 4% fixed $277
| 29 | Statistically Equivalent Probabilities 100% - 50% 50% - 10% 10% 3% 1.5% 1% 0.8% 0.02%
| 30 | Beta Distribution: Establish Probability from Test Cases If you have a set of cases, you can get a probability distribution
| 32 | Using Probability for Complicated Scenarios Calibrate expert Ask expert to assess probability of the event given no other data – “What is the probability of an adversary managing to inject code on the target system in the next 6 months?” Ask expert to re-assess given various conditions – “What if the firewalls are discovered to be misconfigured?” – “What if a Cooperative Vulnerability Inspection team demonstrates code injection?” – “What if a black-box adversarial assessment team demonstrates it?” Use Log-Odds-Ratio – Statistically valid method for combining the effects of multiple conditions on a final probability
| 33 | Log Odds Ratio Example Use Case: Using expert knowledge Initial Prob: P(E) 1.0% Conditions Firewall CVI Finding Adversarial Software Adversary Intel Rogue Network Device Rogue USB P(E|X1) Correct Config Code Injection Code Injection Updated 1 Hop away Detected Detected P(E|X2) Incorrect Config No Injection No Injection Not Updated 2 Hops away Not detected Not Detected P(E|X3) 3+ Hops away P(E|X4) P(E|X1) 0.5% 5.0% 10.0% 1.0% 10.0% 1.0% 1.0% P(E|X2) 2.0% 0.5% 0.5% 5.0% 3.0% 15.0% 15.0% P(E|X3) 1.0% P(E|X4) Condition State Which Applies Correct Config No Injection Code Injection Updated 1 Hop away Detected Detected Conditional Probability 23.2%
| 34 | Now What? For Me – Solidify my risk decompositions – Identify my events to analyze – Calibrate my team – Model and Simulate – Submit Blackhat ‘18 paper For You – Go read Hubbard’s book – Go get my code: https://github.com/richmr/QuantitativeRiskSim – Think about your decompositions – Identify your events – Model and Simulate – Come watch my Blackhat ‘18 presentation
| 35 | Summary Quantitative risk modeling can be a reality in Cybersecurity – Use Case: Risk ranking and prioritization – Use Case: Assessing control audit results – Use Case: Mitigation comparison – Use Case: Quantifying expert knowledge on complex systems – Use Case: Test planning Networks can improve its cybersecurity… Measurably! Python Simulation Code available at: – https://github.com/richmr/QuantitativeRiskSim
| 36 |
Recommend
More recommend
