Notes Cost-benefit analysis Tyler Moore CSE 5/7338 Computer Science & Engineering Department, SMU, Dallas, TX Lecture 10 Review of security investment so far Notes Metrics for quantifying security benefits ALE 0 : expected loss without security investment 1 ALE s : expected loss with security investment 2 EBIS s : ALE 0 − ALE s 3 ENBIS s : ALE 0 − ALE s − c 4 High-level investment metrics ROSI 1 NPV 2 IRR 3 2 / 26 Security investment questions worth answering Notes Q: Should we invest in security? A: Yes, if ENBIS > 0 Q: Should we invest in defense A or B? A: Choose the one with higher ROSI (or NPV if considering longer time horizons) Q: How much should we invest? A: Security investment models (e.g., Gordon-Loeb) say to invest until marginal cost of added security equals marginal benefit Q: Is a security investment cost-effective? A1: Yes, if ENBIS > 0 A2: Probably, if the minimum probability of attack required to break even is high enough 3 / 26 Cost-benefit analysis (CBA) Notes Used widely in public policy to justify expenditures Quite similar to the security metrics presented earlier, especially ENBIS Emphasis placed on making best-effort estimates of key figures Costs of insecurity ( ALE 0 ) 1 Costs of security countermeasures ( c ) 2 Probability of attack ( p 0 ) 3 Risk reduction r = p 0 − p s 4 p 0 In CBA, a security investment is considered cost-effective if ENBIS > 0. CBA exercises estimate the above figures and use the findings as evidence when deciding whether or not to adopt (or continue spending money on) a countermeasure When there is uncertainty over some figures, a range of values is considered 4 / 26
ENBIS using risk reduction Notes ENBIS equations from earlier presentations using Bernoulli loss assumptions used p 0 and the improved probability p s We can equivalently express this in terms of reduced risk ENBIS = ( p 0 − p s ) · λ − c ENBIS = p 0 · p 0 − p s · λ − c p 0 ENBIS = p 0 · r · λ − c 5 / 26 ENBIS for multiple sources of loss Notes Up to now, we have assumed that there is a single financial loss λ associated with an attack In fact, losses can take many forms, each with its own magnitude and probability of occurrence Ideally, we would like to account for each type of loss independently and combine into an aggregate measure Suppose there are n loss types. We can calculate the ENBIS as follows: ENBIS = p 0 · r · λ − c n � ENBIS = p 0 · r · ( P ( λ i | attack) · λ i ) − c i =1 6 / 26 Cost-benefit analysis tasks Notes Estimate p 0 using available data (sometimes hard) Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Estimate (or take as input) security costs c Estimate (or take as input) risk-reduction rate r We discuss cost-benefit efforts for two examples: terrorist attacks targeting highway bridges (reading 1) and sewer overflows at wastewater facilities (reading 2) 7 / 26 Case 1: terrorist attacks targeting highway bridges Notes Estimate p 0 using available data (sometimes hard) No known instances in past, so assign small probability ( p 0 = 10 − 4 ) Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Bridge replacement: $40 million (average of replacement costs 1 for prior collapses), cond. prob. = 1.0 Loss of life: 80 lives with actuarial value $6.3M each, occurring 2 with cond. prob. 0.2 (estimated from prior collapses) Estimate (or take as input) security costs c NPV of 20% of bridge-replacement value amortized over 25 years = $260,000 Estimate (or take as input) risk-reduction rate r Taken to be r = 0 . 9 High value selected to give benefit the best possible chance of exceeding costs 8 / 26
Case 1: terrorist attacks targeting highway bridges Notes n � ENBIS = p 0 · r · ( P ( λ i | attack) · λ i ) − c i =1 ENBIS = Fill in the equation ENBIS = − 247 K Based on this calculation, the security investment does not seem to be justified. 9 / 26 Case 2: sewage overflows at wastewater facilities Notes Estimate p 0 using available data Original goal: estimate probability of malicious attack triggering large overflows, but there have only been a few publicly reported attacks Revised goal: estimate probability of large sewage overflows triggered by accident or attack, since both can be detected and sometimes prevented by incident detection system California Water Board reported 46 large overflows in one year in state They separately reported that facilities cover 110,593 sewer miles Hence the number of overflows can be expressed as 110593 = 4 . 16 × 10 − 4 × # miles. 46 Cities with population over 100,000 have an average of 1,300 sewer miles in their facilities Hence p 0 = 0 . 541 Note that p 0 is more accurately interpreted here as the expected number of overflows during the time period 10 / 26 Case 2: sewage overflows at wastewater facilities Notes Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Loss category Data? Direct losses Cleanup costs yes Property damage yes Regulatory costs yes Lost business for victims no Victim health costs no Indirect losses Lost business for non-victims no Broader environmental impact no Psychological distress no We can estimate the costs for the categories we have to arrive at a lower bound for the total cost 11 / 26 Case 2: sewage overflows at wastewater facilities Notes Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place i Loss category λ i P ( λ i | SO) Comments 1 Cleanup costs 22K 1 Likely underestimate 2 Property damage 1.4M 0.25 no data for cond. prob. 46 violations 2000–2011 3 EPA fine 2.89M 0.01 46 SOs in CA in 2012 12.1% of US pop. in CA × 12 yrs 12 / 26
Case 2: sewage overflows at wastewater facilities Notes Estimate (or take as input) security costs c City Cost factor Cost/year Sewer miles Reference 1 20K 1300 Atlanta 2 39K 2125 DC 3 59K 1800 San Francisco 6 118K 993 New Orleans 8 157K 1600 Estimate (or take as input) risk-reduction rate r Taken to be r = 0 . 4 Argued that some overflows couldn’t be prevented, but some should be 13 / 26 Case 2: sewage overflows at wastewater facilities Notes n � ENBIS = p 0 · r · ( P ( λ i | attack) · λ i ) − c i =1 ENBIS = Fill in the equation ENBIS = 67 K Based on this calculation, the security investment is justified for the “average” city. 14 / 26 Case 2: sewage overflows at wastewater facilities Notes Recall that security investment costs and the expected number of large overflows vary by city City Cost/year Sewer miles ENBIS Reference 20K 1300 67K Atlanta 39K 2125 103K DC 59K 1800 62K San Francisco 118K 993 -51K New Orleans 157K 1600 -50K 15 / 26 Case 2: sewage overflows at wastewater facilities Notes 16 / 26
What if we are uncertain about the accuracy of estimates? Notes When we are uncertain about one or more of the estimated parameters, we can do a breakeven analysis to identify the value a parameter must take for ENBIS = 0. The best parameter to vary is the one that is most uncertain Often, this is p 0 , the probability of attack without security investment 17 / 26 Cybersecurity is not the only discipline where estimating Notes probabilities of rare events is difficult The assessment of the probabilities that adversaries will choose courses of action should be the outputs of analysis, not required input parameters Quote is from National Academies of Science report on bioterrorism risks What does this mean for cost-benefit analysis? 18 / 26 Breakeven analysis with probability of attack as output Notes ENBIS = ( p 0 − p s ) · λ − c ENBIS = p 0 · p 0 − p s · λ − c p 0 ENBIS = p 0 · r · λ − c Setting ENBIS to 0 and solving for p 0 : c p 0 = r · λ We can then see for a range of parameter values what the corresponding breakeven probability of attack must be to justify security investment 19 / 26 Breakeven analysis for case 1 Notes c p 0 = r · λ p 0 = Fill in the equation p 0 = 0 . 002 20 / 26
Breakeven probabilities (as percentages) for case 1 Notes Source: http://politicalscience.osu.edu/faculty/jmueller/CIP.pdf 21 / 26 Breakeven analysis for case 2 Notes c p 0 = r · λ c p 0 = r · (22 K · 1 + 1 . 4 M · 0 . 25 + 2 . 89 M · 0 . 0101) c p 0 = r · 401 K 22 / 26 Breakeven probability of sewage overflow for case 2 Notes 2.5 c=20K c=50K c=100K 2.0 p_0 (Expected # overflows) 1.5 1.0 0.5 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Risk reduction probability 23 / 26 Breakeven analysis with risk reduction as output Notes ENBIS = ( p 0 − p s ) · λ − c ENBIS = p 0 · p 0 − p s · λ − c p 0 ENBIS = p 0 · r · λ − c Setting ENBIS to 0 and solving for r : c r = p 0 · λ We can then see for a range of parameter values what the corresponding breakeven risk reduction must be to justify security investment 24 / 26
Recommend
More recommend
