Modeling trade-offs between false positives and negatives Tyler - PDF document

Notes Modeling trade-offs between false positives and negatives Tyler Moore Computer Science & Engineering Department, SMU, Dallas, TX October 2, 2012 Homework 1 Optimal filter configuration Notes Outline Homework 1 1 Summary statistics Logistical notes Feedback on particular questions Optimal filter configuration 2 ROC curves An economic model of optimal filter configuration 2 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Summary statistics on grades for HW1 8 6 Grade with 4pt curve 4 Mean: 81% Median: 77.5% 2 0 A B C D 4 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Summary statistics on time required ● ● ● Time spent on the assignment not 100 ● correlated with grade ● ● ● Score (curved) 90 Time spent ● ● Mean: 9.3 hours ● 80 ● ● ● ● Median: 6.5 hours ● ● Take as much time as you need 70 ● ● My goal for most assignments: 5-7 ● 0 5 10 15 20 25 hour median hours spent on assignment 5 / 32

Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Remember to put your name on the assignment Please put your name on the assignment Future assignments: on-campus students turn in to Debra McDowell; off-campus submit scanned PDF as before I observed a correlation between students submitting their assignment as a Word file and forgetting to put their name on the assignment No name Name Word 4 3 PDF 1 10 6 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Brevity matters Writing with brevity is an important skill ⇒ Blaise Pascal (1657): “I have made this letter longer than usual because I lack the time to make it shorter.” When questions specify the length of answers, it is important to respect the guidelines 3 paragraphs does not mean 3 pages 1 example means 1 example It can be OK if you go slightly longer than guided, provided that you make a worthwhile point as concise as possible I am not interested in a brain dump of all relevant facts on the question’s topic Including extraneous facts can get you into trouble, particularly if you make a mistake in the extra information 7 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Questions about the assignment Come see me if you’re still unsure about how to solve a problem Questions about your grade? If you think I made a mistake (e.g., your answer is correct according to the answer key), please let me know If you think I took off too many points, please do not let me know I don’t negotiate grades out of fairness to all students 8 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Collaboration and attribution Recall from the syllabus: I encourage collaboration between students on assignments and when studying. Collaboration is an essential skill for engineering, not to mention life in general. Unless I say otherwise, feel free to discuss assignments and the project with your classmates, including ideas for how to solve problems. Please do not, however, share code, equations, or written answers that solve an assignment directly with other students. Solutions to homeworks should be written from scratch and must not be pieced together from other students. It is also important to give credit to others when appropriate. If you implement an idea that you got from another student (or students), please say so . Furthermore, if you consult a web resource that directly assists you, please say so. As a reminder, it is also not acceptable to copy code or equations directly from a web resource that solves a problem on an assignment. 9 / 32

Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Collaboration and attribution It is OK to discuss problems from the assignment with other students It is not OK to share your solutions with other students If you discuss problems at length with other students (especially if you discuss any ideas for how to solve problems), make a note of it on your assignment for each question that you collaborated 10 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Question score distribution mean 100 median 80 % correct 60 40 20 0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 11 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Q4 (15 points) Bruce Schneier maintains a security blog ( http://www.schneier.com/blog/ ) that describes lots of interesting attacks on real-world systems. The site has archives dating to 2004 that are browsable and searchable (see right-hand side under his picture). Your task is to find a post linking to an attack on an information system (i.e., not an attack on a physical system such as terrorism). In 3 paragraphs, describe in your own words the following 1 the threat model of the system that was attacked; 2 how the attack worked; 3 whether and how the attack bypassed the system’s threat model. Be sure to include the URL of the relevant post. Answer varies by response. 12 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Threat models All security is relative, but relative to what? ⇒ Threat models codify assumed adversary behavior Threat models articulate assumed adversary behavior Goal : disrupting defender’s protection goals, make money, 1 wreak havoc Knowledge : does the attacker know how the defense works? 2 Capabilities : Computational power available, time available to 3 target defenders, local vs. global eavesdropping, active vs. passive 13 / 32

Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Threat models Sarah Palin’s email: http://www.schneier.com/blog/ archives/2008/09/sarah_palins_e-.html Matt Honan’s cloud deletion: http://www.schneier.com/ blog/archives/2008/09/sarah_palins_e-.html Middleperson bank fraud: http://www.schneier.com/ blog/archives/2012/09/man-in-the-midd_5.html 14 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Q7 One of the primary motivations for the failed Cybersecurity Act of 2012 was the view that many critical infrastructure operators weren’t investing enough to protect against the risk of attack. In particular, there were concerns that the industrial control systems used to manage heavy industry such as power plants and refineries are inadequately protected against attacks seeking to disrupt operations. Your task is to briefly (in approximately 300 words) present both sides of the argument. First, present any available evidence that industrial control systems are being adequately protected. Second, present an economic argument that the systems are not being protected enough. In particular, explain which market failures, if any, might explain why there has been underinvestment in security. Finally, conclude by briefly explaining which argument you find more convincing and why. 15 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Q7 Points in favor of adequate security: 1 Very few realized attacks 2 Reputational damage of an attack would be harsh Points against adequate security: 1 Security is difficult for consumers to observe (even if they are willing to pay for it) 2 Security as public good 3 Externalities (firm doesn’t pay the full cost) 4 Often critical infrastructures are run by natural monopolists (water supply, electricity grid, etc.) 16 / 32 Summary statistics Homework 1 Logistical notes Optimal filter configuration Feedback on particular questions Notes Q5 1 Suppose you are in charge of tuning the settings of false negative rate β an intrusion detection Indifference system to balance false curves positives and false negatives. The false negative rate is given by β ∈ (0 , 1], while the false #1 positive rate is given by #2 #3 α ∈ (0 , 1]. 0 false positive rate α 0 1 17 / 32