Hackers vs Testers: A Comparison of Software Vulnerability Discovery Processes Daniel Votipka , Rock A. Stevens, Elissa M. Redmiles, Jeremy Hu, and Michelle L. Mazurek 22 May 2018
VULNERABILITY DISCOVERY � 2
VULNERABILITY DISCOVERY Generalists Testers: • Functionality • Performance • Security � 3
VULNERABILITY DISCOVERY Experts Hackers: • Security Team • Contracted Review • Bug Bounty � 4
CHALLENGES •Timeliness •Cognitive Diversity •Communication � 5
RESEARCH QUESTIONS 1. How do testers and hackers search for vulnerabilities? 2. What are the differences between testers and hackers? Interview study: • Task Analysis • Tools, Skills, and Communities � 6
RECRUITMENT Hacker Groups: • Bug Bounty Programs • Top Hacking Teams Tester Groups: • Meetup and LinkedIn • IEEE and AST • Ministry of Testing 106 total groups � 7
PARTICIPANTS Participants 15 10 Vulnerabilities Found 26-50 0-3 Vulnerability Finding Time 10-20 hrs/w 5-10 hrs/w � 8
RESEARCH QUESTIONS 1. How do testers and hackers search for vulnerabilities? 2. What are the differences between testers and hackers? � 9
RESEARCH QUESTIONS 1. How do testers and hackers search for vulnerabilities? 2. What are the differences between testers and hackers? � 10
HACKER AND TESTER PROCESS Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting � 11
Information Gathering Info Gathering • Build context prior to reading or executing code Program • Example actions: Understanding • Identifying libraries • Update history Attack Surface • Previous bug reports Exploration “There were…other functional issues, so I figured that was probably where Vulnerability there was most likely to be security Recognition issues as well. Bugs tend to cluster.” Reporting � 12
Info Gathering Program Understanding • Determine how the program operates Program • Interaction between components Understanding • Interaction with the environment Attack Surface Exploration “You’re touching a little bit everything, and then you are organizing that into Vulnerability Recognition a structure in your head.” Reporting � 13
Info Gathering Program Understanding Attack Surface • Identify how user interacts with program Attack Surface • Direct and indirect inputs Exploration Vulnerability Recognition Reporting � 14
Info Gathering Program Understanding Attack Surface Exploration • Possible inputs to the attack surface • Example actions: Exploration • Fuzzing • Reading code Vulnerability Recognition Reporting � 15
Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Vulnerability • Notice a problem when exploring Recognition • Typically described as intuition-based Reporting � 16
Info Gathering Program Understanding Attack Surface “You do have to [convince] someone that there’s a risk. …It’s quite timely Exploration [time consuming], running a ticket.” Reporting Vulnerability • Tell developers about the problem Recognition • Advocate for remediation • Critical aspects: Reporting • Make report understandable • Importance of fixing � 17
RESEARCH QUESTIONS 1. How do testers and hackers search for vulnerabilities? 2. What are the differences between testers and hackers? � 18
Info Gathering Vulnerability Discovery Experience Program Understanding Underlying System Attack Surface Knowledge Access to Exploration Development Process Vulnerability Recognition Motivation Reporting � 19
Info Gathering Vulnerability Discovery Experience Program Understanding Underlying System Attack Surface Knowledge Access to Exploration Development Process Vulnerability Recognition Motivation Reporting � 20
Attack Surface Attack Surface • More likely to consider indirect input paths Vulnerability Exploration Discovery Exploration • Informs test case selection Experience Vulnerability Recognition Vulnerability • Know vulnerability patterns Recognition • List of common vulnerabilities “As soon as I found the LinkedIn problem, I made sure to test [FB and Twitter input] to make sure [they were processed correctly]. And if we did allow login with another 3rd party in the future, I would check that too.” � 21
Employment Attack Surface Hacking Exercises Vulnerability Discovery Exploration Experience Community Vulnerability Recognition Bug Reports � 22
AMOUNT OF EXPERIENCE Employment Hacking Exercises Vulnerability Discovery Experience Community Bug Reports � 23
“It’s hard to ignore certain details once you know about certain areas already.” Internal Info Gathering • Communicate with developers Access to • Documentation Program Development Understanding Process External • Reverse engineering Reporting • Develop exploits “You can give feedback to your developers. . . .You’re coming back with information, and then they react on it.” � 24
RECOMMENDATIONS ‣ Provide training in known contexts • Hire hackers into the testing team • Bug report-based exercises ‣ Improve hacker communication • Single point of contact � 25
SUMMARY Questions : dvotipka@cs.umd.edu • Similar processes vulnstudy.cs.umd.edu • Impacted by: • Vulnerability Discovery Experience • Underlying System Knowledge, • Access to the Development Process • Motivation • Biggest difference in amount of experience and relationship with the developers Recommendations: • Training in a known context • Hacker/company communication � 26
Recommend
More recommend
