Toward a Field Study on the Impact of Hacking Competitions on Secure Development Daniel Votipka , Hongyi Hu, Bryan Eastes, and Michelle L. Mazurek 12 Aug 2018
SECURE DEVELOPMENT � 2
SECURE DEVELOPMENT � 2
SECURE DEVELOPMENT � 2
SECURE DEVELOPMENT � 2
SECURE DEVELOPMENT • Non-experts lack experience • Experts learn through CTFs [Votipka et al., 2018] � 2
CAPTURE THE FLAG (CTF) • Attack-oriented competitions ‣ Goal: find and exploit vulnerabilities • Simple, vulnerable programs • Expose competitors to several classes of vulnerabilities � 3
CAPTURE THE FLAG (CTF) • Attack-oriented competitions ‣ Goal: find and exploit vulnerabilities • Simple, vulnerable programs • Expose competitors to several classes of vulnerabilities Do these help in practice? � 3
RESEARCH QUESTIONS 1. Do CTFs improve prevention of security issues? 2. Do CTFs improve recognition of security issues? � 4
PILOT STUDY OVERVIEW (1 week) Dropbox Capture- Time the-Flag � 5
PILOT STUDY OVERVIEW (10 mins: 6 weeks, 2x/week, 1x/day) (1 week) Diary Surveys Diary Study Dropbox Capture- Time the-Flag � 5
PILOT STUDY OVERVIEW (10 mins: 6 weeks, 2x/week, 1x/day) (1 week) Diary Surveys Diary Study Dropbox Capture- Time the-Flag Knowledge Assessment Pre-CTF Post-CTF Assessment Assessment (60 mins) (60 mins) � 5
(10 mins: 6 weeks, 2x/week, 1x/day) (1 week) Diary Surveys Time Dropbox Capture- the-Flag DIARY SURVEYS Pre-CTF Post-CTF Assessment Assessment (60 mins) (60 mins) • Survey regarding recent commit ‣ Issues considered ‣ Reasons for considering each issue ‣ Actions taken to resolve • Not security specific • Open to all Dropbox developers � 6
(10 mins: 6 weeks, 2x/week, 1x/day) (1 week) Diary Surveys Time Dropbox Capture- the-Flag DIARY SURVEYS Pre-CTF Post-CTF Assessment Assessment (60 mins) (60 mins) • Survey regarding recent commit ‣ Issues considered ‣ Reasons for considering each issue ‣ Actions taken to resolve Measure impact of CTF on day-to-day decisions • Not security specific • Open to all Dropbox developers � 6
(10 mins: 6 weeks, KNOWLEDGE 2x/week, 1x/day) (1 week) Diary Surveys Time Dropbox Capture- the-Flag ASSESSMENT Pre-CTF Post-CTF Assessment Assessment (60 mins) (60 mins) •Part 1: Find vulnerabilities in insecure code ‣ Copy of the Dropbox codebase ‣ 4 known vulnerabilities •Part 2: Write a secure program •Only CTF participants � 7
(10 mins: 6 weeks, KNOWLEDGE 2x/week, 1x/day) (1 week) Diary Surveys Time Dropbox Capture- the-Flag ASSESSMENT Pre-CTF Post-CTF Assessment Assessment (60 mins) (60 mins) •Part 1: Find vulnerabilities in insecure code ‣ Copy of the Dropbox codebase ‣ 4 known vulnerabilities •Part 2: Write a secure program •Only CTF participants Measure improvements to secure development in a controlled setting � 7
ADDITIONAL METRICS • Number of flagged commits • Communication with the Dropbox security team � 8
PILOT PARTICIPATION • Diary Surveys ‣ 28 participants (12 CTF) ‣ 169 surveys • Knowledge Assessment ‣ 7 participants � 9
PILOT PARTICIPATION • Diary Surveys ‣ 28 participants (12 CTF) • Small sample • Methodological ‣ 169 surveys issues addressed in future iterations • Knowledge Assessment ‣ 7 participants � 9
DIARY SURVEYS • Security considered in 17/124 functionality changes ‣ 19% CTF, 13% non-CTF � 10
DIARY SURVEYS • Security considered in 17/124 functionality changes ‣ 19% CTF, 13% non-CTF CTF participants considered security more often � 10
VULNERABILITIES CONSIDERED XSS XSS CSRF SSRF SQLi SQLi Non-CTF Privacy Privacy CTF Logic Logic Local File Local File Disclosure Disclosure Auth Bug Auth Bug 0 20 40 60 Percentage of functionality changes � 11
VULNERABILITIES CONSIDERED XSS XSS CSRF SSRF SQLi SQLi Non-CTF Privacy Privacy CTF Logic Logic Local File Local File Disclosure Disclosure Everyone considered logic- based vulnerabilities Auth Bug Auth Bug 0 20 40 60 Percentage of functionality changes � 11
VULNERABILITIES CONSIDERED CTF participants considered non- XSS XSS functionality vulnerabilities from the CTF CSRF SSRF SQLi SQLi Non-CTF Privacy Privacy CTF Logic Logic Local File Local File Disclosure Disclosure Everyone considered logic- based vulnerabilities Auth Bug Auth Bug 0 20 40 60 Percentage of functionality changes � 11
REASONS FOR CONSIDERING ISSUES Tool ool Teammate eammate Standard Standard Practice Practice Non-CTF CTF Similar Exp. Similar Sensitive Sensitive Data Data Hacker Hacker 0 20 40 60 80 Percentage of functionality changes � 12
REASONS FOR CONSIDERING ISSUES Tool ool Teammate eammate Standard Standard Practice Practice Non-CTF CTF Similar Exp. Similar Sensitive CTF participants adopted Sensitive Data Data an adversarial mindset Hacker Hacker 0 20 40 60 80 Percentage of functionality changes � 12
ACTIONS TAKEN Teammate eammate System System Doc Doc Previous Previous Experience Exp. Non-CTF Later CTF Later Review Review External External Doc Doc Expert Expert 0 10 20 30 40 50 0 10 20 30 40 50 Percentage of functionality changes � 13
ACTIONS TAKEN Teammate eammate System System Doc Doc Previous Previous Experience Exp. Non-CTF Later CTF Later Review Review External CTF participants sought External Doc Doc help outside of their team Expert Expert 0 10 20 30 40 50 0 10 20 30 40 50 Percentage of functionality changes � 13
KNOWLEDGE ASSESSMENT Change in Assessment Score 5 5.0 2.5 2.5 0.0 0 0 2000 4000 6000 CTF Scores � 14
KNOWLEDGE ASSESSMENT Change in Assessment Score Average CTF Score 1306 5 5.0 2.5 2.5 0.0 0 0 2000 4000 6000 CTF Scores � 15
KNOWLEDGE ASSESSMENT Change in Assessment Score 5 5.0 Average Change in Assessment Score 2.5 2.5 1.36 0.0 0 0 2000 4000 6000 CTF Scores � 16
KNOWLEDGE ASSESSMENT Change in Assessment Score 5 5.0 2.5 2.5 0.0 0 0 2000 4000 6000 CTF Scores � 17
Participants with higher than average CTF scores also had KNOWLEDGE ASSESSMENT higher than average changes in assessment scores Change in Assessment Score 5 5.0 2.5 2.5 0.0 0 0 2000 4000 6000 CTF Scores � 17
Participants with higher than average CTF scores also had KNOWLEDGE ASSESSMENT higher than average changes in assessment scores Change in Assessment Score 5 5.0 2.5 2.5 0.0 0 0 2000 4000 6000 Perfect score on CTF Scores both assessments � 17
ADDITIONAL METRICS • Non-CTF participants’ commits were flagged slightly more often ‣ 2/17 Non-CTF participants flagged ‣ 1/18 CTF participants flagged • 4 CTF participants alerted security team to potential vulnerability � 18
SUMMARY 1. Do CTFs improve prevention of security issues? 2. Do CTFs improve recognition of security issues? � 19
SUMMARY 1. Do CTFs improve prevention of security issues? • Participants who solved more challenges improved in the knowledge assessment • Exposure to non-functionality vulnerabilities 2. Do CTFs improve recognition of security issues? � 19
SUMMARY 1. Do CTFs improve prevention of security issues? • Participants who solved more challenges improved in the knowledge assessment • Exposure to non-functionality vulnerabilities 2. Do CTFs improve recognition of security issues? • Increased consideration of security • Improved security team engagement � 19
Questions : dvotipka@cs.umd.edu SUMMARY vulnstudy.cs.umd.edu 1. Do CTFs improve prevention of security issues? • Participants who solved more challenges improved in the knowledge assessment • Exposure to non-functionality vulnerabilities 2. Do CTFs improve recognition of security issues? • Increased consideration of security • Improved security team engagement � 19
Recommend
More recommend
