Why Cant Johnny Fix Vulnerabilities: A Usability Evaluation of - PowerPoint PPT Presentation

Why Can’t Johnny Fix Vulnerabilities: A Usability Evaluation of Static Analysis Tools for Security Justin Smith (Lafayette College) smithjus@lafayette.edu Lisa Nguyen Quang Do (Google) lisanqd@google.com Emerson Murphy-Hill (Google) emersonm@google.com @JustinSmith0903 smithjus@lafayette.edu https://jssmith1.github.io/

Static Analysis to the Rescue! Static analysis tools detect vulnerabilities early Static Analysis https://metier.jakarman.nl/design_sdlc/design_sdlc.html

Static Analysis to the Rescue? Static analysis tools detect vulnerabilities early Static Analysis https://metier.jakarman.nl/design_sdlc/design_sdlc.html

Unusable static analysis Static analysis tools: - produce “bad warning messages” [Christakis, 2016]; - “may not give enough information” [Johnson, 2013]; - and “miscommunicate” [Johnson, 2016] with developers. “Usable security for developers has been a critically under - investigated area” [ Acar, 2016]. “[Improving] the usability of analysis results significantly increases the utility of analysis tools.” [Sadowski, 2015]

What types of issues detract from the usability of security-oriented static analysis tools?

Tools Evaluated Three open-source tools • Find Security Bugs, RIPS, and Flawfinder One commercial tool FindSecBugs Flawfinder Commercial PHP RIPS Tool

Tools Evaluated YOUR TOOL HERE Replication Package

Approach Heuristic walkthrough evaluation Phase 1: Cognitive walkthrough Phase 2: Heuristic evaluation User study Observed participants (n = 12) as they used the four tools Analysis Identified 194 (heuristic walkthroughs) + 140 (user study) usability issues Open card sort to group issues into unique themes for presentation

Overview of Findings Themes Subthemes Missing Affordances Managing Vulnerabilities Applying Fixes Missing or Buried Vulnerability Prioritization Information Fix Information Scalability of Vulnerability Sorting Interfaces Overlapping Vulnerabilities Scalable Visualizations Inaccuracy of Analysis Code Disconnect Mismatched Examples Immutable Code Workflow Continuity Tracking Progress Batch Processing

Overview of Findings Themes Subthemes Missing Affordances Managing Vulnerabilities Applying Fixes Missing or Buried Vulnerability Prioritization Information Fix Information Scalability of Vulnerability Sorting Interfaces Overlapping Vulnerabilities Scalable Visualizations Inaccuracy of Analysis Code Disconnect Mismatched Examples Immutable Code Workflow Continuity Tracking Progress Batch Processing

Findings Problem: Visual scalability over large programs

Findings Problem: Unclear severity scales

Findings Problem: Buried warnings

Takeaways Usability issues detract from security-oriented static analysis tools. Using relatively inexpensive heuristic walkthroughs, we can identify and address these issues!

Takeaways Usability issues detract from security-oriented static analysis tools. Using relatively inexpensive heuristic walkthroughs, we can identify and ^ address these issues! you } @JustinSmith0903 smithjus@lafayette.edu https://jssmith1.github.io/ Replication Package