Static Analysis versus Software Model Checking for bug finding Dawson Englers, Madanlal Musuvathi Stanford University, CA, USA Presented By: Prateek Agarwal ETH, Zürich
Agenda ● Goal ● Aim ● Scope ● Methodologies used – Meta Compilation – C Model Checker ● Case Studies ● Conclusions
Goal Checking validity of general perception Static Analysis is easy to apply, finds shallow bugs Model Checking is harder but strictly better
Aim ● Documentation of experience ● Set of case studies ● Comparison of two approaches, based on – Bugs/Lines of Code – False positives vs Actual Bugs – Time & Effort required – Types of Bugs: generic vs protocol specific ● Find as many bugs as possible
Scope ● Verification of system/protocol software – Critical – Hard to test/inspect/reason about manually – Highly optimized ● Protocol codes follow event driven model
Meta Compilation (MC) ● Extension of compiler ● Used for Static Analysis & Model Checking ● Metal: Language for – Slice specification by pattern matching – Code translation – State machines ● xg++ over g++ compiler
Meta Compilation: Key Features ● Code should be highly structured – e.g. special MACRO/functions for read/write ● No source code annotation needed ● Emphasis on reducing false positives ● No Simulation of code – No model of heap, tracking variable values – Simulation tends to become model checking
Static Analysis using MC ● Patterns & Actions ● Small State machines for checking Start Example Rule: Sync before read ... start: wait_for_db_full {WAIT_FOR_DB_FULL(addr);} ==> stop | {MISCBUS_READ_DB(addr);} ==> misc_bus_read_db {err(“Buffer read not sync");}; .... Stop Error Engler & Masuvathi [1]
Model Checking using MC ● Extraction & Translation – protocol code Mur Φ Model → ● Eliminates handcrafted system models ● MurΦ – Explicit state enumeration model checker – Each reachable state visited once
Model Checking using MC Translation Pattern State Variable Protocol Code (Metal Printer) (Metal Slicer) (Implementation) xg++ Legend: Automatically Generated Hand Crafted by User Correctness Protocol Model Program Properties Unedited source code Start State MurØ Hardware Model Error List Lie, Chou, Engler, Dill [2]
Example: Model Checking using MC Metal Slicer: ... /* Patterns for n/w and proc mesgs, which use length field */ pat sends = { NI_SEND (type, data, keep,swp, wait,nl);} | { P1_SEND (type, data, keep,swp, wait,nl);} ... Metal Printer Lie, Chou, Engler, Dill [2] ... /* Automatically insert length assertion before send */ {N1_SEND(type, data, keep, swap, wait, null);} ==> { if(mgk_int_cst(data) != 0)a mgk_e(“ assert(nh.len = len_data); ”); else mgk_e(“ assert (nh.len = len_nodata); ”); mgk_e(“ni_send(%t, %t, procNum,nh”,type,swap); ...
Model Checking : C Model Checker ● Motivation – xg++ user needs to know system intricacies – MurΦ lacks many C constructs – Eliminate need of higher level system model ● Use the implementation code itself ● Process scheduling & execution by CMC
CMC Key Features ● Searches for all transitions ● Complete system state saved and restored ● Heuristics & Optimizations for state space explosion ● Correctness properties – Assertions from implementation code – Memory leaks etc. in built – User specified invariants ● Environment Model still handcrafted
Case Study 1: FLASH ● Cache coherence protocol for FLASH – Multiprocessor architecture – Code runs on each cache miss ● 5 FLASH protocols ● Code Size: 10K-18K John Hennessy [3]
Results ● Static Analysis – Bugs Found: 33, False Positives: 28 – Most rules on buffer management – e.g. allocation before use, deallocation after use, not used after deallocation ● Model Checking – 8 bugs – e.g. queue overflows, no invalidation of an exclusive line – Handwritten model lead to delays
Summary ● Static Analysis Advantage: – Better for same set of properties – Works best for code visible rules ● Model Checking Advantage: – Can cover complex properties, invariants – e.g. counter overflow missed by SA ● Model Checking Disadvantage: – Simple models Less Coverage → – Detailed model Increased Time & Effort → – Erroneous model False Positives →
Case Study 2: AODV Protocol ● A d-hoc O n-demand D istance V ector Protocol ● Routing protocol ● Simplified environment – Input: route request, timer interrupt, packet receipt ● 3 publicly available implementations used ● Average 6 KLOC ● Model Checking using CMC
Results ● Model Checking – 42 unique bugs, 1 bug in AODV specification ● Static Analysis: 34 bugs Both Model Checking Static Analysis Generic Properties 21 1 13 Protocol Specific 0 20 0 Total 21 21 13 comparison of bug count classified by properties
Comparison ● Generic Properties: – CMC found only one bug not found by SA – SA found 13 bugs not found by CMC ● Protocol Specific Properties: – Difficult to check statically 0 bugs → ● SA hit more code but CMC hit more properties!
Summary ● Static Analysis Advantage: – Wins when checking same set of properties (again) ● Model Checking Advantage: – Simulates execution of 'entire system' – Looks for actual errors (not 'causes' of error) – Gives actual execution trace ● More checks more bugs → – CMC executed the code
Case Study 3: TCP ● TCP Code size ~ 10 * (AODV code size) ● Frequently audited, Heavily Tested code ● Only Model Checking done (using CMC) ● System tightly coupled with Environment – System: TCP stack – Environment: Linux Kernel, Kernel Modules ● Incremental refinement of model – Start simple, keep adding complexity
Case Study 3: TCP ● External Functions can be included in System ● Advantage – Model Checker will find errors in them – No effort to create stubs ● Disadvantage – Increase in state space – Dependencies on other functions ● Essential Question – Where to draw the line between system and environment?
System Separation ● Cut on narrowest interface ● TCP – Kernel module – Poorly documented – Effectively simulating kernel – Months of effort! – Difficult to determine false positives ● New approach: Use well defined interface – System calls interface – Hardware abstraction layer – Effectively running entire kernel in CMC!
Results ● 4 bugs found ● Metric to measure coverage: – Line Coverage: Lines of code covered – Protocol Coverage: %age of abstract protocol behavior covered Description Line Coverage Protocol Coverage Bugs Standard server, client 47.4 % 64.7 % 2 + Simultaneous connect 51.0 % 66.7% 0 + Partial Close 52.7 % 79.5% 2 + Message corruption 50.6% 84.3% 0 Combined Coverage 55.4% 92.1%
Conclusions ● No model is as good as implementation itself – Simplification , omission missing critical errors → ● Manual work scale badly for complex systems ● Use well defined interfaces ● More code coverage More bugs found →
My Opinion ● Usage of live software project is a big plus ● Meta-Compilation & CMC give huge advantage ● Better comparison if same tools used across case studies ● Time & Effort not measured accurately ● No comparison with other existing tools
Future Scope ● How to increase code coverage in meta compilation and CMC? ● Analysis of class of properties checked
References 1)Experiences using static analysis & model checking for bug finding – Dawson Engler and Madanlal Masuvathi 2)A simple method for extracting models from protocol code – David Lie, Andy Chou, Dawson Engler, David L. Dill 3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans – John Hennessy
Q & A
Recommend
More recommend
