BIO PRESENTATION T6 August 22, 2003 2:00PM R EDUCE R ISK U SING S ECURITY QA A UTOMATION T ECHNIQUES Alexander Mouldovan Cenzic Inc International Conference On Software Test Automation August 19-22, 2003 Boston, MA USA
Alexander Mouldovan As Director of Product Marketing for Cenzic, Alexander Mouldovan is responsible for the definition of Cenzic’s award-winning security QA platform, Hailstorm. He has developed and tested innovative software and hardware for trusted email, digital rights management, security, scanner, and pen computing applications.
Reduce Risk Using Security QA Automation Alexander Mouldovan Cenzic Inc. Presented to Test Automation March, 2003 Update Available at: http://www.cenzic.com/CenzicTASlides.ppt
• 12+ years in software – 4 years product management – 5 years as developer – 4 years testing and evaluating software and hardware products
Outline • Security Background • The Problem • Vulnerabilities (some examples) • Some solutions • Security QA • Opportunities
Introduction
The Problem • Evolution of security –Security by obscurity • Obscurity doesn’t last –Hardened perimeters • “Crunchy on the outside, chewy on the inside”
Evolution –Perimeter security – firewall, IDS, … • Have to let data through –Web services • Built to tunnel through firewalls! • Functionality over security – Security is the #1 reason for not adopting WS –Applications: the final frontier • WIDE OPEN!!!
Security Technology • Firewall, IDS, Intrusion Prevention • Access management, encryption • Scanners (Nessus) • Anti-Virus (Symantec) • Integrated frameworks (Symantec, CA)
The Weakest Link • A system is only as secure as the least secure link • Now that perimeter security has matured, Application logic is the weakest link • Gartner Group, 2002 – "However, close to 75% of today's attacks are tunneling through applications.” – John Pescatore, Security Analyst, Gartner Group
Vulnerabilities
Common Vulnerabilities • Buffer Overflow • Data Input Validation • Buffer Underflow • Random input • SQL Parser • Session ID Hijacking • SQL Disclosure • Parameter tampering • Command Insertion • Cross site scripting • Path traversal • Privilege escalation • Format String Tests • Alternate Encodings
SQL Parser Attack
SQL Disclosure Attack
Where are these Vulnerabilities Found? • Vendors’ platforms • Your custom code – Databases – ASP – Appservers – JSP – Network equipment – Java beans – Messaging – CGI, PHP platforms – ISAPI – Operating systems – .Net – Java – C/C++
Severity of a Defect • Not all bugs are created equal • Compare: – Cosmetic bug – Bug that prevents access to functionality – Bug that crashes the server – Bug that reveals customer data • Credit cards numbers • Passwords
Denver Airport Baggage • Unmanned carts on a track • Bad failure recovery/detection – Piles of fallen bags would not stop the unloaders • Carts got out of sync – Full carts continue to get loaded – Empty carts get unloaded • Delayed airport opening for 11 months – $1 million dollars a day in cost due to interest bond issues
What are Companies Doing?
Finding Security Vulnerability Early Saves Engineering Time and Money 50% 40% 30% Where bugs are found 20% 10% 0% Requirements Code/Unit Test Integration Beta Test Post Prod'n 16 14 12 10 8 Hours per bug fix 6 4 2 0 Requirements Code/Unit Integration Beta Test Post Prod'n Test Source: National Institute of Standards and Technology
Outsourced Testing Penetration tests: • Costly (ex. small engagements run ~$500,000) • Time consuming • Black box • Human driven, not repeatable • Run against production applications
Stretching Scarce Resources • Small internal security groups – Don’t scale, skill levels vary widely – Delay shipment, pressure to ship with holes – Estimated cost to fix a security problem after deployment: $100,000
Nothing – Live with risk and hacks – Rely on “Damage Control” – Over 90% of companies surveyed detected security breaches with over 80% incurring financial losses as a result… * – 44% were willing/able to quantify their losses… averaging over $2M/yr per respondent* *Computer Crime and Security Survey, Computer Security Institute 4/02
Relative Costs • Hacked System Cost per Incident – $1,000,000/hour of downtime* Downtime Due to Hacks • Fix bugs after deployment – $100,000 per bug in finished Scanning at Production product • Outsourced testing Outsource Testing – $10,000 per function point Automation • Automated Security QA – < $1,000 per defect Design Implement Integration Production Phase * Financial Services
Can We Do Better?
Testing Approaches • Black Box – Deep coverage of possible vulnerabilities using broad input set • White Box – Analyze source code, designs for flaws • Grey Box – Use black box tools with monitoring technology
Testing For Security • Destructive testing – QA with attitude! – Try to find weaknesses to exploit • Need standardized methods, metrics – Hunt and peck security testing doesn’t reduce risk reliably • Need to leverages security expertise within organization – Most QA teams do not have security backgrounds
Fault Injection • Critical systems must not fail – BART tested with fault injection • Create failures in a controlled environment – Launching frozen chickens into jet engines to observe failure – Crash test dummies for safety testing • Don’t trust your input – UAL ticket scam for Europe airfares (unchecked text entry)
Fault Injection (2) • Fault Injection is the ultimate black box tool – use the most malicious input possible • Stress the system every way possible prior to deployment > This is how you reduce risk.
Fault Detection • When input was perturbed, did the application behave correctly? – Did it give an error page? • How does the application fail? – Did it reveal stored data or information about the infrastructure? – How long did it take to respond?
Security QA
Integrating Security and QA • Security expertise • Quality Assurance – Precious resource – Close to the developer – Knowledge of – Use repeatable vulnerabilities processes – Can assess severity of – Critical for reliability of vulnerabilities applications – Can teach secure – Automation skill coding – Track and manage defects
Security Test Automation • Create a baseline • Insert inappropriate data, change timings, alter state transitions, break assumptions • Watch for unexpected behavior • Identify, track, remove issues before deployment
How Security and QA Collaborate • Security designs policies that define security testing • QA applies policies to applications in development – Manually run Fault Injection – Embed Fault Injection in build validation suite
Security Personnel and QA collaboration is critical • Policy documentation is a dead-end – Policies must be codified and implemented • QA and Security experts can define appropriate tests for application components • QA team learns about Security, Security team gets visibility into compliance
Benefits
Visibility into Risk
Vulnerability Breakdown
Conclusions
Reliability • Reliability requires: – Functionality – Performance … and … Security!
Pain is High • Companies are ill prepared to deal with Security Testing – Spending lots of time on vulnerabilities – Wasting gobs of money • Chance to impact the bottom line – Reduce the cost of fixing critical bugs – Get applications to market faster
Hottest Sector in QA: Security! • Security: the fastest growing segment in Quality Assurance • Great opportunity: – Risk of being hacked is high • (probability of hack * loss due to hack) – Not many experts in the field
Recommend
More recommend