NF Risk Assessment Framework Increasing Predictability of - PowerPoint PPT Presentation

NF Risk Assessment Framework – Increasing Predictability of Non-Functional Defects 2014

Outline  Conventional approach to NFT  What is not NFT?  Why Risk Assessment?  NF Risk Assessment Framework  Framework Explained  Applicability & Benefits 2

Conventional Approach to Non-Functional Testing (NFT) Will users get Do we have OK. the response enough CPU within 5 I know what to do. and seconds? Memory? Application Let’s do a Load Test should not crash in first and then production !! increase the number Non- of users to Stress I have got a Functional good 3 weeks the system Testing ?? before go-live Somebody’s Also Release Let’s gate crash the gotta do it Management Quality Gateway  wants it  3

Pitfalls of the Conventional Approach to NFT Pre-production Production o Day 1 – so far so good! o Am I focusing too much focus on response time SLA only? o Day 2 – all was well until 3 pm and then seemed kind of o Do I know what I am looking sluggish for ? o Day 3 – Nothing serious but o Am I ensuring adequate ‘system was unavailable’ for 15 coverage besides the happy min this morning (suspected due scenarios? to a Gateway outage the previous night) o Should I test integrated with other applications or defer it to o Day 5 – 4 EoDs were great. But the next release? EoW is a different story  o May be I have taken a one o Day 10 – Not as quick as it was size fits all approach! last week (already aging?) 4

Well, what constitutes Non-functional Testing? Wikipedia Software Quality characteristics as per ISO 9126 Standard  Baseline testing   Compliance testing Functionality  Documentation testing o Interoperability  o Endurance testing Security   Load testing Reliability  Localization testing and o Fault Tolerance Internationalization testing o Recoverability   Performance testing Usability   Recovery testing Efficiency  o Resilience testing Performance   Security testing Maintainability  Scalability testing o Stability   Stress testing Portability  o Usability testing Adaptability  o Volume testing Instability Source: http://www.sqa.net/iso9126.html Source: http://en.wikipedia.org/wiki/Non-functional_testing Non-functional Testing Performance Testing 5

NFT – Where to start and where to end? ? How do I know what non-functional quality attributes to test for? ? How do I know whether my test approach is good enough? ? How do I know how much testing is enough testing? ? How do I know what can possibly go wrong in production? ? How do I know my application is ready for production? Therefore Assess Non-Functional Risk Assessment 6

Non – Functional Risk Assessment Framework Start Risk Assessment System Appreciation & Technical Assessment Develop Risk Matrix Create Risk Catalogue Risks – Tests Traceability 7

Step 1: System Appreciation and Technical Assessment Understand Future Study Product Historical Deployment Architecture Non-Functional and Workload and Design Incidents Analysis Characteristics Incident Description  Architecture,  Infra deployment view Protocols Technical Analysis  Capacity, Network  Transaction Model Non-Functional? (Y/N)  HA and DR  Interfaces NF Domain  Workload pattern  QoS Class of Issue  Volumetrics Sub-Class  Data retention  Customer behavior Potential way of  Data replication  Business growth 8 detecting the issue  Overlapping  Co-existing components Applicable NF Test processes

Step 2: Develop NF Risk Matrix Risk Matrix = Threats Vs Focus Areas “Threat” – technical attribute or event that can impact the non-functional quality of the SUT “Focus Area” – a component or set of functionalities in the SUT that is critical to the non-functional quality of the SUT 9

Risk Matrix – Indicative Threats o Prolonged Usage o Processing Overlaps o Large Volume Workloads o Concurrency Dedicated o Database Size o Integration Complexity center of excellence o Multi Geo Access o Network Latency providing o Offline/Shutdown full- o VM Sharing fledged performan o Incorrect Error Handling o JVM Sharing ce testing solutions o Zone App Crash o Database Sharing o VM Crash o Large Volume Workloads o JVM Crash o Co-existence with Maintenance o Database Crash o Vertical Scalability (lack of) o Shared Resources o Horizontal Scalability (lack of) o Unconstrained Resource Usage o Stress Conditions 10

Risk Matrix – Threats vs Focus Areas NFT Risk Matrix Online Transaction EOD Adhoc Interfaces Global App Zone Infrastructure Non-Functional Focus Areas >> Processing Processing Reports Processing Behaviour Behaviour Utilization Dedicated (Transactional) center of NF Domain Threat TI Processing TI Systems Infrastructure excellence Scalability Processing Overlaps r p a r r p a a providing Concurrency p r g a p r g Integration Complexity a a w p w r Network Latency r g a w P a r full- Reliability w Processing Overlaps a a g g a p fledged a Stress Conditions a p a r r p performan Concurrency p a g r a a w ce testing Prolonged Usage r a a a p p a Performance solutions a Concurrency a a r a r p a Large Volume Workloads p a r a g r Database Size r a r w w a w r Multi Geo Access p w a w p a Resilience & w Offline / Shutdown r r a r r p Recoverability Incorrect Error Handling a p g a a a w w Zone App Crash r r r r w r w VM Crash a r g a a r JVM Crash a p g a a r w w Platform (Appserver/DB) Crash r r r r r r 11

Risk Matrix – Risk Ranking Non Functional and Technical Risks Summary Dedicated Risk Ranking center of Domains of Concern 1 2 3 4 5 Total excellence Scalability 8 25 28 14 9 84 providing Reliability 6 4 13 3 2 28 Performance 4 7 11 1 5 28 full- Resilience and Recoverability 3 18 11 3 7 42 fledged Capacity 0 4 7 3 0 14 performan Interoperability 1 1 3 0 2 7 Compatibility 0 0 14 0 0 14 ce testing solutions Total Count of Risks 22 59 87 24 25 217 Take-aways NF Tests will be designed targeting each of the Rank1 and Rank2 Risks These tests will also include Test Scenarios covering Rank3 Risks 12

Step3: Create Risk Catalog # 1 2 NF Domain Scalability Performance Threat Processing Overlap Multi Geo Access Dedicated Sub- Intra Zone Processing - center of Threats excellence Two or more Multi Bank Entities User sites are spread across the providing (MBE) within a Zone could be globe, however all user access have performing different operations at the full- to pass through the Global Single same time, leveraging the same Risk Sign On (SSO). There will be only one fledged application/OS/database resources primary instance of Global App in one performan and processing the same data set or location and all users will be routed ce testing accessing from the same data source through this single Global app. solutions (table/schema/database) User accesses from multiple geographies to the global App and the There will be intermittent delays in response therefore will potentially be online transaction processing (OLTP) Impact slow, influenced by the bandwidth or delays in Message transmission congestion over the wide area into the Transport Client network (WAN) between the user sites and the global app Parameters OLTP Response Time Global Dashboard Response Time to Measure Focus Area Zone Behavior Global App (SSO, Dashboard) 13

Risks – Tests Traceability Dedicated center of excellence providing full- fledged performan ce testing solutions 14

Non-Functional Risk Assessment Framework Understand Future Study Product Historical Non-Functional Deployment and Architecture and Design Incidents Analysis Workload Characteristics Non-Functional Risk Matrix Online Transaction EOD Adhoc Interfaces Global App Zone Infrastructure Non-Functional Focus/Impact Areas >> Processing Processing Reports Processing Behaviour Behaviour Utilization (Transactional) NF Domain NF Threat Processing Systems Infrastructure Scalability Processing Overlaps r p a r r p a Horizontal Scalability (lack of) r a w a a r w Reliability Processing Overlaps a a g g a p w a Prolonged Usage r a a a p p NF Risk Catalogue Risks – Tests Traceability 15

Applicability  COTS providers/product development initiatives  Large IT initiatives (e.g. platform revamps)  Large infrastructure consolidations/transformations X Stable/matured systems X Periodic releases X Minor enhancements 16

Benefits & Advantages  Gain a precise understanding of the technically Benefits: vulnerable areas of the SUT  Develop an exhaustive repository of non-functional test scenarios  Ability to design tests to simulate the specific technical risks  Ensure maximum possible coverage and traceability of the NF risks in the SUT  Predictability into all probable outcomes in production in the event of a technical failure or an unexpected workload situation or projected business growth 17

Benefits: Thank you! Q ? A Vijayanand Chelliahdhas vijayanand.c@hcl.com vijay.c.anand@gmail.com 18