Sa Safer Dri riving – St Standards for r So Soft ftware Tes esting of f Automotive Systems Stuart Reid PhD, FBCS STA Consulting Inc. (stuart@sta.co.kr)
Scope Automotive Safety Standards – ISO 26262 Testing Standards – ISO 29119, ISO 33063 & ISO 20246 Mappings between ISO 26262 and ISO 29119 – processes, techniques and documentation A co-ordinated approach – using both ISO 26262 and ISO 29119
IE IEC 61508 - Functional l safety of systems ISO 26262 Road Vehicles IEC 61513 IEC 62061 Nuclear Machinery IEC 61508 Generic IEC 61511 IEC 62279 Process Industry Railway Automotive SW Testing Conference 2016
IS ISO 26262 - Overview Vocabular ulary y (1) Managemen gement t (2) Syst stem em Produ duct ct Development lopment (4) Conc ncept ept Hardw dware are Operat ration ion Softw tware are Product duct Phase se Product duct (7) Development lopment (3) Development lopment (6) (5) Suppor portin ting g Process esses es (8) Safety ty Integr egrit ity y Level l Analysis ysis (9) Guideli elines es (10) 0) Automotive SW Testing Conference 2016
ISO ISO 26 26262 – So Software development t process Software Testing Specif ifica icati tion on of Verifica icati tion on of softw tware are safety ty softw tware are safety ty requir quireme ement nts requir quireme ement nts Test Phase Verification Design Phase Verification Software Testing Softw tware are Softw tware are integr grat ation on archit itec ectu tural al design ign and testing ting Test Phase Verification Design Phase Verification Software Testing Softw tware re unit design ign Softw tware are unit testin sting and imp mplemen ementa tati tion on Test Phase Verification Automotive SW Testing Conference 2016
ISO/IEC/IEEE 29119 – Structure IS Part 1 BS 7925-1 Concepts & Vocabulary Part 4 Part 2 Part 3 Testing Processes Documentation Techniques BS 7925-2 IEEE 829 Part 5 ISO/IEC 20246 ISO/IEC 33063 IEEE 1028 Keyword- Process Reviews Driven Testing Assessment Automotive SW Testing Conference 2016
ISO 29119-2 Test Processes IS ORGANIZATIONAL TEST PROCESS TEST MANAGEMENT PROCESSES TEST TEST TEST MONITORING & PLANNING COMPLETION CONTROL DYNAMIC TEST PROCESSES TEST TEST TEST DESIGN & TEST ENVIRONMENT INCIDENT IMPLEMENTATION EXECUTION SET-UP REPORTING Automotive SW Testing Conference 2016
ISO 26262 – Safety In IS Integrity Level (ASIL) • Severity Controllability • S1 – light/moderate injuries Severity Probability C1 C2 C3 • S2 – severe/life threatening injuries E1 • S3 - life threatening/fatal injuries E2 S1 • Probability of exposure E3 A • E1 – v. low probability E4 A B • E2 – low probability E1 E2 ASILs A • E3 – medium probability S2 E3 A B • E4 – high probability E4 B C • Controllability E1 A • C1 – simply controllable E2 A B S3 • C2 – normally controllable E3 A B C • C3 – difficult/uncontrollable E4 B C D Automotive SW Testing Conference 2016
IS ISO 26262-6 6 – Unit it Testing • 9.2 General • A procedure for testing the software unit against the software unit design specifications is established, and the tests are carried out in accordance with this procedure. • 9.4.3 • The software unit testing methods listed in Table 10 shall be applied… Automotive SW Testing Conference 2016
ISO ISO 26 26262-6 6 – So Software In Integratio ion Testing • 10.2 General • In this sub-phase, the particular integration levels and the interfaces between the software elements are tested against the software architectural design. • 10.4.3 • The software integration test methods listed in Table 13 shall be applied… Automotive SW Testing Conference 2016
IS ISO 26262 – Deriv iving Test Cases Does not say 100% Does not say 100% Automotive SW Testing Conference 2016
IS ISO 29119-4 Boundary ry Valu lue Analysis is • Test Case Design ISO 29119 ISO 26262 • Test Coverage 13 pages 1 sentence • Guidelines on Use This method applies to parameters or variables, values approaching and crossing the boundaries and out of range values. Automotive SW Testing Conference 2016
ISO ISO 26 26262-6 Verification soft ftware safety req equirements • 11.2 General • The purpose … is to demonstrate that the embedded software satisfies its requirements in the target environment. • 11.4.2 • To verify that the embedded software fulfils the software safety requirements, tests shall be conducted in the test environments listed in Table 16. Automotive SW Testing Conference 2016
IS ISO 26262 Verif ific ication 9.4.2/10.4.2/11.4.1 Software unit testing/integration testing/verification of software safety requirements shall be planned, specified and executed in accordance with ISO 26262-8:2011, Clause 9. ISO 26262-8 Clause 9 Verification Automotive SW Testing Conference 2016
ISO 26262-8 Verif IS ification Cla lause In the test phases, verification is the evaluation of the work Verification products within a test Verification environment to ensure that they Plan Planning comply with their requirements. Verification Specification Verification Specification Verification Report Verification The tests are planned, specified, Execution & executed, evaluated and documented Evaluation in a systematic manner. Automotive SW Testing Conference 2016
ISO ISO 29 29119-2 2 – Test Management Processes ORGANIZATIONAL TEST PROCESS ORGANIZATIONAL TEST FEEDBACK ON DOCUMENTATION ORGANIZATIONAL TEST DOCUMENTATION TEST MANAGEMENT PROCESSES TEST PLAN UPDATES TEST COMPLETION TEST TEST REPORT TEST TEST PLAN MONITORING PLANNING COMPLETION & CONTROL TEST PLAN, TEST PLAN, TEST PLAN, CONTROL TEST COMPLETION REPORT, CONTROL TEST MEASURES DIRECTIVES TEST MEASURES DIRECTIVES TEST DYNAMIC TEST MANAGEMENT PROCESSES PROCESSES Automotive SW Testing Conference 2016
IS ISO 29119-2 2 - Test Pla lanning Process Scope Understand C Understand Context ontext Organise Organise Test Plan Test Plan Development Development Estimated Risks Identify & Estimate Risks Treatment Identify Risk Approaches Mitigation Approaches Design Test Strategy Schedule, Staffing Determine Profile Staffing and Test Strategy Scheduling Draft Test Plan Document Test Plan Approved Gain Test Plan Consensus on Test Plan Test Plan Publish Test Plan Automotive SW Testing Conference 2016
ISO ISO 26 26262 Mapping to ISO ISO 29 29119-2 Test Pla lanning Process The roles and skills of staff to carry out Scope Understand Co Understand the testing described in the test strategy Context ntext should be identified. Organise Organise Test Plan Test Plan Development Development Estimated Risks Identify & Estimate Risks Treatment Identify Risk Approaches ISO 26262 Mapping Mitigation Approaches 14/29 shalls Design Test 0/7 shoulds Strategy Schedule, Staffing Determine Profile Staffing and Test Strategy Scheduling Draft Test Plan Record Test Plan Approved Gain Test Plan Consensus on Each required test activity in the Test Strategy Test Plan Test Plan shall be scheduled based on the estimates, Publish Test Plan dependencies and staff availability. Automotive SW Testing Conference 2016
IS ISO 26262 - Verif ification Pla lanning • No consideration of organizational test strategy / test policy • No coverage of interaction/approval from stakeholders • Nothing on estimation of required resources • no concept of constraints and compromises • Nothing on staffing or scheduling Automotive SW Testing Conference 2016
ISO ISO 29 29119-2 2 – Test Monitoring & Control Process test status report test progress test control info info Report [testing incomplete] (TMC4) test test plan Set-Up measures Monitor Control test (TMC2) (TMC1) (TMC3) [testing progress complete] info control measures directives DYNAMIC TEST PROCESS Automotive SW Testing Conference 2016
ISO 26 ISO 26262 Mapping to ISO ISO 29 29119-2 2 Test Monitoring & Control Process test status report ISO 26262 Mapping 1/17 shalls test progress test control 0/2 shoulds info info Report [testing incomplete] (TMC4) test test plan Set-Up Monitor Control measures test (TMC2) (TMC1) (TMC3) [testing progress complete] info control measures directives Means of treating DYNAMIC TEST PROCESS newly-identified and changed risks shall be identified. Automotive SW Testing Conference 2016
IS ISO 26262 – Test Management • Appears to assume that once a plan is specified then testing will simply follow the plan and no so no divergence from the plan is possible • does not require new risks to be managed after architectural design • there is no requirement for test progress monitoring while testing is being performed • there is no requirement for test status reporting while testing is being performed • there is no requirement to control the testing • so no test management Automotive SW Testing Conference 2016
ISO ISO 29 29119-2 2 Test Completio ion Process ARCHIVE TEST ASSETS CLEAN UP TEST ENVIRONMENT IDENTIFY LESSONS LEARNED REPORT TEST COMPLETION Automotive SW Testing Conference 2016
Recommend
More recommend
