Dependability Evaluation Robin Bloomfield, Bev Littlewood Centre - PowerPoint PPT Presentation

no: 1 Dependability Evaluation Robin Bloomfield, Bev Littlewood Centre for Software Reliability, City University, London Adelard, London December 2001 CSR

no: 2 Background ! Dependability problems no longer simply concern computer systems (did they ever?). Important dependability issues in the future will concern systems ! that comprise computers, people and organisations. ! Systems may be rapidly evolving, in quite unexpected ways, and face a range of sophisticated yet unknown threats. ! Addressing the dependability issues will involve not only computer scientists and the like, but also statisticians, psychologists, sociologists, economists, etc, in truly inter-disciplinary activity. CSR

no: 3 Problems of dependability requirements, evaluation and confidence We need methods and tools for the development of dependability requirements , to describe and ! trade off different attributes (trust, reliability, security, safety). ! quantitative analysis of efficacy of processes and dependability of products to aid in decision-making at all levels (building systems, procuring systems, using systems). ! to justify high confidence in a system before it has been extensively used. This is a difficult area because of the complex nature of the systems, their disparate make-up, and the paucity of evidence CSR

no: 4 Some technical issues to be addressed ! Quantitative ‘dependability cases’: beyond computers . The problems that arise from human and organisational involvement (and the possible advantages that arise from diversity of roles). Understanding the interactions: software, human, organisation. Understanding the trade offs between security and other attributes. ! Design for assessment. How do we design these complex systems so that their dependability can be assessed? Dependability trade-offs. How to balance re-engineering in systems vs organisations. Development of modelling techniques for large, heterogeneous networks. ! Structuring of dependability arguments. The issue of confidence, ‘strength’ of arguments, diversity in arguments (e.g. ‘independent argument legs’), the use of disparate evidence sources. Marriage of logical and probabilistic approaches. CSR

no: 5 More technical issues to be addressed The role of (human intellectual) diversity. In system design and during system operation. Reasoning about dependability. Understanding the psychology. Relationship between process and product attributes. What does knowledge of design processes allow us to claim? Process efficacy in delivering dependability; interactions between processes. Special examples and problems – e.g. Open Source, COTS. Generation of evidence: Large scale testing and static analysis (e.g. by using the GRID); analysis of large (operating) systems; legacy systems. Validation of dependability assessment practices and procedures. If people can’t trust dependability evaluations, they will not do them. Issues of competency and impact of registration or certification of people. Certification. CSR

no: 6 Application areas There are many ways to consider this: by technology component, application, sector etc. This is a selection: COTS/SOUP, Open Source • • Risks of a surveillance Ubiquitous, pervasive society – how do individuals systems where computers know what is happening? dominate mediation between the user and their Standards – unglamorous but environment; home assistive important impact on market. technologies. Generally not validated by • experience and unjustified. Operational risk in the finance and other commercial sectors CSR

no: 7 Means ! Inter-disciplinarity. ! Focus . ! Genuinely inter-disciplinary research ! Research has many programme, for better understanding meanings. Is the aim in of roles of humans and organisations FP6 to improve the in these complex systems. average, the best or the ! Empirical studies. weakest link? Meaning of Case studies on realistic systems, ! research has changed in and experiments. Cooperation from the different frameworks industry and commerce. and varies from DG to ! Evaluation . DG? ! Need to address quality of evaluation and resources needed to develop proposals.Made worse by inter- disciplinarily and complex nature of dependability CSR