Invariants and State in Testing and Formal Methods Dick Hamlet - PowerPoint PPT Presentation

Invariants and State in Testing and Formal Methods Dick Hamlet Portland State University Supported by NSF CCR-0112654 and SFI E.T.S. Walton Fellowship 1/10

� � � The Simplest Context Meaning of a program with persistent state: input domain ( think: STDIN ) output domain ( think: STDOUT ) state space ( think: permanent R/W file) 2/10

✁ � ✂ ✁ ✆ � ✝✞ ✟ ✂ ☎ ✆ � ☎ ✠ � ✞ ✄ The Simplest Context Meaning of a program with persistent state: input domain ( think: STDIN ) output domain ( think: STDOUT ) state space ( think: permanent R/W file) 2/10

State is Anomalous On the other hand... On the one hand... 3/10

State is Anomalous On the other hand... On the one hand... States are ‘inputs’ that influence program be- havior 3/10

State is Anomalous On the other hand... On the one hand... States are ‘inputs’ that States are ‘outputs’ influence program be- that only the program havior creates 3/10

State is Anomalous On the other hand... On the one hand... States are ‘inputs’ that States are ‘outputs’ influence program be- that only the program havior creates (bottom line) A state variable is not independent – sample at your own risk! 3/10

✂ � ✞ � � ✂ � ✂ ✞ � ✁ Testing Viewpoint Stateless case: Black-box program . Specification function . Test point fails if . Operational profile: Usage P .d.f. on . 4/10

✂ ✞ � ☎ ✁ ☎ � � ✞ � ✂ ✂ � � ✂ ✁ � � � Testing Viewpoint Stateless case: Black-box program . Specification function . Test point fails if . Operational profile: Usage P .d.f. on . Persistent state: Replace by sequences . ✁✄✂ . ( Sequence profile) 4/10

� ✂ � ☎ ✁ ☎ � � ✞ � ✂ ✂ ✞ � ✂ ✁ � � � Testing Viewpoint Stateless case: Black-box program . Specification function . Test point fails if . Operational profile: Usage P .d.f. on . Persistent state: Replace by sequences . ✁✄✂ . ( Sequence profile) State is only implicit — tester may sample ...(?) 4/10

☎ ☎ ✠ ☎ ✆ ☎ ✝ ✆ ☎ ✝ ✄ ✆ ✝ ✄ ☎ ✆ ✝ ✄ � ✠ ✁ ✄ ☎ ✁ ✠ ☎ ✁ ✆ ☎ ✁ ✄ ✆ ✝ Proving Viewpoint Specification is a first-order formula in values of program variables . Type, Symbol Evaluation Variables ( original) Pre-cond before Post-cond after Assertion any Invariant before/after 5/10

✝ ☎ ☎ ✠ ☎ ✆ ☎ ✆ ✆ ☎ ✝ ✄ ✝ ☎ ✄ ✆ ✠ ✄ ☎ ✆ ✄ ✁ ☎ ✆ ✁ ✠ ✝ ✁ ✆ ☎ ✄ ✁ � ✝ Proving Viewpoint Specification is a first-order formula in values of program variables . Type, Symbol Evaluation Variables ( original) Pre-cond before Post-cond after Assertion any Invariant before/after State variable is explicit – specification is state-prescriptive...(?) 5/10

✂ ☎ ✝ ✆ ☎ ✝ ✄ ✂ ✞ ✄ ☎ ✁ ✆ ✝ ✆ ✆ ☎ ✄ ☎ ✝ ✄ � ✠ ✞ ✂ ✄ ☎ ☎ Invariants in Proofs Room for confusion – First-order formulas include implicit evaluation times; Hoare logic hides quantification. For example, correctness of program : 6/10

☎ ✆ ✞ ✂ ✄ ✆ ☎ ✝ ✆ ☎ � ✄ ☎ ✁ ☎ ✁ ✂ ✄ ☎ ✆ ✞ ✁ ✂ ✄ ✞ ✂ ✠ ✆ ✝ ✝ ✂ ✞ ✂ ✠ ☎ � ✄ ✝ ☎ ✄ ☎ ✆ ☎ ☎ ✆ ✁ ✂ ✄ ✞ ✂ ✄ ✝ ☎ ✆ ✝ ✄ Invariants in Proofs Room for confusion – First-order formulas include implicit evaluation times; Hoare logic hides quantification. For example, correctness of program : Invariant role filter out -impossible states. Pre-condition role filter out inputs humans agree not to use. 6/10

☎ ✄ ✂ ✄ ✞ ✞ ✠ ✂ ✄ Testing with Invariants Stateless testing of to approximate proof: Sample , and for each such that , run and check . (TestEra) 7/10

✞ ✞ ✁ ✂ ✂ ✄ ☎ ✆ ✄ ✝ ✁ ✂ ✄ ☎ ✆ ✞ ✄ ☎ ✂ ✄ ✞ ✠ ☎ ✄ ✆ ✂ ✞ ✞ ☎ ✝ ✆ ✂ ✄ ☎ ✠ � Testing with Invariants Stateless testing of to approximate proof: Sample , and for each such that , run and check . (TestEra) With state it’s more complicated. First try: Sample . For each such that , run and check . 7/10

✞ ✆ ✂ ✄ ✝ ☎ ✆ ✝ ☎ ☎ ✞ ✠ ✞ ✆ ☎ � ✄ ✄ ✄ ✞ ☎ ✞ ✄ ✁ ✄ ✂ ✄ ☎ ✆ ✂ ✄ ✁ ✂ ✄ ☎ ✆ � � ✂ ✁ ✄ � ✄ ✆ ✁ ✁ ✆ ☎ ✁ ✂ ✁ ✞ ☎ ✆ ✁ ✞ ✂ ✂ ☎ ✂ ✆ ✄ ✟ ☎ ✂ ☎ ✂ ✂ ✂ ☎ ✄ ✄ ✄ � ✄ ☎ ✁ ✁ ☎ ✞ ✂ ☎ ✂ � � ☎ � � � � � � � � � � � � � � � � � � � � ✞ ✞ ✠ ☎ ✄ ✄ ✂ ✄ ✆ ☎ ☎ ☎ ✂ ✄ ☎ ✠ ✞ ✆ � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ☎ Testing with Invariants Stateless testing of to approximate proof: Sample , and for each such that , run and check . (TestEra) With state it’s more complicated. First try: Sample . For each such that , run and check . Better: Sample , say , such ✁ ✝✆ that . Sample . If , run on the sequence, obtaining state sequence and ✆ ✡✠ check . ✄☞☛ 7/10

Proof-, Testing-like Formulas Let be a logical formula (invariant, post-condition, etc.) applied to a program. 8/10

Proof-, Testing-like Formulas Let be a logical formula (invariant, post-condition, etc.) applied to a program. is Proof-like : No test case can falsify . is Testing-like : There is a low probability that test-case sequences drawn according to a given operational profile will falsify . Since profiles are arbitrary human specifications, proof-like and testing-like can be very different. 8/10

Proof-, Testing-like Formulas Let be a logical formula (invariant, post-condition, etc.) applied to a program. is Proof-like : No test case can falsify . is Testing-like : There is a low probability that test-case sequences drawn according to a given operational profile will falsify . Since profiles are arbitrary human specifications, proof-like and testing-like can be very different. itself can be proof- or testing-like if it is obtained using all possibilities, or only those from a profile. 8/10

Daikon, TestEra, Etc. Daikon TestEra Generates bounded Generates possible exhaustive testset pre- and (BET) from given post-conditions from pre-condition; checks given testset. given post-condition. 9/10

Daikon, TestEra, Etc. Daikon TestEra Generates bounded Generates possible exhaustive testset pre- and (BET) from given post-conditions from pre-condition; checks given testset. given post-condition. +invariant +profile 9/10

� ✁ Daikon, TestEra, Etc. Daikon TestEra Generates bounded Generates possible exhaustive testset pre- and (BET) from given post-conditions from pre-condition; checks given testset. given post-condition. +invariant +profile From invariant and profile, generate BET; check invariant as post-condition. Use BET to generate possible post-condition. 9/10

� � � Summary Testing needs to recognize state and invariants Sample state with care! Drive sampling with invariants 10/10

� � � � Summary Testing needs to recognize state and invariants Sample state with care! Drive sampling with invariants Invariants are inherently prescriptive 10/10

� � � � � Summary Testing needs to recognize state and invariants Sample state with care! Drive sampling with invariants Invariants are inherently prescriptive Operational profiles define ‘usage invariants’ 10/10

� � � � � � Summary Testing needs to recognize state and invariants Sample state with care! Drive sampling with invariants Invariants are inherently prescriptive Operational profiles define ‘usage invariants’ Tools using first-order formulas with tests need specification-based invariants 10/10

� � � � � � Summary Testing needs to recognize state and invariants Sample state with care! Drive sampling with invariants Invariants are inherently prescriptive Operational profiles define ‘usage invariants’ Tools using first-order formulas with tests need specification-based invariants 10/10