"I Wouldn't Have Seen It If I Hadn't Believed It: Confirmation - PDF document

T2 Track 4/29/2010 9:45 AM "I Wouldn't Have Seen It If I Hadn't Believed It: Confirmation Bias in Testing" Presented by: Michael Bolton DevelopSense Brought to you by: 330 Corporate Way, Suite 300, Orange Park, FL 32073 888 ‐ 268 ‐ 8770 ∙ 904 ‐ 278 ‐ 0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Michael Bolton Michael Bolton has been teaching and providing consulting on software testing on five continents for nine years. He is the co-author (with senior author James Bach) of Rapid Software Testing, a course that presents a methodology and mindset for testing software expertly in uncertain conditions and under extreme time pressure. He has been Program Chair for TASSQ and is a co-founder of the Toronto Workshops on Software Testing. Michael wrote the Test Connection columns in Better Software magazine for three years and sporadically produces his own newsletter. Michael lives in Toronto, Canada, with his wife and two children and can be reached at mb@developsense.com or at developsense.com .

“I Wouldn't Have Seen It I Wouldn t Have Seen It If I Hadn't Believed It: Confirmation Bias in Testing” Mi h Michael Bolton l B lt michael@developsense.com DevelopSense http://www.developsense.com STAR EAST April 2010 Testing is About Making Sure That People Aren’t Being Fooled …and we have to start with ourselves 1

The Big Theme of This Workshop • Jerry Weinberg Feedback Loop: Perception & Conception I wouldn’t have What we seen it if I b li believe hadn’t believed it. What we can see What we’ve been trained to do 2

Confirmation Bias The tendency to y see believe behave interpret investigate test filter evaluate choose in ways that fit with your hypotheses hypotheses goals goals predictions expectations beliefs biases preconceptions rather than challenging them . Confirmation Bias: Variations anchoring bias availability bias congr ence he ristic congruence heuristic Pollyanna principle belief perseverance valence effect survivorship bias bandwagon effect selection bias recency bias recency bias endowment effect endowment effect congruence bias concurrency bias assimilation bias choice-supportive bias experimenter bias and, for today’s talk… 3

Positive Test Strategy • “A tendency to test cases that are expected (or known) to have the property of interest rather than known) to have the property of interest rather than those expected (or know) to lack that property.” • “…can be a very good heuristic for determining the truth or falsehood of a hypothesis under realistic conditions.” • It can, however, lead to systematic errors or It can however lead to systematic errors or inefficiencies. • Klayman and Ha, 1987 Positive Test Strategy • “When concrete, task-specific information is lacking or cognitive demands are high people rely lacking, or cognitive demands are high, people rely on the positive test strategy as a general default heuristic .” BUT… • “emphasis on the sufficiency of one’s actions is enhanced when one is rewarded for each individual success rather than only for the final rule individual success rather than only for the final rule discovery.” • Klayman and Ha, 1987 4

The Great Traps of Test Cases If you want to know how the system works, Escaping Confirmation Bias • Practice the skill of factoring • Change a factor in your description Ch f t i d i ti • Change one of the nouns to something else • Be more specific • Add adjectives to your description • Consider the opposite • Manage the focus of your attention M th f f tt ti • Alternate between focusing and defocusing • Reverse figure and ground 5

Factoring: Dimensions of Interest • Factoring is the process of analyzing an object, event or model to determine the elements that event, or model to determine the elements that comprise it. • When we factor for the purposes of test design, we identify elements that we may need to observe or control during a test. We call those factors . • Factors may be Factors may be • intrinsic or relational • variable or static Managing Attention • To test for anticipated problems… • To test a simple product or part of a complex product very • To test a simple product, or part of a complex product very thoroughly… • To pinpoint an observed problem… • To confirm that a fix has been made… • To maximize test integrity… • To stay in grooves… y g 6

Focusing Heuristics • Start the test from a known, clean state • reset the system; observe established pre-conditions reset the system; observe established pre conditions • Prefer simple, deterministic actions • focus on a relatively small number of tests • Vary One Factor At a Time (OFAT) • Develop and test to a specified model • trace test steps to that model • F ll Follow established and consistent lab procedures t bli h d d i t t l b d • Make specific predictions, observations and records • test to observe that the system fits the patterns • Make the test easy to reproduce • automated input may help Managing Attention • To find unexpected problems … • To find elusive problems in sustained field use • To find elusive problems in sustained field use… • To test whether a fix has broken something else… • To discover new dimensions of the product or the testing mission… • To get out of ruts… 7

Defocusing Heuristics • Start from a variety of different states • not necessarily clean; don’t necessarily reset the system • not necessarily clean; don t necessarily reset the system • Prefer complex, challenging actions • considering perform a huge number of tests • Vary Many Factors At a Time (MFAT) • Generate tests from a variety of models • or without reference to a conscious model • Question your lab procedures and tools • Try to see things with open expectations • let the patterns come to you • Make the test hard to pass, instead of easy to reproduce • automatic logging and screen recording may help Reverse Figure and Ground 8

Heuristic: A vs. THE • Example: “A problem…” instead of “THE problem…” • Using “A” instead of “THE” helps us to avoid several kinds of critical thinking errors • single path of causation • confusing correlation and causation • single level of explanation Heuristic: Unless… • To test your description, try adding “unless…” to it, and performing a test based on that 9

Heuristic: The Rule of Three • Special case of the Rule Of At Least Three: Readings • Klayman, Joshua; Young-Won Ha (1987). "Confirmation, Disconfirmation and Information in Hypothesis Testing". yp g http://www.stats.org.uk/statistical-inference/KlaymanHa1987.pdf • Tools of Critical Thinking (Levy) • Exploring Requirements (Weinberg) • Perfect Software and Other Illusions About Testing (Weinberg) • Lessons Learned in Software Testing (Kaner Bach and • Lessons Learned in Software Testing (Kaner, Bach, and Pettichord) • Quality Software Management, Vol. 1: Systems Thinking (Weinberg) • Quality Software Management, Vol. : First-Order Measurement (Weinberg) 10

Readings • How To Lie With Statistics (Huff) • The Black Swan (Taleb) Th Bl k S (T l b) • An Introduction To General Systems Thinking (Weinberg) • Measuring and Managing Performance in Organizations (Austin) • Software Testing as a Social Science (Kaner) • http://www.kaner.com/pdfs/KanerSocialScienceSTEP.pdf • How To Solve It (Polya) • Politics and the English Language (Orwell) 11