Scaling Regression Testing to Large Software Systems Alessandro Orso Co-authors: Nanjuan Shi, Mary Jean Harrold College of Computing Georgia Institute of Technology Supported in part by National Science Foundation (awards CCR-0306372, CCR-0205422, CCR- 9988294, CCR-0209322, SBE-0123532, and CCR-0080900) and Boeing Commercial Airplane Group.
Testing of Evolving Software Program P’ P’’ P … T’ rerun T rerun Test Suite T’ T T T’ - T’ rerun T - T rerun Time to rerun T Analysis Time Time to rerun T rerun Savings time FSE 04
Previous and Related Work [OOPSLA01] Retest T (coverage info) Ideal solution: two-phase approach P Class-Level analysis subset of P Stmt-Level T rerun Stmt-Level analysis on the subset T rerun Analysis P’ Time to rerun T Analysis Time Time to rerun T rerun Savings time FSE 04
Previous and Related Work Precise, Expensive Techniques Retest T (coverage info) Ideal solution: two-phase approach P Class-Level analysis subset of P Stmt-Level Low-Level T rerun Stmt-Level analysis on the subset T rerun Analysis Analysis P’ Jboss, web application server, 1 million LOC Time to rerun T Analysis time Time to rerun T rerun time FSE 04
Previous and Related Work Efficient, Imprecise Techniques T (coverage info) P High-Level T rerun Analysis P’ Time to rerun T Analysis time Time to rerun T rerun time FSE 04
Previous and Related Work Related Work Efficient, Imprecise Techniques T Efficient, less precise techniques (coverage info) White and Leung [CSM92] Chen, Rosenblum, and Vo [ICSE94] Hsia et al. [SMRP97] White and Abdullah [QW97] P High-Level Ren et al. [OOPSLA04] T rerun Expensive, more precise techniques Analysis P’ Binkley [TSE97] Rothermel and Harrold [TOSEM97] Vokolos and Frankl [RQSSIS97] Ball [ISSTA’98] Rothermel, Harrold, and Dedhia [JSTVR00] Time to rerun T Harrold et al. [OOPSLA01] Analysis time Time to rerun T rerun Bible, Rothermel, and Rosenblum [TOSEM01] time FSE 04
Proposed Solution Two-Phase Technique T (coverage info) Subset P of P Class-Level Stmt-Level T rerun Subset Analysis Analysis P’ of P’ Two-phase approach Class-Level analysis subset of P and P’ Stmt-Level analysis on the subset T rerun FSE 04
Outline Background Technique Empirical Evaluation Conclusion FSE 04
Motivating Example P class A { void foo() { … } } class B extends A { } class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo(); } } class E extends D {} class F { void bar(D d) { … } } FSE 04
Motivating Example P P’ class A { class A { void foo() { … } } void foo() { … } } class B extends A { class B extends A { void foo() { … } } } class C extends B {} class C extends B {} class D { class D { void bar() { void bar() { A ref=null; A ref=null; switch(somevar) { switch(somevar) { case ‘1’: ref=new A(); break; case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } case ‘3’: ref=new C(); break; } ref.foo(); ref.foo(); } } } } class E extends D {} class E extends D {} class F { class F { void bar(D d) { … } } void bar(D d) { … } } FSE 04
Motivating Example P P’ class A { class A { void foo() { … } } void foo() { … } } class B extends A { class B extends A { void foo() { … } } } class C extends B {} class C extends B {} class D { class D { void bar() { void bar() { A ref=null; A ref=null; switch(somevar) { switch(somevar) { case ‘1’: ref=new A(); break; case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } case ‘3’: ref=new C(); break; } ref.foo(); ref.foo(); } } } } class E extends D {} class E extends D {} class F { class F { void bar(D d) { … } } void bar(D d) { … } } FSE 04
Motivating Example P P’ class A { class A { void foo() { … } } void foo() { … } } class B extends A { class B extends A { void foo() { … } } } class C extends B {} class C extends B {} class D { class D { void bar() { void bar() { A ref=null; A ref=null; switch(somevar) { switch(somevar) { case ‘1’: ref=new A(); break; case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } case ‘3’: ref=new C(); break; } ref.foo(); ref.foo(); } } } } class E extends D {} class E extends D {} class F { class F { void bar(D d) { … } } void bar(D d) { … } } FSE 04
Class-Level Analysis P P’ class B extends A { class B extends A { void foo() { … } } } FSE 04
Class-Level Analysis P P’ class A { class A { void foo() { … } } void foo() { … } } class B extends A { class B extends A { void foo() { … } } } class C extends B {} class C extends B {} FSE 04
Class-Level Analysis P P’ class A { class A { void foo() { … } } void foo() { … } } class B extends A { class B extends A { void foo() { … } } } class C extends B {} class C extends B {} class D { class D { void bar() { void bar() { A ref=null; A ref=null; switch(somevar) { switch(somevar) { case ‘1’: ref=new A(); break; case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } case ‘3’: ref=new C(); break; } ref.foo(); ref.foo(); } } } } FSE 04
Class-Level Analysis P / P’ Interclass Relation Graph class A { (for P an P’) void foo() { … } } class B extends A { void foo() { … } } A F class C extends B {} class D { D void bar() { B A ref=null; switch(somevar) { E case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; C case ‘3’: ref=new C(); break; } ref.foo(); Inheritance edge } } class E extends D {} Use edge class F { void bar(D d) { … } } FSE 04
Class-Level Analysis P / P’ Interclass Relation Graph class A { (for P an P’) void foo() { … } } class B extends A { void foo() { … } } A F class C extends B {} class D { D void bar() { B A ref=null; switch(somevar) { E case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; C case ‘3’: ref=new C(); break; } ref.foo(); Inheritance edge } } class E extends D {} Use edge class F { void bar(D d) { … } } FSE 04
Class-Level Analysis P / P’ Interclass Relation Graph class A { (for P an P’) void foo() { … } } class B extends A { void foo() { … } } A F class C extends B {} class D { D void bar() { B A ref=null; switch(somevar) { E case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; C case ‘3’: ref=new C(); break; } ref.foo(); Inheritance edge } } class E extends D {} Use edge class F { void bar(D d) { … } } FSE 04
Class-Level Analysis P / P’ Interclass Relation Graph class A { (for P an P’) void foo() { … } } class B extends A { void foo() { … } } A F class C extends B {} class D { D void bar() { B A ref=null; switch(somevar) { E case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; C case ‘3’: ref=new C(); break; } ref.foo(); Inheritance edge } } class E extends D {} Use edge class F { void bar(D d) { … } } FSE 04
Example: Stmt-Level Analysis Subset of P Subset of P’ class A class A class B {…} class B {… void foo() {…} … } class C class C class D { class D { void bar() {…; ref.foo(); …} void bar() {…; ref.foo(); …} } } } } G’ (excerpt) G (excerpt) A A … … A.foo() A.foo() A.foo() B ref.foo() ref.foo() … … B … … C B.foo() B.foo() C Dangerous Edge … FSE 04
Example: Stmt-Level Analysis Subset of P Subset of P’ class A class A class B {…} class B {… void foo() {…} … } class C class C class D { class D { void bar() {…; ref.foo(); …} void bar() {…; ref.foo(); …} } } } } G’ (excerpt) G (excerpt) A A … … Test cases to be rerun: A.foo() A.foo() A.foo() Test cases in T that B execute the call node ref.foo() ref.foo() … … B with ref’ s dynamic type … … C being B or C B.foo() B.foo() C Dangerous Edge … FSE 04
Outline Background Technique Empirical Evaluation Conclusion FSE 04
Empirical Setup Tool DejaVOO Subjects Test Retest Program Versions Classes KLOC Cases Time 525 Jaba 5 70 707 54 min Daikon 5 824 167 200 74 min Jboss 5 2,403 1,000 639 32 min Three empirical studies 1. Effectiveness of Phase 1 2. Precision gain of Phase 2 3. Overall savings in retesting time FSE 04
RQ1: Effectiveness of Phase 1 TwoPhase T (coverage info) Subset P of P Class-Level Stmt-Level T rerun Subset Analysis Analysis P’ of P’ FSE 04
RQ1: Effectiveness of Phase 1 Average: 37% Time < 14 seconds FSE 04
RQ3: Overall Savings in retesting Time Time to rerun T Analysis time Time to rerun T rerun Savings time FSE 04
RQ3: Overall Savings in Retesting Time RerunAll TwoPhase FSE 04
RQ3: Overall Savings in Retesting Time RerunAll TwoPhase Savings in Regression Testing Time: TwoPhase vs. RerunAll Jaba:19% Daikon:36% Jboss: 63% FSE 04
Outline Background Technique Empirical Evaluation Conclusion FSE 04
Recommend
More recommend
