Configuration Sampling Reducing the exponential number of configurations to a manageable size 41
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 {C 1 , C 2 , C 3 , C 4 , C 5 , …} Exponentially large set T = {t 1 , t 2 , t 3 , t 4 , t 5 } 42
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1 , C 3 , C 4 } Sampling {C 1 , C 2 , C 3 , C 4 , C 5 , …} Manageable size set Exponentially large set T = {t 1 , t 2 , t 3 , t 4 , t 5 } 43
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1 , C 3 , C 4 } Sampling {C 1 , C 2 , C 3 , C 4 , C 5 , …} Manageable size set Exponentially large set T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } 44
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1 , C 3 , C 4 } Sampling {C 1 , C 2 , C 3 , C 4 , C 5 , …} Manageable size set Exponentially large set T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } Example: Configuration Interaction Testing (CIT) 45
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1 , C 3 , C 4 } Sampling {C 1 , C 2 , C 3 , C 4 , C 5 , …} Manageable size set Exponentially large set T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } Example: Configuration Interaction Testing (CIT) We choose to test IE7 only under sampled configurations C 1 , C 3 , and C 4 and for each configurations we test IE7 with all tests {t 1 , t 2 , t 3 , t 4 , t 5 } 46
vim : A configurable system 47
vim : A configurable system 2 90 configurations 48
vim : A configurable system CIT selects 2 90 configurations sampling 60 configurations 49
vim : A configurable system CIT selects 2 90 configurations sampling 60 configurations Rerun the full test suite on each 60 configurations 50
vim : A configurable system CIT selects 2 90 configurations sampling 60 configurations Rerun the full test suite on each 60 configurations 7 hours to execute the full test suite Takes 7*60 = 420 hours (~2.5 weeks) to run all test cases under each configuration 51
vim : A configurable system CIT selects 2 90 configurations sampling 60 configurations Rerun the full test suite on each 60 configurations 7 hours to execute the full test suite Takes 7*60 = 420 hours (~2.5 weeks) to run all test cases under each configuration Do we have to run all tests under each configuration? 52
Test case selection when configuration under test changes* * Qu, Acharya, Robinson, “Impact analysis of configuration changes for test case selection”, ISSRE 2011 53
Test case selection when configuration under test changes* IE7 * Qu, Acharya, Robinson, “Impact analysis of configuration changes for test case selection”, ISSRE 2011 54
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, …} C 2 = {ON, Enabled, Yes, …} * Qu, Acharya, Robinson, “Impact analysis of configuration changes for test case selection”, ISSRE 2011 55
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, …} C 2 = {ON, Enabled, Yes, …} What test cases should I re-run for the new configuration? T = {t 1 , t 2 , t 3 , t 4 , t 5 } * Qu, Acharya, Robinson, “Impact analysis of configuration changes for test case selection”, ISSRE 2011 56
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, …} C 2 = {ON, Enabled, Yes, …} What test cases should I re-run for the new configuration? T = {t 1 , t 2 , t 3 , t 4 , t 5 } T’ = {t 2 , t 5 } * Qu, Acharya, Robinson, “Impact analysis of configuration changes for test case selection”, ISSRE 2011 57
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, …} C 2 = {ON, Enabled, Yes, …} What test cases should I re-run for the new configuration? T = {t 1 , t 2 , t 3 , t 4 , t 5 } T’ = {t 2 , t 5 } For the ABB system analyzed, only about 20% of the tests had to be re-run for a configuration change * Qu, Acharya, Robinson, “Impact analysis of configuration changes for test case selection”, ISSRE 2011 58
IE7 Internet Explorer IE6 IE5 Product Evolution IE1.0 & 2.0 59
Configuration prioritization for regression testing* *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 60
Configuration prioritization for regression testing* IE5 {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 61
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 62
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 4 , C 3. … } Prioritization {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 63
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 4 , C 3. … } Prioritization {C 1 , C 2 , C 3 , C 4 , C 5 } Reorder and run as many as you can T = {t 1 , t 2 , t 3 , t 4 , t 5 } *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 64
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 4 , C 3. … } Prioritization {C 1 , C 2 , C 3 , C 4 , C 5 } Reorder and run as many as you can T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 65
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 4 , C 3. … } Prioritization {C 1 , C 2 , C 3 , C 4 , C 5 } Reorder and run as many as you can T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } Increases rate of fault detection. But… *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 66
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 4 , C 3. … } Prioritization {C 1 , C 2 , C 3 , C 4 , C 5 } Reorder and run as many as you can T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } Increases rate of fault detection. But… Does not eliminate redundancy. Does not detect all faults. Not safe. *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 67
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1 , C 4 , C 3. … } Prioritization {C 1 , C 2 , C 3 , C 4 , C 5 } Reorder and run as many as you can T = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } Increases rate of fault detection. But… Does not eliminate redundancy. Does not detect all faults. Not safe. Can we select a subset of {C 1 , C 2 , C 3 , C 4 , C 5 } that is both non-redundant and safe? *Qu, Cohen, Rothermel, “Configuration-aware regression testing: An empirical study of sampling and prioritization”, ISSTA 2008 68
Configuration selection for regression testing (Focus of this talk) 69
Configuration selection for regression testing (Focus of this talk) IE5 {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } 70
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } 71
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1 , C 5 } Selection {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } 72
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1 , C 5 } Selection {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection 73
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1 , C 5 } Selection {C 1 , C 2 , C 3 , C 4 , C 5 } T = {t 1 , t 2 , t 3 , t 4 , t 5 } T’ = {t 1 , t 2 , t 3 , t 4 , t 5 } No test case selection {C 1 , C 5 } is both safe (wrt retest-all configurations) and non redundant 74
State of the Art in Configurable System Testing Focus of this talk 75
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 76
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk Test case selection [ISSRE ‘11] Single version Configuration under test changes Non-redundant Safe 77
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk Configuration prioritization [ISSTA ‘08] Test case selection [ISSRE ‘11] Source code changes Single version Regression Testing Configuration under test changes No test case selection Non-redundant Redundant Safe 78 Not safe
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Configuration selection [ICSM ‘12] Source code changes Focus of Regression testing this talk Non-redundant Safe No test case selection Configuration prioritization [ISSTA ‘08] Test case selection [ISSRE ‘11] Source code changes Single version Regression Testing Configuration under test changes No test case selection Non-redundant Redundant Safe 79 Not safe
Empirical Evaluation Implementation Conclusions Approach Motivation Outline 80
Key Idea: Map configuration options to code 81
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 82
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 IE5 change 83
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 IE5 change 84
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 IE5 change For ABB systems, configurable options (stored in a DB) maps to variables in the source code 85
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 86
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 87
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 88
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 89
Key Idea: statically compute impact of the changes Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 90
Key Idea: Intersect configuration impact with change impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 91
Key Idea: Intersect configuration impact with change impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 Select configuration option “Google Toolbar” for regression testing Safely discard “pop-up blocker” and “Do Not Track” 92
Example Program Mapping configurable options to source code Configurable Options: {P 1 , P 2 , P 3 } 93
Example Program Mapping configurable options to source code Configurable Options: {P 1 , P 2 , P 3 } 94
Example Program Mapping configurable options to source code Configurable Options: {P 1 , P 2 , P 3 } 95
Example Program Mapping configurable options to source code Configurable Options: {P 1 , P 2 , P 3 } 96
Example Program Mapping configurable options to source code Configurable Options: {P 1 , P 2 , P 3 } Function f 1 changes 97
Example 98
False False False Configurable Options Values True True True Options P 1 P 2 P 3 Example 99
Example Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 2 P 1 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 100
Recommend
More recommend
