The Impact of Human Discussions on Just-In-Time Quality Assurance Parastou Tourani, Bram Adams MCIS, Polytechnique Montréal Thursday, March 3, 16
Life Cycle of a Bug Bug Report Fix Change Bug Introducing Change Thursday, March 3, 16
Automatic Identification of Bug Introducing Changes Thursday, March 3, 16
Granularity File Level, Package Level Thursday, March 3, 16
Granularity File Level, Package Level File-Level Thursday, March 3, 16
Granularity File Level, Package Level File-Level Package-Level Thursday, March 3, 16
Granularity File Level, Package Level File-Level Package-Level Thursday, March 3, 16
Granularity File Level, Package Level File-Level Package-Level Thursday, March 3, 16
Granularity: Change or Commit Level (JIT defect model) Thursday, March 3, 16
Granularity: Change or Commit Level (JIT defect model) Clear and Early Thursday, March 3, 16
JIT Defect Model Based on Change Metrics Product Metrics, like: - Lines of code in a file before the change Process Human Factor Metrics, like: Metrics, like: - Number of modified subsystems - Developer Experience JIT Prediction Model Thursday, March 3, 16
Software Collaborative Development Activity Quality of Human Factors Discussions Thursday, March 3, 16
Two Major Discussions Issue Discussions Review Discussion Thursday, March 3, 16
Traditional Meeting Rigid, Limited adoption and, Limited efficiency Thursday, March 3, 16
Regular and quick, With recorded discussions, Traceable Thursday, March 3, 16
Different Aspects of Discussions ✴ IssueTime ✴ ReviewTime ✴ ReplyLag Sentimen Time ✴ Issue_Reporter _Experience ✴ Number_Of_ Patchsets ✴ Reviewer_Experience Focus Length Thursday, March 3, 16
Extracting Sentiments Sentiment Human Language Texts withTheir Sentiment Comments Human Scores Sentistrength Cleaning Extracting Comments Sentiments Thursday, March 3, 16
Different Dimensions of Metrics for JIT prediction Review Issue Change Thursday, March 3, 16
Different Dimensions of Metrics for JIT prediction Review Issue Change Thursday, March 3, 16
Linking Reviews and Issues Launchpad Gerrit Thursday, March 3, 16
Linking Reviews and Issues Launchpad Gerrit Thursday, March 3, 16
Studied Projects (with top most number of linkage between issues and reviews) Cinder, Devstack, Glance, Heat, Keystone, Neutron, Nova, Swift, Tempest, Openstack-Manuals Cdt, Jgit, Egit, Linuxtools, Scout.rt Thursday, March 3, 16
Research Questions RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect- introducing changes? Thursday, March 3, 16
General Approach General Logistics Model ROC Curve and Area Under Curve(AUC) Thursday, March 3, 16
Research Questions RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect- introducing changes? Thursday, March 3, 16
RQ1: Five of the projects see an improvement of 3% up to 10% Enhanced Baseline Enhanced Baseline 70 67 52.5 Precision Recall 64.5 35 62 17.5 59.5 57 0 Glance Tempest Heat Cinder Jgit 61 58.75 AUC Enhanced 56.5 Baseline 54.25 52 Cinder Glance Heat Tempest Thursday, March 3, 16
RQ1: Top Most Important Factors (have negative effect) Change Related Metrics: Type, LA, Number of Developers One Focus Metric: Issue Commenter Experience Thursday, March 3, 16
Research Questions RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect- introducing changes? Thursday, March 3, 16
RQ2: 9 projects improve their precision and/or recall by 3% to 8%. Enhanced Enhanced Baseline Baseline 80 80 60 60 Recall 40 40 Precision 20 20 0 0 neutron ...-manuals jgit egit d h k n . t c e j . g e e . d e e e g i - m t y i t v a u m t s s t t p t a t r o e 80 a o n c s n u n t k e a l s 60 Enhanced 40 Baseline AUC 20 0 Camel neutron...-manuals jgit egit Thursday, March 3, 16
RQ2: Top Most Important Factors From Change Related Metrics: Type, LA/LD, AGE From Focus Metric: Review Commenter Experience From Time Metric: Review Time Thursday, March 3, 16
Research Questions RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect-introducing changes? Thursday, March 3, 16
RQ3: 8 of the projects improve both precision and recall by 3% up to 17%. Enhanced Baseline Enhanced Baseline 90 67.5 Recall 45 Precision 22.5 0 cinder glance heat neutron ...-manuals tempest egit jgit scout.rt cinder glance heat neutron nova ...-manuals swift tempest egit scout.rt 80 60 40 AUC 20 Enhanced Baseline 0 cinder glance heat neutron nova ...-manuals tempest egit scout.rt Thursday, March 3, 16
RQ3: Top Most Important Factors From Change Related Metrics: - Type, AGE - LA From Time Metric: - Review Time - Issue Discussion Lag Thursday, March 3, 16
Thursday, March 3, 16
Conclusion • An enhanced defect prediction model was built considering review discussion metrics. • The proposed model was validated on five open source projects. • There is a connection between defect- prone commits and review discussion metrics. • Review discussion metrics complement change-based metrics. Thursday, March 3, 16
Enhanced Defect Prediction Model Metrics Human Process Factor Metrics, like: Metrics, like: - Developer Experience - Number of modified subsystems (NS) - Change is a defect fix? (FIX) (EXP) Product Review Metrics, like: Discussion Metrics, like: - Lines of code in a file before the change Sentiment of Comments (LT) Thursday, March 3, 16
Thursday, March 3, 16
Thursday, March 3, 16
Thursday, March 3, 16
Review Thursday, March 3, 16
Review Thursday, March 3, 16
Review Issue Thursday, March 3, 16
Review Issue Change Thursday, March 3, 16
Review Issue Change Thursday, March 3, 16
Pattern 3 Pattern 2 Issue Time Pattern 1 Review Time Thursday, March 3, 16
Thursday, March 3, 16
Recommend
More recommend
