the impact of continuous integration on other software
play

The Impact of Continuous Integration on Other Software Development - PowerPoint PPT Presentation

The Impact of Continuous Integration on Other Software Development Practices: A Large-Scale Empirical Study Yangyang Alexander Yuming Vladimir Bogdan Zhao Serebrenik Zhou Filkov Vasilescu DECAL STRUDEL Nanjing U TU Eindhoven Nanjing


  1. The Impact of Continuous Integration on Other Software Development Practices: A Large-Scale Empirical Study Yangyang Alexander Yuming Vladimir Bogdan Zhao Serebrenik Zhou Filkov Vasilescu DECAL STRUDEL Nanjing U TU Eindhoven Nanjing U at UC Davis at CMU @aserebrenik @vfilkov @b_vasilescu

  2. Happy Halloween! https://imgur.com/0TBJ9OW

  3. Interventions are common in software engineering • SVN —> git • push —> pull request • ? —> continuous integration • …

  4. Interventions are common in software engineering • SVN —> git • push —> pull request • ? —> continuous integration • … How to measure effects using trace data?

  5. Evaluating the effects of an intervention: before vs. after change in slope

  6. Evaluating the effects of an intervention: before vs. after change in slope t-test no di ff erence

  7. Evaluating the effects of an intervention: before vs. after change in level

  8. Evaluating the effects of an intervention: before vs. after change in level t-test no di ff erence

  9. Today Methodology to empirically study the e ff ects of an intervention (continuous integration)

  10. Interrupted time series change in level slope slope before after

  11. Interrupted time series Multiple regression w/ controls for confounds change in level slope slope before after

  12. change slope in level before slope after

  13. change slope in level before slope after time : 1 2 3 … … … 100 101 102 … … … 200

  14. change slope in level before slope after time : 1 2 3 … … … 100 101 102 … … … 200 time after intervention : 0 0 0 … … … 0 1 2 … … … 100

  15. change slope in level before slope after time : 1 2 3 … … … 100 101 102 … … … 200 time after intervention : 0 0 0 … … … 0 1 2 … … … 100 intervention : F F F … … … T T T … … … T

  16. ɣ β β + δ time : 1 2 3 … … … 100 101 102 … … … 200 time after intervention : 0 0 0 … … … 0 1 2 … … … 100 intervention : F F F … … … T T T … … … T y i = α + β · time i + ɣ · intervention i + δ · time_after_intervention i + ε i

  17. lm in R ɣ β β + δ y i = β · time i + ɣ · intervention i + δ · time_after_intervention i + ε i Dependent variable: y • β ~ 1 0.991 *** time -48.678 *** intervention • ɣ ~ -50 -1.500 *** time_after_intervention • β + δ ~ -0.5 Constant 1.007 Observations 200 R 2 0.967 Adjusted R 2 0.967 Residual Std. Error 4.844 (df = 196) 1,924.910 *** (df = 3; 196) F Statistic * p<0.1; ** p<0.05; *** p<0.01 Note:

  18. E ff ects of adopting Travis CI

  19. Why CI? Lots of folklore, e.g., Martin Fowler: • Everyone Commits To the Mainline Every Day • Fix Broken Builds Immediately • Keep the Build Fast • … https://martinfowler.com/articles/originalContinuousIntegration.html

  20. Adoption of Travis CI Travis CI adoption Unstable period excluded (first .travis.yml commit) time … … -12 -1 +1 +12 -375 +45 +375 -45 -15 +15 days days days days days days

  21. Adoption of Travis CI Travis CI adoption Unstable period excluded (first .travis.yml commit) time … … -12 -1 +1 +12 -375 +45 +375 -45 -15 +15 days days days days days days Starting sample: 165,549 GitHub projects using Travis

  22. Adoption of Travis CI Travis CI adoption Unstable period excluded (first .travis.yml commit) time … … -12 -1 +1 +12 -375 +45 +375 -45 -15 +15 days days days days days days Starting sample: 165,549 GitHub projects using Travis 24 active periods x 7 programming languages

  23. More frequent commits RQs Impact on Smaller code automated changes testing? More issues and Quick pull requests pull requests closed resolution

  24. Churn Churn in Churn in non-merge commits merge commits 1,336 *** -1,297 ** Intercept ( α ) 0,529 ** 1,113 ** log(TotalCommits) AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 **

  25. Churn Churn in Churn in non-merge commits merge commits 1,336 *** -1,297 ** Intercept ( α ) 0,529 ** 1,113 ** log(TotalCommits) Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 **

  26. Churn Churn in Churn in non-merge commits merge commits 1,336 *** -1,297 ** Intercept ( α ) 0,529 ** 1,113 ** log(TotalCommits) Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI

  27. Churn Churn in non-merge commits Churn in Churn in ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● non-merge commits ● merge commits ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5000 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1,336 *** ● ● ● ● -1,297 ** Intercept ( α ) ● 500 200 0,529 ** 1,113 ** log(TotalCommits) 100 50 20 10 AgeAtTravis -0,003 * -0,005 ** 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 ● ● ● ● log(NumAuthors) -0,233 ** -0,522 ** − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 1 1 2 3 4 5 6 7 8 9 10 11 12 Month index w.r.t. Travis CI adoption time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI

  28. Churn Churn in Churn in non-merge commits merge commits 1,336 *** -1,297 ** Intercept ( α ) 0,529 ** 1,113 ** log(TotalCommits) Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI Discontinuity in merge com. : preparation for transition, clean-up

  29. Churn Churn in Churn in non-merge commits merge commits 1,336 *** -1,297 ** Intercept ( α ) 0,529 ** 1,113 ** log(TotalCommits) Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI Discontinuity in merge com. : preparation for transition, clean-up Decrease in churn in merge commits is amplified by Travis CI

  30. Churn Churn in merge commits Churn in Churn in ● ● Mean merge churn (LOC) non-merge commits merge commits 5000 1,336 *** -1,297 ** Intercept ( α ) 500 200 0,529 ** 1,113 ** log(TotalCommits) 100 50 20 10 Control variables AgeAtTravis -0,003 * -0,005 ** 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ● ● ● ● log(NumAuthors) -0,233 ** -0,522 ** − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 1 1 2 3 4 5 6 7 8 9 10 11 12 Month index w.r.t. Travis CI adoption time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI Discontinuity in merge com. : preparation for transition, clean-up Decrease in churn in merge commits is amplified by Travis CI

Recommend


More recommend