The Impact of Continuous Integration on Other Software Development - PowerPoint PPT Presentation

The Impact of Continuous Integration on Other Software Development Practices: A Large-Scale Empirical Study Yangyang Alexander Yuming Vladimir Bogdan Zhao Serebrenik Zhou Filkov Vasilescu DECAL STRUDEL Nanjing U TU Eindhoven Nanjing U at UC Davis at CMU @aserebrenik @vfilkov @b_vasilescu

Interventions are common in software engineering • SVN —> git • push —> pull request • ? —> continuous integration • … How to measure effects using trace data?

Evaluating the effects of an intervention: before vs. after change in slope

Evaluating the effects of an intervention: before vs. after change in slope t-test no di ff erence

Evaluating the effects of an intervention: before vs. after change in level

Evaluating the effects of an intervention: before vs. after change in level t-test no di ff erence

Today Methodology to empirically study the e ff ects of an intervention (continuous integration)

Interrupted time series Multiple regression w/ controls for confounds change in level slope slope before after

change slope in level before slope after time : 1 2 3 … … … 100 101 102 … … … 200 time after intervention : 0 0 0 … … … 0 1 2 … … … 100 intervention : F F F … … … T T T … … … T

ɣ β β + δ time : 1 2 3 … … … 100 101 102 … … … 200 time after intervention : 0 0 0 … … … 0 1 2 … … … 100 intervention : F F F … … … T T T … … … T y i = α + β · time i + ɣ · intervention i + δ · time_after_intervention i + ε i

lm in R ɣ β β + δ y i = β · time i + ɣ · intervention i + δ · time_after_intervention i + ε i Dependent variable: y • β ~ 1 0.991 *** time -48.678 *** intervention • ɣ ~ -50 -1.500 *** time_after_intervention • β + δ ~ -0.5 Constant 1.007 Observations 200 R 2 0.967 Adjusted R 2 0.967 Residual Std. Error 4.844 (df = 196) 1,924.910 *** (df = 3; 196) F Statistic * p<0.1; ** p<0.05; *** p<0.01 Note:

E ff ects of adopting Travis CI

Why CI? Lots of folklore, e.g., Martin Fowler: • Everyone Commits To the Mainline Every Day • Fix Broken Builds Immediately • Keep the Build Fast • … https://martinfowler.com/articles/originalContinuousIntegration.html

Adoption of Travis CI Travis CI adoption Unstable period excluded (first .travis.yml commit) time … … -12 -1 +1 +12 -375 +45 +375 -45 -15 +15 days days days days days days Starting sample: 165,549 GitHub projects using Travis 24 active periods x 7 programming languages

More frequent commits RQs Impact on Smaller code automated changes testing? More issues and Quick pull requests pull requests closed resolution

Churn Churn in Churn in non-merge commits merge commits Intercept ( α ) 1,336 *** -1,297 ** log(TotalCommits) 0,529 ** 1,113 ** AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 **

Churn Churn in Churn in non-merge commits merge commits Intercept ( α ) 1,336 *** -1,297 ** log(TotalCommits) 0,529 ** 1,113 ** Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 **

Churn Churn in Churn in non-merge commits merge commits Intercept ( α ) 1,336 *** -1,297 ** log(TotalCommits) 0,529 ** 1,113 ** Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI

Churn Churn in non-merge commits Churn in Churn in ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● non-merge commits ● merge commits ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5000 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Intercept ( α ) 1,336 *** -1,297 ** ● 500 200 log(TotalCommits) 0,529 ** 1,113 ** 100 50 20 10 AgeAtTravis -0,003 * -0,005 ** 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 ● ● ● ● log(NumAuthors) − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 -0,233 ** − 1 1 2 3 4 5 6 7 -0,522 ** 8 9 10 11 12 Month index w.r.t. Travis CI adoption time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI

Churn Churn in Churn in non-merge commits merge commits Intercept ( α ) 1,336 *** -1,297 ** log(TotalCommits) 0,529 ** 1,113 ** Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI Discontinuity in merge com. : preparation for transition, clean-up

Churn Churn in Churn in non-merge commits merge commits Intercept ( α ) 1,336 *** -1,297 ** log(TotalCommits) 0,529 ** 1,113 ** Control variables AgeAtTravis -0,003 * -0,005 ** log(NumAuthors) -0,233 ** -0,522 ** time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI Discontinuity in merge com. : preparation for transition, clean-up Decrease in churn in merge commits is amplified by Travis CI

Churn Churn in merge commits Churn in Churn in ● ● Mean merge churn (LOC) non-merge commits merge commits 5000 Intercept ( α ) 1,336 *** -1,297 ** 500 200 log(TotalCommits) 0,529 ** 1,113 ** 100 50 20 10 Control variables AgeAtTravis -0,003 * -0,005 ** 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 ● ● ● ● ● ● ● ● ● ● ● ● log(NumAuthors) -0,233 ** -0,522 ** − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 1 1 2 3 4 5 6 7 8 9 10 11 12 Month index w.r.t. Travis CI adoption time ( β ) -0,007 -0,012 * n.s. interventionTrue ( ɣ ) 0,071 0,220 ** time_after_intervention ( δ ) -0,009 -0,022 ** Churn in non-merge commits is not a ff ected by time or Travis CI Discontinuity in merge com. : preparation for transition, clean-up Decrease in churn in merge commits is amplified by Travis CI

Triangulation: user survey 55 introduced Travis to their projects

R25 : “contributors couldn’t be trusted to run test suite on their Discontinuity in own” merge commits : preparation for R38 : Travis as “a part of transition, clean-up automated package/release e ff ort” Decrease in churn R4 : “commits became smaller and in merge commits more frequent, to check the build; is amplified by pull requests became easier to check” Travis CI