A Conceptual Replication of Continuous Integration Pain Points
A Conceptual Replication of Continuous Integration Pain Points david Michael christian bogdan .. widder hilton kastner vasilescu
What Will I talk about? � 2
What Will I talk about? 1. What is Continuous integration? Why do people use it? � 2
What Will I talk about? 1. What is Continuous integration? Why do people use it? 2. Why do a replication? � 2
What Will I talk about? 1. What is Continuous integration? Why do people use it? 2. Why do a replication? 3. Our Mixed Method � 2
What Will I talk about? 1. What is Continuous integration? Why do people use it? 2. Why do a replication? 3. Our Mixed Method 4. Highlights of Findings: What replicates + what doesn’t? � 2
What is continuous integration? � 3
What is continuous integration? � 3
What is continuous integration? � 3
What is continuous integration? � 3
What is continuous integration? � 3
What is continuous integration? � 3
What is continuous integration? � 3
What is continuous integration? � 3
CI is a Best practice � 4
CI is a Best practice What is 6 Reasons Why we CI and to Use CI need CI why use it? what is CI 7 Reasons Why CI is and why do you should important you need it? use CI � 4
Lots of research Showing CI Benefits � 5
Lots of research Showing CI Benefits Helps Higher pull catch request bugs Throughput faster release Encourages twice as people to often write tests Fewer pull requests rejected � 5
Also, lots of research showing CI pain points � 6
Also, lots of research showing CI pain points Hard to too many options configure long unsupported features builds insecure � 6
We believe it is time to review and replicate CI pain points � 7
Why review and replicate CI pain points? � 8
Why review and replicate CI pain points? - evaluate & synthesize past research � 8
Why review and replicate CI pain points? - evaluate & synthesize past research - provide replicated guidance to practitioners � 8
Why review and replicate CI pain points? - evaluate & synthesize past research - provide replicated guidance to practitioners - focus future research on areas of uncertainty � 8
What’s a conceptual replication? current Switchers & CI users Abandoners � 9
What’s a conceptual replication? current Switchers & CI users Abandoners ? � 9
why study leavers instead of current users? � 10
why study leavers instead of current users? � 10
TravisCI Leavers Current CI Users Industry & OSS OSS ? Survey & lit Triangu interviews lation review Pain Pain points logistic Replicated Points regressions (or not) � 11
TravisCI Leavers Current CI Users Industry & OSS OSS ? Survey & lit Triangu interviews lation review Pain Pain points logistic Replicated Points regressions (or not) � 11
TravisCI Leavers Current CI Users Industry & OSS OSS ? Survey & lit Triangu interviews lation review Pain Pain points logistic Replicated Points regressions (or not) � 11
TravisCI Leavers Current CI Users Industry & OSS OSS ? Survey & lit Triangu interviews lation review Pain Pain points logistic Replicated Points regressions (or not) � 11
TravisCI Leavers Current CI Users Industry & OSS OSS ? Survey & lit Triangu interviews lation review Pain Pain points logistic Replicated Points regressions (or not) � 11
TravisCI Leavers Current CI Users Industry & OSS OSS ? Survey & lit Triangu interviews lation review Pain Pain points logistic Replicated Points regressions (or not) � 11
lit review 37 papers 35 Pain Points � 12
? 132 Survey responses 12 interviews “Why did your project stop using Travis CI, and what has the CI situation been like since then?” � 13
� 14
? 132 Survey responses + 12 interviews � 15
132 Survey responses + 12 interviews � 15
Why logistic regression? People who leave CI vs Those who don’t � 16
Abandonment vs Switching CI { .. } � 17
Abandonment vs Switching CI { .. } � 17
Abandonment vs Switching CI { .. } 2 logistic regressions, 6,239 Repos � 18
Abandonment vs Switching CI { .. } 2 logistic regressions, 6,239 Repos � 18
? Triangulated Results � 19
C # Unsupported technologies � 20
C # Unsupported technologies � 21
C # Unsupported technologies “I switched to Wercker, a container based CI pipeline, which means it can execute any scripts with much more flexibility.” (P89) � 21
C # Unsupported technologies “I switched to Wercker, a container based CI pipeline, which means it can execute any scripts with much more flexibility.” (P89) needing docker: 9.5x increase in switching, 5.25x increase in abandoning � 21
C # Unsupported technologies “I switched to Wercker, a container based CI pipeline, which means it can execute any scripts with much more flexibility.” (P89) needing docker: 9.5x increase in switching, 5.25x increase in abandoning Lacking Language support: 1.5x increase in switching and abandoning � 21
C # Unsupported technologies � 22
C # Unsupported technologies Needing docker affects new TravisCI users only � 22
C # Unsupported technologies Needing docker affects new TravisCI users only when can people hack through technology support challenges, and when are the not able to? � 22
Lack of Tests � 23
Lack of Tests � 24
Lack of Tests “The goal [of adopting Travis] was to ‘force’ myself to add some real tests (to have green Travis badge again!) but this failed so far :)” (P111) � 24
Lack of Tests “The goal [of adopting Travis] was to ‘force’ myself to add some real tests (to have green Travis badge again!) but this failed so far :)” (P111) 1% increase in tests = 16% lower chance of abandoning (no effect on switching) � 24
CI consistency � 25
CI consistency � 26
CI consistency “I had some open source projects running in TravisCI and some in CircleCI. I just wanted to consolidate the project to one place and I’m sorry to say that at that time TravisCI lost the battle.” (P57) � 26
CI consistency “I had some open source projects running in TravisCI and some in CircleCI. I just wanted to consolidate the project to one place and I’m sorry to say that at that time TravisCI lost the battle.” (P57) Exposure to leavers: ~ 1.5x increase in chances of switching & abandoning � 26
CI consistency � 27
CI consistency people’s past experience is very important in determining behavior on future projects! � 27
CI consistency people’s past experience is very important in determining behavior on future projects! but Not observed in past literature, so needs replication. � 27
Single method or conflicting results ? ? � 28
long build times � 29
long build times � 30
long build times “We stopped using Travis CI because it was too slow for us.” (P125) � 30
long build times “We stopped using Travis CI because it was too slow for us.” (P125) longer build times associated with a decreased chance of switching and abandoning. � 30
long build times “We stopped using Travis CI because it was too slow for us.” (P125) longer build times associated with a decreased chance of switching and abandoning. Presence of very long builds associated with increased risk of abandoning. � 30
long build times � 31
long build times So, When do long builds stop providing value, and start becoming annoying? � 31
Full results in the paper � 32
? lit Surveys, Triangulation review Interviews Pain 2 logistic Points regressions davidthewid
we were unable to cleanly ? replicate some pain points, they need further study to validate their existence lit Surveys, Triangulation review Interviews Pain 2 logistic Points regressions davidthewid
we were unable to cleanly ? replicate some pain points, they need further study to validate their existence lit Surveys, Triangulation review Interviews we were able to cleanly replicate some pain points, now we can focus on solutions Pain 2 logistic Points regressions davidthewid
Recommend
More recommend
