Determinants of Pull Request Evaluation Latency on GitHub Yue Yu, - PowerPoint PPT Presentation

MSR 2015, Florence, Italy “Wait For It” Determinants of Pull Request Evaluation Latency on GitHub Yue Yu, Huaimin Wang, Vladimir Filkov, Prem Devanbu, Bogdan Vasilescu

GitHub workflow Personal Main branches / branch forks

Lots of pull requests to review ( rails / rails % Watch ⋆ Star ' Fork 1,887 26,093 10,339 . New pull request Issues Pull requests Labels Milestones Filters is:pr is:open New pull request ) + 467 Open / 12,551 Closed * Author Labels Milestones Assignee Sort + + Move Integer#positive? and Integer#negative? query methods to Numeric 0 2 / #20143 opened an hour ago by meinac , + Deprecate `assert_template`. / 0 8 #20138 opened 9 hours ago by tgxworld - + Add Enumerable#map_with to ActiveSupport 0 0 / #20134 opened 13 hours ago by mlarraz + Allow creating a save callback for same name with parent association 0 2 / #20127 opened 23 hours ago by meinac + ActiveSupport::HashWithIndifferentAccess select and reject should return enumerator if called without block 0 0 / #20125 opened a day ago by imanel + Don't ignore false values for `include_blank` passed to `Tags::Base#select_content_tag` 0 9 / #20124 opened a day ago by greysteil + Fix for irregular inflection inconsistency / 0 0 #20123 opened a day ago by yoongkang + Add openssl_verify_mode and sync other smtp_settings with API docs 0 0 / #20117 opened 2 days ago by jfine

Pull Request evaluation time 3000 Frequency 2000 1000 0 1 hour 1 mo 1 min 10 mins 1 day 1 week half year

Continuous integration ( rails / rails % Watch ⋆ Star ' Fork 1,887 26,093 10,339 . New pull request Issues Pull requests Labels Milestones Filters is:pr is:open New pull request ) + 467 Open / 12,551 Closed * Author Labels Milestones Assignee Sort + + Move Integer#positive? and Integer#negative? query methods to Numeric 0 2 / #20143 opened an hour ago by meinac , Automated testing & + Deprecate `assert_template`. / 0 8 #20138 opened 9 hours ago by tgxworld continuous integration - + Add Enumerable#map_with to ActiveSupport 0 0 / #20134 opened 13 hours ago by mlarraz + Allow creating a save callback for same name with parent association 0 2 / #20127 opened 23 hours ago by meinac + ActiveSupport::HashWithIndifferentAccess select and reject should return enumerator if called without block 0 0 / #20125 opened a day ago by imanel + Don't ignore false values for `include_blank` passed to `Tags::Base#select_content_tag` 0 9 / #20124 opened a day ago by greysteil + Fix for irregular inflection inconsistency / 0 0 #20123 opened a day ago by yoongkang + Add openssl_verify_mode and sync other smtp_settings with API docs 0 0 / #20117 opened 2 days ago by jfine

Median Pull Request times Pull Pull request request closed received 11.2 hours

Median Pull Request times Pull Pull First CI request request human response closed received response 16 mins 39 mins 11.2 hours

What influences PR evaluation time? 3000 ? Frequency 2000 1000 ? 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year

Modeling the Pull Request evaluation time Size Pull-Request • n_additions • n_commits 3000 Review Frequency • n_comments 2000 1000 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year Experience & Social Connections • merge_rate • connection_strength • n_followers [Gousios et al, ICSE’14, ICSE’15] [Tsay et al, ICSE’14, FSE’14]

Modeling the Pull Request evaluation time M1: Previously- Size identified factors Pull-Request • n_additions • n_commits ✓ R 2 = 36.2% 3000 Review Frequency • n_comments 2000 1000 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year Experience & Social Connections • merge_rate • connection_strength • n_followers [Gousios et al, ICSE’14, ICSE’15] [Tsay et al, ICSE’14, FSE’14]

Modeling the Pull Request evaluation time Title & M2: M1 + process-related factors description + continuous integration • n_tokens 3000 Frequency 2000 1000 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year

Modeling the Pull Request evaluation time Title & M2: M1 + process-related factors description + continuous integration • n_tokens 3000 Frequency 2000 1000 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year Management • workload • availability

Modeling the Pull Request evaluation time Title & M2: M1 + process-related factors description + continuous integration • n_tokens Priority • time_to_first _response Continuous 3000 Frequency Integration 2000 • response time 1000 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year Management • workload • availability

Modeling the Pull Request evaluation time Title & M2: M1 + process-related factors description + continuous integration • n_tokens ✓ R 2 = 58.7% Priority • time_to_first _response Continuous 3000 Frequency Integration 2000 • response time 1000 0 1 mo 1 hour 1 min 10 mins 1 day 1 week half year Management • workload • availability Social tagging • @mention • #issue

Determinants of Pull Request Evaluation Latency on GitHub ? 3000 Frequency 2000 ? 1000 0 1 mo 1 min 10 mins 1 hour 1 day 1 week half year

Determinants of Pull Request Evaluation Latency on GitHub ? 3000 Frequency 2000 ? 1000 0 1 mo 1 min 10 mins 1 hour 1 day 1 week half year Pull Pull First CI request request human response closed received response 16 mins 39 mins 11.2 hours

Determinants of Pull Request Evaluation Latency on GitHub ? 3000 Frequency 2000 ? 1000 0 1 mo 1 min 10 mins 1 hour 1 day 1 week half year Pull Pull First CI request request human response closed received response Yue Yu, Huaimin Wang, Vladimir Filkov, Prem Devanbu, Bogdan Vasilescu 16 mins 39 mins 11.2 hours