user feedback in software evolution Andrew J. Ko with Parmit Chilana based on Ko, A.J. and Chilana, P. (2010). How power users help and hinder open bug reporting. ACM Conference on Human Factors in Computing (CHI). Chilana, P., Ko, A.J., & Wobbrock, J.O. (2010). Understanding expressions of unwanted behaviors in open bug reporting. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).
finding and fixing everyday software defects helping developers diagnose failures helping users diagnose failures detecting usability defects automatically where do defects come from? what makes debugging difficult? what do software teams need to fix defects? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 2
finding and fixing everyday software defects software team user community Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 3
finding and fixing everyday software defects software team user community Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 4
finding and fixing everyday software defects users have more opportunities than ever to influence what defects are found and fixed open development processes perpetual beta status open bug reporting web-based user feedback crowdsourced help in support forums software team 5 Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010
users have more opportunities than ever to influence what defects are found and fixed who contributes such feedback? what do they write about? what feedback is addressed? why is some feedback ignored? how can teams get better feedback? how can teams use feedback better? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 6
research from the past two years who contributes such feedback? what do they write about? what feedback is addressed? why is some feedback ignored? research in progress how can teams get better feedback? how can teams use feedback better? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 7
open bug reporting in the Mozilla project } research from the past two years who contributes such feedback? what do they write about? what feedback is addressed? why is some feedback ignored? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 8
outline a brief introduction to bug reporting 0 who contributes such feedback? 1 what do they write about? 2 what feedback is addressed? 3 why is some feedback ignored? 4 Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 9
a user’s expectations are violated “ I wanted this to be a new tab, not a new window! Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 10
user visits bugzilla.mozilla.org Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 11
users are encouraged to look for an existing bug report that describes their issue Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 12
if they don’t find one, they submit a new bug report ... Options for where to open URLs from other applications (reuse tab, new window) This is a dupe of http://bugzilla.mozilla.org/show_bug.cgi?id=75138 (for Mozilla). There’s a great discussion of it over there. However, that RFE has been listed for over a year now and is still marked for “future”. Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 13
Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 14
reporter who created the report date resolution FIXED when the DUPLICATE the reason report was INCOMPLETE the report created WONTFIX was closed WORKSFORME assignee INVALID the developer assigned to resolve this report Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 15
screenshots, patches, test cases, mockups, etc. attachment attacher date Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 16
comments and commenters anyone interested in this bug can add information to the report Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 17
a typical report lifespan original report comment comment screenshot comment comment patch 1.0 comment patch 2.0 code review patch 3.0 closed Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 18
who contributes 1 bug reports? what do they 2 write about? what are the outcomes 3 of their reports? why do their reports have 4 these outcomes? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 19
who contributes 1 bug reports? obtained all 496,766 reports except for private security patches 15 years of reports, including Netscape years 152,877 unique e-mail addresses 64% addresses only authored, attached to, or commented on 1 report who’s e-mail addresses were these? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 20
who’s e-mail addresses were these? CORE developers, drivers, super reviewers, 1% module owners, peers Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 21
who was behind these 152,877 addresses? CORE developers, drivers, super reviewers, 1% module owners, peers 1% ACTIVE developers assigned bug reports Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 22
who was behind these 152,877 addresses? CORE developers, drivers, super reviewers, 1% module owners, peers 1% ACTIVE developers assigned bug reports 80% REPORTERS reported and commented on bug reports responsible for 54% of reports Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 23
who was behind these 152,877 addresses? CORE developers, drivers, super reviewers, 1% module owners, peers 1% ACTIVE developers assigned bug reports 80% 18% REPORTERS USERS reported and only commented on commented bug reports on bug reports responsible for 54% of reports Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 24
# of active contributors by type , per 6 month period reporters and users fluctuate spike before a release (#$!!!" Firefox(2 (!$!!!" '$!!!" 9/829?/9B" &$!!!" Firefox(3 5B/9B" Firefox(1 %$!!!" Firefox(0.1 *>C3/" #$!!!" !" >29/" )*+",-'" ./0",--" )*+",!(" ./0",!#" 489",!%" =>?",!<" 489",!A" =>?",!'" 123",--" 123",!#" :5@",!&" 456",!!" 456",!7" :*;",!<" :*;",!'" Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 25
who contributes 1 bug reports? mostly non-developer, one-time contributors who were active pre-release ( REPORTERS ) what do they 2 write about? what are the outcomes 3 of their reports? why do their reports have 4 these outcomes? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 26
who contributes 1 bug reports? mostly non-developer, one-time contributors who were active pre-release ( REPORTERS ) what do they 2 write about? what are the outcomes 3 of their reports? why do their reports have 4 these outcomes? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 27
what do they 2 write about? selected a sample of 50 REPORTER reports inductively analyzed titles and descriptions Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 28
what do they 2 write about? the most salient feature was the source of expectation what group or person believed the behavior was defective? Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 29
what do they 2 write about? prior runtime logic iterated through 3 specifications individual samples of 100 reports, converging standards community towards a set of 7 sources genre Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 30
what do they 2 write about? developer user intents expectations prior runtime logic individual specifications community standards genre Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 31
runtime violations errors, warnings, assertion violations, crashes, hangs, and language-defined invalid states “…scary deadlock assertions exiting mozilla after referencing nsInstallTrigger…” language Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 32
specification violations an agreed upon functional requirement among the application developers “There's an incorrectly placed PR_MAX in the code for pref width distribution of colspanning cells.” application language Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 33
standards violations industry-wide functional specifications, reaching beyond the application’s developer community “'codebase' attribute of the HTML 4.0 OBJECT element is not supported…” industry application language Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 34
violations of a reporter expectations a reporter’s personal perspective about what the system should do “Every time I Sort By Name by Bookmarks Firefox sorts and closes my Bookmark menu... Why does it do this??” app user Andrew J. Ko – IBM Research Symposium, Sept. 30, 2010 35
Recommend
More recommend
