(Followup/reply-to: m.tools.bmo / tools-bmo@lists.m.o)
For a little over half a year now, I've been following new bugs filed in
Firefox::Untriaged from a separate bugmail account.
With half a year of experience looking at these incoming bugs, I wanted
to improve the flow for people reporting these bugs. I then proceeded to
analyze some actual data to check my gut feelings about what was going
on. I analyzed bugs reported in Fx::Untriaged over a 2-week time period
(2014-05-14 -- 2014-05-28), 4-6 weeks after they had been filed.**
Goals:
- improve the time/effort it takes for bugs to end up in the right
product/component, the frequency with which this happens, and thereby
the frequency with which bugs get fixed.
- reduce the number of bugs that are not actionable (duplicate,
incomplete, invalid, ...) that get filed
- improve the summary, comment #0 (description) and keywords of new bugs
to make them easier to triage, find & fix.
Here are some concrete changes to our 'guided' new bug funnel that I'd
like to propose (to be clear: none of this affects your bug filing if
you have 'canconfirm' privileges, which is probably all mozilla
employees/staff/contractors and a large number of volunteers who know
their way around bugzilla).
I'll list them below, more or less in decreasing order of importance,
with a one-line summary and a brief explanation as to 'why do this?':
1) More aggressively filter out web developers into their own bug
flow, ask explicitly for a testcase, and add a core::untriaged
component for web bugs filed by them
In the sample data that I analyzed, web developers filed 29% of the bugs
that come into Untriaged. 4-6 weeks later, only 15% of those bugs were
still in Fx::Untriaged, as opposed to 30% of bugs filed by end users
(who file 55% of the bugs).
In other words, it is easier to place web dev bugs where they belong and
get picked up, and these bugs are more likely to have good steps to
reproduce as well as testcases. The result is that the fix rate of these
bugs is higher, and the duplicate/incomplete/invalid/wontfix/worksforme
rate is lower.
Concretely, after the "Which product" question, we should add a page
that prompts people if they are web developers / work on the web. I
think the small box at the top of
https://bugzilla.mozilla.org/enter_bug.cgi?format=guided should be
replaced with one that simply says "I need help", with the remainder
being sorted out *after* picking a product.
If the user is a web dev, most bugs they file are either about Gecko
(59%) or the dev tools (14%). We should prioritize those components for
those bugfilers. We could probably ask something like:
"Are you reporting a bug with:
- HTML, CSS, JS, SVG, or some other web technology or combination of web
technologies?
- Firefox's developer tools
- Firefox's browser shell (for example, an issue with bookmarks, tabbed
browsing or the location bar)
"
and automatically select the right product/component for them (being
Core::Untriaged, Firefox::Developer Tools, and Firefox::Untriaged,
respectively).
Many web devs already include testcases when they file bugs, but
especially for the first two cases, I think it would be helpful if we
had 3 clear fields for them to either copy/paste a testcase, upload a
file, or link to a jsbin/codepen/jsfiddle with the relevant bits.
2) More aggressively send end users who aren't on Nightly to SuMo
For people who report Firefox bugs on release/beta channels, if they
aren't web/addon developers (or people with canconfirm/editbugs), the
typical first 5 comments of a lot of the bugs filed against beta/release
are along the lines of "does this happen on a clean profile?" and "does
this happen in safe mode", potentially followed by "can you produce the
output from about:support here ?" and/or "can you list your crash report
IDs" ? We should either ask these questions / suggest these options on
bugzilla, or strongly point them to support.mozilla.org .
Now that we aren't, over 1/3 of the bugs marked INVALID from end users
were add-on or support-related (the reporting period I picked sadly had
a bunch of advocacy issues related to DRM or Australis, and then there's
test bugs that people file which really should be on landfill.m.o, but
there were still more of these non-issues than advocacy/test ones!). And
that's just the bugs marked invalid - there's probably more issues which
would be better off on SUMO that remain open or were marked WFM or INCO.
Why exclude nightly users? Because these instead tend to file recent
regressions that are Nightly-related, and sending them to SuMo would
mostly be unhelpful to them as well as the people on SuMo.
3) Introduce a keyword for potential graphics issues,
add checkboxes for 'this used to work' (ie regression),
'crash', 'hang', and 'graphical corruption' that toggle the
related keywords
This can help with identifying different types of bugs. Ideally, for
crashes, we should prompt for crash report IDs, for graphics issues we
should ask for the relevant about:support output, for regressions we
should ask if people can find a regression window using mozregression.
In the bugset of 312 bugs that I examined, 22 (7.1%) were
graphics-related, and 24 (7.7%) were crashes or hangs. That's not
particularly much, but equally, both sets of bugs tend to make the
browser hard or very uncomfortable to use - in other words, they're more
important than things like "The focus highlight of the address bar
remains after going into Customize Mode" (which is a semi-randomly
selected other bug in my sample).
4) Provide people who file bugs against Nightly with a filter-able
list of recently filed bugs with the regression keyword, sorted
by number of duplicates and recency (frecency).
We already show a box to type in a summary which automatically
identifies duplicates. Especially when people are on nightly, I think we
should pre-empt that by showing recently filed nightly regression bugs
(if we need a new keyword for this to distinguish them from regressions
in the more general sense, we should add one).
Rationale: we regularly have issues with nightly. The next morning
(according to timezone), nightly testers will duly notice and/or file
issues that they discover. To reduce the number of duplicates and to
make it easier for nightly users to find the existing issues, it would
be nice if we prompted with these immediately, rather than only when
filing the new bug.
For all the resolved bugs, 'duplicate' was by far the most common
resolution, and anything we can do to reduce that will be helpful in
reducing the amount of noise in our bugtracker. This particular group
seems like low-hanging fruit, although admittedly I have no data on how
many of the duplicates were from nightly users.
5) Improve the icons/layout on the first page of
https://bugzilla.mozilla.org/enter_bug.cgi?format=guided
Because 4 identical Firefox icons don't help anyone. We should have
separate icons for Android and FxOS, and probably remove Fx for Metro
from the list (?). We could also do with using columns so it's not such
a long page using only 1/3 of the horizontal size of my screen. In other
words, someone with a design background should look at this page and
improve it. :-)
(admittedly this is the most "gut feeling" and least data-driven point
of all of the above)
----
Comments, questions, concerns? If nobody objects strongly, I'll be
filing these as enhancement requests for bmo in about a week, with exact
details (which buttons go where, designs, wording, etc.) to be fleshed
out in the bug, with design/language input from actual
designers/wordsmiths rather than just me. :-)
Gijs
** I have data in a messy and potentially unclear google spreadsheet.
You can look at it here:
https://docs.google.com/spreadsheets/d/1b5_3xUaT9Dr8NznBjJzTIphv9-R-CWjJHVFsW511NAs
The bug states ("is this still in untriaged" and resolution) were
collected in one go using a small script, run at the end of last week -
I have purposefully not updated them so that the data is static.
You can make your own clone to filter and pivot table at will using File
> Make a Copy...
While there are undoubtedly individual calls I've made in categorizing
things that can be questioned ("Is this really a chrome/content/whatever
issue"), I tried to move quickly in order to deal with the entire
dataset in a reasonable time, and I am fairly confident about the
overall trends that showed up.
