This discussion is a follow-up discussion to some emails sent privately by
accident.
If you have not followed, I will quote David Bryant:
> Improving release quality is one of the three fundamental goals Platform
> Engineering committed to this year. To this end, lmandel built a Bugzilla
> dashboard that allows us to track regressions found in any given release
> cycle. This dashboard […] can
> also be found at: http://mozilla.github.io/releasehealth/
To David's email, I answered the following:
----------
tl;dr: If we want to improve the quality of our products we should
split Gecko in standalone programs which are fuzzing-friendly.
One thing which strikes me, is the ratio of regressions per component
that we have for each versions, and more over who are the persons
opening these bugs:
- Release:
https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox44&f2=OP&f3=cf_status_firefox43&f4=cf_status_firefox43&f5=cf_status_firefox43&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898533&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on=
- Beta:
https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox45&f2=OP&f3=cf_status_firefox44&f4=cf_status_firefox44&f5=cf_status_firefox44&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898534&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on=
- Aurora:
https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox46&f2=OP&f3=cf_status_firefox45&f4=cf_status_firefox45&f5=cf_status_firefox45&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898536&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on=
- Nightly:
https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox47&f2=OP&f3=cf_status_firefox46&f4=cf_status_firefox46&f5=cf_status_firefox46&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898528&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on=
To be more precise:
- The small number of regression we have in the JS engine on the
release channel, versus the Extremely Huge number of regressions we
have on nightly.
- And the fact that (almost) all the bugs opened against the JS
engine are opened by our fuzzing team.
What I want to remark is the fact that our automated fuzzing is better
at finding recently introduced regressions. And as far as I know,
Alice is not a bot.
From what I know, the reason fuzzing team is so efficient on the JS
engine is because we have a *standalone* JS shell.
The *standalone* JS shell is also the reason why our build time is
below 2 minutes as opposed to 18 minutes.
So, I think that if we want to improve our quality we should focus on
making fuzzing-friendly standalone programs for the different
components of the platform.
Thus reducing, the compilation time, reducing the test suite time, and
improving the ability of the fuzzing team to find recently added
regressions.
Maybe I am wrong, in which case the other alternative might be to
staff the JS Team to get rid of all these nightly issues before they
ride the train to release.
----------
To which I got the following replies:
On Wed, Mar 9, 2016 at 3:05 PM, Kyle Huey wrote:
> The ratio of engineers to code in the js engine is so much higher than
> the rest of the product that I'm not sure this is a sensible comparison.
> The js engine also doesn't depend on things like 3rd party gfx drivers ...
On Wed, Mar 9, 2016 at 3:05 PM, Olli Pettay wrote:
> Fuzzing captures only a fraction of issues.
On Wed, Mar 9, 2016 at 3:42 PM, Chris Hofmann wrote:
> On Wed, Mar 9, 2016 at 7:05 AM, Kyle Huey wrote:
>>
>> The ratio of engineers to code in the js engine is so much higher than the
>> rest of the product that I'm not sure this is a sensible comparison. The js
>> engine also doesn't depend on things like 3rd party gfx drivers ...
>
> This is probably not the only step that we need to take to substantially
> improve quality so setting up a place to have those discussions is good. It
> really is worth some time and effort to brainstorm about all the things we
> might do to raise the bar, poke some holes in those ideas, then decide on
> and push forward on a few more in the next few quarters.
On Wed, Mar 9, 2016 at 5:23 PM, Al Billings wrote:
> On 3/9/16 6:58 AM, Nicolas B. Pierron wrote:
>> So, I think that if we want to improve our quality we should focus on
>> making fuzzing-friendly standalone programs for the different
>> components of the platform.
>> Thus reducing, the compilation time, reducing the test suite time, and
>> improving the ability of the fuzzing team to find recently added
>> regressions.
> The fuzzing team has asked for this from different teams for various
> components, with different degrees of support and resistance, depending
> on the component and team.
>
> It is work to do this but it changes the fuzzing efficiency and ability
> to find focused issues when we have this level of support, such as we
> have with the JS shell. It allows the team to fuzz directly on a
> component and not have to find a way to reach it through Firefox and all
> the additional noise/assertions/etc that it churns up.
--
Nicolas B. Pierron
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
