To make automated regression emails more useful, I propose that several of the benchmarks that we run not send regression alerts by default. This proposal is *not* a proposal to discontinue the benchmarks entirely (yet); the numbers they generate will remain available on the graphserver and are there for your perusal. This proposal simply covers tests that generate a large number of emails that are not useful in identifying problems.
The proposed tests: - Dromaeo (JS); we have better JS benchmarks available. - # of constructors; important to track, not important to bug everybody about until we bring the number down and fix the reporting (https://bugzilla.mozilla.org/show_bug.cgi?id=721387) - Trace Malloc tests (all of them); it's not clear how useful these tests are, given the pageload we run them with and the availability of tools like about:memory. - Tp5 system measurements (%CPU, Content RSS, Main RSS, Private Bytes); These tests tend to not provide any extra information as opposed to other tests. - a11y No Network Row Major MozAfterPaint; this does not seem to be providing value above and beyond what other tests already tell us; the a11y team has made noises in the past about putting together a more a11y-focused benchmark. Again, this is not a proposal to stop running the tests; it is a proposal to halt the flood of ignored email to dev-tree-management, so that people can begin to pay attention to such emails again. I think some of the above tests are unnecessary and don't need to be run, but that is a discussion for a separate time. Comments welcome on the validity and/or history of the above tests. Additional proposals also welcome. I'll let a suitable period of time pass (probably about a week) and then I will start filing bugs on turning email from benchmarks off. -Nathan _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
