On 12-08-31 6:01 AM, Justin Lebar wrote:
I'm not saying it should be OK to regress our performance tests, as a rule. But I think we need to acknowledge that hunting regressions can be time-consuming, and that a policy requiring that all regressions be understood may hamstring our ability to get anything else done. There's a trade-off here that we seem to be ignoring.
There is definitely a trade-off here, and at least for the past year (and maybe for the past two years) we have in practice been weighing on the side of the difficulty of tracking down performance regression to the point that we've been ignoring them (except for perhaps a few people.)
It is a mistake to take Rafael's example and extend it to the average regression that we measure on Talos. It's true that sometimes those things happen, and in practice we cannot deal with them all, because we don't have an army of Rafaels. But it bothers me when people take an example of a very difficult to understand regression encountered by a person who bravely dwells with low-level compiler code generation stuff and extend it to come up with a policy covering all regressions. Please, let's not do that.
And let's remember the other side of the trade-off too. A lot of blood and tears has gone into shaving off milliseconds from our startup time. Taking a ~5% hit on startup time within a 6-week cycle effectively means that we have undone man-months of optimizations which have happened to the startup time. So it's not like letting these regressions in beneath our noses is going to make us all more productive.
There are extremely non-stable Talos tests, and relatively stable ones. Let's focus on the relatively stable ones. There are extremely hard to diagnose performance regressions, and extremely easy ones (i.e., let's not wait on this lock, do this I/O, run this exponential algorithm, load tons of XUL/XBL when a window opens, etc.) We have many great tools for the job, so not all regressions need to be treated the same.
Cheers, Ehsan _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
