On 09/09/2016 13:29, zbranie...@mozilla.com wrote:
The test at the moment is both noisy, and has a steady regression with each run
which makes it hard to feel confident about it and also makes it impossible to
just increase the number of runs in order to increase significance and
reliability of the result.
Basically, tpaint does 20 runs and currently each run is slower than the
previous one. I believe it may affect your experience of using the test.
Note that the number of runs is configurable, and that you can run these
tests locally with ./mach (more is usually better for local testing).
That will hopefully help your investigation if you're struggling with
perf. You can also automatically create SPS profiles which you can then
analyze to see why one set of runs is systematically slower than another
set.
As for the apparently monotonically increasing runtime, I just landed a
patch on inbound to "fix" this. The only issue was that we *sort* the
results of the window opening, so there is no clear trend where
subsequent window opens are slower - the sorting just created the
illusion of a pattern where none existed.
What's more, the test doesn't only test opening a new window, but it also loads
a document into it (via data: protocol) and measures first paint after the
document is loaded.
This is documented, as far I understand it. If it isn't, it should be.
We found out that the regression is visible in tpaint (window.open + document
load), but not in MozAfterPaint of the browser chrome UI, so your patch may
have an impact on the tpaint *without* affecting the new window performance.
I haven't investigated this in detail, but the following links may be
helpful in understanding what's going on here:
https://groups.google.com/forum/#!topic/mozilla.dev.platform/pCLwWdYc_GY
https://bugzilla.mozilla.org/show_bug.cgi?id=1287938
It's probably a good idea to talk to Mike Conley about this.
~ Gijs
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
