James Willcox <jwill...@mozilla.com> writes:
> I guess there's not currently an easy way to enable Marionette for
> non-debug builds?
You can set the ENABLE_MARIONETTE output variable to something exciting
in your mozconfig to enable it in optimised builds.
> One thing I looked into before was using ChromeDriver and Marionette
> to do head-to-head performance tests against Chrome on Android, and it
> would be great if we could use Nightly or something.
One should be aware that no software instrumentation comes at zero cost.
WebDriver was not designed to be a performance measurement tool, and
cross-browser performance measurement is a particularly difficult area.
It does however provide ways to remote control the browser to _initiate_
the necessary measurements and _collect_ the data.
There are a lot of considerations to be taken with regards to variance
and outliers: you can’t test against live sites; you need to
forensically examine cached documents to ensure they don’t serve
different content based on the UA; the test environment needs to be
hermetic and network conditions reproducible; considerations must be
made with regards to cold-start vs. hot-start; population of various
caches could prove challenging &c. &c.
But given a sufficiently sophisticated test environment, there is
probably data available in Gecko that it would be interesting to harvest
and look at, especially if we could compare it to other browsers’.
The core issue preventing that is that vendors haven’t agreed
on _what_ categories of data it would make sense to expose in a
cross-browser API. The little discussion we’ve had so far about this
can be found here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1250290
As the Browser Tools- and Testing Working Group is finishing up the last
pieces of the puzzle in the base specification…
http://w3c.github.io/webdriver/webdriver-spec.html
… it will probably soon be time to start thinking about what
additional benefit we can garner from an out-of-process remote control
protocol like this in the future. We have already discussed permissions
management with the WebBluetooth WG. Logging- and performance APIs also
make a great deal of sense.
At the last WebDriver WG meeting we spoke with the Edge performance
PM Todd Reifsteck who is doing things similar to what you describe.
He had noted Firefox was slower at loading pages when run under
instrumentation, which can probably be attributed to the way Marionette
injects code on navigation. It would be good to track down and fix
that, but it proves the point that no automation comes for free.
For the record I don’t think that externally measured load time
comparisons are very interesting: there are likely less coarse metrics
to look at that has less variance, such as MOZ_PROFILER dumps or the
performance timeline.
_______________________________________________
mobile-firefox-dev mailing list
mobile-firefox-dev@mozilla.org
https://mail.mozilla.org/listinfo/mobile-firefox-dev
