On Mon, Aug 4, 2014 at 6:21 PM, Gregory Szorc <g...@mozilla.com> wrote:
> On 8/4/14, 10:39 AM, Dave Townsend wrote: > >> I've done a little investigation into marionette and I've found a few >> issues with it: >> >> Firstly it doesn't look like running marionette directly or through mach >> allows developers to select individual directories or test files to run, >> rather it is a one-shot affair. This is very inconvenient for development. >> >> Secondly marionette doesn't seem to be built to scale to many test >> types. It uses regular expressions on filenames to determine the test >> type, as it happens the Jetpack tests do use a different form to the >> existing marionette tests so it's not out of the question but still >> makes me wary of adding a new test type. >> >> Thirdly I can't run marionette tests locally, they consistently fail >> quite badly. >> >> These problems make marionette a less than desirable option for use as a >> base for our test harness right now so I plan to get my work to make >> mochitests run jetpack tests completed this week and submit it for >> review. If Marionette becomes a better choice in the future a lot of the >> work I'm doing right now carries over, it will be simpler to switch from >> mochitest to marionette than it is switching to mochitest. >> > > The issues listed seem fixable. I would rather we spend energy improving > Marionette than piling yet more things on top of mochitest's haphazard base. > > The various automation "failures" in the past few weeks should be reason > enough to avoid mochitest and go with a better-engineered and tested > solution (marionette). > Who is going to do that work? I have patches that vastly improve the testing situation for jetpack tests by allowing other developers run them more easily, making them easier for releng to manage and most importantly making them meet tbpl visibility requirements. They involve tiny changes to the mochitest harness code. We're already hidden by default on tinderbox and are hitting problems because of it, I'd rather go ahead and finish up this work than wait for some future time when marionette can be upgraded to meet our requirements. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
