Hi Joel, I spent about an hour tonight trying to debug a test failure, and I'm writing this email in frustration at how difficult it is. It seems like the process has actually gotten a lot worse over the last few years (although it was never good). Here's the situation I ran into:
A test is failing on try. I want to reproduce it. Assume that running the test by itself isn't sufficient. I would like to run whatever set of tests actually ran together on the test machine in a single Firefox invocation. I don't want any more tests to run than those. I can't figure out any way to do that. I can pass a directory to |mach mochitest|. But that has a number of problems: - It also runs every subdirectory recursively inside that directory. Often that includes way more tests. I can't figure out any way to stop it from doing this. I tried the "--chunk-by-dir" option, but it complains that the argument is supposed to be an integer. What is this option for? - |mach mochitest| runs all flavor of tests even though I only want one. There is the --flavor option to disable that. But I have never figured out how to use it correctly. No matter what I do, some irrelevant devtools are a11y or plugin tests seem to run instead of what I want. - There is a --start-at option that should allow me to start running tests near the one that I want. But it never seems to work either. I'm not sure if it's confounded by the two problems above, or if it's just completely broken. We could easily fix this by printing in the tinderbox log the mach command that you need to run in order to run the tests for a particular directory (and making that discoverable through treeherder). I want to emphasize that, from a developer's perspective, this is the second most basic thing that I could ask for. (Simply running a test by itself is the most basic, and it works fine.) Running tests by directory in automation has been a huge improvement, but we're not really earning the dividends from it because it's so hard to get the same behavior locally. Anyway, sorry about the rant. And sorry to pick on your email. But it's frustrating to see all these advanced ideas being proposed when we can't even get basic stuff right. As an aside, I would also like to complain about the decision to strip a lot of the useful information out of test logs. I understand this was done to make the tests faster, and I can "just" check in a patch to add "SimpleTest.requestCompleteLog()" to the intermittent test. But why didn't we instead figure out why logging was so slow and fix that? Fundamentally, it doesn't seem like saving 50MB of log data to S3 should take very long. -Bill On Tue, Feb 7, 2017 at 9:40 AM, <jma...@mozilla.com> wrote: > This is the second update of project stockwell (first update: > https://goo.gl/1X31t8). > > This month we will be recommending and asking that intermittent failures > that occur >=30 times/week be resolved within 2 weeks of triaging them. > > Yesterday we had these stats: > Orange Factor: 10.75 (https://goo.gl/qvFbeB) > count(high_frequency_bugs): 61 > > Last month we had these stats: > Orange Factor: 13.76 (https://goo.gl/o5XOof) > count(high_frequency_bugs): 42 > > For more details of the bugs and what we are working on, you can read more > on this recent blog post: > https://elvis314.wordpress.com/2017/02/07/project-stockwell-february-2017/ > > Thanks for helping out with intermittent test failures when we ping you > about them! > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
