There's good feedback in here. Are some of these known, jmaher? Are any intentional choices, or should we just start turning these into bugs to get fixed?
On 08/02/2017 12:33 AM, Bill McCloskey wrote: > Hi Joel, > I spent about an hour tonight trying to debug a test failure, and I'm > writing this email in frustration at how difficult it is. It seems like the > process has actually gotten a lot worse over the last few years (although > it was never good). Here's the situation I ran into: > > A test is failing on try. I want to reproduce it. Assume that running the > test by itself isn't sufficient. I would like to run whatever set of tests > actually ran together on the test machine in a single Firefox invocation. I > don't want any more tests to run than those. I can't figure out any way to > do that. > > I can pass a directory to |mach mochitest|. But that has a number of > problems: > - It also runs every subdirectory recursively inside that directory. Often > that includes way more tests. I can't figure out any way to stop it from > doing this. I tried the "--chunk-by-dir" option, but it complains that the > argument is supposed to be an integer. What is this option for? > - |mach mochitest| runs all flavor of tests even though I only want one. > There is the --flavor option to disable that. But I have never figured out > how to use it correctly. No matter what I do, some irrelevant devtools are > a11y or plugin tests seem to run instead of what I want. > - There is a --start-at option that should allow me to start running tests > near the one that I want. But it never seems to work either. I'm not sure > if it's confounded by the two problems above, or if it's just completely > broken. > > We could easily fix this by printing in the tinderbox log the mach command > that you need to run in order to run the tests for a particular directory > (and making that discoverable through treeherder). > > I want to emphasize that, from a developer's perspective, this is the > second most basic thing that I could ask for. (Simply running a test by > itself is the most basic, and it works fine.) Running tests by directory in > automation has been a huge improvement, but we're not really earning the > dividends from it because it's so hard to get the same behavior locally. > > Anyway, sorry about the rant. And sorry to pick on your email. But it's > frustrating to see all these advanced ideas being proposed when we can't > even get basic stuff right. > > As an aside, I would also like to complain about the decision to strip a > lot of the useful information out of test logs. I understand this was done > to make the tests faster, and I can "just" check in a patch to add > "SimpleTest.requestCompleteLog()" to the intermittent test. But why didn't > we instead figure out why logging was so slow and fix that? Fundamentally, > it doesn't seem like saving 50MB of log data to S3 should take very long. > > -Bill > > On Tue, Feb 7, 2017 at 9:40 AM, <jma...@mozilla.com> wrote: > >> This is the second update of project stockwell (first update: >> https://goo.gl/1X31t8). >> >> This month we will be recommending and asking that intermittent failures >> that occur >=30 times/week be resolved within 2 weeks of triaging them. >> >> Yesterday we had these stats: >> Orange Factor: 10.75 (https://goo.gl/qvFbeB) >> count(high_frequency_bugs): 61 >> >> Last month we had these stats: >> Orange Factor: 13.76 (https://goo.gl/o5XOof) >> count(high_frequency_bugs): 42 >> >> For more details of the bugs and what we are working on, you can read more >> on this recent blog post: >> https://elvis314.wordpress.com/2017/02/07/project-stockwell-february-2017/ >> >> Thanks for helping out with intermittent test failures when we ping you >> about them! >> _______________________________________________ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform >> > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
