Thank you for this investigation James. Catalan and I can spend an additional cycle of manual coverage on the continuing finicky and disabled tests to at least ensure we aren’t flying blind.
Aaron Train Sr. QE Engineer Mozilla Corporation https://quality.mozilla.org > On May 1, 2016, at 8:32 PM, James Hugman <jhug...@mozilla.com> wrote: > > Hey, > > I've spent a few spare cycles looking at the UI tests, which are currently > failing consistently. > > I thought this might be a set of fixes which roughly translate to "sprinkle > in a few `tester().waitForAnimationsToFinish()` statements", but it turns out > to not be as simple as that. > > I'll probably get some more time later on in the week, but thought I'd > document what I'd seen/discovered before the next Testing Tuesday. > > I'll reproduce some of my findings/data below the sig, but the most immediate > conclusions/executive summary/TL;DR are here in a bullet list. > there is an ordering dependency in UITests. NavigationTests and > PrivateBrowsingTests both have side effects that cause subsequent tests to > fail. > There are genuinely failing tests, some intermittent, some not. > While I haven't got much to show for it (there are still 97 tests, with 25 – > 26 failing), I feel like I'm making progress. > > Let me know if you have any ideas, or extra information I could've easily > missed. > > Many thanks, and regards, > > – James > > > The most interesting tests are the classes that not all methods fail. My > assumption is that the ones where all fail are caused by a teardown or lack > of from a previous test case. > > The following seem to consistently fail in the runs I've done. > NavigationTests.testBackForwardNavigationRestoresMobileOrDesktopSite > PrivateBrowsingTests.testClosePrivateTabsClosesPrivateTabs (fails on smaller > devices) > ReaderViewUITests.testReaderViewUI > ViewMemoryLeakTests.testWebViewDisposed > > The following I've see fail when run within their TestCase, but not when > running all of UI Tests. > ViewMemoryLeakTests.testAboutHomeDisposed > > The following I've seen fail when run when running all of UI Tests, but pass > when running a single test case. > DomainAutocompleteTests.testAutocomplete > PrivateBrowsingTests.testPrivateTabDoesntTrackHistory > SearchTests.testURLBarContextMenu > > Starting at these more interesting cases, I've found two test cases that > cause cascading failures. > > Disable nothing extra. > 97 tests, 25 failing, 1 disabled. 41x > > Disable: Navigationtests, PrivateBrowsingTests > 97 tests, 3 failing, 15 disabled. 5x. > 97 tests, 4 failing, 15 disabled. 6x > > Disable: PrivateBrowsingTests > 97 tests, 24 failing, 5 disabled 41x? > > Disable NavigationTests > 97 tests, 24 failing, 11 disabled 41x > > I haven't been able to change the execution order of the test cases (the > classes), though I do note that the test methods themselves are run > alphabetically. > > So far, I haven't seen anything obvious in the teardown of NavigationTests or > PrivateBrowsingTests that allow them to pass themselves, but cause subsequent > failures. > > Other interesting things: > the whole UITests test suite is taking about 25 minutes to run. > Xcode occasionally stops compiling tests, even though you know it should > compile. > Xcode occasionally declares something fails an integrity check, but changes > its mind after a restart. > The simulator seems to stop passing tests after you run a few test runs. I > haven't observed it losing its way, just I get to it and see it's timing out > for every test. I haven't checked for memory leaks. > There's a bunch of opportunities for sharing code between UI tests and L10N > screenshotting. We tend to use accessibility labels in the tests, though > probably should be using accessibilityIdentifiers. A simple navigate to/from > screen DSL would be really helpful here. > Next steps: > FxKIFTestCase common super class to help logging/instrumentation and probably > lifecycle. We should implement a common afterAll() method. > UITestUtils to help teardown/setup. This ideally would be resetting the > profile, and setting a pref for IntroViewControllerSeen, though end up being > working on BrowserUtils and adding a clearPrivateDataFromHome class method > which can be called judiciously. > Add to the iOS/Technical Debt page: profile reset, consider sharing code > between l10n screen shot code and UI tests. > Ask for help! HALP! There are a bunch of tests which are failing because of > regressions. If you see a test you wrote please could you take a look at it, > and see if you can get it to run as a TestCase on its own. > > > > > > > > > Message ends. > > > > _______________________________________________ > mobile-firefox-dev mailing list > mobile-firefox-dev@mozilla.org > https://mail.mozilla.org/listinfo/mobile-firefox-dev
_______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev@mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev
