Hey, I've spent a few spare cycles looking at the UI tests, which are currently failing consistently.
I thought this might be a set of fixes which roughly translate to "sprinkle in a few `tester().waitForAnimationsToFinish()` statements", but it turns out to not be as simple as that. I'll probably get some more time later on in the week, but thought I'd document what I'd seen/discovered before the next Testing Tuesday. I'll reproduce some of my findings/data below the sig, but the most immediate conclusions/executive summary/TL;DR are here in a bullet list. - there is an ordering dependency in UITests. NavigationTests and PrivateBrowsingTests both have side effects that cause subsequent tests to fail. - There are genuinely failing tests, some intermittent, some not. While I haven't got much to show for it (there are still 97 tests, with 25 – 26 failing), I feel like I'm making progress. Let me know if you have any ideas, or extra information I could've easily missed. Many thanks, and regards, – James The most interesting tests are the classes that not all methods fail. My assumption is that the ones where all fail are caused by a teardown or lack of from a previous test case. The following seem to consistently fail in the runs I've done. NavigationTests.testBackForwardNavigationRestoresMobileOrDesktopSite PrivateBrowsingTests.testClosePrivateTabsClosesPrivateTabs (fails on smaller devices) ReaderViewUITests.testReaderViewUI ViewMemoryLeakTests.testWebViewDisposed The following I've see fail when run within their TestCase, but not when running all of UI Tests. ViewMemoryLeakTests.testAboutHomeDisposed The following I've seen fail when run when running all of UI Tests, but pass when running a single test case. DomainAutocompleteTests.testAutocomplete PrivateBrowsingTests.testPrivateTabDoesntTrackHistory SearchTests.testURLBarContextMenu Starting at these more interesting cases, I've found two test cases that cause cascading failures. Disable nothing extra. 97 tests, 25 failing, 1 disabled. 41x Disable: Navigationtests, PrivateBrowsingTests 97 tests, 3 failing, 15 disabled. 5x. 97 tests, 4 failing, 15 disabled. 6x Disable: PrivateBrowsingTests 97 tests, 24 failing, 5 disabled 41x? Disable NavigationTests 97 tests, 24 failing, 11 disabled 41x I haven't been able to change the execution order of the test cases (the classes), though I do note that the test methods themselves are run alphabetically. So far, I haven't seen anything obvious in the teardown of NavigationTests or PrivateBrowsingTests that allow them to pass themselves, but cause subsequent failures. Other interesting things: - the whole UITests test suite is taking about 25 minutes to run. - Xcode occasionally stops compiling tests, even though you know it should compile. - Xcode occasionally declares something fails an integrity check, but changes its mind after a restart. - The simulator seems to stop passing tests after you run a few test runs. I haven't observed it losing its way, just I get to it and see it's timing out for every test. I haven't checked for memory leaks. - There's a bunch of opportunities for sharing code between UI tests and L10N screenshotting. We tend to use accessibility labels in the tests, though probably should be using accessibilityIdentifiers. A simple navigate to/from screen DSL would be really helpful here. Next steps: - FxKIFTestCase common super class to help logging/instrumentation and probably lifecycle. We should implement a common afterAll() method. - UITestUtils to help teardown/setup. This ideally would be resetting the profile, and setting a pref for IntroViewControllerSeen, though end up being working on BrowserUtils and adding a clearPrivateDataFromHome class method which can be called judiciously. - Add to the iOS/Technical Debt page: profile reset, consider sharing code between l10n screen shot code and UI tests. - Ask for help! HALP! There are a bunch of tests which are failing because of regressions. If you see a test you wrote please could you take a look at it, and see if you can get it to run as a TestCase on its own. Message ends.
_______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev@mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev
