The Android test environments used for continuous integration have been through many changes over the last year or two; here's a review of what we have today. [1]
Most of our Android tests run on emulators. Some run on hardware: real phones. Our Android hardware tests run on physical devices -- Motorola G5 and Pixel 2 phones currently -- and those phones are physically managed by bitbar, a device farm provider. These test platforms appear on treeherder as "Android 7.0 MotoG5" and "Android 8.0 Pixel2". Running tests on hardware is relatively expensive so we make deliberate choices about which tests run on hardware. All of our performance (raptor) tests, architecture-sensitive tests like jittest and jsreftest, and select tests requiring special capabilities run on hardware. All other tests -- web-platform tests, most mochitests, reftests, xpcshell tests, etc -- run on emulators. The emulator test platform appears on treeherder as "Android 7.0 x86-64". These tests run in the Android x86_64 emulator on a Linux host. Unlike previous generations of our emulator test environment, today's emulator tests are fast: Thanks to hardware acceleration, tests run at about the same rate as they do on our desktop platforms. The Android tests running on trunk today are testing geckoview apps (Fennec tests continue to run on the esr68 branch). Most raptor tests run in the geckoview_example app; additional raptor tests run in Fenix and the Reference Browser; most other tests run in the geckoview test app. Both emulator and hardware tests have a fixed pool of instances: Regardless of load, we can only run N emulator tasks, or M hardware tasks at a time. Release Engineering Operations monitors backlog for both pools, but temporary backlogs are expected and tolerated. Since our hardware testing capacity is particularly limited, to run Android hardware tests on try, you must use the --full option with 'mach try fuzzy' [2]. For instance, you can see the available tests with 'mach try fuzzy --no-push --full --query "android-hw"' and you could run android-hw mochitest-media tests with 'mach try fuzzy --full --query "android-hw mochitest-media"'. There are no similar restrictions on try runs for emulator tests -- but please use responsibly! [1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2Ctest [2] https://firefox-source-docs.mozilla.org/tools/try/selectors/fuzzy.html _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
