I think we're all in agreement. Adding support for manifest and mach/make option with the specific directory makes sense. To summarize: * land tests in a directory name "external" so it's easily identified as needing external access. * implement support in test manifest and mach/make to enable execution of external tests only (default would be not to run external tests)
I've updated the wiki. https://wiki.mozilla.org/QA/External_Tests -edwin ----- Original Message ----- From: "Jonathan Griffin" <jgrif...@mozilla.com> To: dev-platform@lists.mozilla.org Sent: Thursday, August 14, 2014 10:31:28 AM Subject: Re: External dependent tests in gecko and gaia I think this is a great idea, although like others have said, I'd like to have this implemented inside the test manifests, regardless of directory structure. A related piece is reporting; for years, we've had tests like this run on separate systems, reporting to custom dashboards, because they weren't appropriate for buildbot, and TBPL can't display data for tests running anywhere else. With Treeherder, we're now in a position to be able to display results for tests running anywhere, and this could lead to greater visibility and adoption of non-buildbot automation. Such tests would still need to be sheriffed differently (by different people and/or with different rules), so we'd have to work out what the views for these tests should be, but using Treeherder to expose test results for the variety of automation that's run in Mozilla will be a big win. Jonathan On 8/13/2014 9:21 AM, Edwin Wong wrote: > Hi dev-platform, > > TL;DR - Cloud Services and Quality Engineering would like to propose the > creation of a directory named “external" in gecko and gaia repos for > externally dependent tests. > > This enables features married to Cloud Services such as Loop, FindMyDevice, > FirefoxAccounts, and Sync to have centralized tests that can be run locally > or on other continuous integration systems. These tests would live in this > "external” directory along side existing tests (so they live together). These > will be run and sheriffed independently from the main tests. Reviews would > be governed by modules and feature teams. > > More detail: > https://wiki.mozilla.org/QA/External_Tests > > Cheers, > Edwin Wong > Cloud Services QE Manager > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
