Re: External dependent tests in gecko and gaia

2014-08-15 Thread Jared Hirsch
Hey Edwin, >From my perspective (building service-connected Gaia apps), this proposal >makes sense: we should give developers the option of testing against real >servers. However, I think the proposal would be improved if it described the tradeoffs of the mock server and real server approaches

Re: External dependent tests in gecko and gaia

2014-08-15 Thread Edwin Wong
I’m open to either options. So the current proposal is keeping tests with external access *in the same directory* as existing tests governed by manifest files. Good to know that adding another directory is something to avoid. -edwin On Aug 14, 2014, at 2:31 PM, Ehsan Akhgari wrote: > On 20

Re: External dependent tests in gecko and gaia

2014-08-14 Thread Ehsan Akhgari
On 2014-08-14, 5:00 PM, Edwin Wong wrote: Hi Ms2ger, Thanks for calling me on that, just trying to move things along. Test location and using manifests concepts can also be decoupled. Wherever the location the tests, we will use manifests to define it’s execution. There isn’t really any oth

Re: External dependent tests in gecko and gaia

2014-08-14 Thread Edwin Wong
Hi Ms2ger, Thanks for calling me on that, just trying to move things along. Test location and using manifests concepts can also be decoupled. Wherever the location the tests, we will use manifests to define it’s execution. There isn’t really any other way around that. So now the decision

Re: External dependent tests in gecko and gaia

2014-08-14 Thread Ms2ger
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/14/2014 08:22 PM, Edwin Wong wrote: > > I think we're all in agreement. Adding support for manifest and > mach/make option with the specific directory makes sense. To > summarize: > > * land tests in a directory name "external" so it's easil

Re: External dependent tests in gecko and gaia

2014-08-14 Thread Edwin Wong
August 14, 2014 10:31:28 AM Subject: Re: External dependent tests in gecko and gaia I think this is a great idea, although like others have said, I'd like to have this implemented inside the test manifests, regardless of directory structure. A related piece is reporting; for years, we've

Re: External dependent tests in gecko and gaia

2014-08-14 Thread Jonathan Griffin
I think this is a great idea, although like others have said, I'd like to have this implemented inside the test manifests, regardless of directory structure. A related piece is reporting; for years, we've had tests like this run on separate systems, reporting to custom dashboards, because they

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Doug Turner
> Anyways, now that gaia revisions are tied to m-c revisions with > b2g/config/gaia.json, regression range hunting should be a matter of > tooling around that, now. > That is excellent news. ___ dev-platform mailing list dev-platform@lists.mozilla.or

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Mike Hommey
On Wed, Aug 13, 2014 at 10:52:29AM -0700, Doug Turner wrote: > Did we ever reach agreement about moving gaia into m-c? It would make > this type of problem go away, and make regression range hunting much > easier. The gaia repo is massive, importing it in m-c would more than double m-c's size. An

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Ted Mielczarek
On 8/13/2014 12:21 PM, Edwin Wong wrote: > Hi dev-platform, > > TL;DR - Cloud Services and Quality Engineering would like to propose the > creation of a directory named “external" in gecko and gaia repos for > externally dependent tests. > > This enables features married to Cloud Services such

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Ehsan Akhgari
On 2014-08-13, 3:07 PM, Edwin Wong wrote: Ehsan, This wouldn’t be a top level dir, it would be as such: • [area]/[framework]/external/test_example.js • content/media/test/mochitest/external/test_loop.html • gaia/apps/findmydevice/test/marionette/external/fmd_locate.py

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Edwin Wong
Ehsan, This wouldn’t be a top level dir, it would be as such: • [area]/[framework]/external/test_example.js • content/media/test/mochitest/external/test_loop.html • gaia/apps/findmydevice/test/marionette/external/fmd_locate.py -edwin On Aug 13, 2014, at 11:58 AM, Ehsan

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Edwin Wong
Hi Greg, A few had discussed manifest vs directory separation of tests. I think the directory idea took root because it’s easily identifiable from it’s path, reducing collision with existing tests. I also agree, using directory as a form of meta data is limiting. That said, both concepts can

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Ehsan Akhgari
On 2014-08-13, 12:21 PM, Edwin Wong wrote: Hi dev-platform, TL;DR - Cloud Services and Quality Engineering would like to propose the creation of a directory named “external" in gecko and gaia repos for externally dependent tests. This enables features married to Cloud Services such as Loop, F

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Doug Turner
Did we ever reach agreement about moving gaia into m-c? It would make this type of problem go away, and make regression range hunting much easier. On Aug 13, 2014, at 9:21 AM, Edwin Wong wrote: > Hi dev-platform, > > TL;DR - Cloud Services and Quality Engineering would like to propose the

Re: External dependent tests in gecko and gaia

2014-08-13 Thread Gregory Szorc
On 8/13/14 9:21 AM, Edwin Wong wrote: Hi dev-platform, TL;DR - Cloud Services and Quality Engineering would like to propose the creation of a directory named “external" in gecko and gaia repos for externally dependent tests. This enables features married to Cloud Services such as Loop, FindMy

External dependent tests in gecko and gaia

2014-08-13 Thread Edwin Wong
Hi dev-platform, TL;DR - Cloud Services and Quality Engineering would like to propose the creation of a directory named “external" in gecko and gaia repos for externally dependent tests. This enables features married to Cloud Services such as Loop, FindMyDevice, FirefoxAccounts, and Sync to