Thanks everyone for chiming in here. I see this isn't as simple as a binary decision and to simplify things, I think turning on all non-e10s tests that were running for windows7-debug would give us reasonable coverage and ensure that users on our most popular OS (and focus for 57) have a stable experience if they choose to go without e10s.
Instead of a date to turn things off, I would like to just focus on each framework one at a time and ideally in a few months we would have a clear set of requirements (in the form of bugs) to resolve before turning off the specific non-e10s tests. If there is a different configuration other than windows7-debug I would like to hear about it. Joel On Wed, Aug 16, 2017 at 12:53 PM, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote: > On 08/15/2017 04:29 PM, jma...@mozilla.com wrote: > >> I would propose running these above tests on windows7-opt (we don't have >> these running yet in windows 10, although we are close), and only running >> specific tests which are not run in e10s mode, turning them off December >> 29th, 2017. >> >> Keep in mind we have had many years to get all our tests running in e10s >> mode and we have known since last year that Firefox 57 would be the first >> release that we ship e10s by default to our users- my proposal is a 4 month >> temporary measure to let test owners finish ensuring their tests are >> running in e10s mode. >> > Hi Joel, > > I think Ben is right on here. Please note that it's not just service > workers that have a different implementation in e10s vs. non-e10s, many > other super important parts of the browser are in the same boat. For > example, our entire HTTP(S) stack has different code paths in e10s vs. > non-e10s. Furthermore, even in places where we share a lot of code in the > two modes, there are often subtle differences in the behavior in between > the two modes. I hope this is convincing in terms of the necessity of > maintaining the test coverage across the two modes. > > To slightly readjust what you stated above from the perspective of the > Firefox developers, we have known since last years that Firefox 57 would be > the first release that we ship e10s by default for our *desktop* users, and > we have also known that Firefox 57 for Android will be shipped without e10s > for our *mobile* users, so it shouldn't be any surprise why no large scale > effort has been put into porting all of our tests to run into e10s mode. > Also please note that some of the test frameworks you have listed above > like browser-chrome aren't relevant to Android, and some like jsreftests > aren't relevant to e10s, so we should probably have a more detailed > conversation about the remaining ones. > > I think a better way to think about this problem is perhaps how to get to > a point where we never end up losing test coverage on code that we ship on > our tier 1 platforms. Thinking about this in terms of date-based deadlines > where we don't have the human power to do the work necessary isn't > particularly helpful, and would ultimately result us in losing invaluable > test coverage. If past history is any lesson for us, regressions will > creep in as a result, and especially due to the fact that this is mostly > affecting Firefox for Android where we have the lowest pre-release testing > population among our tier1 products (AFAIK) makes that extremely risky. > > Therefore, I think we should: > > * start running all of the non-e10s tests that can affect Android again > immediately. > * work out a realistic plan between engineering and the automation team > to address the existing gaps that prevent us from turning off non-e10s > tests without losing coverage on Android, and execute that plan. > * turn non-e10s tests off when the above has been done. > > Please let me know what you think! > > Thanks, > Ehsan > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
