On Fri, Dec 14, 2018 at 9:41 AM Philip Jägenstedt <foo...@chromium.org> wrote: > > On Fri, Oct 19, 2018 at 2:42 PM Philip Jägenstedt <foo...@chromium.org> wrote: > > > > On Wed, Oct 17, 2018 at 11:53 PM Boris Zbarsky <bzbar...@mit.edu> wrote: > > > > > > On 10/13/18 3:27 AM, Philip Jägenstedt wrote: > > > > Fiddling with these rules can reveal lots > > > > more potential issues, and if you like I could provide reports on that > > > > too. > > > > > > I would be pretty interested in that, yes. In particular, a report > > > where there is 1 "not PASS and not FAIL" and 3 "PASS" would be pretty > > > helpful, I suspect. > > > > Rerunning my script it's apparent that unreliable Edge results [1] > > leads to the same tests being considered lone failures or not for the > > other browsers. So, I've use the same set of runs for this report of > > what you suggested: > > https://gist.github.com/foolip/e6014c9bcc8ca405219bf18542eb5d69 > > > > It's not a long list, so I checked them all and they are timeouts. > > This is sometimes the failure mode for genuine problems, so looking > > over these might be valuable. > > Given the recent news [1] it won't be as relevant to consider the > status of EdgeHTML for prioritization in other engines. Given that and > the unreliable results, I've updated my script to consider only > Chrome, Firefox and Safari. I also the reports auto-updating on a > daily basis: > https://foolip.github.io/ad-hoc-wpt-results-analysis/chrome-lone-failures.html > https://foolip.github.io/ad-hoc-wpt-results-analysis/firefox-lone-failures.html > https://foolip.github.io/ad-hoc-wpt-results-analysis/safari-lone-failures.html > > [1] https://github.com/MicrosoftEdge/MSEdge/blob/master/README.md
And, to spell it out, the effect of that is to increase the number of product-specific for all three by quite a lot. Firefox goes from ~700 to ~1300. Chrome went from ~300 to ~900, and I'm suggesting that we get to at least <500 and stay there. (I suspect many failures are for trivial reasons, so that it'll be easy to make progress in the beginning.) _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
