Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-12-15 Thread David Burns
Thanks for this Philip. I have started raising bugs and blocking https://bugzilla.mozilla.org/show_bug.cgi?id=1498357. David On Fri, 14 Dec 2018 at 08:41, Philip Jägenstedt wrote: > On Fri, Oct 19, 2018 at 2:42 PM Philip Jägenstedt > wrote: > > > > On Wed, Oct 17, 2018 at 11:53 PM Boris Zbars

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-12-15 Thread Philip Jägenstedt
That's fantastic, there's a lot to triage but hopefully it's well worth it. If you create any ad-hoc mapping between failures and bugs, please comment on https://github.com/web-platform-tests/wpt.fyi/issues/64 and perhaps we can populate using that data when the linking feature exists. +Luke Bjerri

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-12-14 Thread Philip Jägenstedt
On Fri, Dec 14, 2018 at 9:41 AM Philip Jägenstedt wrote: > > On Fri, Oct 19, 2018 at 2:42 PM Philip Jägenstedt wrote: > > > > On Wed, Oct 17, 2018 at 11:53 PM Boris Zbarsky wrote: > > > > > > On 10/13/18 3:27 AM, Philip Jägenstedt wrote: > > > > Fiddling with these rules can reveal lots > > > >

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-12-14 Thread Philip Jägenstedt
On Fri, Oct 19, 2018 at 2:42 PM Philip Jägenstedt wrote: > > On Wed, Oct 17, 2018 at 11:53 PM Boris Zbarsky wrote: > > > > On 10/13/18 3:27 AM, Philip Jägenstedt wrote: > > > Fiddling with these rules can reveal lots > > > more potential issues, and if you like I could provide reports on that >

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-19 Thread Boris Zbarsky
On 10/19/18 8:42 AM, Philip Jägenstedt wrote: That's a bit odd, the is in the markup and would be when running manually or under automation. Are you sure that explains the difference? Yes. I filed https://github.com/web-platform-tests/wpt/issues/13625 -Boris

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-19 Thread Philip Jägenstedt
On Wed, Oct 17, 2018 at 11:53 PM Boris Zbarsky wrote: > > On 10/13/18 3:27 AM, Philip Jägenstedt wrote: > > Fiddling with these rules can reveal lots > > more potential issues, and if you like I could provide reports on that too. > > I would be pretty interested in that, yes. In particular, a rep

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-17 Thread Boris Zbarsky
On 10/13/18 3:27 AM, Philip Jägenstedt wrote: Fiddling with these rules can reveal lots more potential issues, and if you like I could provide reports on that too. I would be pretty interested in that, yes. In particular, a report where there is 1 "not PASS and not FAIL" and 3 "PASS" would be

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-17 Thread Philip Jägenstedt
On Wed, Oct 17, 2018 at 2:03 PM Emilio Cobos Álvarez wrote: > > On 10/17/18 11:56 AM, James Graham wrote: > > On 17/10/2018 10:12, James Graham wrote: > >> On 17/10/2018 01:23, Emilio Cobos Álvarez wrote: > >>> Hi Philip, > >>> > >>> Do you know how do reftests run in order to get that data? > >>>

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-17 Thread Emilio Cobos Álvarez
On 10/17/18 11:56 AM, James Graham wrote: On 17/10/2018 10:12, James Graham wrote: On 17/10/2018 01:23, Emilio Cobos Álvarez wrote: Hi Philip, Do you know how do reftests run in order to get that data? I'm particularly curious about this Firefox-only failure:    css/selectors/selection-image

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-17 Thread James Graham
On 17/10/2018 10:12, James Graham wrote: On 17/10/2018 01:23, Emilio Cobos Álvarez wrote: Hi Philip, Do you know how do reftests run in order to get that data? I'm particularly curious about this Firefox-only failure:    css/selectors/selection-image-001.html It passes both on our automation

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-17 Thread James Graham
On 17/10/2018 01:23, Emilio Cobos Álvarez wrote: Hi Philip, Do you know how do reftests run in order to get that data? I'm particularly curious about this Firefox-only failure:   css/selectors/selection-image-001.html It passes both on our automation and locally. I'm curious because I was t

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-16 Thread Emilio Cobos Álvarez
Hi Philip, Do you know how do reftests run in order to get that data? I'm particularly curious about this Firefox-only failure: css/selectors/selection-image-001.html It passes both on our automation and locally. I'm curious because I was the author of that test (whoops) and the Firefox fix

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-13 Thread Philip Jägenstedt
On Sat, Oct 13, 2018, 09:17 Philip Jägenstedt wrote: > On Thu, Oct 11, 2018, 22:34 Boris Zbarsky wrote: > >> On 10/11/18 4:22 PM, Philip Jägenstedt wrote: >> > https://gist.github.com/foolip/a77c88e62aa3cfc461c2879f3e5d4855 is a >> > list of tests that fail in Firefox Nightly, but pass in stable

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-13 Thread Philip Jägenstedt
On Thu, Oct 11, 2018, 22:34 Boris Zbarsky wrote: > On 10/11/18 4:22 PM, Philip Jägenstedt wrote: > > https://gist.github.com/foolip/a77c88e62aa3cfc461c2879f3e5d4855 is a > > list of tests that fail in Firefox Nightly, but pass in stable > > versions of Chrome, Edge and Safari. > > Or more precise

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-11 Thread Boris Zbarsky
I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1498357 to track these failures. -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform

Re: web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-11 Thread Boris Zbarsky
On 10/11/18 4:22 PM, Philip Jägenstedt wrote: https://gist.github.com/foolip/a77c88e62aa3cfc461c2879f3e5d4855 is a list of tests that fail in Firefox Nightly, but pass in stable versions of Chrome, Edge and Safari. Or more precisely have some sub-test that has that property, right? Thank you f

web-platform-tests that fail only in Firefox (from wpt.fyi data)

2018-10-11 Thread Philip Jägenstedt
Hi all, I sent the result of some investigation to webkit-dev [1] today and thought you might be interested to take a look the equivalent list for Firefox. https://gist.github.com/foolip/a77c88e62aa3cfc461c2879f3e5d4855 is a list of tests that fail in Firefox Nightly, but pass in stable versions