Then I am on board. We still do need to revisit the handling of flaky tests. The current approach is an absolute disaster. (I normally love exaggerating, but in this case, I feel no satisfaction in doing so because it is at best an understatement.)
-Filip On Aug 17, 2012, at 5:36 PM, Dirk Pranke <[email protected]> wrote: > All non-flaky failures, yes. > > Flaky tests would still require entries in the TestExpectations files > at this time; discussion of how to treat them is a separate topic. > > -- Dirk > > On Fri, Aug 17, 2012 at 5:35 PM, Filip Pizlo <[email protected]> wrote: >> +1, contingent upon the following: are we agreeing that all current uses of >> TEXT, IMAGE, and so forth in TestExpectations should be in the *very near >> term* following Dirk's change be turned into -failing files? >> >> -Filip >> >> >> On Aug 17, 2012, at 5:01 PM, Ryosuke Niwa <[email protected]> wrote: >> >> On Fri, Aug 17, 2012 at 4:55 PM, Ojan Vafai <[email protected]> wrote: >>> >>> Asserting a test case is 100% correct is nearly impossible for a large >>> percentage of tests. The main advantage it gives us is the ability to have >>> -expected mean "unsure". >>> >>> Lets instead only add -failing (i.e. no -passing). Leaving -expected to >>> mean roughly what it does today to Chromium folk (roughly, as best we can >>> tell this test is passing). -failing means it's *probably* an incorrect >>> result but needs an expert to look at it to either mark it correct (i.e. >>> rename it to -expected) or figure out how the root cause of the bug. >>> >>> This actually matches exactly what Chromium gardeners do today, except >>> instead of putting a line in TestExpectations/Skipped to look at later, they >>> checkin the -failing file to look at later, which has all the advantages >>> Dirk listed in the other thread. >> >> >> I'm much more comfortable with this proposal. >> >> - Ryosuke >> >> _______________________________________________ >> webkit-dev mailing list >> [email protected] >> http://lists.webkit.org/mailman/listinfo/webkit-dev >> >> _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
