On Tue, Aug 21, 2012 at 4:16 PM, Maciej Stachowiak <[email protected]> wrote: > > On Aug 21, 2012, at 3:23 PM, Ojan Vafai <[email protected]> wrote: > > On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <[email protected]> wrote: >> >> Here's how I imagine the workflow when a sheriff or just innocent >> bystander notices a deterministically failing test. Follow this two-step >> algorithm: >> >> 1) Are you confident that the new result is an improvement or no worse? If >> so, then simply update -expected.txt. >> 2) Otherwise, copy the old result to >> -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new >> result as -<whatever-we-call-the-expected-failure-result.txt>. > > > I think we should do this. I don't care much about the naming. > >> >> This replaces all other approaches to marking expected failures, including >> the Skipped list, overwriting -expected even you know the result is a >> regression, marking the test in TestExpectations as Skip, Wontfix, Image, >> Text, or Text+Image, or any of the other legacy techniques for marking an >> expected failure reult. > > > Don't forget suffixing the test with "-disabled"! We have 109 such tests at > the moment according to > http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs. > I think we should also get rid of this. If we need a way to disable a test > across ports (e.g. because it crashes in cross-platform code), we should > make a Skipped/TestExpectations file in LayoutTest/platform instead of > renaming the test file. > > > I agree that renaming to -disabled should be phased out as well. I > specifically did not cover failure modes that produce no result, such as > crashes or hangs. Those should still be tracked via TestExpectations IMO. > Likewise for nondeterministic expectations failures. >
Since we support cascading TestExpectations now, we can replace -disabled with a common TestExpectations file that all ports share, as appropriate (this would require ORWT to recognize TestExpectations, of course, which I plan to make happen soonish). -- Dirk _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
