On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson <[email protected]> wrote:
> On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa <[email protected]> wrote: > >> It should be fairly straight forward to create a tool that analyzes files >> changed in each commit and deduce which tests' expected results have been >> changed. The tool can then fetch results from each port' bot for those >> tests and automatically land them. It can then comment on the bug >> automatically about these rebaseline commits. There is no need to add & >> remove entries from TestExpectation files. >> > > I think it's implied in other messages in the thread, but just to be > explicit: this automated rebaseline commit feels like exactly the wrong > thing to do. How can you rebaseline a test without SOME manual inspection? > I know that mass layout rebaselines may not have every pixel checked, but > there's tooling to help with that. > > Changes that are benign in one port may not be benign in other ports. > Automatically landing changes to other ports' baselines risks corrupting > our test data. > I hate to repeat myself every time this conversation comes up but here it is: historically, only Chromium port used expected results as "correct results" and non-Chromium ports used expected results files to store "the current state of the world" including expected failures although this may have changed a little since non-Chromium ports started using TestExpectations instead of Skipped files. There is a significant danger in adding test expectations as opposed to committing expected failures because tests with Failure or ImageOnlyFailure expectation can regress further in any minute. I've encountered several severe regressions that would have caught if relevant tests had not been marked Failure or ImageOnlyFailure (in some cases needing rebaselines) for some minor rendering issues in the past. So no, it's not exactly the wrong thing to do. - R. Niwa
_______________________________________________ webkit-dev mailing list [email protected] https://lists.webkit.org/mailman/listinfo/webkit-dev
