On Tue, Nov 6, 2012 at 11:58 PM, Žan Doberšek <[email protected]> wrote: > Hi WebKitties, > > I'd like to see in what scale the --test-list option in NRWT is used and > whether anyone would object a change in how its usage behaves. > > Currently the test gathering takes into account any paths that were passed > in through the command line and any paths listed in the file to which a > possible --test-file option points[1]. These paths are then expanded if > necessary (due to platform-specific tests and virtual test suites) and all > the tests to which these paths apply are gathered[2]. All the gathered tests > are then either sorted into a natural order or randomized (if the > --randomize-order option is used)[3]. > > I'd like to propose a change in the behavior when using the --test-list > option. When used, the tests would be gathered only from the paths listed in > the file to which the option points. Also, the gathered tests would be > sorted to follow the order of the paths in the test list file. For instance, > consider the following two entries being placed in the test list file: > > c/d/e.html > a/b/ > > The current behavior would gather all the tests to which these two paths > apply and sort them in this order (on the premise that no other paths are > passed through the command line arguments): > > a/b/1.html > a/b/2.html > c/d/e.html > > The new behavior would place the c/d/e.html test at the top of the list but > sort the other two tests in order, like this: > > c/d/e.html > a/b/1.html > a/b/2.html > > The idea behind this is to be able to specify the exact order in which the > tests should be run. I'm currently playing around with a tool that randomly > chooses a certain number of tests, runs them and bisects down the first > regression that occurs (if any) to the smallest possible ordered list of > tests that reproduces that regression. The proposed change makes this a > simple task from the test listing perspective and I'd argue that exact test > ordering is the main point of an option such as --test-list. > > Currently I'm using an untested workaround of adding a new option that > specifies the tests should be reordered back to the original order that the > test list file provided, but I'd like to move on with implementing the > change in behavior if everyone feels it's OK and won't tamper with his/her > workflow. > > -Z > > > [1] > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46 > [2] > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576 > [3] > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309 >
Hi, I have used the option from time to time, and I have also wanted from time to time the ability to run tests in a specific order. This change would be fine by me. If people want to ensure that tests run in the old order, they can always sort the test list. I would also be fine changing the code so that order is preserved for tests passed in on the command line as well. (e.g. so "run-webkit-tests foo.html bar.html" runs them in that order). I think it would be good to keep the behavior consistent across the two methods of specifying names. -- Dirk _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
