On Mon, Oct 29, 2012 at 4:17 AM, Maciej Stachowiak <[email protected]> wrote: > > On Oct 28, 2012, at 3:30 PM, Antti Koivisto <[email protected]> wrote: > >> We could clear the cache between tests but run each test twice in a row. >> Second run will then happen with deterministically pre-populated cache. That >> would both make things more predictable and improve our test coverage for >> cached cases. Unfortunately it would also slow down testing significantly, >> though less than 2x. > > I actually really like this idea. Doing it this way would effectively run > each test both completely uncached, and fully cached, which would be better > test coverage than our current approach. Can we get an estimate on what this > would cost if applied to our whole test suite? Could we do it for just a > subset of the tests? > > (BTW I think this is better than the "virtual test suite" approach suggested > by Dirk; running the test with all its resources cached from having loaded it > immediately before is more reliable and better test coverage than running it > later as part of some sequence that doesn't clear the cache.) > > Does anyone strongly object to this approach? It seems way better to me than > other options discussed on this thread. >
I would like to understand the proposal and thinking behind it a little better. First, I'm not quite sure what the intent is here ... are you thinking that maybe this would help ensure tests get more isolated environments and yet we still exercise the cache all the time? Second, today you could get close by saying NRWT --repeat-each=2 --batch-size=2; that would run each test twice in succession and then restart DRT. Are you suggesting that we'd change something such that we'd clear the cache rather than restarting DRT? Third, it seems like it would probably introduce a *lot* of extra redundant test-running. I would imagine the very large majority of our tests are not sensitive to the cache contents at all and so running them twice doesn't buy you anything. An advantage of the virtual test suite approach is that you can specify which subsets of tests you'd like to run; perhaps you could combine that with running them twice? Lastly, it seems like this is focusing on the wrong thing; wouldn't it be better to try and write specific tests to provoke different configurations and code paths through the cache? Are you suggesting we'd run all the tests twice as an interim measure until we had better specific test suites? -- Dirk > Regards, > Maciej > _______________________________________________ > webkit-dev mailing list > [email protected] > http://lists.webkit.org/mailman/listinfo/webkit-dev _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
