On Wed, Aug 29, 2012 at 04:22:33PM -0700, Chase Douglas wrote: > On 08/29/2012 03:36 PM, Peter Hutterer wrote: > >On Wed, Aug 29, 2012 at 01:14:35PM -0700, Chase Douglas wrote: > >>On 08/28/2012 03:57 AM, Peter Hutterer wrote: > >>>One of the things I've spent quite a bit of time on over the last weeks > >>>is a test suite. Chase tried to get some integration tests into the X > >>>server repo a while ago but I think a standalone repo is best (for now > >>>anyway). > >>> > >>>I've pushed the current set of tests to > >>>http://cgit.freedesktop.org/~whot/xorg-integration-tests/, with a > >>>lengthier explanation here: > >>>http://who-t.blogspot.com.au/2012/08/xorg-integration-test-suite.html > >>> > >>>Long story short, the xorg integration tests (XIT) are built on > >>>googletest and xorg-gtest, i.e. written in C++. Most of them write out a > >>>config, fire up a server, and then either check the log or query the > >>>server for state. A few tests use evemu devices to send events. > >>> > >>>Input testing is relatively simple with these tests since we can emulate > >>>virtually anything, but I'm not sure yet if such test cases can scale to > >>>output tests as well. Or even if there are tests that can be fully > >>>automatic without anyone staring at the screen. I know mesa already has > >>>such tests. > >>> > >>>Any feedback is appreciated, I intend to talk about this a bit more at > >>>XDC, but in the meantime you can see if it is useful. Right now, the > >>>bigger issues I'm facing are the build system and scalability if we end > >>>up with a ton of tests. And the ifdef hell that already started with > >>>tests covering different X server versions, RHEL support, etc. Any > >>>epiphanies on how to avoid a train wreck would be appreciated. > >> > >>Overall, I'm quite pleased with how things have progressed :). > >> > >>As for the run time issue, I assume you are mostly hitting it during > >>device tests. Could you find a way to set up all the input config > >>blocks you need into the same context, and then start the tests > >>against a single server? > > > >most of the tests (so far) require a specific InputDevice section that may > >or may not be CorePointer, or require some xorg.conf option to toggle the > >defaults. there are plenty of normal bug tests that don't need that, though > >then we do run the chance of having state-dependent tests. instead of > >starting with a clean slate for all of them. > > For those that require specific and different InputDevice sections, > you could use MatchProductName (or whatever it is) on the name of > the device, and use different named devices for each test. > > Normal tests should be run on the same server instance. If there is > any state dependency, that's also the sign of a bug. In the end, > we'd want to be running all of the test in a sequential order, and > then in a random order. The sequential order will be a stable test > suite, the random order will give us clues if some server state gets > messed up as we go.
the problem with random orders are that they're hard to reproduce. It doesn't help me when a random-order test fails if I can't run the tests in exactly the same order to reproduce. so we'd have to store or print the seed somewhere. Cheers, Peter _______________________________________________ [email protected]: X.Org development Archives: http://lists.x.org/archives/xorg-devel Info: http://lists.x.org/mailman/listinfo/xorg-devel
