On Sat, Jul 26, 2025 at 08:54:51AM +0200, Thomas Huth wrote: > On 25/07/2025 16.48, Daniel P. Berrangé wrote: > > On Fri, Jul 25, 2025 at 02:25:46PM +0100, Daniel P. Berrangé wrote: > > > On Fri, Jul 25, 2025 at 12:41:25PM +0300, Manos Pitsidianakis wrote: > > > > Add a CLI argument that takes fnmatch(3)-style patterns as value and can > > > > be specified many times. Only tests that match the pattern will be > > > > executed. This argument is passed to unittest.main which takes the same > > > > argument. > > > > > > > > Acked-by: Thomas Huth <[email protected]> > > > > Signed-off-by: Manos Pitsidianakis <[email protected]> > > > > --- > > > > tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++-- > > > > 1 file changed, 21 insertions(+), 2 deletions(-) > > > > > > > > > One of the goals with the new functional test system was that we stop > > > trying > > > to (re-)invent a custom test runner harness, as was the case with Avocado, > > > in favour of relying on the pre-existing python infrastructure to the > > > greatest extent possible. > > > > > > Seeing this, and all the other CLI arg handling added in this series, > > > makes > > > me fairly uncomfortable, as it is effectively inventing a custom test > > > runner > > > once again which is exactly what we wanted to get away from. > > > > > > At the same time, there are some pieces in this series that do things that > > > unittest.main() can't do on its own. > > > > So considering the broader picture, we already have a load of tunables > > on the test execution that we control exclusively via envirnoment > > variables, as that gives us independance of the test runner, which > > owns sys.argv processing. > > > > So in terms of this series, IMHO, we should just add support for > > QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables, > > and not touch sys.argv at all. > > I basically agree, but I wonder if we could maybe also have both? First > parse the sys.argv for our parameters, then pass the unknown ones to > unittest.main() ?
Any argv that we process are subject to clashing with that defined by the test harness and it is desirable to avoid making assumptions about what test harness args exist. To avoid that they would all have to be long options named with a fixed prefix --qemu-XXXX. Also that sould be something applied to all env vars, not just the couple of params in this series. > > With that, the only thing we're missing is a way to enumerate the > > test cases, but IMHO that's the job of the test runner and thus > > out of scope for QEMU to solve. > > With the patch applied that passes the argv to unittest.main(), there also > seems to be a "discover" mode: > > ~/devel/qemu/tests/functional/test_vnc.py -h | grep usage: > usage: test_vnc.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b] > usage: test_vnc.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f] > > ... however, when I try to run it, it does not work as expected (I get a > "ModuleNotFoundError: No module named 'discover'" error). Hmm, not sure that the 'discover' option makes sense when we're asking to run a single test file. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
