Hi Karl, On Wed, 19 Aug 2020 19:17:13 -0600 Karl Berry <k...@freefriends.org> wrote:
> Hi Glenn, > > I noticed that when specifying a subset of tests to run via the > TESTS environment variable, I could have all tests complete > successfully > > Thanks for the report. That does seem suboptimal, but on the other > hand, subsets of tests are a complication. Does it work to override > SUBDIRS along with TESTS? As in (thanks to Jim): > env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS= Yes this does work in the special case of grub, thanks for the suggestion. I think at a minimum this behavior should be documented with the test subsets documentation and this work around mentioned. Ideally, the subset tests feature design should be updated to play nice with makefile recursion. Setting SUBDIRS= is just disabling the makefile recusion. If I had a test in grub-core/unittest1 and I wanted to run just the test subset of unittest1 and grub_cmd_echo, how would I do that such that make returns success? If I run `TESTS="unittest1 grub_cmd_echo" make -e check', make will fail even if both test individually succeed because the recursive make check in grub-core will try to run grub_cmd_echo (in addition to unittest1) and fail. Another issue is that if there is a test X in the root makefile and a test X in SUBDIR, how does one choose to run the X in SUBDIR and not the X in root? Right now TESTS=X will run both tests. Sure one could do `TESTS=X make -e -C SUBDIR check', but then again you're just bypassing the makefile recursion. And thanks Karl for helping to maintain this venerable and essential, yet not super glamorous project. Glenn
