Hi Tomas, Tomas Volf <[email protected]> writes:
[...] > For sure, this is documented in: > > (automake)Log files generation and test results recording > > Quoting the relevant parts: > > What happens when two or more ‘:recheck:’ fields are present in the > same ‘.trs’ file is undefined behavior. > > What happens when two or more ‘:copy-in-global-log:’ fields are > present in the same ‘.trs’ file is undefined behavior. > > What happens when two or more ‘:test-global-result:’ fields are > present in the same ‘.trs’ file is undefined behavior. [...] > The following is a valid test files, and it has 2 outermost test-ends. > > (test-begin "foo") > (test-begin "xx") > (test-end) ;not an outermost > (test-end) ;first outermost > > (test-begin "bar") > (test-end) ;second outermost > > The SRFI-64 mandates that test-runner-on-final will, on this specific > file, be invoked twice. Thank you for the reference and explanation! >>> * build-aux/test-driver.scm (test-runner-gnu): Define new procedure >>> `finalize' >>> and return it together with the runner. Do not call >>> test-runner-on-group-end!. >>> (main): Call the `finalize' after all tests are done. >> >> Applied, and tested with the same example above. It seems to work >> great, thank you. >> >> By the way, I had to come up with a small workaround for your new SRFI >> 64 implementation that landed in Guile 3.0.11 (now available in Guix); >> see commit 5fada9a751f as well as upstream issue >> <https://codeberg.org/guile/guile/issues/133>. > > Thank you for the ping. It is some time since I implemented it, so my > memory is bit fuzzy, but looking at the code and SRFI, I do not believe > this is a bug. Not having a test on the run list is not the same as > skipping it. > > The specification says the following for the `Result kind', emphasis > mine: > > *Running* a test may yield one of the following status symbols: > > When the test is not on the run list (== list of tests to run), it was > never *ran* (== the action of *running* was never performed on it), so > the kind is not supposed to be set. At least per my reading of the > specification. > > If the kind would be set to 'skip even in the case of the test was not > on the run list, there would be no way to tell whether test was skipped > due to skips, or not ran due to not being requested via run list, which > would quite limit the options of custom test runners (they need to be > able to tell the difference). Hm, OK. That's definitely a breaking change from the previous behavior, but perhaps it's a good one. But I must say it seemed intuitive to me that if I use a specifiers to run just a selection of the tests, that implicitly means I want the other tests to not run, which is the same as requesting them to be skipped. > I would appreciate if you could add the above to the Guile's bug report, > I do not have Codeberg account, so I cannot react there. Done; please consider joining us there directly :-). -- Thanks, Maxim
