Dne 12. 10. 18 v 17:58 Brian J. Murrell napsal(a): > If I run an external test tool with process.system(), say, and that > test tool produces it's own junit output, is there any way to have > avocado consume that and make it available in it's own job- > results/*/results.xml file rather than just the simplified "pass/fail" > result that it puts in there for a simple test? > > Cheers, > b. >
Hello Brian,
we've had these many times and unfortunately there is no solution so far. We've
had suggestions for allowing sub-test results (one test reporting many stages)
that could be used for that but it was rejected for simplicity reasons (one
job, many tests, no subtests)
We've had import-tap discussion as well, the problem is test should not grow.
Once you define a job, it's given, that is the current definition. Reporting
multiple results imported from tap/xml breaks this definition as test can't
report multiple test results. It'd be a level violation.
All of those discussions always finished with job-api discussion, which is the
way we intend to take, but was postponed due to avocado-qemu work. But we
intend to pursue this once we have more resources (external contributions are
also welcome, see the job-api RFCs on mailing list). The way it would work is
one would define a job, triggered it's suite that reports xml/tap and inject
those into the job results. That is not a level violation, because one creates
the job.
Anyway there is a "workaround" I'm actually using in my CI for reporting failed
libguestfs results. I run the `make check` prior Avocado execution, then
process the results and create fake tests executing "/bin/false ...":
# Now let's run the installcheck
make installcheck LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 &&
LIBGUESTFS_TESTS+=("/bin/true libguestfs_testsuite_status") ||
LIBGUESTFS_TESTS+=("/bin/false libguestfs_testsuite_status")
# Populate LIBGUESTFS_TESTS with status received from logs
for match in $(find -name test-suite.log | xargs grep "^FAIL:" | tr
' ' '_'); do
LIBGUESTFS_TESTS+=("'/bin/false libguestfs_testsuite_$match'")
done
cd ..
mkdir -p "libguestfs_testsuite_logs"
for log in $(find libguestfs -iname test-suite.log); do
mkdir -p $(dirname "libguestfs_testsuite_logs/$log")
cp "$log" "libguestfs_testsuite_logs/$log"
done
# Run the tests
avocado run --xunit "$WORKSPACE/results.xml" --job-results-dir
"$WORKSPACE" --loaders file external:"/bin/sh -c" --mux-inject
/run:timeout:7200 -- "${{LIBGUESTFS_TESTS[@]}}" "$(which libguestfs-test-tool)"
...
Is it nice? No. Does it work? Yes. And in a way it's similar to job-api
approach only it's not part of the job execution.
Regards,
Lukáš
signature.asc
Description: OpenPGP digital signature
