On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
> > if [ -n "$retrieve_xml" ]; then
> > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> > + # Findout if one of the test failed
> > + if ! grep -q '</testsuites>' tests-junit.xml; then
> > + echo "ERROR: tests-junit.xml is incomplete or missing."
> > + TEST_RESULT=1
> > + elif grep -q '</failure>' tests-junit.xml; then
> > + TEST_RESULT=1
> > + fi
> > fi
> >
> > exit "$TEST_RESULT"
>
> A couple of things.
>
> From my experimentation with junit,
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
> we can also use </error> for classification. I'm also very disappointed
> in Gitlab classifying <warning> as success.
According to the documentation [1] which point to this junit xml format [2]
the only elements (and path) are:
testsuites.testsuite.testcase.failure
"error" or "warning" don't exist.
There's the attributes `type` in <failure> but this isn't explained how
it's used.
But I guess if we follow the link in [2], go through web.archive.org, we
can find [3] which has "skipped", "error", "failure", but still no
"warning".
[1]
https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow
[2]
https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format
[3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd
> Not for this patch, but for XTF I need to be able to express "tolerable
> failure". (All branches of Xen will run the same tests, and we don't
> have OSSTest to deem "fail never passed" as non-blocking.)
According to [1], there's a notion of "Existing failures", but that
might show up only on merge request.
> Even if the job passes overall, I want tolerable failures to show up in
> the UI, so I have to use <failure> in junit.xml. But that means needing
> to be more selective, and I don't have a good idea of how to do this.
> (I have one terrible idea, which is </failure type=tolerable"> which
> will escape that grep, but it feels like (ab)buse of XML.)
At the moment, `run-tools-tests` write '<failure type="failure"' on
failure, so we could grep on that instead event if it is sligtly more
fragile. I've choosen to grep on '</failure>' at first because that's
much less likely to be written differently, while the attributes in the
tag could be written in a different order.
Then, we can always use `sed` and extract the "type" to check it:
sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml |
while read type; do
case $type in
failure) echo fail;;
tolerable) echo ok;;
*) echo error unknwon type $type;;
esac
done
But that maybe going a bit too far :-)
Cheers,
--
Anthony PERARD
