The reduction in verbosity is very welcome, especially since we retain the
full log in the downloads. Thank you Tor Arne!
Regarding the label time summary, it is actually different output than
what is before it. For reasons I don't know (feedback welcome), we assign
a LABEL to each test that includes its full path. I believe a group of
test can even have the same LABEL. That is the summary of
these labels.
They are assigned in cmake/QTestHelpers.cmake, function qt_internal_add_test():
602 # Generate a label in the form tests/auto/foo/bar/tst_baz
603 # and use it also for XML output
604 set(label_base_directory "${PROJECT_SOURCE_DIR}")
605 if (QT_SUPERBUILD)
606 # Prepend repository name for qt5 builds, so that tests can be
run for
607 # individual repositories.
608 set(label_base_directory "${label_base_directory}/..")
609 endif()
610 file(RELATIVE_PATH label "${label_base_directory}"
"${CMAKE_CURRENT_SOURCE_DIR}/${name}")
...
755 set_tests_properties("${testname}" PROPERTIES RUN_SERIAL
"${arg_RUN_SERIAL}" LABELS "${label}")
Example output from CTest:
agent:2024/09/24 20:04:49 build.go:404: 100% tests passed, 0 tests failed out
of 605
agent:2024/09/24 20:04:49 build.go:404: Label Time Summary:
agent:2024/09/24 20:04:49 build.go:404:
tests/auto/concurrent/qtconcurrentfilter/tst_qtconcurrentfilter
= 0.77 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404:
tests/auto/concurrent/qtconcurrentfiltermapgenerated/tst_qtconcurrentfiltermapgenerated
= 0.48 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404:
tests/auto/concurrent/qtconcurrentiteratekernel/tst_qtconcurrentiteratekernel
= 1.80 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404:
tests/auto/concurrent/qtconcurrentmap/tst_qtconcurrentmap
= 1.26 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404:
tests/auto/concurrent/qtconcurrentmedian/tst_qtconcurrentmedian
= 0.46 sec*proc (1 test)
...
I support removing this summary, if we don't have any particular use for
the labels.
Dimitris
On Wed, 25 Sep 2024, Assam Boudjelthia via Development wrote:
This would be very nice!
Also, consider removing the "Label Time Summary:” section from the logs which
outputs the run tests with their execution times, which is just duplicate
information to what ctest printed before it.
— Assam
On Sep 25, 2024, at 12:01 PM, Tor Arne Vestbø via Development
<development@qt-project.org> wrote:
Hello,
As described in https://bugreports.qt.io/browse/QTQAINFRA-6592, we currently
execute ctest -V (verbose), which logs all of the output from all tests to the
resulting test log, even for passing tests.
This practice presents challenges in investigating test failures, as it
requires loading and navigating through approximately 25MB (qtbase) of logs
within a (mobile) web browser or editor to identify a test failure.
To address this issue, we propose implementing
https://codereview.qt-project.org/c/qt/qtbase/+/592470, which will enable us to
pass —output-on-failure to ctest. This modification will significantly reduce
the ctest output and
signal-to-noise ratio.
In certain cases, it may still be beneficial to examine the full verbose log.
Therefore, we will continue to store this information as CTest.log within the
testresults.tar.gz for those who require a comprehensive analysis.
It is important to note that enabling —output-on-failure will not impact our
test reporting to https://testresults.qt.io/grafana. This reporting is based on
XML output that we still generate for each test.
An example of how this change would manifest is available here:
https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1727203984
Let us know if you have any concerns not addressed in the corresponding bug
reports.
Cheers,
Tor Arne
--
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
--
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
