The reduction in verbosity is very welcome, especially since we retain the full log in the downloads. Thank you Tor Arne!

Regarding the label time summary, it is actually different output than what is before it. For reasons I don't know (feedback welcome), we assign a LABEL to each test that includes its full path. I believe a group of test can even have the same LABEL. That is the summary of these labels.

They are assigned in cmake/QTestHelpers.cmake, function qt_internal_add_test():

    602     # Generate a label in the form tests/auto/foo/bar/tst_baz
    603     # and use it also for XML output
    604     set(label_base_directory "${PROJECT_SOURCE_DIR}")
    605     if (QT_SUPERBUILD)
    606         # Prepend repository name for qt5 builds, so that tests can be 
run for
    607         # individual repositories.
    608         set(label_base_directory "${label_base_directory}/..")
    609     endif()
    610     file(RELATIVE_PATH label "${label_base_directory}" 
"${CMAKE_CURRENT_SOURCE_DIR}/${name}")
...
    755         set_tests_properties("${testname}" PROPERTIES RUN_SERIAL 
"${arg_RUN_SERIAL}" LABELS "${label}")


Example output from CTest:

agent:2024/09/24 20:04:49 build.go:404: 100% tests passed, 0 tests failed out 
of 605
agent:2024/09/24 20:04:49 build.go:404: Label Time Summary:
agent:2024/09/24 20:04:49 build.go:404: 
tests/auto/concurrent/qtconcurrentfilter/tst_qtconcurrentfilter                 
                            =   0.77 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404: 
tests/auto/concurrent/qtconcurrentfiltermapgenerated/tst_qtconcurrentfiltermapgenerated
                     =   0.48 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404: 
tests/auto/concurrent/qtconcurrentiteratekernel/tst_qtconcurrentiteratekernel   
                            =   1.80 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404: 
tests/auto/concurrent/qtconcurrentmap/tst_qtconcurrentmap                       
                            =   1.26 sec*proc (1 test)
agent:2024/09/24 20:04:49 build.go:404: 
tests/auto/concurrent/qtconcurrentmedian/tst_qtconcurrentmedian                 
                            =   0.46 sec*proc (1 test)
...


I support removing this summary, if we don't have any particular use for the labels.


Dimitris



On Wed, 25 Sep 2024, Assam Boudjelthia via Development wrote:

This would be very nice!

Also, consider removing the "Label Time Summary:” section from the logs which 
outputs the run tests with their execution times, which is just duplicate 
information to what ctest printed before it.

— Assam


      On Sep 25, 2024, at 12:01 PM, Tor Arne Vestbø via Development 
<development@qt-project.org> wrote:

Hello,
As described in https://bugreports.qt.io/browse/QTQAINFRA-6592, we currently 
execute ctest -V (verbose), which logs all of the output from all tests to the 
resulting test log, even for passing tests.

This practice presents challenges in investigating test failures, as it 
requires loading and navigating through approximately 25MB (qtbase) of logs 
within a (mobile) web browser or editor to identify a test failure.

To address this issue, we propose implementing 
https://codereview.qt-project.org/c/qt/qtbase/+/592470, which will enable us to 
pass —output-on-failure to ctest. This modification will significantly reduce 
the ctest output and
signal-to-noise ratio.

In certain cases, it may still be beneficial to examine the full verbose log. 
Therefore, we will continue to store this information as CTest.log within the 
testresults.tar.gz for those who require a comprehensive analysis.

It is important to note that enabling —output-on-failure will not impact our 
test reporting to https://testresults.qt.io/grafana. This reporting is based on 
XML output that we still generate for each test.

An example of how this change would manifest is available here: 
https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1727203984

Let us know if you have any concerns not addressed in the corresponding bug 
reports.

Cheers,
Tor Arne 
--
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development



-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Reply via email to