[Development] Reducing CI test run verbosity

[Tor Arne Vestbø]

Hello,


As described in https://bugreports.qt.io/browse/QTQAINFRA-6592, we currently execute ctest -V (verbose), which logs all of the output from all tests to the resulting test log, even for passing tests.


This practice presents challenges in investigating test failures, as it requires loading and navigating through approximately 25MB (qtbase) of logs within a (mobile) web browser or editor to identify a test failure.


To address this issue, we propose implementing https://codereview.qt-project.org/c/qt/qtbase/+/592470, which will enable us to pass —output-on-failure to ctest. This modification will significantly reduce the ctest output and signal-to-noise ratio.


In certain cases, it may still be beneficial to examine the full verbose log. Therefore, we will continue to store this information as CTest.log within the testresults.tar.gz for those who require a comprehensive analysis.


It is important to note that enabling —output-on-failure will not impact our test reporting to https://testresults.qt.io/grafana. This reporting is based on XML output that we still generate for each test.


An example of how this change would manifest is available here: https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1727203984


Let us know if you have any concerns not addressed in the corresponding bug reports.


Cheers,

Tor Arne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20240925/1375ff4f/attachment-0001.htm>