[Development] Automatic bug reporting for top flaky test in CI

[Anna Wojciechowska]

Hello,

I guess we are all frustrated by the test flakiness

Starting today top 5 cases will be added to Jira bug reports weekly, and those reports will be labeled as "top flaky".

Below you will find links to already already-created report for this week:
https://bugreports.qt.io/browse/QTBUG-119429

https://bugreports.qt.io/browse/QTBUG-119430


Fixing them will improve the stability of the CI.
I understand that correcting flaky tests is not an easy task - in some cases, rewriting can be the solution.
If the test is not supposed to be run on a given platform - please use QSKIP macro functionality. Blacklisting should be the last resort and used only for short-term test disabling.

You can read more about the test system here<https://wiki.qt.io/Qt_test_system>, or more about flakiness data here.<https://wiki.qt.io/Flakiness_information>

We will analyze the effort and inform you about the results.


Good luck!

Anna Wojciechowska


ps

In a rare case if a false report would be created, just feel free to close it as invalid.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20231124/eb1d9063/attachment.htm>