[Development] Flaky auto tests on macOS 10.12

[J-P Nurmi]

Hi all,

After a few successful qt5.git CI rounds, auto tests were enabled for macOS 10.12 in the CI last week. [*] Even though many flaky auto tests had been already blacklisted on macOS 10.12 earlier (https://bugreports.qt.io/browse/QTBUG-58968), it became nearly impossible to integrate any qtbase patch due to random timer/eventloop/animation related auto test failures on macOS 10.12. As agreed earlier (http://lists.qt-project.org/pipermail/development/2017-February/028715.html), an action was taken to blacklist/skip several more auto tests on macOS 10.12:


Extend blacklisting of tst_QElapsedTimer::elapsed to cover macOS 10.12
- https://bugreports.qt.io/browse/QTBUG-58713
- https://codereview.qt-project.org/#/c/195625/

Blacklist tst_QGuiEventLoop::processEvents in macOS 10.12
- https://bugreports.qt.io/browse/QTBUG-61131
- https://codereview.qt-project.org/#/c/196087/

Blacklist flaky tst_QTimeLine tests on macOS 10.12
- https://bugreports.qt.io/browse/QTBUG-61037
- https://codereview.qt-project.org/#/c/195608/

Skip unreliable tst_QTimer::moveToThread() on macOS 10.12
- https://bugreports.qt.io/browse/QTBUG-59679
- https://codereview.qt-project.org/#/c/197769/

Blacklist flaky tst_QGuiEventLoop::testQuitLock() on macOS 10.12
- https://bugreports.qt.io/browse/QTBUG-61499
- https://codereview.qt-project.org/#/c/197791/

Blacklist tst_QParallelAnimationGroup::deleteChildrenWithRunningGroup()
- https://bugreports.qt.io/browse/QTBUG-61500
- https://codereview.qt-project.org/#/c/197792/


Quoting Lars:
"
Anybody who get’s one of those bug reports assigned please look at them immediately and try to see what’s going wrong. This is not always easy or straightforward, as they aren’t always reproducible outside the CI system. Here are a few pointers to help:

Have a look at https://wiki.qt.io/Writing_good_tests , and check that the test is following the rules there. Often tests fail more easily under high load, so this is something to check as well. If you’re working for The Qt Company, you have the additional option of creating a VM inside the CI system or running test builds of a pushed change in the CI system. If you’re not working for TQtC, ask someone who does and we can schedule a build of any patch you push to gerrit inside the CI on the platform of your choice (with results being reported back to the gerrit change).

If the test is flaky due to some external dependency (e.g. the network test server), you might want to file them as a subtask for getting a better test server in place and keep it blacklisted. In almost all other cases, you should try to fix the test (or the bug in Qt). If it's not possible to fix the test, think about how it could be rewritten. If the test is worthless (for example because it doesn’t test anything we haven’t covered in other ways), remove it.

In any case, please handle those bug reports quickly, as said above they will block the release until handled. Please don’t down-prioritize these bugs reports without very good reasons (and talking to the module maintainer).

I hope that as many people as possible will help in the effort. Fixing those flaky tests is quite some work right now, but in the longer term we will benefit us all when integrations go in more smoothly and we can more easily update qt5.git and get releases out.
"

[*] The timing, during the 5.9.1 soft freeze week, was a bit unfortunate, but this is a separate discussion.

--
J-P Nurmi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20170619/f8ba78ec/attachment.html>