[Development] Flaky auto tests on macOS 10.12

[Morten Sørvig]

> On 19 Jun 2017, at 08:34, J-P Nurmi <jpnurmi at qt.io> wrote:
> 
> Hi all,
> 
> After a few successful qt5.git CI rounds, auto tests were enabled for macOS 10.12 in the CI last week. [*] Even though many flaky auto tests had been already blacklisted on macOS 10.12 earlier (https://bugreports.qt.io/browse/QTBUG-58968), it became nearly impossible to integrate any qtbase patch due to random timer/eventloop/animation related auto test failures on macOS 10.12. As agreed earlier (http://lists.qt-project.org/pipermail/development/2017-February/028715.html), an action was taken to blacklist/skip several more auto tests on macOS 10.12:

I went through the list and most of these are not reproducible locally. (see
list below for details). This matches the pattern we’ve been seeing for some
time with CI-only failures.

So I think the priority for the reports should be reduced once the tests have
been blacklisted and they are no longer blocking development, due to them being
hard to reproduce as well as being less likely to be hit by users.

Morten


> 
> Extend blacklisting of tst_QElapsedTimer::elapsed to cover macOS 10.12
> - https://bugreports.qt.io/browse/QTBUG-58713
> - https://codereview.qt-project.org/#/c/195625/

Not reproducible.

> 
> Blacklist tst_QGuiEventLoop::processEvents in macOS 10.12
> - https://bugreports.qt.io/browse/QTBUG-61131
> - https://codereview.qt-project.org/#/c/196087/

Reproducible. Looks like 10.12 is sending an extra event at startup
which the test fails to account for (see QTBUG-61131 for details).

> 
> Blacklist flaky tst_QTimeLine tests on macOS 10.12
> - https://bugreports.qt.io/browse/QTBUG-61037
> - https://codereview.qt-project.org/#/c/195608/

Not reproducible.

> 
> Skip unreliable tst_QTimer::moveToThread() on macOS 10.12
> - https://bugreports.qt.io/browse/QTBUG-59679
> - https://codereview.qt-project.org/#/c/197769/

Not reproducible (however, warnings).
QWARN  : tst_QTimer::moveToThread() QObject::killTimer: Timers cannot be stopped from another thread
QWARN  : tst_QTimer::moveToThread() QObject::~QObject: Timers cannot be stopped from another thread

> 
> Blacklist flaky tst_QGuiEventLoop::testQuitLock() on macOS 10.12
> - https://bugreports.qt.io/browse/QTBUG-61499
> - https://codereview.qt-project.org/#/c/197791/

Not reproducible.

> 
> Blacklist tst_QParallelAnimationGroup::deleteChildrenWithRunningGroup()
> - https://bugreports.qt.io/browse/QTBUG-61500
> - https://codereview.qt-project.org/#/c/197792/

Not reproducible.