[Development] PSA: Blacklisting test for CI-runs only

[Simon Hausmann]

To emphasize on aspect of this PSA: This only works for tests in Qt 5.9 and onwards (once qtbase merged), not for older branches of Qt.




Simon

________________________________
From: Development <development-bounces+simon.hausmann=qt.io at qt-project.org> on behalf of Tor Arne Vestbø <tor.arne.vestbo at qt.io>
Sent: Thursday, March 23, 2017 3:55:30 PM
To: development at qt-project.org
Subject: [Development] PSA: Blacklisting test for CI-runs only

Hi everyone,

It's now possible to blacklist tests only when run in the CI. You do
this by adding the 'ci' keyword to the normal list of keywords when
blacklisting a test, eg:

[someTestFunction]
osx-10.8 ci

See also https://bugreports.qt.io/browse/QTBUG-59564

This results in the test being blacklisted when running on OSX 10.8 in
the CI (BPASS/BFAIL), but not on local machines (PASS/FAIL). This should
typically be used for tests that are flakey, but can only be reproduced
in the CI. That way we'll detect when the test fails in local testing
and have a higher chance of fixing it.

Please consider if the 'ci' keyword should be added when blacklisting
new tests (or reviewing blacklistings). In most cases it should, as
tests that can be reproduced locally should ideally be fixed instead of
blacklisted.

Cheers,
Tor Arne

_______________________________________________
Development mailing list
Development at qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20170323/6c9f30d6/attachment.html>