[Development] PSA: Blacklisting test for CI-runs only
Tor Arne Vestbø
tor.arne.vestbo at qt.io
Thu Mar 23 15:55:30 CET 2017
Hi everyone,
It's now possible to blacklist tests only when run in the CI. You do
this by adding the 'ci' keyword to the normal list of keywords when
blacklisting a test, eg:
[someTestFunction]
osx-10.8 ci
See also https://bugreports.qt.io/browse/QTBUG-59564
This results in the test being blacklisted when running on OSX 10.8 in
the CI (BPASS/BFAIL), but not on local machines (PASS/FAIL). This should
typically be used for tests that are flakey, but can only be reproduced
in the CI. That way we'll detect when the test fails in local testing
and have a higher chance of fixing it.
Please consider if the 'ci' keyword should be added when blacklisting
new tests (or reviewing blacklistings). In most cases it should, as
tests that can be reproduced locally should ideally be fixed instead of
blacklisted.
Cheers,
Tor Arne
More information about the Development
mailing list