[Development] PSA: Blacklisting test for CI-runs only

Tor Arne Vestbø tor.arne.vestbo at qt.io
Thu Mar 23 15:55:30 CET 2017


Hi everyone,

It's now possible to blacklist tests only when run in the CI. You do 
this by adding the 'ci' keyword to the normal list of keywords when 
blacklisting a test, eg:

[someTestFunction]
osx-10.8 ci

See also https://bugreports.qt.io/browse/QTBUG-59564

This results in the test being blacklisted when running on OSX 10.8 in 
the CI (BPASS/BFAIL), but not on local machines (PASS/FAIL). This should 
typically be used for tests that are flakey, but can only be reproduced 
in the CI. That way we'll detect when the test fails in local testing 
and have a higher chance of fixing it.

Please consider if the 'ci' keyword should be added when blacklisting 
new tests (or reviewing blacklistings). In most cases it should, as 
tests that can be reproduced locally should ideally be fixed instead of 
blacklisted.

Cheers,
Tor Arne




More information about the Development mailing list