[Development] changes to qtbase continuous integration and testing

[Simon Hausmann]

Hi,

In the past two weeks we've made some "interesting" discoveries regarding our 
test coverage, specifically regarding qtbase, but the same applies also to 
qt5.git:

  * Our networking stack is only tested on OS X 10.10. Due to what seems like 
a bug in the test planner, the tests are not fully included on the Windows 
configurations. Regarding linux, see the second bullet below.

 * We do not run _any_ tests on Linux for the dev branch of qtbase and 
qt5.git. We actually run the tests, but the results are ignored - therefore 
test failures on Linux do not block bad changes from integrating. Any change 
in qtbase that accidentally breaks for example qtscript on Linux will go into 
qt5.git undetected.

As to how it happened that we do not do testing on Linux anymore: We've had 
enforcing test coverage with our Ubuntu 12.04 configurations. Then we added 
Ubuntu 14.04 but with ignore-failures mode (called "insignificant" in the 
system). Then in the beginning of February this year we removed the 12.04 
configurations from the CI. When that happened nobody had checked if the 
Ubuntu 14.04 configurations were a proper replacement, i.e. if they would 
enforce test failures.

In addition we've had OpenSuSE and RHEL 6.6 configurations added to the CI as 
well, but those were also in ignore-failures mode, despite RHEL 6.6 being the 
configuration we're using for our binary packages. Tests were failing, but 
nobody looked at the concrete results and fixed the tests.

In the 5.4/5.5/5.5.0 branches the situation is actually similar, but meanwhile 
some fixes helped to regain some coverage.

In the future we must avoid these situations through much more thorough review 
of changes to the CI system configuration. At the same time we are not having 
this ignore-failures mode in the new CI system anymore. If some configuration 
gets added, the tests have to pass. Otherwise we cannot add it and we cannot 
claim support for a certain OS/compiler configuration.

Additionally we're taking this also as a sign to re-prioritize CI system 
related work to bring qtbase and the rest of the dev branches of qt5 modules 
over to the new CI system much sooner. I've been doing repeated builds of 
qtbase dev in the new CI and with a larger set of blacklisted tests - I'll 
send a separate email about this - and I would like to switch the module over 
this week. We will run tests on all non-cross compiling Linux configurations 
(Ubuntu 14.04, OpenSUSE 13.01 and RHEL 6.6) and if a test fails, the change 
doesn't go in. Generally we have the same matrix of os/compiler combinations 
as for the dev branch in the current CI system.

Re-prioritization means for us that we'll do this switch sooner in order to 
maintain a certain level of quality, at the expense of not having a public web 
frontend and other features in the CI system available before the switch but 
some later point in time.


There are a few more lessons to learn from this, but one step at a time :). 


Simon