[Development] qtbase CI now takes 6 hours

[Stephen Kelly]

Hi there,

Adding to the other major problem of CI (non-deterministic results), CI in 
qtbase now takes 6 hours. See the most recent successful build here: 

 https://codereview.qt-project.org/#change,55977

and mouse-over the date to get the timestamps.

As a reminder, you can see all CI reports here:

 http://thread.gmane.org/gmane.comp.lib.qt.ci-reports

As you can see, things fail very often. If you look into the failures, you can 
see that many many of them do not relate to the patches under test (they are 
flaky, or the test machines hit a problem etc).

I recommend that anyone who can do anything about the CI keep track of it by 
subscribing to the reports mailing list:

 http://lists.qt-project.org/mailman/listinfo/ci-reports

It is very easy to see if failures are related to the patch under test or not, 
and then that can be reacted to. The reaction might be to mark a test as 
insignificant, or to mark a broken platform as having insignificant tests (as 
was done recently for macx-clang_developer-build_qtnamespace_OSX_10.7 for the 
qt5.git repo).

I really don't know if 'the CI people' are already tracking failures like 
that. The CI really is a mess these days, and I get the feeling that it's not 
something anyone is tracking or systematically trying to fix. I could notify 
about individual problems, but a) they should be so obvious that I don't have 
to and b) that doesn't solve the systematic problems. 

Someone who can get more information and actually solve the problems needs to 
be on top of it without being notified. I have no way of finding out why 
integrations suddenly take 6 hours instead of 2. I don't have the karma to 
mark macx-clang_developer-build_qtnamespace_OSX_10.7 or anything else as 
insignificant (or significant). I don't have any way to get the information 
that a widgets test is failing because the widget is bigger than the 'screen 
size' used by the vm (eg https://codereview.qt-project.org/51289). But someone 
must have access to all that information and karma, and they should be on top 
of the CI failures (by following the above mailing list) without being 
notified by me or someone else.

Currently qtwebkit stable can't be merged to dev, which means qt5.git dev 
branch can't be updated at all. We need a better way to track issues like 
that. It is important that qt5.git can be updated. So, I think submodule 
merges and qt5.git integrations should be of particular importance for any 'CI 
people' tracking the reports mailing list.

If such tracking is not possible by the people with more-direct access to the 
CI system, or no such person exists who can track that stuff, then we need a 
better way to track these failures as a community. We at least need to all be 
aware that there is no person who is going to take responsibility for CI 
issues like that, and it's up to everyone else to 'scramble and do what they 
can to fix it', using publically available information. If that's the best 
we're going to be able to do, then we should know that. 

I also recommend adding the duration of an integration run to the mails sent 
to the reports mailing list. That way, the mailing list is a 'one stop shop' 
for seeing when CI is having systematic or unusual problems.

Thanks,

-- 
Stephen Kelly <stephen.kelly at kdab.com> | Software Engineer
KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company
www.kdab.com || Germany +49-30-521325470 || Sweden (HQ) +46-563-540090
KDAB - Qt Experts - Platform-Independent Software Solutions
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3636 bytes
Desc: not available
URL: <http://lists.qt-project.org/pipermail/development/attachments/20130516/a034422d/attachment.bin>