[Development] Qt CI reliability

[Sean Harmer]

Hi,

after yet another 5 hour wait just to be greeted with yet another random 
failure with no build logs I'm getting really tired of the poor reliability of 
the Qt CI system.

https://codereview.qt-project.org/#/c/157590/

has been greeted with genuine failures, failures in qtdeclarative, 
qtxmlpatterns, multiple random failures in qt3d despite being a simple change 
which I suspect are due to issues on one or more CI nodes.

I appreciate the new CI is more efficient than the previous implementation but 
it still has a very high rate of failed integration attempts. For a piece of 
infrastructure that is critical to the Qt Project the CI system should have 
24x7 sysadmin support. I'm not expecting devs to be doing this. The Qt Company 
has sys admins supported by license fees yet we see example after example of 
the support being substandard. Certificates expiring, disks filling up, upgrades 
being performed to coincide with feature freezes, upgrades performed then 
going home for the weekend leaving systems in an unfit state.

Such long waits resulting in random failures is killing both productivity and 
motivation. This whole week seems to have been nothing other than an endless 
cycle of staging and re-staging. We're doing our best to get Qt 3D into shape 
for Qt 5.7 but this is being hamstrung by the infrastructure.

I don't understand how sysadmins can be caught out by disks filling up without 
them being notified. Why is there no server monitoring in place? Likewise for 
the numerous SSL certificate expirations over the last couple of years.

Now being in the same situation of seemingly endless failures over a holiday 
weekend, it seems my backlog will continue to get longer whilst my will power 
to dedicate my personal time to meeting the release deadline so that TQC 
sysadmins can go on their summer vacations wanes.

I would be very grateful if anybody happens to read this and is available to 
look at the current CI issues. Please can TQC management dedicate resources to 
sorting out the underlying resource shortages for administering the CI, 
putting on place some reliable server monitoring that actually gets acted upon 
and giving the project the system it should have.

Thank you and enjoy what is left of the holiday weekend.

Sean
--
Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
Klarälvdalens Datakonsult AB, a KDAB Group company
Tel. UK +44 (0)1625 809908, Sweden (HQ) +46-563-540090
KDAB - Qt Experts - Platform-independent software solutions