[Development] Qt CI reliability

[Tuukka Turunen]

Hi Sean,

Firstofall, I do apologize for the inconvenience caused by the CI system. We are fully aware of the situation, and the effect is has for productivity. All the developers of The Qt Company are using exactly the same CI system. 

To address the problems, we had with Jenkins based CI we started to develop new CI system built from the ground up to serve the needs of Qt. Unfortunately, it has taken us longer than we anticipated to get the new CI system stabile, and there are still a lot of failures caused by the CI itself. We are also continuously improving the test asset to make it less prone for errors, including identification of flaky cases and fixing and/or temporarily blacklisting these. 

While we unfortunately do not have 24x7 sysadmins, we do have persons dedicated to operate the CI system, as well as support from IT as needed for the infrastructure. In addition, we are still putting a significant effort into developing the CI further and stabilizing it. And, yes, we are also continuously monitoring the infrastructure of the CI systems as well as planning how to improve it further in the future.

Yours,

	Tuukka

> -----Original Message-----
> From: Development [mailto:development-
> bounces+tuukka.turunen=qt.io at qt-project.org] On Behalf Of Sean Harmer
> Sent: lauantaina 30. huhtikuuta 2016 22.26
> To: development at qt-project.org
> Subject: [Development] Qt CI reliability
> 
> Hi,
> 
> after yet another 5 hour wait just to be greeted with yet another random
> failure with no build logs I'm getting really tired of the poor reliability of the
> Qt CI system.
> 
> https://codereview.qt-project.org/#/c/157590/
> 
> has been greeted with genuine failures, failures in qtdeclarative,
> qtxmlpatterns, multiple random failures in qt3d despite being a simple
> change which I suspect are due to issues on one or more CI nodes.
> 
> I appreciate the new CI is more efficient than the previous implementation
> but it still has a very high rate of failed integration attempts. For a piece of
> infrastructure that is critical to the Qt Project the CI system should have
> 24x7 sysadmin support. I'm not expecting devs to be doing this. The Qt
> Company has sys admins supported by license fees yet we see example after
> example of the support being substandard. Certificates expiring, disks filling
> up, upgrades being performed to coincide with feature freezes, upgrades
> performed then going home for the weekend leaving systems in an unfit
> state.
> 
> Such long waits resulting in random failures is killing both productivity and
> motivation. This whole week seems to have been nothing other than an
> endless cycle of staging and re-staging. We're doing our best to get Qt 3D into
> shape for Qt 5.7 but this is being hamstrung by the infrastructure.
> 
> I don't understand how sysadmins can be caught out by disks filling up
> without them being notified. Why is there no server monitoring in place?
> Likewise for the numerous SSL certificate expirations over the last couple of
> years.
> 
> Now being in the same situation of seemingly endless failures over a holiday
> weekend, it seems my backlog will continue to get longer whilst my will
> power to dedicate my personal time to meeting the release deadline so that
> TQC sysadmins can go on their summer vacations wanes.
> 
> I would be very grateful if anybody happens to read this and is available to
> look at the current CI issues. Please can TQC management dedicate
> resources to sorting out the underlying resource shortages for administering
> the CI, putting on place some reliable server monitoring that actually gets
> acted upon and giving the project the system it should have.
> 
> Thank you and enjoy what is left of the holiday weekend.
> 
> Sean
> --
> Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
> Klarälvdalens Datakonsult AB, a KDAB Group company Tel. UK +44 (0)1625
> 809908, Sweden (HQ) +46-563-540090 KDAB - Qt Experts - Platform-
> independent software solutions
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development