[Development] Why are qtbase integrations taking so long?

Hausmann Simon Simon.Hausmann at theqtcompany.com
Fri Apr 3 12:47:51 CEST 2015


Hi,

I believe what we are seeing is caused by instability in the network that connects the Jenkins service with the Jenkins slave machines. Occasionally network connectivity between the slaves and the master is lost, causing the running build as a whole to abort - all other still running builds are aborted and the results from builds that had already finished are discarded. In an attempt to recover, a whole new integration with builds for all configurations is started.

We have observed that this scenario repeats itself several times, causing overall integration of many hours.

As part of the work on the new CI system, we have observed similar network connectivity related symptoms. We are treating them more gracefully by not discarding otherwise successful results. Nevertheless it is a major annoyance.

Based on rumors and observation of symptoms it is a theory ‎of Frederik and I that there is a firewall service centrally installed in this virtual network. It shows symptoms of connection tracking and - more importantly - signs of being able to handle only an insufficient amount of traffic or connections. Beyond that limit, connection attempts time out and existing connections become "spotty".

I would like to get to the bottom of this at some point, because it severely affects the efficiency of the current ci system as well.

Tony, do you happen to have any more details about this?

I'll see about filing a ticket with IT next week unless we conclude anything different.

Simon

  Original Message
From: Thiago Macieira
Sent: Friday, April 3, 2015 07:11
To: development at qt-project.org
Subject: [Development] Why are qtbase integrations taking so long?


qtbase integrations used to take around 3 hours as recently as two weeks ago.

In the past week, I've caught several integrations lasting more than 6 hours.
The one currently running is integrating a single commit and has been running
for 6h30. I've seen one for 12 hours.

Is this a timeout not caught by the coordinator?

http://testresults.qt.io/ci/status/ says that it is in state "monitor-jenkins-
build" and "build_attempt: 6". For attempt 5, the only stage not to be at
SUCCESS was linux-g++_developer-build_qtnamespace_qtlibinfix_RHEL65_x64. The
same for attempts 3 and 4.
--
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

_______________________________________________
Development mailing list
Development at qt-project.org
http://lists.qt-project.org/mailman/listinfo/development


More information about the Development mailing list