[Development] dev CI integration stuck for 3½ days

[Simon Hausmann]

The best theory I can come up with at this point looks like this:

Once there was a "newer" staging branch that contained changes that were still in 'STAGED' state but another change was in 'INTEGRATING' (which
was in the _previous_ staging branch), the conditions for an endless "loop" were met, I think. Upon any of the CI restarts, the CI thought it would
"resume" the integration of the one change that was in 'INTEGRATING' state and picked up from the latest staging branch, which had the other (two or three)
changes that were still in 'STAGED' state. When the build finished (pass or fail), the staging-approve command must have failed. It failed for me when I tried
to manually "reject" the changes - that's when I realized the discrepancy. However the change that was in 'INTEGRATING' state remained, because the staging
branch that we tried to approve/reject did not contain that change, so nothing changed there. As the state of the 'INTEGRATING' and 'STAGED' changes didn't
change, the CI didn't pick up any new integrations or try to create new staging branches.

How it is possible that a new staging branch was created while another change remained in 'INTEGRATING' state is a mystery to me. Perhaps something happened
in Gerrit at that moment? Perhaps there's a bug in the CI that allows for creating this "impossible" state?

Simon
________________________________________
From: Development <development-bounces+simon.hausmann=qt.io at qt-project.org> on behalf of Jędrzej Nowacki <jedrzej.nowacki at qt.io>
Sent: Wednesday, May 18, 2016 9:42:41 AM
To: development at qt-project.org
Subject: Re: [Development]      dev CI integration stuck for 3½ days

Yes and I fixed them all on Friday with a "catch all" command. I have no clue
how just one integration could stay locked.

Cheers,
 Jędrek

On Tuesday 17 of May 2016 12:49:17 Oswald Buddenhagen wrote:
> On Tue, May 17, 2016 at 06:08:50AM +0000, Simon Hausmann wrote:
> > I just looked into it and it looks like an inconsistency in the gerrit
> > database. The latest builds branch points to a set of changes that are in
> > staged state, while the change that is in integrating change is not in
> > that branch. I've found the build branch that had the integrating change
> > and rejected the change (and staged it again).
> yes, fregl (or nierob?) diagnosed that there was a network outage during
> the time the CI was supposed to report the result to gerrit, and the
> system apparently has no queue/retry mechanism for this. so it's out of
> sync now.
> Somebody (TM) needs to (re-)execute the relevant commands by hand ...
>
> > It appears that the change was staged Friday morning and nobody noticed it
> > during Friday. Then came a long weekend, with Monday off and Tuesday also
> > off in Norway.
> >
> > Simon
> > ________________________________________
> > From: Development
> > <development-bounces+simon.hausmann=qt.io at qt-project.org> on behalf of
> > Thiago Macieira <thiago.macieira at intel.com> Sent: Monday, May 16, 2016
> > 10:42:32 PM
> > To: development at qt-project.org
> > Subject: [Development] dev CI integration stuck for 3½ days
> >
> > Will someone PLEASE look into the qtbase/dev integration?
> >
> > https://codereview.qt-project.org/156523 has been integrating for (at the
> > time of writing this email) 84 and a half hours -- 3.5 days -- and that's
> > including two full working days in Finland, one in Germany and Norway as
> > today is is a bank holiday in those countries. I believe we'll break the
> > record by the time someone gets around to fixing this, if we haven't yet.
> >
> > Let's not hope we have to wait until after tomorrow's holiday in Norway
> > for it to get back to working.
> >
> > --
> > Thiago Macieira - thiago.macieira (AT) intel.com
> >
> >   Software Architect - Intel Open Source Technology Center
> >
> > _______________________________________________
> > Development mailing list
> > Development at qt-project.org
> > http://lists.qt-project.org/mailman/listinfo/development
> > _______________________________________________
> > Development mailing list
> > Development at qt-project.org
> > http://lists.qt-project.org/mailman/listinfo/development
>
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

_______________________________________________
Development mailing list
Development at qt-project.org
http://lists.qt-project.org/mailman/listinfo/development