[Development] CI problems

[Tony Sarajärvi]

Better new this time. Aprox 3 hours ago I got green light to start the CI again. An hour into the run I stumbled upon a non-infra related small issue, which got fixed by a simple quick restart. After that builds have been running OK and as I saw first green results and commits being merged I felt confident to send this e-mail. So, all seem to be working again. (Fingers crossed)

We have extra pair of eyes on the system this weekend, and I’ll keep you informed if it goes to a halt again.

Thanks for your patience
-Tony

From: Development [mailto:development-bounces+tony.sarajarvi=qt.io at qt-project.org] On Behalf Of Tony Sarajärvi
Sent: perjantaina 19. toukokuuta 2017 12.17
To: development at qt-project.org
Subject: Re: [Development] CI problems

Hi again ☹

Something hit the fan again, and all CI systems came to a halt. IT has been contacted who will contact Dell immediately. I’ll keep you updated as I have receive info.

-Tony

From: Tuukka Turunen
Sent: perjantaina 19. toukokuuta 2017 0.54
To: Tony Sarajärvi <tony.sarajarvi at qt.io<mailto:tony.sarajarvi at qt.io>>; development at qt-project.org<mailto:development at qt-project.org>
Subject: Re: [Development] CI problems


Awesome. Thanks, Tony (and others who worked to get this resolved quickly)!

Yours,

                Tuukka

From: Development <development-bounces+tuukka.turunen=qt.io at qt-project.org<mailto:development-bounces+tuukka.turunen=qt.io at qt-project.org>> on behalf of Tony Sarajärvi <tony.sarajarvi at qt.io<mailto:tony.sarajarvi at qt.io>>
Date: Thursday, 18 May 2017 at 22.46
To: "development at qt-project.org<mailto:development at qt-project.org>" <development at qt-project.org<mailto:development at qt-project.org>>
Subject: Re: [Development] CI problems

(Apparently it was just my own inbox not being updated 😉)

It now looks like we have a running CI again!
I started Coin and everything seems to be running as normal. I’ll keep watching it for a while longer and then resume work in the morning. Thanks for your patience.

-Tony

From: Development [mailto:development-bounces+tony.sarajarvi=qt.io at qt-project.org] On Behalf Of Tony Sarajärvi
Sent: torstaina 18. toukokuuta 2017 20.17
To: development at qt-project.org<mailto:development at qt-project.org>
Subject: [Development] CI problems

Hi

Seems my earlier 2 mails haven’t reached you. I hope this one does.

We’ve suffered hardware malfunction today. Our CI environment began behaving oddly aprox 20 hours ago and early this morning we noticed that not everything was OK. To prevent data loss and further problems, we shut down the CI.

It took a lot of time to narrow the problem down, and we still aren’t 100% sure where the problem is, but we have a good suspect currently. We have 24/7 teams on this issue trying to get everything fixed.

Bear with us. We’ll be back as soon as possible.

-Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20170519/5b77c044/attachment.html>