[Development] Maintenance break tomorrow Apr 1st

[Tony Sarajärvi]

Update.

We are now running on the new setup. I did forget that I had to do some manual BIOS / UEFI tweaking on every single host, so it took us this time to get all up and running, but we were live at aprox 11:00 EEST.

But that didn't last long... 2 hours later our new NFS backend crashed for what it seems like overwhelming load. So we 10-20 folded the CPU and RAM it had and heavily reduced the pre-fetching of data as we set up a new host. Now it's been up and kicking since.

I hope the VMs themselves are more steady now 😊

-Tony


-----Original Message-----
From: Tony Sarajärvi 
Sent: Wednesday, April 1, 2020 7:58 AM
To: development at qt-project.org
Subject: RE: Maintenance break tomorrow Apr 1st

Good morning!

We'll be bringing the infra down now.

A piece of good news I didn't even think of yesterday. The new setup has been tested with 5 hosts and they already exist there. So the shutdown will be very brief, but at launch you will only see the 5 hosts to build with in the beginning. The schedule I presented remains for the rest of the hosts.

It starts now...

-Tony

-----Original Message-----
From: Tony Sarajärvi 
Sent: Tuesday, March 31, 2020 3:03 PM
To: development at qt-project.org
Subject: Maintenance break tomorrow Apr 1st

Hi

We’ll be bringing the OpenNebula down tomorrow morning at 8:00 EEST for an upgrade.

We’ll update OpenNebula itself to version 5.10.1 as well as an implementation of using hugepages when running virtual machines.
We’ve seen it possibly eliminating the slowness they’ve been suffering. It seems like the slowness is due to vast amounts of memory allocations in a fragmented memory space.

Also, the latest firmware’s will be installed on the hosts. The firmware install will take aprox 1h per host, but that can be done for all of them in parallel.
After that we’ll have to deploy all the hosts, and it will take some time, since we have to upload all tier1 images to them. Because of this upload we’ll do it in batches of 5-6 hosts at a time.

First ones should be coming online at around 10 o’clock, if everything goes as planned. So give it an extra 2 hours as usual 😉

For our internal employees: the parked VMs have to be nuked. Save your work!!!

-Tony