[Development] QT infra weekly report

[Tony Sarajärvi]

Hi

We’ve decided to start sharing a bit more information about the state of our infra and what we’re up to. This will be a brief e-mail that will cover the main topics. 

State of the CI
- We've been running 2 weeks now with a stable kernel. The hosts have actually been up without reboots now for 2 weeks, which must be a record for us. This means, we won't be touching them now as they work. 

Update of OpenNebula
- 5.6 has been installed. Scripting it will require still some work, as the same deployment procedure doesn’t work right of the bat. Will be tested in the dev environment.

Update of CI hosts to Ubuntu 18.04 and upgrade MAAS
- Current MAAS version doesn't support Ubuntu 18.04, so we need to update MAAS.
- A newer version of MAAS requires it to run on Ubuntu 18.04. So we need to update the host of MAAS as well (no chicken and the egg problem here).
- This all lead us to need a dev/sandbox environment with all new VLAN tags. That's done now and we can now test everything before breaking production (again).
- One host has been allocated into the dev environment (So CI is down one from 33 we have in total)

Mac hardware in the CI
- Currently OpenNebula doesn’t have Mac hardware attached to it. A Mac Pro could be attached to it for testing, but we don’t have a PDU (power distribution unit) for it yet. This needs some further planning what we really want to do with these.
 
Our packet loss situation
- We’ve had problems with odd packet loss. We suspected that our firewall was to blame, and surely it hit high peaks in load.
- QoS was disabled, as manufacturer told us it will increase load on the firewall by 20-50%
- Packet loss was reduced from 5-8% down to 0.5%.
- If we can pin point the culprit of the remaining 0.5% to be the firewall, we might get a budget to replace those firewalls ASAP (work ongoing).

Our Blade infrastructure is going to have a backup Blade allocated to the pool currently running our infra. It needed more RAM however, and the order has been placed.

Coin’s server will be allocated terabytes of disk space from the Compellent to cope with vast amount of builds produced by product repo development.
This became an issue as the Compellent is getting quite full, so we’re currently dumping old pre Qt 5.6 images into USB drives for disposal (read: storage).

You might have seen issues with Licenses in the INTEGRITY builds. These have now been resolved, but if you see more of these or have any license related questions, contact mailto:it.helpdesk at qt.io and CC andrey.leman at qt.io who takes care of them and mailto:qt.ci at qt.io (that’s us) as well.  Naturally we keep an eye on builds as well, but quite often is the case such that people staging the commits see the problem right after the failure and not the next time we glance Coin's front page.


This first e-mail had some longer back stories attached to the points above. In the future I'll try to be more brief, as you all will know the back stories from this e-mail 😊

-Tony
----
Click _here_ if you wish to unsubscribe.