[Development] CI wrap up of 2014

[Sarajärvi Tony]

Hi all!

Last day of the year and time to wrap up what has changed in the CI during the last year, and a few words about what's to come.


·         New Apple hardware was installed

o   24 x quad core mac minis with upgraded memory

o   Configured them to use our LUN instead of internal hard drives

o   With these we were able to get rid of stand-alone minis running things and replaced them with virtual machines

o   This immediately helped the situation where we were running out of hardware to run all mac configurations which caused queues to build up

·         Autotest blacklisting was introduced which has helped getting builds passing where tests are flaky

·         Lots of different platforms were added to add the variety were we test Qt on:

o   OpenSUSE

o   RHEL 6.5

o   QNX 6.5.0 and 6.6.0

o   OS X 10.10

o   Windows 8.1

o   Winphone

·         QtMetrics got new cool features like Build time graphs and detailed failure tables from autotests.

o   http://testresults.qt-project.org/qtmetrics/metricspageci.php

o   http://qt-project.org/wiki/QtMetrics_How_To_Read

·         Nightly builds are set up (builds with postfix '_state')

o   Once a day a broader set of configurations are run with all autotests run

o   Logs are published to testresults and inserted into QtMetrics

o   Doesn't block, but is only informative of the current state

·         And naturally we keept upgrading xcodes, visual studios, mesa drivers, openssl etc that's just daily maintenance work

---

To be able to cope with this increasingly complex environment we have developed something called VM-cloner. The main idea behind it is that we no longer have dozens of virtual machines up and running that are maintained with puppet scripts or similar. VM-cloner will clone the virtual machines on-the-fly from templates as is needed by any particular job. And after a build is done with them, the machines are deleted and the resources are freed.
In bullets:

·         Maintenance is reduced to template images which can be verified to work before taken into use

·         No unnecessary virtual machines occupy resources

·         Load is balanced throughout our servers enabling more efficient use of our hardware (builds are faster ;))

·         Less disk space is needed as cloned virtual machines are linked clones of the template image

·         We can now support several machine configurations simultaneously. Meaning we can build different Qt branches simultaneously without the need to reconfigure the virtual machines.

·         Possibility to not delete a machine after build if debugging is needed

·         Doesn't face Jenkins' problems with rebooting nodes

VM-cloner is already partly in use and will expand to cover more submodules in upcoming weeks. We will gradually transfer builds over to it, so that we can verify everything working.

---

We also have a "new CI" coming. This is planned to replace the very unstable Junkins. It will also optimize the building so that we will have an artifact storage where we store already built binaries and reuse them whenever possible making builds a lot faster...in theory. More about this next year.

---

We will also clean up the configuration a bit. We will move stuff from blocking CI to nightly builds and also remove old platforms. Other threads are already ongoing about this, but it's difficult to get a consensus of what should remain in the CI. We don't have resources to test everything after all.

---

Listing of dependencies. Currently we say that Qt 5.4 builds in RHEL 6.5 for example. This might not be the whole truth as you might need to install a devtoolset from a custom repository to get things working. And on top of that you might need to say -no-c++11 to configure. So personally I'd like to see a list of dependencies per submodule as they are, and not a list of operating systems the submodules work on if you modify them. Per submodule, because QtWebEngine for example will need a newer GCC version than the other modules. This doesn't prevent us from building on a specific platform and claiming we do that, but we can't say that Qt works on that platform right out of the box. Something to think about...

Qt New Year 2015!
-Tony

Tony Sarajärvi
CI Tech Lead
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20141231/ca7fb259/attachment.html>