[Development] Building additional components with Conan for Qt 6

[Toni Saario]

> Most of provisioning scripts fall into 2 categories:
> 1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.

Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with their versions in one place.

[1] https://docs.conan.io/en/latest/devtools/create_installer_packages.html#create-installer-packages
That is what the automatic documentation tools are asking. Might be stupid questions due to the knowledge. We would install all dependencies during the provisioning and during the builds conan only exposes dependencies listed in the conanfile? One thing that conflicts a little bit here is that we want to use cached stuff from our servers instead of downloading from the internet in the provisioning and also have the provisioning to be runnable by anyone, but i guess that if the conan caches all the packages then it equals pretty much the same.
>> As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.
>
> Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).

In this case each testing activity should have its own conanfile with a list of dependencies, as well as each Qt project which uses them for building.
Ultimately where would all the conanfiles sit? Currently we only use qt5 repository during provisioning and with conan each repo would have its own conanfiles which needs to be read when running provisioning? How big maintenance burden those conanfiles become, are common files across repos possible?

I do not see why it would be impossible to use conan in provisioning, the transition is quite big effort though, and some cherry-picks need to be translated to the old format afterwards. Theres also many other things I cannot answer now related to artifactory and any legal sides to distributing the packages etc. There seems to be a lot of research into conan by Iikka, but i cannot find any related to building all the qt modules with it. I am sure it requires developer attention also, related to the CMake changes that needs to be done. It has a low priority now. https://bugreports.qt.io/browse/QTBUG-74397

________________________________
From: Konstantin Tokarev <annulen at yandex.ru>
Sent: Wednesday, October 7, 2020 6:44 PM
To: Toni Saario <toni.saario at qt.io>; Iikka Eklund <iikka.eklund at qt.io>; Richard Weickelt <richard at weickelt.de>; development at qt-project.org <development at qt-project.org>
Subject: Re: [Development] Building additional components with Conan for Qt 6



02.10.2020, 16:20, "Konstantin Tokarev" <annulen at yandex.ru>:
> 02.10.2020, 15:53, "Toni Saario" <toni.saario at qt.io>:
>>  One thing to check is that how much the provisioning it is possible to translate to Conan, I would believe that in the end it does not work for everything.
>
> Most of provisioning scripts fall into 2 categories:
> 1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.

Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with their versions in one place.

[1] https://docs.conan.io/en/latest/devtools/create_installer_packages.html#create-installer-packages

>>  As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.
>
> Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).

In this case each testing activity should have its own conanfile with a list of dependencies, as well as each Qt project which uses them for building.

>
> I can see another possible way to use Conan in CI - install packages into the system via "deploy" generator so that Qt build system can find them without any knowledge of Conan. I guess this way may be called "Ossi-friendly because Ossi was adamantly opposed to any explicit support of Conan in build system, see e.g. https://codereview.qt-project.org/c/qt/qtbase/+/184241, however this way indeed may cause dependency slips.


--
Regards,
Konstantin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20201008/ea7bc829/attachment-0001.html>