[Development] Building additional components with Conan for Qt 6

Wed Oct 14 14:06:39 CEST 2020

Hi,

qt3d has been added as an add-on which can be built with Conan.
The qtimageformats and qtnetworkauth conan recipes got a minor update as well.

We've been testing with Android and B2Qt (iMX8) builds as well.
Some fixes still need to land into qtbase for Android so that the conan build recipe
does not require that many workarounds. We still need to figure out details regarding
the profile files for cross compilation targets.

Iikka Eklund
Senior Software Engineer

The Qt Company
http://qt.io<http://qt.io/>

________________________________
From: Heikki Halmet <heikki.halmet at qt.io>
Sent: Wednesday, October 14, 2020 9:14 AM
To: Konstantin Tokarev <annulen at yandex.ru>; Toni Saario <toni.saario at qt.io>; Iikka Eklund <iikka.eklund at qt.io>; Richard Weickelt <richard at weickelt.de>; development at qt-project.org <development at qt-project.org>
Subject: RE: [Development] Building additional components with Conan for Qt 6

Hi,

I'm not the expert when it comes to Conan so I'm asking what would be the main gain we would achieve by using Conan instead of scripts we currently have..? List of packages and versions in one place? With quick review this would take lot of effort and honestly I'm pretty skeptical about the gain we would have vs time we would spent to make needed changes.

>>> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.
>> Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with >> their versions in one place.
You mean reducing provisioning time when everything would be prebuilt already? This is something we could change to existing scripts already. Script could check from the local cache do we have the prebuilt stuff and if don't then build those and add those to cache. And next time use prebuilt package. Currently, e.g. with gcc installation, script will use prebuilt package if it exist and if don't it will be built from sources. But in some cases we have just prebuilt the package and added it to cache without source build option in the script.
With installers I don't think there's no point to start putting those under Conan packages. I think the only thing we would gain here is the listing thing.
And what comes to complexity - if we would start using Conan for everything we still would need to start creating  quite complexity conanfiles for it, right? I guess the gain in some cases would be that we could use the same conanfile in unix and windows.

>>> In current implementation stuff is loaded from Internet, but all packages are verified against manifests stored in provisioning repo. But you can also set up internal Conan repository by running instance of Artifactory CE.
Yes, we would have to use our own internal Conan repo and the repo should be kept up to date automatically.
Do we know what kind of regression Conan repos have? Is there a risk that some package is not reachable etc. E.g. with brew we have stumbled to connection problems or their repositories haven't been available.

Back in days we used Puppet for provisioning. With puppet we stumbled to problems which were quite hard to debug. Reason why we dump it and moved to shell/powershell scripts was to simplify things and make debugging easier.

We definitely could improve the readability of our provisioning scripts but it's all matter of prioritizing.

Br
Heikki

-----Original Message-----
From: Development <development-bounces at qt-project.org> On Behalf Of Konstantin Tokarev
Sent: Thursday, October 8, 2020 12:24 PM
To: Toni Saario <toni.saario at qt.io>; Iikka Eklund <iikka.eklund at qt.io>; Richard Weickelt <richard at weickelt.de>; development at qt-project.org
Subject: Re: [Development] Building additional components with Conan for Qt 6

08.10.2020, 11:59, "Toni Saario" <toni.saario at qt.io>:
>>> Most of provisioning scripts fall into 2 categories:
>>> 1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
>>> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.
>>
>> Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with their versions in one place.
>>
>> [1] https://docs.conan.io/en/latest/devtools/create_installer_packages.html#create-installer-packages
> That is what the automatic documentation tools are asking. Might be stupid questions due to the knowledge. We would install all dependencies during the provisioning and during the builds conan only exposes dependencies listed in the conanfile?

Yes. There would be no network downloads or unnecessary compilations during product builds.

> One thing that conflicts a little bit here is that we want to use cached stuff from our servers instead of downloading from the internet in the provisioning and also have the provisioning to be runnable by anyone, but i guess that if the conan caches all the packages then it equals pretty much the same.

In current implementation stuff is loaded from Internet, but all packages are verified against manifests stored in provisioning repo. But you can also set up internal Conan repository by running instance of Artifactory CE.

>>>> As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.
>>>
>>> Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).
>>
>> In this case each testing activity should have its own conanfile with a list of dependencies, as well as each Qt project which uses them for building.
> Ultimately where would all the conanfiles sit? Currently we only use qt5 repository during provisioning and with conan each repo would have its own conanfiles which needs to be read when running provisioning? How big maintenance burden those conanfiles become, are common files across repos possible?

One possible way (which current implementation uses) is to store all conanfiles in qt5 repository. For local building of individual modules it's possible to use conanfiles right from there, as there are no hard requirements on location or filename of conanfile.

>
> I do not see why it would be impossible to use conan in provisioning, the transition is quite big effort though, and some cherry-picks need to be translated to the old format afterwards. Theres also many other things I cannot answer now related to artifactory and any legal sides to distributing the packages etc.

IANAL, but I don't see why there should be any legal differences between distributing 3rd party code as bundled library or Conan package. Of course, adding new dependencies will require attention to their licensing terms. Also, usage of Conan packages in Coin is not a distribution, only files imported to final packages do count.

See also https://docs.conan.io/en/latest/howtos/collect_licenses.html

>There seems to be a lot of research into conan by Iikka, but i cannot find any related to building all the qt modules with it. I am sure it requires developer attention also, related to the CMake changes that needs to be done. It has a low priority now. https://bugreports.qt.io/browse/QTBUG-74397
>
> ----------------------------------------
> From: Konstantin Tokarev <annulen at yandex.ru>
> Sent: Wednesday, October 7, 2020 6:44 PM
> To: Toni Saario <toni.saario at qt.io>; Iikka Eklund <iikka.eklund at qt.io>; Richard Weickelt <richard at weickelt.de>; development at qt-project.org <development at qt-project.org>
> Subject: Re: [Development] Building additional components with Conan for Qt 6
>
> 02.10.2020, 16:20, "Konstantin Tokarev" <annulen at yandex.ru>:
>> 02.10.2020, 15:53, "Toni Saario" <toni.saario at qt.io>:
>>>  One thing to check is that how much the provisioning it is possible to translate to Conan, I would believe that in the end it does not work for everything.
>>
>> Most of provisioning scripts fall into 2 categories:
>> 1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
>> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.
>
> Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with their versions in one place.
>
> [1] https://docs.conan.io/en/latest/devtools/create_installer_packages.html#create-installer-packages
>
>>>  As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.
>>
>> Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).
>
> In this case each testing activity should have its own conanfile with a list of dependencies, as well as each Qt project which uses them for building.
>
>>
>> I can see another possible way to use Conan in CI - install packages into the system via "deploy" generator so that Qt build system can find them without any knowledge of Conan. I guess this way may be called "Ossi-friendly because Ossi was adamantly opposed to any explicit support of Conan in build system, see e.g. https://codereview.qt-project.org/c/qt/qtbase/+/184241, however this way indeed may cause dependency slips.
>
> --
> Regards,
> Konstantin

--
Regards,
Konstantin
_______________________________________________
Development mailing list
Development at qt-project.org
https://lists.qt-project.org/listinfo/development
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20201014/85105299/attachment-0001.html>