[Development] Building additional components with Conan for Qt 6

[Konstantin Tokarev]

08.10.2020, 11:59, "Toni Saario" <toni.saario at qt.io>:
>>> Most of provisioning scripts fall into 2 categories:
>>> 1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
>>> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.
>>
>> Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with their versions in one place.
>>
>> [1] https://docs.conan.io/en/latest/devtools/create_installer_packages.html#create-installer-packages
> That is what the automatic documentation tools are asking. Might be stupid questions due to the knowledge. We would install all dependencies during the provisioning and during the builds conan only exposes dependencies listed in the conanfile?

Yes. There would be no network downloads or unnecessary compilations during product builds.

> One thing that conflicts a little bit here is that we want to use cached stuff from our servers instead of downloading from the internet in the provisioning and also have the provisioning to be runnable by anyone, but i guess that if the conan caches all the packages then it equals pretty much the same.

In current implementation stuff is loaded from Internet, but all packages are verified against manifests stored in provisioning repo. But you can also set up internal Conan repository by running instance of Artifactory CE.


>>>> As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.
>>>
>>> Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).
>>
>> In this case each testing activity should have its own conanfile with a list of dependencies, as well as each Qt project which uses them for building.
> Ultimately where would all the conanfiles sit? Currently we only use qt5 repository during provisioning and with conan each repo would have its own conanfiles which needs to be read when running provisioning? How big maintenance burden those conanfiles become, are common files across repos possible?

One possible way (which current implementation uses) is to store all conanfiles in qt5 repository. For local building of individual modules it's possible to use conanfiles right from there, as there are no hard requirements on location or filename of conanfile.

>
> I do not see why it would be impossible to use conan in provisioning, the transition is quite big effort though, and some cherry-picks need to be translated to the old format afterwards. Theres also many other things I cannot answer now related to artifactory and any legal sides to distributing the packages etc. 

IANAL, but I don't see why there should be any legal differences between distributing 3rd party code as bundled library or Conan package. Of course, adding new dependencies will require attention to their licensing terms. Also, usage of Conan packages in Coin is not a distribution, only files imported to final packages do count.

See also https://docs.conan.io/en/latest/howtos/collect_licenses.html

>There seems to be a lot of research into conan by Iikka, but i cannot find any related to building all the qt modules with it. I am sure it requires developer attention also, related to the CMake changes that needs to be done. It has a low priority now. https://bugreports.qt.io/browse/QTBUG-74397
>
> ----------------------------------------
> From: Konstantin Tokarev <annulen at yandex.ru>
> Sent: Wednesday, October 7, 2020 6:44 PM
> To: Toni Saario <toni.saario at qt.io>; Iikka Eklund <iikka.eklund at qt.io>; Richard Weickelt <richard at weickelt.de>; development at qt-project.org <development at qt-project.org>
> Subject: Re: [Development] Building additional components with Conan for Qt 6
>
> 02.10.2020, 16:20, "Konstantin Tokarev" <annulen at yandex.ru>:
>> 02.10.2020, 15:53, "Toni Saario" <toni.saario at qt.io>:
>>>  One thing to check is that how much the provisioning it is possible to translate to Conan, I would believe that in the end it does not work for everything.
>>
>> Most of provisioning scripts fall into 2 categories:
>> 1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
>> 2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.
>
> Of course, it's also possible to use conan packages for category (1), however this may require writing quite a few custom recipes (see manual [1]), possibly containing more boilerplate than existing ps1 snippets. However, this would allow to have a nice list of all provisioned packages with their versions in one place.
>
> [1] https://docs.conan.io/en/latest/devtools/create_installer_packages.html#create-installer-packages
>
>>>  As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.
>>
>> Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).
>
> In this case each testing activity should have its own conanfile with a list of dependencies, as well as each Qt project which uses them for building.
>
>>
>> I can see another possible way to use Conan in CI - install packages into the system via "deploy" generator so that Qt build system can find them without any knowledge of Conan. I guess this way may be called "Ossi-friendly because Ossi was adamantly opposed to any explicit support of Conan in build system, see e.g. https://codereview.qt-project.org/c/qt/qtbase/+/184241, however this way indeed may cause dependency slips.
>
> --
> Regards,
> Konstantin


-- 
Regards,
Konstantin