[Development] Building additional components with Conan for Qt 6

Konstantin Tokarev annulen at yandex.ru
Fri Oct 2 15:16:15 CEST 2020



02.10.2020, 15:53, "Toni Saario" <toni.saario at qt.io>:
> As Iikka said that there has not been plans to use Conan when provisioning the images for CI.

That's sad, I hoped plan was to use Conan for QtBase dependencies eventually to avoid custom bundling code and 3rd-party sources checked into the repo. And also eliminate potential for conflicts when user's project wants to use the same libraries as bundled into QtBase

>
> However there has been some discussions how to make the provisioning easier to manage and have some documentability in them. Currently the provisioning is just a pile of scripts which have zero value for documentation tools. Easiest solution for this would be to list the dependencies on some other file in common format and then install listed dependencies with simple scripts. About using Conan there, I cannot really say anything about it with my knowledge about it.
>
> Another thing is that there are many items built during provisioning which become dependencies, can we reduce the provisioning workload by having these cached in Conan storage?

Yes, I've done this for QtWebKit dependencies. Packages of dependencies are built on public CI services for all necessary compilers and uploaded to https://bintray.com/qtproject/, then during provisioning they are downloaded and verified.

> One thing to check is that how much the provisioning it is possible to translate to Conan, I would believe that in the end it does not work for everything.

Most of provisioning scripts fall into 2 categories:
1) Download installer (or other binary package) of 3rd party stuff, verify checksum, install. These scripts are quite simple.
2) Build stuff from sources in provisioning time, sometimes in a sophisticated way (e.g. https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1). These can be (and IMO should be) replaced with Conan, to reduce provisioning time and complexity.

>
> As for provisioning and why the CI images are different from what are documented, biggest problem there is that the images are used by other components also e.g. RTA testing. Those require some additional stuff and provisioning currently installs everything. AFAIK also all the optional dependencies are installed, this makes it hard to distinguish between them and required dependencies because both are always present. These make it easy that new dependencies etc. slip through unnoticed.

Note that when Conan packages are used via conanfiles (i.e. as it's originally intended by Conan developers), there cannot be any dependency slips because Conan dependencies don't become visible to any project which doesn't require them via conanfile (directly or indirectly).

I can see another possible way to use Conan in CI - install packages into the system via "deploy" generator so that Qt build system can find them without any knowledge of Conan. I guess this way may be called "Ossi-friendly because Ossi was adamantly opposed to any explicit support of Conan in build system, see e.g. https://codereview.qt-project.org/c/qt/qtbase/+/184241, however this way indeed may cause dependency slips.

>
> -Toni
>
> ----------------------------------------
> From: Konstantin Tokarev <annulen at yandex.ru>
> Sent: Friday, October 2, 2020 3:20 PM
> To: Iikka Eklund <iikka.eklund at qt.io>; Richard Weickelt <richard at weickelt.de>; development at qt-project.org <development at qt-project.org>; Toni Saario <toni.saario at qt.io>
> Subject: Re: [Development] Building additional components with Conan for Qt 6
>
> 02.10.2020, 14:47, "Iikka Eklund" <iikka.eklund at qt.io>:
>> Hi Konstantin, all,
>>
>>>>  thanks for the heads up. I have some further questions:
>>>>
>>>>  1. Will Conan be used to manage dependencies of Qt as well?
>>>>
>>>> If this is the plan, note that there is infrastructure [1] for using Conan in provisioning images,
>>>> with verification of downloaded file checksums and build reproducibility for each qt5 commit [2].
>>
>> Regarding Qt CI @Toni Saario is the best one to answer on this, though I haven't heard
>> if there are plans to use Conan more extensively by CI/provisioning at this point?
>
> From my experience, when the way project is built in CI is radically different from a way suggested to
> the end user, it inevitably leads to troubles. Users hit unexpected errors (in this case they may be
> caused by changes in 3rd party Conan recipes/packages, or by contributors to Qt module which
> don't use Conan), and they come back to me asking for solution.
>
>>
>> What comes to add-ons (https://doc-snapshots.qt.io/qt6-dev/qtmodules.html#qt-add-ons) as mentioned
>> earlier the idea is to release add-ons as src packages accompanied with conanfile.py i.e. the build recipe.
>> If the add-on depends on 3rd party libraries which are available e.g. in conan-center the recipe could
>> naturally utilize packages from there (openssl, zlib, etc.).
>>
>>>     2. How will the recipes be managed and where on code.qt.io can I find them?
>>
>> The recipes of the first two add-ons are not part of the source repositories yet, probably should be.
>> Though there are some drawbacks in keeping the recipes and sources in the same repo
>> (some related discussion https://github.com/conan-io/conan/issues/4734).
>>
>> Currently the recipes are part of an internal packaging repository but the current recipes can be found from the src
>> installation directory: "6.0.0/Src/qtnetworkauth/conanfile.py" after you have installed an add-on package.
>>
>> Iikka Eklund
>> Senior Software Engineer
>>
>> iikka.eklund at qt.io
>> http://qt.io
>>
>> ----------------------------------------
>> From: Konstantin Tokarev <annulen at yandex.ru>
>> Sent: Friday, October 2, 2020 12:15 PM
>> To: Richard Weickelt <richard at weickelt.de>; Iikka Eklund <iikka.eklund at qt.io>; development at qt-project.org <development at qt-project.org>
>> Subject: Spam: Re: [Development] Building additional components with Conan for Qt 6
>>
>> 01.10.2020, 16:16, "Konstantin Tokarev" <annulen at yandex.ru>:
>>> 01.10.2020, 16:07, "Richard Weickelt" <richard at weickelt.de>:
>>>>  Hello Ilkka,
>>>>
>>>>  thanks for the heads up. I have some further questions:
>>>>
>>>>  1. Will Conan be used to manage dependencies of Qt as well?
>>>
>>> If this is the plan, note that there is infrastructure [1] for using Conan in provisioning images,
>>> with verification of downloaded file checksums and build reproducibility for each qt5 commit [2].
>>>
>>> If you are interested in details we could discuss them in depth in IRC.
>>>
>>> [1] https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/conan.ps1
>>> [2] requires https://codereview.qt-project.org/c/qt/qt5/+/313930 to handle situations when the same package version is rebuilt without version bump
>>
>> Iikka, any comments?
>>
>> --
>> Regards,
>> Konstantin
>
> --
> Regards,
> Konstantin


-- 
Regards,
Konstantin


More information about the Development mailing list