[Development] Proposal: move Qt provisioning scripts and 3rd party components into a dedicated repo

[Volker Hilsheimer]

> On 14 Jul 2022, at 13:51, Edward Welbourne <edward.welbourne at qt.io> wrote:
> 
> Volker Hilsheimer (14 July 2022 11:23) wrote:
>> And it makes it in general messy to maintain an overview of those 3rd
>> party components. We have a responsibility to keep track of those,
> 
> That's supposed to be handled by qt_attribution.json files; do you have
> any evidence that we're failing at it ?

From the perspective of keeping track of things, one qt_attribution.json file per submodule is only marginally better than searching src/3rdparty for each submodule.

It does not help people needing a 3rd party component in one submodule see what 3rd party components we use elsewhere already. It also makes it very difficult for auditors at companies using Qt to see which 3rd party components there are buried in the Qt code.

>> and to keep things up-to-date.
> 
> That, indeed, is a chore; but I don't see how moving them to another
> module would help.  Indeed, when we do updates to a 3rdparty component,
> sometimes it requires parallel changes to the code that calls it.  That
> is easy to synchronise if they're in the same module.  Our dependency
> tracking system does provide a way to do it with them in separate
> modules, but it's one more complication to have to struggle with.

Agree, if there is very tight coupling between an exact version of a 3rd party module and our code, then for those 3rd party modules it will be difficult. But then it will be next to impossible anyway to use the system library of those modules anyway.

I think that’s by far the exception though. Most 3rd party components we use have well defined, stable APIs.


>> I think that would be easier if we move them into a single repo with
>> all 3rd party libraries.
> 
> Aside from the case of duplicated 3rdparty components, I don't really
> see how this would make it easier.  Those who maintain the code that
> exercises the 3rdparty component would still need to take care of its
> updates, along with any needed changes to their code using it, but this
> work would now be split across two modules, with a dependency gate in
> between, instead of being localised to one module.


The assumption is that for the vast majority of our 3rd party dependencies, there is a stable API. Stuff that’s tightly coupled is the exception, and can be treated as such.


> Indeed, even for a third-party component that's presently used by two Qt
> git modules, if one of those is actively maintained and the other is in
> maintenance mode (particularly if its maintainer has quietly slipped out
> of touch), an update to the third-party component driven by the former
> may break the latter (if we switch to making them use the same
> version). You'd either have to kick the latter out of Qt (in order to be
> able to take in the update) or do the needed maintenance on it, and we
> might not have anyone sufficiently familiar with it to do that robustly.
> 
> That might still be a good thing (forcing us to address our lack of
> adequate cover for a module we still ship), but it'd be a new source of
> pain.

If we have a submodule in Qt that doesn’t keep up with the 3rd party libraries it depends on, then it needs to go. Otherwise it becomes a security liability.

>> Our provisioning process in the CI system could then use the
>> respective “native" build systems of each 3rd party component in that
>> repo to install them as system libraries.
> 
> That's great as long as your builds are always in virtual machines, but
> very very unwelcome for local native builds, unless I'm misunderstanding
> what you mean by "as system libraries".  First, I do not want the
> software I build to install random crap all over where my operating
> system keeps libraries; second, I routinely build several versions of Qt
> in parallel, from different source trees into different build trees.
> That "might" lead to conflicts - and, if you're using the third-party
> code's own build systems, you're opting to accept whatever bad policy
> they have for where to install themselves as system libraries.
> So I hope you meant something else by "as system libraries."


I was specifically talking about the CI system.

For local builds, you have the option today already to use the system libraries in most cases. Building the 3rd party code from the repo of 3rd party modules would not have to be done differently than how it’s done today, except that the sources are not searched in $QTCLONE/qtbase/src/3rdparty/zlib, but in $QTCLONE/qt3rdpartystuff/zlib (or whatever).

If you have neither the system SDK installed, nor the qt3rdpartystuff repo checked out, then you will miss features or fail to configure. Just as today with many other 3rd party libraries.


> If we're going to use their native build systems, we should engineer
> that they install into some part of Qt's build tree, that Qt's build
> system then adds to the "system" library and header paths for builds in
> that tree.  We need this, in any case, so that we can bundle those
> libraries up with the resulting Qt binaries when we want to build
> packages that can be installed.  (One exception to that is the 3rdparty
> components only used in building, e.g. gradle.  The packaging scripts
> don't want to pick those up.)

Yep.


>> This will speed up the build of Qt (and removes noise from those
>> libraries, some of which are not exactly free of warnings),
> 
> That, indeed, would be welcome.
> However, see note below on stray 3rdparty .c files.
> 
>> and it allows us to gradually stop shipping Qt binaries that use the
>> bundled 3rd party stuff.
> 
> I fail to follow - how exactly does this let us do that ?  The only case
> of it can think of is anywhere that we can defer to the system libraries
> and, surely, all distributions that have those system libraries ship
> their Qt with a dependency on the relevant system library already, so
> build Qt without the internal 3rdparty dependency.

We can expect that a system that runs Qt has a system libpng.so. We don’t need to build libpng sources into Qt just because our build environment doesn’t have libpng-dev provisioned.

>> That will be a big step towards reducing our security relevant body of
>> code. It won’t be possible for all 3rd party components
> 
> Indeed, we have some third-party components that are just a stray .c
> file without a build system; and more where we've extracted the subset
> we need from a larger upstream, discarding the rest along with their
> build system.  I think we're stuck with those as part of our source tree
> (and their warnings as part of our build logs).


Again, by far the exception, in the greater scheme of things. Nobody is going to raise a CVE against a stray .c file that then makes customers asks what we are going to do about it.


>> and on all platforms, but at least it allows us to move into that
>> direction.
> 
> If it lets us merely take the few duplicated third-party components out,
> so that there's just one authoritative version of them, so much the
> better.  If it's convenient and practical to move some other 3rdparty
> libraries out to join them, that's fine.  Let's not, however, take for
> granted that we can do more than that.
> 
>> At the same time, and because it’s quite related, I’d also like to see
>> if we can move all the coin provisioning scripts out of qt5.git
> 
> That would be nice.
> 
>> and into that same repository.
> 
> Do you have a compelling reason for it to be the same repo ?
> Why can't it simply be another module ?
> (And, thus, a separate task / project / topic.)


Having things that are required when we provision our CI machines in the same place seems to be an advantage. Our CI provisioning scripts will run the builds of the 3rd party components (at least those that can’t be sourced from elsewhere, like officially maintained conan packages).


>> That qt5.git is not just an empty meta-repository that keeps track of
>> the consistent set of submodules, but also has code that might need to
>> be changed in order for submodule updates to succeed, has more than
>> once caused problems. Moving the provisioning code and coin rules into
>> a submodule untangles that.
> 
> But what new tangles will it introduce ?
> I hope the Coin team can answer that,


Yop. Nothing is going to happen immediately. If we can agree that the goal is worthy the effort it will take, then setting up a new submodule where we can experiment with some of the obvious candidates (zlib, libpng, freetype) will probably find the new tangles :)


Volker