[Development] Proposal: move Qt provisioning scripts and 3rd party components into a dedicated repo

[Edward Welbourne]

Volker Hilsheimer (14 July 2022 11:23) wrote:
> Our 3rd party dependencies currently live in the submodules where they
> are used. For some 3rd party components, that means we have two,
> sometimes different copies (e.g. assimp in both Qt Quick 3D and Qt 3D,
> only one of them patched).

Having two checkouts of the same thing is not nice; having them not even
on the same version is positively nasty.

> And it makes it in general messy to maintain an overview of those 3rd
> party components. We have a responsibility to keep track of those,

That's supposed to be handled by qt_attribution.json files; do you have
any evidence that we're failing at it ?

> and to keep things up-to-date.

That, indeed, is a chore; but I don't see how moving them to another
module would help.  Indeed, when we do updates to a 3rdparty component,
sometimes it requires parallel changes to the code that calls it.  That
is easy to synchronise if they're in the same module.  Our dependency
tracking system does provide a way to do it with them in separate
modules, but it's one more complication to have to struggle with.

> I think that would be easier if we move them into a single repo with
> all 3rd party libraries.

Aside from the case of duplicated 3rdparty components, I don't really
see how this would make it easier.  Those who maintain the code that
exercises the 3rdparty component would still need to take care of its
updates, along with any needed changes to their code using it, but this
work would now be split across two modules, with a dependency gate in
between, instead of being localised to one module.

Indeed, even for a third-party component that's presently used by two Qt
git modules, if one of those is actively maintained and the other is in
maintenance mode (particularly if its maintainer has quietly slipped out
of touch), an update to the third-party component driven by the former
may break the latter (if we switch to making them use the same
version). You'd either have to kick the latter out of Qt (in order to be
able to take in the update) or do the needed maintenance on it, and we
might not have anyone sufficiently familiar with it to do that robustly.

That might still be a good thing (forcing us to address our lack of
adequate cover for a module we still ship), but it'd be a new source of
pain.

> Our provisioning process in the CI system could then use the
> respective “native" build systems of each 3rd party component in that
> repo to install them as system libraries.

That's great as long as your builds are always in virtual machines, but
very very unwelcome for local native builds, unless I'm misunderstanding
what you mean by "as system libraries".  First, I do not want the
software I build to install random crap all over where my operating
system keeps libraries; second, I routinely build several versions of Qt
in parallel, from different source trees into different build trees.
That "might" lead to conflicts - and, if you're using the third-party
code's own build systems, you're opting to accept whatever bad policy
they have for where to install themselves as system libraries.
So I hope you meant something else by "as system libraries."

If we're going to use their native build systems, we should engineer
that they install into some part of Qt's build tree, that Qt's build
system then adds to the "system" library and header paths for builds in
that tree.  We need this, in any case, so that we can bundle those
libraries up with the resulting Qt binaries when we want to build
packages that can be installed.  (One exception to that is the 3rdparty
components only used in building, e.g. gradle.  The packaging scripts
don't want to pick those up.)

> This will speed up the build of Qt (and removes noise from those
> libraries, some of which are not exactly free of warnings),

That, indeed, would be welcome.
However, see note below on stray 3rdparty .c files.

> and it allows us to gradually stop shipping Qt binaries that use the
> bundled 3rd party stuff.

I fail to follow - how exactly does this let us do that ?  The only case
of it can think of is anywhere that we can defer to the system libraries
and, surely, all distributions that have those system libraries ship
their Qt with a dependency on the relevant system library already, so
build Qt without the internal 3rdparty dependency.

> That will be a big step towards reducing our security relevant body of
> code. It won’t be possible for all 3rd party components

Indeed, we have some third-party components that are just a stray .c
file without a build system; and more where we've extracted the subset
we need from a larger upstream, discarding the rest along with their
build system.  I think we're stuck with those as part of our source tree
(and their warnings as part of our build logs).

> and on all platforms, but at least it allows us to move into that
> direction.

If it lets us merely take the few duplicated third-party components out,
so that there's just one authoritative version of them, so much the
better.  If it's convenient and practical to move some other 3rdparty
libraries out to join them, that's fine.  Let's not, however, take for
granted that we can do more than that.

> At the same time, and because it’s quite related, I’d also like to see
> if we can move all the coin provisioning scripts out of qt5.git

That would be nice.

> and into that same repository.

Do you have a compelling reason for it to be the same repo ?
Why can't it simply be another module ?
(And, thus, a separate task / project / topic.)

> That qt5.git is not just an empty meta-repository that keeps track of
> the consistent set of submodules, but also has code that might need to
> be changed in order for submodule updates to succeed, has more than
> once caused problems. Moving the provisioning code and coin rules into
> a submodule untangles that.

But what new tangles will it introduce ?
I hope the Coin team can answer that,

	Eddy.