[Development] Proposal: move Qt provisioning scripts and 3rd party components into a dedicated repo

[Thiago Macieira]

On Thursday, 14 July 2022 04:22:59 PDT Laszlo Agocs wrote:
> 1. Can we assume every single 3rd party library out there has a
> sophisticated build system that builds (or at least allows building after
> appropriate configuration) exactly the way Qt needs, with first class
> cross-compilation and embedded support, including the exotic targets like
> INTEGRITY? (unlikely)  How about Yocto where you wouldn't need the upstream
> build system but rather appropriate recipes for packages? Integrating the
> external code into Qt's build system have some benefits that are probably
> not obvious at first glance, esp. when it comes to the less straightforward
> non-desktopish targets.

I don't think we can *assume* that, but we can make it so. Both we and Yocto 
Project somehow manage to build all of those, so it must be possible somehow. 
Sometimes it might require patching, or it may require injecting a different 
build system to make it compile for that target. But there is a solution.

Of course, the next issue is what we should do if that need for a solution 
arises. I'm hoping that most projects will accept patches to fix them to build 
them properly and cleanly. They may balk at having an INTEGRITY requirement, 
but the burden falls on the interested party to keep that working. This is 
particularly true for OSes and SDKs that aren't freely available or even 
remotely easy to obtain. This may be an area that we can get Blackberry and 
Green Hills to contribute directly to, actually: get their own people to keep 
those low-level infrastructure libraries working.

> If the end result here is a partial solution for the common desktop
> platforms, while most Qt modules will still be stuck with having to support
> alternative solutions (e.g. still bundle) for the more exotic (but
> essential) targets, then that might not be much of an improvement over all.

There's a separate meaning of "bundle" here. Some of those libraries are 
mandatory or at least highly-recommended requirements of the Qt modules, so 
they must be present on the user's device, regardless of how they were built. 
The difference here is how they're build and I think Volker's proposal makes an 
improvement, if at the cost of having to deal with a foreign build system. See 
my direct reply for more.

> 2. Not every 3rd party library offers Qt-style compatibility guarantees. We
> may not even have proper source/behaviorial compatiblity promises in some
> cases. This means blindly updating stuff to the latest version (no matter
> how important or great that release may seem) or sharing the same 3rd party
> content between multiple Qt modules is not always feasible or wise.

The libraries where sharing is not feasible (Assimp) need to be fixed, right 
now. One radical solution is to yank them away as crappy code that they are 
and never look back. Why are we building our own code over this house of 
cards?

If necessary, we fork it. But then I'll insist on those warnings getting fixed, 
which means the code will greatly diverge from upstream.

> A 3rd party library is often an integral part of a module, with custom
> patches applied and *careful testing* with every single update of the 3rd
> party code to make sure it plays perfectly together with the particular
> module.  (may not be the case everywhere in Qt, but certainly true for some
> of the graphics/3D dependencies)

We must attempt to upstream those patches. If upstream refuses, we can discuss 
it. Hopefully they'll only apply to "exotic" platforms, so the standard third 
party will work for almost everyone. Then we simply store the Git patchset to 
apply on top of third party, like KDE is doing to the Qt LTS branches.

There's no such thing as a maximum version of a module: any version of that 
third party above the minimum MUST WORK at the moment of our release. That 
means we maintainers of a given code must test Qt against the latest versions 
of the third party, including making sure we are in the cycle for their 
releases and correcting anything in our code, or reporting issues in theirs.

We can't be blamed for issues that arise with updates after our releases, if 
we misused the library. Users will know which version was current at the time 
of the release and they MAY want to stick to the same release series. But 
that's their choice to make.

And if the third party in question is so fragile that every release of theirs 
breaks Qt? I repeat: Yank. It. Out. Why are we building our code on top of 
this fragile house of cards?

> Dumping the full upstream contents to a faraway external repository and
> claiming it will just work as-is (or that any change we apply on top is
> fine for all clients of the 3rd party lib within Qt) sounds a bit
> optimistic. The 3rd party content that is essential for a Qt module should
> rather be "near the eyes of the developers", ideally next to the module
> sources. (again, this may not apply uniformly true within Qt, but certainly
> true in graphics/3D for certain 3rd party deps that provide core and
> essential functionality)

No, it should not. You shouldn't have to debug it unless you're also a 
developer in that particular library or, at a minimum, are trying to report a 
bug against it as a good user.

libpng and fontconfig/freetype are a core/essential functionality of QtGui, for 
example. But I don't expect any of our developers have debugged those in a 
decade or more. That is not true for libwayland, but that's because our 
QtWayland developers are very active in the Wayland community. And moreover, 
it's not bundled.

> 3. Many 3rd party components are stripped down, meaning we build only what
> is needed by Qt. In the form that is best, even if that is something the
> upstream build system would never do, with settings or changes upstream may
> not support out of the box. (e.g. sometimes we can just compile in the 3rd
> party sources directly to the module or maybe use a static lib etc.,
> instead of polluting everything with tens of shared libraries and
> unnecessary tools and artifacts from the upstream build system)
> 
> That this makes updating more difficult is not a valid argument, since
> updating *should* be an involved process - the impact of updating the 3rd
> party code must be carefully evaluated and tested. (as always, thinking of
> graphics/3D 3rd party examples here, perhaps the bar is lower elsewhere
> with other 3rd party deps)

Those are arguments for removing the bundled copy, because updating the 
content is made more difficult. Updating is required, both by us and by our 
users, so we should make it as easy as possible.

> 4. Simplicity of customization. Applying custom patches to a stripped down
> bundled version may be significantly simpler than fighting the entire
> upstream repo and its build system. (and customization will be needed,
> given Qt's extensive platform and compiler coverage)

Upstream it.

If upstream isn't willing to deal with an exotic platform, then we should 
reconsider this particular feature in that library. Either find another 
library, or develop ourselves, or don't support this feature in that 
configuration.

> 5. Manually building and installing all 3rd party dependencies for every
> single Qt module we work with (thinking here of devs with no toplevel
> builds and whatnot, just building individual repos) is not very tempting.

It should simply be an extra build before the qtbase one.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering