[Development] Unity Build on CI for Qt Base for 3 Platforms

[Marc Mutz]

Hi,

All my -unity-build-batch-size 100'000 patches except for one are now 
merged: https://codereview.qt-project.org/q/topic:unity-build

I'm currently testing without the unmerged one, because it won't apply 
to 6.5, anyway, and the goal is to have unity-builds from 6.5 onwards. 
Maybe a few more TUs will have to be excluded, but you should be able to 
see the pattern. So far it looks like it's not needed.

Compile with -unity-build -unity-build-batch-size 100000 and add all TUs 
that create problems with a small comment as to what's breaking to 
NO_UNITY_BUILD_SOURCES.

I've done the work on Linux, we need someone that does the (considerably 
less involved) work on at least Windows and OSX (though the immediate 
problem that caused the CI to block was Integrity, of course).

Please use topic:unity-build, and link to the most fitting Jira tickets 
in the https://bugreports.qt.io/browse/QTBUG-109394 epic.

We should then be able to re-enable unity builds on the CI, and maintain 
a 100000 build as a nightly.

Thanks,
Marc

On 21.07.23 10:25, Marc Mutz via Development wrote:
> Hi all,
> 
> I don't like the "what _you_ should expect". IMNSHO, _we_ should expect
> it to break only when _we_ add code that causes it to break. IOW: before
> we enable this feature on the production CI for a given configuration X,
> the submitter should have tried configuration X with a
> -unity-build-batch-size of 100000, or whatever is necessary to ensure
> that each sub-project gets built with exactly one unity_<N>_cxx.cxx,
> and exclude all the .cpp files that cause problems in those builds with
> NO_UNITY_BUILD_SOURCES. Only then can we enable the feature for that
> configuration, being (reasonably) certain that a) the batch size
> configured doesn't matter and b) introducing new .cpp files into a
> sub-project's SOURCES causes unity-builds to blow up _only_ due to
> content in _that_ .cpp file.
> 
> I have done that work for qtbase and qtdeclarative and Linux yesterday
> instead of my normal work. I can continue with the rest for Linux
> (already on qttools), but someone else needs to do the same for Windows,
> OSX, Integrity, and whatever other configuration we want to enable
> unity-builds for, _before_ enabling them.
> 
> As it is now, with unity-builds only checked to some insufficient finite
> limit, any unrelated change, even Wednesday's _decrease_ (if it was that)
> of the batch size from 32 to 16, can break the build of totally unrelated
> changes.
> 
> Yesterday was therefore a lost day for the dev CI, and everyone that
> depends on it to integrate their patches. This should be enough of a
> shot across the bow to reconsider strategy.
> 
> I therefore propose to disable all unity-builds on the production CI
> until the proper tests have been done locally (preferably, with a
> nightly build to prove it to reviewers).
> 
> Proposed reverts end in
> https://codereview.qt-project.org/c/qt/qt5/+/492193 (already approved by
> Volker, thanks).
> 
> Thanks,
> Marc
> 
> On 26.06.23 09:39, EXT Amir Abdol via Development wrote:
>> Hi,
>>
>> Unity Build is now enabled on CI for *Qt Base*, on *dev* branch, for the
>> following platforms: *INTEGRITY, Windows 10 MinGW, Windows 11 MinGW*,
>> which are among the slowest platforms on our CI.
>>
>> Here is a brief note about the unity build, and what you should expect:
>>
>>    * Unity build achieves a faster build by pooling several source files
>>      into one translation unit and compiling that instead of individually
>>      compiling every source file. This means the build is especially
>>      susceptible to duplicate symbol collision. If you notice such instances,
>>        o Try resolving the issue, as it is often achievable in a
>>          reasonable timeframe. If not,
>>        o You can exclude the file from the unity build, by adding it to
>>          |NO_UNITY_BUILD_SOURCES|, which is available in most of our
>>          |qt_internal_*| CMake APIs, e.g., |qt_internal_add_tool|. When
>>          doing so, please leave a brief comment and/or TODO on why/how to
>>          resolve the issue.
>>        o If an entire |target| is troublesome, and resolving the issue
>>          needs major work, you can exclude it from the unity build by
>>          passing |NO_UNITY_BUILD|to the respective |qt_internal_*| call.
>>    * You can enable the unity build locally by passing |-unity-build| and
>>      |-unity-build-batch-size| to |configure|, or use
>>      |-DQT_UNITY_BUILD|if you are using CMake. Default batch size is 32.
>>    * We have tested the build with batch size of 32. Changing the batch
>>      size may result in undiscovered symbol collisions, or other issues.
>>    * Tests and examples are not being built using unity build at the moment.
>>
>> While at the moment, we are only partially enabling the unity build for
>> Qt Base on CI, the build should be successful on other platforms as
>> well. Other Qt modules can be built with unity build as well, with
>> varying degrees of success. Please give it a try, and let us know if you
>> ran into issues, either by opening a bug report, or submitting a patch!
>>
>> P.S. Unity build for Qt was initially discussed here:
>> https://lists.qt-project.org/pipermail/development/2023-January/043518.html <https://lists.qt-project.org/pipermail/development/2023-January/043518.html>.
>>
>> Best,
>>
>> Amir.
>>
>>
-- 
Marc Mutz <marc.mutz at qt.io>
Principal Software Engineer

The Qt Company
Erich-Thilo-Str. 10 12489
Berlin, Germany
www.qt.io

Geschäftsführer: Mika Pälsi, Juha Varelius, Jouni Lintunen
Sitz der Gesellschaft: Berlin,
Registergericht: Amtsgericht Charlottenburg,
HRB 144331 B