[Development] Introducing: test batching

[Mikołaj Boć]

Hello,

The WASM team has developed a need to batch multiple tests together into a single binary. The rationale is that on WASM it is required to enable asyncify<https://emscripten.org/docs/porting/asyncify.html> support for tests and linking times and disk size for multiple binaries with that feature on are unacceptable. iOS might be struggling with the problem as well.

The feature that attempts to resolve the problem is called ‘batch_test_support’ and has been implemented mostly in cmake via qt_internal_add_test and its subordinate functions. See https://codereview.qt-project.org/c/qt/qtbase/+/421446 for details.

Enable support for the feature in your Qt build by providing the switch -feature-batch_test_support to configure.
Batch tests together by supplying -batch-tests (support does not imply batching them, only allows one to order the batching).

On C++ level, the macros that register the test’s main function have been changed to register test entry functions in a central registry instead. The test names in a batch may be queried by a launch of the batch binary lacking input parameters. The first parameter to the batched test binary is the name of the test to run. This works identically across platforms.

There are two (known) things that may become an issue for batched tests, which used to work for standard standalone binaries:

  *   Symbol name duplication. The existence of two identically named symbols with external linkage will cause linking errors – the symbols will clash if linked together into a single binary. Please either prefer internal linkage (via e.g., unnamed namespaces) or, if this is not possible in that context, assign unique names to symbols.
Example:
class CustomLayoutStyle is a duplicate symbol with batched tests across tst_qgraphicsanchorlayout.cpp and tst_qboxlayout.cpp

  *   Conflicting test target parameters. In case of conflicts in parameters (parameters directly contradict each other. e.g., one test specifies exception support, the other specifies lack thereof), the tests won’t batch together and cmake will output an error.

There is no complete list now of which tests we would like to batch – we would like to be able to run as many as possible, understandably. The WASM team will determine this in a couple of following weeks. Currently there is work ongoing to adapt corelib/kernel and widgets/ tests for batching.

We plan to enable the feature on CI for WASM. Since that has not yet been done, breaking the tests unintentionally is possible. With CI on, it will become everyone’s responsibility to keep them running (at least on WASM, which will be checked by the CI). Following the guideline on symbol name duplication will save us a considerable amount of work.

Batching tests for other platforms is an opt-in, please do check whether that would simplify the testing pipeline on the platform you are proficient with.

Best regards,
Mikołaj Boć


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20220829/271161f7/attachment-0001.htm>