[Development] Suggestion for new CI: "early warning" for qt5.git

[Rohan McGovern]

Kent Hansen said:
> Hi,
> Sometimes there are changes to qtbase that the author and/or reviewers 
> strongly suspect will break one or more modules on top of it (e.g. in 
> qt5.git). For example, source-incompatible changes (removing 
> QEventLoop::DeferredDeletion), or changes to the meta-type/QObject 
> internals (which qtdeclarative is relying heavily on).
> 
> When such a change goes into qtbase, you only get the failure for the 
> dependent modules the next time you stage something (anything) in that 
> other module; suddenly the module no longer builds, or tests seemingly 
> unrelated to the developer's own change start failing.
> 
> This means developers will be scratching their heads and spending time 
> tracking down what was the actual cause of the failure (first in their 
> own changes/module, then in the dependencies). It also means that the CI 
> is wasting cycles on subsequent (failed) attempts to fix the failure, 
> which is disruptive to everyone trying to stage new and unrelated 
> changes. If there is no quick fix, the last resort is to "pin" one or 
> more of the dependencies' to a particular SHA1 (e.g. from the qt5.git 
> submodule) in the sync.profile until the issue has been resolved, then 
> "unpin" it again later.
> 
> An idea to avoid such disruption would be a CI where you can try to 
> merge your change, and it will build all of qt5.git with it (_and_ run 
> all the module's tests). This could be a separate button in Gerrit, for 
> example.
> 
> You then get a report in Gerrit with the status from the complete build 
> & tests run. Even if everything succeeds, the change will not actually 
> be integrated; this would just be a tool for getting an indication of 
> potential inter-module problems with your patch. If there are problems, 
> the developer can then either fix the affected modules in advance if 
> possible, or at least give a heads-up and/or prepare patches that will 
> fix the upcoming breakage in the other modules.
> 
> What do you think?
> 

As Joao mentioned, this relates to "a dry run of the CI" feature.
I think that we need a method of requesting that some test procedure is
run over your change in gerrit, without also testing some set of everyone
else's changes at the same time, and without requesting that the change
is actually merged into the repository at the end of the test.

If we have that, then we can add all sorts of test procedures into that
workflow.  "test all of qt5 on top of my change" would be a natural test
to provide.

You mentioned a button in gerrit, but what do you think of soliciting
machine reviewers instead, as you currently do for humans?  e.g.

  1. push change to gerrit
  2. go to web UI, add reviewer named something like
  "Test : qt5.git tester : compile/test all of qt5.git with your change"
  3. qt5.git tester immediately adds comment "I'll start testing your
  change in (~ some time estimate)"
  4. qt5.git tester later adds pass or fail comment, with a link to a
  build/test log.

3 could be omitted if that's perceived as too noisy.

You could add multiple testers with different goals this way, and they'd
operate independent from each other.