[Development] Integration of qt5.git

[André Pönitz]

On Fri, Nov 29, 2013 at 09:19:33PM +0000, Hausmann Simon wrote:
> Once scenario I'm worried about is when a less active module (say qtscript) has
> a regression test failure due to a change in an active module (qtbase) and it's
> only visible in the qt5.git integration (because nobody submits a change to
> qtscript).
> 
> We will see the failure still, but we will also be very very tempted to still
> cut a release from qt5.git because of schedules etc. , despite the failure.

That's not really different from severe performance regressions, or regressions
that hit most user setups, but "accidentally" not the CI machines etc etc.

As an example, for a bit more than two weeks now I have quite a few test cases
failing locally which happily pass CI. _Of course_ this means that something
is broken with my local setup, especially since this happened after a distro
upgrade, but what system makes sure that the Real World is more similar to
what the CI system sees to what my system sees?

I can easily produce dozens of examples from the past where fundamentally
wrong changes got CI approval, as well as examples of case where good
changes got rejected. There are examples of unwanted behaviour changes that
passed CI because the same change also changed the auto-test, etc.

No CI system is infallible, and if we accept that, it's better to have a way
to recover quickly, and not wait a full day, or two, to get "obvious" fixes in.

> Currently we have a mechanism in place that makes it very hard to release in
> such a situation.

I am afraid this is a misconception. And even if it was not, the benefits
would have to be weighed against the costs. Not every CI failure is fatal
for the release, and CI checks only a small fraction of possible failures.
This should be handled at face value: an advice, strong advice even, but
not as a final judgement.

> We are removing this protection and am replacing an automated
> mechanism with a human gate keeper.

Indeed. And that's a good thing, since it allows a balanced decision.
 
> I'm not against trying this, but we have to be very careful I think, more
> discipline will be required from the release side or else we will release with
> known regressions.

Not different from now.

Andre'