[Releasing] Avoiding platform test coverage regressions

[Frederik Gladhorn]

On onsdag 25. januar 2017 13.17.40 CET Jani Heikkinen wrote:
> Hi!
> 
> I totally agree with you that missing platforms should block the release. I
> would be even tighter, we should get all new platforms we are planning to
> support in before feature freeze. But lately it has been really hard to get
> new platforms in and that's why we have accepted the situation where
> missing platform hasn't been the release blocker.

Don't accept. You are the one to say: "no, then we don't release". Scream out 
loud and early that we won't release instead :) I think there was a bit of 
screaming now so I'm happy that the missing platforms got into many people's 
mind and focus.
Especially you, Jani, are in a position to demand things from the rest of the 
organization. When you see something that is going to make us unable to 
release, try to talk to the people that can do something about it directly. If 
people don't react, raise a big red flag with Olli and/or Tuukka. Considering 
our track record in releasing on time and Tuukka's concern with releases we'll 
hopefully get people to help out.

> And that is one of most important things to be fixed. We need to get new
> platforms in ci much easier that it is now. I have seen us fighting to get
> new platforms in several months. And that should be only few hours work. It
> seems that it is really hard to get all tests passing at same time for that
> new platform. We try to get new platform in & find some test failing.
> Failures are fixed and new try... Argh, some other test are now failing ;)
> And test are fixed and new test are failing... All that has prevented us to
> get stuff in. So we need to find a way to get platforms in easier or then
> block other development until failed test are fixed & new platform is in.

I agree. Creative proposals and code that implements that please :)

We (in Oslo and Berlin) have probably been quite hard to deal with in this 
regard, since we made some experiences that are really on our minds when 
dealing with this: We don't want platforms added when they actually don't 
work, because history has shown again and again that nobody will fix them.
To this day I know we don't test qtxmlpatterns (well, we run 5 out of 100 
tests or so) and Simon mentioned the network example.
I think maybe it needs someone with the will to do a bit of leadership and who 
has passion to write it on their flags and fight the battle.

Cheers,
Frederik


> 
> br,
> Jani
> 
> ________________________________________
> From: Releasing <releasing-bounces+jani.heikkinen=qt.io at qt-project.org> on
> behalf of Simon Hausmann <Simon.Hausmann at qt.io> Sent: Wednesday, January
> 25, 2017 2:31 PM
> To: releasing at qt-project.org
> Subject: [Releasing] Avoiding platform test coverage regressions
> 
> Hi,
> 
> 
> I'm writing this email because of a recurring problem with our test coverage
> in combination with releases of Qt.
> 
> 
> Many times in the past we've had the situation where a change was applied to
> the CI system that temporarily removed an entire platform, for example in
> favor of adding a new one in steps. I'll cite two examples right off the
> top of my head, but upon request I can dig up more:
> 
> 
>     (1) During times of qtqa-testconfig we added and removed configurations
> without observing what their test coverage was, and at some point near the
> 5.3/5.4 release ended up in a situation where our entire cross-platform
> network stack was only tested partially on Linux, partly as a consequence
> of removing an older Windows (where we ran tests) in favor of a newer
> Windows (where we "insignifified" all network tests). We made several
> releases with a network stack that was untested on Windows and macOS.
> 
> 
>     (2) Today we are in the situation where we are building Qt and running
> tests on macOS 10.11 in the 5.6 branch of Qt. However the changes that
> added 10.11 to the CI configuration to provide build and test coverage were
> never applied to the 5.7, 5.8 or the dev branch of Qt, leaving us with no
> tests run on 10.11.
> 
> 
> During a related discussion in https://codereview.qt-project.org/#/c/183219/
> I suggested that we use a release blocking bug to track the progress of
> changing platforms and that way ensure that we don't release Qt before the
> regression of temporarily removing build and/or test coverage is fixed.
> 
> 
> Tony declined the suggestion with the following statement:
> 
> "We had that. We had a P0 bug to put 10.11 in to 5.7 as it was already in
> 5.6, but it got show down as it was seen as definitely not being able to
> block a release. And this happened again in 5.8. If everything else is
> fine, a missing platform from CI was seen as not to be able to block the
> release."
> 
> 
> I could not find any mention of that in the release team meeting logs
> regarding such a decision.
> 
> 
> I would like to raise the attention to this topic and request either
> approval to use blocking release bugs to track these regressions or ask for
> suggestions of alternate methods to prevent us from repeatedly running into
> the scenario of releasing Qt with regressed build and test coverage.
> 
> 
> 
> 
> Simon
> 
> _______________________________________________
> Releasing mailing list
> Releasing at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/releasing