[Releasing] Avoiding platform test coverage regressions

Thu Jan 26 13:58:38 CET 2017

On torsdag 26. januar 2017 09.48.56 CET Jani Heikkinen wrote:
> Hi,
> 
> >>We continue to accept today's situation of knowingly regressing in
> >>platform test coverage for releases of Qt.
> I didn't mean that and I think we shouldn't do this ;)
> 
> We have accepted that earlier with
> a) Not defining missing platform as a release blocker
> b) Blacklisting big mass of autotest
> 
> And I do agree we need to stop that. That is causing us lots of problems &
> false impression about our quality, test coverage & supported platforms.
> 
> But we cannot change all that at once; we are committed to deliver new
> releases & new features. So we need to improve the situation by doing baby
> steps to right direction. For example now; I think most important thing is
> to get missing platforms in the builds first (macOS 10.11, 10.12, msvc
> 2017, Ubuntu 16.10 etc). Even without running tests there. But we do need
> to make sure that all possible test must be run there eventuallyl.

To me new platforms including testing on them are features.
Either they make feature freeze or not. Let's either postpone the feature 
freeze or not add them past the poing of the feature freeze.
We all agreed we want a strict feature freeze, so let's not make exceptions 
here. This is exactly why I don't think 5.9 will be out according to schedule.

> As soon
> as possible, well before official release. And I would state now that
> missing test coverage (meaning if we temporarily remove some test now to
> get new platform in) is p0 bug for the Qt 5.9.0 release and so on creating
> P0 bug for that is definitely OK. As you wrote p0 bug is good (I think
> best) way to get someone focusing to that issue...

I think it's a matter of perception, but P0 is not the answer because the 
definition to me is "blocks other people from working" and this doesn't.
So for me P1 would feel correct. But we have a big inflation about the meaning 
of the bug priorities. I still think we should have P1s being release 
blockers. And de-mote every P1 to P2 if it's determined not to be a blocker.

> 
> And in addition to these new platforms I think we must put effort to reduce
> the amount of those blacklisted autotest. It is just wrong to have those in
> as much as there is now. I do understand that sometimes it is wise to
> blacklist the test to be able to proceed but in that case we must create a
> plan how to remove that blacklisting as soon as possible. p0 bug there
> would be good option as well. We tried that but it didn't work. Maybe it
> could be good time now to retry? But what to do for this existing mass of
> blacklisted autotest, it is also test coverage regression from previous
> releases? How we should fix that?

I think we need to make individuals responsible for each of these P1s and make 
sure that they know they are the only ones working on them and that these will 
block the release.

Cheers,
Frederik

> 
> br,
> Jani
> 
> ________________________________________
> From: Simon Hausmann
> Sent: Thursday, January 26, 2017 10:19 AM
> To: Jani Heikkinen; releasing at qt-project.org
> Subject: Re: Avoiding platform test coverage regressions
> 
> Hi,
> 
> 
> You wrote "So we need to find a way to get platforms in easier or then block
> other development until failed test are fixed & new platform is in".
> 
> 
> I do think that both go hand-in-hand. There is infrastructural work that has
> been done and still needs to be done to make it easier to get those
> platforms in. At the same time there is a last mile of actual test fixing
> that often requires attention from a developer who is familiar enough with
> the platform details to fix the issue in some way. I was under the
> impression that release blocking bugs are the most effective instrument we
> have today to direct the attention of development to a particular issue,
> because we have buy-in from the rest of the organization and project that
> release blocking issues are really easy to justify in terms of priorities.
> 
> 
> However relating you statement of "accepted the situation where missing
> platform hasn't been the release blocker" to my question of whether to use
> jira release blockers or if there's an alternate method of blocking, I am
> therefore inclined to conclude:
> 
> 
>     We continue to accept today's situation of knowingly regressing in
> platform test coverage for releases of Qt.
> 
> 
> 
> This is important to me, because when reviewing changes that relate to this
> it really helps to know where we stand with our goals. Knowing that
> retaining that coverage is not a release goal means we can approve changes
> that temporarily remove platforms without the need to follow up that they
> are added again. I personally find this very sad from a quality point of
> view.
> 
> 
> 
> Simon
> 
> ________________________________
> From: Jani Heikkinen
> Sent: Wednesday, January 25, 2017 2:17:40 PM
> To: Simon Hausmann; releasing at qt-project.org
> Subject: Re: Avoiding platform test coverage regressions
> 
> Hi!
> 
> I totally agree with you that missing platforms should block the release. I
> would be even tighter, we should get all new platforms we are planning to
> support in before feature freeze. But lately it has been really hard to get
> new platforms in and that's why we have accepted the situation where
> missing platform hasn't been the release blocker.
> 
> And that is one of most important things to be fixed. We need to get new
> platforms in ci much easier that it is now. I have seen us fighting to get
> new platforms in several months. And that should be only few hours work. It
> seems that it is really hard to get all tests passing at same time for that
> new platform. We try to get new platform in & find some test failing.
> Failures are fixed and new try... Argh, some other test are now failing ;)
> And test are fixed and new test are failing... All that has prevented us to
> get stuff in. So we need to find a way to get platforms in easier or then
> block other development until failed test are fixed & new platform is in.
> 
> br,
> Jani
> 
> ________________________________________
> From: Releasing <releasing-bounces+jani.heikkinen=qt.io at qt-project.org> on
> behalf of Simon Hausmann <Simon.Hausmann at qt.io> Sent: Wednesday, January
> 25, 2017 2:31 PM
> To: releasing at qt-project.org
> Subject: [Releasing] Avoiding platform test coverage regressions
> 
> Hi,
> 
> 
> I'm writing this email because of a recurring problem with our test coverage
> in combination with releases of Qt.
> 
> 
> Many times in the past we've had the situation where a change was applied to
> the CI system that temporarily removed an entire platform, for example in
> favor of adding a new one in steps. I'll cite two examples right off the
> top of my head, but upon request I can dig up more:
> 
> 
>     (1) During times of qtqa-testconfig we added and removed configurations
> without observing what their test coverage was, and at some point near the
> 5.3/5.4 release ended up in a situation where our entire cross-platform
> network stack was only tested partially on Linux, partly as a consequence
> of removing an older Windows (where we ran tests) in favor of a newer
> Windows (where we "insignifified" all network tests). We made several
> releases with a network stack that was untested on Windows and macOS.
> 
> 
>     (2) Today we are in the situation where we are building Qt and running
> tests on macOS 10.11 in the 5.6 branch of Qt. However the changes that
> added 10.11 to the CI configuration to provide build and test coverage were
> never applied to the 5.7, 5.8 or the dev branch of Qt, leaving us with no
> tests run on 10.11.
> 
> 
> During a related discussion in https://codereview.qt-project.org/#/c/183219/
> I suggested that we use a release blocking bug to track the progress of
> changing platforms and that way ensure that we don't release Qt before the
> regression of temporarily removing build and/or test coverage is fixed.
> 
> 
> Tony declined the suggestion with the following statement:
> 
> "We had that. We had a P0 bug to put 10.11 in to 5.7 as it was already in
> 5.6, but it got show down as it was seen as definitely not being able to
> block a release. And this happened again in 5.8. If everything else is
> fine, a missing platform from CI was seen as not to be able to block the
> release."
> 
> 
> I could not find any mention of that in the release team meeting logs
> regarding such a decision.
> 
> 
> I would like to raise the attention to this topic and request either
> approval to use blocking release bugs to track these regressions or ask for
> suggestions of alternate methods to prevent us from repeatedly running into
> the scenario of releasing Qt with regressed build and test coverage.
> 
> 
> 
> 
> Simon
> 
> _______________________________________________
> Releasing mailing list
> Releasing at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/releasing