[Releasing] Proposal to adjust release candidate process

[Tuukka Turunen]

> -----Original Message-----
> From: Releasing [mailto:releasing-bounces+tuukka.turunen=qt.io at qt-
> project.org] On Behalf Of Maurice Kalinowski
> Sent: keskiviikkona 21. joulukuuta 2016 9.34
> To: Giuseppe D'Angelo <giuseppe.dangelo at kdab.com>; releasing at qt-
> project.org
> Subject: Re: [Releasing] Proposal to adjust release candidate process
> 
> > >
> > > 1.       Process to make a RC that is as flawless as final causes
> > > inefficiency as we only get full test coverage with the RC itself
> >
> > Well, "Release Candidate" by definition means "(hopefully) suitable
> > *as-is* for final release"; as such, we must enter that phase only
> > when we have already reached a state as flawless as possible. It's sad
> > that we get "test coverage" with the RC. Do you mind to elaborate
> > more, however? What does "test coverage" mean here?
> 
>  [Kalinowski Maurice]
> Test coverage implies active user manual testing outside the Qt Company.
> Whenever new packages are ready, R&D is asked to test all available
> packages. Those are actually more packages than the ones we send out to
> everyone asking to test.
> 
> However, the feedback we get from the community on package testing
> before the RC is almost zero, but then raises significantly when a package is
> claimed to be a Release Candidate and sometimes a Release Candidate
> Candidate. But as mentioned before, for many reported items, this is too late
> in the process.
> 

Exactly. The main point in my proposal is to get focus from users to the "RC1" (or whatever we decide to call it). We are in practice already doing most of this, only issue is that in my opinion we wait too long before making the first release out of the .0 release branch. Then we aim to do the final in 2 weeks after, which means it is difficult to get enough feedback in time.  

> [...]
> >
> >
> > What I really feel here is that these are symptoms of a more serious
> > issue, which is our commitment to time-based releases.
> >
> > Instead of entering the RC phase when the product is ready for it, we
> > forcibly push for it because of the approaching deadlines.
> >
> > This results in lack of proper testing before the RC (since there's no
> > real stabilization phase). Also, because in the RC phase only critical
> > bugfixes are accepted, we end up releasing a product with known
> > serious bugs (causing .0 releases to be frowned upon because full of bugs).
> 
>  [Kalinowski Maurice]
> I would object to that. The reason we are having so many delays for minor
> releases is that we are not taking the feature freeze seriously, push a huge
> amount of stuff still into it (with at least questionable significance) and pray
> that it'll work. As you mention, it never does, but neither does everyone
> accept the fact that each and every individual contributing raises the chance
> of causing regressions and further bugs at that stage.
> 
> The idea of having patch level releases more frequent is certainly welcomed,
> but then again the problems are the same, way too many changes for those
> causing a lot of effort for testing and verification.
> 

Qt 5.7.1 took way too long to get out, no arguments there. We should make the first patch release soon after the .0 release because there always are already many changes waiting in the stable branch at the time of making the .0 release. 

In my (probably biased :) opinion our current quality level of the .0 releases has been quite good recently. I do not expect my proposal to make RC1 earlier to make it worse - on the contrary I expect to get feedback earlier thus resulting in a better .0 release.

> 
> While I can understand the background for this proposal, I still do not like the
> idea to call it Release Candidate, because right from the description it is not
> and hence distinguishes from any other software product doing releases
> including release candidates.
> 

Name of the release can be in essence whatever that does not confuse users. My gut feeling is that Beta 2 is not good name as we have already moved to the next phase in going to the release branch. We have not released the current process RC as final for any of the Qt 5 releases - there always have been a couple of fixes between RC and final. I think that as long as everyone is aware what the release is, it does not confuse users to call it RC (or RC1).  

Yours,

	Tuukka

> Maurice
> _______________________________________________
> Releasing mailing list
> Releasing at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/releasing