[Releasing] Proposal to adjust release candidate process

[Giuseppe D'Angelo]

Hello,

first and foremost, thanks for raising these issues.


Il 20/12/2016 14:34, Tuukka Turunen ha scritto:

> I think we have three major problems with our current release process
> regarding the release candidate phase:
> 
> 1.       Process to make a RC that is as flawless as final causes
> inefficiency as we only get full test coverage with the RC itself

Well, "Release Candidate" by definition means "(hopefully) suitable
*as-is* for final release"; as such, we must enter that phase only when
we have already reached a state as flawless as possible. It's sad that
we get "test coverage" with the RC. Do you mind to elaborate more,
however? What does "test coverage" mean here?

> 2.       We get full attention for testing a bit too late, many fixes
> are still coming in close to the planned RC time causing instability
>
> 3.       Current time between RC and final is planned to be 2 weeks,
> which is very little in order to take in the feedback and fix things

IMHO this time should be enough, if

1) it doesn't happen over holidays (I sent an email a few days ago about
this very issue. Proper testing can't happen across holidays. As such we
should refrain from releasing just before them, and/or not consider them
as part of the timeout period)

2) Release candidate really means "candidate suitable for final
release". As such we would've failed the release process if we put a RC
out with blatant bugs. Two weeks should be enough to confirm that indeed
there are no such bugs.



What I really feel here is that these are symptoms of a more serious
issue, which is our commitment to time-based releases.

Instead of entering the RC phase when the product is ready for it, we
forcibly push for it because of the approaching deadlines.

This results in lack of proper testing before the RC (since there's no
real stabilization phase). Also, because in the RC phase only critical
bugfixes are accepted, we end up releasing a product with known serious
bugs (causing .0 releases to be frowned upon because full of bugs).

(Side note: we haven't managed to make a release happen on time since
5.2; actually, we've got pretty significant delays:

* Qt 5.2.0: +2 days
* Qt 5.3.0: +21 days
* Qt 5.4.0: +48 days
* Qt 5.5.0: +64 days
* Qt 5.6.0: +99 days
* Qt 5.7.0: +49 days
* Qt 5.8.0: +22 days and counting (won't be less than 50 considering the
RC is not out yet, plus the 2 weeks of holidays))

I would therefore ask to _stop_ the timed-based release process for new
minor releases, and keep it only for patch releases. For those, I would
actually like it to be at a stable and high frequency (say, every 2
months for the stable branches, and 3-4 for the 5.6 branch).


> Therefore, I would like to propose the following:
> 
> a.       Consider “Release Candidate” to be a phase rather than an
> individual delivery
> 
> b.       Create the first “RC1” almost immediately after release branch
> (e.g. 5.9.0) is operational
> 
> c.       Criteria for the “RC1” is that no known P0 bugs exist (i.e.
> there can be other issues that would not be acceptable in a final release)
> 
> d.       During the “RC” phase P1 (and possible P0 of course) bugs and
> documentation are fixed
> 
> e.       Public “RC” release is similar development release as Alpha and
> Beta in that it starts a phase of work
> 
> f.        Multiple snapshots / new candidates are created during the
> “RC” phase until one of them is considered the final release

Honestly I'm not too convinced of this plan; how do you decide when to
branch 5.9.0 out of 5.9? "No known P0" isn't an acceptable bar -- that's
actually true most of the time. When does the stabilization actually
happen in the above plan?

> 
> If desired, we could use some other name than “Release Candidate 1” for
> the release that begins the last phase of the release. It could be
> called “Beta 2” or “Technology preview”, if so desired. Personally, I
> would call it “Release Candidate 1”.

Yes, please, the correct name for all of this is "Beta", not "Release
Candidate". And during the beta we should accept fixes for P2/P3 bugs
too, if they affect the product quality in a significant way.

(We should also put betas and RC outs for patch releases; hopefully, the
stabilization period for those should be considerably shorter given
they're already supposed to be branched out of a "stable" tree.)


> The difference to our current process is quite small. In essence it
> would be about considering the “RC1” the beginning of the final
> releasing phase (.0 branch), not something we do almost at the end of
> it. I believe that lowering the quality criterial for “RC1” helps us in
> being more efficient as it has been in practice impossible to really
> fulfill the current process goal and have already the first RC as good
> as the final.

As I said before, I think this is a symptom, not the cause. Lowering the
quality criteria for release phase just doesn't seem right to me. Let's
try to get more betas out, and more frequently, instead.


> In case of Qt 5.9 it would mean that we have the first “RC” out around
> end of April, soon after the branching to 5.9.0 has been completed. We
> then have 4 or so weeks to make all the needed amount of candidates /
> snapshots until one of them will be released as Qt 5.9.0 final. If it
> happens earlier than in 4 weeks, great. If it takes more time, then so
> be it (although in such case we have probably missed something in the
> earlier steps of the release creation).

We've missed to evaluate whether we were ready to branch off 5.9.0 out
of the 5.9 branch and enter the release process. I think that's the part
where we shouldn't rush things out by having fixed deadlines, but allow
stabilization and bugfixing to happen by releasing several betas before
getting into the RC phase.


Cheers,
-- 
Giuseppe D'Angelo | giuseppe.dangelo at kdab.com | Senior Software Engineer
KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908
KDAB - Qt, C++ and OpenGL Experts

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4007 bytes
Desc: Firma crittografica S/MIME
URL: <http://lists.qt-project.org/pipermail/releasing/attachments/20161221/404d2864/attachment.bin>