[Development] Proposal to adjust release candidate process

[Robin Burchell]

My (slightly delayed) $0.02:

On releasing in general: I would agree that releases take too long,
often, but I am of the belief that a good part of this is a mess of our
own making. For instance, due to an end-user or project requiring a
specific version or purely out of fear that the next release may be a
long time away, I think we (collectively) often end up pushing changes
to branches that might be less than ideally suited for the change in
question.

I think we should recognise that these pressures do exist, and consider
ways we can work to mitigate them (things like: nightly or at least
semi-"nightly" builds? working on pushing out patch releases more often?
encouraging more of our consumers to apply patches that we already took
in upstream and do their own builds? anything else?)

It goes without saying that these are hard problems, but I think that
working at the root causes is the only way to ultimately fix this.
Moving away from a time-based schedule is just going to encourage the
schedule to slip further, because now you have to wait [potentially
forever] to get a change released rather than roughly [x months], which
leads to even more pressure to get changes in an earlier version.

Ultimately, I think we need to be better at saying "no" to changes that
aren't explicitly necessary, especially on version branches (5.x.y) and
in general taking care to evaluate how "done" something is before taking
it in too. This is not an exact science, though, which makes it
difficult to balance. Even if we truly are on the side of being too
aggressive now - it may be perfectly possible that we could end up too
conservative (and delaying fixes that our end users would like to see
sooner), e.g. by implementing some kind of "blanket" restriction like
"only P0 bugs get fixed in this branch" (not a suggestion I am making I
will emphasise, just a note that we need to approach this carefully).

To the proposal at hand: I think that we should not be afraid of
releasing more release candidates. Some of them may well be broken, and
while that's unfortunate, it's still certainly better that we release
broken release candidates than a broken final release in the end.

It may be the case that releasing more betas or using a different name
may help get feedback earlier, but IMHO we have to accept to some extent
that our end users are not us: they have limited time to build & test Qt
in addition to their normal product/application work, and thus, overall,
we should not expect too much of them. If we want more feedback, we need
to work on making that testing process easier for them, rather than
playing naming tricks, I think.

On Tue, Dec 20, 2016, at 02:34 PM, Tuukka Turunen wrote:
> 
> Hi,
> 
> I think we have three major problems with our current release process
> regarding the release candidate phase:
> 
> 1.       Process to make a RC that is as flawless as final causes
> inefficiency as we only get full test coverage with the RC itself
> 
> 2.       We get full attention for testing a bit too late, many fixes are
> still coming in close to the planned RC time causing instability
> 
> 3.       Current time between RC and final is planned to be 2 weeks,
> which is very little in order to take in the feedback and fix things
> 
> Therefore, I would like to propose the following:
> 
> a.       Consider "Release Candidate" to be a phase rather than an
> individual delivery
> 
> b.       Create the first "RC1" almost immediately after release branch
> (e.g. 5.9.0) is operational
> 
> c.       Criteria for the "RC1" is that no known P0 bugs exist (i.e.
> there can be other issues that would not be acceptable in a final
> release)
> 
> d.       During the "RC" phase P1 (and possible P0 of course) bugs and
> documentation are fixed
> 
> e.       Public "RC" release is similar development release as Alpha and
> Beta in that it starts a phase of work
> 
> f.        Multiple snapshots / new candidates are created during the "RC"
> phase until one of them is considered the final release
> 
> If desired, we could use some other name than "Release Candidate 1" for
> the release that begins the last phase of the release. It could be called
> "Beta 2" or "Technology preview", if so desired. Personally, I would call
> it "Release Candidate 1".
> 
> The difference to our current process is quite small. In essence it would
> be about considering the "RC1" the beginning of the final releasing phase
> (.0 branch), not something we do almost at the end of it. I believe that
> lowering the quality criterial for "RC1" helps us in being more efficient
> as it has been in practice impossible to really fulfill the current
> process goal and have already the first RC as good as the final.
> 
> In case of Qt 5.9 it would mean that we have the first "RC" out around
> end of April, soon after the branching to 5.9.0 has been completed. We
> then have 4 or so weeks to make all the needed amount of candidates /
> snapshots until one of them will be released as Qt 5.9.0 final. If it
> happens earlier than in 4 weeks, great. If it takes more time, then so be
> it (although in such case we have probably missed something in the
> earlier steps of the release creation).
> 
> Yours,
> 
> ---
> Tuukka Turunen
> Director, R&D
> 
> The Qt Company
> Lutakonaukio 1
> 40100 Jyväskylä, Finland
> tuukka.turunen at qt.io
> +358 40 7655 800
> http://qt.io
> ---
> 
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development