[Development] Proposal to adjust release candidate process

[Sean Harmer]

Hi Tuukka,

I agree with your proposal, however I think there is also another issue or two 
that we should address.

At present we have ~ 4 releases per year 2 minor versions and 2 further patch 
releases. One issue is that it looks like 5.8.0 will soon render the very new 
5.7.1 release redundant. With that in mind was it worth splitting the effort 
and having the 5.7.1 release at all? Don't get me wrong I'm all for additional 
patch releases just that I'd hope they wouldn't coincide so closely with a 
minor version release.

This brings me onto another issue which perhaps we feel more strongly than 
average in Qt 3D as a new module that is undergoing rapid bug-fixing, 
improvements and with lots of new feature development starting to open up. For 
example, although Qt 3D in Qt 5.7.0 was a nice release after lots of effort, 
we still fixed a huge number of issues for 5.7.1, which took 181 days to 
release from 5.7.0.

Would it be possible for us to release more frequent patch releases? If not 
for the whole of Qt, then at least for addon modules such as Qt 3D? The latter 
would require splitting the packaging of such modules but would reduce the 
amount of overall testing required for a new patch release. If we'd 
collectively rather not go down that street I still think more frequent patch 
releases would be nice. Users have the option of upgrading or not if they want 
to reduce their own internal regression testing burden and freeze on a specific 
version. But it would have the benefit that other users don't need to wait 6 
months for a set of bug fixes - roughly the same time to wait as for new 
features.

I wonder how much of the current pressure on releases is driven by the time-
based release policy. We aim for a new minor release every 6 months which is 
admirable. But I wonder about the consequences of trying to stick to that at 
the expense of quality of the 5.x.0 releases, especially given the long turn 
around for subsequent patch releases.

With the above in mind, how about this proposal in addition to yours regarding 
the RC phase?

* Branch off new feature branch every 6 months as at present.

* Do not rush the release at the expense of quality or when it's convenient 
due to packagers going on vacation etc. We release only when the release is 
deemed to meet quality standards.

* Aim for a patch release every alternate month if there are fixes on the minor 
version branch and packaging/release staff are available. i.e. don't try to 
force one out before the summer/Xmas vacations.

* Consider releasing at least one patch release of the previous minor series 
after the release of the next minor series' .0 release. For example, in the 
current situation, potentially release a 5.7.2 after 5.8.0. (We already have 
fixes on 5.8.0 and 5.8 that could have been submitted to the 5.7 branch if 
there were to be a 5.7.2 release).

Additionally, is there any way that contributors outside of The Qt Company can 
assist with the practicalities of packaging?

Just some food for thought.

Cheers,

Sean

On Tuesday 20 December 2016 13:34:39 Tuukka Turunen wrote:
> Hi,
> 
> I think we have three major problems with our current release process
> regarding the release candidate phase:
> 
> 1.       Process to make a RC that is as flawless as final causes
> inefficiency as we only get full test coverage with the RC itself
> 
> 2.       We get full attention for testing a bit too late, many fixes are
> still coming in close to the planned RC time causing instability
> 
> 3.       Current time between RC and final is planned to be 2 weeks, which
> is very little in order to take in the feedback and fix things
> 
> Therefore, I would like to propose the following:
> 
> a.       Consider "Release Candidate" to be a phase rather than an
> individual delivery
> 
> b.       Create the first "RC1" almost immediately after release branch
> (e.g. 5.9.0) is operational
> 
> c.       Criteria for the "RC1" is that no known P0 bugs exist (i.e. there
> can be other issues that would not be acceptable in a final release)
> 
> d.       During the "RC" phase P1 (and possible P0 of course) bugs and
> documentation are fixed
> 
> e.       Public "RC" release is similar development release as Alpha and
> Beta in that it starts a phase of work
> 
> f.        Multiple snapshots / new candidates are created during the "RC"
> phase until one of them is considered the final release
> 
> If desired, we could use some other name than "Release Candidate 1" for the
> release that begins the last phase of the release. It could be called "Beta
> 2" or "Technology preview", if so desired. Personally, I would call it
> "Release Candidate 1".
> 
> The difference to our current process is quite small. In essence it would be
> about considering the "RC1" the beginning of the final releasing phase (.0
> branch), not something we do almost at the end of it. I believe that
> lowering the quality criterial for "RC1" helps us in being more efficient
> as it has been in practice impossible to really fulfill the current process
> goal and have already the first RC as good as the final.
> 
> In case of Qt 5.9 it would mean that we have the first "RC" out around end
> of April, soon after the branching to 5.9.0 has been completed. We then
> have 4 or so weeks to make all the needed amount of candidates / snapshots
> until one of them will be released as Qt 5.9.0 final. If it happens earlier
> than in 4 weeks, great. If it takes more time, then so be it (although in
> such case we have probably missed something in the earlier steps of the
> release creation).
> 
> Yours,
> 
> ---
> Tuukka Turunen
> Director, R&D
> 
> The Qt Company
> Lutakonaukio 1
> 40100 Jyväskylä, Finland
> tuukka.turunen at qt.io
> +358 40 7655 800
> http://qt.io
> ---

-- 
Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
KDAB (UK) Ltd, a KDAB Group company
Tel. +44 (0)1625 809908; Sweden (HQ) +46-563-540090
Mobile: +44 (0)7545 140604
KDAB - Qt Experts