[Development] Proposal to adjust release candidate process

[Sean Harmer]

On Tuesday 20 December 2016 15:14:17 Sean Harmer wrote:
> Hi Tuukka,
> 
> I agree with your proposal, however I think there is also another issue or
> two that we should address.

Except that instead of calling it RC1 why not call it another beta? The RC 
should be something that *may* be suitable for release. But at present we get 
the RC too late so it gets difficult to get fixes for issues accepted. Sure we 
might need a new RC after that but calling the initial packages after the 
release branch is branched an RC seems like a misnomer.

Cheers,

Sean

> 
> At present we have ~ 4 releases per year 2 minor versions and 2 further
> patch releases. One issue is that it looks like 5.8.0 will soon render the
> very new 5.7.1 release redundant. With that in mind was it worth splitting
> the effort and having the 5.7.1 release at all? Don't get me wrong I'm all
> for additional patch releases just that I'd hope they wouldn't coincide so
> closely with a minor version release.
> 
> This brings me onto another issue which perhaps we feel more strongly than
> average in Qt 3D as a new module that is undergoing rapid bug-fixing,
> improvements and with lots of new feature development starting to open up.
> For example, although Qt 3D in Qt 5.7.0 was a nice release after lots of
> effort, we still fixed a huge number of issues for 5.7.1, which took 181
> days to release from 5.7.0.
> 
> Would it be possible for us to release more frequent patch releases? If not
> for the whole of Qt, then at least for addon modules such as Qt 3D? The
> latter would require splitting the packaging of such modules but would
> reduce the amount of overall testing required for a new patch release. If
> we'd collectively rather not go down that street I still think more
> frequent patch releases would be nice. Users have the option of upgrading
> or not if they want to reduce their own internal regression testing burden
> and freeze on a specific version. But it would have the benefit that other
> users don't need to wait 6 months for a set of bug fixes - roughly the same
> time to wait as for new features.
> 
> I wonder how much of the current pressure on releases is driven by the time-
> based release policy. We aim for a new minor release every 6 months which
> is admirable. But I wonder about the consequences of trying to stick to
> that at the expense of quality of the 5.x.0 releases, especially given the
> long turn around for subsequent patch releases.
> 
> With the above in mind, how about this proposal in addition to yours
> regarding the RC phase?
> 
> * Branch off new feature branch every 6 months as at present.
> 
> * Do not rush the release at the expense of quality or when it's convenient
> due to packagers going on vacation etc. We release only when the release is
> deemed to meet quality standards.
> 
> * Aim for a patch release every alternate month if there are fixes on the
> minor version branch and packaging/release staff are available. i.e. don't
> try to force one out before the summer/Xmas vacations.
> 
> * Consider releasing at least one patch release of the previous minor series
> after the release of the next minor series' .0 release. For example, in the
> current situation, potentially release a 5.7.2 after 5.8.0. (We already
> have fixes on 5.8.0 and 5.8 that could have been submitted to the 5.7
> branch if there were to be a 5.7.2 release).
> 
> Additionally, is there any way that contributors outside of The Qt Company
> can assist with the practicalities of packaging?
> 
> Just some food for thought.
> 
> Cheers,
> 
> Sean
> 
> On Tuesday 20 December 2016 13:34:39 Tuukka Turunen wrote:
> > Hi,
> > 
> > I think we have three major problems with our current release process
> > regarding the release candidate phase:
> > 
> > 1.       Process to make a RC that is as flawless as final causes
> > inefficiency as we only get full test coverage with the RC itself
> > 
> > 2.       We get full attention for testing a bit too late, many fixes are
> > still coming in close to the planned RC time causing instability
> > 
> > 3.       Current time between RC and final is planned to be 2 weeks, which
> > is very little in order to take in the feedback and fix things
> > 
> > Therefore, I would like to propose the following:
> > 
> > a.       Consider "Release Candidate" to be a phase rather than an
> > individual delivery
> > 
> > b.       Create the first "RC1" almost immediately after release branch
> > (e.g. 5.9.0) is operational
> > 
> > c.       Criteria for the "RC1" is that no known P0 bugs exist (i.e. there
> > can be other issues that would not be acceptable in a final release)
> > 
> > d.       During the "RC" phase P1 (and possible P0 of course) bugs and
> > documentation are fixed
> > 
> > e.       Public "RC" release is similar development release as Alpha and
> > Beta in that it starts a phase of work
> > 
> > f.        Multiple snapshots / new candidates are created during the "RC"
> > phase until one of them is considered the final release
> > 
> > If desired, we could use some other name than "Release Candidate 1" for
> > the
> > release that begins the last phase of the release. It could be called
> > "Beta
> > 2" or "Technology preview", if so desired. Personally, I would call it
> > "Release Candidate 1".
> > 
> > The difference to our current process is quite small. In essence it would
> > be about considering the "RC1" the beginning of the final releasing phase
> > (.0 branch), not something we do almost at the end of it. I believe that
> > lowering the quality criterial for "RC1" helps us in being more efficient
> > as it has been in practice impossible to really fulfill the current
> > process goal and have already the first RC as good as the final.
> > 
> > In case of Qt 5.9 it would mean that we have the first "RC" out around end
> > of April, soon after the branching to 5.9.0 has been completed. We then
> > have 4 or so weeks to make all the needed amount of candidates / snapshots
> > until one of them will be released as Qt 5.9.0 final. If it happens
> > earlier
> > than in 4 weeks, great. If it takes more time, then so be it (although in
> > such case we have probably missed something in the earlier steps of the
> > release creation).
> > 
> > Yours,
> > 
> > ---
> > Tuukka Turunen
> > Director, R&D
> > 
> > The Qt Company
> > Lutakonaukio 1
> > 40100 Jyväskylä, Finland
> > tuukka.turunen at qt.io
> > +358 40 7655 800
> > http://qt.io
> > ---

-- 
Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
KDAB (UK) Ltd, a KDAB Group company
Tel. +44 (0)1625 809908; Sweden (HQ) +46-563-540090
Mobile: +44 (0)7545 140604
KDAB - Qt Experts