[Development] Proposal to adjust release candidate process

[Tuukka Turunen]

> -----Original Message-----
> From: sean.harmer On Behalf Of Sean Harmer
> Sent: tiistaina 20. joulukuuta 2016 17.14
> To: development at qt-project.org
> Cc: Tuukka Turunen <tuukka.turunen at qt.io>; releasing at qt-project.org
> Subject: Re: [Development] Proposal to adjust release candidate process
> 
> Hi Tuukka,
> 
> I agree with your proposal, however I think there is also another issue or two
> that we should address.
> 
> At present we have ~ 4 releases per year 2 minor versions and 2 further
> patch releases. One issue is that it looks like 5.8.0 will soon render the very
> new
> 5.7.1 release redundant. With that in mind was it worth splitting the effort
> and having the 5.7.1 release at all? Don't get me wrong I'm all for additional
> patch releases just that I'd hope they wouldn't coincide so closely with a
> minor version release.
> 

Did we need Qt 5.7.1 in my opinion, yes we did. Was it too late, yes it was. We can not change the past, but for Qt 5.8, I would like to branch 5.8.1 quite soon after the 5.8.0 is released. This way there will be more time between 5.8.1 and 5.9.0 releases. There are already quite many changes in 5.8 branch that are not in 5.8.0 branch.

> This brings me onto another issue which perhaps we feel more strongly than
> average in Qt 3D as a new module that is undergoing rapid bug-fixing,
> improvements and with lots of new feature development starting to open
> up. For example, although Qt 3D in Qt 5.7.0 was a nice release after lots of
> effort, we still fixed a huge number of issues for 5.7.1, which took 181 days to
> release from 5.7.0.
> 
> Would it be possible for us to release more frequent patch releases? If not
> for the whole of Qt, then at least for addon modules such as Qt 3D? 

I would like to have more patch level releases. Perhaps not between the feature releases, but to continue patch releases of Qt 5.x also after 5.x+1 is released. Currently this has not been feasible, but it is something I am interested in for 5.9 (after we have completed the big CI HW and virtualization infrastructure change and relocated the machine room).

>The
> latter would require splitting the packaging of such modules but would
> reduce the amount of overall testing required for a new patch release. If
> we'd collectively rather not go down that street I still think more frequent
> patch releases would be nice. Users have the option of upgrading or not if
> they want to reduce their own internal regression testing burden and freeze
> on a specific version. But it would have the benefit that other users don't
> need to wait 6 months for a set of bug fixes - roughly the same time to wait
> as for new features.
> 
> I wonder how much of the current pressure on releases is driven by the
> time- based release policy. We aim for a new minor release every 6 months
> which is admirable. But I wonder about the consequences of trying to stick to
> that at the expense of quality of the 5.x.0 releases, especially given the long
> turn around for subsequent patch releases.
> 

I think the two biggest reasons to the pressure are: 1. We end of having too big amount of changes in a patch release and 2. Test automation does not yet catch enough issues on embedded and mobile, and we need to use slow manual testing a lot.

Number #1 is something we can fix by our processes and behavior. It is also a "self filling prophecy" as we try to push a lot of things in to current release because it may take long time before next patch release is coming - thus making it more difficult to make patch releases. 

Number #2 is a continuous effort by our release and QA team and we are already quite good on desktop for automated package testing.

> With the above in mind, how about this proposal in addition to yours
> regarding the RC phase?
> 
> * Branch off new feature branch every 6 months as at present.
> 
> * Do not rush the release at the expense of quality or when it's convenient
> due to packagers going on vacation etc. We release only when the release is
> deemed to meet quality standards.
> 

We already try not to rush on expense of quality. I do think the current ~17 weeks is too long time. It may be so long already that some lose focus. I would like to get it shorter, 10 or 12 weeks would be good in my opinion. The first step to reach this is to keep the feature freeze (no exceptions to complete some feature weeks after the FF).

> * Aim for a patch release every alternate month if there are fixes on the
> minor version branch and packaging/release staff are available. i.e. don't try
> to force one out before the summer/Xmas vacations.
> 
> * Consider releasing at least one patch release of the previous minor series
> after the release of the next minor series' .0 release. For example, in the
> current situation, potentially release a 5.7.2 after 5.8.0. (We already have
> fixes on 5.8.0 and 5.8 that could have been submitted to the 5.7 branch if
> there were to be a 5.7.2 release).
> 

I think this would be very good for our users and something I would like to reach for 5.9 some way (see above for the reason why not yet)

> Additionally, is there any way that contributors outside of The Qt Company
> can assist with the practicalities of packaging?
> 

Most important thing from wider community is testing, reporting and fixing the found issues. From maintainers getting the changes files in early and being active to freeze the modules early etc is extremely valuable. The sooner we identify and fix issues, the better it is. 

Yours,

	Tuukka

> Just some food for thought.
> 
> Cheers,
> 
> Sean
> 
> On Tuesday 20 December 2016 13:34:39 Tuukka Turunen wrote:
> > Hi,
> >
> > I think we have three major problems with our current release process
> > regarding the release candidate phase:
> >
> > 1.       Process to make a RC that is as flawless as final causes
> > inefficiency as we only get full test coverage with the RC itself
> >
> > 2.       We get full attention for testing a bit too late, many fixes are
> > still coming in close to the planned RC time causing instability
> >
> > 3.       Current time between RC and final is planned to be 2 weeks, which
> > is very little in order to take in the feedback and fix things
> >
> > Therefore, I would like to propose the following:
> >
> > a.       Consider "Release Candidate" to be a phase rather than an
> > individual delivery
> >
> > b.       Create the first "RC1" almost immediately after release branch
> > (e.g. 5.9.0) is operational
> >
> > c.       Criteria for the "RC1" is that no known P0 bugs exist (i.e. there
> > can be other issues that would not be acceptable in a final release)
> >
> > d.       During the "RC" phase P1 (and possible P0 of course) bugs and
> > documentation are fixed
> >
> > e.       Public "RC" release is similar development release as Alpha and
> > Beta in that it starts a phase of work
> >
> > f.        Multiple snapshots / new candidates are created during the "RC"
> > phase until one of them is considered the final release
> >
> > If desired, we could use some other name than "Release Candidate 1"
> > for the release that begins the last phase of the release. It could be
> > called "Beta 2" or "Technology preview", if so desired. Personally, I
> > would call it "Release Candidate 1".
> >
> > The difference to our current process is quite small. In essence it
> > would be about considering the "RC1" the beginning of the final
> > releasing phase (.0 branch), not something we do almost at the end of
> > it. I believe that lowering the quality criterial for "RC1" helps us
> > in being more efficient as it has been in practice impossible to
> > really fulfill the current process goal and have already the first RC as good as
> the final.
> >
> > In case of Qt 5.9 it would mean that we have the first "RC" out around
> > end of April, soon after the branching to 5.9.0 has been completed. We
> > then have 4 or so weeks to make all the needed amount of candidates /
> > snapshots until one of them will be released as Qt 5.9.0 final. If it
> > happens earlier than in 4 weeks, great. If it takes more time, then so
> > be it (although in such case we have probably missed something in the
> > earlier steps of the release creation).
> >
> > Yours,
> >
> > ---
> > Tuukka Turunen
> > Director, R&D
> >
> > The Qt Company
> > Lutakonaukio 1
> > 40100 Jyväskylä, Finland
> > tuukka.turunen at qt.io
> > +358 40 7655 800
> > http://qt.io
> > ---
> 
> --
> Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK KDAB
> (UK) Ltd, a KDAB Group company Tel. +44 (0)1625 809908; Sweden (HQ) +46-
> 563-540090
> Mobile: +44 (0)7545 140604
> KDAB - Qt Experts