[Development] Proposal to adjust release candidate process

Fri Dec 23 08:33:24 CET 2016

> -----Original Message-----
> From: Development [mailto:development-
> bounces+tuukka.turunen=qt.io at qt-project.org] On Behalf Of Robin Burchell
> Sent: perjantaina 23. joulukuuta 2016 3.40
> To: development at qt-project.org
> Subject: Re: [Development] Proposal to adjust release candidate process
> 
> My (slightly delayed) $0.02:
> 
> On releasing in general: I would agree that releases take too long, often, but I
> am of the belief that a good part of this is a mess of our own making. For
> instance, due to an end-user or project requiring a specific version or purely
> out of fear that the next release may be a long time away, I think we
> (collectively) often end up pushing changes to branches that might be less
> than ideally suited for the change in question.
> 

I think this is true and something we need to address. Another symptom of this is the time we reserve between feature freeze and Alpha. Why does it always take many weeks (4 is reserved in the current release process http://wiki.qt.io/Qt5Releasing). There is still a lot of commits coming during this phase, perhaps some of them are such that actually should have been in before the FF. I would like us to reach a state where Alpha is released almost immediately (within 1 week) of the Feature freeze. I know that we are not yet there, but in order to reduce the overall release time that is necessary. 

> I think we should recognise that these pressures do exist, and consider ways
> we can work to mitigate them (things like: nightly or at least semi-"nightly"
> builds? working on pushing out patch releases more often?
> encouraging more of our consumers to apply patches that we already took in
> upstream and do their own builds? anything else?)
> 
> It goes without saying that these are hard problems, but I think that working
> at the root causes is the only way to ultimately fix this.
> Moving away from a time-based schedule is just going to encourage the
> schedule to slip further, because now you have to wait [potentially forever]
> to get a change released rather than roughly [x months], which leads to even
> more pressure to get changes in an earlier version.
> 

We do not want to move away from the time based releasing, just trying to become better in 1. keeping the schedule and after that 2. reducing the time it takes to make a feature release of Qt.

> Ultimately, I think we need to be better at saying "no" to changes that aren't
> explicitly necessary, especially on version branches (5.x.y) and in general
> taking care to evaluate how "done" something is before taking it in too. This
> is not an exact science, though, which makes it difficult to balance. Even if we
> truly are on the side of being too aggressive now - it may be perfectly
> possible that we could end up too conservative (and delaying fixes that our
> end users would like to see sooner), e.g. by implementing some kind of
> "blanket" restriction like "only P0 bugs get fixed in this branch" (not a
> suggestion I am making I will emphasise, just a note that we need to
> approach this carefully).
> 

This, I think, especially has been an issue with some of our patch level releases lately. While it sounds good to have fixes, too big amount of fixes is prone to cause instability - and certainly it increases the effort of making the patch releases.

> To the proposal at hand: I think that we should not be afraid of releasing
> more release candidates. Some of them may well be broken, and while that's
> unfortunate, it's still certainly better that we release broken release
> candidates than a broken final release in the end.
> 

Thanks, fully agree. Our users are intelligent people. I do not think anyone thinks such an early preview releases can be used in production (or if they would, then there is a good reason).

> It may be the case that releasing more betas or using a different name may
> help get feedback earlier, but IMHO we have to accept to some extent that
> our end users are not us: they have limited time to build & test Qt in addition
> to their normal product/application work, and thus, overall, we should not
> expect too much of them. If we want more feedback, we need to work on
> making that testing process easier for them, rather than playing naming
> tricks, I think.
> 

Actually the problem currently has not so much been lack of feedback, but the fact that feedback comes so late in the release phase that it is hard to react to it properly. I hope we could slightly ease this by having the first release from release branch earlier than we now do. 

Yours,

	Tuukka

> On Tue, Dec 20, 2016, at 02:34 PM, Tuukka Turunen wrote:
> >
> > Hi,
> >
> > I think we have three major problems with our current release process
> > regarding the release candidate phase:
> >
> > 1.       Process to make a RC that is as flawless as final causes
> > inefficiency as we only get full test coverage with the RC itself
> >
> > 2.       We get full attention for testing a bit too late, many fixes are
> > still coming in close to the planned RC time causing instability
> >
> > 3.       Current time between RC and final is planned to be 2 weeks,
> > which is very little in order to take in the feedback and fix things
> >
> > Therefore, I would like to propose the following:
> >
> > a.       Consider "Release Candidate" to be a phase rather than an
> > individual delivery
> >
> > b.       Create the first "RC1" almost immediately after release branch
> > (e.g. 5.9.0) is operational
> >
> > c.       Criteria for the "RC1" is that no known P0 bugs exist (i.e.
> > there can be other issues that would not be acceptable in a final
> > release)
> >
> > d.       During the "RC" phase P1 (and possible P0 of course) bugs and
> > documentation are fixed
> >
> > e.       Public "RC" release is similar development release as Alpha and
> > Beta in that it starts a phase of work
> >
> > f.        Multiple snapshots / new candidates are created during the "RC"
> > phase until one of them is considered the final release
> >
> > If desired, we could use some other name than "Release Candidate 1"
> > for the release that begins the last phase of the release. It could be
> > called "Beta 2" or "Technology preview", if so desired. Personally, I
> > would call it "Release Candidate 1".
> >
> > The difference to our current process is quite small. In essence it
> > would be about considering the "RC1" the beginning of the final
> > releasing phase
> > (.0 branch), not something we do almost at the end of it. I believe
> > that lowering the quality criterial for "RC1" helps us in being more
> > efficient as it has been in practice impossible to really fulfill the
> > current process goal and have already the first RC as good as the final.
> >
> > In case of Qt 5.9 it would mean that we have the first "RC" out around
> > end of April, soon after the branching to 5.9.0 has been completed. We
> > then have 4 or so weeks to make all the needed amount of candidates /
> > snapshots until one of them will be released as Qt 5.9.0 final. If it
> > happens earlier than in 4 weeks, great. If it takes more time, then so
> > be it (although in such case we have probably missed something in the
> > earlier steps of the release creation).
> >
> > Yours,
> >
> > ---
> > Tuukka Turunen
> > Director, R&D
> >
> > The Qt Company
> > Lutakonaukio 1
> > 40100 Jyväskylä, Finland
> > tuukka.turunen at qt.io
> > +358 40 7655 800
> > http://qt.io
> > ---
> >
> > _______________________________________________
> > Development mailing list
> > Development at qt-project.org
> > http://lists.qt-project.org/mailman/listinfo/development
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development