[Releasing] [Development] Focusing bug fixes to 5.9 branch and patch releases during H1/17

[Tuukka Turunen]

Hi Sean et al,

First I want to say that I am also very sorry that we need to skip 5.8.1. Unfortunately I do not see any other approach that would allow us to: 1) catch the delay caused by 5.8 being late (and 5.7 being late, and 5.6 being late) and 2) enable implementation the much needed CI changes in good time before Qt 5.10 feature freeze.

We tried really hard to keep somehow Qt 5.8 schedule or even get close to it, but the fact that we were maintaining 4 active branches (5.6, 5.7, 5.8 and dev) and feature freeze of 5.8 being late for some items seen critical to be in made it impossible to be faster. 

What could have been done, is to see this coming and plan 5.9 release for H2/17. But it was decided to keep the 6 month cycle and therefore development of 5.9 features was completed with the original schedule. New platforms to CI missed this due to 5.8 delay, but otherwise features were completed in time. Now that the FF is reached we should be as fast as possible to get the release out. We should hopefully be even faster than the planned 17 weeks. Getting the Alpha out next week would be a great start.

As I wrote earlier, we have many much awaited changes coming to CI this year. These will help capacity definitely and stability hopefully. In addition, we are putting more focus into fixing the flaky cases, which has been a constant effort, but still an item that causes failed runs quite frequently. These changes need to go in well before the Qt 5.10 FF and due to summer holidays in June-August timeframe, we prefer June for most to put them into production. 

So due to these reasons postponing Qt 5.9 to June or July is not feasible. Making the first patch release is expected to take 4-6 weeks effort from release team, machines and also many developers. The tests run in the CI do not catch all issues and there will be multiple rounds needed to verify a new release on all our supported platforms. So, if we would do 5.8.1 now it would mean delay of 4-6 weeks to 5.9 schedule, and thus put the CI changes at risk due to vacation period and Qt 5.10 FF approaching. Even if there would be much higher amount of volunteers from the community to help in making 5.8.1 release, it would still cause delay to 5.9 schedule because of the load to the system and necessity of having The Qt Company developers and release engineers participate. It is a bit work to make even a patch release of Qt and we need to ensure high quality with every release we do.

I think making just 1 patch release of 5.7 and none for 5.8 is really bad. Even with the help from our support team and possibility to work around issues. I would like us to use the time we now have to make sure Qt 5.9.0 is a solid release and that it is easy to make Qt 5.9.x patch releases during H2/17 with the CI improvements in place. 

Yours,

	Tuukka

On 15/02/2017, 11.36, "sean.harmer on behalf of Sean Harmer" <sean.harmer at kdab.com> wrote:

    First of all, apologies for not being able to make the release meeting 
    yesterday. I was in a workshop all day.
    
    For the record I think skipping 5.8.1 is a big mistake. I would much rather 
    delay 5.9 by a few weeks and have a 5.8.1 release out than skip it and try for 
    a quick 5.9.0.
    
    Why would releasing 5.8.1 take as long as a .0 release if nothing much has 
    changed in the packaging and build system? I think our users, your paying 
    customers, would rather have a .1 release than another .0. In our experience, 
    many customers explicitly do not use .0 releases. So for these we are 
    condemning them to wait 3/4 of a year or so to be able to go from 5.7.1 to 
    5.9.1.
    
    Imho, a bug fix release should take priority over a feature release. Also, the 
    number of downloads is no measure of the quality of a release.
    
    There are plenty of fixes available on the 5.8 branch across pretty much all 
    modules ready to go now. I appreciate the packages will still need testing, 
    but surely this is mainly functional testing for 5.8.1 given the packaging 
    should not have changed since 5.8.0. We could branch 5.8.1 now and be 
    generating the first set of test packages before the 5.9 beta.
    
    The reason of not wanting to do a bug fix release in conjunction with a feature 
    release due to limited resources is particularly galling when we read 
    yesterday that TQC have decided to release an *old* 5.5.1 release for Vx 
    Works. Those resources could have been used towards a Qt 5.8.1 over the last 
    month.
    
    There is still a very real risk that we will be late with the 5.9.0 release 
    given the large changes to the configuration system in 5.9. Having a 5.8.1 
    release is a way to mitigate that risk too.
    
    It wasn't so long ago Tuukka, that you wanted to do a quick 5.8.1 release 
    after there had been such a huge delay between 5.7.0 and 5.7.1. Please do not 
    go back on this and let the users have a bug fix release. Are there really 
    features in 5.9 that customers cannot live without for an extra 4-6 weeks 
    whilst we prepare a 5.8.1 release? I very much doubt it.
    
    Kind regards,
    
    Sean
    
    On Friday 10 February 2017 10:59:56 Tuukka Turunen wrote:
    > Hi,
    > 
    > Short summary: The Qt Company has decided to focus development effort to 5.9
    > and dev branches in order to reach the planned timeline for Qt 5.9 release
    > and make improvements in the CI system. Next scheduled patch level release
    > would be Qt 5.6.3 straight after Qt 5.9 is out. In case of severe security
    > vulnerability we would make Qt 5.8.1 directly based on 5.8.0 with just the
    > security fix(es) added.
     
    > Qt 5.8.0 looks to be a good and successful release. It has already been
    > downloaded 150.000 times even though it was released just two weeks ago.
    > The reception has been good overall and there are no blockers reported
    > based on the release. Qt 5.9 and subsequent releases will leverage the new
    > configuration system and other major improvements introduced with it, as
    > well as add new functionality to make Qt even better.
     
    > It took us a more time than planned to get Qt 5.8 out and because of that we
    > are already past Qt 5.9 feature freeze and approaching the alpha release
    > soon. Looking into the reasons why Qt 5.8.0 was delayed there are 3 items
    > that had a significant effect:
     1. We did not keep the feature freeze
    > properly as there were some items we wanted to get in – doing so we should
    > have replanned the whole release time, but we tried to keep schedule and
    > failed. 2. We were not able to focus enough to Qt 5.8 finalization as there
    > was quite a lot of effort going still to Qt 5.6.x and 5.7.x patch releases
    > in parallel – and our systems and processes are not yet good enough for
    > this. 3. We have had some issues in the CI system that cause integrations
    > to fail and thus increase system load – these are now being addressed
    > (getting new HW in place, change to a better virtualization platform, use
    > local disks instead of central disk, reduce number of flaky tests, etc). 
    > We believe it will be possible to significantly reduce the time from feature
    > freeze to release. To reach this we need to improve the items that have
    > been problematic with Qt 5.8 and other releases. After we have done the
    > improvements, it will be easier to keep the timelines and also be more
    > productive. We absolutely do not want to rush a new release out with severe
    > bugs in it – not now and not in the future – for this reason we want to
    > focus into the processes and systems part and allow more efficient release
    > creation. First we want to be able to reach the current 17 weeks timeline
    > from FF to release with Qt 5.9. After that we want to reduce it gradually
    > to 10-12 weeks (for a new minor release of Qt). Increased productivity will
    > allow us to make more patch releases than before.
     
    > In the CI system side we have major improvements happening during this year.
    > We will be purchasing more blades and other HW during H1/17, which will add
    > capacity, so we will be able to run more parallel integrations. We are also
    > looking into changing the system architecture away from central disk into
    > using local ssd for build machines. This is expected to bring improvements
    > to both performance and reliability (especially now as we have had issues
    > with disks failing despite the disk system still being in the mid of its
    > life-cycle). Third item we are looking into is change of the virtualization
    > platform, which we hope will bring improvements in stability as well as new
    > capabilities.
     
    > Arranging enough time to implement the improvements in our systems as well
    > as keeping the timeline of Qt 5.9 despite delayed 5.8 release unfortunately
    > means that we can’t make any scheduled patch releases until June, after Qt
    > 5.9 is out. We will keep the ability to release patch release for urgent
    > security vulnerabilities, as always. Such release would be on top of the
    > previous patch release and not include all other changes in the
    > corresponding branch. Even with a lot more help from community than before,
    > making a scheduled Qt 5.8.1 release parallel with Qt 5.9 has impact to the
    > system load as well as to the work needed from the release team and
    > others.
     
    > I am well aware that a Qt 5.8.1 release would be beneficial, but making one
    > now would impact Qt 5.9 schedule and our capability to improve CI system
    > stability. Qt 5.6.3 LTS patch release is done straight after Qt 5.9
    > release, but not before. After the Qt 5.9.0 in May we should then focus
    > into making Qt 5.9.1 release instead of making Qt 5.8.1 any more as Qt
    > 5.9.0 is already out. With the CI improvements in place, we should
    > definitely be able to make more than one patch release for Qt 5.9.
     
    > As a scheduled Qt 5.8.1 release is not planned, should we start pushing
    > fixes directly to 5.9 branch now? It would reduce the workload as we do not
    > need to first integrate changes into 5.8 and then merge upwards to 5.9. It
    > would also reduce the time it takes to get fixes into to 5.9 and dev as
    > well as reduce the load to the CI system as less integrations are needed.
    > We can keep 5.8 branch open for a while still similarly as 5.6 is in case
    > there is some fix that must be in the 5.8 branch. Let’s discuss in the next
    > release team meeting (Tuesday 14th Feb 16.00 CET) if pushing fixes to 5.9
    > branch directly would be good approach.
     
    > Yours,
    > 
    >                 Tuukka
    > 
    > 
    > 
    > 
    > 
    
    -- 
    Dr Sean Harmer | sean.harmer at kdab.com | Managing Director UK
    KDAB (UK) Ltd, a KDAB Group company
    Tel. +44 (0)1625 809908; Sweden (HQ) +46-563-540090
    Mobile: +44 (0)7545 140604
    KDAB - Qt Experts