[Releasing] Avoiding platform test coverage regressions

[Tuukka Turunen]

> -----Original Message-----
> From: Releasing [mailto:releasing-bounces+tuukka.turunen=qt.io at qt-
> project.org] On Behalf Of Frederik Gladhorn
> Sent: torstaina 26. tammikuuta 2017 14.59
> To: releasing at qt-project.org
> Subject: Re: [Releasing] Avoiding platform test coverage regressions
> 
> To me new platforms including testing on them are features.
> Either they make feature freeze or not. Let's either postpone the feature
> freeze or not add them past the poing of the feature freeze.
> We all agreed we want a strict feature freeze, so let's not make exceptions
> here. This is exactly why I don't think 5.9 will be out according to schedule.

I do agree about this in principle, but we can also think it the other way around for the new version of an already supported platform: If the code already works on a new platform version and we know users are using it, should we increase the amount of automated testing even beyond FF if it is possible? For example macOS 10.12 - if we fail to add it by 5.9 FF but can add it after without affecting the release schedule, should we do it? I think we should. But if we consider it to be a blocker for the release, then it should be there already at the FF. 

Similarly, I think having first the build enforced in CI and then adding the tests step by step is also a viable option. Of course we could take a stand that we block a release of Qt if we can not make it fully tested of a new version of an important platform, but to me that would feel a drastic measure. 

I would like us to add a new release process step e.g. 4 weeks before FF to have all the supported platforms in place and also all new modules (including possible TP modules) in place. This would help reach FF in time and lower the risk to overall release schedule.

Yours,

	Tuukka