[Development] Commercial-only LTS phase starts: Closing the 5.15 branch(es) on 5th January

[Max Paperno]

On 1/6/2021 6:10 AM, Volker Hilsheimer wrote:
> Hi Max and Adam,
> 
> What can do better to avoid such regressions from making it into a release, or preferably into the code, in the first place? Nobody, not even the Qt Company management :P *wants* to release crappy quality on time.
> 
> What we know about those bugs is that they passed all code reviews, and didn’t get caught by any of the thousands of tests we run for every change on half a dozen platforms. And we know that the only way they were discovered is real users testing real applications against the released version of Qt.
> 
> So, what we have is clearly not good enough, but if the last 15 years of writing unit tests etc hasn’t gotten us to a better place, then maybe “more of the same” can’t be the only strategy.
> 
> Is your experience that we release stuff “come hell or high water" in spite of severe bugs being reported during beta testing? We do spend a lot of time triaging incoming bug reports, and a severe enough bug can always block a release.
> 
> Or do we not discover the issues until the .0 release because few people test the pre-releases? That seems to be supported by the data we have about downloads and general activity in response to pre-releases.
> 
> 
> Volker


Hi Voker,

Thanks for your thoughtful response.  You bring up good points, all of 
which certainly merit serious discussion.

Honestly I don't care what the releases are called or how they're 
numbered.  The main issue right now is the locking down of stable 
release branches.  It should be no surprise that this is "still" a 
sensitive issue and calling Qt6 the next usable version (or however it's 
stated officially) is pretty much rubbing salt in the wounds. And who 
knows what's next.

As for getting people to test things... obviously there's no magic 
bullet here. Any project which relies on their users to test 
functionality is going to have this issue.  These days this seems to 
include just about everything, like commercial applications, firmware, 
and airplanes.  I think maybe people are getting tired of it and burnt 
out.  Also there's only so much time in a day.  So on one hand it's a 
popularity contest... how to get users excited to be your testers?  OTOH 
it's a catch-22 because the obvious way to get users excited is with New 
Features, which may not leave much time or motivation to fix stuff that 
doesn't seem vital. But users who don't need the exciting NFs may not 
bother. So a few versions go by before those others catch up and then 
start discovering new bugs from several version ago. At that point 
you're lucky to get a P3, because hey, no one noticed this for over a 
year, "obviously" not a big deal.

I would also say it's discouraging when there are already 5K+ open P2+ 
QTBUGs in the system. Double that with P3. Some going way back and 
still, seemingly, relevant.  (e.g. a P1 from 2014: "Affects Version/s: 
5.4.0 Alpha, 5.6.2, 5.9, 5.12.3, 5.14")  This was brought up here just 
recently, of course.

Even if half of them are irrelevant now, this is time people spent doing 
"half" the work of finding the bug in the first place. Stuff unit tests 
obviously couldn't/didn't catch, for whatever reason. In some cases 
there are even patches, or suggested fixes, or pointers to exact source 
of the issue. So one could argue that there's already plenty to work on 
before, say, new features or higher version numbers.

Look, when one can control the severity level of an issue, and one knows 
a certain level will block the on-time release, it seems logical that 
one gets to pick and choose what constitutes "severe" or "critical" or 
whatever.  It's a circular argument.  "Well, we don't want to delay 
release _just_ for this, so it's not critical."  Someone (or group) gets 
to make the choice, any way you look at it.  Which is fine really, 
someone has to drive the boat.  It's inevitable that some of the 
passengers like the direction, and some might not.  And some won't know 
until they get there...  :-)  Whether the drivers are happy with the 
outcome, and who they care to listen to on the way, is up to them.

Personally I prefer to clear out backlogs of known issues before 
building further upon that foundation.  I mean real issues, not feature 
requests or nice-to-haves, or pixel-perfect everywhere.  Fresh paint on 
a house with crumbling walls will only last so long.

Thanks again for listening,
-Max