[Development] Qt branches & proposal how to continue with those

[Ville Voutilainen]

On 9 February 2018 at 20:46, Ville Voutilainen
<ville.voutilainen at gmail.com> wrote:
>> But I am asking to do a minimal investigation. In most cases of blacklisting,
>> the test has been failing for days, if not months. Spending an hour or two to
>> understand why it's failing and whether it's something that only happens in
>> the CI should be the norm.
>
> The problem is that a large amount of tests have been failing, for
> weeks. In some cases,
> months. In some cases, for a year. That results in a restage storm,
> and carves a doubt
> in every submitter's mind whether CI failures are something to really
> bother about beyond
> hitting the restage button. I doubt either of us thinks that to be optimal.


Also, our blacklist patch submitters *do* minimal investigation and
more. There's been a whole shebang
of suggested flaky tests, which have been deemed to not be caused by
flakiness in tests,
after proper investigation. And another matter to consider is that
both because of CI infra problems
and flaky tests, it's been next to impossible to get anything into
e.g. qtbase for two weeks.
Why some of those flaky tests seem to be hit more often isn't exactly
known, but our statistics
information does support their being flaky, and we've seen every one
of these tests fail before
in spurious manners, we just haven't done anything to it, due to some
extent some people adamantly
opposing blacklisting in general, and some demanding that there's a
thorough investigation almost tantamount
to 100% proof that a test is flaky before a possibly temporary
blacklisting is even considered.

It's also perhaps worth realizing that the current flakiness fixes and
blacklists haven't gone in, because flaky tests here and
there and everywhere prevent integrating those fixes.