[Development] CI stability

[Lars Knoll]

> On 09 Feb 2017, at 11:52, Marc Mutz <marc.mutz at kdab.com> wrote:
> 
> Hi Lars,
> 
> On Wednesday 08 February 2017 10:46:36 Lars Knoll wrote:
>> Anybody who identifies a flaky test (ie. a test that is randomly failing in
>> CI), can blacklist that test; under one condition. He needs to at the same
>> time create a P0 bug report about it. Please also add the labels
>> ‘autotest’ and ‘flaky’ to the bug report, so that we can follow up on
>> those.
> 
> A request for clarification:
> 
> If a test fails on 5.8, it's clear: blacklist it in 5.8, too, backport to 5.6 
> as needed.
> 
> What about 5.9 and dev? I'd push the blacklist to 5.8 if the test in question 
> didn't change since 5.8, but maybe the implementation of the functionality did 
> change and made the test flaky?
> 
> Still submit to 5.8 or to the branch it was seen failing on, even if that 
> means multiple identical commits to different branches down the line?

I’d say we push the branch we saw it failing on (but please feel free to do a quick check whether you can find it failing in 5.8 as well before pushing). That might imply that we get multiple changes, but we won’t lower our test coverage more than strictly required.

Cheers,
Lars