[Development] Removal of some of the blacklisted (non-working) autotests?

[Jędrzej Nowacki]

On torsdag 3. november 2016 09.17.57 CET Milla Pohjanheimo wrote:
> I would like to challenge you a bit about removal of (some of) the
> blacklisted autotests. 
> (...)

Hi,
 
    In your email you wrote that blacklisted is just a burden for CI. In 
general it is true, but mark that currently they are compiling and they are 
_not_ crashing. So they do contribute to the quality of Qt. On the other hand 
they artificially increase test coverage level hiding untested code paths and 
they may affect other tests.

Partial conclusion: delete them, but watch code coverage and add new ones 
where needed.

    Eddy mentioned that tests are documenting certain Qt usages. There are two 
aspects of it. In many cases, we do not know what author of a test wanted to 
test, sometimes it was a use case, sometimes it was just an internal code path 
in the implementation. On the other hand, we know that the code is not working 
in a reliable way, therefore in reality it is a misleading documentation. 
Moreover developers tends to do copy&paste coding and coping a blacklisted 
test is just plain wrong, while being very difficult to catch in code review.

Partial conclusion: delete them, but add a policy that every test should 
contain a short comment what is the purpose of the test.

Some tests are blacklisted only on certain platforms. Depending on the reason 
why the test is not working on a specific platform, we need to handle them 
differently. In general it is ok to assume that Qt as well as Qt bugs are 
cross platform. So a test failing only on one of them is failing because of a 
platform specific code (we do not have so much of it) or it is wrongly 
blacklisted or it hits a bug in a platform itself.

Partial conclusion: the test delivers an interesting information. It is worth 
to spend a bit of time and check what is going on. A tool that do a stress 
test of a test would be a really good addition, as it would simplify 
debugging.

Conclusion: I believe we should delete them, but be smart about that. Don't 
just go and remove all of them, but first build infrastructure / process that 
allows us to not decrease the test coverage. Even more, we need something that 
would not allow flaky, badly written test to be merged to Qt in the first place, 
otherwise we would have that discussion again in next 12 months.

Cheers,
  Jędrek