[Development] Removal of some of the blacklisted (non-working) autotests?

[Edward Welbourne]

Milla Pohjanheimo said:
> I would like to challenge you a bit about removal of (some of) the
> blacklisted autotests.

Do we have a documented policy on when black-listing should be used ?
We can mark tests as expected fail if they reliably fail, so I presume
black-listing is reserved for flaky tests.  Flakiness should be
considered a bug in the test, so indeed every blacklisting should come
with a bug report, which should be recorded in a comment in the
BLACKLIST file.  (Lines starting with # are comments in the relevant
format.)

There is value in having flaky tests in the source tree, even if it is
only to illustrate how to use the relevant APIs (dealing with all the
complications); they can also serve as start-points from which folk can
attempt to develop less flaky tests.  If the test *isn't* good in either
of these senses, though - if it uses the APIs badly and a proper test of
them wouldn't start from here - then deleting the test makes sense.  We
should ideally replace bad tests with ones that do at least illustrate
best practice use of APIs; but deleting a badly-written test will do.

If there's no reliable way to test some API, I have trouble seeing how
that API can reliably be used, so we should treat that as an issue with
the API itself.  It's still worth having some test of such an API, even
if it is unreliable, if only to give folk investigating the problem a
way to reproduce the flakiness.  We just don't want to burden CI with
having to run such tests - it's essentially a manual test, in practice.

Perhaps the right answer is for the test runner to simply leave out
black-listed tests - have CI not run them.  This may be tricky to
implement, though.  It may make sense to move some flaky tests from auto
to manual, rather than deleting them.

	Eddy.