[Development] Removal of some of the blacklisted (non-working) autotests?

[Giuseppe D'Angelo]

Howdy,

On 03/11/16 09:17, Milla Pohjanheimo wrote:
> I would like to challenge you a bit about removal of (some of) the
> blacklisted autotests. We have around fifty autotest functions in
> different Qt modules that have been blacklisted on all platforms. (The
> list of those can be found at the end of this email). In total there are
> over 300 tests in all Qt modules that are blacklisted on some plaforms
> including those fifty I mentioned earlier. The blacklisted tests are one
> (big) problem we have, but another problem is that the blacklisted tests
> also burden our CI, since even though a test is blacklisted, it is still
> compiled and run in the CI. So basically we have dead code that gets
> compiled and run, but it's not used anywhere, just sucking up CI resources.

The very fact that this test code successfully compiles *is* a test. It
is also be beneficial to run these tests from time to time (say, every
1-2 days or so, instead of every integration): this may reveal that a
blacklisted test is actually passing, so it should be removed from the
blacklist.

So my personal vote would be against removing any code, but (if possible
and wanted) just not running these tests at every CI integration.

> Should we just delete at least those tests that are blacklisted on all
> platforms? Or remove all those that have been blacklisted more than 6
> months ago? Ideally those should be fixed/rewritten of course, but who
> would do that? Fixing the tests requires knowledge of the code itself
> it's testing.
> 
> Don't be scared of loosing the tests completely. If something is removed
> from gerrit it still leaves a mark and one can get the deleted tests
> back rather easily, right? There should be a bug report in jira for
> every blacklisted test and if one adds 'Task-number' to the commit
> message, then the commit is attached to the bug report too, which makes
> it easy to spot what has been removed. If there is no such bug report,
> it needs to be done so that the tracking of the removed tests is easy.

While this is true from a technical point of view, a crude removal of
the code makes it way harder to figure out if certain code paths have a
test at all.

If I'm working on some code and look into adding or extending or fixing
its tests, I will never realize that some useful test code existed but
got removed and I can search into the history to find it. (But this is
just me, so I realize that this isn't an argument, it's an opinion.)

My 2 cents,
-- 
Giuseppe D'Angelo | giuseppe.dangelo at kdab.com | Senior Software Engineer
KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908
KDAB - The Qt, C++ and OpenGL Experts

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4007 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.qt-project.org/pipermail/development/attachments/20161103/128b2fcc/attachment.bin>