[Development] Removal of some of the blacklisted (non-working) autotests?

[Milla Pohjanheimo]

Hi all,

I would like to challenge you a bit about removal of (some of) the blacklisted autotests. We have around fifty autotest functions in different Qt modules that have been blacklisted on all platforms. (The list of those can be found at the end of this email). In total there are over 300 tests in all Qt modules that are blacklisted on some plaforms including those fifty I mentioned earlier. The blacklisted tests are one (big) problem we have, but another problem is that the blacklisted tests also burden our CI, since even though a test is blacklisted, it is still compiled and run in the CI. So basically we have dead code that gets compiled and run, but it's not used anywhere, just sucking up CI resources.

Should we just delete at least those tests that are blacklisted on all platforms? Or remove all those that have been blacklisted more than 6 months ago? Ideally those should be fixed/rewritten of course, but who would do that? Fixing the tests requires knowledge of the code itself it's testing.

Don't be scared of loosing the tests completely. If something is removed from gerrit it still leaves a mark and one can get the deleted tests back rather easily, right? There should be a bug report in jira for every blacklisted test and if one adds 'Task-number' to the commit message, then the commit is attached to the bug report too, which makes it easy to spot what has been removed. If there is no such bug report, it needs to be done so that the tracking of the removed tests is easy.

I would like to get your comments on this and also if you have improvement ideas on the matter, I'm all ears. Also if you have an itch to fix some autotests, feel free to do so. :) It's a common effort to improve the overall quality of Qt and the autotests play an essential role in spotting the possible defects quickly, so let's keep Qt in a good shape without unnecessarily burdening our CI system.

In hope of getting feedback from you,
Milla Pohjanheimo (the official autotest snoop in the Qt Company)


Blacklisted tests on all platforms (dev branch) (with '*' on the BLACKLIST file) (blacklisted date):
- qtnetworkauth/tests/auto/oauth1/BLACKLIST
2016-08-22 [getToken]
2016-08-22 [grant]
2016-08-22 [authenticatedCalls]

- qtdeclarative/tests/auto/qml/animation/qsequentialanimationgroupjob/BLACKLIST
2015-06-04 [finishWithUncontrolledAnimation]

- qtdeclarative/tests/auto/qmltest/BLACKLIST
2015-06-02 [SelfTests::test_blacklisted_fail]
2015-06-02 [SelfTests::test_blacklistWithData:test2]

- qtdeclarative/tests/auto/quick/qquicktext/BLACKLIST (2015-06-04)
2015-06-04[dependentImplicitSizes]
2015-06-04 [mouseSelection]

- qtdeclarative/tests/auto/quick/qquickmultipointtoucharea/BLACKLIST
2015-06-04 [inFlickable]

- qtdeclarative/tests/auto/quick/qquicktextedit/BLACKLIST
2015-06-04 [mouseSelection]

- qtdeclarative/tests/auto/quick/qquicklistview/BLACKLIST
2015-03-26 [QTBUG_38209]

- qtbase/tests/auto/corelib/animation/qpauseanimation/BLACKLIST
2015-05-24 [pauseAndPropertyAnimations]

- qtbase/tests/auto/corelib/thread/qsemaphore/BLACKLIST
2015-06-15 [tryAcquireWithTimeout]

- qtbase/tests/auto/testlib/selftests/blacklisted/BLACKLIST
2015-04-28 [pass]
2015-04-28 [skip]
2015-04-28 [fail]
2015-04-28 [xpass]
2015-04-28 [xfail]
2015-04-28 [messages]

- qtbase/tests/auto/network/socket/qsocks5socketengine/BLACKLIST
2014-11-12 [udpTest]
2014-11-12 [passwordAuth]

- qtbase/tests/auto/network/socket/qudpsocket/BLACKLIST
2015-10-29 [multicast:same bind, group ipv6 address]

- qtbase/tests/auto/network/access/qftp/BLACKLIST
2015-11-17 [list:epsvNotSupported]

- qtbase/tests/auto/network/access/qnetworkreply/BLACKLIST
2014-09-16 [httpAbort]
2014-11-13 [backgroundRequestInterruption:ftp, bg, nobg]

- qtbase/tests/auto/network/ssl/qsslsocket/BLACKLIST
2014-11-25 [waitForConnectedEncryptedReadyRead:WithSocks5ProxyAuth]

- qtbase/tests/auto/network/ssl/qsslcertificate/BLACKLIST
2015-08-04 [subjectAndIssuerAttributes]

- qtspeech/tests/auto/texttospeech/BLACKLIST
2016-08-11 [say_hello]
2016-08-11 [speech_rate]
2016-08-11 [pitch]
2016-08-11 [set_voice]

- qtwebengine/tests/auto/quick/qmltests/BLACKLIST
2016-02-29 [DesktopWebEngineViewLinkHovered::test_linkHovered]
2016-02-29 [WebViewGeopermission::test_geoPermissionRequest]

- qtwebengine/tests/auto/quick/qquickwebengineviewgraphics/BLACKLIST
2016-07-15 [showHideShow]

- qtwebengine/tests/auto/widgets/qwebenginepage/BLACKLIST
2015-10-29 [geolocationRequestJS]

- qtquickcontrols2/tests/auto/sanity/BLACKLIST
2016-04-12 [signalHandlers:material/SpinBox.qml]
2015-10-30 [signalHandlers:material/TextArea.qml]
2015-10-30 [signalHandlers:material/TextField.qml]
2015-11-26 [attachedObjects:controls/ComboBox.qml]
2015-11-26 [attachedObjects:material/ComboBox.qml]
2016-05-10 [attachedObjects:material/Switch.qml]
2016-05-10 [attachedObjects:material/SwitchDelegate.qml]
2015-11-26 [attachedObjects:universal/ComboBox.qml]
2016-07-12 [functions:material/RectangularGlow.qml]
2016-07-12 [anchors:material/RectangularGlow.qml]

- qtlocation/tests/auto/declarative_ui/BLACKLIST
2016-08-10 [MapKeepGrabAndPreventSteal::test_map_preventsteal]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20161103/3424c2e6/attachment.html>