[Development] Towards a Qt 5 beta

[jason.mcdonald at nokia.com]

> I've grepped through all the modules to create a list of these insignificant tests, and listed them below. Please note that a test marked as insignificant in essence provides us with zero coverage, since all results from those test cases (although run) are completely ignored.

> IMO, if there are tasks we cannot repair, due to the tests being inheritly unstable, or our infrastructure not being able to handle it, then we should remove them. Having test cases which adds no value, but is hard to see unless you find the "insignificant_test" keyword, is bad since you don't know exactly what the coverage is and where it is lacking.

I agree that having useless tests adds no value.  From my examination of the insignificant tests so far, I would say that most of them should be able to be fixed rather than discarded.

I have also noticed that some other tests are disabled by removing them from SUBDIRS in their parent directory .pro file.  Sometimes this is legitimate (test doesn't apply to a particular platform or build config), but sometimes they are disabled this way because the test is/was broken. The result is effectively the same as marking the test insignificant, but it's much harder to grep for. so please avoid this practice in future and use insignificant_test instead.

It may be a while before I can find time to investigate these properly, so any help would be appreciated.

> How should we mark that we are working on a specific test case to fix the instability etc, to ensure we don't do double work? Suggestions? Is a Jira task with '<module>: insignificant <test name>' good enough?

I am currently going through all of the insignificant tests that have no associated bug report and creating appropriate Jira tasks with suggestions for how to proceed based on analysis of recent CI logs. This analysis is proving to be rather time consuming, but shows that a lot of the insignificant_test markers could be replaced by QEXPECT_FAIL calls because a lot of the insignificant test appear to be failing in a consistent fashion (at least over the last ten CI runs).

Some trivial commits to add the bug numbers next to the insignificant_test markers will filter through the CI system over the next few days, which should cure any worries about doing duplicate work as every insignificant_test marker in Qt will have a bug number next to it.

I'm happy to be added to gerrit reviews for anyone fixing insignificant tests and also to try to provide further advice on ways to fix specific insignificant tests.

--
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20120413/0647a575/attachment.html>