[Development] Towards a Qt 5 beta

[Rohan McGovern]

marius.storm-olsen at nokia.com said:
> On 04/11/2012 07:49 AM, ext lars.knoll at nokia.com<mailto:lars.knoll at nokia.com> wrote:
> 
> ** insignificant tests **
> 
> We still have quite some tests (around 110 in total) marked as
> insignificant. This means they will be ignored by the CI system. Any help
> to reduce this number would be great.
> 
> 
> 
> I've grepped through all the modules to create a list of these insignificant tests, and listed them below.
> 
> Please note that a test marked as insignificant in essence provides us with zero coverage, since all results from those test cases (although run) are completely ignored.
> 
> Yes, we used to have a (quite large) list of black-listed tests for Qt 4 as well. However, back then we could mark only a test function as black-listed, while the rest of the testcase would still provide some test coverage. That's an important difference.
> 
> IMO, if there are tasks we cannot repair, due to the tests being inheritly unstable, or our infrastructure not being able to handle it, then we should remove them. Having test cases which adds no value, but is hard to see unless you find the "insignificant_test" keyword, is bad since you don't know exactly what the coverage is and where it is lacking.
> 
> How should we mark that we are working on a specific test case to fix the instability etc, to ensure we don't do double work? Suggestions? Is a Jira task with '<module>: insignificant <test name>' good enough?
> 

Tests marked as insignificant _should_ have had a task raised
when they were marked as such - for those, it would make sense for the
person who is working on it to assign the task to themselves and/or add
a comment.

In reality it seems like quite many tests were marked as insignificant
with no task raised, it would probably make sense to raise a task when
beginning work on it.