[Development] Qt 4.8 autotests need attention; over 100 insignificant tests!

[Friedemann Kleint]

Hi,

to emphasize, the situation is quite serious and there were already 
regressions like https://bugreports.qt-project.org/browse/QTBUG-25906 
(QFile::append no longer working).

In most cases, the problem is that most tests using test data files were 
written assuming those files can always be accessed from the working 
directory "./test.data".
This was true for the old autotest system used in Qt 4.

The new test infrastructure of the Qt Project executes 'make check' from 
the test root directory, causing the working directory to be set to the 
location of the binary. Since they are located in the 'debug' or 
'release' subfolders on Windows or in the application.app/MacOs... 
folder on Mac, respectively, they can no longer find the test data files 
from there.

In  Qt 5, this was solved by introducing the QFINDTESTDATA macro, but 
porting it to Qt 4 is a non-trivial task.

https://codereview.qt-project.org/#change,28579 (Fix QLibrary test) and 
https://codereview.qt-project.org/#change,27579 (QApplication) 
illustrate what needs to be done for such cases:

- Removing CONFIG -= app_bundle causes the Mac binary to be created in 
the test root folder

- On Windows, cd up one level if the current working directory is 
'debug' or 'release' in the test constructor

- Remove the insignification from the test profile

Additionally, it would be nice if the test printed a nice error report 
about the missing files.

In case the test cannot be completely fixed, it already helps to 
restrict the insignification to the offending platform or move it to the 
code (QSKIP, QEXPECT_FAIL).

Thanks,
Friedemann

-- 
Friedemann Kleint
Qt Open Source Team
Nokia, Qt Development Frameworks