[Development] Qt 4.8 autotests need attention; over 100 insignificant tests!

[Rohan McGovern]

Hi all,

When Qt 4.8 was moved to gerrit, corners were cut to get it working with
the Qt 5 test infrastructure.  This included the disabling of many
autotests, ostensibly as a temporary measure.

Unfortunately this temporary situation has, to date, not been followed
up on, and we still have greater than 100 autotests effectively disabled
(marked as insignificant) in Qt 4.8's CI system.

These are the relevant tests, in need of attention.  Currently, _none_
of these tests are being used to block regressions:

  tests/auto/bic
  tests/auto/compilerwarnings
  tests/auto/declarative/examples
  tests/auto/declarative/qdeclarativeborderimage
  tests/auto/declarative/qdeclarativedebugjs
  tests/auto/declarative/qdeclarativeerror
  tests/auto/declarative/qdeclarativeimage
  tests/auto/declarative/qdeclarativetext
  tests/auto/declarative/qdeclarativexmlhttprequest
  tests/auto/declarative/qmlvisual
  tests/auto/exceptionsafety_objects
  tests/auto/guiapplauncher
  tests/auto/headers
  tests/auto/languagechange
  tests/auto/linguist/lupdate
  tests/auto/macgui
  tests/auto/macnativeevents
  tests/auto/mediaobject
  tests/auto/q3actiongroup
  tests/auto/q3process
  tests/auto/q3socket
  tests/auto/q3sqlcursor
  tests/auto/q3urloperator
  tests/auto/qaccessibility
  tests/auto/qapplication
  tests/auto/qaudioinput
  tests/auto/qclipboard
  tests/auto/qcolumnview
  tests/auto/qdbusmarshall
  tests/auto/qeventloop
  tests/auto/qfilesystemmodel
  tests/auto/qfilesystemwatcher
  tests/auto/qfile
  tests/auto/qfocusevent
  tests/auto/qfont
  tests/auto/qftp
  tests/auto/qgl
  tests/auto/qglthreads
  tests/auto/qgraphicsanchorlayout1
  tests/auto/qgraphicsanchorlayout
  tests/auto/qgraphicseffect
  tests/auto/qgraphicsitem
  tests/auto/qgraphicsproxywidget
  tests/auto/qgraphicsscene
  tests/auto/qgraphicstransform
  tests/auto/qgraphicsview
  tests/auto/qgraphicswidget
  tests/auto/qhostinfo
  tests/auto/qhttp
  tests/auto/qimagereader
  tests/auto/qinputcontext
  tests/auto/qinputdialog
  tests/auto/qitemdelegate
  tests/auto/qlibrary
  tests/auto/qlistwidget
  tests/auto/qlocale
  tests/auto/qlocalsocket
  tests/auto/qmacstyle
  tests/auto/qmdiarea
  tests/auto/qmdisubwindow
  tests/auto/qmenu
  tests/auto/qnetworkconfigurationmanager
  tests/auto/qnetworkconfiguration
  tests/auto/qnetworkreply
  tests/auto/qnetworksession
  tests/auto/qpainter
  tests/auto/qpauseanimation
  tests/auto/qpixmap
  tests/auto/qpluginloader
  tests/auto/qprinter
  tests/auto/qscriptcontext
  tests/auto/qscriptextensionplugin
  tests/auto/qscriptv8testsuite
  tests/auto/qscrollbar
  tests/auto/qsettings
  tests/auto/qsharedpointer_and_qwidget
  tests/auto/qsharedpointer
  tests/auto/qsocketnotifier
  tests/auto/qsocks5socketengine
  tests/auto/qsslsocket_onDemandCertificates_member
  tests/auto/qsslsocket_onDemandCertificates_static
  tests/auto/qsslsocket
  tests/auto/qstylesheetstyle
  tests/auto/qtableview
  tests/auto/qtcpserver
  tests/auto/qtcpsocket
  tests/auto/qtextlayout
  tests/auto/qtextscriptengine
  tests/auto/qtextstream
  tests/auto/qtimeline
  tests/auto/qtipc/qsharedmemory
  tests/auto/qtooltip
  tests/auto/qtreeview
  tests/auto/qudpsocket
  tests/auto/quuid
  tests/auto/qwebframe
  tests/auto/qwebpage
  tests/auto/qwidget
  tests/auto/qwidget_window
  tests/auto/qwizard
  tests/auto/selftests
  tests/auto/symbols
  tests/auto/xmlpatterns

( found by: git grep insignificant_test -- tests/auto )

I need your help to improve the situation.  For anyone interested in
ensuring the quality of Qt 4.8, this is a great way to help out.
Nokia's own QA staff is almost exclusively working on Qt 5 or related
projects and we won't be able to fix this ourselves.

The goal is to remove CONFIG+=insignificant_test from as many of these
autotests as possible.

How you can help:

  - check the current status of these tests for each platform.  This can
    always be ascertained from these links:

  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/linux-g++-32_Ubuntu_10.04_x86/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/linux-g++_developer-build_qtnamespace_qtlibinfix_Ubuntu_11.10_x64/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/macx-g++_OSX_10.6/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/macx-g++_OSX_10.7/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/macx-g++_developer-build_OSX_10.7/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/qws_linux-x86-g++/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/win32-msvc2010_Windows_7/log.txt.gz
  http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/win32-msvc2010_developer-build_Windows_7/log.txt.gz

  - if the test is already passing on some platforms, you can amend the
    insignificant_test condition to apply only to the platforms where it
    is failing, at least allowing the test to detect regressions on some
    platforms while the remaining failures are investigated.

    It looks to me like many of the failing tests are only failing on
    Windows, but incorrectly marked as insignificant for all platforms.

  - if the test is failing consistently, you can replace the
    insignificant_test marker with a QEXPECT_FAIL for the failing
    portion of the test, and raise a bug for the test's failure.  This
    allows the rest of the test to be used to detect regressions, until
    the whole test can be fixed.

  - if the test failure looks easy to fix, you can of course fix it
    directly.

I'd also like to apologize for not bringing this up sooner, as I've
known for some time the Qt 4.8 autotest situation is quite bad and
haven't attempted to make that more widely known.