[Development] Towards a Qt 5 beta

marius.storm-olsen at nokia.com marius.storm-olsen at nokia.com
Wed Apr 11 15:40:26 CEST 2012 

On 04/11/2012 07:49 AM, ext lars.knoll at nokia.com<mailto:lars.knoll at nokia.com> wrote:

** insignificant tests **

We still have quite some tests (around 110 in total) marked as
insignificant. This means they will be ignored by the CI system. Any help
to reduce this number would be great.



I've grepped through all the modules to create a list of these insignificant tests, and listed them below.

Please note that a test marked as insignificant in essence provides us with zero coverage, since all results from those test cases (although run) are completely ignored.

Yes, we used to have a (quite large) list of black-listed tests for Qt 4 as well. However, back then we could mark only a test function as black-listed, while the rest of the testcase would still provide some test coverage. That's an important difference.

IMO, if there are tasks we cannot repair, due to the tests being inheritly unstable, or our infrastructure not being able to handle it, then we should remove them. Having test cases which adds no value, but is hard to see unless you find the "insignificant_test" keyword, is bad since you don't know exactly what the coverage is and where it is lacking.

How should we mark that we are working on a specific test case to fix the instability etc, to ensure we don't do double work? Suggestions? Is a Jira task with '<module>: insignificant <test name>' good enough?



  1.
qt3d/tests/auto/qml3d/qml3d.pro:26:CONFIG+=insignificant_test
  2.
qt3d/tests/auto/threed/qgeometrydata/qgeometrydata.pro:12:CONFIG+=insignificant_test
  3.
qt3d/tests/auto/threed/qglabstractsurface/qglabstractsurface.pro:9:CONFIG+=insignificant_test
  4.
qt3d/tests/auto/threed/qglmaterial/qglmaterial.pro:9:CONFIG+=insignificant_test
  5.
qt3d/tests/auto/threed/qglpainter/qglpainter.pro:18:CONFIG+=insignificant_test
  6.
qt3d/tests/auto/threed/qglrender/qglrender.pro:9:CONFIG+=insignificant_test
  7.
qt3d/tests/auto/threed/qglscenenode/qglscenenode.pro:10:CONFIG+=insignificant_test
  8.
qt3d/tests/auto/threed/qglvertexbundle/qglvertexbundle.pro:9:macx:CONFIG+=insignificant_test
  9.
qtbase/tests/auto/concurrent/qtconcurrentfilter/qtconcurrentfilter.pro:7:CONFIG += insignificant_test # See QTBUG-20688
  10.
qtbase/tests/auto/corelib/io/qfile/test/test.pro:14:mac*:CONFIG+=insignificant_test
  11.
qtbase/tests/auto/corelib/io/qiodevice/qiodevice.pro:9:mac: CONFIG += insignificant_test # QTBUG-22766
  12.
qtbase/tests/auto/corelib/io/qtextstream/test/test.pro:23:mac: CONFIG += insignificant_test # QTBUG-22767
  13.
qtbase/tests/auto/corelib/plugin/qfactoryloader/test/test.pro:21:mac: CONFIG += insignificant_test # QTBUG-22765
  14.
qtbase/tests/auto/corelib/plugin/qpluginloader/tst/tst.pro:17:mac: CONFIG += insignificant_test # QTBUG-22765
  15.
qtbase/tests/auto/corelib/tools/qlocale/test/test.pro:19:mac: CONFIG += insignificant_test # QTBUG-22769
  16.
qtbase/tests/auto/corelib/tools/qtimeline/qtimeline.pro:5:win32:CONFIG+=insignificant_test    # unstable, QTBUG-24796
  17.
qtbase/tests/auto/gui/kernel/qclipboard/test/test.pro:18:mac: CONFIG += insignificant_test # QTBUG-23057
  18.
qtbase/tests/auto/gui/kernel/qguiapplication/qguiapplication.pro:6:win32:CONFIG += insignificant_test # QTBUG-24186
  19.
qtbase/tests/auto/gui/qopengl/qopengl.pro:11:mac: CONFIG += insignificant_test # QTBUG-23061
  20.
qtbase/tests/auto/gui/qopengl/qopengl.pro:12:win32:CONFIG += insignificant_test # QTBUG-24192
  21.
qtbase/tests/auto/gui/text/qfontdatabase/qfontdatabase.pro:13:mac: CONFIG += insignificant_test # QTBUG-23062
  22.
qtbase/tests/auto/gui/text/qrawfont/qrawfont.pro:13:win32:CONFIG += insignificant_test # QTBUG-24197
  23.
qtbase/tests/auto/gui/text/qstatictext/qstatictext.pro:7:mac: CONFIG += insignificant_test # QTBUG-23063
  24.
qtbase/tests/auto/gui/text/qtextscriptengine/qtextscriptengine.pro:10:mac: CONFIG += insignificant_test # QTBUG-23064
  25.
qtbase/tests/auto/network/access/qabstractnetworkcache/qabstractnetworkcache.pro:9:CONFIG += insignificant_test  # QTBUG-20686; note, assumed unstable on all platforms
  26.
qtbase/tests/auto/network/access/qftp/qftp.pro:17:CONFIG+=insignificant_test  # uses live qt-test-server, inherently unstable
  27.
qtbase/tests/auto/network/access/qnetworkreply/test/test.pro:6:contains(QT_CONFIG,xcb): CONFIG+=insignificant_test  # unstable, QTBUG-21102
  28.
qtbase/tests/auto/network/access/qnetworkreply/test/test.pro:15:win32:CONFIG += insignificant_test # QTBUG-24226
  29.
qtbase/tests/auto/network/kernel/qhostinfo/qhostinfo.pro:14:linux-*:CONFIG+=insignificant_test    # QTBUG-23837 - test is unstable
  30.
qtbase/tests/auto/network/kernel/qnetworkinterface/qnetworkinterface.pro:7:win32:CONFIG+=insignificant_test      # QTBUG-24451 - localAddress()
  31.
qtbase/tests/auto/network/socket/qsocks5socketengine/qsocks5socketengine.pro:14:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG += insignificant_test
  32.
qtbase/tests/auto/network/socket/qudpsocket/test/test.pro:25:CONFIG+=insignificant_test
  33.
qtbase/tests/auto/network/ssl/qsslsocket/qsslsocket.pro:39:win32:CONFIG += insignificant_test # QTBUG-24234
  34.
qtbase/tests/auto/network/ssl/qsslsocket_onDemandCertificates_member/qsslsocket_onDemandCertificates_member.pro:26:win32:CONFIG+=insignificant_test      # QTBUG-24451 - all
  35.
qtbase/tests/auto/network/ssl/qsslsocket_onDemandCertificates_static/qsslsocket_onDemandCertificates_static.pro:26:win32:CONFIG+=insignificant_test      # QTBUG-24451 - all
  36.
qtbase/tests/auto/opengl/qgl/qgl.pro:13:CONFIG+=insignificant_test
  37.
qtbase/tests/auto/opengl/qglthreads/qglthreads.pro:13:CONFIG+=insignificant_test # QTBUG-22560
  38.
qtbase/tests/auto/other/lancelot/lancelot.pro:15:CONFIG += insignificant_test # QTBUG-21402
  39.
qtbase/tests/auto/other/macgui/macgui.pro:11:CONFIG+=insignificant_test  # QTBUG-20984, fails unstably
  40.
qtbase/tests/auto/other/macnativeevents/macnativeevents.pro:14:CONFIG += insignificant_test # QTBUG-22775
  41.
qtbase/tests/auto/other/qaccessibility/qaccessibility.pro:15:mac: CONFIG += insignificant_test # QTBUG-22812
  42.
qtbase/tests/auto/other/qfocusevent/qfocusevent.pro:5:mac: CONFIG += insignificant_test # QTBUG-22815
  43.
qtbase/tests/auto/printsupport/kernel/qprinter/qprinter.pro:6:mac*:CONFIG+=insignificant_test
  44.
qtbase/tests/auto/printsupport/kernel/qprinterinfo/qprinterinfo.pro:9:mac: CONFIG += insignificant_test # QTBUG-23060
  45.
qtbase/tests/auto/widgets/dialogs/qdialog/qdialog.pro:5:mac:CONFIG += insignificant_test    # QTBUG-24977
  46.
qtbase/tests/auto/widgets/graphicsview/qgraphicsgridlayout/qgraphicsgridlayout.pro:8:contains(QT_CONFIG,xcb):CONFIG+=insignificant_test
  47.
qtbase/tests/auto/widgets/graphicsview/qgraphicsproxywidget/qgraphicsproxywidget.pro:10:contains(QT_CONFIG,xcb):CONFIG+=insignificant_test
  48.
qtbase/tests/auto/widgets/graphicsview/qgraphicsproxywidget/qgraphicsproxywidget.pro:12:win32:CONFIG += insignificant_test # QTBUG-24294
  49.
qtbase/tests/auto/widgets/graphicsview/qgraphicsscene/qgraphicsscene.pro:21:win32:CONFIG += insignificant_test # QTBUG-24295
  50.
qtbase/tests/auto/widgets/graphicsview/qgraphicsview/qgraphicsview.pro:10:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG+=insignificant_test
  51.
qtbase/tests/auto/widgets/graphicsview/qgraphicsview/qgraphicsview.pro:12:win32|mac:CONFIG += insignificant_test # QTBUG-24296
  52.
qtbase/tests/auto/widgets/graphicsview/qgraphicswidget/qgraphicswidget.pro:10:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG += insignificant_test
  53.
qtbase/tests/auto/widgets/itemviews/qitemdelegate/qitemdelegate.pro:9:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG+=insignificant_test
  54.
qtbase/tests/auto/widgets/itemviews/qlistwidget/qlistwidget.pro:7:contains(QT_CONFIG,xcb):CONFIG+=insignificant_test  # QTBUG-21098, fails unstably
  55.
qtbase/tests/auto/widgets/itemviews/qlistwidget/qlistwidget.pro:9:win32:CONFIG += insignificant_test # QTBUG-24299
  56.
qtbase/tests/auto/widgets/itemviews/qtreeview/qtreeview.pro:8:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG+=insignificant_test
  57.
qtbase/tests/auto/widgets/kernel/qwidget/qwidget.pro:24:CONFIG+=insignificant_test
  58.
qtbase/tests/auto/widgets/styles/qmacstyle/qmacstyle.pro:7:mac*:CONFIG+=insignificant_test
  59.
qtbase/tests/auto/widgets/util/qcompleter/qcompleter.pro:8:CONFIG += insignificant_test # QTBUG-21424
  60.
qtbase/tests/auto/widgets/widgets/qcombobox/qcombobox.pro:7:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG += insignificant_test
  61.
qtbase/tests/auto/widgets/widgets/qdatetimeedit/qdatetimeedit.pro:7:mac:CONFIG += insignificant_test # numerous failures, see QTBUG-23674
  62.
qtbase/tests/auto/widgets/widgets/qdoublespinbox/qdoublespinbox.pro:7:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG += insignificant_test
  63.
qtbase/tests/auto/widgets/widgets/qmdiarea/qmdiarea.pro:15:CONFIG+=insignificant_test
  64.
qtbase/tests/auto/widgets/widgets/qmdisubwindow/qmdisubwindow.pro:9:mac*:CONFIG+=insignificant_test
  65.
qtbase/tests/auto/widgets/widgets/qmenubar/qmenubar.pro:7:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG += insignificant_test
  66.
qtbase/tests/auto/widgets/widgets/qscrollbar/qscrollbar.pro:6:mac*:CONFIG+=insignificant_test
  67.
qtbase/tests/auto/widgets/widgets/qsizegrip/qsizegrip.pro:7:mac: CONFIG += insignificant_test # failures on mac, QTBUG-23681
  68.
qtbase/tests/auto/widgets/widgets/qtextedit/qtextedit.pro:19:contains(QT_CONFIG,xcb):CONFIG+=insignificant_test  # QTBUG-20756 crashes on xcb
  69.
qtconnectivity/tests/auto/qbluetoothdevicediscoveryagent/qbluetoothdevicediscoveryagent.pro:7:CONFIG += insignificant_test    # QTBUG-22017
  70.
qtconnectivity/tests/auto/qbluetoothdeviceinfo/qbluetoothdeviceinfo.pro:7:CONFIG += insignificant_test    # QTBUG-22017
  71.
qtconnectivity/tests/auto/qbluetoothlocaldevice/qbluetoothlocaldevice.pro:7:CONFIG += insignificant_test    # QTBUG-22017
  72.
qtconnectivity/tests/auto/qbluetoothservicediscoveryagent/qbluetoothservicediscoveryagent.pro:7:CONFIG += insignificant_test    # QTBUG-22017
  73.
qtconnectivity/tests/auto/qbluetoothserviceinfo/qbluetoothserviceinfo.pro:7:CONFIG += insignificant_test    # QTBUG-22017
  74.
qtconnectivity/tests/auto/qbluetoothsocket/qbluetoothsocket.pro:13:CONFIG += insignificant_test    # QTBUG-22017
  75.
qtconnectivity/tests/auto/qbluetoothtransfermanager/qbluetoothtransfermanager.pro:10:CONFIG += insignificant_test    # QTBUG-22017
  76.
qtconnectivity/tests/auto/qbluetoothtransferrequest/qbluetoothtransferrequest.pro:7:CONFIG += insignificant_test    # QTBUG-22017
  77.
qtconnectivity/tests/auto/qndefmessage/qndefmessage.pro:7:CONFIG += insignificant_test    # QTBUG-22016
  78.
qtconnectivity/tests/auto/qnearfieldmanager/qnearfieldmanager.pro:9:maemo*:CONFIG += insignificant_test
  79.
qtconnectivity/tests/auto/qnearfieldtagtype1/qnearfieldtagtype1.pro:10:maemo*:CONFIG += insignificant_test
  80.
qtconnectivity/tests/auto/qnearfieldtagtype2/qnearfieldtagtype2.pro:9:maemo*:CONFIG += insignificant_test
  81.
qtconnectivity/tests/auto/qrfcommserver/qrfcommserver.pro:10:CONFIG += insignificant_test    # QTBUG-22017
  82.
qtdeclarative/tests/auto/qmltest/qmltest.pro:12:CONFIG+=insignificant_test
  83.
qtdeclarative/tests/auto/quick/qquickdroparea/qquickdroparea.pro:11:mac: CONFIG += insignificant_test # QTBUG-24588
  84.
qtdeclarative/tests/auto/quick/qquickpixmapcache/qquickpixmapcache.pro:18:CONFIG += insignificant_test
  85.
qtfeedback/tests/auto/qdeclarativefeedback/qdeclarativefeedback.pro:23:maemo*:CONFIG += insignificant_test
  86.
qtfeedback/tests/auto/qfeedbackmmk/qfeedbackmmk.pro:17:linux-*:CONFIG += insignificant_test    # QTBUG-22779, segfaults
  87.
qtmultimedia/tests/auto/unit/qcamerawidgets/qcamerawidgets.pro:12:maemo*:CONFIG += insignificant_test
  88.
qtmultimedia/tests/auto/unit/qgraphicsvideoitem/qgraphicsvideoitem.pro:10:CONFIG += insignificant_test
  89.
qtmultimedia/tests/auto/unit/qpaintervideosurface/qpaintervideosurface.pro:17:CONFIG += insignificant_test
  90.
qtmultimedia/tests/auto/unit/qvideowidget/qvideowidget.pro:10:CONFIG += insignificant_test
  91.
qtphonon/tests/auto/mediaobject/mediaobject.pro:13:CONFIG += insignificant_test    # QTBUG-19539
  92.
qtpim/tests/auto/contacts/qcontactmanagerdetails/qcontactmanagerdetails.pro:27:#maemo*:CONFIG += insignificant_test
  93.
qtpim/tests/auto/contacts/qcontactmanagerfiltering/unittest/unittest.pro:28:maemo*:CONFIG += insignificant_test
  94.
qtqa/tests/postbuild/bic/bic.pro:13:CONFIG += insignificant_test    # QTQAINFRA-321
  95.
qtqa/tests/postbuild/compilerwarnings/compilerwarnings.pro:7:CONFIG += insignificant_test    # QTQAINFRA-322
  96.
qtqa/tests/postbuild/guiapplauncher/guiapplauncher.pro:24:CONFIG += insignificant_test    # QTQAINFRA-323
  97.
qtqa/tests/postbuild/headers/headers.pro:6:CONFIG += insignificant_test    # QTQAINFRA-324
  98.
qtqa/tests/postbuild/symbols/symbols.pro:9:CONFIG += insignificant_test    # QTQAINFRA-325
  99.
qtquick1/tests/auto/declarative/examples/examples.pro:24:CONFIG+=insignificant_test # QTQAINFRA-428
  100.
qtquick1/tests/auto/declarative/qdeclarativeborderimage/qdeclarativeborderimage.pro:21:CONFIG+=insignificant_test # QTQAINFRA-428
  101.
qtquick1/tests/auto/declarative/qdeclarativedebugjs/tst_qdeclarativedebugjs.pro:21:CONFIG+=insignificant_test # QTQAINFRA-428
  102.
qtquick1/tests/auto/declarative/qdeclarativeerror/qdeclarativeerror.pro:15:CONFIG+=insignificant_test # QTQAINFRA-428
  103.
qtquick1/tests/auto/declarative/qdeclarativefontloader/qdeclarativefontloader.pro:21:win32:CONFIG += insignificant_test # QTBUG-24475
  104.
qtquick1/tests/auto/declarative/qdeclarativeimage/qdeclarativeimage.pro:21:CONFIG+=insignificant_test # QTQAINFRA-428
  105.
qtquick1/tests/auto/declarative/qdeclarativetext/qdeclarativetext.pro:24:CONFIG+=insignificant_test # QTQAINFRA-428
  106.
qtquick1/tests/auto/declarative/qdeclarativetimer/qdeclarativetimer.pro:16:CONFIG+=insignificant_test # QTQAINFRA-428
  107.
qtquick1/tests/auto/declarative/qdeclarativexmlhttprequest/qdeclarativexmlhttprequest.pro:17:CONFIG+=insignificant_test # QTQAINFRA-428
  108.
qtquick1/tests/auto/declarative/qmlvisual/qmlvisual.pro:40:CONFIG+=insignificant_test # QTQAINFRA-428
  109.
qtscript/tests/auto/qscriptjstestsuite/qscriptjstestsuite.pro:16:win32:CONFIG += insignificant_test # QTBUG-24426
  110.
qtscript/tests/auto/qscriptv8testsuite/qscriptv8testsuite.pro:8:win32:CONFIG += insignificant_test # QTBUG-24427
  111.
qttools/tests/auto/linguist/lupdate/lupdate.pro:6:CONFIG+=insignificant_test

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep mac | wc -l
26

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep linux | wc -l
10

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep win32 | wc -l
16

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep maemo | wc -l
7

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep oneiric | wc -l
8

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep widget | wc -l
26

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep -v -e linux -e mac -e win32 -e maemo -e oneiric | wc -l
53

[marius at LinuxError:~/source/qt5] (gerrit-master)
$ cat insignificant_test.txt  | grep -v -e linux -e mac -e win32 -e maemo -e oneiric -e widget | wc -l
45

--
.marius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20120411/26e5b570/attachment.html>

More information about the Development mailing list