[Development] Failure in tst_qwidget

[Olivier Goffart]

On Saturday 02 March 2013 09:45:28 Alberto Mardegan wrote:
> Hi all!
>   The CI bot reported a failure in tst_qwidget when merging
> https://codereview.qt-project.org/#change,42990,patchset=8
> (see the last comment from "Qt Continuous Integration System").
> 
> I can reproduce the same failure under ubuntu 13.04 running XFCE.
> Then I removed my patch (by checking out an older version of the dev
> branch, c65ecc50ae4c322294f01685a2b2d2a748a9c940), and ran "make clean"
> from the qtbase root directory.
> After having rebuild everything (by just running "make"), I tried
> running the tst_qwidget again, but I still get the same failure.
> 
> Interestingly, the failure does not occur if I run the individual test
> which is failing (both with my patch and without):
> 
> tests/auto/widgets/kernel/qwidget$ ./tst_qwidget showHideShowX11
> ********* Start testing of tst_QWidget *********
> Config: Using QTest library 5.1.0, Qt 5.1.0
> PASS   : tst_QWidget::initTestCase()
> PASS   : tst_QWidget::showHideShowX11()
> PASS   : tst_QWidget::cleanupTestCase()
> Totals: 3 passed, 0 failed, 0 skipped
> ********* Finished testing of tst_QWidget *********
> 
> So, I wonder:
> 1) Did I take to correct steps to rebuild qtbase?
> 2) Did someone else experience this failure as well? Or is it definitely
> related to my patch?
> 3) Any idea why the individual test works? Could it be a WM timing issue?


I beleive the test is "unstable".  That means that it may often passes, but on 
some cases, it does not.
If i understand the test properly, it waits two seconds for a window to 
receive some events.  But it may very well happen that the seconds are not 
enough, because the tests are run on some busy virtual machine or because the 
window manager is busy or anything.  
The test should be stabilized by sending a signal when they receive an event, 
and waiting for this event to be received.
Test that waits an arbitrary amount of time are flawed. It can either be too 
small and then the test is unstable, or too long and then the running time of 
the tests increase.  (If every test was taking two second, the CI system would 
take forever)
It is better to fire stop the event loop when the condition is met, or use 
QTRY_VERIFY or something.

-- 
Olivier 

Woboq - Qt services and support - http://woboq.com - http://code.woboq.org