[Development] Staging of qtdeclarative changes has been blocked while we try to unpin the qtbase SHA1

[Kent Hansen]

Den 12. mars 2012 23:45, skrev ext Rohan McGovern:
> Richard Moore said:
>> On 12 March 2012 17:56,<kent.hansen at nokia.com>  wrote:
>>> Besides flaky tests, we also have the general/recurring problem of changes going into qtbase that break qtdeclarative (and possibly/likely other modules). While I realize it's time-consuming for everyone to manually build and run the qtdeclarative tests (or entire qt5.git) when you're working on qtbase, please at least take the time to consider how your change might affect other modules. If you're unsure and/or would like someone else's opinion on the possible impact, feel free to send an email to this list, or ask on IRC. Thanks.
>> Do you have any idea of how many of these were due to QtDeclarative
>> making assumptions that aren't part of the defined API, vs how many
>> were changes in expected behaviour?
> Here's the commits which were needed:

Thanks for the overview, Rohan.

>      http://codereview.qt-project.org/18909 - 16e29f3 Remove unneeded dependencies to QtWidgets and QtOpenGL
>        ->  I don't think this one resolves any test failures.

It does; by switching to using QGuiApplication for those tests, we 
effectively sidestep a change in QApplication font loading behavior due 
to http://codereview.qt-project.org/#change,18621, which broke a test on 
Mac.


>      http://codereview.qt-project.org/19656 - c787809 Mark presumed unstable test as insignificant.
>        ->  marks tst_qquickpixmapcache as insignificant, doesn't actually
>           resolve the problem, so the real issue may not yet be understood

Yep, it's still not understood. I added a comment to the change about 
the symptom, but I don't know how to reproduce it.


>      http://codereview.qt-project.org/19552 - dda130f Fix MouseArea autotest.
>      http://codereview.qt-project.org/19534 - 6cf36b2 Fix tst_qquicktextedit.
>      http://codereview.qt-project.org/19427 - cb1ff7a Fix double click handler in QQuickItem.
>        ->  all of these were failing due to changes in double-click
>           semantics, apparently a bug fix:
>
>              commit b371f3f943703840d0dfbe30505018bcca06e260
>              Author: Laszlo Agocs<laszlo.p.agocs at nokia.com>
>              Date:   Tue Mar 6 16:09:09 2012 +0200
>
>              Fix double click handling.
>

Yes, the mouse event handling change was the most serious source of 
failures.

In addition, there was a change in QtNetwork last week that I reverted 
because it caused qtdeclarative autotests to crash. That one is being 
reapplied again in http://codereview.qt-project.org/#change,19591, 
hopefully with the crashes gone.

>      http://codereview.qt-project.org/19598 - cbb7f8b Skip test that accesses deleted QML engine
>        ->  apparently the test reads invalid memory, but doesn't actually
>           crash on most platforms.  It might be that the qtbase changes,
>           due to changing the layout of a few things in memory, caused it
>           to start crashing on at least one platform.

It turned out that valgrind was already complaining about this on 
platforms where it didn't crash. Would be good if there were a CI that 
ran the tests through valgrind, maybe once a week or so.

In the qtbase/api_changes branch, there was a change in QString::mid() 
(the handling of a negative position argument) that caused 
tst_qquicktextinput::getText to fail.

It's difficult to know which type of changes can break stuff in other 
modules, but things like

- QtCore/Gui event/timer handling
- Gui text/font changes
- Network changes (declarative uses networking extensively)
- Changing the semantics of input arguments of public functions

might be worth validating against dependent modules.
By doing so we can save a lot of time we otherwise have to spend on 
failed CI rounds and tracking down changes that broke other modules.

Regarding flaky tests: This is a nightmare right now since it makes 
_any_ integration so darn unpredictable. The CI takes ~1.5 hours to 
build qtdeclarative and run tests on all platforms. Everything might 
look green and dandy on the seven first machines, but then the last one 
gets to 99% before failing some test. Then we need to investigate 
whether that failure is due to a new change in e.g. qtbase, or whether 
it's flaky, apply some fix (e.g. mark the test as insignificant), and do 
a whole new round of CI.

So please, be careful when adding new tests that rely on timing/event 
loop specifics. Due to the very nature of QtQuick (interaction tests, 
networking), I guess it has a higher likelyhood of having flaky tests 
than other modules.

Best regards,
Kent