[Development] Qt 4.8 CI problems

[Anttila Janne]

Hi,

There have been various problems on Qt 4.8 CI since we moved that project
from Nokia to Digia hosting - I'm sorry that those problems have been 
preventing you to get anything in to Qt 4.8 since last Thursday. 

Apparently I made wrong decision to move 4.8 CI from Nokia hosting to Digia 
hosting before build succeeded on Digia hosting. I made my decision 
based on false assumption that 4.8 CI should pass in our machines because 
all Qt5 modules were passing.

As a latest (hopeless) action we decided to make autotests results 
temporarily insignicant for Windows configurations with Qt 4.8 CI. 
This _should_ now make 4.8 CI passing again. Once we get complete list of 
failing autotests we will go them through one by one and fix them in 
order to get tests significant again.

===
 
Summary about the problems for those interested:

1. Digia hosted environment had newer version of Curys installed 
    on auto test server, which made many network auto tests to fail: 
    https://codereview.qt-project.org/37849 

2. Microsoft DNS server used in Digia hosted Qt-Project CI system did not 
    respond as expected by one of the network autotest: 
    https://codereview.qt-project.org/37853 

3. QHttp autotest failed because expected 200 OK from qt.nokia.com, 
    but the server had redirect activated to qt.digia.com and returned 301:
    https://codereview.qt-project.org/37858 

Fixes for these were implemented and initially run through Nokia hosted 
4.8 CI but unfortunately these builds failed due flaky tests. To avoid these 
failures on Digia hosted system, the following additional change was made
before 4.8 CI Project was moved to Digia hosting:

4. Marked flaky tst_qlocalsocket autotest insignificant: 
    https://codereview.qt-project.org/38137  

Then CI builds and testing proceeded further under Digia hosting and it was
found out that:

5. QEXPECT_FAIL in tst_qaccessibility::textEditTest test started to pass on 
    Digia hosted Ubuntu 10.04 machines: https://codereview.qt-project.org/38221 

6. Digia hosted Windows CI testers did not have OpenSSL installed and 
    tst_qsslsocket failed to compile when OpenSSL was not available: 
    https://codereview.qt-project.org/38263  

Builds/testing got a bit further again and it was found out that also:

7. Network selftest started to fail on Windows because OpenSSL was not 
    installed: https://codereview.qt-project.org/38363 

8. Execution of tst_qthread and tst_qxmlquery timed out after 450 seconds
    in Digia hosted environment: https://codereview.qt-project.org/38363  

Once all of these changes were in Gerrit, approved and staged; the Gerrit 
bug https://bugreports.qt-project.org/browse/QTQAINFRA-509 stepped in and 
prevented 4.8 CI to run during Sunday. The change I6ffa96a was manually 
changed back to integrating state and 4.8 CI restarted this morning.

Then we faced the next error in tst_QHttpNetworkConnection::nossl autotest,
and made a decision to mark all autotests on Windows temporarily insignificant
for 4.8 CI. All these problems compined with quite long turn around time of 
Qt 4.8 CI, made CI to be broken as long as it has been.

Next steps:

OpenSSL will be installed to Digia hosted Windows CI testers soon,
but tst_networkselftest probably need to stay insignificant until the reason
for this is found: https://bugreports.qt-project.org/browse/QTBUG-27571. 
 
The reason for tst_qthread and tst_qxmlquery time out; and flakiness of 
tst_qlocalsocket and failures of other autotests will be investigated and 
tests reverted back to significant when the root cause(s) for failures 
is found and fixed.

Br,
--
Janne Anttila
Senior Architect - Digia, Qt
Visit us on: http://qt.digia.com