[Development] The qtbase CI should run the qtdeclarative tests

[lars.knoll at nokia.com]

On 3/19/12 2:11 AM, "ext Rohan McGovern" <rohan.mcgovern at nokia.com> wrote:

>marius.storm-olsen at nokia.com said:
>> While it's not normal to do this type of dependency "lock" for upwards
>>dependencies, i think in this case we should indeed do the extra
>>checking. Not just because QtDeclarative is such an important tech for
>>Qt5, but also because it's a very good test case for QtBase.
>> 
>
>What would be the proposed workflow to resolve this situation?
>
>  - I have a change in qtbase which I can't push because it breaks a
>    qtdeclarative test.
>
>  - I have a fix to the test in qtdeclarative but I can't push it
>    because it doesn't pass without my qtbase change.
>
>One solution I can think of is to introduce a %reverse_dependencies
>in sync.profile, used for CI and nothing else, which can control the
>SHA1 of qtdeclarative used while testing qtbase.

The easiest way is probably to do it in 3 stages:
* Pin the SHA1 of qtbase in declarative
* Submit the qtbase change
* Unpin the sha1 and submit the declarative fix.

>
>Another is to skip the testing of qtdeclarative in qtbase
><branchname>'s CI if qtdeclarative's sync.profile does not contain
>"qtbase => refs/heads/<branchname>".
>
>> The extra time used for checking both repos will give much greater
>>stability as a whole.
>> 
>
>Well, we don't really know that.  Note that introducing qtdeclarative
>into the qtbase tests means that both directions of breakage are now
>possible (i.e. not only can qtbase changes block qtdeclarative's CI,
>but now qtdeclarative changes will be able to block qtbase's CI - though
>it should be unlikely).
>
>However it's certainly reasonable to expect that, and give it a try.
>
>> Maybe the fast platforms can do the extra checking, while slow
>>platforms still handle only QtBase? At least we will capture most
>>compile issues?
>
>I think we'd start with adding new configurations rather than modifying
>existing ones.  e.g. as there is currently a "linux-g++-32 Ubuntu 10.04
>x86" configuration for qtbase, there could be a new "linux-g++-32 Ubuntu
>10.04 x86 with declarative" configuration.
>
>Doing it that way also means the "with declarative" configuration can
>run just the qtdeclarative autotests (we assert that there's no need to
>run the qtbase autotests again), theoretically not increasing the
>overall runtime.
>
>Unfortunately there's probably not enough resources to do that for some
>platforms (like Mac).

If we can do it in a new configuration that only tests declarative that'll
help the total runtime of the tests. Not having it for some platforms is
IMO acceptable. No CI system will ever be perfect :)
>
>> (The CI rules might get very complicated though.)
>
>Yes, in some ways I feel this is adding complexity to the test setup to
>work around a "false simplicity" in our source code setup.  We claim
>that Qt is modular, but actually we know some parts of it are not really,
>so we add gates to enforce some level of de-modularization.

The main problem here is that our test coverage of qtbase in itself is not
good enough in some areas. So we have to cover this up by adding
declarative tests in thus implicitly raising test coverage of qtbase.
>
>> 
>> Rohan/Sergio, what will be the turn-around increase if we always test
>>Declarative together with QtBase?
>> 
>
>The runtime should be somewhere between the current runtime of qtbase,
>and the qtbase runtime plus the qtdeclarative runtime.  Anything more
>would be speculation ... but I don't think the runtime would be a big
>issue :)
>
>Anyway, overall I think it's worth a try, but we should accept that this
>could also create some new problems as well as solving some existing
>ones.

Let's give it a try and see how it goes. Given the submit policies we now
have in place this should work most of the times, and we'll simply need to
deal with exceptions (where we need a change in both base and declarative)
manually. I don't think this should happen too often anymore.

Cheers,
Lars