[Development] The qtbase CI should run the qtdeclarative tests

[kent.hansen at nokia.com]

Hi,
Again the qtdeclarative CI has been blocked an entire day because of qtbase changes.

It's great that the CI catches all these issues, but at the same time it's ridiculous how much time is spent suffering through failed qtdeclarative CI runs and fixing those failures up after the fact!

If the turn-around time will increase by one hour or whatever by also testing qtdeclarative as part of the qtbase CI runs, so what? The increased stability and confidence in the results should easily make it a net win. No longer will there be a need to wonder whether a failed qtdeclarative CI run was due to one of the actual staged changed, or a recent qtbase change, and dreading the aftermath that comes from that.

The current option of pinning the qtbase SHA1 used by qtdeclarative to some older, known good SHA1 _after_ a breaking qtbase change has gone through, is bad. BAD! Don't ask me to write a paper called "Pinning of SHA1s considered harmful", because I will.

Pinning just masks the failure. C'mon, after first locating a "good" qtbase SHA1 and being able to commit your own changes again, wouldn't you ideally want to go on with your own work instead of spending the rest of the day chasing down some unknown change in qtbase, contacting the author (if possible), waiting for the fix/revert, and babysitting that through the CI? But someone certainly needs to fix it, and while that's ongoing, new changes are blissfully making their way into qtbase, which means new qtdeclarative failures can appear, and it gets progressively harder to get out of the mess. (Happened twice this week.)

Marking qtdeclarative tests as insignificant due to qtbase-introduced failures, just to get stuff through the CI again, is also a bit ho-humm. Who follows up that backlog of ignored tests?

Yes, there are some qtbase changes that require qtdeclarative to be adapted (AKA "intentional breakages"), and then you need to run the qtbase CI in isolation. But that should be the exception, not the rule. It's in those cases one should first pin the qtbase SHA1 in qtdeclarative before staging the qtbase change, and afterwards adapt qtdeclarative (with a change already prepared and reviewed) and unpin the qtbase SHA1 at once. Takes away all the surprise.

Hey, I know, we can just move qtdeclarative into qtbase. Bring back the -no-declarative configure option and we're golden.

With best wishes for the weekend,
Kent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20120316/13682cc5/attachment.html>