[Development] [Qt5-feedback] 3rdpart platform plugins, tests and CI

[Rohan McGovern]

Holger Freyther said:
> 
> I understand that in the beginning infrastructure will be solely operated
> by Nokia, would it make sense to setup a Jenkins outside of Nokia?

I am not sure.  The only immediate benefits I can see from doing that is
(1) removes one blocker for non-Nokia entities to add test machines with
their own configurations, (2) the full build logs could immediately be
made public.  The disadvantage would be disruption on the existing
codereview/CI setup for all developers.

The build logs issue is tracked by
https://bugreports.qt.nokia.com/browse/QTQAINFRA-139 and should be
solvable with a less drastic solution than an entire switchover of the CI
tool.

> Is there
> a list of what Pulse/Scripts do that Jenkins is not capable of doing right
> now?

No, not right now.  It would make sense to create this.

In our setup, the responsibilities of Pulse/Jenkins/whatever else at
that layer consists of "merely":

  - schedule arbitrary test scripts over a pool of machines
  - gather and store build logs, test results
  - reading the contents of some git repositories
  - run arbitrary pre/post build commands for a few things (e.g.
    automatic reboot of virtual machines after each test run)

I expect these requirements are simple enough to be satisfied by many
tools.  The actual test logic and test machine setups are not
implemented in Pulse at all, they are almost all public and could be
immediately used outside of Nokia's current Pulse setup.

  - http://qt.gitorious.org/qt/qtqa <- the scripts used for testing Qt5
    in CI
  - http://qt.gitorious.org/qtqa/testconfig <- glue between CI tool
    (e.g. Pulse) and the above scripts
  - http://qt.gitorious.org/qtqa/sysadmin <- metadata for configuration
    of test machines

There is some documentation at http://wiki.qt-project.org/index.php/Public_Autotest_Infrastructure
which should be expanded.

> I assume the Qt Project starts off while still using your internal
> and non-free pulse system while an alternative is built?
> 

Yes, as far as the CI system is concerned, the Qt project going live meant
nothing other than the firewall rules blocking access to
codereview.qt-project.org from outside of Nokia being dropped.

> > If they are generic enough to be usable for e.g. "any" DirectFB backend,
> > then it may be appropriate to check them into the source tree, if they
> > are fairly stable.  If they're likely to be unstable, or they are very
> > large, we don't have any good solution at the moment :(
> 
> The question is a general one. How do we have custom results? In this
> specific case the DirectFB software image decoders generate different
> results to Qt, but if I would build a SoC platform and I decide to run
> Qt tests and my hardware is giving a slightly different result i would
> want to override some results? Any idea?
> 
> In case of image results it would make sense to look for 'foo.png'
> in images/foo.png, images/QPA-DirectFB/foo.png,
> images/QPA-DirectFB/HW/foo.png, 

That would be fine, we've used that approach for testdata sometimes in
the past.  However it's hard to decide how much detail needs to go into
the identification of the testdata.
For example, maybe changing the version of system libraries such as
freetype could change the font rendering slightly, so would you then
include the freetype version number in the string as well?

> maybe even have qrc:/ look into an external path?
> 

Not sure about that.