[Development] Use make before you push and stage

Rohan McGovern rohan.mcgovern at nokia.com
Mon Jan 30 04:15:30 CET 2012


Jedrzej Nowacki said:
> 
>     You are assuming that we need to test every patchest which would be nice, 
> but it is not necessary. A bot-tester would be sufficient, it might be added as 
> a reviewer by human reviewer only if it makes sense.
> 

I like this idea and had been thinking along these lines for a while.
Similar to how all changes at the moment automatically end up with "Qt
Sanity Bot" as a reviewer, a developer could explicitly add other test
bots as reviewers to their change.  This could also include testing
which takes too long to perform in the main CI.

If anyone thinks this is _not_ a good idea then I would be interested to hear
the reasons :)

>     We do not need to test every platform, only the fastest one. The point of 
> this system would be to reduce failures count because of simple mistakes; like 
> broken compilation because missing function. It doesn't have to find problems 
> related to differences in compilers.
> 
>     Another thing is, can it happen in parallel? Do we use 100% of test server 
> capacity? Btw. I do not think that ccache is a hack, I was using it for a few 
> months working on Qt and it was ok.
> 
>     Currently, there is no way to test properly difficult patches without 
> staging them. Only CI knows how to test Qt correctly, which tests are flaky, 
> which tests may be ignored, which may be run in parallel and which needs to be 
> run without touching mouse/keyboard because of focus issues. Parsing output of 
> "make check" is not funny too. Everything that improves that situation even 
> slightly is awesome.
> 

I agree with the main point of this paragraph but would like to correct
a couple of minor things:

"Only CI knows ... which tests may be ignored" => mostly false, the build
system knows this.  CONFIG+=insignificant_test in a .pro file causes
`make check' to ignore the exit code of a testcase, both in CI and when
you run `make check' locally.  The exception is that the CI system can
be configured to ignore _all_ autotests for a particular configuration,
and nothing in the tested repo tells you this.

"Only CI knows ... which may be run in parallel" => none of them are run
in parallel by CI.  If you are thinking of CONFIG+=parallel_test: that
was added for this utility http://labs.qt.nokia.com/2010/10/29/making-auto-testing-easier/
by Yoann Lopes, and as far as I know has never been used by any other
tools.



More information about the Development mailing list