[Development] Use make before you push and stage

[Jedrzej Nowacki]

On Wednesday 25. January 2012 20.14.41 ext marius.storm-olsen at nokia.com wrote:
> On 1/25/12 11:32 AM, "ext
> morten.sorvig at nokia.com<mailto:morten.sorvig at nokia.com>"
> <morten.sorvig at nokia.com<mailto:morten.sorvig at nokia.com>> wrote: On Jan
> 25, 2012, at 10:47 AM, ext
> jiang.jiang at nokia.com<mailto:jiang.jiang at nokia.com> wrote:
> 
> For somewhat big changes and refactor changes, it usually took many
> iterations of compile/testing until they finally went in, because it's not
> always possible for the developers to try all the different configurations
> on their development machine. Would it make sense to setup a semi-final
> staging area to test that kind of changes before we finally cherry-pick
> them to master? It should make smaller changes integration smoother.
> 
> I like the idea, but let's not make the gerrit interface more complicated.
> Instead the CI system could test all changes that are uploaded, in
> parallel (and perhaps at off-peak hours to better use available capasity).
> 
> While that it would be awesome to have a system which could compile test
> each uploaded patch set, and preferably have the results available before
> developers would review them, I think it something that will not scale; at
> least not without a system which can guarantee proper incremental builds
> (ccache would be a hack, still waste lots of resources, and only works
> with GCC.)
> 
> Lets take a look at the current numbers:
> 
> In the last week (now-7 days) we have across all projects merged 592
> uploaded patch sets. Lets assume that we stage 3 patch sets per
> integration round, and that they are all successful on the first try. That
> would give us 197 rounds in the CI system. A round in the CI system
> currently takes ~3h, although that does include running auto tests of
> course.
> 
> In the same time period, we uploaded 1461 patch sets, since most review
> tasks go through several revisions. To properly evaluate each uploaded
> patch set, we would need 4383h of processing time, equal to 182.63 days if
> processed in order. To scale it up to manage that in parallel within 1
> day, we'd need 182 more Mac machines, Linux boxes and Windows machines;
> and that's at the current contribution rate :)
> 
> Of course, there are things we can do to streamline it more, and make such
> a system more likely. So, first we drop running auto tests, just simply
> compile the code. We'd perhaps be down to ~1h on VMs; measured by the
> slowest platform in VMs. Let's also drop webkit in the compile, and
> perhaps reduce the compile time be down to 20min? Then we're down to 20.3
> days of compilation.
> 
> Could we get there? Perhaps. But we would need quite a few compromising
> shortcuts to get there, I think :)
> 
> --
> .marius

Hi,

    You are assuming that we need to test every patchest which would be nice, 
but it is not necessary. A bot-tester would be sufficient, it might be added as 
a reviewer by human reviewer only if it makes sense.

    We do not need to test every platform, only the fastest one. The point of 
this system would be to reduce failures count because of simple mistakes; like 
broken compilation because missing function. It doesn't have to find problems 
related to differences in compilers.

    Another thing is, can it happen in parallel? Do we use 100% of test server 
capacity? Btw. I do not think that ccache is a hack, I was using it for a few 
months working on Qt and it was ok.

    Currently, there is no way to test properly difficult patches without 
staging them. Only CI knows how to test Qt correctly, which tests are flaky, 
which tests may be ignored, which may be run in parallel and which needs to be 
run without touching mouse/keyboard because of focus issues. Parsing output of 
"make check" is not funny too. Everything that improves that situation even 
slightly is awesome.

Cheers,
  Jędrek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20120126/54232f89/attachment.html>