[Development] On the reliability of CI

[Shawn Rutledge]

On Fri, Oct 26, 2012 at 09:45:14AM +1000, Rohan McGovern wrote:
> Personally I think the logical next step, which could be implemented
> in parallel to the current setup without drastic new hardware purchases,
> would be to provide a service where changes in gerrit can be submitted
> for testing separately from the "please merge to stable branch"
> mechanism.  Right now, if you know your change is risky, you have best
> intentions and you would like it tested separately, you have no
> mechanism for doing that.

+1 to that.  Do we own the web UI enough to be able to add a feature like that?

> Extending the Early Warning System (sanity bot and other similar bots)
> to also include compile and autotesting might help, but I have a feeling
> that system could never be extended to cover as many configurations as
> the full CI, which is likely to always have some un-sandboxable or
> un-virtualizable platforms.

Yes.  For example there was the idea that maybe we should still be testing
on big-endian platforms.  So I was thinking of writing another bot (like the 
sanity and doc bots) which would at least build, maybe even test, every patch 
on a spare G5 Mac which we have available for that purpose.  But I haven't 
taken the time to write the script.

The nice thing about bots is they don't completely block integration.  It's an
adjunct: if it decides not to work one day, it's not such a crisis.  So maybe 
we could use more of those and less of the kind of CI testing that does block it, 
to avoid the kind of "log jams" we've seen lately.  We can separate tests into 
sets which take longer to run and are warning-only (and can be done ahead of time
on a "bot" machine), and tests which are required to pass before each integration.  
The UI needs to be able to handle any number of columns of results that come back 
from them.  Currently only the doc bot is able to really -1 a change, so that it 
requires a manual override to submit.  But the other bots should be able to do the 
same.  I heard there's something hard about making that happen though.