[Development] Blacklisting: CI vs non-CI

[Allan Sandfeld Jensen]

On Friday 26 May 2017, Thiago Macieira wrote:
> I've seen some blacklists, in the past, that were real flaky tests: things
> that would be unstable on any machine, given the proper conditions.
> 
> But most of the recent blacklists, and certainly all the BPASS I see when I
> run tests, are due to the tests being run in an environment that runs 300x
> slower than normal. Can we invent a blacklist that only applies to slow
> machines?
> 
> I don't need to see those BPASS on 100% of the time that the test is run on
> my machine.

What about a flaky list for tests that are allowed to be run multiple times 
until they pass more than 50% of the time? I think we already have something 
like that for integration, but it could be narrowed to specific tests and then 
allowed even more runs and be used outside of integration.

Best regards
`Allan