[Development] Blacklisting: CI vs non-CI

[Allan Sandfeld Jensen]

On Friday 26 May 2017, Konstantin Tokarev wrote:
> 26.05.2017, 09:45, "Thiago Macieira" <thiago.macieira at intel.com>:
> > I've seen some blacklists, in the past, that were real flaky tests:
> > things that would be unstable on any machine, given the proper
> > conditions.
> > 
> > But most of the recent blacklists, and certainly all the BPASS I see when
> > I run tests, are due to the tests being run in an environment that runs
> > 300x slower than normal. Can we invent a blacklist that only applies to
> > slow machines?
> > 
> > I don't need to see those BPASS on 100% of the time that the test is run
> > on my machine.
> 
> If problem is that tests just don't have enough time to complete in CI
> environment, it would be a good idea to mark such tests as "slow" so they
> get higher timeout settings.
> 
That is not always the issue. Sometimes timings are just so far off the tests 
no longer work. Be it due to the test testing timing specifically or because 
it is doing simulated input. There are almost as many reason for the flakiness 
as there are flaky tests.

Best regards
`Allan