[Development] Need advise on acceptable timeouts for autotests

[Olivier Goffart]

On Donnerstag, 16. März 2017 10:00:55 CET Marc Mutz wrote:
> Hi,
> 
> We repeatedly have the problem that timeouts that developers think are ample
> (because they exceed typical runtime by, say, two orders of magnitude) are
> found to be insufficient on the CI.
> 
> Latest example:
> http://testresults.qt.io/coin/integration/qt/qtbase/tasks/1489618366
> 
> The timeout to run update-mime-database was recently increased to 2mins. But
> that still does not seem to be enough. For a call that hardly takes a
> second to run on unloaded machines.
> 
> We can of course crank up timeouts to insane amounts like 1h, but that means
> that a test will just sit there idling for an hour in the worst case.
> 
> I have two questions:
> 
> 1. Where do these huge slowdowns come from? Is the VM live-migrated? Is the
>    SAN, if any, down? At this point it lools like no overcommitting of
> CPU/RAM could ever explain how update-mime-database can take 2mins to run.

In this particular case, it seems there is an actual problem with 
tst_QMimeDatabase,  it is broken in every branches because update-mime-
database does not stops or stops with errors.  Maybe there is a bug in update-
mime-database in this particular machine?

> 2. What should we choose as timeouts? I understand that tests which are
> stuck are killed after some time (how long?). Maybe timeouts should be set
> to the same value?

It should be set to a shorter value

-- 
Olivier

Woboq - Qt services and support - https://woboq.com - https://code.woboq.org